github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/datasette/pull/1967#issuecomment-1368265354 | https://api.github.com/repos/simonw/datasette/issues/1967 | 1368265354 | IC_kwDOBm6k_c5RjhKK | 9599 | 2022-12-31T18:56:35Z | 2022-12-31T18:56:35Z | OWNER | Thanks for this! Datasette has a favicon in this file: https://github.com/simonw/datasette/blob/2b4a8aa0833068da25072b6247bf349949ca00ee/datasette/static/favicon.png - so I'll copy that over and use it here as well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1503010009 | |
https://github.com/simonw/datasette/issues/1970#issuecomment-1368264882 | https://api.github.com/repos/simonw/datasette/issues/1970 | 1368264882 | IC_kwDOBm6k_c5RjhCy | 9599 | 2022-12-31T18:52:01Z | 2022-12-31T18:52:01Z | OWNER | Relevant code: https://github.com/simonw/datasette/blob/e03aed00026cc2e59c09ca41f69a247e1a85cc89/datasette/app.py#L418-L424 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1515182998 | |
https://github.com/simonw/datasette/issues/1614#issuecomment-1364345119 | https://api.github.com/repos/simonw/datasette/issues/1614 | 1364345119 | IC_kwDOBm6k_c5RUkEf | 536941 | 2022-12-23T21:27:10Z | 2022-12-23T21:27:10Z | CONTRIBUTOR | is this issue closed by #1893? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1115435536 | |
https://github.com/simonw/datasette/issues/1796#issuecomment-1364345071 | https://api.github.com/repos/simonw/datasette/issues/1796 | 1364345071 | IC_kwDOBm6k_c5RUkDv | 536941 | 2022-12-23T21:27:02Z | 2022-12-23T21:27:02Z | CONTRIBUTOR | @simonw is this issue closed by #1893? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1355148385 | |
https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1364141224 | https://api.github.com/repos/simonw/sqlite-utils/issues/488 | 1364141224 | IC_kwDOCGYnMM5RTySo | 1176293 | 2022-12-23T17:38:55Z | 2022-12-23T17:38:55Z | NONE | > text columns containing empty strings should not be rewritten to null. I would actually appreciate an option to do just that for text columns as well. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1373224657 | |
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-1359468823 | https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46 | 1359468823 | IC_kwDODFdgUs5RB9kX | 1839645 | 2022-12-20T14:39:39Z | 2022-12-20T14:40:15Z | NONE | Just a quick +1 to this one from me - I would like to do a better job of tracking who is reviewing one another's pull requests in repositories, since this is a specific kind of maintenance work that I think often goes unrewarded. I can't seem to figure this out just by looking at the `pull_request` or `issue_comments` tables, so I think it would be helpful to support PR reviews natively (even if just for summary statistics). Alternatively if there is a way in the API to tell if an issue comment is part of a review, then perhaps you could quickly calculate the number of unique reviews that an author performed. But that was beyond my SQL-foo :-) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
664485022 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1357084279 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1357084279 | IC_kwDOBm6k_c5Q43Z3 | 178162 | 2022-12-19T04:34:16Z | 2022-12-19T04:34:16Z | NONE | You were super-close on the python version of the test here, changing `http` to `https` on 8b73fc6b47dffd8836f5c58aae1e57c1f66a5754 is enough to pass the test: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b4a3f..ba07a11d37f6 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -207,7 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds("http://localhost:8042/", verify=client_cert) + wait_until_responds("https://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert ``` My speculation about what was happening here: when `wait_until_responds()` would time out due to SSL connection problems, because `.terminate()` isn’t in a `finally`, the datasette process wouldn’t get killed. That could (1) hang CI and (2) cause all your future local test runs to mysteriously fail because they’d be secretly talking to that old datasette process still hanging around from a past test run with an old temporary server certificate, and that old server cert wouldn’t validate against your newly-created ca cert. A `finally` for `.terminate()` would help; a fancier version could be a context manager for running the external `datasette` process that could: - ensure the process always exited when no longer needed - if you want to be fancy, call `terminate()`, `wait()` for a short timeout for the process to exit, then try `kill()` and `wait()` again; raise an exception complaining about the seemingly-unkillable process if all that fails - raise an error if the process exited with a non-zero error code; here it’s likely that some `datasette` executions were secretly failing with `[Errno 48] error while attempting to bind on address ('127.0.0.1', 8042): address already in use` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1886#issuecomment-1356842576 | https://api.github.com/repos/simonw/datasette/issues/1886 | 1356842576 | IC_kwDOBm6k_c5Q38ZQ | 18738650 | 2022-12-18T17:34:20Z | 2022-12-18T17:34:20Z | NONE | A bit late to this, but I have made an app to publish air quality data in Bristol, UK. [air quality data in Bristol, UK.](https://brisaq-wfzqhmj43q-ew.a.run.app/) Next step to see if I can make a streamlit app based on this to produce some nice charts. | { "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1447050738 | |
https://github.com/simonw/datasette/pull/1965#issuecomment-1356827218 | https://api.github.com/repos/simonw/datasette/issues/1965 | 1356827218 | IC_kwDOBm6k_c5Q34pS | 9599 | 2022-12-18T16:01:36Z | 2022-12-18T16:01:36Z | OWNER | Will link to this from my TIL shortly. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1501843596 | |
https://github.com/simonw/datasette/pull/1965#issuecomment-1356827167 | https://api.github.com/repos/simonw/datasette/issues/1965 | 1356827167 | IC_kwDOBm6k_c5Q34of | 9599 | 2022-12-18T16:01:22Z | 2022-12-18T16:01:22Z | OWNER | This is great, thank you! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1501843596 | |
https://github.com/simonw/datasette/issues/1964#issuecomment-1356697705 | https://api.github.com/repos/simonw/datasette/issues/1964 | 1356697705 | IC_kwDOBm6k_c5Q3ZBp | 9599 | 2022-12-18T06:37:23Z | 2022-12-18T06:37:23Z | OWNER | I'm certain the two other cog menus (the app menu on the right of the nav bar and the column action menus) have the same problem. Would be great to figure out the right ARIA attributes for these too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1501778647 | |
https://github.com/simonw/datasette/issues/1771#issuecomment-1356694671 | https://api.github.com/repos/simonw/datasette/issues/1771 | 1356694671 | IC_kwDOBm6k_c5Q3YSP | 9599 | 2022-12-18T06:34:20Z | 2022-12-18T06:34:20Z | OWNER | Now live on https://latest.datasette.io/fixtures/attraction_characteristic | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1306984363 | |
https://github.com/simonw/datasette/issues/1771#issuecomment-1356680769 | https://api.github.com/repos/simonw/datasette/issues/1771 | 1356680769 | IC_kwDOBm6k_c5Q3U5B | 9599 | 2022-12-18T05:56:05Z | 2022-12-18T05:56:05Z | OWNER | This does the trick: ```css div.select-wrapper:focus-within { border: 1px solid black; } ``` ![tab-select-border-fix](https://user-images.githubusercontent.com/9599/208283826-de48212f-a213-40fc-9b37-9d66f0858f21.gif) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1306984363 | |
https://github.com/simonw/datasette/issues/1771#issuecomment-1356657451 | https://api.github.com/repos/simonw/datasette/issues/1771 | 1356657451 | IC_kwDOBm6k_c5Q3PMr | 1473102 | 2022-12-18T04:04:32Z | 2022-12-18T04:04:32Z | NONE | the problem is: ``` .select-wrapper select:focus { outline: none; } ``` I sometimes add this js: ``` window.addEventListener('keydown', function check_tab(e) { if (e.key === 'Tab') { document.documentElement.classList.add('user-is-tabbing') window.removeEventListener('keydown', check_tab) } }) ``` and then in the css, using a `html.user-is-tabbing` selector undo any outlines I removed. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1306984363 | |
https://github.com/simonw/datasette/issues/1771#issuecomment-1356655630 | https://api.github.com/repos/simonw/datasette/issues/1771 | 1356655630 | IC_kwDOBm6k_c5Q3OwO | 9599 | 2022-12-18T03:43:12Z | 2022-12-18T03:43:12Z | OWNER | The border is actually on the div that wraps the select box: <img width="1171" alt="image" src="https://user-images.githubusercontent.com/9599/208280249-9d9848b5-6e33-46c3-ad99-bc1948e08f25.png"> I tried adding a `border: 1px dotted black` to `select:focus` but it's not quite right - it jumps around a bit like this: ![Tabbing to the selects shows a 1px border but the element expands in size by one pixel, causing a visual jump](https://user-images.githubusercontent.com/9599/208280271-41a07f68-b8b1-4908-a4e2-aac4304d6c09.gif) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1306984363 | |
https://github.com/simonw/datasette/issues/1771#issuecomment-1356655217 | https://api.github.com/repos/simonw/datasette/issues/1771 | 1356655217 | IC_kwDOBm6k_c5Q3Opx | 9599 | 2022-12-18T03:38:16Z | 2022-12-18T03:38:16Z | OWNER | OK I see what you mean: https://latest.datasette.io/fixtures/attraction_characteristic ![Animated GIF of the table page hitting tab a bunch - the cog icon highlights and so does the text input but the two select boxes do not](https://user-images.githubusercontent.com/9599/208280176-1e2de671-fe69-43e8-8d62-bf7aa8f4d36e.gif) | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1306984363 | |
https://github.com/simonw/datasette/issues/1963#issuecomment-1356652057 | https://api.github.com/repos/simonw/datasette/issues/1963 | 1356652057 | IC_kwDOBm6k_c5Q3N4Z | 9599 | 2022-12-18T03:23:22Z | 2022-12-18T03:23:22Z | OWNER | https://pypi.org/project/datasette/0.63.3/ is released. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1501713288 | |
https://github.com/simonw/datasette/issues/1963#issuecomment-1356651943 | https://api.github.com/repos/simonw/datasette/issues/1963 | 1356651943 | IC_kwDOBm6k_c5Q3N2n | 9599 | 2022-12-18T03:23:03Z | 2022-12-18T03:23:03Z | OWNER | Oh that's annoying... every step in publish succeeded except the static docs one: <img width="1305" alt="image" src="https://user-images.githubusercontent.com/9599/208279782-8582c130-820c-48c5-a4ca-a77438570b02.png"> https://github.com/simonw/datasette/actions/runs/3723015082/jobs/6314292722 This means the documentation database used to update the search engine on https://datasette.io/ won't reflect the very latest changelog. I'm OK with that - I'll fix this workflow so that next time I publish a release this will work correctly. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1501713288 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356640463 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356640463 | IC_kwDOBm6k_c5Q3LDP | 9599 | 2022-12-18T02:45:18Z | 2022-12-18T02:45:18Z | OWNER | ... and with this change, the following now works correctly: ``` % datasette install datasette-gunicorn % datasette gunicorn fixtures.db -p 8855 [2022-12-17 18:44:29 -0800] [7651] [INFO] Starting gunicorn 20.1.0 [2022-12-17 18:44:29 -0800] [7651] [INFO] Listening at: http://127.0.0.1:8855 (7651) [2022-12-17 18:44:29 -0800] [7651] [INFO] Using worker: uvicorn.workers.UvicornWorker [2022-12-17 18:44:29 -0800] [7653] [INFO] Booting worker with pid: 7653 [2022-12-17 18:44:29 -0800] [7653] [INFO] Started server process [7653] [2022-12-17 18:44:29 -0800] [7653] [INFO] Waiting for application startup. [2022-12-17 18:44:29 -0800] [7653] [INFO] Application startup complete. ``` So this issue is now fixed! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356640266 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356640266 | IC_kwDOBm6k_c5Q3LAK | 9599 | 2022-12-18T02:43:00Z | 2022-12-18T02:43:00Z | OWNER | https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI: ``` Generated a certificate for 'localhost', '127.0.0.1', '::1' Configure your server to use the following files: cert=/home/runner/work/datasette/datasette/server.pem key=/home/runner/work/datasette/datasette/server.key Configure your client to use the following files: cert=/home/runner/work/datasette/datasette/client.pem INFO: Started server process [4036] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed INFO: 127.0.0.1:56726 - "GET /_memory.json HTTP/1.1" 200 OK 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 213 0 213 0 0 11542 0 --:--:-- --:--:-- --:--:-- 11833 INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4036] {"database": "_memory", "private": false, "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": true, "table_columns": {}, "query_ms": 1.4545189999921604}0 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1221#issuecomment-1356639873 | https://api.github.com/repos/simonw/datasette/issues/1221 | 1356639873 | IC_kwDOBm6k_c5Q3K6B | 9599 | 2022-12-18T02:39:04Z | 2022-12-18T02:39:04Z | OWNER | I ended up moving this test out of Python and into a `bash` script here: https://github.com/simonw/datasette/commit/d1d369456a7319b9de39175605568cbc9b852478 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
806849424 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356633937 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356633937 | IC_kwDOBm6k_c5Q3JdR | 9599 | 2022-12-18T02:28:22Z | 2022-12-18T02:28:22Z | OWNER | This `bash` script does the job: ```bash #!/bin/bash # Generate certificates python -m trustme # This creates server.pem, server.key, client.pem # Start the server in the background datasette --memory \ --ssl-keyfile=server.key \ --ssl-certfile=server.pem \ -p 8152 & # Store the background process ID in a variable server_pid=$! # Wait for the server to start sleep 2 # Make a test request using curl curl -f --cacert client.pem 'https://localhost:8152/_memory.json' # Save curl's exit code (-f option causes it to return one on HTTP errors) curl_exit_code=$? # Shut down the server kill $server_pid sleep 1 # Clean up the certificates rm server.pem server.key client.pem echo $curl_exit_code exit $curl_exit_code ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356630092 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356630092 | IC_kwDOBm6k_c5Q3IhM | 9599 | 2022-12-18T02:20:01Z | 2022-12-18T02:20:01Z | OWNER | Asked ChatGPT: > Write a bash script which starts a server in the background using "datasette -p 8002", then uses curl to make a test request against it, then shuts the server down again at the end It gave me: ```bash #!/bin/bash # Start the server in the background datasette -p 8002 & # Store the background process ID in a variable server_pid=$! # Make a test request using curl curl http://localhost:8002 # Shut down the server kill $server_pid ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356629783 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356629783 | IC_kwDOBm6k_c5Q3IcX | 9599 | 2022-12-18T02:18:43Z | 2022-12-18T02:18:43Z | OWNER | Various attempts at a fix which didn't work: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b..899d36fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -6,6 +5,7 @@ import pytest import pytest_asyncio import re import subprocess +import sys import tempfile import time import trustme @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None -def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): +def wait_until_responds(url, timeout=5.0, client=None, **kwargs): + client = client or httpx.Client(**kwargs) start = time.time() while time.time() - start < timeout: try: - client.get(url, **kwargs) + if "verify" in kwargs: + print(kwargs["verify"]) + print( + "Contents of verify file: {}".format( + open(kwargs.get("verify")).read() + ) + ) + print("client = {}, kwargs = {}".format(client, kwargs)) + client.get(url) return - except httpx.ConnectError: + except (httpx.ConnectError, httpx.RemoteProtocolError) as ex: + print(ex) time.sleep(0.1) raise AssertionError("Timed out waiting for {} to respond".format(url)) @@ -166,7 +176,7 @@ def check_permission_actions_are_documented(): @pytest.fixture(scope="session") def ds_localhost_http_server(): ds_proc = subprocess.Popen( - ["datasette", "--memory", "-p", "8041"], + [sys.executable, "-m", "datasette", "--memory", "-p", "8041"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # Avoid FileNotFoundError: [Errno 2] No such file or directory: @@ -180,7 +190,7 @@ def ds_localhost_http_server(): ds_proc.terminate() -@pytest.fixture(scope="session") +@pytest.fixture def ds_localhost_https_serve… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356627931 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356627931 | IC_kwDOBm6k_c5Q3H_b | 9599 | 2022-12-18T02:13:01Z | 2022-12-18T02:13:01Z | OWNER | Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356627331 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356627331 | IC_kwDOBm6k_c5Q3H2D | 9599 | 2022-12-18T02:11:17Z | 2022-12-18T02:11:17Z | OWNER | This issue might be relevant, but I tried the suggested fix in there (`Connection: close` on the incoming requests) and it didn't fix my problem: - https://github.com/encode/httpx/discussions/2056 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356626334 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356626334 | IC_kwDOBm6k_c5Q3Hme | 9599 | 2022-12-18T02:04:01Z | 2022-12-18T02:04:07Z | OWNER | I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052 In one terminal: ``` cd /tmp python -m trustme datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem -p 8003 ``` Then in another terminal: ``` curl --cacert /tmp/client.pem 'https://localhost:8003/_memory.json' ``` This worked correctly, outputting the expected JSON. So the feature still works, it's just the test that is broken for some reason. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356625642 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356625642 | IC_kwDOBm6k_c5Q3Hbq | 9599 | 2022-12-18T02:00:57Z | 2022-12-18T02:00:57Z | OWNER | I added the TLS support here: - https://github.com/simonw/datasette/issues/1221 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356625556 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356625556 | IC_kwDOBm6k_c5Q3HaU | 9599 | 2022-12-18T02:00:18Z | 2022-12-18T02:00:18Z | OWNER | Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356620233 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356620233 | IC_kwDOBm6k_c5Q3GHJ | 9599 | 2022-12-18T01:31:10Z | 2022-12-18T01:31:10Z | OWNER | During the polling loop it constantly raises: `httpx.RemoteProtocolError`: Server disconnected without sending a response | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356618913 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356618913 | IC_kwDOBm6k_c5Q3Fyh | 9599 | 2022-12-18T01:29:05Z | 2022-12-18T01:29:05Z | OWNER | Now the only failure is in the `https` test - which fails like this (in CI and on my laptop): ``` message = str(exc) > raise mapped_exc(message) from exc E httpx.RemoteProtocolError: Server disconnected without sending a response. /opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError =========================== short test summary info ============================ ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response. ================= 30 passed, 1264 deselected, 1 error in 6.15s ================= ``` That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24 And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356610089 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356610089 | IC_kwDOBm6k_c5Q3Dop | 9599 | 2022-12-18T01:12:39Z | 2022-12-18T01:12:39Z | OWNER | ... and it turns out those tests saved me. Because I forgot to check if `datasette` would actually start a server correctly! ``` % datasette fixtures.db -p 8852 INFO: Started server process [3538] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 86, in main await app(scope, self.receive, self.send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/asgi.py", line 437, in __call__ return await self.asgi(scope, receive, send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/asgi_csrf.py", line 39, in app_wrapped_with_csrf await app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1457, in __call__ path = scope["path"] KeyError: 'path' ERROR: Application startup failed. Exiting. ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356609095 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356609095 | IC_kwDOBm6k_c5Q3DZH | 9599 | 2022-12-18T01:10:43Z | 2022-12-18T01:10:43Z | OWNER | Improved version of that fixture: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 44c44f87..69dee68b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None +def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs): + start = time.time() + while time.time() - start < timeout: + try: + client.get(url, **kwargs) + return + except httpx.ConnectError: + time.sleep(0.1) + raise AssertionError("Timed out waiting for {} to respond".format(url)) + + @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette @@ -161,13 +172,7 @@ def ds_localhost_http_server(): # Avoid FileNotFoundError: [Errno 2] No such file or directory: cwd=tempfile.gettempdir(), ) - # Loop until port 8041 serves traffic - while True: - try: - httpx.get("http://localhost:8041/") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8041/") # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - while True: - try: - httpx.get("https://localhost:8042/", verify=client_cert) - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory): # Poll until available transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) - while T… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356600917 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356600917 | IC_kwDOBm6k_c5Q3BZV | 9599 | 2022-12-18T01:02:26Z | 2022-12-18T01:02:26Z | OWNER | This bit here looks like it could hang! ```python # Loop until port 8041 serves traffic while True: try: httpx.get("http://localhost:8041/") break except httpx.ConnectError: time.sleep(0.1) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356599930 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356599930 | IC_kwDOBm6k_c5Q3BJ6 | 9599 | 2022-12-18T01:01:47Z | 2022-12-18T01:01:47Z | OWNER | I think that's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L6-L13 Using this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L155-L175 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356596740 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356596740 | IC_kwDOBm6k_c5Q3AYE | 9599 | 2022-12-18T00:59:47Z | 2022-12-18T00:59:47Z | OWNER | Hitting `Ctrl+C` while using `--full-trace` gave me more clues: ``` % pytest -m serial tests/test_cli_serve_server.py --full-trace ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 3 items tests/test_cli_serve_server.py ^C^C ====================================================== no tests ran in 3.49s ======================================================= Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp sock = socket.create_connection( File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused [...] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356595665 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356595665 | IC_kwDOBm6k_c5Q3AHR | 9599 | 2022-12-18T00:58:16Z | 2022-12-18T00:58:16Z | OWNER | `pytest -m serial` on my Mac laptop also freezes: ``` (datasette) datasette % pytest -m serial ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items / 1264 deselected / 31 selected tests/test_package.py . [ 3%] tests/test_cli_serve_server.py ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356489200 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356489200 | IC_kwDOBm6k_c5Q2mHw | 9599 | 2022-12-17T22:29:51Z | 2022-12-17T22:29:51Z | OWNER | No, it still causes the tests to hang (I let them run for 12 minutes): <img width="664" alt="image" src="https://user-images.githubusercontent.com/9599/208268223-7fb40e3f-833b-4204-9acb-079adce6452f.png"> Interesting that the regular tests passed an then the `pytest -m serial` ones seem to have failed. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1356487139 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1356487139 | IC_kwDOBm6k_c5Q2lnj | 9599 | 2022-12-17T22:16:52Z | 2022-12-17T22:16:52Z | OWNER | I'm trying this fix again, after a bunch of work on the test suite in: - #1959 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1356481595 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1356481595 | IC_kwDOBm6k_c5Q2kQ7 | 9599 | 2022-12-17T22:02:40Z | 2022-12-17T22:02:40Z | OWNER | This is interesting: ```python import resource print(resource.getrlimit(resource.RLIMIT_NOFILE)) ``` On Mac: ```pycon >>> print(resource.getrlimit(resource.RLIMIT_NOFILE)) (256, 9223372036854775807) ``` On Ubuntu (in a Docker `ubuntu:22.04` container): ```pycon >>> resource.getrlimit(resource.RLIMIT_NOFILE) (1048576, 1048576) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1356480256 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1356480256 | IC_kwDOBm6k_c5Q2j8A | 9599 | 2022-12-17T22:01:13Z | 2022-12-17T22:01:13Z | OWNER | The refactor in the following issue did NOT prevent this error from occurring when I try to run the full `pytest` suite on my Mac laptop: - #1959 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1356478792 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1356478792 | IC_kwDOBm6k_c5Q2jlI | 9599 | 2022-12-17T21:49:36Z | 2022-12-17T21:49:36Z | OWNER | Made a really good start on this in the just-merged PR: - #1960 The follow-up work will happen in: - #1962 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1356478575 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1356478575 | IC_kwDOBm6k_c5Q2jhv | 9599 | 2022-12-17T21:47:48Z | 2022-12-17T21:47:48Z | OWNER | Stick a twig in it, this will do for the moment. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354062939 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354062939 | IC_kwDOBm6k_c5QtVxb | 22429695 | 2022-12-16T01:48:06Z | 2022-12-17T21:40:43Z | NONE | # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.19**% // Head: **92.05**% // Decreases project coverage by **`-0.13%`** :warning: > Coverage data is based on head [(`770879a`)](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`0b68996`)](https://codecov.io/gh/simonw/datasette/commit/0b68996cc511b3a801f0cd0157bd66332d75f46f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered. > :exclamation: Current head 770879a differs from pull request most recent head f42bca8. Consider uploading reports for the commit f42bca8 to get more accurate results <details><summary>Additional details and impacted files</summary> ```diff @@ Coverage Diff @@ ## main #1960 +/- ## ========================================== - Coverage 92.19% 92.05% -0.14% ========================================== Files 38 38 Lines 5521 5527 +6 ========================================== - Hits 5090 5088 -2 - Misses 431 439 +8 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.13% <100.00%> (-0.57%)` | :arrow_down: | | [datasette/utils/testing.py](https://codecov.io/gh/simonw/… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1356476886 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1356476886 | IC_kwDOBm6k_c5Q2jHW | 9599 | 2022-12-17T21:37:05Z | 2022-12-17T21:37:05Z | OWNER | I think this test may be to blame: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/test_plugins.py#L950-L972 It's over-riding `_metadata_local` and then failing to set it back to original in a `finally:` block at the end. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1356476583 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1356476583 | IC_kwDOBm6k_c5Q2jCn | 9599 | 2022-12-17T21:34:51Z | 2022-12-17T21:34:51Z | OWNER | These are all the places that tests touch `_metadata_local` at the moment: ``` (venv) root@76a81d2417f5:/tmp/datasette/tests# rg _metadata_local test_facets.py 596: ds._metadata_local = { 605: ds._metadata_local["databases"]["test_facet_size"]["tables"]["neighbourhoods"][ test_permissions.py 62: padlock_client.ds._metadata_local["allow"] = allow 77: del padlock_client.ds._metadata_local["allow"] 522: cascade_app_client.ds._metadata_local = updated_metadata 533: cascade_app_client.ds._metadata_local = previous_metadata 549: previous_metadata = cascade_app_client.ds._metadata_local 551: cascade_app_client.ds._metadata_local = metadata 566: cascade_app_client.ds._metadata_local = previous_metadata 842: perms_ds._metadata_local = updated_metadata 849: perms_ds._metadata_local = previous_metadata test_html.py 1114: orig = ds_client.ds._metadata_local 1115: ds_client.ds._metadata_local = metadata 1123: ds_client.ds._metadata_local = orig test_plugins.py 1034: ds_client.ds._metadata_local = { ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1356056018 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1356056018 | IC_kwDOBm6k_c5Q08XS | 9599 | 2022-12-17T05:33:26Z | 2022-12-17T05:33:26Z | OWNER | I think I've found the problem. The failing test is this one: `paginated_view.json-201-9` That's this example: https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L179-L180 Why is it expected to take 9 pages and not 5, when the definition of that view is this: ```sql CREATE VIEW paginated_view AS SELECT content, '- ' || content || ' -' AS content_extra FROM no_primary_key; ``` Because `paginated_view` has extra configuration in `metadata.json`: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/fixtures.py#L357 So this test is showing that `metadata` can be used to set an alternative page size for a view. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1356038242 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1356038242 | IC_kwDOBm6k_c5Q04Bi | 9599 | 2022-12-17T04:56:22Z | 2022-12-17T04:57:04Z | OWNER | May have spotted the problem with that `test_paginate_tables_and_views` test: ``` (Pdb) path, expected_rows, expected_pages (None, 201, 9) ``` `path` should not be `None` here: https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L175-L212 No that's not it either - `path` is reassigned on purpose. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355823260 | IC_kwDOBm6k_c5Q0Dic | 9599 | 2022-12-16T23:36:07Z | 2022-12-16T23:36:07Z | OWNER | I ran `pytest --pdb -x` to drop into the debugger on the first failing test. ``` assert expected_rows == len(fetched) > assert expected_pages == count E assert 9 == 5 ``` That's in `test_paginate_tables_and_views()`. On a hunch, I checked the current settings: ``` (Pdb) ds_client.ds._settings {'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'} ``` Looks like something changed `default_page_size` to 50 and forgot to change it back! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595 | https://api.github.com/repos/simonw/datasette/issues/1962 | 1355691595 | IC_kwDOBm6k_c5QzjZL | 9599 | 2022-12-16T21:53:45Z | 2022-12-16T21:55:29Z | OWNER | Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/datasette/utils/testing.py#L90-L103 One option would be adding that mechanism to `datasette.client.post(...)` - as a `_csrf_from=` parameter (with an underscore because it's mainly intended for use in tests, though perhaps that's a weird convention that I shouldn't introduce). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1500636982 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355685828 | IC_kwDOBm6k_c5Qzh_E | 9599 | 2022-12-16T21:50:01Z | 2022-12-16T21:50:01Z | OWNER | Looks like that `@pytest.mark.ds_client` mark I've been using isn't necessary - I added that so I could easily run `pytest -m ds_client` to execute all tests that I had ported to the new feature, but actually this achieves the same thing: pytest -k ds_client So I'm going to remove the mark. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355478743 | IC_kwDOBm6k_c5QyvbX | 9599 | 2022-12-16T19:27:12Z | 2022-12-16T19:27:12Z | OWNER | Bad news: they're definitely caused by tests that are subtly affected by other tests. This passes without errors: pytest -k test_paginate_tables_and_views | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355475671 | IC_kwDOBm6k_c5QyurX | 9599 | 2022-12-16T19:26:08Z | 2022-12-16T19:26:08Z | OWNER | Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI: ``` ======== short test summary info ======== FAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5 FAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error' FAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']] FAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance' FAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip' FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...>1</a></td>']] FAILED tests/test_table_html.py::test_view_html - assert <a href="/fixtures/simple_view?_size=3&_sort=upper_content" rel="nofollow">upper_content</a> is None FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key' FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name'] FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll' ======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ======== ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355471341 | IC_kwDOBm6k_c5Qytnt | 9599 | 2022-12-16T19:23:33Z | 2022-12-16T19:23:33Z | OWNER | Trying this instead: ``` docker run -it ubuntu:22.04 /bin/bash ``` Then in that shell: ``` apt-get update apt-get install python3.11 python3.11-venv git -y cd /tmp git clone https://github.com/simonw/datasette cd datasette git checkout async-tests python3.11 -m venv venv pip install -e '.[test]' pytest ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355445710 | IC_kwDOBm6k_c5QynXO | 9599 | 2022-12-16T19:06:49Z | 2022-12-16T19:09:35Z | OWNER | This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop. https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though. https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/ | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355325426 | IC_kwDOBm6k_c5QyJ_y | 9599 | 2022-12-16T18:00:40Z | 2022-12-16T18:00:40Z | OWNER | Many of the failing tests pass on my laptop but fail in CI. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355319541 | IC_kwDOBm6k_c5QyIj1 | 9599 | 2022-12-16T17:58:24Z | 2022-12-16T17:58:46Z | OWNER | > I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this. It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884 ====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) ======= | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1355317782 | IC_kwDOBm6k_c5QyIIW | 9599 | 2022-12-16T17:57:25Z | 2022-12-16T17:57:25Z | OWNER | Opened a follow-up issue here: - #1962 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355317369 | IC_kwDOBm6k_c5QyIB5 | 9599 | 2022-12-16T17:57:14Z | 2022-12-16T17:57:14Z | OWNER | Opened a follow-up issue here: - #1962 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355313058 | IC_kwDOBm6k_c5QyG-i | 9599 | 2022-12-16T17:53:17Z | 2022-12-16T17:53:17Z | OWNER | Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ``` > assert response.json() == [{"id": 1, "name": "San Francisco"}] E AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] E Full diff: E - [{'id': 1, 'name': 'San Francisco'}] E + {'error': 'Table not found: 𝐜𝐢𝐭𝐢𝐞𝐬', 'ok': False, 'status': 404, 'title': None} ``` A hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen when `python -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated. I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1355300217 | IC_kwDOBm6k_c5QyD15 | 9599 | 2022-12-16T17:44:55Z | 2022-12-16T17:44:55Z | OWNER | That's enough for this round. I'll get the tests passing and land this. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354269873 | IC_kwDOBm6k_c5QuISx | 9599 | 2022-12-16T06:11:43Z | 2022-12-16T06:11:43Z | OWNER | This is quite fast: ``` % pytest -m ds_client -n auto ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict gw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291] ................................................................................................................................................................................. [ 60%] .................................................................................................................. [100%] ================================================================================== 291 passed in 6.30s ================================================================================== ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168 | https://api.github.com/repos/simonw/sqlite-utils/issues/235 | 1354192168 | IC_kwDOCGYnMM5Qt1Uo | 9020979 | 2022-12-16T04:35:30Z | 2022-12-16T04:35:38Z | NONE | A related historical problem: https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720 I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
810618495 | |
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286 | https://api.github.com/repos/simonw/sqlite-utils/issues/235 | 1354160286 | IC_kwDOCGYnMM5Qttie | 9020979 | 2022-12-16T03:51:19Z | 2022-12-16T03:52:13Z | NONE | Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with `Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30` Unfortunately, `3.10.8` isn't listed in `pyenv` so I'm unable to install it. For me, the trigger is trying to use the `add-foreign-key` command on its own: ```bash sqlite-utils add-foreign-key library.db book_creators creator_id creators id ``` Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
810618495 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354148139 | IC_kwDOBm6k_c5Qtqkr | 9599 | 2022-12-16T03:32:52Z | 2022-12-16T03:32:52Z | OWNER | Got that done to: 68 passed in 14.92s By implementing my own global variable - since `pytest` won't use a global fixture for me, I decided to do it for myself. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354072344 | IC_kwDOBm6k_c5QtYEY | 9599 | 2022-12-16T02:00:07Z | 2022-12-16T02:00:07Z | OWNER | It did NOT speed it up: 68 passed in 26.26s | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354061440 | IC_kwDOBm6k_c5QtVaA | 9599 | 2022-12-16T01:45:38Z | 2022-12-16T01:45:38Z | OWNER | I'm going to do `test_table_html.py` next. Currently: 68 passed in 17.20s Will this speed it up? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354053151 | IC_kwDOBm6k_c5QtTYf | 9599 | 2022-12-16T01:33:22Z | 2022-12-16T01:33:22Z | OWNER | The thing with `Datasette(memory=False)` is tripping me up. The problem is that the tests written against `app_client` - which I want to replace - all assume that there is no `_memory` database, because when you start Datasette with at least one database file it doesn't enable `_memory` unless you explicitly tell it to. But the new `ds_client` fixture works by creating a named in-memory database called `fixtures`, which it does with a call to `ds.add_memory_database("fixtures")` after the object has been instantiated. This results in a datasette instance that DOES have a `_memory` database, when we didn't want one. My initial solution attempt was a huge hack - I decided that if you pass `memory=False` to the `Datasette` constructor it should mean "don't add a `_memory` database even though I didn't pass any files". I set a the default `memory` argument to `None`. This is weird and surprising (`memory=False` no does something different from `memory=None`?) and I found other tests that it broke, like this one: ```python def test_sql_errors_logged_to_stderr(): runner = CliRunner(mix_stderr=False) result = runner.invoke(cli, ["--get", "/_memory.json?sql=select+blah"]) assert result.exit_code == 1 assert "sql = 'select blah', params = {}: no such column: blah\n" in result.stderr ``` It ended up with no `_memory` database because it turns out `datasette serve ...` passes `memory=False` without me realizing it. So I'm going to undo that hack and teach the fixture to do this instead: ```python db = ds.add_memory_database("fixtures") ds.remove_database("_memory") ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354046627 | IC_kwDOBm6k_c5QtRyj | 9599 | 2022-12-16T01:23:21Z | 2022-12-16T01:23:21Z | OWNER | This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { } -@pytest.fixture(scope="session") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope="session") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database("fixtures") def prepare(conn): - conn.executescript(TABLES) - for sql, params in TABLE_PARAMETERIZED_SQL: - with conn: - conn.execute(sql, params) + if not conn.execute("select count(*) from sqlite_master").fetchone()[0]: + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) await db.execute_write_fn(prepare) return ds.client diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py index 4f7bf08c..d588342c 100644 --- a/tests/plugins/my_plugin_2.py +++ b/tests/plugins/my_plugin_2.py @@ -117,7 +117,12 @@ def actor_from_request(datasette, request): def permission_allowed(datasette, actor, action): # Testing asyncio version of permission_allowed async def inner(): - assert 2 == (await datasette.get_database().execute("select 1 + 1")).first()[0] + assert ( + 2 + == ( + await datasette.get_database("_internal").execute("select 1 + 1") + ).first()[0] + ) if action == "this_is_allowed_async": return True elif action == "this_is_denied_async": ``` `pytest -m ds_client` now passes 134 tests. Need to get `pytest -n auto` passing too. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1354036967 | IC_kwDOBm6k_c5QtPbn | 9599 | 2022-12-16T01:10:12Z | 2022-12-16T01:10:12Z | OWNER | If it does turn out that I can't use `scope="session"` on this fixture it might not actually be a showstopper: I can take advantage of the fact that `memory_name="..."` databases stay present in memory for the duration of the process, so I could have it such that each test that uses the `ds_client` fixture DOES construct a fresh `Datasette` instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/issues/1958#issuecomment-1354025319 | https://api.github.com/repos/simonw/datasette/issues/1958 | 1354025319 | IC_kwDOBm6k_c5QtMln | 11729897 | 2022-12-16T00:59:12Z | 2022-12-16T00:59:12Z | NONE | Awesome. Thank you @simonw. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1497909798 | |
https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960 | https://api.github.com/repos/simonw/datasette/issues/1958 | 1354023960 | IC_kwDOBm6k_c5QtMQY | 9599 | 2022-12-16T00:58:14Z | 2022-12-16T00:58:19Z | OWNER | This is in the `0.63.x` branch now, ready to go out in a bug fix release. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1497909798 | |
https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543 | https://api.github.com/repos/simonw/datasette/issues/1958 | 1354019543 | IC_kwDOBm6k_c5QtLLX | 9599 | 2022-12-16T00:54:11Z | 2022-12-16T00:54:11Z | OWNER | To test the fix, I did a fresh checkout of `simonw/datasette` and edited `Dockerfile` to look like this: ```dockerfile FROM python:3.11.0-slim-bullseye as build RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \ find /usr/local/lib -name '__pycache__' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001 CMD ["datasette"] ``` Then I built it like this: ``` docker build -f Dockerfile -t datasette-pre-click . ``` And ran like this: ``` docker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0 ``` This exhibited the same problem. Then I pushed a changed to branch and changed the line to: ``` RUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \ ``` Ran this: ``` docker build -f Dockerfile -t datasette-post-click . ``` And this: ``` docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 ``` It fixed the problem! ``` datasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0 http://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1497909798 | |
https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688 | https://api.github.com/repos/simonw/datasette/issues/1958 | 1354008688 | IC_kwDOBm6k_c5QtIhw | 9599 | 2022-12-16T00:45:40Z | 2022-12-16T00:45:40Z | OWNER | The fix may just be to switch to `click.echo()`. https://click.palletsprojects.com/en/8.1.x/api/#click.echo | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1497909798 | |
https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605 | https://api.github.com/repos/simonw/datasette/issues/1958 | 1353977605 | IC_kwDOBm6k_c5QtA8F | 9599 | 2022-12-16T00:38:23Z | 2022-12-16T00:38:23Z | OWNER | Confirmed, I just got the same result: ``` - % docker run datasetteproject/datasette pip install datasette-upload-csvs ~ % docker commit $(docker ps -lq) datasette-with-plugins sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ^CINFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [1] http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98 ``` Note how the auth-token URL is only displayed after you hit `Ctrl+C`! | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1497909798 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1353812913 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1353812913 | IC_kwDOBm6k_c5QsYux | 9599 | 2022-12-15T22:48:54Z | 2022-12-15T22:48:54Z | OWNER | This is all very broken: ``` % pytest -x --pdb ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items tests/test_package.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_cli_serve_get.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_black.py . [ 0%] tests/test_api.py E >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fixturedef = <FixtureDef argname='event_loop' scope='session' baseid='tests'>, request = <SubRequest 'event_loop' for <Function test_homepage>> @pytest.hookimpl(hookwrapper=True) def pytest_fixture… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1353805839 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1353805839 | IC_kwDOBm6k_c5QsXAP | 9599 | 2022-12-15T22:38:37Z | 2022-12-15T22:38:37Z | OWNER | I'm going to make `.status_code` work on `TestClient` response too, so I don't have to worry about using both `status` or `status_code` depending on which kind of object I am using. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1353765125 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1353765125 | IC_kwDOBm6k_c5QsNEF | 9599 | 2022-12-15T22:00:04Z | 2022-12-15T22:00:04Z | OWNER | I'm going to punt on that for the moment and continue to use `app_client` for tests that use that mechanism. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1353763837 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1353763837 | IC_kwDOBm6k_c5QsMv9 | 9599 | 2022-12-15T21:59:05Z | 2022-12-15T21:59:05Z | OWNER | Here's an annoying error: ``` > response4 = await ds_client.post( "/-/logout", csrftoken_from=True, cookies={"ds_actor": ds_client.actor_cookie({"id": "test"})}, ) tests/test_auth.py:88: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <datasette.app.DatasetteClient object at 0x1033cc4f0>, path = '/-/logout' kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client = <httpx.AsyncClient object at 0x10347a9b0> async def post(self, path, **kwargs): await self.ds.invoke_startup() async with httpx.AsyncClient(app=self.app) as client: > return await client.post(self._fix(path), **kwargs) E TypeError: AsyncClient.post() got an unexpected keyword argument 'csrftoken_from' ``` I need an alternative to the `csrftoken_from` mechanism I built for `TestClient`: https://github.com/simonw/datasette/blob/0b68996cc511b3a801f0cd0157bd66332d75f46f/datasette/utils/testing.py#L77-L103 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/pull/1960#issuecomment-1353749401 | https://api.github.com/repos/simonw/datasette/issues/1960 | 1353749401 | IC_kwDOBm6k_c5QsJOZ | 9599 | 2022-12-15T21:47:27Z | 2022-12-15T21:47:27Z | OWNER | I'm using this new mark: ```python @pytest.mark.ds_client ``` Purely so I can run all of the tests that I've refactored using: ``` pytest -m ds_client ``` I'll likely remove this once the test refactoring project is complete. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499150951 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1353747370 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1353747370 | IC_kwDOBm6k_c5QsIuq | 9599 | 2022-12-15T21:45:14Z | 2022-12-15T21:45:14Z | OWNER | I'm going to do this in a PR. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1353738075 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1353738075 | IC_kwDOBm6k_c5QsGdb | 9599 | 2022-12-15T21:35:56Z | 2022-12-15T21:35:56Z | OWNER | I built that `OldResponse` class: ```diff diff --git a/tests/utils.py b/tests/utils.py index 191ead9b..f39ac434 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -30,3 +30,25 @@ def inner_html(soup): def has_load_extension(): conn = sqlite3.connect(":memory:") return hasattr(conn, "enable_load_extension") + + +class OldResponse: + "Transform an HTTPX response to simulate the older TestClient responses" + # https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091 + def __init__(self, response): + self.response = response + self._json = None + + @property + def headers(self): + return self.response.headers + + @property + def status(self): + return self.response.status_code + + @property + def json(self): + if self._json is None: + self._json = self.response.json() + return self._json ``` I can use it in tests like this: ```python @pytest.mark.asyncio async def test_homepage(ds_client): response = OldResponse(await ds_client.get("/.json")) assert response.status == 200 assert "application/json; charset=utf-8" == response.headers["content-type"] assert response.json.keys() == {"fixtures": 0}.keys() d = response.json["fixtures"] assert d["name"] == "fixtures" assert d["tables_count"] == 24 assert len(d["tables_and_views_truncated"]) == 5 assert d["tables_and_views_more"] is True # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d["hidden_tables_count"] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: assert d["hidden_table_rows_sum"] == 207, response.json assert d["views_count"] == 4 ``` But as I work through the tests I'm finding it's actually not too hard to port them over, so I likely won't use it after all. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1353728682 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1353728682 | IC_kwDOBm6k_c5QsEKq | 9599 | 2022-12-15T21:28:35Z | 2022-12-15T21:28:35Z | OWNER | Got this error trying to have two tests use the same `ds_client` async fixture when I added `scope="session"` to that fixture: - https://github.com/tortoise/tortoise-orm/issues/638 Adding this to `conftest.py` (as suggested in that issue thread) seemed to fix it: ```python @pytest.fixture(scope="session") def event_loop(): return asyncio.get_event_loop() ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/issues/1619#issuecomment-1353721442 | https://api.github.com/repos/simonw/datasette/issues/1619 | 1353721442 | IC_kwDOBm6k_c5QsCZi | 2090382 | 2022-12-15T21:20:53Z | 2022-12-15T21:20:53Z | NONE | i'm also getting bit by this. I'm trying to set up an nginx reverse proxy in front of multiple datasette backends. When I run it locally or behind the proxy, I see the `base_url` value added a second time to the path for various action links on table pages (view as JSON, sort by column, etc). | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1121583414 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1353721091 | IC_kwDOBm6k_c5QsCUD | 9599 | 2022-12-15T21:20:32Z | 2022-12-15T21:20:32Z | OWNER | Rather than tediously rewriting every single test to the new shape I'm going to try a wrapper for that HTTPX response that transforms it into an imitation of the one returned by the existing `TestClient` class. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1353720559 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1353720559 | IC_kwDOBm6k_c5QsCLv | 9599 | 2022-12-15T21:19:56Z | 2022-12-15T21:19:56Z | OWNER | Here's a port of the first `def ...(app_client)` test. Note that the TestClient object works slightly differently from the HTTPX response returned by `await datasette.client.get(...)`: ```diff diff --git a/datasette/app.py b/datasette/app.py index f3cb8876..b770b469 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -281,7 +281,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) diff --git a/pytest.ini b/pytest.ini index 559e518c..0bcb0d1e 100644 --- a/pytest.ini +++ b/pytest.ini @@ -8,4 +8,5 @@ filterwarnings= ignore:.*current_task.*:PendingDeprecationWarning markers = serial: tests to avoid using with pytest-xdist + ds_client: tests using the ds_client fixture asyncio_mode = strict diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..648423ba 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,6 +2,7 @@ import httpx import os import pathlib import pytest +import pytest_asyncio import re import subprocess import tempfile @@ -23,6 +24,22 @@ UNDOCUMENTED_PERMISSIONS = { } +@pytest_asyncio.fixture +async def ds_client(): + from datasette.app import Datasette + from .fixtures import METADATA, PLUGINS_DIR + ds = Datasette(memory=False, metadata=METADATA, plugins_dir=PLUGINS_DIR) + from .fixtures import TABLES, TABLE_PARAMETERIZED_SQL + db = ds.add_memory_database("fixtures") + def prepare(conn): + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + await db.execute_write_fn(prepare) + return ds.client + + def pytest_report_header(config): return "SQLite: {}".format( … | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1353707828 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1353707828 | IC_kwDOBm6k_c5Qr_E0 | 9599 | 2022-12-15T21:06:29Z | 2022-12-15T21:06:29Z | OWNER | Previous, abandoned attempt at this work (for #1843): ```diff diff --git a/datasette/app.py b/datasette/app.py index 7e682498..cf35c3a2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -228,7 +228,7 @@ class Datasette: template_dir=None, plugins_dir=None, static_mounts=None, - memory=False, + memory=None, settings=None, secret=None, version_note=None, @@ -238,6 +238,7 @@ class Datasette: nolock=False, ): self._startup_invoked = False + self._extra_on_startup = [] assert config_dir is None or isinstance( config_dir, Path ), "config_dir= should be a pathlib.Path" @@ -278,7 +279,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) @@ -391,6 +392,9 @@ class Datasette: self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self) + def _add_on_startup(self, fn): + self._extra_on_startup.append(fn) + async def refresh_schemas(self): if self._refresh_schemas_lock.locked(): return @@ -431,6 +435,8 @@ class Datasette: # This must be called for Datasette to be in a usable state if self._startup_invoked: return + for fn in self._extra_on_startup: + await fn() # Register permissions, but watch out for duplicate name/abbr names = {} abbrs = {} @@ -1431,9 +1437,9 @@ class Datasette: ) if self.setting("trace_debug"): asgi = AsgiTracer(asgi) - asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup]) for wrapper in pm.hook.asgi… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/issues/1959#issuecomment-1353705072 | https://api.github.com/repos/simonw/datasette/issues/1959 | 1353705072 | IC_kwDOBm6k_c5Qr-Zw | 9599 | 2022-12-15T21:04:07Z | 2022-12-15T21:04:07Z | OWNER | I'm going to start by getting every test that uses the raw `(app_client)` fixture and nothing else (194 at the moment) to switch to `async def` using a shared Datasette instance and `datasette.client.get()`. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1499081664 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353701674 | IC_kwDOBm6k_c5Qr9kq | 9599 | 2022-12-15T21:00:51Z | 2022-12-15T21:00:51Z | OWNER | OK, I've broken the test suite here. I'm going to revert these two commits: - https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447 - https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7 Then I'll do a bunch of work making the test suite more robust before I try this again. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353694582 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353694582 | IC_kwDOBm6k_c5Qr712 | 9599 | 2022-12-15T20:52:46Z | 2022-12-15T20:52:46Z | OWNER | Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135 <img width="746" alt="image" src="https://user-images.githubusercontent.com/9599/207964564-46c31685-db7a-494f-9aab-a611d25e5511.png"> This suggests that the regular tests passed in CI fine, but the non-serial ones failed. I'm going to try running everything using `pytest -n auto` without splitting serial and non-serial tests. Maybe the serial thing isn't needed any more? | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1353690591 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1353690591 | IC_kwDOBm6k_c5Qr63f | 9599 | 2022-12-15T20:49:05Z | 2022-12-15T20:49:05Z | OWNER | I have a nasty feeling the cleaner solution for this would involve porting my entire test suite from `def test_blah(app_client)` sync functions (which work due to a `@async_to_sync` call in `TestClient`) to `async def test_blah(async_fixture):` functions instead. I've been using that latter pattern for new tests (and plugin tests) for quite a while now, but I never took on the job of refactoring all of the old ones. A search for `(app_client):` across the whole project currently returns 194 results which might be a reasonable target to try switching to the new pattern as a starting point. No idea if it will have much impact on the "Too many open files" errors though. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353683238 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353683238 | IC_kwDOBm6k_c5Qr5Em | 9599 | 2022-12-15T20:42:18Z | 2022-12-15T20:42:18Z | OWNER | Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353680261 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353680261 | IC_kwDOBm6k_c5Qr4WF | 9599 | 2022-12-15T20:39:19Z | 2022-12-15T20:39:19Z | OWNER | When I hit `Ctr+C` here's the traceback I get: ``` ^C^CException ignored in: <module 'threading' from '/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py'> Traceback (most recent call last): File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1530, in _shutdown !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py:324: KeyboardInterrupt (to show a full traceback on KeyboardInterrupt use --full-trace) Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/pytest", line 8, in <module> atexit_call() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit sys.exit(console_main()) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 187, in console_main t.join() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1089, in join self._wait_for_tstate_lock() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock if lock.acquire(block, timeout): KeyboardInterrupt: code = main() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 164, in main ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AW… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1353522652 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1353522652 | IC_kwDOBm6k_c5QrR3c | 9599 | 2022-12-15T18:21:27Z | 2022-12-15T18:21:27Z | OWNER | I'll still use on-disk test databases for `is_immutable=True`, but not for the majority of tests. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1353522211 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1353522211 | IC_kwDOBm6k_c5QrRwj | 9599 | 2022-12-15T18:21:02Z | 2022-12-15T18:21:02Z | OWNER | When I initially built this test suite Datasette didn't have the `memory_name=` mechanism for creating persistent in-memory databases. I'm going to see if I can switch to that for the majority of Datasette's tests. Might find that doing so both fixes this "too many open files" issue AND gives me a significant speed improvement to the test site too! Hopefully I can do most of the work on that in this big ugly function: https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/fixtures.py#L104-L173 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1843#issuecomment-1353520615 | https://api.github.com/repos/simonw/datasette/issues/1843 | 1353520615 | IC_kwDOBm6k_c5QrRXn | 9599 | 2022-12-15T18:19:25Z | 2022-12-15T18:19:25Z | OWNER | I've been seeing this error again: ``` ERROR tests/test_api_write.py::test_create_table[input16-400-expected_response16] - OSError: [Errno 24] Too ... ERROR tests/test_api_write.py::test_create_table[input17-400-expected_response17] - OSError: [Errno 24] Too ... ERROR tests/test_api_write.py::test_create_table[input18-400-expected_response18] - OSError: [Errno 24] Too ... ``` It doesn't happen in CI, and it turns out that's because CI runs `pytest -n auto` which splits the tests across multiple parallel processes. I've been working around the error on my laptop using `pytest -n auto` there too, but I'd rather not have to do that. This is also getting in my way when I try to debug other issues, like this one: - #1955 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1408757705 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353516572 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353516572 | IC_kwDOBm6k_c5QrQYc | 9599 | 2022-12-15T18:15:28Z | 2022-12-15T18:15:28Z | OWNER | I added `return` to the first line of that test to disable it, then ran again - and now it's hanging at about the same progress point through the tests but in a different test: ![Image](https://user-images.githubusercontent.com/9599/207936587-30ebf780-c0da-4e62-b20b-e274e0adaa19.png) So this time it was hanging at `test_urlsafe_components()`. So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353512099 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353512099 | IC_kwDOBm6k_c5QrPSj | 9599 | 2022-12-15T18:11:27Z | 2022-12-15T18:11:27Z | OWNER | This is surprising! ![Image](https://user-images.githubusercontent.com/9599/207935885-e1f51983-0621-4490-86a6-fafd4c876f41.png) The logs suggest that the test suite hung running this test here: https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/test_utils.py#L55-L58 I find that very hard to believe. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353509776 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353509776 | IC_kwDOBm6k_c5QrOuQ | 9599 | 2022-12-15T18:09:26Z | 2022-12-15T18:09:26Z | OWNER | I added this to `conftest.py`: ```python @pytest.fixture(autouse=True) def log_name_of_test_before_test(request): # To help identify tests that are hanging name = str(request.node) with open("/tmp/test.log", "a") as f: f.write(name + "\n") yield ``` This logs out the name of each test to `/tmp/test.log` before running the test - so I can wait until it hangs and see which test it was that caused that. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353473571 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353473571 | IC_kwDOBm6k_c5QrF4j | 9599 | 2022-12-15T17:43:28Z | 2022-12-15T17:43:48Z | OWNER | Running: pytest -n auto -x -v On may laptop to see if I can replicate. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 | |
https://github.com/simonw/datasette/issues/1955#issuecomment-1353473086 | https://api.github.com/repos/simonw/datasette/issues/1955 | 1353473086 | IC_kwDOBm6k_c5QrFw- | 9599 | 2022-12-15T17:43:08Z | 2022-12-15T17:43:08Z | OWNER | It looks like that fix _almost_ works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1496652622 |