issue_comments
8,358 rows where author_association = "OWNER" sorted by updated_at descending
This data as json, CSV (advanced)
created_at (date) >30 ✖
- 2021-03-22 63
- 2021-11-19 59
- 2020-10-15 52
- 2020-09-22 51
- 2020-10-30 47
- 2022-03-21 45
- 2020-06-09 42
- 2020-12-18 42
- 2022-10-26 42
- 2020-06-18 41
- 2022-12-13 40
- 2022-06-14 39
- 2022-10-25 39
- 2022-11-16 39
- 2020-05-27 38
- 2020-12-30 38
- 2021-12-16 38
- 2022-01-09 38
- 2022-10-27 38
- 2020-10-20 37
- 2021-11-16 36
- 2022-12-15 36
- 2020-09-15 34
- 2021-11-20 34
- 2022-01-20 34
- 2022-03-19 34
- 2020-06-08 33
- 2020-06-01 32
- 2020-10-09 32
- 2021-05-27 31
- …
user 1
- simonw 7,534
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1356651943 | https://github.com/simonw/datasette/issues/1963#issuecomment-1356651943 | https://api.github.com/repos/simonw/datasette/issues/1963 | IC_kwDOBm6k_c5Q3N2n | simonw 9599 | 2022-12-18T03:23:03Z | 2022-12-18T03:23:03Z | OWNER | Oh that's annoying... every step in publish succeeded except the static docs one: https://github.com/simonw/datasette/actions/runs/3723015082/jobs/6314292722 This means the documentation database used to update the search engine on https://datasette.io/ won't reflect the very latest changelog. I'm OK with that - I'll fix this workflow so that next time I publish a release this will work correctly. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
0.63.3 bugfix release 1501713288 | |
1356640463 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356640463 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3LDP | simonw 9599 | 2022-12-18T02:45:18Z | 2022-12-18T02:45:18Z | OWNER | ... and with this change, the following now works correctly:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356640266 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356640266 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3LAK | simonw 9599 | 2022-12-18T02:43:00Z | 2022-12-18T02:43:00Z | OWNER | https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI: ``` Generated a certificate for 'localhost', '127.0.0.1', '::1' Configure your server to use the following files: cert=/home/runner/work/datasette/datasette/server.pem key=/home/runner/work/datasette/datasette/server.key Configure your client to use the following files: cert=/home/runner/work/datasette/datasette/client.pem INFO: Started server process [4036] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed INFO: 127.0.0.1:56726 - "GET /_memory.json HTTP/1.1" 200 OK 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 213 0 213 0 0 11542 0 --:--:-- --:--:-- --:--:-- 11833 INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4036] {"database": "_memory", "private": false, "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": true, "table_columns": {}, "query_ms": 1.4545189999921604}0 ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356639873 | https://github.com/simonw/datasette/issues/1221#issuecomment-1356639873 | https://api.github.com/repos/simonw/datasette/issues/1221 | IC_kwDOBm6k_c5Q3K6B | simonw 9599 | 2022-12-18T02:39:04Z | 2022-12-18T02:39:04Z | OWNER | I ended up moving this test out of Python and into a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support SSL/TLS directly 806849424 | |
1356633937 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356633937 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3JdR | simonw 9599 | 2022-12-18T02:28:22Z | 2022-12-18T02:28:22Z | OWNER | This !/bin/bashGenerate certificatespython -m trustme This creates server.pem, server.key, client.pemStart the server in the backgrounddatasette --memory \ --ssl-keyfile=server.key \ --ssl-certfile=server.pem \ -p 8152 & Store the background process ID in a variableserver_pid=$! Wait for the server to startsleep 2 Make a test request using curlcurl -f --cacert client.pem 'https://localhost:8152/_memory.json' Save curl's exit code (-f option causes it to return one on HTTP errors)curl_exit_code=$? Shut down the serverkill $server_pid sleep 1 Clean up the certificatesrm server.pem server.key client.pem echo $curl_exit_code exit $curl_exit_code ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356630092 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356630092 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3IhM | simonw 9599 | 2022-12-18T02:20:01Z | 2022-12-18T02:20:01Z | OWNER | Asked ChatGPT:
It gave me: ```bash !/bin/bashStart the server in the backgrounddatasette -p 8002 & Store the background process ID in a variableserver_pid=$! Make a test request using curlcurl http://localhost:8002 Shut down the serverkill $server_pid ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356629783 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356629783 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3IcX | simonw 9599 | 2022-12-18T02:18:43Z | 2022-12-18T02:18:43Z | OWNER | Various attempts at a fix which didn't work: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b..899d36fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -6,6 +5,7 @@ import pytest import pytest_asyncio import re import subprocess +import sys import tempfile import time import trustme @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None -def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): +def wait_until_responds(url, timeout=5.0, client=None, kwargs): + client = client or httpx.Client(kwargs) start = time.time() while time.time() - start < timeout: try: - client.get(url, kwargs) + if "verify" in kwargs: + print(kwargs["verify"]) + print( + "Contents of verify file: {}".format( + open(kwargs.get("verify")).read() + ) + ) + print("client = {}, kwargs = {}".format(client, kwargs)) + client.get(url) return - except httpx.ConnectError: + except (httpx.ConnectError, httpx.RemoteProtocolError) as ex: + print(ex) time.sleep(0.1) raise AssertionError("Timed out waiting for {} to respond".format(url)) @@ -166,7 +176,7 @@ def check_permission_actions_are_documented(): @pytest.fixture(scope="session") def ds_localhost_http_server(): ds_proc = subprocess.Popen( - ["datasette", "--memory", "-p", "8041"], + [sys.executable, "-m", "datasette", "--memory", "-p", "8041"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # Avoid FileNotFoundError: [Errno 2] No such file or directory: @@ -180,7 +190,7 @@ def ds_localhost_http_server(): ds_proc.terminate() -@pytest.fixture(scope="session") +@pytest.fixture def ds_localhost_https_server(tmp_path_factory): cert_directory = tmp_path_factory.mktemp("certs") ca = trustme.CA() @@ -194,6 +204,8 @@ def ds_localhost_https_server(tmp_path_factory): ca.cert_pem.write_to_path(path=client_cert) ds_proc = subprocess.Popen( [ + sys.executable, + "-m", "datasette", "--memory", "-p", @@ -207,7 +219,11 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds("http://localhost:8042/", verify=client_cert) + wait_until_responds( + "http://localhost:8042/memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py index 1c31e2a3..9320b623 100644 --- a/tests/test_cli_serve_server.py +++ b/tests/test_cli_serve_server.py @@ -16,7 +16,11 @@ def test_serve_localhost_http(ds_localhost_http_server): @pytest.mark.serial def test_serve_localhost_https(ds_localhost_https_server): , client_cert = ds_localhost_https_server - response = httpx.get("https://localhost:8042/_memory.json", verify=client_cert) + response = httpx.get( + "https://localhost:8042/_memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) assert { "database": "_memory", "path": "/_memory", ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356627931 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356627931 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3H_b | simonw 9599 | 2022-12-18T02:13:01Z | 2022-12-18T02:13:01Z | OWNER | Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356627331 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356627331 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3H2D | simonw 9599 | 2022-12-18T02:11:17Z | 2022-12-18T02:11:17Z | OWNER | This issue might be relevant, but I tried the suggested fix in there ( |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356626334 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356626334 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Hme | simonw 9599 | 2022-12-18T02:04:01Z | 2022-12-18T02:04:07Z | OWNER | I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052 In one terminal:
So the feature still works, it's just the test that is broken for some reason. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356625642 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356625642 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Hbq | simonw 9599 | 2022-12-18T02:00:57Z | 2022-12-18T02:00:57Z | OWNER | I added the TLS support here: - https://github.com/simonw/datasette/issues/1221 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356625556 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356625556 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3HaU | simonw 9599 | 2022-12-18T02:00:18Z | 2022-12-18T02:00:18Z | OWNER | Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356620233 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356620233 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3GHJ | simonw 9599 | 2022-12-18T01:31:10Z | 2022-12-18T01:31:10Z | OWNER | During the polling loop it constantly raises:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356618913 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356618913 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Fyh | simonw 9599 | 2022-12-18T01:29:05Z | 2022-12-18T01:29:05Z | OWNER | Now the only failure is in the ``` message = str(exc)
/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError =========================== short test summary info ============================ ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response. ================= 30 passed, 1264 deselected, 1 error in 6.15s ================= ``` That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24 And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356610089 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356610089 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3Dop | simonw 9599 | 2022-12-18T01:12:39Z | 2022-12-18T01:12:39Z | OWNER | ... and it turns out those tests saved me. Because I forgot to check if
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356609095 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356609095 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3DZH | simonw 9599 | 2022-12-18T01:10:43Z | 2022-12-18T01:10:43Z | OWNER | Improved version of that fixture: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 44c44f87..69dee68b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None +def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): + start = time.time() + while time.time() - start < timeout: + try: + client.get(url, kwargs) + return + except httpx.ConnectError: + time.sleep(0.1) + raise AssertionError("Timed out waiting for {} to respond".format(url)) + + @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette @@ -161,13 +172,7 @@ def ds_localhost_http_server(): # Avoid FileNotFoundError: [Errno 2] No such file or directory: cwd=tempfile.gettempdir(), ) - # Loop until port 8041 serves traffic - while True: - try: - httpx.get("http://localhost:8041/") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8041/") # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - while True: - try: - httpx.get("https://localhost:8042/", verify=client_cert) - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory): # Poll until available transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) - while True: - try: - client.get("http://localhost/_memory.json") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost/_memory.json", client=client) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, uds ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356600917 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356600917 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3BZV | simonw 9599 | 2022-12-18T01:02:26Z | 2022-12-18T01:02:26Z | OWNER | This bit here looks like it could hang!
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356599930 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356599930 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3BJ6 | simonw 9599 | 2022-12-18T01:01:47Z | 2022-12-18T01:01:47Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | ||
1356596740 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356596740 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3AYE | simonw 9599 | 2022-12-18T00:59:47Z | 2022-12-18T00:59:47Z | OWNER | Hitting ``` % pytest -m serial tests/test_cli_serve_server.py --full-trace ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 3 items tests/test_cli_serve_server.py ^C^C ====================================================== no tests ran in 3.49s ======================================================= Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp sock = socket.create_connection( File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused [...] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356595665 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356595665 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q3AHR | simonw 9599 | 2022-12-18T00:58:16Z | 2022-12-18T00:58:16Z | OWNER |
``` (datasette) datasette % pytest -m serial ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items / 1264 deselected / 31 selected tests/test_package.py . [ 3%] tests/test_cli_serve_server.py ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356489200 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356489200 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q2mHw | simonw 9599 | 2022-12-17T22:29:51Z | 2022-12-17T22:29:51Z | OWNER | No, it still causes the tests to hang (I let them run for 12 minutes): Interesting that the regular tests passed an then the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356487139 | https://github.com/simonw/datasette/issues/1955#issuecomment-1356487139 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Q2lnj | simonw 9599 | 2022-12-17T22:16:52Z | 2022-12-17T22:16:52Z | OWNER | I'm trying this fix again, after a bunch of work on the test suite in: - #1959 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1356481595 | https://github.com/simonw/datasette/issues/1843#issuecomment-1356481595 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Q2kQ7 | simonw 9599 | 2022-12-17T22:02:40Z | 2022-12-17T22:02:40Z | OWNER | This is interesting:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1356480256 | https://github.com/simonw/datasette/issues/1843#issuecomment-1356480256 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Q2j8A | simonw 9599 | 2022-12-17T22:01:13Z | 2022-12-17T22:01:13Z | OWNER | The refactor in the following issue did NOT prevent this error from occurring when I try to run the full
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1356478792 | https://github.com/simonw/datasette/issues/1959#issuecomment-1356478792 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Q2jlI | simonw 9599 | 2022-12-17T21:49:36Z | 2022-12-17T21:49:36Z | OWNER | Made a really good start on this in the just-merged PR: - #1960 The follow-up work will happen in: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1356478575 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356478575 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jhv | simonw 9599 | 2022-12-17T21:47:48Z | 2022-12-17T21:47:48Z | OWNER | Stick a twig in it, this will do for the moment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356476886 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356476886 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jHW | simonw 9599 | 2022-12-17T21:37:05Z | 2022-12-17T21:37:05Z | OWNER | I think this test may be to blame: It's over-riding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356476583 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356476583 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q2jCn | simonw 9599 | 2022-12-17T21:34:51Z | 2022-12-17T21:34:51Z | OWNER | These are all the places that tests touch test_permissions.py 62: padlock_client.ds._metadata_local["allow"] = allow 77: del padlock_client.ds._metadata_local["allow"] 522: cascade_app_client.ds._metadata_local = updated_metadata 533: cascade_app_client.ds._metadata_local = previous_metadata 549: previous_metadata = cascade_app_client.ds._metadata_local 551: cascade_app_client.ds._metadata_local = metadata 566: cascade_app_client.ds._metadata_local = previous_metadata 842: perms_ds._metadata_local = updated_metadata 849: perms_ds._metadata_local = previous_metadata test_html.py 1114: orig = ds_client.ds._metadata_local 1115: ds_client.ds._metadata_local = metadata 1123: ds_client.ds._metadata_local = orig test_plugins.py 1034: ds_client.ds._metadata_local = { ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356056018 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356056018 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q08XS | simonw 9599 | 2022-12-17T05:33:26Z | 2022-12-17T05:33:26Z | OWNER | I think I've found the problem. The failing test is this one:
That's this example: Why is it expected to take 9 pages and not 5, when the definition of that view is this:
So this test is showing that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1356038242 | https://github.com/simonw/datasette/pull/1960#issuecomment-1356038242 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q04Bi | simonw 9599 | 2022-12-17T04:56:22Z | 2022-12-17T04:57:04Z | OWNER | May have spotted the problem with that
No that's not it either - |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355823260 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Q0Dic | simonw 9599 | 2022-12-16T23:36:07Z | 2022-12-16T23:36:07Z | OWNER | I ran ``` assert expected_rows == len(fetched)
On a hunch, I checked the current settings:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355691595 | https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595 | https://api.github.com/repos/simonw/datasette/issues/1962 | IC_kwDOBm6k_c5QzjZL | simonw 9599 | 2022-12-16T21:53:45Z | 2022-12-16T21:55:29Z | OWNER | Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST: One option would be adding that mechanism to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Alternative, async-friendly pattern for `make_app_client()` and similar - fully retire `TestClient` 1500636982 | |
1355685828 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qzh_E | simonw 9599 | 2022-12-16T21:50:01Z | 2022-12-16T21:50:01Z | OWNER | Looks like that
So I'm going to remove the mark. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355478743 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyvbX | simonw 9599 | 2022-12-16T19:27:12Z | 2022-12-16T19:27:12Z | OWNER | Bad news: they're definitely caused by tests that are subtly affected by other tests. This passes without errors:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355475671 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyurX | simonw 9599 | 2022-12-16T19:26:08Z | 2022-12-16T19:26:08Z | OWNER | Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355471341 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qytnt | simonw 9599 | 2022-12-16T19:23:33Z | 2022-12-16T19:23:33Z | OWNER | Trying this instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355445710 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QynXO | simonw 9599 | 2022-12-16T19:06:49Z | 2022-12-16T19:09:35Z | OWNER | This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop. https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though. https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355325426 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355325426 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyJ_y | simonw 9599 | 2022-12-16T18:00:40Z | 2022-12-16T18:00:40Z | OWNER | Many of the failing tests pass on my laptop but fail in CI. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355319541 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyIj1 | simonw 9599 | 2022-12-16T17:58:24Z | 2022-12-16T17:58:46Z | OWNER |
It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355317782 | https://github.com/simonw/datasette/issues/1959#issuecomment-1355317782 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QyIIW | simonw 9599 | 2022-12-16T17:57:25Z | 2022-12-16T17:57:25Z | OWNER | Opened a follow-up issue here: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1355317369 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355317369 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyIB5 | simonw 9599 | 2022-12-16T17:57:14Z | 2022-12-16T17:57:14Z | OWNER | Opened a follow-up issue here: - #1962 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355313058 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyG-i | simonw 9599 | 2022-12-16T17:53:17Z | 2022-12-16T17:53:17Z | OWNER | Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626 ```
I tried adding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1355300217 | https://github.com/simonw/datasette/pull/1960#issuecomment-1355300217 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QyD15 | simonw 9599 | 2022-12-16T17:44:55Z | 2022-12-16T17:44:55Z | OWNER | That's enough for this round. I'll get the tests passing and land this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354269873 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QuISx | simonw 9599 | 2022-12-16T06:11:43Z | 2022-12-16T06:11:43Z | OWNER | This is quite fast:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354148139 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5Qtqkr | simonw 9599 | 2022-12-16T03:32:52Z | 2022-12-16T03:32:52Z | OWNER | Got that done to: 68 passed in 14.92s By implementing my own global variable - since |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354072344 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354072344 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtYEY | simonw 9599 | 2022-12-16T02:00:07Z | 2022-12-16T02:00:07Z | OWNER | It did NOT speed it up: 68 passed in 26.26s |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354061440 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtVaA | simonw 9599 | 2022-12-16T01:45:38Z | 2022-12-16T01:45:38Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | ||
1354053151 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtTYf | simonw 9599 | 2022-12-16T01:33:22Z | 2022-12-16T01:33:22Z | OWNER | The thing with The problem is that the tests written against But the new This results in a datasette instance that DOES have a My initial solution attempt was a huge hack - I decided that if you pass This is weird and surprising (
So I'm going to undo that hack and teach the fixture to do this instead:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354046627 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtRyj | simonw 9599 | 2022-12-16T01:23:21Z | 2022-12-16T01:23:21Z | OWNER | This does seem to help: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 1306c407..af9c7696 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = { } -@pytest.fixture(scope="session") -def event_loop(): - return asyncio.get_event_loop() - - -@pytest_asyncio.fixture(scope="session") +@pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette from .fixtures import METADATA, PLUGINS_DIR @@ -53,10 +48,11 @@ async def ds_client(): db = ds.add_memory_database("fixtures")
Need to get |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354036967 | https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QtPbn | simonw 9599 | 2022-12-16T01:10:12Z | 2022-12-16T01:10:12Z | OWNER | If it does turn out that I can't use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1354023960 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354023960 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtMQY | simonw 9599 | 2022-12-16T00:58:14Z | 2022-12-16T00:58:19Z | OWNER | This is in the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1354019543 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtLLX | simonw 9599 | 2022-12-16T00:54:11Z | 2022-12-16T00:54:11Z | OWNER | To test the fix, I did a fresh checkout of RUN apt-get update && \ apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg/info/* RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \ find /usr/local/lib -name 'pycache' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001
CMD ["datasette"]
Then I pushed a changed to branch and changed the line to:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1354008688 | https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtIhw | simonw 9599 | 2022-12-16T00:45:40Z | 2022-12-16T00:45:40Z | OWNER | The fix may just be to switch to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1353977605 | https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5QtA8F | simonw 9599 | 2022-12-16T00:38:23Z | 2022-12-16T00:38:23Z | OWNER | Confirmed, I just got the same result: ``` - % docker run datasetteproject/datasette pip install datasette-upload-csvs ~ % docker commit $(docker ps -lq) datasette-with-plugins sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0 INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit) ^CINFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1]
http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1353812913 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353812913 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsYux | simonw 9599 | 2022-12-15T22:48:54Z | 2022-12-15T22:48:54Z | OWNER | This is all very broken: ``` % pytest -x --pdb ================================================================================== test session starts ================================================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items tests/test_package.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_cli_serve_get.py .. [ 0%] tests/test_cli.py . [ 0%] tests/test_black.py . [ 0%] tests/test_api.py E
fixturedef = <FixtureDef argname='event_loop' scope='session' baseid='tests'>, request = <SubRequest 'event_loop' for \<Function test_homepage>>
/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pytest_asyncio/plugin.py:377: /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: in event_loop return asyncio.get_event_loop() self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x111e34670>
/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py:656: RuntimeError
=================================================================================== warnings summary ==================================================================================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /Users/simon/Dropbox/Development/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:596: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error tests/test_cli_serve_get.py::test_serve_with_get /Users/simon/Dropbox/Development/datasette/datasette/cli.py:599: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds)) tests/test_api.py::test_homepage /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: DeprecationWarning: There is no current event loop return asyncio.get_event_loop() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================ short test summary info ================================================================================ ERROR tests/test_api.py::test_homepage - RuntimeError: There is no current event loop in thread 'MainThread'. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ======================================================================== 7 passed, 7 warnings, 1 error in 19.15s ======================================================================== (datasette) datasette % ``` The problem looks to be caused by this: Which I found necessary in order to have Can I work around this, or is |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353805839 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353805839 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsXAP | simonw 9599 | 2022-12-15T22:38:37Z | 2022-12-15T22:38:37Z | OWNER | I'm going to make |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353765125 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353765125 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsNEF | simonw 9599 | 2022-12-15T22:00:04Z | 2022-12-15T22:00:04Z | OWNER | I'm going to punt on that for the moment and continue to use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353763837 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353763837 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsMv9 | simonw 9599 | 2022-12-15T21:59:05Z | 2022-12-15T21:59:05Z | OWNER | Here's an annoying error: ```
tests/test_auth.py:88: self = <datasette.app.DatasetteClient object at 0x1033cc4f0>, path = '/-/logout' kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client = <httpx.AsyncClient object at 0x10347a9b0>
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353749401 | https://github.com/simonw/datasette/pull/1960#issuecomment-1353749401 | https://api.github.com/repos/simonw/datasette/issues/1960 | IC_kwDOBm6k_c5QsJOZ | simonw 9599 | 2022-12-15T21:47:27Z | 2022-12-15T21:47:27Z | OWNER | I'm using this new mark:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Port as many tests as possible to async def tests against ds_client 1499150951 | |
1353747370 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353747370 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsIuq | simonw 9599 | 2022-12-15T21:45:14Z | 2022-12-15T21:45:14Z | OWNER | I'm going to do this in a PR. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353738075 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353738075 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsGdb | simonw 9599 | 2022-12-15T21:35:56Z | 2022-12-15T21:35:56Z | OWNER | I built that |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353728682 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353728682 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsEKq | simonw 9599 | 2022-12-15T21:28:35Z | 2022-12-15T21:28:35Z | OWNER | Got this error trying to have two tests use the same Adding this to
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353721091 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsCUD | simonw 9599 | 2022-12-15T21:20:32Z | 2022-12-15T21:20:32Z | OWNER | Rather than tediously rewriting every single test to the new shape I'm going to try a wrapper for that HTTPX response that transforms it into an imitation of the one returned by the existing |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353720559 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353720559 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5QsCLv | simonw 9599 | 2022-12-15T21:19:56Z | 2022-12-15T21:19:56Z | OWNER | Here's a port of the first ```diff diff --git a/datasette/app.py b/datasette/app.py index f3cb8876..b770b469 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -281,7 +281,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) diff --git a/pytest.ini b/pytest.ini index 559e518c..0bcb0d1e 100644 --- a/pytest.ini +++ b/pytest.ini @@ -8,4 +8,5 @@ filterwarnings= ignore:.current_task.:PendingDeprecationWarning markers = serial: tests to avoid using with pytest-xdist + ds_client: tests using the ds_client fixture asyncio_mode = strict diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..648423ba 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -2,6 +2,7 @@ import httpx import os import pathlib import pytest +import pytest_asyncio import re import subprocess import tempfile @@ -23,6 +24,22 @@ UNDOCUMENTED_PERMISSIONS = { } +@pytest_asyncio.fixture +async def ds_client(): + from datasette.app import Datasette + from .fixtures import METADATA, PLUGINS_DIR + ds = Datasette(memory=False, metadata=METADATA, plugins_dir=PLUGINS_DIR) + from .fixtures import TABLES, TABLE_PARAMETERIZED_SQL + db = ds.add_memory_database("fixtures") + def prepare(conn): + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + await db.execute_write_fn(prepare) + return ds.client + + def pytest_report_header(config): return "SQLite: {}".format( sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0] diff --git a/tests/test_api.py b/tests/test_api.py index 5f2a6ea6..ddf4219c 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -23,12 +23,15 @@ import sys import urllib -def test_homepage(app_client): - response = app_client.get("/.json") - assert response.status == 200 +@pytest.mark.ds_client +@pytest.mark.asyncio +async def test_homepage(ds_client): + response = await ds_client.get("/.json") + assert response.status_code == 200 assert "application/json; charset=utf-8" == response.headers["content-type"] - assert response.json.keys() == {"fixtures": 0}.keys() - d = response.json["fixtures"] + data = response.json() + assert data.keys() == {"fixtures": 0}.keys() + d = data["fixtures"] assert d["name"] == "fixtures" assert d["tables_count"] == 24 assert len(d["tables_and_views_truncated"]) == 5 @@ -36,7 +39,7 @@ def test_homepage(app_client): # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert d["hidden_tables_count"] == 6 # 201 in no_primary_key, plus 6 in other hidden tables: - assert d["hidden_table_rows_sum"] == 207, response.json + assert d["hidden_table_rows_sum"] == 207, data assert d["views_count"] == 4 ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353707828 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353707828 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Qr_E0 | simonw 9599 | 2022-12-15T21:06:29Z | 2022-12-15T21:06:29Z | OWNER | Previous, abandoned attempt at this work (for #1843): ```diff diff --git a/datasette/app.py b/datasette/app.py index 7e682498..cf35c3a2 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -228,7 +228,7 @@ class Datasette: template_dir=None, plugins_dir=None, static_mounts=None, - memory=False, + memory=None, settings=None, secret=None, version_note=None, @@ -238,6 +238,7 @@ class Datasette: nolock=False, ): self._startup_invoked = False + self._extra_on_startup = [] assert config_dir is None or isinstance( config_dir, Path ), "config_dir= should be a pathlib.Path" @@ -278,7 +279,7 @@ class Datasette: raise self.crossdb = crossdb self.nolock = nolock - if memory or crossdb or not self.files: + if memory or crossdb or (not self.files and memory is not False): self.add_database( Database(self, is_mutable=False, is_memory=True), name="_memory" ) @@ -391,6 +392,9 @@ class Datasette: self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self)
diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py index 56690251..986755cb 100644 --- a/datasette/utils/asgi.py +++ b/datasette/utils/asgi.py @@ -423,9 +423,9 @@ class AsgiFileDownload: class AsgiRunOnFirstRequest: - def init(self, asgi, on_startup): + def init(self, app, on_startup): assert isinstance(on_startup, list) - self.asgi = asgi + self.app = app self.on_startup = on_startup self._started = False @@ -434,4 +434,4 @@ class AsgiRunOnFirstRequest: self._started = True for hook in self.on_startup: await hook() - return await self.asgi(scope, receive, send) + return await self.app(scope, receive, send) diff --git a/tests/conftest.py b/tests/conftest.py index cd735e12..d1301943 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -23,6 +23,15 @@ UNDOCUMENTED_PERMISSIONS = { } +# @pytest.fixture(autouse=True) +# def log_name_of_test_before_test(request): +# # To help identify tests that are hanging +# name = str(request.node) +# with open("/tmp/test.log", "a") as f: +# f.write(name + "\n") +# yield + + def pytest_report_header(config): return "SQLite: {}".format( sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0] diff --git a/tests/fixtures.py b/tests/fixtures.py index a6700239..18d3f1b7 100644 --- a/tests/fixtures.py +++ b/tests/fixtures.py @@ -101,6 +101,19 @@ EXPECTED_PLUGINS = [ ] +def _populate_connection(conn): + # Drop any tables and views that exist + to_drop = conn.execute( + "SELECT name, type FROM sqlite_master where type in ('table', 'view')" + ).fetchall() + for name, type in to_drop: + conn.execute(f"DROP {type} IF EXISTS [{name}]") + conn.executescript(TABLES) + for sql, params in TABLE_PARAMETERIZED_SQL: + with conn: + conn.execute(sql, params) + + @contextlib.contextmanager def make_app_client( sql_time_limit_ms=None, @@ -117,45 +130,22 @@ def make_app_client( metadata=None, crossdb=False, ): - with tempfile.TemporaryDirectory() as tmpdir: - filepath = os.path.join(tmpdir, filename) - if is_immutable: - files = [] - immutables = [filepath] - else: - files = [filepath] - immutables = [] - conn = sqlite3.connect(filepath) - conn.executescript(TABLES) - for sql, params in TABLE_PARAMETERIZED_SQL: - with conn: - conn.execute(sql, params) - # Close the connection to avoid "too many open files" errors - conn.close() - if extra_databases is not None: - for extra_filename, extra_sql in extra_databases.items(): - extra_filepath = os.path.join(tmpdir, extra_filename) - c2 = sqlite3.connect(extra_filepath) - c2.executescript(extra_sql) - c2.close() - # Insert at start to help test /-/databases ordering: - files.insert(0, extra_filepath) - os.chdir(os.path.dirname(filepath)) - settings = settings or {} - for key, value in { - "default_page_size": 50, - "max_returned_rows": max_returned_rows or 100, - "sql_time_limit_ms": sql_time_limit_ms or 200, - # Default is 3 but this results in "too many open files" - # errors when running the full test suite: - "num_sql_threads": 1, - }.items(): - if key not in settings: - settings[key] = value + settings = settings or {} + for key, value in { + "default_page_size": 50, + "max_returned_rows": max_returned_rows or 100, + "sql_time_limit_ms": sql_time_limit_ms or 200, + # Default is 3 but this results in "too many open files" + # errors when running the full test suite: + "num_sql_threads": 1, + }.items(): + if key not in settings: + settings[key] = value + # We can use an in-memory database, but only if we're not doing anything + # with is_immutable or extra_databases and filename is the default + if not is_immutable and not extra_databases and filename == "fixtures.db": ds = Datasette( - files, - immutables=immutables, - memory=memory, + memory=memory or False, cors=cors, metadata=metadata or METADATA, plugins_dir=PLUGINS_DIR, @@ -165,12 +155,57 @@ def make_app_client( template_dir=template_dir, crossdb=crossdb, ) + db = ds.add_memory_database("fixtures") + + async def populate_fixtures(): + print("Here we go... populating fixtures") + await db.execute_write_fn(_populate_connection) + + ds._add_on_startup(populate_fixtures) yield TestClient(ds) - # Close as many database connections as possible - # to try and avoid too many open files error - for db in ds.databases.values(): - if not db.is_memory: - db.close() + else: + with tempfile.TemporaryDirectory() as tmpdir: + filepath = os.path.join(tmpdir, filename) + if is_immutable: + files = [] + immutables = [filepath] + else: + files = [filepath] + immutables = [] + + conn = sqlite3.connect(filepath) + _populate_connection(conn) + # Close the connection to reduce "too many open files" errors + conn.close() + + if extra_databases is not None: + for extra_filename, extra_sql in extra_databases.items(): + extra_filepath = os.path.join(tmpdir, extra_filename) + c2 = sqlite3.connect(extra_filepath) + c2.executescript(extra_sql) + c2.close() + # Insert at start to help test /-/databases ordering: + files.insert(0, extra_filepath) + os.chdir(os.path.dirname(filepath)) + ds = Datasette( + files, + immutables=immutables, + memory=memory, + cors=cors, + metadata=metadata or METADATA, + plugins_dir=PLUGINS_DIR, + settings=settings, + inspect_data=inspect_data, + static_mounts=static_mounts, + template_dir=template_dir, + crossdb=crossdb, + ) + yield TestClient(ds) + # Close as many database connections as possible + # to try and avoid too many open files error + for db in ds.databases.values(): + if not db.is_memory: + db.close() @pytest.fixture(scope="session") diff --git a/tests/test_cli.py b/tests/test_cli.py index d3e015fa..d9e4e457 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1,5 +1,6 @@ from .fixtures import ( app_client, + app_client_with_cors, make_app_client, TestClient as _TestClient, EXPECTED_PLUGINS, @@ -38,7 +39,7 @@ def test_inspect_cli(app_client): assert expected_count == database["tables"][table_name]["count"] -def test_inspect_cli_writes_to_file(app_client): +def test_inspect_cli_writes_to_file(app_client_with_cors): runner = CliRunner() result = runner.invoke( cli, ["inspect", "fixtures.db", "--inspect-file", "foo.json"] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353705072 | https://github.com/simonw/datasette/issues/1959#issuecomment-1353705072 | https://api.github.com/repos/simonw/datasette/issues/1959 | IC_kwDOBm6k_c5Qr-Zw | simonw 9599 | 2022-12-15T21:04:07Z | 2022-12-15T21:04:07Z | OWNER | I'm going to start by getting every test that uses the raw |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Refactor test suite to use mostly `async def` tests 1499081664 | |
1353701674 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr9kq | simonw 9599 | 2022-12-15T21:00:51Z | 2022-12-15T21:00:51Z | OWNER | OK, I've broken the test suite here. I'm going to revert these two commits:
Then I'll do a bunch of work making the test suite more robust before I try this again. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353694582 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353694582 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr712 | simonw 9599 | 2022-12-15T20:52:46Z | 2022-12-15T20:52:46Z | OWNER | Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135 This suggests that the regular tests passed in CI fine, but the non-serial ones failed. I'm going to try running everything using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353690591 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353690591 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5Qr63f | simonw 9599 | 2022-12-15T20:49:05Z | 2022-12-15T20:49:05Z | OWNER | I have a nasty feeling the cleaner solution for this would involve porting my entire test suite from I've been using that latter pattern for new tests (and plugin tests) for quite a while now, but I never took on the job of refactoring all of the old ones. A search for No idea if it will have much impact on the "Too many open files" errors though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353683238 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353683238 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr5Em | simonw 9599 | 2022-12-15T20:42:18Z | 2022-12-15T20:42:18Z | OWNER | Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353680261 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353680261 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qr4WF | simonw 9599 | 2022-12-15T20:39:19Z | 2022-12-15T20:39:19Z | OWNER | When I hit |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353522652 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353522652 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5QrR3c | simonw 9599 | 2022-12-15T18:21:27Z | 2022-12-15T18:21:27Z | OWNER | I'll still use on-disk test databases for |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353522211 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353522211 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5QrRwj | simonw 9599 | 2022-12-15T18:21:02Z | 2022-12-15T18:21:02Z | OWNER | When I initially built this test suite Datasette didn't have the I'm going to see if I can switch to that for the majority of Datasette's tests. Might find that doing so both fixes this "too many open files" issue AND gives me a significant speed improvement to the test site too! Hopefully I can do most of the work on that in this big ugly function: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353520615 | https://github.com/simonw/datasette/issues/1843#issuecomment-1353520615 | https://api.github.com/repos/simonw/datasette/issues/1843 | IC_kwDOBm6k_c5QrRXn | simonw 9599 | 2022-12-15T18:19:25Z | 2022-12-15T18:19:25Z | OWNER | I've been seeing this error again:
It doesn't happen in CI, and it turns out that's because CI runs I've been working around the error on my laptop using This is also getting in my way when I try to debug other issues, like this one:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Intermittent "Too many open files" error running tests 1408757705 | |
1353516572 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353516572 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrQYc | simonw 9599 | 2022-12-15T18:15:28Z | 2022-12-15T18:15:28Z | OWNER | I added So this time it was hanging at So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353512099 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353512099 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrPSj | simonw 9599 | 2022-12-15T18:11:27Z | 2022-12-15T18:11:27Z | OWNER | This is surprising! The logs suggest that the test suite hung running this test here: I find that very hard to believe. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353509776 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353509776 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrOuQ | simonw 9599 | 2022-12-15T18:09:26Z | 2022-12-15T18:09:26Z | OWNER | I added this to
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353473571 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353473571 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrF4j | simonw 9599 | 2022-12-15T17:43:28Z | 2022-12-15T17:43:48Z | OWNER | Running:
On may laptop to see if I can replicate. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353473086 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353473086 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QrFw- | simonw 9599 | 2022-12-15T17:43:08Z | 2022-12-15T17:43:08Z | OWNER | It looks like that fix almost works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353448095 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353448095 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq_qf | simonw 9599 | 2022-12-15T17:25:05Z | 2022-12-15T17:25:05Z | OWNER | So actually that I'm inclined to ditch |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353443718 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353443718 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq-mG | simonw 9599 | 2022-12-15T17:23:12Z | 2022-12-15T17:23:55Z | OWNER | That may not be the best fix here. It turns out this pattern:
I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1353423584 | https://github.com/simonw/datasette/issues/1955#issuecomment-1353423584 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qq5rg | simonw 9599 | 2022-12-15T17:13:18Z | 2022-12-15T17:22:59Z | OWNER | Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352674924 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352674924 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5QoC5s | simonw 9599 | 2022-12-15T07:46:36Z | 2022-12-15T07:46:36Z | OWNER | It's possible the fix for this might be for the first incoming HTTP request to trigger This would be a much more elegant fix, I could remove those multiple |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352644281 | https://github.com/simonw/datasette/issues/1958#issuecomment-1352644281 | https://api.github.com/repos/simonw/datasette/issues/1958 | IC_kwDOBm6k_c5Qn7a5 | simonw 9599 | 2022-12-15T07:08:14Z | 2022-12-15T07:08:14Z | OWNER | Thanks for the details write-up! This looks like a bug in Datasette itself when run with Docker. Moving this issue there. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette --root running in Docker doesn't reliably show the magic URL 1497909798 | |
1352643333 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352643333 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qn7MF | simonw 9599 | 2022-12-15T07:07:29Z | 2022-12-15T07:07:29Z | OWNER | Datasette 0.63 is the release that broke this, thanks to this issue: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352643049 | https://github.com/simonw/datasette/issues/1955#issuecomment-1352643049 | https://api.github.com/repos/simonw/datasette/issues/1955 | IC_kwDOBm6k_c5Qn7Hp | simonw 9599 | 2022-12-15T07:07:10Z | 2022-12-15T07:07:10Z | OWNER | This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago: https://datasette.io/plugins/datasette-gunicorn And I just tested and it has the same bug you describe here! Filed: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 1496652622 | |
1352459146 | https://github.com/simonw/datasette/issues/1953#issuecomment-1352459146 | https://api.github.com/repos/simonw/datasette/issues/1953 | IC_kwDOBm6k_c5QnOOK | simonw 9599 | 2022-12-15T02:02:15Z | 2022-12-15T02:02:15Z | OWNER | ``` The third Datasette 1.0 alpha release adds upsert support to the JSON API, plus the ability to specify finely grained permissions when creating an API token.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release notes for Datasette 1.0a2 1495821607 | |
1352411327 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352411327 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QnCi_ | simonw 9599 | 2022-12-15T00:46:27Z | 2022-12-15T00:46:27Z | OWNER | I got this far: ```diff diff --git a/datasette/handle_exception.py b/datasette/handle_exception.py index 8b7e83e3..31d41e00 100644 --- a/datasette/handle_exception.py +++ b/datasette/handle_exception.py @@ -54,7 +54,17 @@ def handle_exception(datasette, request, exception): headers = {} if datasette.cors: add_cors_headers(headers) - if request.path.split("?")[0].endswith(".json"): + # Return JSON error under certain conditions + should_return_json = ( + # URL ends in .json + request.path.split("?")[0].endswith(".json") + or + # Hints from incoming request headers + request.headers.get("content-type") == "application/json" + or "application/json" in request.headers.get("accept", "") + ) + breakpoint() + if should_return_json: return Response.json(info, status=status, headers=headers) else: template = datasette.jinja_env.select_template(templates) diff --git a/tests/test_api_write.py b/tests/test_api_write.py index f27d143f..982543a6 100644 --- a/tests/test_api_write.py +++ b/tests/test_api_write.py @@ -1140,6 +1140,38 @@ async def test_create_table_permissions( assert data["errors"] == expected_errors +@pytest.mark.asyncio +@pytest.mark.parametrize( + "headers,expect_json", + ( + ({}, False), + ({"Accept": "text/html"}, True), + ({"Accept": "application/json"}, True), + ({"Content-Type": "application/json"}, True), + ({"Accept": "application/json, text/plain, /"}, True), + ({"Content-Type": "application/json"}, True), + ({"accept": "application/json, text/plain, /"}, True), + ({"content-type": "application/json"}, True), + ), +) +async def test_permission_errors_html_and_json(ds_write, headers, expect_json): + request_headers = {"Authorization": "Bearer bad_token"} + request_headers.update(headers) + response = await ds_write.client.post( + "/data/-/create", + json={}, + headers=request_headers, + ) + assert response.status_code == 403 + if expect_json: + data = response.json() + assert data["ok"] is False + assert data["errors"] == ["Permission denied"] + else: + assert response.headers["Content-Type"] == "text/html; charset=utf-8" + assert "Permission denied" in response.text + + @pytest.mark.asyncio @pytest.mark.parametrize( "input,expected_rows_after", ``` Then decided I would punt this until the next milestone. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352410078 | https://github.com/simonw/datasette/issues/1953#issuecomment-1352410078 | https://api.github.com/repos/simonw/datasette/issues/1953 | IC_kwDOBm6k_c5QnCPe | simonw 9599 | 2022-12-15T00:44:56Z | 2022-12-15T00:44:56Z | OWNER | Highlights:
Plus some smaller things. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Release notes for Datasette 1.0a2 1495821607 | |
1352378370 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352378370 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qm6gC | simonw 9599 | 2022-12-15T00:02:08Z | 2022-12-15T00:04:54Z | OWNER | I fixed this issue to help research this further: - https://github.com/simonw/datasette-ripgrep/issues/26 Now this search works: https://ripgrep.datasette.io/-/ripgrep?pattern=return+_error&literal=on&glob=datasette%2F** I wish I had this feature! - https://github.com/simonw/datasette-ripgrep/issues/24 Looks like I have both |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352371019 | https://github.com/simonw/datasette/issues/1957#issuecomment-1352371019 | https://api.github.com/repos/simonw/datasette/issues/1957 | IC_kwDOBm6k_c5Qm4tL | simonw 9599 | 2022-12-14T23:50:50Z | 2022-12-14T23:50:50Z | OWNER | One option: if any rows were truncated, show a button you can click to run the query again with truncation disabled - maybe with But... still want to truncate if the user runs a query that would return multiple MBs of HTML (assuming the other Datasette query limits don't stop that from happening). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Reconsider row value truncation on query page 1497577017 | |
1352357322 | https://github.com/simonw/datasette/issues/1295#issuecomment-1352357322 | https://api.github.com/repos/simonw/datasette/issues/1295 | IC_kwDOBm6k_c5Qm1XK | simonw 9599 | 2022-12-14T23:28:49Z | 2022-12-14T23:28:49Z | OWNER | Related: - #1875 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Errors should have links to further information 855296937 | |
1352356356 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352356356 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qm1IE | simonw 9599 | 2022-12-14T23:27:25Z | 2022-12-14T23:28:16Z | OWNER | Also weird: errors returned by that mechanism look like this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352354927 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352354927 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qm0xv | simonw 9599 | 2022-12-14T23:25:06Z | 2022-12-14T23:25:14Z | OWNER | Looks like the code I've written for permission checking on Which uses this: Having two different patterns to return errors is bad, I should fix that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352340518 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352340518 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmxQm | simonw 9599 | 2022-12-14T23:07:01Z | 2022-12-14T23:07:01Z | OWNER | Easiest fix would be to look for Not bullet-proof, so people might occasionally make JSON requests and get back an HTML error - but the documentation can tell people that they need to send those headers if they want to reliably get back JSON error messages. I'm happy with this as a solution. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352338620 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352338620 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qmwy8 | simonw 9599 | 2022-12-14T23:05:17Z | 2022-12-14T23:05:17Z | OWNER | Sniffing for a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352335503 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352335503 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmwCP | simonw 9599 | 2022-12-14T23:03:28Z | 2022-12-14T23:03:28Z | OWNER | This raises a more complicated issue At some point I'm likely to want to add an HTML interface for creating tables and inserting and updating rows. The obvious URLs for that are the same as for the JSON API: Those endpoints are currently POST only - and can return JSON all the time. If they start accepting form POSTs too they'll need to be able to accept form-encoded data and return HTML instead. That's OK - they can detect incoming JSON thanks to the I think it can still work though: I'll only set |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352331314 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352331314 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmvAy | simonw 9599 | 2022-12-14T22:59:36Z | 2022-12-14T22:59:36Z | OWNER | I'm going to prototype that up to see what it looks like. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352330825 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352330825 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5Qmu5J | simonw 9599 | 2022-12-14T22:58:51Z | 2022-12-14T22:59:27Z | OWNER | I need a way for those JSON endpoints to communicate back to the Since it gets the It's a bit of a cludge though! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 | |
1352329027 | https://github.com/simonw/datasette/issues/1949#issuecomment-1352329027 | https://api.github.com/repos/simonw/datasette/issues/1949 | IC_kwDOBm6k_c5QmudD | simonw 9599 | 2022-12-14T22:56:24Z | 2022-12-14T22:57:19Z | OWNER | Most But that code triggers when the URL ends with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`.json` errors should be returned as JSON 1493471221 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
issue >30