... and with this change, the following now works correctly:

% datasette install datasette-gunicorn % datasette gunicorn fixtures.db -p 8855 [2022-12-17 18:44:29 -0800] [7651] [INFO] Starting gunicorn 20.1.0 [2022-12-17 18:44:29 -0800] [7651] [INFO] Listening at: http://127.0.0.1:8855 (7651) [2022-12-17 18:44:29 -0800] [7651] [INFO] Using worker: uvicorn.workers.UvicornWorker [2022-12-17 18:44:29 -0800] [7653] [INFO] Booting worker with pid: 7653 [2022-12-17 18:44:29 -0800] [7653] [INFO] Started server process [7653] [2022-12-17 18:44:29 -0800] [7653] [INFO] Waiting for application startup. [2022-12-17 18:44:29 -0800] [7653] [INFO] Application startup complete. So this issue is now fixed!

https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI:

``` Generated a certificate for 'localhost', '127.0.0.1', '::1' Configure your server to use the following files: cert=/home/runner/work/datasette/datasette/server.pem key=/home/runner/work/datasette/datasette/server.key Configure your client to use the following files: cert=/home/runner/work/datasette/datasette/client.pem INFO: Started server process [4036] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

INFO: 127.0.0.1:56726 - "GET /_memory.json HTTP/1.1" 200 OK 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 213 0 213 0 0 11542 0 --:--:-- --:--:-- --:--:-- 11833 INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4036] {"database": "_memory", "private": false, "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": true, "table_columns": {}, "query_ms": 1.4545189999921604}0 ```

This bash script does the job: ```bash

!/bin/bash

Generate certificates

python -m trustme

This creates server.pem, server.key, client.pem

Start the server in the background

datasette --memory \ --ssl-keyfile=server.key \ --ssl-certfile=server.pem \ -p 8152 &

Store the background process ID in a variable

server_pid=$!

Wait for the server to start

sleep 2

Make a test request using curl

curl -f --cacert client.pem 'https://localhost:8152/_memory.json'

Save curl's exit code (-f option causes it to return one on HTTP errors)

curl_exit_code=$?

Shut down the server

kill $server_pid sleep 1

Clean up the certificates

rm server.pem server.key client.pem

echo $curl_exit_code exit $curl_exit_code ```

Asked ChatGPT:

Write a bash script which starts a server in the background using "datasette -p 8002", then uses curl to make a test request against it, then shuts the server down again at the end

It gave me:

```bash

!/bin/bash

Start the server in the background

datasette -p 8002 &

Store the background process ID in a variable

server_pid=$!

Make a test request using curl

curl http://localhost:8002

Shut down the server

kill $server_pid ```

Various attempts at a fix which didn't work:

```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b..899d36fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -6,6 +5,7 @@ import pytest import pytest_asyncio import re import subprocess +import sys import tempfile import time import trustme @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None

-def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): +def wait_until_responds(url, timeout=5.0, client=None, kwargs): + client = client or httpx.Client(kwargs) start = time.time() while time.time() - start < timeout: try: - client.get(url, kwargs) + if "verify" in kwargs: + print(kwargs["verify"]) + print( + "Contents of verify file: {}".format( + open(kwargs.get("verify")).read() + ) + ) + print("client = {}, kwargs = {}".format(client, kwargs)) + client.get(url) return - except httpx.ConnectError: + except (httpx.ConnectError, httpx.RemoteProtocolError) as ex: + print(ex) time.sleep(0.1) raise AssertionError("Timed out waiting for {} to respond".format(url))

@@ -166,7 +176,7 @@ def check_permission_actions_are_documented(): @pytest.fixture(scope="session") def ds_localhost_http_server(): ds_proc = subprocess.Popen( - ["datasette", "--memory", "-p", "8041"], + [sys.executable, "-m", "datasette", "--memory", "-p", "8041"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # Avoid FileNotFoundError: [Errno 2] No such file or directory: @@ -180,7 +190,7 @@ def ds_localhost_http_server(): ds_proc.terminate()

-@pytest.fixture(scope="session") +@pytest.fixture def ds_localhost_https_server(tmp_path_factory): cert_directory = tmp_path_factory.mktemp("certs") ca = trustme.CA() @@ -194,6 +204,8 @@ def ds_localhost_https_server(tmp_path_factory): ca.cert_pem.write_to_path(path=client_cert) ds_proc = subprocess.Popen( [ + sys.executable, + "-m", "datasette", "--memory", "-p", @@ -207,7 +219,11 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds("http://localhost:8042/", verify=client_cert) + wait_until_responds( + "http://localhost:8042/memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py index 1c31e2a3..9320b623 100644 --- a/tests/test_cli_serve_server.py +++ b/tests/test_cli_serve_server.py @@ -16,7 +16,11 @@ def test_serve_localhost_http(ds_localhost_http_server): @pytest.mark.serial def test_serve_localhost_https(ds_localhost_https_server): , client_cert = ds_localhost_https_server - response = httpx.get("https://localhost:8042/_memory.json", verify=client_cert) + response = httpx.get( + "https://localhost:8042/_memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) assert { "database": "_memory", "path": "/_memory", ```

Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example.

This issue might be relevant, but I tried the suggested fix in there (Connection: close on the incoming requests) and it didn't fix my problem: - https://github.com/encode/httpx/discussions/2056

I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052

In one terminal: cd /tmp python -m trustme datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem -p 8003 Then in another terminal: curl --cacert /tmp/client.pem 'https://localhost:8003/_memory.json' This worked correctly, outputting the expected JSON.

So the feature still works, it's just the test that is broken for some reason.

I added the TLS support here: - https://github.com/simonw/datasette/issues/1221

Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632

During the polling loop it constantly raises:

httpx.RemoteProtocolError: Server disconnected without sending a response

Now the only failure is in the https test - which fails like this (in CI and on my laptop):

``` message = str(exc)

      raise mapped_exc(message) from exc

E httpx.RemoteProtocolError: Server disconnected without sending a response.

/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError =========================== short test summary info ============================ ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response. ================= 30 passed, 1264 deselected, 1 error in 6.15s ================= ``` That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24

And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215

... and it turns out those tests saved me. Because I forgot to check if datasette would actually start a server correctly!

% datasette fixtures.db -p 8852 INFO: Started server process [3538] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 86, in main await app(scope, self.receive, self.send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/asgi.py", line 437, in __call__ return await self.asgi(scope, receive, send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/asgi_csrf.py", line 39, in app_wrapped_with_csrf await app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1457, in __call__ path = scope["path"] KeyError: 'path' ERROR: Application startup failed. Exiting.

Improved version of that fixture: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 44c44f87..69dee68b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None

+def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): + start = time.time() + while time.time() - start < timeout: + try: + client.get(url, kwargs) + return + except httpx.ConnectError: + time.sleep(0.1) + raise AssertionError("Timed out waiting for {} to respond".format(url)) + + @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette @@ -161,13 +172,7 @@ def ds_localhost_http_server(): # Avoid FileNotFoundError: [Errno 2] No such file or directory: cwd=tempfile.gettempdir(), ) - # Loop until port 8041 serves traffic - while True: - try: - httpx.get("http://localhost:8041/") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8041/") # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - while True: - try: - httpx.get("https://localhost:8042/", verify=client_cert) - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory): # Poll until available transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) - while True: - try: - client.get("http://localhost/_memory.json") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost/_memory.json", client=client) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, uds ```

This bit here looks like it could hang! python # Loop until port 8041 serves traffic while True: try: httpx.get("http://localhost:8041/") break except httpx.ConnectError: time.sleep(0.1)

I think that's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L6-L13

Using this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L155-L175

Hitting Ctrl+C while using --full-trace gave me more clues:

``` % pytest -m serial tests/test_cli_serve_server.py --full-trace ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 3 items

tests/test_cli_serve_server.py ^C^C

====================================================== no tests ran in 3.49s ======================================================= Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp sock = socket.create_connection( File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused [...] ```

pytest -m serial on my Mac laptop also freezes:

``` (datasette) datasette % pytest -m serial ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items / 1264 deselected / 31 selected

tests/test_package.py . [ 3%] tests/test_cli_serve_server.py ```

No, it still causes the tests to hang (I let them run for 12 minutes):

Interesting that the regular tests passed an then the pytest -m serial ones seem to have failed.

I'm trying this fix again, after a bunch of work on the test suite in: - #1959

OK, I've broken the test suite here.

I'm going to revert these two commits:

• https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447
• https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7

Then I'll do a bunch of work making the test suite more robust before I try this again.

Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135

This suggests that the regular tests passed in CI fine, but the non-serial ones failed.

I'm going to try running everything using pytest -n auto without splitting serial and non-serial tests. Maybe the serial thing isn't needed any more?

Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60

When I hit Ctr+C here's the traceback I get: ^C^CException ignored in: <module 'threading' from '/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py'> Traceback (most recent call last): File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1530, in _shutdown !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py:324: KeyboardInterrupt (to show a full traceback on KeyboardInterrupt use --full-trace) Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/pytest", line 8, in <module> atexit_call() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit sys.exit(console_main()) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 187, in console_main t.join() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1089, in join self._wait_for_tstate_lock() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock if lock.acquire(block, timeout): KeyboardInterrupt: code = main() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 164, in main ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall return outcome.get_result() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py", line 315, in pytest_cmdline_main return wrap_session(config, _main) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py", line 303, in wrap_session config.hook.pytest_sessionfinish( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/terminal.py", line 798, in pytest_sessionfinish outcome.get_result() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/dsession.py", line 88, in pytest_sessionfinish nm.teardown_nodes() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/workermanage.py", line 79, in teardown_nodes self.group.terminate(self.EXIT_TIMEOUT) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py", line 215, in terminate safe_terminate( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py", line 311, in safe_terminate reply.get() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py", line 206, in get self.waitfinish(timeout) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py", line 213, in waitfinish if not self._result_ready.wait(timeout): File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 600, in wait signaled = self._cond.wait(timeout) File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 320, in wait waiter.acquire() KeyboardInterrupt It looks to me like this relates to pytest-xdist istelf - it's waiting on some locks but site-packages/xdist/workermanage.py shows up in that track.

I added return to the first line of that test to disable it, then ran again - and now it's hanging at about the same progress point through the tests but in a different test:

So this time it was hanging at test_urlsafe_components().

So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason.

This is surprising!

The logs suggest that the test suite hung running this test here:

https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/test_utils.py#L55-L58

I find that very hard to believe.

I added this to conftest.py:

python @pytest.fixture(autouse=True) def log_name_of_test_before_test(request): # To help identify tests that are hanging name = str(request.node) with open("/tmp/test.log", "a") as f: f.write(name + "\n") yield This logs out the name of each test to /tmp/test.log before running the test - so I can wait until it hangs and see which test it was that caused that.

Running:

pytest -n auto -x -v

On may laptop to see if I can replicate.

It looks like that fix almost works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see.

So actually that setup_db() function I wrote back in 2019 has not been executing for most of Datasette's tests. Which seems bad.

I'm inclined to ditch AsgiLifespan entirely in favour of the mechanism I described above, where invoke_startup() is called for every request on the first request processed by the server.

That may not be the best fix here. It turns out this pattern: python async def get(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) Doesn't trigger that AsgiLifespan class.

I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx

Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced invoke_startup()!

https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1416-L1440

It's possible the fix for this might be for the first incoming HTTP request to trigger invoke_startup() if it hasn't been called yet - similar to the hack I put in place for datasette.client.get() in tests:

https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1728-L1731

This would be a much more elegant fix, I could remove those multiple invoke_startup() calls entirely - and remove this tip from the documentation too: https://docs.datasette.io/en/0.63.2/testing_plugins.html#setting-up-a-datasette-test-instance

Datasette 0.63 is the release that broke this, thanks to this issue:

• https://github.com/simonw/datasette/issues/1809

This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the invoke_startup() hook.

Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago:

https://datasette.io/plugins/datasette-gunicorn

And I just tested and it has the same bug you describe here! Filed:

• https://github.com/simonw/datasette-gunicorn/issues/5