issue_comments

OK, I've broken the test suite here.

I'm going to revert these two commits:

• https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447
• https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7

Then I'll do a bunch of work making the test suite more robust before I try this again.

Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135

This suggests that the regular tests passed in CI fine, but the non-serial ones failed.

I'm going to try running everything using pytest -n auto without splitting serial and non-serial tests. Maybe the serial thing isn't needed any more?

Possibly related issue: - https://github.com/pytest-dev/pytest-xdist/issues/60

When I hit Ctr+C here's the traceback I get: ^C^CException ignored in: <module 'threading' from '/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py'> Traceback (most recent call last): File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1530, in _shutdown !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py:324: KeyboardInterrupt (to show a full traceback on KeyboardInterrupt use --full-trace) Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/pytest", line 8, in <module> atexit_call() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit sys.exit(console_main()) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 187, in console_main t.join() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1089, in join self._wait_for_tstate_lock() File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock if lock.acquire(block, timeout): KeyboardInterrupt: code = main() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 164, in main ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall return outcome.get_result() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py", line 315, in pytest_cmdline_main return wrap_session(config, _main) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py", line 303, in wrap_session config.hook.pytest_sessionfinish( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/terminal.py", line 798, in pytest_sessionfinish outcome.get_result() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/dsession.py", line 88, in pytest_sessionfinish nm.teardown_nodes() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/workermanage.py", line 79, in teardown_nodes self.group.terminate(self.EXIT_TIMEOUT) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py", line 215, in terminate safe_terminate( File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py", line 311, in safe_terminate reply.get() File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py", line 206, in get self.waitfinish(timeout) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py", line 213, in waitfinish if not self._result_ready.wait(timeout): File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 600, in wait signaled = self._cond.wait(timeout) File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 320, in wait waiter.acquire() KeyboardInterrupt It looks to me like this relates to pytest-xdist istelf - it's waiting on some locks but site-packages/xdist/workermanage.py shows up in that track.

I added return to the first line of that test to disable it, then ran again - and now it's hanging at about the same progress point through the tests but in a different test:

So this time it was hanging at test_urlsafe_components().

So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason.

This is surprising!

The logs suggest that the test suite hung running this test here:

https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/test_utils.py#L55-L58

I find that very hard to believe.

I added this to conftest.py:

python @pytest.fixture(autouse=True) def log_name_of_test_before_test(request): # To help identify tests that are hanging name = str(request.node) with open("/tmp/test.log", "a") as f: f.write(name + "\n") yield This logs out the name of each test to /tmp/test.log before running the test - so I can wait until it hangs and see which test it was that caused that.

Running:

pytest -n auto -x -v

On may laptop to see if I can replicate.

It looks like that fix almost works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see.

So actually that setup_db() function I wrote back in 2019 has not been executing for most of Datasette's tests. Which seems bad.

I'm inclined to ditch AsgiLifespan entirely in favour of the mechanism I described above, where invoke_startup() is called for every request on the first request processed by the server.

That may not be the best fix here. It turns out this pattern: python async def get(self, path, **kwargs): async with httpx.AsyncClient(app=self.app) as client: return await client.get(self._fix(path), **kwargs) Doesn't trigger that AsgiLifespan class.

I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx

Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced invoke_startup()!

https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1416-L1440

It's possible the fix for this might be for the first incoming HTTP request to trigger invoke_startup() if it hasn't been called yet - similar to the hack I put in place for datasette.client.get() in tests:

https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1728-L1731

This would be a much more elegant fix, I could remove those multiple invoke_startup() calls entirely - and remove this tip from the documentation too: https://docs.datasette.io/en/0.63.2/testing_plugins.html#setting-up-a-datasette-test-instance

Datasette 0.63 is the release that broke this, thanks to this issue:

• https://github.com/simonw/datasette/issues/1809

This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the invoke_startup() hook.

Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago:

https://datasette.io/plugins/datasette-gunicorn

And I just tested and it has the same bug you describe here! Filed:

• https://github.com/simonw/datasette-gunicorn/issues/5