Will link to this from my TIL shortly.

This is great, thank you!

I'm certain the two other cog menus (the app menu on the right of the nav bar and the column action menus) have the same problem.

Would be great to figure out the right ARIA attributes for these too.

Now live on https://latest.datasette.io/fixtures/attraction_characteristic

This does the trick:

css div.select-wrapper:focus-within { border: 1px solid black; }

The border is actually on the div that wraps the select box:

I tried adding a border: 1px dotted black to select:focus but it's not quite right - it jumps around a bit like this:

OK I see what you mean:

https://latest.datasette.io/fixtures/attraction_characteristic

https://pypi.org/project/datasette/0.63.3/ is released.

Oh that's annoying... every step in publish succeeded except the static docs one:

https://github.com/simonw/datasette/actions/runs/3723015082/jobs/6314292722

This means the documentation database used to update the search engine on https://datasette.io/ won't reflect the very latest changelog. I'm OK with that - I'll fix this workflow so that next time I publish a release this will work correctly.

... and with this change, the following now works correctly:

% datasette install datasette-gunicorn % datasette gunicorn fixtures.db -p 8855 [2022-12-17 18:44:29 -0800] [7651] [INFO] Starting gunicorn 20.1.0 [2022-12-17 18:44:29 -0800] [7651] [INFO] Listening at: http://127.0.0.1:8855 (7651) [2022-12-17 18:44:29 -0800] [7651] [INFO] Using worker: uvicorn.workers.UvicornWorker [2022-12-17 18:44:29 -0800] [7653] [INFO] Booting worker with pid: 7653 [2022-12-17 18:44:29 -0800] [7653] [INFO] Started server process [7653] [2022-12-17 18:44:29 -0800] [7653] [INFO] Waiting for application startup. [2022-12-17 18:44:29 -0800] [7653] [INFO] Application startup complete. So this issue is now fixed!

https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI:

``` Generated a certificate for 'localhost', '127.0.0.1', '::1' Configure your server to use the following files: cert=/home/runner/work/datasette/datasette/server.pem key=/home/runner/work/datasette/datasette/server.key Configure your client to use the following files: cert=/home/runner/work/datasette/datasette/client.pem INFO: Started server process [4036] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

INFO: 127.0.0.1:56726 - "GET /_memory.json HTTP/1.1" 200 OK 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 213 0 213 0 0 11542 0 --:--:-- --:--:-- --:--:-- 11833 INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [4036] {"database": "_memory", "private": false, "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": true, "table_columns": {}, "query_ms": 1.4545189999921604}0 ```

I ended up moving this test out of Python and into a bash script here: https://github.com/simonw/datasette/commit/d1d369456a7319b9de39175605568cbc9b852478

This bash script does the job: ```bash

!/bin/bash

Generate certificates

python -m trustme

This creates server.pem, server.key, client.pem

Start the server in the background

datasette --memory \ --ssl-keyfile=server.key \ --ssl-certfile=server.pem \ -p 8152 &

Store the background process ID in a variable

server_pid=$!

Wait for the server to start

sleep 2

Make a test request using curl

curl -f --cacert client.pem 'https://localhost:8152/_memory.json'

Save curl's exit code (-f option causes it to return one on HTTP errors)

curl_exit_code=$?

Shut down the server

kill $server_pid sleep 1

Clean up the certificates

rm server.pem server.key client.pem

echo $curl_exit_code exit $curl_exit_code ```

Asked ChatGPT:

Write a bash script which starts a server in the background using "datasette -p 8002", then uses curl to make a test request against it, then shuts the server down again at the end

It gave me:

```bash

!/bin/bash

Start the server in the background

datasette -p 8002 &

Store the background process ID in a variable

server_pid=$!

Make a test request using curl

curl http://localhost:8002

Shut down the server

kill $server_pid ```

Various attempts at a fix which didn't work:

```diff diff --git a/tests/conftest.py b/tests/conftest.py index 69dee68b..899d36fd 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,3 @@ -import asyncio import httpx import os import pathlib @@ -6,6 +5,7 @@ import pytest import pytest_asyncio import re import subprocess +import sys import tempfile import time import trustme @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None

-def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): +def wait_until_responds(url, timeout=5.0, client=None, kwargs): + client = client or httpx.Client(kwargs) start = time.time() while time.time() - start < timeout: try: - client.get(url, kwargs) + if "verify" in kwargs: + print(kwargs["verify"]) + print( + "Contents of verify file: {}".format( + open(kwargs.get("verify")).read() + ) + ) + print("client = {}, kwargs = {}".format(client, kwargs)) + client.get(url) return - except httpx.ConnectError: + except (httpx.ConnectError, httpx.RemoteProtocolError) as ex: + print(ex) time.sleep(0.1) raise AssertionError("Timed out waiting for {} to respond".format(url))

@@ -166,7 +176,7 @@ def check_permission_actions_are_documented(): @pytest.fixture(scope="session") def ds_localhost_http_server(): ds_proc = subprocess.Popen( - ["datasette", "--memory", "-p", "8041"], + [sys.executable, "-m", "datasette", "--memory", "-p", "8041"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, # Avoid FileNotFoundError: [Errno 2] No such file or directory: @@ -180,7 +190,7 @@ def ds_localhost_http_server(): ds_proc.terminate()

-@pytest.fixture(scope="session") +@pytest.fixture def ds_localhost_https_server(tmp_path_factory): cert_directory = tmp_path_factory.mktemp("certs") ca = trustme.CA() @@ -194,6 +204,8 @@ def ds_localhost_https_server(tmp_path_factory): ca.cert_pem.write_to_path(path=client_cert) ds_proc = subprocess.Popen( [ + sys.executable, + "-m", "datasette", "--memory", "-p", @@ -207,7 +219,11 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - wait_until_responds("http://localhost:8042/", verify=client_cert) + wait_until_responds( + "http://localhost:8042/memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py index 1c31e2a3..9320b623 100644 --- a/tests/test_cli_serve_server.py +++ b/tests/test_cli_serve_server.py @@ -16,7 +16,11 @@ def test_serve_localhost_http(ds_localhost_http_server): @pytest.mark.serial def test_serve_localhost_https(ds_localhost_https_server): , client_cert = ds_localhost_https_server - response = httpx.get("https://localhost:8042/_memory.json", verify=client_cert) + response = httpx.get( + "https://localhost:8042/_memory.json", + verify=client_cert, + headers={"Connection": "close"}, + ) assert { "database": "_memory", "path": "/_memory", ```

Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example.

This issue might be relevant, but I tried the suggested fix in there (Connection: close on the incoming requests) and it didn't fix my problem: - https://github.com/encode/httpx/discussions/2056

I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052

In one terminal: cd /tmp python -m trustme datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem -p 8003 Then in another terminal: curl --cacert /tmp/client.pem 'https://localhost:8003/_memory.json' This worked correctly, outputting the expected JSON.

So the feature still works, it's just the test that is broken for some reason.

I added the TLS support here: - https://github.com/simonw/datasette/issues/1221

Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632

During the polling loop it constantly raises:

httpx.RemoteProtocolError: Server disconnected without sending a response

Now the only failure is in the https test - which fails like this (in CI and on my laptop):

``` message = str(exc)

      raise mapped_exc(message) from exc

E httpx.RemoteProtocolError: Server disconnected without sending a response.

/opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError =========================== short test summary info ============================ ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response. ================= 30 passed, 1264 deselected, 1 error in 6.15s ================= ``` That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24

And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215

... and it turns out those tests saved me. Because I forgot to check if datasette would actually start a server correctly!

% datasette fixtures.db -p 8852 INFO: Started server process [3538] INFO: Waiting for application startup. ERROR: Exception in 'lifespan' protocol Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 86, in main await app(scope, self.receive, self.send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__ return await self.app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/asgi.py", line 437, in __call__ return await self.asgi(scope, receive, send) File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/asgi_csrf.py", line 39, in app_wrapped_with_csrf await app(scope, receive, send) File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1457, in __call__ path = scope["path"] KeyError: 'path' ERROR: Application startup failed. Exiting.

Improved version of that fixture: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index 44c44f87..69dee68b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = { _ds_client = None

+def wait_until_responds(url, timeout=5.0, client=httpx, kwargs): + start = time.time() + while time.time() - start < timeout: + try: + client.get(url, kwargs) + return + except httpx.ConnectError: + time.sleep(0.1) + raise AssertionError("Timed out waiting for {} to respond".format(url)) + + @pytest_asyncio.fixture async def ds_client(): from datasette.app import Datasette @@ -161,13 +172,7 @@ def ds_localhost_http_server(): # Avoid FileNotFoundError: [Errno 2] No such file or directory: cwd=tempfile.gettempdir(), ) - # Loop until port 8041 serves traffic - while True: - try: - httpx.get("http://localhost:8041/") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8041/") # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory): stderr=subprocess.STDOUT, cwd=tempfile.gettempdir(), ) - while True: - try: - httpx.get("https://localhost:8042/", verify=client_cert) - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost:8042/", verify=client_cert) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, client_cert @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory): # Poll until available transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) - while True: - try: - client.get("http://localhost/_memory.json") - break - except httpx.ConnectError: - time.sleep(0.1) + wait_until_responds("http://localhost/_memory.json", client=client) # Check it started successfully assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8") yield ds_proc, uds ```

This bit here looks like it could hang! python # Loop until port 8041 serves traffic while True: try: httpx.get("http://localhost:8041/") break except httpx.ConnectError: time.sleep(0.1)

I think that's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L6-L13

Using this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L155-L175

Hitting Ctrl+C while using --full-trace gave me more clues:

``` % pytest -m serial tests/test_cli_serve_server.py --full-trace ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 3 items

tests/test_cli_serve_server.py ^C^C

====================================================== no tests ran in 3.49s ======================================================= Traceback (most recent call last): File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions yield File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp sock = socket.create_connection( File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 845, in create_connection raise err File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 61] Connection refused [...] ```

pytest -m serial on my Mac laptop also freezes:

``` (datasette) datasette % pytest -m serial ======================================================= test session starts ======================================================== platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0 SQLite: 3.39.4 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0 asyncio: mode=strict collected 1295 items / 1264 deselected / 31 selected

tests/test_package.py . [ 3%] tests/test_cli_serve_server.py ```