# issue_comments

html_url: https://github.com/simonw/datasette/issues/1953#issuecomment-1352459146
issue_url: https://api.github.com/repos/simonw/datasette/issues/1953
node_id: IC_kwDOBm6k_c5QnOOK
created_at: 2022-12-15T02:02:15Z
updated_at: 2022-12-15T02:02:15Z
author_association: OWNER
body:
  ```
  The third Datasette 1.0 alpha release adds upsert support to the JSON API, plus the ability to specify finely grained permissions when creating an API token.

  - New `/db/table/-/upsert` API, [documented here](https://docs.datasette.io/en/latest/json_api.html#tableupsertview). upsert is an update-or-replace: existing rows will have specified keys updated, but if no row matches the incoming primary key a brand new row will be inserted instead. ([#1878](https://github.com/simonw/datasette/issues/1878))
  - New [register_permissions(datasette)](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-register-permissions) plugin hook. Plugins can now register named permissions, which will then be listed in various interfaces that show available permissions. ([#1940](https://github.com/simonw/datasette/issues/1940))
  - The `/db/-/create` API for [creating a table](https://docs.datasette.io/en/latest/json_api.html#tablecreateview) now accepts `"ignore": true` and `"replace": true` options when called with the `"rows"` property that creates a new table based on an example set of rows. This means the API can be called multiple times with different rows, setting rules for what should happen if a primary key collides with an existing row. ([#1927](https://github.com/simonw/datasette/issues/1927))
  - Arbitrary permissions can now be configured at the instance, database and resource (table, SQL view or canned query) level in Datasette's [Metadata](https://docs.datasette.io/en/latest/metadata.html#metadata) JSON and YAML files. The new `"permissions"` key can be used to specify which actors should have which permissions. See [Other permissions in metadata](https://docs.datasette.io/en/latest/authentication.html#authentication-permissions-other) for details. ([#1636](https://github.com/simonw/datasette/issues/1636))
  - The `/-/create-token` page can now be used to create API tokens which are restricted to just a subset of actions, including against specific databases or resources. See [API Tokens](https://docs.datasette.io/en/latest/authentication.html#createtokenview) for details. ([#1947](https://github.com/simonw/datasette/issues/1947))
  - Likewise, the `datasette create-token` CLI command can now create tokens with [a subset of permissions](https://docs.datasette.io/en/latest/authentication.html#authentication-cli-create-token-restrict). ([#1855](https://github.com/simonw/datasette/issues/1855))
  - New datasette.create_token() API method <datasette_create_token>` for programmatically creating signed API tokens. ([#1951](https://github.com/simonw/datasette/issues/1951))
  - `/db/-/create` API now requires actor to have `insert-row` permission in order to use the `"row"` or `"rows"` properties. ([#1937](https://github.com/simonw/datasette/issues/1937))
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1352643049
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qn7Hp
created_at: 2022-12-15T07:07:10Z
updated_at: 2022-12-15T07:07:10Z
author_association: OWNER
body:
  This is definitely a regression: Datasette is meant to work in those environments, and I didn't think to test them when I added the `invoke_startup()` hook.

  Coincidentally I actually built a plugin for running Datasette with Gunicorn just a couple of months ago:

  https://datasette.io/plugins/datasette-gunicorn

  And I just tested and it has the same bug you describe here! Filed:

  - https://github.com/simonw/datasette-gunicorn/issues/5

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1352643333
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qn7MF
created_at: 2022-12-15T07:07:29Z
updated_at: 2022-12-15T07:07:29Z
author_association: OWNER
body:
  Datasette 0.63 is the release that broke this, thanks to this issue:

  - https://github.com/simonw/datasette/issues/1809

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1352644262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5Qn7am
created_at: 2022-12-13T16:49:31Z
updated_at: 2022-12-13T16:49:52Z
author_association: NONE
body:
  I may have found the reason. I don't think the `--root` option is taking effect.

  Visited: http://127.0.0.1:8001/-/permissions

  ![image](https://user-images.githubusercontent.com/11729897/207394127-59f8320f-3410-4ff8-869d-b19ab955d67c.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1352644267
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5Qn7ar
created_at: 2022-12-13T18:33:32Z
updated_at: 2022-12-13T18:33:32Z
author_association: OWNER
body:
  When you run `--root` you need to follow the special link that gets output to the console:

  ```
  % datasette --root
  http://127.0.0.1:8001/-/auth-token?token=036d8055cc8000e9667f21c1dd08722a9358c066463873ad9566d23d88765c52
  INFO:     Started server process [53934]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  ```
  That `/-/auth-token?...` link is the one that sets the cookie and lets you in.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1352644270
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5Qn7au
created_at: 2022-12-14T00:05:16Z
updated_at: 2022-12-14T00:05:16Z
author_association: NONE
body:
  FYI @simonw, I don't see that message.

  ![image](https://user-images.githubusercontent.com/11729897/207471893-1ec5d4d0-81db-4eaa-ba40-adedcc7c9a32.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1352644274
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5Qn7ay
created_at: 2022-12-14T14:19:24Z
updated_at: 2022-12-14T14:19:24Z
author_association: NONE
body:
  Hmm, it appears after I kill the process with `Ctrl+c`:

  ![image](https://user-images.githubusercontent.com/11729897/207619736-a49b2e97-c5d0-4e54-ad3c-3c3c279927c7.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1352644276
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5Qn7a0
created_at: 2022-12-14T14:53:53Z
updated_at: 2022-12-14T14:53:53Z
author_association: NONE
body:
  I don't have much experience with Python; however, I wonder if this print statement needs `flush=True`?

  https://github.com/simonw/datasette/blob/fdf7c27b5438f02153c3a7f8ad1b320e4b29e4f4/datasette/cli.py#L621

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1352644281
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5Qn7a5
created_at: 2022-12-15T07:08:14Z
updated_at: 2022-12-15T07:08:14Z
author_association: OWNER
body:
  Thanks for the details write-up! This looks like a bug in Datasette itself when run with Docker. Moving this issue there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1352674924
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5QoC5s
created_at: 2022-12-15T07:46:36Z
updated_at: 2022-12-15T07:46:36Z
author_association: OWNER
body:
  It's possible the fix for this might be for the first incoming HTTP request to trigger `invoke_startup()` if it hasn't been called yet - similar to the hack I put in place for `datasette.client.get()` in tests:

  https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1728-L1731

  This would be a much more elegant fix, I could remove those multiple `invoke_startup()` calls entirely - and remove this tip from the documentation too: https://docs.datasette.io/en/0.63.2/testing_plugins.html#setting-up-a-datasette-test-instance

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353423584
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qq5rg
created_at: 2022-12-15T17:13:18Z
updated_at: 2022-12-15T17:22:59Z
author_association: OWNER
body:
  Wow, just spotted this in the code - it turns out I solved this problem a different (and better) way long before i introduced `invoke_startup()`!

  https://github.com/simonw/datasette/blob/e054704fb64d1f23154ec43b81b6c9481ff8202f/datasette/app.py#L1416-L1440

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353443718
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qq-mG
created_at: 2022-12-15T17:23:12Z
updated_at: 2022-12-15T17:23:55Z
author_association: OWNER
body:
  That may not be the best fix here. It turns out this pattern:
  ```python
      async def get(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.get(self._fix(path), **kwargs)
  ```
  Doesn't trigger that `AsgiLifespan` class.

  I wrote about that previously in this TIL: https://til.simonwillison.net/asgi/lifespan-test-httpx

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353448095
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qq_qf
created_at: 2022-12-15T17:25:05Z
updated_at: 2022-12-15T17:25:05Z
author_association: OWNER
body:
  So actually that `setup_db()` function I wrote back in 2019 has not been executing for most of Datasette's tests. Which seems bad.

  I'm inclined to ditch `AsgiLifespan` entirely in favour of the mechanism I described above, where `invoke_startup()` is called for every request on the first request processed by the server.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353473086
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5QrFw-
created_at: 2022-12-15T17:43:08Z
updated_at: 2022-12-15T17:43:08Z
author_association: OWNER
body:
  It looks like that fix _almost_ works... except it seems to push the tests into an infinite loop or similar? They're not finishing their runs from what I can see.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353509776
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5QrOuQ
created_at: 2022-12-15T18:09:26Z
updated_at: 2022-12-15T18:09:26Z
author_association: OWNER
body:
  I added this to `conftest.py`:

  ```python
  @pytest.fixture(autouse=True)
  def log_name_of_test_before_test(request):
      # To help identify tests that are hanging
      name = str(request.node)
      with open("/tmp/test.log", "a") as f:
          f.write(name + "\n")
      yield
  ```
  This logs out the name of each test to `/tmp/test.log` before running the test - so I can wait until it hangs and see which test it was that caused that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353512099
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5QrPSj
created_at: 2022-12-15T18:11:27Z
updated_at: 2022-12-15T18:11:27Z
author_association: OWNER
body:
  This is surprising!

  ![Image](https://user-images.githubusercontent.com/9599/207935885-e1f51983-0621-4490-86a6-fafd4c876f41.png)

  The logs suggest that the test suite hung running this test here:

  https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/test_utils.py#L55-L58

  I find that very hard to believe.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353516572
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5QrQYc
created_at: 2022-12-15T18:15:28Z
updated_at: 2022-12-15T18:15:28Z
author_association: OWNER
body:
  I added `return` to the first line of that test to disable it, then ran again - and now it's hanging at about the same progress point through the tests but in a different test:

  ![Image](https://user-images.githubusercontent.com/9599/207936587-30ebf780-c0da-4e62-b20b-e274e0adaa19.png)

  So this time it was hanging at `test_urlsafe_components()`.

  So it's clearly not the individual tests themselves that are the problem - something about running the entire test suite in one go is incompatible with this change for some reason.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1353520615
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5QrRXn
created_at: 2022-12-15T18:19:25Z
updated_at: 2022-12-15T18:19:25Z
author_association: OWNER
body:
  I've been seeing this error again:

  ```
  ERROR tests/test_api_write.py::test_create_table[input16-400-expected_response16] - OSError: [Errno 24] Too ...
  ERROR tests/test_api_write.py::test_create_table[input17-400-expected_response17] - OSError: [Errno 24] Too ...
  ERROR tests/test_api_write.py::test_create_table[input18-400-expected_response18] - OSError: [Errno 24] Too ...
  ```

  It doesn't happen in CI, and it turns out that's because CI runs `pytest -n auto` which splits the tests across multiple parallel processes.

  I've been working around the error on my laptop using `pytest -n auto` there too, but I'd rather not have to do that.

  This is also getting in my way when I try to debug other issues, like this one:

  - #1955

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1353522211
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5QrRwj
created_at: 2022-12-15T18:21:02Z
updated_at: 2022-12-15T18:21:02Z
author_association: OWNER
body:
  When I initially built this test suite Datasette didn't have the `memory_name=` mechanism for creating persistent in-memory databases.

  I'm going to see if I can switch to that for the majority of Datasette's tests. Might find that doing so both fixes this "too many open files" issue AND gives me a significant speed improvement to the test site too!

  Hopefully I can do most of the work on that in this big ugly function:

  https://github.com/simonw/datasette/blob/dc18f62089e5672d03176f217d7840cdafa5c447/tests/fixtures.py#L104-L173

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353680261
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qr4WF
created_at: 2022-12-15T20:39:19Z
updated_at: 2022-12-15T20:39:19Z
author_association: OWNER
body:
  When I hit `Ctr+C` here's the traceback I get:
  ```
  ^C^CException ignored in: <module 'threading' from '/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py'>
  Traceback (most recent call last):
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1530, in _shutdown
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py:324: KeyboardInterrupt
  (to show a full traceback on KeyboardInterrupt use --full-trace)
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/pytest", line 8, in <module>
      atexit_call()
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit
      sys.exit(console_main())
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 187, in console_main
      t.join()
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1089, in join
      self._wait_for_tstate_lock()
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 1109, in _wait_for_tstate_lock
      if lock.acquire(block, timeout):
  KeyboardInterrupt: 
      code = main()
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/config/__init__.py", line 164, in main
      ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
      return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
      return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
      return outcome.get_result()
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py", line 315, in pytest_cmdline_main
      return wrap_session(config, _main)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/main.py", line 303, in wrap_session
      config.hook.pytest_sessionfinish(
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
      return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
      return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall
      gen.send(outcome)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/_pytest/terminal.py", line 798, in pytest_sessionfinish
      outcome.get_result()
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
      raise ex[1].with_traceback(ex[2])
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
      res = hook_impl.function(*args)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/dsession.py", line 88, in pytest_sessionfinish
      nm.teardown_nodes()
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/xdist/workermanage.py", line 79, in teardown_nodes
      self.group.terminate(self.EXIT_TIMEOUT)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py", line 215, in terminate
      safe_terminate(
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/multi.py", line 311, in safe_terminate
      reply.get()
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py", line 206, in get
      self.waitfinish(timeout)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/execnet/gateway_base.py", line 213, in waitfinish
      if not self._result_ready.wait(timeout):
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 600, in wait
      signaled = self._cond.wait(timeout)
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/threading.py", line 320, in wait
      waiter.acquire()
  KeyboardInterrupt
  ```
  It looks to me like this relates to `pytest-xdist` istelf - it's waiting on some locks but `site-packages/xdist/workermanage.py` shows up in that track.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1353690591
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5Qr63f
created_at: 2022-12-15T20:49:05Z
updated_at: 2022-12-15T20:49:05Z
author_association: OWNER
body:
  I have a nasty feeling the cleaner solution for this would involve porting my entire test suite from `def test_blah(app_client)` sync functions (which work due to a `@async_to_sync` call in `TestClient`) to `async def test_blah(async_fixture):` functions instead.

  I've been using that latter pattern for new tests (and plugin tests) for quite a while now, but I never took on the job of refactoring all of the old ones.

  A search for `(app_client):` across the whole project currently returns 194 results which might be a reasonable target to try switching to the new pattern as a starting point.

  No idea if it will have much impact on the "Too many open files" errors though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353694582
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qr712
created_at: 2022-12-15T20:52:46Z
updated_at: 2022-12-15T20:52:46Z
author_association: OWNER
body:
  Just noticed this: https://github.com/simonw/datasette/actions/runs/3706504228/jobs/6281796135

  <img width="746" alt="image" src="https://user-images.githubusercontent.com/9599/207964564-46c31685-db7a-494f-9aab-a611d25e5511.png">

  This suggests that the regular tests passed in CI fine, but the non-serial ones failed.

  I'm going to try running everything using `pytest -n auto` without splitting serial and non-serial tests. Maybe the serial thing isn't needed any more?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1353701674
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Qr9kq
created_at: 2022-12-15T21:00:51Z
updated_at: 2022-12-15T21:00:51Z
author_association: OWNER
body:
  OK, I've broken the test suite here.

  I'm going to revert these two commits:

  - https://github.com/simonw/datasette/commit/dc18f62089e5672d03176f217d7840cdafa5c447
  - https://github.com/simonw/datasette/commit/51ee8caa4a697fa3f4120e93b1c205b714a6cdc7

  Then I'll do a bunch of work making the test suite more robust before I try this again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1959#issuecomment-1353705072
issue_url: https://api.github.com/repos/simonw/datasette/issues/1959
node_id: IC_kwDOBm6k_c5Qr-Zw
created_at: 2022-12-15T21:04:07Z
updated_at: 2022-12-15T21:04:07Z
author_association: OWNER
body:
  I'm going to start by getting every test that uses the raw `(app_client)` fixture and nothing else (194 at the moment) to switch to `async def` using a shared Datasette instance and `datasette.client.get()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1959#issuecomment-1353707828
issue_url: https://api.github.com/repos/simonw/datasette/issues/1959
node_id: IC_kwDOBm6k_c5Qr_E0
created_at: 2022-12-15T21:06:29Z
updated_at: 2022-12-15T21:06:29Z
author_association: OWNER
body:
  Previous, abandoned attempt at this work (for #1843):
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 7e682498..cf35c3a2 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -228,7 +228,7 @@ class Datasette:
           template_dir=None,
           plugins_dir=None,
           static_mounts=None,
  -        memory=False,
  +        memory=None,
           settings=None,
           secret=None,
           version_note=None,
  @@ -238,6 +238,7 @@ class Datasette:
           nolock=False,
       ):
           self._startup_invoked = False
  +        self._extra_on_startup = []
           assert config_dir is None or isinstance(
               config_dir, Path
           ), "config_dir= should be a pathlib.Path"
  @@ -278,7 +279,7 @@ class Datasette:
                   raise
           self.crossdb = crossdb
           self.nolock = nolock
  -        if memory or crossdb or not self.files:
  +        if memory or crossdb or (not self.files and memory is not False):
               self.add_database(
                   Database(self, is_mutable=False, is_memory=True), name="_memory"
               )
  @@ -391,6 +392,9 @@ class Datasette:
           self._root_token = secrets.token_hex(32)
           self.client = DatasetteClient(self)
   
  +    def _add_on_startup(self, fn):
  +        self._extra_on_startup.append(fn)
  +
       async def refresh_schemas(self):
           if self._refresh_schemas_lock.locked():
               return
  @@ -431,6 +435,8 @@ class Datasette:
           # This must be called for Datasette to be in a usable state
           if self._startup_invoked:
               return
  +        for fn in self._extra_on_startup:
  +            await fn()
           # Register permissions, but watch out for duplicate name/abbr
           names = {}
           abbrs = {}
  @@ -1431,9 +1437,9 @@ class Datasette:
           )
           if self.setting("trace_debug"):
               asgi = AsgiTracer(asgi)
  -        asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup])
           for wrapper in pm.hook.asgi_wrapper(datasette=self):
               asgi = wrapper(asgi)
  +        asgi = AsgiRunOnFirstRequest(asgi, on_startup=[setup_db, self.invoke_startup])
           return asgi
   
   
  diff --git a/datasette/utils/asgi.py b/datasette/utils/asgi.py
  index 56690251..986755cb 100644
  --- a/datasette/utils/asgi.py
  +++ b/datasette/utils/asgi.py
  @@ -423,9 +423,9 @@ class AsgiFileDownload:
   
   
   class AsgiRunOnFirstRequest:
  -    def __init__(self, asgi, on_startup):
  +    def __init__(self, app, on_startup):
           assert isinstance(on_startup, list)
  -        self.asgi = asgi
  +        self.app = app
           self.on_startup = on_startup
           self._started = False
   
  @@ -434,4 +434,4 @@ class AsgiRunOnFirstRequest:
               self._started = True
               for hook in self.on_startup:
                   await hook()
  -        return await self.asgi(scope, receive, send)
  +        return await self.app(scope, receive, send)
  diff --git a/tests/conftest.py b/tests/conftest.py
  index cd735e12..d1301943 100644
  --- a/tests/conftest.py
  +++ b/tests/conftest.py
  @@ -23,6 +23,15 @@ UNDOCUMENTED_PERMISSIONS = {
   }
   
   
  +# @pytest.fixture(autouse=True)
  +# def log_name_of_test_before_test(request):
  +#     # To help identify tests that are hanging
  +#     name = str(request.node)
  +#     with open("/tmp/test.log", "a") as f:
  +#         f.write(name + "\n")
  +#     yield
  +
  +
   def pytest_report_header(config):
       return "SQLite: {}".format(
           sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0]
  diff --git a/tests/fixtures.py b/tests/fixtures.py
  index a6700239..18d3f1b7 100644
  --- a/tests/fixtures.py
  +++ b/tests/fixtures.py
  @@ -101,6 +101,19 @@ EXPECTED_PLUGINS = [
   ]
   
   
  +def _populate_connection(conn):
  +    # Drop any tables and views that exist
  +    to_drop = conn.execute(
  +        "SELECT name, type FROM sqlite_master where type in ('table', 'view')"
  +    ).fetchall()
  +    for name, type in to_drop:
  +        conn.execute(f"DROP {type} IF EXISTS [{name}]")
  +    conn.executescript(TABLES)
  +    for sql, params in TABLE_PARAMETERIZED_SQL:
  +        with conn:
  +            conn.execute(sql, params)
  +
  +
   @contextlib.contextmanager
   def make_app_client(
       sql_time_limit_ms=None,
  @@ -117,45 +130,22 @@ def make_app_client(
       metadata=None,
       crossdb=False,
   ):
  -    with tempfile.TemporaryDirectory() as tmpdir:
  -        filepath = os.path.join(tmpdir, filename)
  -        if is_immutable:
  -            files = []
  -            immutables = [filepath]
  -        else:
  -            files = [filepath]
  -            immutables = []
  -        conn = sqlite3.connect(filepath)
  -        conn.executescript(TABLES)
  -        for sql, params in TABLE_PARAMETERIZED_SQL:
  -            with conn:
  -                conn.execute(sql, params)
  -        # Close the connection to avoid "too many open files" errors
  -        conn.close()
  -        if extra_databases is not None:
  -            for extra_filename, extra_sql in extra_databases.items():
  -                extra_filepath = os.path.join(tmpdir, extra_filename)
  -                c2 = sqlite3.connect(extra_filepath)
  -                c2.executescript(extra_sql)
  -                c2.close()
  -                # Insert at start to help test /-/databases ordering:
  -                files.insert(0, extra_filepath)
  -        os.chdir(os.path.dirname(filepath))
  -        settings = settings or {}
  -        for key, value in {
  -            "default_page_size": 50,
  -            "max_returned_rows": max_returned_rows or 100,
  -            "sql_time_limit_ms": sql_time_limit_ms or 200,
  -            # Default is 3 but this results in "too many open files"
  -            # errors when running the full test suite:
  -            "num_sql_threads": 1,
  -        }.items():
  -            if key not in settings:
  -                settings[key] = value
  +    settings = settings or {}
  +    for key, value in {
  +        "default_page_size": 50,
  +        "max_returned_rows": max_returned_rows or 100,
  +        "sql_time_limit_ms": sql_time_limit_ms or 200,
  +        # Default is 3 but this results in "too many open files"
  +        # errors when running the full test suite:
  +        "num_sql_threads": 1,
  +    }.items():
  +        if key not in settings:
  +            settings[key] = value
  +    # We can use an in-memory database, but only if we're not doing anything
  +    # with is_immutable or extra_databases and filename is the default
  +    if not is_immutable and not extra_databases and filename == "fixtures.db":
           ds = Datasette(
  -            files,
  -            immutables=immutables,
  -            memory=memory,
  +            memory=memory or False,
               cors=cors,
               metadata=metadata or METADATA,
               plugins_dir=PLUGINS_DIR,
  @@ -165,12 +155,57 @@ def make_app_client(
               template_dir=template_dir,
               crossdb=crossdb,
           )
  +        db = ds.add_memory_database("fixtures")
  +
  +        async def populate_fixtures():
  +            print("Here we go... populating fixtures")
  +            await db.execute_write_fn(_populate_connection)
  +
  +        ds._add_on_startup(populate_fixtures)
           yield TestClient(ds)
  -        # Close as many database connections as possible
  -        # to try and avoid too many open files error
  -        for db in ds.databases.values():
  -            if not db.is_memory:
  -                db.close()
  +    else:
  +        with tempfile.TemporaryDirectory() as tmpdir:
  +            filepath = os.path.join(tmpdir, filename)
  +            if is_immutable:
  +                files = []
  +                immutables = [filepath]
  +            else:
  +                files = [filepath]
  +                immutables = []
  +
  +            conn = sqlite3.connect(filepath)
  +            _populate_connection(conn)
  +            # Close the connection to reduce "too many open files" errors
  +            conn.close()
  +
  +            if extra_databases is not None:
  +                for extra_filename, extra_sql in extra_databases.items():
  +                    extra_filepath = os.path.join(tmpdir, extra_filename)
  +                    c2 = sqlite3.connect(extra_filepath)
  +                    c2.executescript(extra_sql)
  +                    c2.close()
  +                    # Insert at start to help test /-/databases ordering:
  +                    files.insert(0, extra_filepath)
  +            os.chdir(os.path.dirname(filepath))
  +            ds = Datasette(
  +                files,
  +                immutables=immutables,
  +                memory=memory,
  +                cors=cors,
  +                metadata=metadata or METADATA,
  +                plugins_dir=PLUGINS_DIR,
  +                settings=settings,
  +                inspect_data=inspect_data,
  +                static_mounts=static_mounts,
  +                template_dir=template_dir,
  +                crossdb=crossdb,
  +            )
  +            yield TestClient(ds)
  +            # Close as many database connections as possible
  +            # to try and avoid too many open files error
  +            for db in ds.databases.values():
  +                if not db.is_memory:
  +                    db.close()
   
   
   @pytest.fixture(scope="session")
  diff --git a/tests/test_cli.py b/tests/test_cli.py
  index d3e015fa..d9e4e457 100644
  --- a/tests/test_cli.py
  +++ b/tests/test_cli.py
  @@ -1,5 +1,6 @@
   from .fixtures import (
       app_client,
  +    app_client_with_cors,
       make_app_client,
       TestClient as _TestClient,
       EXPECTED_PLUGINS,
  @@ -38,7 +39,7 @@ def test_inspect_cli(app_client):
           assert expected_count == database["tables"][table_name]["count"]
   
   
  -def test_inspect_cli_writes_to_file(app_client):
  +def test_inspect_cli_writes_to_file(app_client_with_cors):
       runner = CliRunner()
       result = runner.invoke(
           cli, ["inspect", "fixtures.db", "--inspect-file", "foo.json"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1959#issuecomment-1353720559
issue_url: https://api.github.com/repos/simonw/datasette/issues/1959
node_id: IC_kwDOBm6k_c5QsCLv
created_at: 2022-12-15T21:19:56Z
updated_at: 2022-12-15T21:19:56Z
author_association: OWNER
body:
  Here's a port of the first `def ...(app_client)` test. Note that the TestClient object works slightly differently from the HTTPX response returned by `await datasette.client.get(...)`:

  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index f3cb8876..b770b469 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -281,7 +281,7 @@ class Datasette:
                   raise
           self.crossdb = crossdb
           self.nolock = nolock
  -        if memory or crossdb or not self.files:
  +        if memory or crossdb or (not self.files and memory is not False):
               self.add_database(
                   Database(self, is_mutable=False, is_memory=True), name="_memory"
               )
  diff --git a/pytest.ini b/pytest.ini
  index 559e518c..0bcb0d1e 100644
  --- a/pytest.ini
  +++ b/pytest.ini
  @@ -8,4 +8,5 @@ filterwarnings=
       ignore:.*current_task.*:PendingDeprecationWarning
   markers =
       serial: tests to avoid using with pytest-xdist
  +    ds_client: tests using the ds_client fixture
   asyncio_mode = strict
  diff --git a/tests/conftest.py b/tests/conftest.py
  index cd735e12..648423ba 100644
  --- a/tests/conftest.py
  +++ b/tests/conftest.py
  @@ -2,6 +2,7 @@ import httpx
   import os
   import pathlib
   import pytest
  +import pytest_asyncio
   import re
   import subprocess
   import tempfile
  @@ -23,6 +24,22 @@ UNDOCUMENTED_PERMISSIONS = {
   }
   
   
  +@pytest_asyncio.fixture
  +async def ds_client():
  +    from datasette.app import Datasette
  +    from .fixtures import METADATA, PLUGINS_DIR
  +    ds = Datasette(memory=False, metadata=METADATA, plugins_dir=PLUGINS_DIR)
  +    from .fixtures import TABLES, TABLE_PARAMETERIZED_SQL
  +    db = ds.add_memory_database("fixtures")
  +    def prepare(conn):
  +        conn.executescript(TABLES)
  +        for sql, params in TABLE_PARAMETERIZED_SQL:
  +            with conn:
  +                conn.execute(sql, params)
  +    await db.execute_write_fn(prepare)
  +    return ds.client
  +
  +
   def pytest_report_header(config):
       return "SQLite: {}".format(
           sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0]
  diff --git a/tests/test_api.py b/tests/test_api.py
  index 5f2a6ea6..ddf4219c 100644
  --- a/tests/test_api.py
  +++ b/tests/test_api.py
  @@ -23,12 +23,15 @@ import sys
   import urllib
   
   
  -def test_homepage(app_client):
  -    response = app_client.get("/.json")
  -    assert response.status == 200
  +@pytest.mark.ds_client
  +@pytest.mark.asyncio
  +async def test_homepage(ds_client):
  +    response = await ds_client.get("/.json")
  +    assert response.status_code == 200
       assert "application/json; charset=utf-8" == response.headers["content-type"]
  -    assert response.json.keys() == {"fixtures": 0}.keys()
  -    d = response.json["fixtures"]
  +    data = response.json()
  +    assert data.keys() == {"fixtures": 0}.keys()
  +    d = data["fixtures"]
       assert d["name"] == "fixtures"
       assert d["tables_count"] == 24
       assert len(d["tables_and_views_truncated"]) == 5
  @@ -36,7 +39,7 @@ def test_homepage(app_client):
       # 4 hidden FTS tables + no_primary_key (hidden in metadata)
       assert d["hidden_tables_count"] == 6
       # 201 in no_primary_key, plus 6 in other hidden tables:
  -    assert d["hidden_table_rows_sum"] == 207, response.json
  +    assert d["hidden_table_rows_sum"] == 207, data
       assert d["views_count"] == 4
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091
issue_url: https://api.github.com/repos/simonw/datasette/issues/1959
node_id: IC_kwDOBm6k_c5QsCUD
created_at: 2022-12-15T21:20:32Z
updated_at: 2022-12-15T21:20:32Z
author_association: OWNER
body:
  Rather than tediously rewriting every single test to the new shape I'm going to try a wrapper for that HTTPX response that transforms it into an imitation of the one returned by the existing `TestClient` class.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1619#issuecomment-1353721442
issue_url: https://api.github.com/repos/simonw/datasette/issues/1619
node_id: IC_kwDOBm6k_c5QsCZi
created_at: 2022-12-15T21:20:53Z
updated_at: 2022-12-15T21:20:53Z
author_association: NONE
body:
  i'm also getting bit by this. I'm trying to set up an nginx reverse proxy in front of multiple datasette backends.

  When I run it locally or behind the proxy, I see the `base_url` value added a second time to the path for various action links on table pages (view as JSON, sort by column, etc).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1959#issuecomment-1353728682
issue_url: https://api.github.com/repos/simonw/datasette/issues/1959
node_id: IC_kwDOBm6k_c5QsEKq
created_at: 2022-12-15T21:28:35Z
updated_at: 2022-12-15T21:28:35Z
author_association: OWNER
body:
  Got this error trying to have two tests use the same `ds_client` async fixture when I added `scope="session"` to that fixture:

  - https://github.com/tortoise/tortoise-orm/issues/638

  Adding this to `conftest.py` (as suggested in that issue thread) seemed to fix it:

  ```python
  @pytest.fixture(scope="session")
  def event_loop():
      return asyncio.get_event_loop()
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1959#issuecomment-1353738075
issue_url: https://api.github.com/repos/simonw/datasette/issues/1959
node_id: IC_kwDOBm6k_c5QsGdb
created_at: 2022-12-15T21:35:56Z
updated_at: 2022-12-15T21:35:56Z
author_association: OWNER
body:
  I built that `OldResponse` class:
  ```diff
  diff --git a/tests/utils.py b/tests/utils.py
  index 191ead9b..f39ac434 100644
  --- a/tests/utils.py
  +++ b/tests/utils.py
  @@ -30,3 +30,25 @@ def inner_html(soup):
   def has_load_extension():
       conn = sqlite3.connect(":memory:")
       return hasattr(conn, "enable_load_extension")
  +
  +
  +class OldResponse:
  +    "Transform an HTTPX response to simulate the older TestClient responses"
  +    # https://github.com/simonw/datasette/issues/1959#issuecomment-1353721091
  +    def __init__(self, response):
  +        self.response = response
  +        self._json = None
  +
  +    @property
  +    def headers(self):
  +        return self.response.headers
  +
  +    @property
  +    def status(self):
  +        return self.response.status_code
  +
  +    @property
  +    def json(self):
  +        if self._json is None:
  +            self._json = self.response.json()
  +        return self._json
  ```
  I can use it in tests like this:
  ```python
  @pytest.mark.asyncio
  async def test_homepage(ds_client):
      response = OldResponse(await ds_client.get("/.json"))
      assert response.status == 200
      assert "application/json; charset=utf-8" == response.headers["content-type"]
      assert response.json.keys() == {"fixtures": 0}.keys()
      d = response.json["fixtures"]
      assert d["name"] == "fixtures"
      assert d["tables_count"] == 24
      assert len(d["tables_and_views_truncated"]) == 5
      assert d["tables_and_views_more"] is True
      # 4 hidden FTS tables + no_primary_key (hidden in metadata)
      assert d["hidden_tables_count"] == 6
      # 201 in no_primary_key, plus 6 in other hidden tables:
      assert d["hidden_table_rows_sum"] == 207, response.json
      assert d["views_count"] == 4
  ```
  But as I work through the tests I'm finding it's actually not too hard to port them over, so I likely won't use it after all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1353749401
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QsJOZ
created_at: 2022-12-15T21:47:27Z
updated_at: 2022-12-15T21:47:27Z
author_association: OWNER
body:
  I'm using this new mark:
  ```python
  @pytest.mark.ds_client
  ```
  Purely so I can run all of the tests that I've refactored using:

  ```
  pytest -m ds_client
  ```
  I'll likely remove this once the test refactoring project is complete.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1353763837
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QsMv9
created_at: 2022-12-15T21:59:05Z
updated_at: 2022-12-15T21:59:05Z
author_association: OWNER
body:
  Here's an annoying error:
  ```
  >       response4 = await ds_client.post(
              "/-/logout",
              csrftoken_from=True,
              cookies={"ds_actor": ds_client.actor_cookie({"id": "test"})},
          )

  tests/test_auth.py:88: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  self = <datasette.app.DatasetteClient object at 0x1033cc4f0>, path = '/-/logout'
  kwargs = {'cookies': {'ds_actor': 'eyJhIjp7ImlkIjoidGVzdCJ9fQ.fuFCTJG5XE-RNnUM7dcnXx9sPvE'}, 'csrftoken_from': True}, client = <httpx.AsyncClient object at 0x10347a9b0>

      async def post(self, path, **kwargs):
          await self.ds.invoke_startup()
          async with httpx.AsyncClient(app=self.app) as client:
  >           return await client.post(self._fix(path), **kwargs)
  E           TypeError: AsyncClient.post() got an unexpected keyword argument 'csrftoken_from'
  ```
  I need an alternative to the `csrftoken_from` mechanism I built for `TestClient`:

  https://github.com/simonw/datasette/blob/0b68996cc511b3a801f0cd0157bd66332d75f46f/datasette/utils/testing.py#L77-L103

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1353765125
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QsNEF
created_at: 2022-12-15T22:00:04Z
updated_at: 2022-12-15T22:00:04Z
author_association: OWNER
body:
  I'm going to punt on that for the moment and continue to use `app_client` for tests that use that mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1353805839
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QsXAP
created_at: 2022-12-15T22:38:37Z
updated_at: 2022-12-15T22:38:37Z
author_association: OWNER
body:
  I'm going to make `.status_code` work on `TestClient` response too, so I don't have to worry about using both `status` or `status_code` depending on which kind of object I am using.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1353812913
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QsYux
created_at: 2022-12-15T22:48:54Z
updated_at: 2022-12-15T22:48:54Z
author_association: OWNER
body:
  This is all very broken:
  ```
  % pytest -x --pdb
  ================================================================================== test session starts ==================================================================================
  platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0
  SQLite: 3.39.4
  rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
  plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0
  asyncio: mode=strict
  collected 1295 items                                                                                                                                                                    

  tests/test_package.py ..                                                                                                                                                          [  0%]
  tests/test_cli.py .                                                                                                                                                               [  0%]
  tests/test_cli_serve_get.py ..                                                                                                                                                    [  0%]
  tests/test_cli.py .                                                                                                                                                               [  0%]
  tests/test_black.py .                                                                                                                                                             [  0%]
  tests/test_api.py E
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  fixturedef = <FixtureDef argname='event_loop' scope='session' baseid='tests'>, request = <SubRequest 'event_loop' for <Function test_homepage>>

      @pytest.hookimpl(hookwrapper=True)
      def pytest_fixture_setup(
          fixturedef: FixtureDef, request: SubRequest
      ) -> Optional[object]:
          """Adjust the event loop policy when an event loop is produced."""
          if fixturedef.argname == "event_loop":
              outcome = yield
  >           loop = outcome.get_result()

  /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/pytest_asyncio/plugin.py:377: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
  /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: in event_loop
      return asyncio.get_event_loop()
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

  self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x111e34670>

      def get_event_loop(self):
          """Get the event loop for the current context.
      
          Returns an instance of EventLoop or raises an exception.
          """
          if (self._local._loop is None and
                  not self._local._set_called and
                  threading.current_thread() is threading.main_thread()):
              self.set_event_loop(self.new_event_loop())
      
          if self._local._loop is None:
  >           raise RuntimeError('There is no current event loop in thread %r.'
                                 % threading.current_thread().name)
  E           RuntimeError: There is no current event loop in thread 'MainThread'.

  /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py:656: RuntimeError
  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  > /Users/simon/.pyenv/versions/3.10.3/lib/python3.10/asyncio/events.py(656)get_event_loop()
  -> raise RuntimeError('There is no current event loop in thread %r.'
  (Pdb) q


  =================================================================================== warnings summary ====================================================================================
  tests/test_cli.py::test_inspect_cli_writes_to_file
  tests/test_cli.py::test_inspect_cli
    /Users/simon/Dropbox/Development/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop
      loop = asyncio.get_event_loop()

  tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error
  tests/test_cli_serve_get.py::test_serve_with_get
    /Users/simon/Dropbox/Development/datasette/datasette/cli.py:596: DeprecationWarning: There is no current event loop
      asyncio.get_event_loop().run_until_complete(ds.invoke_startup())

  tests/test_cli_serve_get.py::test_serve_with_get_exit_code_for_error
  tests/test_cli_serve_get.py::test_serve_with_get
    /Users/simon/Dropbox/Development/datasette/datasette/cli.py:599: DeprecationWarning: There is no current event loop
      asyncio.get_event_loop().run_until_complete(check_databases(ds))

  tests/test_api.py::test_homepage
    /Users/simon/Dropbox/Development/datasette/tests/conftest.py:30: DeprecationWarning: There is no current event loop
      return asyncio.get_event_loop()

  -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
  ================================================================================ short test summary info ================================================================================
  ERROR tests/test_api.py::test_homepage - RuntimeError: There is no current event loop in thread 'MainThread'.
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! _pytest.outcomes.Exit: Quitting debugger !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ======================================================================== 7 passed, 7 warnings, 1 error in 19.15s ========================================================================
  (datasette) datasette % 
  ```
  The problem looks to be caused by this:

  https://github.com/simonw/datasette/blob/87737aa1ace82fa7b54c60c41471ec9a661f5299/tests/conftest.py#L28-L30

  Which I found necessary in order to have `async def` fixtures that could be shared on the `scope="session"` basis.

  Can I work around this, or is `scope="session"` for async fixtures incompatible with my test suite for some reason?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1353977605
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5QtA8F
created_at: 2022-12-16T00:38:23Z
updated_at: 2022-12-16T00:38:23Z
author_association: OWNER
body:
  Confirmed, I just got the same result:
  ```
  - % docker run datasetteproject/datasette pip install datasette-upload-csvs
  ~ % docker commit $(docker ps -lq) datasette-with-plugins
  sha256:8cde4a6357b9221d6f9e15887a314f2b4d9d1b87b517764d207ccbaec7c0a69f
  ~ % docker run -p 8001:8001 -v $(pwd):/mnt datasette-with-plugins datasette --root -p 8001 -h 0.0.0.0
  INFO:     Started server process [1]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)



  ^CINFO:     Shutting down
  INFO:     Waiting for application shutdown.
  INFO:     Application shutdown complete.
  INFO:     Finished server process [1]
  http://0.0.0.0:8001/-/auth-token?token=4bd70fdbca215ea55c874eaf889adf8c09f2a00231f7e5e6d0470f3176407a98
  ```
  Note how the auth-token URL is only displayed after you hit `Ctrl+C`!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1354008688
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5QtIhw
created_at: 2022-12-16T00:45:40Z
updated_at: 2022-12-16T00:45:40Z
author_association: OWNER
body:
  The fix may just be to switch to `click.echo()`. https://click.palletsprojects.com/en/8.1.x/api/#click.echo

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1958#issuecomment-1354019543
issue_url: https://api.github.com/repos/simonw/datasette/issues/1958
node_id: IC_kwDOBm6k_c5QtLLX
created_at: 2022-12-16T00:54:11Z
updated_at: 2022-12-16T00:54:11Z
author_association: OWNER
body:
  To test the fix, I did a fresh checkout of `simonw/datasette` and edited `Dockerfile` to look like this:
  ```dockerfile
  FROM python:3.11.0-slim-bullseye as build

  RUN apt-get update && \
      apt-get install -y --no-install-recommends libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/apt && \
      rm -rf /var/lib/dpkg/info/*

  RUN pip install https://github.com/simonw/datasette/archive/refs/heads/main.zip && \
      find /usr/local/lib -name '__pycache__' | xargs rm -r && \
      rm -rf /root/.cache/pip

  EXPOSE 8001
  CMD ["datasette"]
  ```
  Then I built it like this:
  ```
  docker build -f Dockerfile -t datasette-pre-click .
  ```
  And ran like this:
  ```
  docker run -p 8001:8001 -v $(pwd):/mnt datasette-pre-click datasette --root -p 8001 -h 0.0.0.0
  ```
  This exhibited the same problem.

  Then I pushed a changed to branch and changed the line to:
  ```
  RUN pip install https://github.com/simonw/datasette/archive/refs/heads/click-echo-root.zip && \
  ```
  Ran this:
  ```
  docker build -f Dockerfile -t datasette-post-click .
  ```
  And this:
  ```
  docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0
  ```
  It fixed the problem!
  ```
  datasette % docker run -p 8001:8001 -v $(pwd):/mnt datasette-post-click datasette --root -p 8001 -h 0.0.0.0
  http://0.0.0.0:8001/-/auth-token?token=6542dcf5c8f34f8d13f4af9ce728359c602469efb54029098562bd06c87ad26d
  INFO:     Started server process [1]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1354036967
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QtPbn
created_at: 2022-12-16T01:10:12Z
updated_at: 2022-12-16T01:10:12Z
author_association: OWNER
body:
  If it does turn out that I can't use `scope="session"` on this fixture it might not actually be a showstopper: I can take advantage of the fact that `memory_name="..."` databases stay present in memory for the duration of the process, so I could have it such that each test that uses the `ds_client` fixture DOES construct a fresh `Datasette` instance, but doesn't need to populate the database since they can re-use the in-memory database from the previous object.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1354046627
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QtRyj
created_at: 2022-12-16T01:23:21Z
updated_at: 2022-12-16T01:23:21Z
author_association: OWNER
body:
  This does seem to help:
  ```diff
  diff --git a/tests/conftest.py b/tests/conftest.py
  index 1306c407..af9c7696 100644
  --- a/tests/conftest.py
  +++ b/tests/conftest.py
  @@ -25,12 +25,7 @@ UNDOCUMENTED_PERMISSIONS = {
   }
   
   
  -@pytest.fixture(scope="session")
  -def event_loop():
  -    return asyncio.get_event_loop()
  -
  -
  -@pytest_asyncio.fixture(scope="session")
  +@pytest_asyncio.fixture
   async def ds_client():
       from datasette.app import Datasette
       from .fixtures import METADATA, PLUGINS_DIR
  @@ -53,10 +48,11 @@ async def ds_client():
       db = ds.add_memory_database("fixtures")
   
       def prepare(conn):
  -        conn.executescript(TABLES)
  -        for sql, params in TABLE_PARAMETERIZED_SQL:
  -            with conn:
  -                conn.execute(sql, params)
  +        if not conn.execute("select count(*) from sqlite_master").fetchone()[0]:
  +            conn.executescript(TABLES)
  +            for sql, params in TABLE_PARAMETERIZED_SQL:
  +                with conn:
  +                    conn.execute(sql, params)
   
       await db.execute_write_fn(prepare)
       return ds.client
  diff --git a/tests/plugins/my_plugin_2.py b/tests/plugins/my_plugin_2.py
  index 4f7bf08c..d588342c 100644
  --- a/tests/plugins/my_plugin_2.py
  +++ b/tests/plugins/my_plugin_2.py
  @@ -117,7 +117,12 @@ def actor_from_request(datasette, request):
   def permission_allowed(datasette, actor, action):
       # Testing asyncio version of permission_allowed
       async def inner():
  -        assert 2 == (await datasette.get_database().execute("select 1 + 1")).first()[0]
  +        assert (
  +            2
  +            == (
  +                await datasette.get_database("_internal").execute("select 1 + 1")
  +            ).first()[0]
  +        )
           if action == "this_is_allowed_async":
               return True
           elif action == "this_is_denied_async":
  ```
  `pytest -m ds_client` now passes 134 tests.

  Need to get `pytest -n auto` passing too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1354053151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QtTYf
created_at: 2022-12-16T01:33:22Z
updated_at: 2022-12-16T01:33:22Z
author_association: OWNER
body:
  The thing with `Datasette(memory=False)` is tripping me up.

  The problem is that the tests written against `app_client` - which I want to replace - all assume that there is no `_memory` database, because when you start Datasette with at least one database file it doesn't enable `_memory` unless you explicitly tell it to.

  But the new `ds_client` fixture works by creating a named in-memory database called `fixtures`, which it does with a call to `ds.add_memory_database("fixtures")` after the object has been instantiated.

  This results in a datasette instance that DOES have a `_memory` database, when we didn't want one.

  My initial solution attempt was a huge hack - I decided that if you pass `memory=False` to the `Datasette` constructor it should mean "don't add a `_memory` database even though I didn't pass any files". I set a the default `memory` argument to `None`.

  This is weird and surprising (`memory=False` no does something different from `memory=None`?) and I found other tests that it broke, like this one:

  ```python
  def test_sql_errors_logged_to_stderr():
      runner = CliRunner(mix_stderr=False)
      result = runner.invoke(cli, ["--get", "/_memory.json?sql=select+blah"])
      assert result.exit_code == 1
      assert "sql = 'select blah', params = {}: no such column: blah\n" in result.stderr
  ```
  It ended up with no `_memory` database because it turns out `datasette serve ...` passes `memory=False` without me realizing it.

  So I'm going to undo that hack and teach the fixture to do this instead:

  ```python
  db = ds.add_memory_database("fixtures")
  ds.remove_database("_memory")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1354061440
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QtVaA
created_at: 2022-12-16T01:45:38Z
updated_at: 2022-12-16T01:45:38Z
author_association: OWNER
body:
  I'm going to do `test_table_html.py` next.

  Currently: 68 passed in 17.20s

  Will this speed it up?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1354062939
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QtVxb
created_at: 2022-12-16T01:48:06Z
updated_at: 2022-12-17T21:40:43Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.19**% // Head: **92.05**% // Decreases project coverage by **`-0.13%`** :warning:
  > Coverage data is based on head [(`770879a`)](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`0b68996`)](https://codecov.io/gh/simonw/datasette/commit/0b68996cc511b3a801f0cd0157bd66332d75f46f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  > :exclamation: Current head 770879a differs from pull request most recent head f42bca8. Consider uploading reports for the commit f42bca8 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1960      +/-   ##
  ==========================================
  - Coverage   92.19%   92.05%   -0.14%     
  ==========================================
    Files          38       38              
    Lines        5521     5527       +6     
  ==========================================
  - Hits         5090     5088       -2     
  - Misses        431      439       +8     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.13% <100.00%> (-0.57%)` | :arrow_down: |
  | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.83% <100.00%> (+0.24%)` | :arrow_up: |
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: |
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1960/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.57% <0.00%> (-0.61%)` | :arrow_down: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1960?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1354148139
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Qtqkr
created_at: 2022-12-16T03:32:52Z
updated_at: 2022-12-16T03:32:52Z
author_association: OWNER
body:
  Got that done to: 68 passed in 14.92s

  By implementing my own global variable - since `pytest` won't use a global fixture for me, I decided to do it for myself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5Qttie
created_at: 2022-12-16T03:51:19Z
updated_at: 2022-12-16T03:52:13Z
author_association: NONE
body:
  Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with 

  `Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30`

  Unfortunately, `3.10.8` isn't listed in `pyenv` so I'm unable to install it.

  For me, the trigger is trying to use the `add-foreign-key` command on its own:

  ```bash
  sqlite-utils add-foreign-key library.db book_creators creator_id creators id
  ```

  Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/235
node_id: IC_kwDOCGYnMM5Qt1Uo
created_at: 2022-12-16T04:35:30Z
updated_at: 2022-12-16T04:35:38Z
author_association: NONE
body:
  A related historical problem:

  https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720

  I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1354269873
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QuISx
created_at: 2022-12-16T06:11:43Z
updated_at: 2022-12-16T06:11:43Z
author_association: OWNER
body:
  This is quite fast:
  ```
  % pytest -m ds_client -n auto
  ================================================================================== test session starts ==================================================================================
  platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0
  SQLite: 3.39.4
  rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
  plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0
  asyncio: mode=strict
  gw0 [291] / gw1 [291] / gw2 [291] / gw3 [291] / gw4 [291] / gw5 [291]
  ................................................................................................................................................................................. [ 60%]
  ..................................................................................................................                                                                [100%]
  ================================================================================== 291 passed in 6.30s ==================================================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355313058
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QyG-i
created_at: 2022-12-16T17:53:17Z
updated_at: 2022-12-16T17:53:17Z
author_association: OWNER
body:
  Got some surprising test failures here: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336626
  ```
  >       assert response.json() == [{"id": 1, "name": "San Francisco"}]
  E       AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}]
  E         Full diff:
  E         - [{'id': 1, 'name': 'San Francisco'}]
  E         + {'error': 'Table not found: 𝐜𝐢𝐭𝐢𝐞𝐬', 'ok': False, 'status': 404, 'title': None}
  ```
  A hunch: this failure suggests that maybe the fixtures tables were not correctly created when this test run. Maybe that can happen when `python -n auto` runs a bunch of separate processes and hence one of the tests randomly gets run in a fresh process and executes before the in-memory fixtures database has been fully populated.

  I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355319541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QyIj1
created_at: 2022-12-16T17:58:24Z
updated_at: 2022-12-16T17:58:46Z
author_association: OWNER
body:
  > I tried adding `invoke_startup()` to the `ds_client()` fixture to see if that would fix this.

  It did not: I'm still seeing those same failures. Frustrating: https://github.com/simonw/datasette/actions/runs/3715317653/jobs/6300336884

      ====== 11 failed, 1252 passed, 1 skipped, 1 warning in 185.77s (0:03:05) =======

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355445710
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QynXO
created_at: 2022-12-16T19:06:49Z
updated_at: 2022-12-16T19:09:35Z
author_association: OWNER
body:
  This would be much easier to debug if I could use Docker to run the GitHub Actions image directly on my own laptop.

  https://github.com/actions/runner-images/blob/releases/ubuntu22/20221212/images/linux/Ubuntu2204-Readme.md is the README for their most recent image. Not sure if there's an easy way to run it in Docker though.

  https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md is instructions for building them locally - looks fiddly though, involves https://www.packer.io/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355471341
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Qytnt
created_at: 2022-12-16T19:23:33Z
updated_at: 2022-12-16T19:23:33Z
author_association: OWNER
body:
  Trying this instead:
  ```
  docker run -it ubuntu:22.04 /bin/bash
  ```
  Then in that shell:
  ```
  apt-get update
  apt-get install python3.11 python3.11-venv git -y
  cd /tmp
  git clone https://github.com/simonw/datasette
  cd datasette
  git checkout async-tests
  python3.11 -m venv venv
  pip install -e '.[test]'
  pytest
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QyurX
created_at: 2022-12-16T19:26:08Z
updated_at: 2022-12-16T19:26:08Z
author_association: OWNER
body:
  Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI:

  ```
  ======== short test summary info ========
  FAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5
  FAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error'
  FAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']]
  FAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance'
  FAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}]
  FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...]
  FAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip'
  FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...>1</a></td>']]
  FAILED tests/test_table_html.py::test_view_html - assert <a href="/fixtures/simple_view?_size=3&amp;_sort=upper_content" rel="nofollow">upper_content</a> is None
  FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This &lt;em&...t; is escaped' == 'simple_primary_key'
  FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name']
  FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name']
  FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll'
  ======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ========
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5QyvbX
created_at: 2022-12-16T19:27:12Z
updated_at: 2022-12-16T19:27:12Z
author_association: OWNER
body:
  Bad news: they're definitely caused by tests that are subtly affected by other tests.

  This passes without errors:

      pytest -k test_paginate_tables_and_views

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Qzh_E
created_at: 2022-12-16T21:50:01Z
updated_at: 2022-12-16T21:50:01Z
author_association: OWNER
body:
  Looks like that `@pytest.mark.ds_client` mark I've been using isn't necessary - I added that so I could easily run `pytest -m ds_client` to execute all tests that I had ported to the new feature, but actually this achieves the same thing:

      pytest -k ds_client

  So I'm going to remove the mark.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595
issue_url: https://api.github.com/repos/simonw/datasette/issues/1962
node_id: IC_kwDOBm6k_c5QzjZL
created_at: 2022-12-16T21:53:45Z
updated_at: 2022-12-16T21:55:29Z
author_association: OWNER
body:
  Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST:

  https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/datasette/utils/testing.py#L90-L103

  One option would be adding that mechanism to `datasette.client.post(...)` - as a `_csrf_from=` parameter (with an underscore because it's mainly intended for use in tests, though perhaps that's a weird convention that I shouldn't introduce).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Q0Dic
created_at: 2022-12-16T23:36:07Z
updated_at: 2022-12-16T23:36:07Z
author_association: OWNER
body:
  I ran `pytest --pdb -x` to drop into the debugger on the first failing test.

  ```
          assert expected_rows == len(fetched)
  >       assert expected_pages == count
  E       assert 9 == 5
  ```
  That's in `test_paginate_tables_and_views()`.

  On a hunch, I checked the current settings:

  ```
  (Pdb) ds_client.ds._settings
  {'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'}
  ```
  Looks like something changed `default_page_size` to 50 and forgot to change it back!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1356038242
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Q04Bi
created_at: 2022-12-17T04:56:22Z
updated_at: 2022-12-17T04:57:04Z
author_association: OWNER
body:
  May have spotted the problem with that `test_paginate_tables_and_views` test:

  ```
  (Pdb) path, expected_rows, expected_pages
  (None, 201, 9)
  ```
  `path` should not be `None` here:

  https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L175-L212

  No that's not it either - `path` is reassigned on purpose.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1356056018
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Q08XS
created_at: 2022-12-17T05:33:26Z
updated_at: 2022-12-17T05:33:26Z
author_association: OWNER
body:
  I think I've found the problem. The failing test is this one:

  `paginated_view.json-201-9`

  That's this example:

  https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L179-L180

  Why is it expected to take 9 pages and not 5, when the definition of that view is this:

  ```sql
  CREATE VIEW paginated_view AS
      SELECT
          content,
          '- ' || content || ' -' AS content_extra
      FROM no_primary_key;
  ```
  Because `paginated_view` has extra configuration in `metadata.json`:

  https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/fixtures.py#L357

  So this test is showing that `metadata` can be used to set an alternative page size for a view.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1356476583
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Q2jCn
created_at: 2022-12-17T21:34:51Z
updated_at: 2022-12-17T21:34:51Z
author_association: OWNER
body:
  These are all the places that tests touch `_metadata_local` at the moment:
  ```
  (venv) root@76a81d2417f5:/tmp/datasette/tests# rg _metadata_local         
  test_facets.py
  596:    ds._metadata_local = {
  605:    ds._metadata_local["databases"]["test_facet_size"]["tables"]["neighbourhoods"][

  test_permissions.py
  62:    padlock_client.ds._metadata_local["allow"] = allow
  77:    del padlock_client.ds._metadata_local["allow"]
  522:        cascade_app_client.ds._metadata_local = updated_metadata
  533:        cascade_app_client.ds._metadata_local = previous_metadata
  549:    previous_metadata = cascade_app_client.ds._metadata_local
  551:        cascade_app_client.ds._metadata_local = metadata
  566:        cascade_app_client.ds._metadata_local = previous_metadata
  842:    perms_ds._metadata_local = updated_metadata
  849:        perms_ds._metadata_local = previous_metadata

  test_html.py
  1114:    orig = ds_client.ds._metadata_local
  1115:    ds_client.ds._metadata_local = metadata
  1123:        ds_client.ds._metadata_local = orig

  test_plugins.py
  1034:    ds_client.ds._metadata_local = {
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1960#issuecomment-1356476886
issue_url: https://api.github.com/repos/simonw/datasette/issues/1960
node_id: IC_kwDOBm6k_c5Q2jHW
created_at: 2022-12-17T21:37:05Z
updated_at: 2022-12-17T21:37:05Z
author_association: OWNER
body:
  I think this test may be to blame:

  https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/test_plugins.py#L950-L972

  It's over-riding `_metadata_local` and then failing to set it back to original in a `finally:` block at the end.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1959#issuecomment-1356478792
issue_url: https://api.github.com/repos/simonw/datasette/issues/1959
node_id: IC_kwDOBm6k_c5Q2jlI
created_at: 2022-12-17T21:49:36Z
updated_at: 2022-12-17T21:49:36Z
author_association: OWNER
body:
  Made a really good start on this in the just-merged PR:
  - #1960

  The follow-up work will happen in:
  - #1962

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1356480256
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5Q2j8A
created_at: 2022-12-17T22:01:13Z
updated_at: 2022-12-17T22:01:13Z
author_association: OWNER
body:
  The refactor in the following issue did NOT prevent this error from occurring when I try to run the full `pytest` suite on my Mac laptop:

  - #1959

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1843#issuecomment-1356481595
issue_url: https://api.github.com/repos/simonw/datasette/issues/1843
node_id: IC_kwDOBm6k_c5Q2kQ7
created_at: 2022-12-17T22:02:40Z
updated_at: 2022-12-17T22:02:40Z
author_association: OWNER
body:
  This is interesting:

  ```python
  import resource
  print(resource.getrlimit(resource.RLIMIT_NOFILE))
  ```
  On Mac:
  ```pycon
  >>> print(resource.getrlimit(resource.RLIMIT_NOFILE))
  (256, 9223372036854775807)
  ```
  On Ubuntu (in a Docker `ubuntu:22.04` container):
  ```pycon
  >>> resource.getrlimit(resource.RLIMIT_NOFILE)
  (1048576, 1048576)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356489200
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q2mHw
created_at: 2022-12-17T22:29:51Z
updated_at: 2022-12-17T22:29:51Z
author_association: OWNER
body:
  No, it still causes the tests to hang (I let them run for 12 minutes):

  <img width="664" alt="image" src="https://user-images.githubusercontent.com/9599/208268223-7fb40e3f-833b-4204-9acb-079adce6452f.png">

  Interesting that the regular tests passed an then the `pytest -m serial` ones seem to have failed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356595665
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3AHR
created_at: 2022-12-18T00:58:16Z
updated_at: 2022-12-18T00:58:16Z
author_association: OWNER
body:
  `pytest -m serial` on my Mac laptop also freezes:

  ```
  (datasette) datasette % pytest -m serial
  ======================================================= test session starts ========================================================
  platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0
  SQLite: 3.39.4
  rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
  plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0
  asyncio: mode=strict
  collected 1295 items / 1264 deselected / 31 selected                                                                               

  tests/test_package.py .                                                                                                      [  3%]
  tests/test_cli_serve_server.py 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356596740
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3AYE
created_at: 2022-12-18T00:59:47Z
updated_at: 2022-12-18T00:59:47Z
author_association: OWNER
body:
  Hitting `Ctrl+C` while using `--full-trace` gave me more clues:

  ```
  % pytest -m serial tests/test_cli_serve_server.py --full-trace
  ======================================================= test session starts ========================================================
  platform darwin -- Python 3.10.3, pytest-7.1.3, pluggy-1.0.0
  SQLite: 3.39.4
  rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
  plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, asyncio-0.19.0, timeout-2.1.0, profiling-1.7.0
  asyncio: mode=strict
  collected 3 items                                                                                                                  

  tests/test_cli_serve_server.py ^C^C

  ====================================================== no tests ran in 3.49s =======================================================
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
      yield
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/httpcore/backends/sync.py", line 86, in connect_tcp
      sock = socket.create_connection(
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 845, in create_connection
      raise err
    File "/Users/simon/.pyenv/versions/3.10.3/lib/python3.10/socket.py", line 833, in create_connection
      sock.connect(sa)
  ConnectionRefusedError: [Errno 61] Connection refused
  [...]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356599930
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3BJ6
created_at: 2022-12-18T01:01:47Z
updated_at: 2022-12-18T01:01:47Z
author_association: OWNER
body:
  I think that's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L6-L13

  Using this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L155-L175

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356600917
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3BZV
created_at: 2022-12-18T01:02:26Z
updated_at: 2022-12-18T01:02:26Z
author_association: OWNER
body:
  This bit here looks like it could hang!
  ```python
       # Loop until port 8041 serves traffic 
       while True: 
           try: 
               httpx.get("http://localhost:8041/") 
               break 
           except httpx.ConnectError: 
               time.sleep(0.1) 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356609095
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3DZH
created_at: 2022-12-18T01:10:43Z
updated_at: 2022-12-18T01:10:43Z
author_association: OWNER
body:
  Improved version of that fixture:
  ```diff
  diff --git a/tests/conftest.py b/tests/conftest.py
  index 44c44f87..69dee68b 100644
  --- a/tests/conftest.py
  +++ b/tests/conftest.py
  @@ -27,6 +27,17 @@ UNDOCUMENTED_PERMISSIONS = {
   _ds_client = None
   
   
  +def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs):
  +    start = time.time()
  +    while time.time() - start < timeout:
  +        try:
  +            client.get(url, **kwargs)
  +            return
  +        except httpx.ConnectError:
  +            time.sleep(0.1)
  +    raise AssertionError("Timed out waiting for {} to respond".format(url))
  +
  +
   @pytest_asyncio.fixture
   async def ds_client():
       from datasette.app import Datasette
  @@ -161,13 +172,7 @@ def ds_localhost_http_server():
           # Avoid FileNotFoundError: [Errno 2] No such file or directory:
           cwd=tempfile.gettempdir(),
       )
  -    # Loop until port 8041 serves traffic
  -    while True:
  -        try:
  -            httpx.get("http://localhost:8041/")
  -            break
  -        except httpx.ConnectError:
  -            time.sleep(0.1)
  +    wait_until_responds("http://localhost:8041/")
       # Check it started successfully
       assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
       yield ds_proc
  @@ -202,12 +207,7 @@ def ds_localhost_https_server(tmp_path_factory):
           stderr=subprocess.STDOUT,
           cwd=tempfile.gettempdir(),
       )
  -    while True:
  -        try:
  -            httpx.get("https://localhost:8042/", verify=client_cert)
  -            break
  -        except httpx.ConnectError:
  -            time.sleep(0.1)
  +    wait_until_responds("http://localhost:8042/", verify=client_cert)
       # Check it started successfully
       assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
       yield ds_proc, client_cert
  @@ -231,12 +231,7 @@ def ds_unix_domain_socket_server(tmp_path_factory):
       # Poll until available
       transport = httpx.HTTPTransport(uds=uds)
       client = httpx.Client(transport=transport)
  -    while True:
  -        try:
  -            client.get("http://localhost/_memory.json")
  -            break
  -        except httpx.ConnectError:
  -            time.sleep(0.1)
  +    wait_until_responds("http://localhost/_memory.json", client=client)
       # Check it started successfully
       assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
       yield ds_proc, uds
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356610089
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3Dop
created_at: 2022-12-18T01:12:39Z
updated_at: 2022-12-18T01:12:39Z
author_association: OWNER
body:
  ... and it turns out those tests saved me. Because I forgot to check if `datasette` would actually start a server correctly!

  ```
   % datasette fixtures.db -p 8852
  INFO:     Started server process [3538]
  INFO:     Waiting for application startup.
  ERROR:    Exception in 'lifespan' protocol
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/lifespan/on.py", line 86, in main
      await app(scope, self.receive, self.send)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
      return await self.app(scope, receive, send)
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/asgi.py", line 437, in __call__
      return await self.asgi(scope, receive, send)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/asgi_csrf.py", line 39, in app_wrapped_with_csrf
      await app(scope, receive, send)
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1457, in __call__
      path = scope["path"]
  KeyError: 'path'
  ERROR:    Application startup failed. Exiting.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356618913
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3Fyh
created_at: 2022-12-18T01:29:05Z
updated_at: 2022-12-18T01:29:05Z
author_association: OWNER
body:
  Now the only failure is in the `https` test - which fails like this (in CI and on my laptop):

  ```
              message = str(exc)
  >           raise mapped_exc(message) from exc
  E           httpx.RemoteProtocolError: Server disconnected without sending a response.

  /opt/hostedtoolcache/Python/3.11.1/x64/lib/python3.11/site-packages/httpx/_transports/default.py:77: RemoteProtocolError
  =========================== short test summary info ============================
  ERROR tests/test_cli_serve_server.py::test_serve_localhost_https - httpx.RemoteProtocolError: Server disconnected without sending a response.
  ================= 30 passed, 1264 deselected, 1 error in 6.15s =================
  ```
  That's this test: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/test_cli_serve_server.py#L16-L24

  And this fixture: https://github.com/simonw/datasette/blob/63fb750f39cac6f49b451387fdff659ecd9edc5c/tests/conftest.py#L178-L215

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356620233
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3GHJ
created_at: 2022-12-18T01:31:10Z
updated_at: 2022-12-18T01:31:10Z
author_association: OWNER
body:
  During the polling loop it constantly raises:

  `httpx.RemoteProtocolError`: Server disconnected without sending a response

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356625556
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3HaU
created_at: 2022-12-18T02:00:18Z
updated_at: 2022-12-18T02:00:18Z
author_association: OWNER
body:
  Maybe the reason the ASGI lifespan stuff broke was this line: https://github.com/simonw/datasette/blob/8b73fc6b47dffd8836f5c58aae1e57c1f66a5754/datasette/cli.py#L630-L632

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356626334
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3Hme
created_at: 2022-12-18T02:04:01Z
updated_at: 2022-12-18T02:04:07Z
author_association: OWNER
body:
  I used the steps to test manually from this comment: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052

  In one terminal:
  ```
  cd /tmp
  python -m trustme
  datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem -p 8003
  ```
  Then in another terminal:
  ```
  curl --cacert /tmp/client.pem 'https://localhost:8003/_memory.json'
  ```
  This worked correctly, outputting the expected JSON.

  So the feature still works, it's just the test that is broken for some reason.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356627331
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3H2D
created_at: 2022-12-18T02:11:17Z
updated_at: 2022-12-18T02:11:17Z
author_association: OWNER
body:
  This issue might be relevant, but I tried the suggested fix in there (`Connection: close` on the incoming requests) and it didn't fix my problem:
  - https://github.com/encode/httpx/discussions/2056

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356627931
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3H_b
created_at: 2022-12-18T02:13:01Z
updated_at: 2022-12-18T02:13:01Z
author_association: OWNER
body:
  Rather than continue to bang my head against this, I'm tempted to rewrite this test to happen outside of Python world - in a bash script run by GitHub Actions, for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356629783
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3IcX
created_at: 2022-12-18T02:18:43Z
updated_at: 2022-12-18T02:18:43Z
author_association: OWNER
body:
  Various attempts at a fix which didn't work:

  ```diff
  diff --git a/tests/conftest.py b/tests/conftest.py
  index 69dee68b..899d36fd 100644
  --- a/tests/conftest.py
  +++ b/tests/conftest.py
  @@ -1,4 +1,3 @@
  -import asyncio
   import httpx
   import os
   import pathlib
  @@ -6,6 +5,7 @@ import pytest
   import pytest_asyncio
   import re
   import subprocess
  +import sys
   import tempfile
   import time
   import trustme
  @@ -27,13 +27,23 @@ UNDOCUMENTED_PERMISSIONS = {
   _ds_client = None
   
   
  -def wait_until_responds(url, timeout=5.0, client=httpx, **kwargs):
  +def wait_until_responds(url, timeout=5.0, client=None, **kwargs):
  +    client = client or httpx.Client(**kwargs)
       start = time.time()
       while time.time() - start < timeout:
           try:
  -            client.get(url, **kwargs)
  +            if "verify" in kwargs:
  +                print(kwargs["verify"])
  +                print(
  +                    "Contents of verify file: {}".format(
  +                        open(kwargs.get("verify")).read()
  +                    )
  +                )
  +                print("client = {}, kwargs = {}".format(client, kwargs))
  +            client.get(url)
               return
  -        except httpx.ConnectError:
  +        except (httpx.ConnectError, httpx.RemoteProtocolError) as ex:
  +            print(ex)
               time.sleep(0.1)
       raise AssertionError("Timed out waiting for {} to respond".format(url))
   
  @@ -166,7 +176,7 @@ def check_permission_actions_are_documented():
   @pytest.fixture(scope="session")
   def ds_localhost_http_server():
       ds_proc = subprocess.Popen(
  -        ["datasette", "--memory", "-p", "8041"],
  +        [sys.executable, "-m", "datasette", "--memory", "-p", "8041"],
           stdout=subprocess.PIPE,
           stderr=subprocess.STDOUT,
           # Avoid FileNotFoundError: [Errno 2] No such file or directory:
  @@ -180,7 +190,7 @@ def ds_localhost_http_server():
       ds_proc.terminate()
   
   
  -@pytest.fixture(scope="session")
  +@pytest.fixture
   def ds_localhost_https_server(tmp_path_factory):
       cert_directory = tmp_path_factory.mktemp("certs")
       ca = trustme.CA()
  @@ -194,6 +204,8 @@ def ds_localhost_https_server(tmp_path_factory):
       ca.cert_pem.write_to_path(path=client_cert)
       ds_proc = subprocess.Popen(
           [
  +            sys.executable,
  +            "-m",
               "datasette",
               "--memory",
               "-p",
  @@ -207,7 +219,11 @@ def ds_localhost_https_server(tmp_path_factory):
           stderr=subprocess.STDOUT,
           cwd=tempfile.gettempdir(),
       )
  -    wait_until_responds("http://localhost:8042/", verify=client_cert)
  +    wait_until_responds(
  +        "http://localhost:8042/_memory.json",
  +        verify=client_cert,
  +        headers={"Connection": "close"},
  +    )
       # Check it started successfully
       assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
       yield ds_proc, client_cert
  diff --git a/tests/test_cli_serve_server.py b/tests/test_cli_serve_server.py
  index 1c31e2a3..9320b623 100644
  --- a/tests/test_cli_serve_server.py
  +++ b/tests/test_cli_serve_server.py
  @@ -16,7 +16,11 @@ def test_serve_localhost_http(ds_localhost_http_server):
   @pytest.mark.serial
   def test_serve_localhost_https(ds_localhost_https_server):
       _, client_cert = ds_localhost_https_server
  -    response = httpx.get("https://localhost:8042/_memory.json", verify=client_cert)
  +    response = httpx.get(
  +        "https://localhost:8042/_memory.json",
  +        verify=client_cert,
  +        headers={"Connection": "close"},
  +    )
       assert {
           "database": "_memory",
           "path": "/_memory",
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356630092
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3IhM
created_at: 2022-12-18T02:20:01Z
updated_at: 2022-12-18T02:20:01Z
author_association: OWNER
body:
  Asked ChatGPT:

  > Write a bash script which starts a server in the background using "datasette -p 8002", then uses curl to make a test request against it, then shuts the server down again at the end

  It gave me:

  ```bash
  #!/bin/bash

  # Start the server in the background
  datasette -p 8002 &

  # Store the background process ID in a variable
  server_pid=$!

  # Make a test request using curl
  curl http://localhost:8002

  # Shut down the server
  kill $server_pid
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356633937
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3JdR
created_at: 2022-12-18T02:28:22Z
updated_at: 2022-12-18T02:28:22Z
author_association: OWNER
body:
  This `bash` script does the job:
  ```bash
  #!/bin/bash

  # Generate certificates
  python -m trustme
  # This creates server.pem, server.key, client.pem

  # Start the server in the background
  datasette --memory \
      --ssl-keyfile=server.key \
      --ssl-certfile=server.pem \
      -p 8152 &

  # Store the background process ID in a variable
  server_pid=$!

  # Wait for the server to start
  sleep 2

  # Make a test request using curl
  curl -f --cacert client.pem 'https://localhost:8152/_memory.json'

  # Save curl's exit code (-f option causes it to return one on HTTP errors)
  curl_exit_code=$?

  # Shut down the server
  kill $server_pid
  sleep 1

  # Clean up the certificates
  rm server.pem server.key client.pem

  echo $curl_exit_code
  exit $curl_exit_code
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1221#issuecomment-1356639873
issue_url: https://api.github.com/repos/simonw/datasette/issues/1221
node_id: IC_kwDOBm6k_c5Q3K6B
created_at: 2022-12-18T02:39:04Z
updated_at: 2022-12-18T02:39:04Z
author_association: OWNER
body:
  I ended up moving this test out of Python and into a `bash` script here: https://github.com/simonw/datasette/commit/d1d369456a7319b9de39175605568cbc9b852478

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356640266
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3LAK
created_at: 2022-12-18T02:43:00Z
updated_at: 2022-12-18T02:43:00Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/actions/runs/3722908296/jobs/6314093163 shows that new test passing in CI:

  ```
  Generated a certificate for 'localhost', '127.0.0.1', '::1'
  Configure your server to use the following files:
    cert=/home/runner/work/datasette/datasette/server.pem
    key=/home/runner/work/datasette/datasette/server.key
  Configure your client to use the following files:
    cert=/home/runner/work/datasette/datasette/client.pem
  INFO:     Started server process [4036]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on https://127.0.0.1:8152/ (Press CTRL+C to quit)
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

  INFO:     127.0.0.1:56726 - "GET /_memory.json HTTP/1.1" 200 OK
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  100   213    0   213    0     0  11542      0 --:--:-- --:--:-- --:--:-- 11833
  INFO:     Shutting down
  INFO:     Waiting for application shutdown.
  INFO:     Application shutdown complete.
  INFO:     Finished server process [4036]
  {"database": "_memory", "private": false, "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [], "allow_execute_sql": true, "table_columns": {}, "query_ms": 1.4545189999921604}0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1356640463
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q3LDP
created_at: 2022-12-18T02:45:18Z
updated_at: 2022-12-18T02:45:18Z
author_association: OWNER
body:
  ... and with this change, the following now works correctly:

  ```
  % datasette install datasette-gunicorn
  % datasette gunicorn fixtures.db -p 8855
  [2022-12-17 18:44:29 -0800] [7651] [INFO] Starting gunicorn 20.1.0
  [2022-12-17 18:44:29 -0800] [7651] [INFO] Listening at: http://127.0.0.1:8855 (7651)
  [2022-12-17 18:44:29 -0800] [7651] [INFO] Using worker: uvicorn.workers.UvicornWorker
  [2022-12-17 18:44:29 -0800] [7653] [INFO] Booting worker with pid: 7653
  [2022-12-17 18:44:29 -0800] [7653] [INFO] Started server process [7653]
  [2022-12-17 18:44:29 -0800] [7653] [INFO] Waiting for application startup.
  [2022-12-17 18:44:29 -0800] [7653] [INFO] Application startup complete.
  ```
  So this issue is now fixed!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1963#issuecomment-1356651943
issue_url: https://api.github.com/repos/simonw/datasette/issues/1963
node_id: IC_kwDOBm6k_c5Q3N2n
created_at: 2022-12-18T03:23:03Z
updated_at: 2022-12-18T03:23:03Z
author_association: OWNER
body:
  Oh that's annoying... every step in publish succeeded except the static docs one:

  <img width="1305" alt="image" src="https://user-images.githubusercontent.com/9599/208279782-8582c130-820c-48c5-a4ca-a77438570b02.png">

  https://github.com/simonw/datasette/actions/runs/3723015082/jobs/6314292722

  This means the documentation database used to update the search engine on https://datasette.io/ won't reflect the very latest changelog. I'm OK with that - I'll fix this workflow so that next time I publish a release this will work correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1771#issuecomment-1356655217
issue_url: https://api.github.com/repos/simonw/datasette/issues/1771
node_id: IC_kwDOBm6k_c5Q3Opx
created_at: 2022-12-18T03:38:16Z
updated_at: 2022-12-18T03:38:16Z
author_association: OWNER
body:
  OK I see what you mean:

  https://latest.datasette.io/fixtures/attraction_characteristic

  ![Animated GIF of the table page hitting tab a bunch - the cog icon highlights and so does the text input but the two select boxes do not](https://user-images.githubusercontent.com/9599/208280176-1e2de671-fe69-43e8-8d62-bf7aa8f4d36e.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1771#issuecomment-1356655630
issue_url: https://api.github.com/repos/simonw/datasette/issues/1771
node_id: IC_kwDOBm6k_c5Q3OwO
created_at: 2022-12-18T03:43:12Z
updated_at: 2022-12-18T03:43:12Z
author_association: OWNER
body:
  The border is actually on the div that wraps the select box:

  <img width="1171" alt="image" src="https://user-images.githubusercontent.com/9599/208280249-9d9848b5-6e33-46c3-ad99-bc1948e08f25.png">

  I tried adding a `border: 1px dotted black` to `select:focus` but it's not quite right - it jumps around a bit like this:

  ![Tabbing to the selects shows a 1px border but the element expands in size by one pixel, causing a visual jump](https://user-images.githubusercontent.com/9599/208280271-41a07f68-b8b1-4908-a4e2-aac4304d6c09.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1771#issuecomment-1356657451
issue_url: https://api.github.com/repos/simonw/datasette/issues/1771
node_id: IC_kwDOBm6k_c5Q3PMr
created_at: 2022-12-18T04:04:32Z
updated_at: 2022-12-18T04:04:32Z
author_association: NONE
body:
  the problem is:
  ```
  .select-wrapper select:focus {
    outline: none;
  }
  ```

  I sometimes add this js:
  ```
  window.addEventListener('keydown', function check_tab(e) {
      if (e.key === 'Tab') {
          document.documentElement.classList.add('user-is-tabbing')
          window.removeEventListener('keydown', check_tab)
      }
  })
  ```

  and then in the css, using a `html.user-is-tabbing` selector undo any outlines I removed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1771#issuecomment-1356680769
issue_url: https://api.github.com/repos/simonw/datasette/issues/1771
node_id: IC_kwDOBm6k_c5Q3U5B
created_at: 2022-12-18T05:56:05Z
updated_at: 2022-12-18T05:56:05Z
author_association: OWNER
body:
  This does the trick:

  ```css
  div.select-wrapper:focus-within {
    border: 1px solid black;
  }
  ```
  ![tab-select-border-fix](https://user-images.githubusercontent.com/9599/208283826-de48212f-a213-40fc-9b37-9d66f0858f21.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1964#issuecomment-1356697705
issue_url: https://api.github.com/repos/simonw/datasette/issues/1964
node_id: IC_kwDOBm6k_c5Q3ZBp
created_at: 2022-12-18T06:37:23Z
updated_at: 2022-12-18T06:37:23Z
author_association: OWNER
body:
  I'm certain the two other cog menus (the app menu on the right of the nav bar and the column action menus) have the same problem.

  Would be great to figure out the right ARIA attributes for these too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1886#issuecomment-1356842576
issue_url: https://api.github.com/repos/simonw/datasette/issues/1886
node_id: IC_kwDOBm6k_c5Q38ZQ
created_at: 2022-12-18T17:34:20Z
updated_at: 2022-12-18T17:34:20Z
author_association: NONE
body:
  A bit late to this, but I have made an app to publish air quality data in Bristol, UK. 
  [air quality data in Bristol, UK.](https://brisaq-wfzqhmj43q-ew.a.run.app/)
  Next step to see if I can make a streamlit app based on this to produce some nice charts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1955#issuecomment-1357084279
issue_url: https://api.github.com/repos/simonw/datasette/issues/1955
node_id: IC_kwDOBm6k_c5Q43Z3
created_at: 2022-12-19T04:34:16Z
updated_at: 2022-12-19T04:34:16Z
author_association: NONE
body:
  You were super-close on the python version of the test here, changing `http` to `https` on 8b73fc6b47dffd8836f5c58aae1e57c1f66a5754 is enough to pass the test:

  ```diff
  diff --git a/tests/conftest.py b/tests/conftest.py
  index 69dee68b4a3f..ba07a11d37f6 100644
  --- a/tests/conftest.py
  +++ b/tests/conftest.py
  @@ -207,7 +207,7 @@ def ds_localhost_https_server(tmp_path_factory):
           stderr=subprocess.STDOUT,
           cwd=tempfile.gettempdir(),
       )
  -    wait_until_responds("http://localhost:8042/", verify=client_cert)
  +    wait_until_responds("https://localhost:8042/", verify=client_cert)
       # Check it started successfully
       assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
       yield ds_proc, client_cert
  ```

  My speculation about what was happening here: when `wait_until_responds()` would time out due to SSL connection problems, because `.terminate()` isn’t in a `finally`, the datasette process wouldn’t get killed. That could (1) hang CI and (2) cause all your future local test runs to mysteriously fail because they’d be secretly talking to that old datasette process still hanging around from a past test run with an old temporary server certificate, and that old server cert wouldn’t validate against your newly-created ca cert.

  A `finally` for `.terminate()` would help; a fancier version could be a context manager for running the external `datasette` process that could:
    - ensure the process always exited when no longer needed
        - if you want to be fancy, call `terminate()`, `wait()` for a short timeout for the process to exit, then try `kill()` and `wait()` again; raise an exception complaining about the seemingly-unkillable process if all that fails
    - raise an error if the process exited with a non-zero error code; here it’s likely that some `datasette` executions were secretly failing with `[Errno 48] error while attempting to bind on address ('127.0.0.1', 8042): address already in use`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-1359468823
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46
node_id: IC_kwDODFdgUs5RB9kX
created_at: 2022-12-20T14:39:39Z
updated_at: 2022-12-20T14:40:15Z
author_association: NONE
body:
  Just a quick +1 to this one from me - I would like to do a better job of tracking who is reviewing one another's pull requests in repositories, since this is a specific kind of maintenance work that I think often goes unrewarded. I can't seem to figure this out just by looking at the `pull_request` or `issue_comments` tables, so I think it would be helpful to support PR reviews natively (even if just for summary statistics). Alternatively if there is a way in the API to tell if an issue comment is part of a review, then perhaps you could quickly calculate the number of unique reviews that an author performed. But that was beyond my SQL-foo :-) 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/488#issuecomment-1364141224
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/488
node_id: IC_kwDOCGYnMM5RTySo
created_at: 2022-12-23T17:38:55Z
updated_at: 2022-12-23T17:38:55Z
author_association: NONE
body:
  > text columns containing empty strings should not be rewritten to null.

  I would actually appreciate an option to do just that for text columns as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1970#issuecomment-1368264882
issue_url: https://api.github.com/repos/simonw/datasette/issues/1970
node_id: IC_kwDOBm6k_c5RjhCy
created_at: 2022-12-31T18:52:01Z
updated_at: 2022-12-31T18:52:01Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/e03aed00026cc2e59c09ca41f69a247e1a85cc89/datasette/app.py#L418-L424

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1967#issuecomment-1368265354
issue_url: https://api.github.com/repos/simonw/datasette/issues/1967
node_id: IC_kwDOBm6k_c5RjhKK
created_at: 2022-12-31T18:56:35Z
updated_at: 2022-12-31T18:56:35Z
author_association: OWNER
body:
  Thanks for this!

  Datasette has a favicon in this file: https://github.com/simonw/datasette/blob/2b4a8aa0833068da25072b6247bf349949ca00ee/datasette/static/favicon.png - so I'll copy that over and use it here as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1971#issuecomment-1368266457
issue_url: https://api.github.com/repos/simonw/datasette/issues/1971
node_id: IC_kwDOBm6k_c5RjhbZ
created_at: 2022-12-31T19:05:26Z
updated_at: 2022-12-31T19:05:26Z
author_association: OWNER
body:
  Ran this locally:

      pip install -U Sphinx

  Got these warnings:

  ```
  Installing collected packages: Sphinx
    Attempting uninstall: Sphinx
      Found existing installation: Sphinx 5.1.1
      Uninstalling Sphinx-5.1.1:
        Successfully uninstalled Sphinx-5.1.1
  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
  sphinx-basic-ng 0.0.1a12 requires sphinx<6.0,>=4.0, but you have sphinx 6.0.0 which is incompatible.
  furo 2022.6.21 requires sphinx<6.0,>=4.0, but you have sphinx 6.0.0 which is incompatible.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1971#issuecomment-1368266658
issue_url: https://api.github.com/repos/simonw/datasette/issues/1971
node_id: IC_kwDOBm6k_c5Rjhei
created_at: 2022-12-31T19:07:19Z
updated_at: 2022-12-31T19:07:19Z
author_association: OWNER
body:
  So it looks like Furo doesn't work with Sphinx 6.0 yet - there' s a fix on `main` but it hasn't been released:

  - https://github.com/pradyunsg/furo/discussions/569

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1971#issuecomment-1368266904
issue_url: https://api.github.com/repos/simonw/datasette/issues/1971
node_id: IC_kwDOBm6k_c5RjhiY
created_at: 2022-12-31T19:09:48Z
updated_at: 2022-12-31T19:09:48Z
author_association: OWNER
body:
  For the moment I'm going to pin to Sphinx 5.3.0

  Note that I got this warning there:

  ```
  [sphinx-autobuild] > sphinx-build -b html /Users/simon/Dropbox/Development/datasette/docs /Users/simon/Dropbox/Development/datasette/docs/_build
  Running Sphinx v5.3.0
  loading pickled environment... done
  WARNING: extlinks: Sphinx-6.0 will require a caption string to contain exactly one '%s' and all other '%' need to be escaped as '%%'.
  ```
  So I should probably fix that in advance of the 6.0 upgrade.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1967#issuecomment-1368267484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1967
node_id: IC_kwDOBm6k_c5Rjhrc
created_at: 2022-12-31T19:15:50Z
updated_at: 2022-12-31T19:15:50Z
author_association: OWNER
body:
  My Firefox tab before:

  <img width="169" alt="image" src="https://user-images.githubusercontent.com/9599/210153361-42241d3b-7681-42ed-a491-c0d513ef41fe.png">

  And after:

  <img width="169" alt="image" src="https://user-images.githubusercontent.com/9599/210153655-f74ef94f-7223-468b-9ca0-cf944393d993.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-1368268148
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: IC_kwDOBm6k_c5Rjh10
created_at: 2022-12-31T19:22:07Z
updated_at: 2022-12-31T19:22:07Z
author_association: OWNER
body:
  It turned out the most significant part of this change - switching from an array of arrays to an array of objects for the `"rows"` key - was really easy: Datasette already had a `?_shape=arrays` v.s. `?_shape=objects` mechanism, so I switched which one was the default in https://github.com/simonw/datasette/commit/234230e59574ccb8d8a24c45ccd325f725812377

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1970#issuecomment-1368268233
issue_url: https://api.github.com/repos/simonw/datasette/issues/1970
node_id: IC_kwDOBm6k_c5Rjh3J
created_at: 2022-12-31T19:22:58Z
updated_at: 2022-12-31T19:22:58Z
author_association: OWNER
body:
  https://latest.datasette.io/_internal/databases shows the fix now:

  <img width="542" alt="image" src="https://user-images.githubusercontent.com/9599/210153822-11a94a8e-9049-42e9-89b5-c709cde63467.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-1368269283
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: IC_kwDOBm6k_c5RjiHj
created_at: 2022-12-31T19:29:45Z
updated_at: 2022-12-31T19:29:45Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2 now shows the new default:
  ```json
  {
    "database": "fixtures",
    "table": "compound_three_primary_keys",
    "is_view": false,
    "human_description_en": "",
    "rows": [
      {
        "pk1": "a",
        "pk2": "a",
        "pk3": "a",
        "content": "a-a-a"
      },
      {
        "pk1": "a",
        "pk2": "a",
        "pk3": "b",
        "content": "a-a-b"
      }
    ],
  ```
  The old format can be had like this: https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2&_shape=arrays

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-1368269732
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: IC_kwDOBm6k_c5RjiOk
created_at: 2022-12-31T19:32:33Z
updated_at: 2023-01-17T02:05:45Z
author_association: OWNER
body:
  New thinking on the trimmed-down default. Previously I was going to use `"row"` and `"next_url"` - I now want to do this instead:
  ```json
  {
    "ok": true,
    "rows": [
      {
        "pk1": "a",
        "pk2": "a",
        "pk3": "a",
        "content": "a-a-a"
      },
      {
        "pk1": "a",
        "pk2": "a",
        "pk3": "b",
        "content": "a-a-b"
      }
    ],
    "next": "a,a,b"
  }
  ```
  If there isn't a next page it will return `"next": null`.

  This is even more succinct. I'm OK with people having to request `next_url` if they don't want to construct the new URL themselves.

  The `"ok": true` is there so it can be `false` for errors, consistently.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-1368269811
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: IC_kwDOBm6k_c5RjiPz
created_at: 2022-12-31T19:33:09Z
updated_at: 2022-12-31T19:33:09Z
author_association: OWNER
body:
  Here's the so-far updated documentation for this change: https://github.com/simonw/datasette/blob/a2dca62360ad4a961d4c46f68eae41b7d5c7b2c9/docs/json_api.rst#different-shapes

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-1368285442
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: IC_kwDOBm6k_c5RjmEC
created_at: 2022-12-31T22:02:16Z
updated_at: 2022-12-31T22:02:16Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_size=2 now returns `count`:

  ```json
  {
    "database": "fixtures",
    "table": "compound_three_primary_keys",
    "is_view": false,
    "human_description_en": "",
    "rows": [
      {
        "pk1": "a",
        "pk2": "a",
        "pk3": "a",
        "content": "a-a-a"
      },
      {
        "pk1": "a",
        "pk2": "a",
        "pk3": "b",
        "content": "a-a-b"
      }
    ],
    "truncated": false,
    "count": 1001,
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1973#issuecomment-1368935320
issue_url: https://api.github.com/repos/simonw/datasette/issues/1973
node_id: IC_kwDOBm6k_c5RmEuY
created_at: 2023-01-02T13:12:00Z
updated_at: 2023-01-02T13:12:00Z
author_association: OWNER
body:
  Thanks for catching this!

  Definitely a bug. This might be a good opportunity to reconsider how his should work though.

  It's possible that Datasette will eventually support alternative database backends - PostgreSQL, DuckDB and ClickHouse are all on my radar here.

  If that happens, this plugin hook returning a SQLite Row wouldn't make sense.

  Some kind of thing abstraction layer - maybe CustomRow, maybe something a little bit better - might be a good idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1974#issuecomment-1368935738
issue_url: https://api.github.com/repos/simonw/datasette/issues/1974
node_id: IC_kwDOBm6k_c5RmE06
created_at: 2023-01-02T13:12:30Z
updated_at: 2023-01-02T13:12:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1974?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.10**% // Head: **92.10**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`1e213a3`)](https://codecov.io/gh/simonw/datasette/pull/1974?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`572bdb5`)](https://codecov.io/gh/simonw/datasette/commit/572bdb5b8095bbdc5ec2aa95400055e49ea028cb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1974   +/-   ##
  =======================================
    Coverage   92.10%   92.10%           
  =======================================
    Files          38       38           
    Lines        5538     5538           
  =======================================
    Hits         5101     5101           
    Misses        437      437           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1974?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1973#issuecomment-1368936146
issue_url: https://api.github.com/repos/simonw/datasette/issues/1973
node_id: IC_kwDOBm6k_c5RmE7S
created_at: 2023-01-02T13:13:03Z
updated_at: 2023-01-02T13:13:03Z
author_association: OWNER
body:
  It's quite handy that the label for foreign keys is made available to the custom render function here - if that was properly documented (and maybe had a slightly more intuitive API) I think it might be a good solution here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1973#issuecomment-1369044959
issue_url: https://api.github.com/repos/simonw/datasette/issues/1973
node_id: IC_kwDOBm6k_c5Rmfff
created_at: 2023-01-02T15:41:40Z
updated_at: 2023-01-02T15:41:40Z
author_association: CONTRIBUTOR
body:
  Thanks for the response!

  Yes, it does seem like a pretty nice developer experience--both the automagical labelling of fkeys, and the ability to index the row by column name in addition to column index.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1369333759
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/520
node_id: IC_kwDOCGYnMM5Rnl__
created_at: 2023-01-03T02:23:43Z
updated_at: 2023-01-03T02:23:43Z
author_association: OWNER
body:
  The documentation here does at least say the following: https://sqlite-utils.datasette.io/en/3.30/python-api.html#reading-rows-from-a-file

  > -   **fp** (*BinaryIO*) -- a file-like object containing binary data

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-1370786026
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54
node_id: IC_kwDODEm0Qs5RtIjq
created_at: 2023-01-04T11:06:44Z
updated_at: 2023-01-04T11:06:44Z
author_association: NONE
body:
  as of 2023 it appears that `tweets: not yet implemented` is happening.. pretty important for a twitter export functionality!

  this seems an impossible task with twitter liable to switch this around every other day

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1976#issuecomment-1372203113
issue_url: https://api.github.com/repos/simonw/datasette/issues/1976
node_id: IC_kwDOBm6k_c5Ryihp
created_at: 2023-01-05T13:15:04Z
updated_at: 2023-01-05T13:15:04Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1976?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.15**% // Head: **92.15**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`af76abc`)](https://codecov.io/gh/simonw/datasette/pull/1976?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`c41278b`)](https://codecov.io/gh/simonw/datasette/commit/c41278b46f7936b4b1a8a14bf285bed82c81c609?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1976   +/-   ##
  =======================================
    Coverage   92.15%   92.15%           
  =======================================
    Files          38       38           
    Lines        5542     5542           
  =======================================
    Hits         5107     5107           
    Misses        435      435           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1976?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1977#issuecomment-1373600402
issue_url: https://api.github.com/repos/simonw/datasette/issues/1977
node_id: IC_kwDOBm6k_c5R33qS
created_at: 2023-01-06T13:09:51Z
updated_at: 2023-01-06T13:09:51Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1977?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.15**% // Head: **92.15**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`a4eebee`)](https://codecov.io/gh/simonw/datasette/pull/1977?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`c41278b`)](https://codecov.io/gh/simonw/datasette/commit/c41278b46f7936b4b1a8a14bf285bed82c81c609?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1977   +/-   ##
  =======================================
    Coverage   92.15%   92.15%           
  =======================================
    Files          38       38           
    Lines        5542     5542           
  =======================================
    Hits         5107     5107           
    Misses        435      435           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1977?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1409#issuecomment-1374555507
issue_url: https://api.github.com/repos/simonw/datasette/issues/1409
node_id: IC_kwDOBm6k_c5R7g1z
created_at: 2023-01-07T18:06:01Z
updated_at: 2023-01-07T18:06:01Z
author_association: OWNER
body:
  Documentation:
  - https://docs.datasette.io/en/latest/authentication.html#controlling-the-ability-to-execute-arbitrary-sql
  - https://docs.datasette.io/en/latest/settings.html#setting-default-allow-sql

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1979#issuecomment-1374582375
issue_url: https://api.github.com/repos/simonw/datasette/issues/1979
node_id: IC_kwDOBm6k_c5R7nZn
created_at: 2023-01-07T19:22:39Z
updated_at: 2023-01-07T19:22:39Z
author_association: OWNER
body:
  This helps:
  ```diff
  diff --git a/datasette/cli.py b/datasette/cli.py
  index 2b61292b..ea98879c 100644
  --- a/datasette/cli.py
  +++ b/datasette/cli.py
  @@ -4,13 +4,15 @@ import click
   from click import formatting
   from click.types import CompositeParamType
   from click_default_group import DefaultGroup
  +import functools
   import json
   import os
   import pathlib
   import shutil
   from subprocess import call
  -import sys
   from runpy import run_module
  +import sys
  +import textwrap
   import webbrowser
   from .app import (
       OBSOLETE_SETTINGS,
  @@ -126,7 +128,7 @@ class Setting(CompositeParamType):
   
   
   def sqlite_extensions(fn):
  -    return click.option(
  +    fn = click.option(
           "sqlite_extensions",
           "--load-extension",
           type=LoadExtension(),
  @@ -134,6 +136,25 @@ def sqlite_extensions(fn):
           multiple=True,
           help="Path to a SQLite extension to load, and optional entrypoint",
       )(fn)
  +    # Wrap it in a custom error handler
  +    @functools.wraps(fn)
  +    def wrapped(*args, **kwargs):
  +        try:
  +            return fn(*args, **kwargs)
  +        except AttributeError as e:
  +            if "enable_load_extension" in str(e):
  +                raise click.ClickException(
  +                    textwrap.dedent(
  +                        """
  +                    Your Python installation does not have the ability to load SQLite extensions.
  +
  +                    More information: https://docs.datasette.io/en/stable/installation.html#extensions
  +                    """
  +                    ).strip()
  +                )
  +            raise
  +
  +    return wrapped
   
   
   @click.group(cls=DefaultGroup, default="serve", default_if_no_args=True)
  ```
  Need to write help for that to link to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1979#issuecomment-1374657204
issue_url: https://api.github.com/repos/simonw/datasette/issues/1979
node_id: IC_kwDOBm6k_c5R75q0
created_at: 2023-01-07T23:48:42Z
updated_at: 2023-01-07T23:48:42Z
author_association: OWNER
body:
  Once I ship the next release I should change that `/help/extensions` link to go to `/stable/` and not `/latest/`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1979#issuecomment-1374658468
issue_url: https://api.github.com/repos/simonw/datasette/issues/1979
node_id: IC_kwDOBm6k_c5R75-k
created_at: 2023-01-07T23:53:41Z
updated_at: 2023-01-07T23:53:41Z
author_association: OWNER
body:
  https://datasette.io/help/extensions now redirects to this new section of documentation: https://docs.datasette.io/en/latest/installation.html#a-note-about-extensions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1979#issuecomment-1374659233
issue_url: https://api.github.com/repos/simonw/datasette/issues/1979
node_id: IC_kwDOBm6k_c5R76Kh
created_at: 2023-01-07T23:56:25Z
updated_at: 2023-01-07T23:56:25Z
author_association: OWNER
body:
  ```
  % datasette --load-extension foo                        
  Error: Your Python installation does not have the ability to load SQLite extensions.

  More information: https://datasette.io/help/extensions
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1978#issuecomment-1374916593
issue_url: https://api.github.com/repos/simonw/datasette/issues/1978
node_id: IC_kwDOBm6k_c5R84_x
created_at: 2023-01-08T20:06:16Z
updated_at: 2023-01-08T20:06:16Z
author_association: OWNER
body:
  Related issue - I'm not 100% settled on the design for these yet:
  - https://github.com/simonw/datasette/issues/1048#issuecomment-1179757912

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1980#issuecomment-1375061747
issue_url: https://api.github.com/repos/simonw/datasette/issues/1980
node_id: IC_kwDOBm6k_c5R9cbz
created_at: 2023-01-09T03:22:36Z
updated_at: 2023-01-09T03:22:36Z
author_association: OWNER
body:
  The problem here is that the HTML ended up with this HTML in it:
  ```html
                  <select name="_sort" id="sort_by">
                      <option value="">Sort...</option>
                              <option value="id" selected>Sort by id</option>
  ```
  But `id` is not a valid sort option any more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1980#issuecomment-1375062405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1980
node_id: IC_kwDOBm6k_c5R9cmF
created_at: 2023-01-09T03:23:53Z
updated_at: 2023-01-09T03:23:53Z
author_association: OWNER
body:
  Some potential solutions:
  - Make it so the primary key is _always_ sortable - not a bad idea, the `sortable_columns` thing was designed to protect against sorting GIANT tables by a non-indexed column, and the primary key always has an index
  - Make sure that `<select>` menu never includes keys that cannot be sorted - this would be a good fix anyway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1981#issuecomment-1375142205
issue_url: https://api.github.com/repos/simonw/datasette/issues/1981
node_id: IC_kwDOBm6k_c5R9wE9
created_at: 2023-01-09T06:05:44Z
updated_at: 2023-01-09T06:05:44Z
author_association: OWNER
body:
  I want the labels to take as much space as they need, but line up with each other.

  If they are more than half the width of the page then the form fields should break to another line.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1983#issuecomment-1375795441
issue_url: https://api.github.com/repos/simonw/datasette/issues/1983
node_id: IC_kwDOBm6k_c5SAPjx
created_at: 2023-01-09T15:27:53Z
updated_at: 2023-01-09T15:27:53Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/7b48664d751e59222c01c97a2739d53d6363bdda/datasette/utils/__init__.py#L163-L178

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1983#issuecomment-1375797856
issue_url: https://api.github.com/repos/simonw/datasette/issues/1983
node_id: IC_kwDOBm6k_c5SAQJg
created_at: 2023-01-09T15:29:34Z
updated_at: 2023-01-09T15:29:34Z
author_association: OWNER
body:
  I had forgotten this had special handling for cursors!

  The default representation of a row as a tuple rather than a dictionary might not be the right thing to do based on:
  - #782 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1983#issuecomment-1375810027
issue_url: https://api.github.com/repos/simonw/datasette/issues/1983
node_id: IC_kwDOBm6k_c5SATHr
created_at: 2023-01-09T15:35:58Z
updated_at: 2023-01-09T15:35:58Z
author_association: CONTRIBUTOR
body:
  Yes please, and thank you. I realized I was maybe getting myself in trouble using that, but I think it's a good way to standardize JSON handling.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1987#issuecomment-1379276484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1987
node_id: IC_kwDOBm6k_c5SNhbE
created_at: 2023-01-11T18:00:35Z
updated_at: 2023-01-11T18:00:35Z
author_association: OWNER
body:
  Next challenge: how to best deploy this.

  Ideally I'd like to fix all of the older versions of the docs too - pages like https://docs.datasette.io/en/0.59.4/installation.html#using-pip

  Not sure if ReadTheDocs has a mechanism for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1987#issuecomment-1379282756
issue_url: https://api.github.com/repos/simonw/datasette/issues/1987
node_id: IC_kwDOBm6k_c5SNi9E
created_at: 2023-01-11T18:04:22Z
updated_at: 2023-01-11T18:04:22Z
author_association: OWNER
body:
  That change is now live on https://docs.datasette.io/en/latest/installation.html#using-pip

  I may need to ship a new stable dot-release to update the `/en/stable/` page though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1987#issuecomment-1379311498
issue_url: https://api.github.com/repos/simonw/datasette/issues/1987
node_id: IC_kwDOBm6k_c5SNp-K
created_at: 2023-01-11T18:27:30Z
updated_at: 2023-01-11T18:27:46Z
author_association: OWNER
body:
  I'm going to leave the bad link in the old versions, because I can't figure out a way to update them using ReadTheDocs - shipping 0.64.1 will at least mean that both `/latest/` and `/stable/` have a good link.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1987#issuecomment-1379313564
issue_url: https://api.github.com/repos/simonw/datasette/issues/1987
node_id: IC_kwDOBm6k_c5SNqec
created_at: 2023-01-11T18:29:25Z
updated_at: 2023-01-11T18:29:25Z
author_association: OWNER
body:
  Fixed:
  - https://docs.datasette.io/en/latest/installation.html#using-pip
  - https://docs.datasette.io/en/stable/installation.html#using-pip
  - https://docs.datasette.io/en/0.64.1/installation.html#using-pip

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1988#issuecomment-1379492499
issue_url: https://api.github.com/repos/simonw/datasette/issues/1988
node_id: IC_kwDOBm6k_c5SOWKT
created_at: 2023-01-11T21:15:01Z
updated_at: 2023-01-11T21:20:14Z
author_association: OWNER
body:
  This is tricky. `datasette-template-sql` registers a custom `sql()` function, but @cldellow spotted a situation where this would interfere with an existing `{{ sql }}` variable in a template context!

  https://github.com/simonw/datasette-template-sql/blob/70483fd94bb0349761830e37c9ab928e8966841a/datasette_template_sql/__init__.py

  ```python
  @hookimpl
  def extra_template_vars(datasette):
      async def execute_sql(sql, args=None, database=None):
          db = datasette.get_database(database)
          return (await db.execute(sql, args)).rows

      return {"sql": execute_sql}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1988#issuecomment-1379495504
issue_url: https://api.github.com/repos/simonw/datasette/issues/1988
node_id: IC_kwDOBm6k_c5SOW5Q
created_at: 2023-01-11T21:18:00Z
updated_at: 2023-01-11T21:18:25Z
author_association: OWNER
body:
  A few options:
  - Ensure that the explicit template context overrides anything that plugins might do - so in this case the `sql(...)` function would not be available on that page. This would break people who have custom templates that use that function though.
  - Encourage naming conventions where functions and variables  from plugins are less likely to interfere with existing functionality.
  - Completely change how templates work, so you never have a variable called `{{ sql }}` - it is always accessed via some parent object instead, such as `{{ page.sql }}`.

  That last option actually fits quite well with my efforts to unify template rendering with JSON (and JSON extras) so it might be the best way to address this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1988#issuecomment-1379500131
issue_url: https://api.github.com/repos/simonw/datasette/issues/1988
node_id: IC_kwDOBm6k_c5SOYBj
created_at: 2023-01-11T21:22:31Z
updated_at: 2023-01-11T21:22:31Z
author_association: OWNER
body:
  Relevant code:

  https://github.com/simonw/datasette/blob/6a352e99ab988dbf8fd22a100049caa6ad33f1ec/datasette/app.py#L1138-L1149

  https://github.com/simonw/datasette/blob/6a352e99ab988dbf8fd22a100049caa6ad33f1ec/datasette/app.py#L1193-L1195

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1988#issuecomment-1379502161
issue_url: https://api.github.com/repos/simonw/datasette/issues/1988
node_id: IC_kwDOBm6k_c5SOYhR
created_at: 2023-01-11T21:24:24Z
updated_at: 2023-01-11T21:25:05Z
author_association: OWNER
body:
  Renaming `sql()` to `template_sql()` in https://github.com/simonw/datasette-template-sql and shipping a 2.0 release of that plugin (it is currently at 1.0.2) would be a sensible short-term fix too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1382655354
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31
node_id: IC_kwDOD079W85SaaV6
created_at: 2023-01-14T04:08:36Z
updated_at: 2023-01-14T04:08:36Z
author_association: NONE
body:
  I just tried this branch and saw some errors. 

  I installed this PR locally with:

  ```bash
  pip install https://github.com/RhetTbull/dogsheep-photos/archive/update_for_bigsur.zip
  ```

  <details>
  <summary>System Details</summary>

  **OS:** MacOS Monterey
  **Python Version:** Python 3.10.8

  </details>

  <details>
  <summary>Stacktrace</summary>

  ```python
  Traceback (most recent call last):
    File "/Users/df/.venvs/photo-experiments/bin/dogsheep-photos", line 8, in <module>
      sys.exit(cli())
    File "/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
      return self.main(*args, **kwargs)
    File "/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py", line 1055, in main
      rv = self.invoke(ctx)
    File "/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/click/core.py", line 760, in invoke
      return __callback(*args, **kwargs)
    File "/Users/df/.venvs/photo-experiments/lib/python3.10/site-packages/dogsheep_photos/cli.py", line 254, in apple_photos
      sha256 = calculate_hash(pathlib.Path(photo.path))
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py", line 960, in __new__
      self = cls._from_parts(args)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py", line 594, in _from_parts
      drv, root, parts = self._parse_args(args)
    File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/pathlib.py", line 578, in _parse_args
      a = os.fspath(a)
  TypeError: expected str, bytes or os.PathLike object, not NoneType
  ```
  </details>

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1992#issuecomment-1384039611
issue_url: https://api.github.com/repos/simonw/datasette/issues/1992
node_id: IC_kwDOBm6k_c5SfsS7
created_at: 2023-01-16T13:12:32Z
updated_at: 2023-03-29T06:10:23Z
author_association: NONE
body:
  ## [Codecov](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage has no change and project coverage change: **`-0.31`** :warning:
  > Comparison is base [(`c025b01`)](https://codecov.io/gh/simonw/datasette/commit/c025b0180fe344e4f28177ecd7cf388d492d3064?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41% compared to head [(`bfaa59b`)](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11%.

  > :exclamation: Current head bfaa59b differs from pull request most recent head 64dd698. Consider uploading reports for the commit 64dd698 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1992      +/-   ##
  ==========================================
  - Coverage   92.41%   92.11%   -0.31%     
  ==========================================
    Files          38       38              
    Lines        5750     5555     -195     
  ==========================================
  - Hits         5314     5117     -197     
  - Misses        436      438       +2     
  ```


  [see 9 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/1992/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1384741055
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5SiXi_
created_at: 2023-01-17T01:58:24Z
updated_at: 2023-01-17T01:58:24Z
author_association: OWNER
body:
  As suggested in this issue:
  - #1721 

  There are three parts of the Datasette API that need to support extras:

  - Table, e.g. https://latest.datasette.io/fixtures/facetable.json
  - Row, e.g. https://latest.datasette.io/fixtures/facetable/1.json
  - Query, e.g. https://latest.datasette.io/fixtures/neighborhood_search.json or https://latest.datasette.io/fixtures.json?sql=%0Aselect+_neighborhood%2C+facet_cities.name%2C+state%0Afrom+facetable%0A++++join+facet_cities%0A++++++++on+facetable._city_id+%3D+facet_cities.id%0Awhere+_neighborhood+like+%27%25%27+||+%3Atext+||+%27%25%27%0Aorder+by+_neighborhood%3B%0A&text=

  There are two other pages I should consider though:

  - https://latest.datasette.io/.json - the JSON version of the https://latest.datasette.io/ homepage
  - https://latest.datasette.io/fixtures.json - note that this is different from the same URL with `?sql=...` appended to it. This is the index of tables in a specific database

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1384742385
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5SiX3x
created_at: 2023-01-17T02:00:23Z
updated_at: 2023-01-17T02:00:38Z
author_association: OWNER
body:
  I'm not actually too happy about how `/fixtures.json` currently entirely changes shape based on whether or not you pass a `?sql=` argument to it.

  Maybe I can fix that disparity with extras too?

  The list of tables you see on `/fixtures.json` without the `?sql=` could become another extra. The HTML version of that page could know to request that extra by default.

  This would also support running a SQL query but also returning a list of tables - which can be useful for building a SQL editor interface which hints at the tables that are available to the user - or even for generating the configuration needed by the CodeMirror editor's SQL completion, added in:
  - #1893

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1384743243
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5SiYFL
created_at: 2023-01-17T02:01:26Z
updated_at: 2023-01-17T02:01:26Z
author_association: OWNER
body:
  I'm tempted NOT to document the JSON for the `/.json` page, simply because I'm not at all convinced that the current homepage design is the best possible use of that space - and I'd like to reserve the opportunity to redesign that in e.g. Datasette 1.1 without it being a breaking change to the documented JSON API.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1384752452
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5SiaVE
created_at: 2023-01-17T02:14:41Z
updated_at: 2023-01-17T02:15:58Z
author_association: OWNER
body:
  Thinking about `?_extra=` values just for the table JSON. The default shape will look like this:

  ```json
  {
    "ok": true,
    "rows": [{"id": 1, "name": "Name"}],
    "next": null,
  }
  ```
  The table extras could be:

  - `count` - adds a `"count"` field with a full `count(*)` for that filtered table
  - `next_url` - the full URL to the next page
  - `columns` - adds `"columns": ["id", "name"]`
  - `expandable_columns` - a list of columns that can be expanded (note that `"expanded_columns": [...]` shows up automatically if the user passes any `?_label=` options, like on https://latest.datasette.io/fixtures/facetable.json?_label=_city_id ) - I'm tempted to rename this to `label_columns` and have it add both `label_columns` and `label_columns_selected` or similar.
  - `primary_keys` - a list of primary keys e.g. `["id"]` - not sure what to do about `rowid` columns here
  - `query` - a `{"sql": "select ...", "params": {"p0": "1"}}` object
  - `units` - the units feature
  - `suggested_facets` - suggested facets
  - `metadata` - a `{"metadata": {"source_url": "..."}}` etc block - differs from current in that it would be nested in `"metadata": {...}`.

  Stuff currently in https://latest.datasette.io/fixtures/facetable.json that is not yet covered by the above:
  ```
    "database": "fixtures",
    "table": "facetable",
    "is_view": false,
    "human_description_en": "where id = 1",
    "private": false,
    "allow_execute_sql": true,
    "query_ms": 16.749476999393664,
  ```
  I'm tempted to bundle `database`, `table`, `is_view` and `human_description_en` into one (not sure what to call it though, perhaps `display_details`?) - and then drop `allow_execute_sql` entirely and have `private` and `query_ms` as their own named extras.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1385805702
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5SmbeG
created_at: 2023-01-17T17:50:17Z
updated_at: 2023-01-17T17:50:17Z
author_association: OWNER
body:
  Or maybe have a `permissions` extra which includes `allow_execute_sql` and `private`? Could anything else go in there?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1385807684
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5Smb9E
created_at: 2023-01-17T17:51:54Z
updated_at: 2023-01-19T23:20:59Z
author_association: OWNER
body:
  In most cases, the `?_extra=xxx` name exactly corresponds to the additional key that is added to the JSON.

  `?_facet=...` is one example of a query string argument that causes an extra key - `"facet_results"` - to be added to the JSON even though it wasn't requested by name in a `?_extra=`.

  Am I OK with that? I think so.

  Related issue:
  - #1558 

  Actually there's an edge-case here that's worth considering: it's possible to use metadata to set default facets for a table. If you do this for a table, then `.json` for that table will always calculate and return those facets - which may be an expensive and unnecessary operation.

  So maybe we don't include `facet_results` in the JSON unless explicitly asked for in that case, but have a rule that `?_facet` implies `?_extra=facet_results`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1497#issuecomment-1387433455
issue_url: https://api.github.com/repos/simonw/datasette/issues/1497
node_id: IC_kwDOBm6k_c5Sso3v
created_at: 2023-01-18T17:13:45Z
updated_at: 2023-01-18T17:13:45Z
author_association: NONE
body:
  You may have just been talking to yourself here, but I found your documentation of the process incredibly useful when I hit the same error myself. Thanks! 🌈 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1996#issuecomment-1396933348
issue_url: https://api.github.com/repos/simonw/datasette/issues/1996
node_id: IC_kwDOBm6k_c5TQ4Lk
created_at: 2023-01-19T12:55:57Z
updated_at: 2023-01-19T12:55:57Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.11**% // Head: **92.11**% // Increases project coverage by **`+0.00%`** :tada:
  > Coverage data is based on head [(`96b8b7f`)](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`6a352e9`)](https://codecov.io/gh/simonw/datasette/commit/6a352e99ab988dbf8fd22a100049caa6ad33f1ec?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #1996   +/-   ##
  =======================================
    Coverage   92.11%   92.11%           
  =======================================
    Files          38       38           
    Lines        5555     5556    +1     
  =======================================
  + Hits         5117     5118    +1     
    Misses        438      438           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.87% <100.00%> (+<0.01%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1989#issuecomment-1397734424
issue_url: https://api.github.com/repos/simonw/datasette/issues/1989
node_id: IC_kwDOBm6k_c5TT7wY
created_at: 2023-01-19T23:18:45Z
updated_at: 2023-01-19T23:18:45Z
author_association: OWNER
body:
  There's a `?_nocol=x` argument you can use to hide a column when you link to a table, but that won't help you if you need to hide the column for privacy reasons: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments

  One solution right now is to define a SQL view for the things that you DO want people to be able to see, and then use Datasette's permission system to hide the tables (`'"allow": false` in metadata for each table) but show the views.

  If you want to redact specific columns there's a plugin for doing that: https://datasette.io/plugins/datasette-mask-columns

  This does make sense as more of a core Datasette feature though - tagging it as a feature suggestion.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1397942113
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5TUudh
created_at: 2023-01-20T05:33:00Z
updated_at: 2023-01-20T05:33:00Z
author_association: OWNER
body:
  I'm going to write code which parses `?_extra=` in the comma separated or multiple parameter format and then looks up functions in a dictionary.

  It will return an error if you ask for an extra that does not exist.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1998#issuecomment-1398767813
issue_url: https://api.github.com/repos/simonw/datasette/issues/1998
node_id: IC_kwDOBm6k_c5TX4DF
created_at: 2023-01-20T18:18:27Z
updated_at: 2023-01-20T18:18:27Z
author_association: OWNER
body:
  Fell down a bit of a rabbit hole trying to figure out how to get Click's `version_option()` to evaluate a custom message. Got this far:

  ```python
  class _VersionMessage(UserString):
      @property
      def data(self):
          return "%(prog)s, version %(version)s (SQLite {})".format(
              sqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0]
          )

      @data.setter
      def data(self, value):
          pass


  @click.group(cls=DefaultGroup, default="serve", default_if_no_args=True)
  @click.version_option(version=__version__, message=_VersionMessage(""))
  def cli():
      """
      Datasette is an open source multi-tool for exploring and publishing data

      \b
      About Datasette: https://datasette.io/
      Full documentation: https://docs.datasette.io/
      """
  ```
  But now:

  ```
  % datasette --version
  %(prog)s, version %(version)s (SQLite 3.40.1)
  ```

  I was trying to avoid running that `select sqlite_version()` thing unless the `--version` option was used.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1998#issuecomment-1398768399
issue_url: https://api.github.com/repos/simonw/datasette/issues/1998
node_id: IC_kwDOBm6k_c5TX4MP
created_at: 2023-01-20T18:19:06Z
updated_at: 2023-01-20T18:19:06Z
author_association: OWNER
body:
  Simplest solution would be to ditch the `version_option` decorator and roll a custom option based on it instead, imitating what this code does:

  https://github.com/pallets/click/blob/7586834cab38c5592d9d6de3ee0ebe75d4353bfb/src/click/decorators.py#L413-L524

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1399145981
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5TZUX9
created_at: 2023-01-21T01:56:52Z
updated_at: 2023-01-21T01:56:52Z
author_association: OWNER
body:
  Got first prototype working using `asyncinject` and it's pretty nice:
  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index ad45ecd3..c8690b22 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -2,6 +2,7 @@ import asyncio
   import itertools
   import json
   
  +from asyncinject import Registry
   import markupsafe
   
   from datasette.plugins import pm
  @@ -538,57 +539,60 @@ class TableView(DataView):
           # Execute the main query!
           results = await db.execute(sql, params, truncate=True, **extra_args)
   
  -        # Calculate the total count for this query
  -        count = None
  -        if (
  -            not db.is_mutable
  -            and self.ds.inspect_data
  -            and count_sql == f"select count(*) from {table_name} "
  -        ):
  -            # We can use a previously cached table row count
  -            try:
  -                count = self.ds.inspect_data[database_name]["tables"][table_name][
  -                    "count"
  -                ]
  -            except KeyError:
  -                pass
  -
  -        # Otherwise run a select count(*) ...
  -        if count_sql and count is None and not nocount:
  -            try:
  -                count_rows = list(await db.execute(count_sql, from_sql_params))
  -                count = count_rows[0][0]
  -            except QueryInterrupted:
  -                pass
  -
  -        # Faceting
  -        if not self.ds.setting("allow_facet") and any(
  -            arg.startswith("_facet") for arg in request.args
  -        ):
  -            raise BadRequest("_facet= is not allowed")
  +        # Resolve extras
  +        extras = _get_extras(request)
  +        if request.args.getlist("_facet"):
  +            extras.add("facet_results")
   
  -        # pylint: disable=no-member
  -        facet_classes = list(
  -            itertools.chain.from_iterable(pm.hook.register_facet_classes())
  -        )
  -        facet_results = {}
  -        facets_timed_out = []
  -        facet_instances = []
  -        for klass in facet_classes:
  -            facet_instances.append(
  -                klass(
  -                    self.ds,
  -                    request,
  -                    database_name,
  -                    sql=sql_no_order_no_limit,
  -                    params=params,
  -                    table=table_name,
  -                    metadata=table_metadata,
  -                    row_count=count,
  -                )
  +        async def extra_count():
  +            # Calculate the total count for this query
  +            count = None
  +            if (
  +                not db.is_mutable
  +                and self.ds.inspect_data
  +                and count_sql == f"select count(*) from {table_name} "
  +            ):
  +                # We can use a previously cached table row count
  +                try:
  +                    count = self.ds.inspect_data[database_name]["tables"][table_name][
  +                        "count"
  +                    ]
  +                except KeyError:
  +                    pass
  +
  +            # Otherwise run a select count(*) ...
  +            if count_sql and count is None and not nocount:
  +                try:
  +                    count_rows = list(await db.execute(count_sql, from_sql_params))
  +                    count = count_rows[0][0]
  +                except QueryInterrupted:
  +                    pass
  +            return count
  +
  +        async def facet_instances(extra_count):
  +            facet_instances = []
  +            facet_classes = list(
  +                itertools.chain.from_iterable(pm.hook.register_facet_classes())
               )
  +            for facet_class in facet_classes:
  +                facet_instances.append(
  +                    facet_class(
  +                        self.ds,
  +                        request,
  +                        database_name,
  +                        sql=sql_no_order_no_limit,
  +                        params=params,
  +                        table=table_name,
  +                        metadata=table_metadata,
  +                        row_count=extra_count,
  +                    )
  +                )
  +            return facet_instances
  +
  +        async def extra_facet_results(facet_instances):
  +            facet_results = {}
  +            facets_timed_out = []
   
  -        async def execute_facets():
               if not nofacet:
                   # Run them in parallel
                   facet_awaitables = [facet.facet_results() for facet in facet_instances]
  @@ -607,9 +611,13 @@ class TableView(DataView):
                           facet_results[key] = facet_info
                       facets_timed_out.extend(instance_facets_timed_out)
   
  -        suggested_facets = []
  +            return {
  +                "results": facet_results,
  +                "timed_out": facets_timed_out,
  +            }
   
  -        async def execute_suggested_facets():
  +        async def extra_suggested_facets(facet_instances):
  +            suggested_facets = []
               # Calculate suggested facets
               if (
                   self.ds.setting("suggest_facets")
  @@ -624,8 +632,15 @@ class TableView(DataView):
                   ]
                   for suggest_result in await gather(*facet_suggest_awaitables):
                       suggested_facets.extend(suggest_result)
  +            return suggested_facets
  +
  +        # Faceting
  +        if not self.ds.setting("allow_facet") and any(
  +            arg.startswith("_facet") for arg in request.args
  +        ):
  +            raise BadRequest("_facet= is not allowed")
   
  -        await gather(execute_facets(), execute_suggested_facets())
  +        # pylint: disable=no-member
   
           # Figure out columns and rows for the query
           columns = [r[0] for r in results.description]
  @@ -732,17 +747,56 @@ class TableView(DataView):
               rows = rows[:page_size]
   
           # human_description_en combines filters AND search, if provided
  -        human_description_en = filters.human_description_en(
  -            extra=extra_human_descriptions
  -        )
  +        async def extra_human_description_en():
  +            human_description_en = filters.human_description_en(
  +                extra=extra_human_descriptions
  +            )
  +            if sort or sort_desc:
  +                human_description_en = " ".join(
  +                    [b for b in [human_description_en, sorted_by] if b]
  +                )
  +            return human_description_en
   
           if sort or sort_desc:
               sorted_by = "sorted by {}{}".format(
                   (sort or sort_desc), " descending" if sort_desc else ""
               )
  -            human_description_en = " ".join(
  -                [b for b in [human_description_en, sorted_by] if b]
  -            )
  +
  +        async def extra_next_url():
  +            return next_url
  +
  +        async def extra_columns():
  +            return columns
  +
  +        async def extra_primary_keys():
  +            return pks
  +
  +        registry = Registry(
  +            extra_count,
  +            extra_facet_results,
  +            extra_suggested_facets,
  +            facet_instances,
  +            extra_human_description_en,
  +            extra_next_url,
  +            extra_columns,
  +            extra_primary_keys,
  +        )
  +
  +        results = await registry.resolve_multi(
  +            ["extra_{}".format(extra) for extra in extras]
  +        )
  +        data = {
  +            "ok": True,
  +            "rows": rows[:page_size],
  +            "next": next_value and str(next_value) or None,
  +        }
  +        data.update({
  +            key.replace("extra_", ""): value
  +            for key, value in results.items()
  +            if key.startswith("extra_")
  +            and key.replace("extra_", "") in extras
  +        })
  +        return Response.json(data, default=repr)
   
           async def extra_template():
               nonlocal sort
  @@ -1334,3 +1388,11 @@ class TableDropView(BaseView):
   
           await db.execute_write_fn(drop_table)
           return Response.json({"ok": True}, status=200)
  +
  +
  +def _get_extras(request):
  +    extra_bits = request.args.getlist("_extra")
  +    extras = set()
  +    for bit in extra_bits:
  +        extras.update(bit.split(","))
  +    return extras
  ```
  With that in place, `http://127.0.0.1:8001/content/releases?author=25778&_size=1&_extra=count,primary_keys,columns&_facet=author` returns:
  ```json
  {
    "ok": true,
    "rows": [
      {
        "html_url": "https://github.com/eyeseast/geocode-sqlite/releases/tag/0.1.2",
        "id": 30926270,
        "author": {
          "value": 25778,
          "label": "eyeseast"
        },
        "node_id": "MDc6UmVsZWFzZTMwOTI2Mjcw",
        "tag_name": "0.1.2",
        "target_commitish": "master",
        "name": "v0.1.2",
        "draft": 0,
        "prerelease": 1,
        "created_at": "2020-09-08T17:48:24Z",
        "published_at": "2020-09-08T17:50:15Z",
        "body": "Basic API is in place, with CLI support for Google, Bing, MapQuest and Nominatum (OSM) geocoders.",
        "repo": {
          "value": 293361514,
          "label": "geocode-sqlite"
        },
        "reactions": null,
        "mentions_count": null
      }
    ],
    "next": "30926270",
    "primary_keys": [
      "id"
    ],
    "columns": [
      "html_url",
      "id",
      "author",
      "node_id",
      "tag_name",
      "target_commitish",
      "name",
      "draft",
      "prerelease",
      "created_at",
      "published_at",
      "body",
      "repo",
      "reactions",
      "mentions_count"
    ],
    "count": 25,
    "facet_results": {
      "results": {
        "author": {
          "name": "author",
          "type": "column",
          "hideable": true,
          "toggle_url": "/content/releases?author=25778&_size=1&_extra=count%2Cprimary_keys%2Ccolumns",
          "results": [
            {
              "value": 25778,
              "label": "eyeseast",
              "count": 25,
              "toggle_url": "http://127.0.0.1:8001/content/releases?_size=1&_extra=count%2Cprimary_keys%2Ccolumns&_facet=author",
              "selected": true
            }
          ],
          "truncated": false
        }
      },
      "timed_out": []
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1399178591
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5TZcVf
created_at: 2023-01-21T04:53:15Z
updated_at: 2023-01-21T04:53:15Z
author_association: OWNER
body:
  Implementing this to work with the `.json` extension is going to be a lot harder.

  The challenge here is that we're working with the whole `BaseView()` v.s. `TableView()` abstraction, which I've been wanting to get rid of for a long time.

  `BaseView()` calls `.data()` and expects to get back a `(data, extra_template_data, templates)` tuple - then if a format is in play (`.json` or `.geojson` or similar from a plugin) it hands off `data` to that. If `.csv` is involved it does something special, in order to support streaming responses. And if it's regular HTML it calls `await extra_template_data()` and combines that with `data` and passes it to the template.

  I want this to work completely differently: I want the formats (including HTML) to have the option of adding some extra `?_extra=` extras, then I want HTML to be able to render the page entirely from the JSON if necessary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1399184540
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5TZdyc
created_at: 2023-01-21T05:35:32Z
updated_at: 2023-01-21T05:35:32Z
author_association: OWNER
body:
  It's annoying that the https://docs.datasette.io/en/0.64.1/plugin_hooks.html#register-output-renderer-datasette plugin hook passes `rows` as "list of sqlite3.Row objects" - I'd prefer it if that plugin hook worked with JSON data, not `sqlite3.Row`.

  https://docs.datasette.io/en/0.64.1/plugin_hooks.html#render-cell-row-value-column-table-database-datasette is documented as accepting `Row` but actually gets `CustomRow`, see:

  - #1973

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1399184642
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5TZd0C
created_at: 2023-01-21T05:36:22Z
updated_at: 2023-01-21T05:41:06Z
author_association: OWNER
body:
  Maybe `"rows"` should be a default `?_extra=`... but it should be possible to request `"arrays"` instead which would be a list of arrays, more suitable perhaps for custom renderers such as the CSV one.

  This could be quite neat, in that EVERY key in the JSON representation would be defined as an extra - just some would be on by default. There could even be a mechanism for turning them back off again, maybe using `?_extra=-rows`.

  In which case maybe `?_extra=` isn't actually the right name for this feature. It could be `?_key=` perhaps, or `?_field=`.

  Being able to pass `?_field=count,-rows` to get back just the count (and skip executing the count entirely) would be pretty neat.

  Although `?_only=count` would be tidier. So maybe the pair of `?_only=` and `?_extra=` would make sense.

  Would `?_only=rows` still return the `"ok"` field so you can always look at that to confirm an error didn't occur?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1399341151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5TaEBf
created_at: 2023-01-21T22:03:20Z
updated_at: 2023-01-21T22:03:20Z
author_association: OWNER
body:
  I think I'm going to have to write a new view function from scratch which completely ignores the existing BaseView/DataView/TableView hierarchy.

  Here's what I get on the incoming request:
  ```
  (Pdb) request.url, request.full_path, request.host, request.url_vars
  ('http://127.0.0.1:8001/content/repos.json', '/content/repos.json', '127.0.0.1:8001',
    {'database': 'content', 'table': 'repos', 'format': 'json'})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-1399341761
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: IC_kwDOBm6k_c5TaELB
created_at: 2023-01-21T22:07:19Z
updated_at: 2023-01-21T22:07:19Z
author_association: OWNER
body:
  Idea for supporting streaming with the `register_output_renderer` hook:

  ```python
  @hookimpl
  def register_output_renderer(datasette):
      return {
          "extension": "test",
          "render": render_demo,
          "can_render": can_render_demo,
          "render_stream": render_demo_stream, # This is new
      }
  ```
  So there's a new `"render_stream"` key which can be returned, which if present means that the output renderer supports streaming.

  I'll play around with the design of that function signature in:

  - #1999
  - #1062 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1399343659
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5TaEor
created_at: 2023-01-21T22:19:20Z
updated_at: 2023-02-06T23:02:12Z
author_association: OWNER
body:
  HTML mode needs a list of renderers so it can show links to `.geojson` etc - can do that as a hidden extra (maybe called `renderers`), repeating this code:

  https://github.com/simonw/datasette/blob/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d/datasette/views/base.py#L477-L497

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1159#issuecomment-1399589414
issue_url: https://api.github.com/repos/simonw/datasette/issues/1159
node_id: IC_kwDOBm6k_c5TbAom
created_at: 2023-01-22T19:48:41Z
updated_at: 2023-01-22T19:48:41Z
author_association: CONTRIBUTOR
body:
  Hey @lovasoa, I hope you don't mind - I pulled this PR into [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras), a plugin I'm making that collects UI tweaks to Datasette.

  You can apply it to your own Datasette instance by running `datasette install datasette-ui-extras`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1159#issuecomment-1399606104
issue_url: https://api.github.com/repos/simonw/datasette/issues/1159
node_id: IC_kwDOBm6k_c5TbEtY
created_at: 2023-01-22T20:58:10Z
updated_at: 2023-01-22T20:58:10Z
author_association: NONE
body:
  great initiative, @cldellow :+1: 

  @simonw, if you want to merge this, that would still be welcome :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2000#issuecomment-1399847946
issue_url: https://api.github.com/repos/simonw/datasette/issues/2000
node_id: IC_kwDOBm6k_c5Tb_wK
created_at: 2023-01-23T06:08:00Z
updated_at: 2023-01-23T06:08:00Z
author_association: CONTRIBUTOR
body:
  Actually, I discovered [your post](https://til.simonwillison.net/datasette/register-new-plugin-hooks) showing how a plugin can add a Datasette hook. That's wild! I've released `datasette-rewrite-sql` that adds this ability, albeit via monkey patching.

  I had hoped to be able to expose `request` to the hook (or, even better `actor`) when the SQL was being run as a result of a user's HTTP request.

  But some spelunking in the code makes me suspect that would actually require co-operation from Datasette itself. I'd be happy to be wrong and pointed in the right direction, though!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1994#issuecomment-1399957897
issue_url: https://api.github.com/repos/simonw/datasette/issues/1994
node_id: IC_kwDOBm6k_c5TcamJ
created_at: 2023-01-23T08:21:08Z
updated_at: 2023-01-23T08:21:08Z
author_association: NONE
body: Me too, on a M1. Not sure if it's compatible?
reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 1, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1989#issuecomment-1402347667
issue_url: https://api.github.com/repos/simonw/datasette/issues/1989
node_id: IC_kwDOBm6k_c5TliCT
created_at: 2023-01-24T17:48:59Z
updated_at: 2023-01-24T17:48:59Z
author_association: NONE
body:
  The problem (in my particular use case) with using a VIEW is that I'd need one of the columns to be searchable – but that ([enable-fts](https://github.com/simonw/datasette-search-all)) doesn't work with views :/

  __
  side-suggestion: I don't know how feasible this might be, but when one column (or table) would be marked as hidden, could the _Download SQLite DB_ link take that into account? 🧐

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1099#issuecomment-1402563930
issue_url: https://api.github.com/repos/simonw/datasette/issues/1099
node_id: IC_kwDOBm6k_c5TmW1a
created_at: 2023-01-24T20:11:11Z
updated_at: 2023-01-24T20:11:11Z
author_association: CONTRIBUTOR
body:
  hi @simonw, this bug bit me today.

  the UX for linking from a table to the foreign key seems tough! 

  the design in the other direction seems a lot easier, for a given primary key detail page, add links back to the tables that refer to the row.

  would you be open to a PR that solved the second problem but not the first?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2003#issuecomment-1402894191
issue_url: https://api.github.com/repos/simonw/datasette/issues/2003
node_id: IC_kwDOBm6k_c5Tnndv
created_at: 2023-01-25T00:49:23Z
updated_at: 2023-01-25T00:49:23Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.11**% // Head: **92.12**% // Increases project coverage by **`+0.01%`** :tada:
  > Coverage data is based on head [(`1e5b42f`)](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`e4ebef0`)](https://codecov.io/gh/simonw/datasette/commit/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch coverage: 100.00% of modified lines in pull request are covered.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2003      +/-   ##
  ==========================================
  + Coverage   92.11%   92.12%   +0.01%     
  ==========================================
    Files          38       38              
    Lines        5555     5565      +10     
  ==========================================
  + Hits         5117     5127      +10     
    Misses        438      438              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.50% <100.00%> (+0.01%)` | :arrow_up: |
  | [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `95.73% <100.00%> (+0.04%)` | :arrow_up: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.89% <100.00%> (+0.02%)` | :arrow_up: |
  | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.82% <100.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.61% <100.00%> (+0.03%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/2003?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1099#issuecomment-1402900354
issue_url: https://api.github.com/repos/simonw/datasette/issues/1099
node_id: IC_kwDOBm6k_c5Tno-C
created_at: 2023-01-25T00:58:26Z
updated_at: 2023-01-25T00:58:26Z
author_association: CONTRIBUTOR
body:
  > My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.

  it's pretty hard to know what the right thing to do is if a field is part of multiple foreign keys. 

  but, if that's not the case, what about making each of the columns a link. seems like an improvement over the status quo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2001#issuecomment-1403053144
issue_url: https://api.github.com/repos/simonw/datasette/issues/2001
node_id: IC_kwDOBm6k_c5ToORY
created_at: 2023-01-25T03:34:53Z
updated_at: 2023-01-25T03:34:53Z
author_association: CONTRIBUTOR
body:
  Your comment introduced me to this issue in sqlite and to the `ctypes` module - thanks!

  > I also hope that the datasette developers will enable this mode in a test environment [...]
  > perhaps we could figure out how to invoke it using `ctypes`

  I'm not a Datasette developer, but I _am_ curious to learn more about getting unholy access to the sqlite C APIs inside of Datasette. (Such access could also help #1293, and if done without grovelling inside of pysqlite's Connection object for the db handle, could even be relatively safe.)

  I experimented a bit. I came up with https://gist.github.com/cldellow/85bba507c314b127f85563869cd94820

  If you run `python3 enable-strict-quoting-sqlite3.py`, it seems to set those flags correctly -- `SELECT "foo"` fails where it would normally succeed.

  But if you put it in a `plugins/` dir and run `datasette --plugins-dir plugins/`, it segfaults when it tries to call `sqlite3_db_config` on the connections created by Datasette.

  I am... confused. I'm _pretty_ sure I'm using the same python and the same libsqlite3 in both scenarios, so I would expect it to work.

  @gwk do you know anything that might help me debug the segfault? I gather that my approach of going grovelling inside of a `PyObject` is particularly dangerous, but I was thinking (a) it's necessary in order to test Datasette's use of the sqlite3 library and (b) even if it's not portable, it'd be good enough for running the tests on a single machine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2001#issuecomment-1403071122
issue_url: https://api.github.com/repos/simonw/datasette/issues/2001
node_id: IC_kwDOBm6k_c5ToSqS
created_at: 2023-01-25T04:12:41Z
updated_at: 2023-01-25T04:12:41Z
author_association: NONE
body:
  @cldellow glad to hear you tried it, as I got grossed out by my own suggestion ;) If you are on macOS I do have one trick for debugging segfaults using lldb.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2001#issuecomment-1403078134
issue_url: https://api.github.com/repos/simonw/datasette/issues/2001
node_id: IC_kwDOBm6k_c5ToUX2
created_at: 2023-01-25T04:20:43Z
updated_at: 2023-01-25T04:22:28Z
author_association: CONTRIBUTOR
body:
  I'm on Ubuntu, unfortunately. :( Would it still be relevant?

  I think I've narrowed things down a bit more.

  Even `sqlite3_free(sqlite3_malloc(128))` segfaults -- this suggests to me that it's something about the sqlite3 library that was loaded, vs, say, getting the wrong db handle when I go spelunking in the Connection object.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2001#issuecomment-1403084856
issue_url: https://api.github.com/repos/simonw/datasette/issues/2001
node_id: IC_kwDOBm6k_c5ToWA4
created_at: 2023-01-25T04:31:02Z
updated_at: 2023-01-25T04:31:02Z
author_association: CONTRIBUTOR
body:
  Aha, it's user error on my part.

  Adding

  ```
  sqlite3_db_config.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_int]
  ```

  makes it work reliably both on the CLI and from datasette, and now I can reproduce the errors you mentioned in the issue description.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2004#issuecomment-1403110269
issue_url: https://api.github.com/repos/simonw/datasette/issues/2004
node_id: IC_kwDOBm6k_c5TocN9
created_at: 2023-01-25T05:18:54Z
updated_at: 2023-01-25T05:18:54Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/2004?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.11**% // Head: **92.11**% // No change to project coverage :thumbsup:
  > Coverage data is based on head [(`dca7634`)](https://codecov.io/gh/simonw/datasette/pull/2004?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`e4ebef0`)](https://codecov.io/gh/simonw/datasette/commit/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2004   +/-   ##
  =======================================
    Coverage   92.11%   92.11%           
  =======================================
    Files          38       38           
    Lines        5555     5555           
  =======================================
    Hits         5117     5117           
    Misses        438      438           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/2004?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/2004?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.86% <ø> (ø)` | |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/2004?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2003#issuecomment-1404065571
issue_url: https://api.github.com/repos/simonw/datasette/issues/2003
node_id: IC_kwDOBm6k_c5TsFcj
created_at: 2023-01-25T18:44:42Z
updated_at: 2023-01-25T18:44:42Z
author_association: CONTRIBUTOR
body:
  see this related discussion to a change in API in sqlite-utils https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/203#issuecomment-1404070841
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/203
node_id: IC_kwDOCGYnMM5TsGu5
created_at: 2023-01-25T18:47:18Z
updated_at: 2023-01-25T18:47:18Z
author_association: CONTRIBUTOR
body:
  i'll adopt this PR to make the changes @simonw suggested https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1404253358
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5TszSu
created_at: 2023-01-25T21:35:32Z
updated_at: 2023-01-25T21:35:32Z
author_association: OWNER
body:
  This issue here would benefit from some kid of mechanism for returning just the HTML of the table itself, without any of the surrounding material. I'm not sure if that would make sense as an extra or not:
  - https://github.com/simonw/datasette-search-all/issues/17

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2005#issuecomment-1404458011
issue_url: https://api.github.com/repos/simonw/datasette/issues/2005
node_id: IC_kwDOBm6k_c5TtlQb
created_at: 2023-01-26T01:41:30Z
updated_at: 2023-01-26T01:41:50Z
author_association: OWNER
body:
  My code looked like this:
  ```python
  @hookimpl
  def extra_template_vars(datasette, view_name, database):
      async def inner():
          if view_name == "database":
              return {"blah": 1}

      return inner
  ```
  It returns `None` in the case where the `if` condition does not match.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2006#issuecomment-1405488523
issue_url: https://api.github.com/repos/simonw/datasette/issues/2006
node_id: IC_kwDOBm6k_c5Txg2L
created_at: 2023-01-26T19:20:32Z
updated_at: 2023-01-26T19:20:32Z
author_association: OWNER
body:
  This won't actually help that much if the user's GitHub Actions workflow does the equivalent of this:
  ```
  pip install datasette
  datasette publish cloudrun mydb.db ...
  ```
  Since they'll get the 1.0 release too.

  Since they'll need to make changes anyway, maybe a better solution is to document this and tell people they should use this instead:

  ```
  datasette publish cloudrun mydb.db ... --branch 0.64.1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2006#issuecomment-1405488884
issue_url: https://api.github.com/repos/simonw/datasette/issues/2006
node_id: IC_kwDOBm6k_c5Txg70
created_at: 2023-01-26T19:20:53Z
updated_at: 2023-01-26T19:20:53Z
author_association: OWNER
body:
  I can run a GitHub code search a week or so before the release to give me time to get in touch with anyone who has actions that look like they might break.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/523#issuecomment-1407264466
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/523
node_id: IC_kwDOCGYnMM5T4SbS
created_at: 2023-01-28T02:41:14Z
updated_at: 2023-01-28T02:41:14Z
author_association: CONTRIBUTOR
body:
  I also often then run another little script to cast all empty strings to null, but i save that for another issue if this gets accepted.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2008#issuecomment-1407471459
issue_url: https://api.github.com/repos/simonw/datasette/issues/2008
node_id: IC_kwDOBm6k_c5T5E9j
created_at: 2023-01-28T19:40:18Z
updated_at: 2023-01-29T04:55:39Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Base: **92.11**% // Head: **91.78**% // Decreases project coverage by **`-0.34%`** :warning:
  > Coverage data is based on head [(`f529a30`)](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`e4ebef0`)](https://codecov.io/gh/simonw/datasette/commit/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > Patch has no changes to coverable lines.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2008      +/-   ##
  ==========================================
  - Coverage   92.11%   91.78%   -0.34%     
  ==========================================
    Files          38       39       +1     
    Lines        5555     5599      +44     
  ==========================================
  + Hits         5117     5139      +22     
  - Misses        438      460      +22     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/facets.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZhY2V0cy5weQ==) | `91.84% <ø> (ø)` | |
  | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.82% <0.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.57% <0.00%> (ø)` | |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.61% <0.00%> (ø)` | |
  | [datasette/utils/shutil\_backport.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3NodXRpbF9iYWNrcG9ydC5weQ==) | `9.09% <0.00%> (ø)` | |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.40% <0.00%> (+2.77%)` | :arrow_up: |
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `85.29% <0.00%> (+2.94%)` | :arrow_up: |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `93.12% <0.00%> (+3.05%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/2008?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1973#issuecomment-1407523547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1973
node_id: IC_kwDOBm6k_c5T5Rrb
created_at: 2023-01-29T00:40:31Z
updated_at: 2023-01-29T00:40:31Z
author_association: CONTRIBUTOR
body:
  A +1 for switching to `CustomRow`: I think you currently only get a `CustomRow` if the result set had a column that was an fkey ([this code](https://github.com/simonw/datasette/blob/3c352b7132ef09b829abb69a0da0ad00be5edef9/datasette/views/table.py#L667-L682))

  Otherwise you get vanilla `sqlite3.Row`s, which will fail if you try to access `.columns` or lookup the cell by name, which surprised me recently

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2008#issuecomment-1407558284
issue_url: https://api.github.com/repos/simonw/datasette/issues/2008
node_id: IC_kwDOBm6k_c5T5aKM
created_at: 2023-01-29T04:23:58Z
updated_at: 2023-01-29T04:24:27Z
author_association: CONTRIBUTOR
body:
  Ack, this PR is broken. I see now that the `inner.*` is necessary for ensuring the correct count in the face of rows having duplicate values in views.

  That fixes the overcounting, but I think can undercount when the rows have the same data, eg a view like:

  ```sql
  SELECT '["bar"]' tags UNION ALL SELECT '["bar"]'
  ```

  will produce a count of `{"bar": 1 }`, when it should be `{"bar": 2}`. In fact, this could apply in tables without primary keys, too.

  If `inner` came from a base table that had a primary key or a rowid, we could use those column(s) to solve that case.

  I guess a general solution would be to compute a window function so we have a distinct ID for each row. Will fiddle to see if I can get that working.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2008#issuecomment-1407561308
issue_url: https://api.github.com/repos/simonw/datasette/issues/2008
node_id: IC_kwDOBm6k_c5T5a5c
created_at: 2023-01-29T04:50:50Z
updated_at: 2023-01-29T04:50:50Z
author_association: CONTRIBUTOR
body:
  I pushed a revised version which ends up being faster -- the example which currently takes 4 seconds now runs in 500ms.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2008#issuecomment-1407567753
issue_url: https://api.github.com/repos/simonw/datasette/issues/2008
node_id: IC_kwDOBm6k_c5T5ceJ
created_at: 2023-01-29T05:39:54Z
updated_at: 2023-01-29T05:40:34Z
author_association: OWNER
body:
  I absolutely _love_ this performance boost - really nice find.

  One concern: this will be the first time Datasette ships a core feature that uses window functions.

  Window functions were added to SQLite in [version 3.25.0](https://www.sqlite.org/releaselog/3_25_0.html) on 2018-09-15 - which means it's still very common for Datasette to run on versions that don't yet support them.

  So I see two options:
  - Detect window function support and switch between the old implementation and this better, new one
  - Detect window functions and disable the facet-by-JSON feature entirely if they are missing

  I like the first option a bit better.

  This also leads to a tricky CI challenge: Datasette needs to be able to run its test suite against more than one SQLite version to confidently test this feature going forward.

  I don't yet have a good GitHub Actions recipe for this, but I _really_ need one - for `sqlite-utils` too.

  Might be able to use this trick for that: https://til.simonwillison.net/sqlite/ld-preload

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2008#issuecomment-1407568923
issue_url: https://api.github.com/repos/simonw/datasette/issues/2008
node_id: IC_kwDOBm6k_c5T5cwb
created_at: 2023-01-29T05:47:36Z
updated_at: 2023-01-29T05:47:36Z
author_association: OWNER
body:
  > I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.

  We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

  I see not having to change the tests at all for this change as a really positive sign. If you find any behaviour differences between this and the previous that's a sign we should add a mother test or two specifying the behaviour we want.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2008#issuecomment-1407716963
issue_url: https://api.github.com/repos/simonw/datasette/issues/2008
node_id: IC_kwDOBm6k_c5T6A5j
created_at: 2023-01-29T17:04:03Z
updated_at: 2023-01-29T17:04:03Z
author_association: CONTRIBUTOR
body:
  Performance tests - I think most places don't have them as a formal gate enforced by CI. TypeScript and scalac seem to have tests that run to capture timings. The timings are included by a bot as a comment or build check, and also stored in a database so you can graph changes over time to spot regressions. Probably overkill for Datasette!

  Window functions - oh, good point. Looks like Ubuntu shipped JSON1 support as far back as sqlite 3.11. I'll let this PR linger until there's a way to run against different SQLite versions. For now, I'm shipping this with `datasette-ui-extras`, since I think it's OK for a plugin to enforce a higher minimum requirement.

  Tests - there actually did end up being test changes to capture the undercount bug of the current implementation, so the current implementation would fail against the new tests.

  Perhaps a non-window function version could be written that uses `random()` instead of `row_number() over ()` in order to get a unique key. It's technically not unique, but in practice, I imagine it'll work well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2008#issuecomment-1407733793
issue_url: https://api.github.com/repos/simonw/datasette/issues/2008
node_id: IC_kwDOBm6k_c5T6FAh
created_at: 2023-01-29T18:17:40Z
updated_at: 2023-01-29T18:17:40Z
author_association: OWNER
body:
  > We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

  Had an interesting conversation about this just now: https://fedi.simonwillison.net/@simon/109773800944614366

  There's a risk that different runs will return different results due to the shared resource nature of GitHub Actions runners, but a good fix for that is to run comparative tests where you run the benchmark against e.g. both `main` and the incoming PR branch and report back on any differences.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1696#issuecomment-1407767434
issue_url: https://api.github.com/repos/simonw/datasette/issues/1696
node_id: IC_kwDOBm6k_c5T6NOK
created_at: 2023-01-29T20:56:20Z
updated_at: 2023-01-29T20:56:20Z
author_association: CONTRIBUTOR
body:
  I did some horrible things in https://github.com/cldellow/datasette-ui-extras/issues/2 to enable this in my plugin -- example here: https://dux-demo.fly.dev/cooking/posts?_facet=owner_user_id&owner_user_id=67

  The implementation relies on two things:

  - a `filters_from_request` hook that adds a good human description (unfortunately, without the benefit of the CSS styling you mention)
  - doing something evil to hijack the `exact` and `not` operators in the `Filters` class. We can't leave them as is, or we'll get 2 human descriptions -- the built-in Datasette one and the one from my plugin. We can't remove them, or the filters UI will stop supporting the `=` and `!=` operators

  This got me thinking: it'd be neat if the list of operators that the filters UI supported wasn't a closed set.

  A motivating example: adding a geospatial `NEAR` operator. Ideally it'd take two arguments - a target point and a radius, so you could express a filter like `find me all rows whose lat/lng are within 10km of 43.4516° N, 80.4925° W`. (Optionally, the UI could be enhanced if the geonames database was loaded and queried, so a user could say `find me all rows whose lat/lng are within 10km of Kitchener, ON`, and the city gets translated to a lat/lng for them)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2010#issuecomment-1409406327
issue_url: https://api.github.com/repos/simonw/datasette/issues/2010
node_id: IC_kwDOBm6k_c5UAdV3
created_at: 2023-01-30T21:51:58Z
updated_at: 2023-01-30T21:51:58Z
author_association: OWNER
body:
  Here's a quick prototype I knocked up for this:
  ```diff
  diff --git a/datasette/static/app.css b/datasette/static/app.css
  index 71437bd4..d763bcff 100644
  --- a/datasette/static/app.css
  +++ b/datasette/static/app.css
  @@ -695,7 +695,48 @@ p.zero-results {
   
   
   
  +/* Force table to not be like tables anymore */
  +body.row table.rows-and-columns,
  +body.row .rows-and-columns thead,
  +body.row .rows-and-columns tbody,
  +body.row .rows-and-columns th,
  +body.row .rows-and-columns td,
  +body.row .rows-and-columns tr {
  +    display: block;
  +}
  +
  +/* Hide table headers (but not display: none;, for accessibility) */
  +body.row .rows-and-columns thead tr {
  +    position: absolute;
  +    top: -9999px;
  +    left: -9999px;
  +}
  +
  +body.row .rows-and-columns tr {
  +    border: 1px solid #ccc;
  +    margin-bottom: 1em;
  +    border-radius: 10px;
  +    background-color: white;
  +    padding: 0.2rem;
  +}
   
  +body.row .rows-and-columns td {
  +    /* Behave  like a "row" */
  +    border: none;
  +    border-bottom: 1px solid #eee;
  +    padding: 0;
  +    padding-left: 10%;
  +    padding-bottom: 0.3em;
  +}
  +
  +body.row .rows-and-columns td:before {
  +    display: block;
  +    color: black;
  +    padding-bottom: 0.2em;
  +    font-size: 0.8em;
  +    font-weight: bold;
  +    background-color: #f5f5f5;
  +}
   
   
   /* Overrides ===============================================================*/
  diff --git a/datasette/templates/row.html b/datasette/templates/row.html
  index 1d1b0bfd..339eb643 100644
  --- a/datasette/templates/row.html
  +++ b/datasette/templates/row.html
  @@ -5,6 +5,9 @@
   {% block extra_head %}
   {{- super() -}}
   <style>
  +{% for column in columns %}
  +body.row .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: "{{ column|escape_css_string }}"; }
  +{% endfor %}
   @media only screen and (max-width: 576px) {
   {% for column in columns %}
       .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: "{{ column|escape_css_string }}"; }
  ```
  Now the row page looks like this at all page widths:

  <img width="776" alt="image" src="https://user-images.githubusercontent.com/9599/215603142-4d020ac0-cc33-436c-a1c6-17c9bcf9edb1.png">

  I think that's better (could do with a bit of tightening up).

  One catch: you can't copy and paste the column labels, since they are added using generated content like this:

  https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/static/app.css#L752-L757

  https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/templates/row.html#L9-L11

  I think the row page should switch to different HTML entirely, rather than continuing to share the `<table>` that's used by the table page.

  This will be a breaking change for users who customize Datasette, so I should aim to ship it before 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2011#issuecomment-1410827249
issue_url: https://api.github.com/repos/simonw/datasette/issues/2011
node_id: IC_kwDOBm6k_c5UF4Px
created_at: 2023-01-31T17:58:54Z
updated_at: 2023-01-31T17:58:54Z
author_association: OWNER
body:
  I think this is the relevant code:

  https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/facets.py#L260-L268

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2016#issuecomment-1418288077
issue_url: https://api.github.com/repos/simonw/datasette/issues/2016
node_id: IC_kwDOBm6k_c5UiVvN
created_at: 2023-02-05T22:56:43Z
updated_at: 2023-02-05T22:56:43Z
author_association: OWNER
body:
  This absolutely makes sense. One of the biggest goals for Datasette 1.0 is "documented template contexts" - for any default template in Datasette that people might want to over-ride there should be documentation that describes the available context variables, plus tests that ensure they don't accidentally get broken by future changes.

  Ensuring description/title/etc are available on the index page feels like it fits well into that bucket.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419357290
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Umaxq
created_at: 2023-02-06T16:21:44Z
updated_at: 2023-02-06T16:21:44Z
author_association: CONTRIBUTOR
body:
  SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419390560
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Umi5g
created_at: 2023-02-06T16:43:47Z
updated_at: 2023-02-06T16:43:47Z
author_association: NONE
body:
  > SQLite doesn't have a native `DATETIME` type. It stores dates internally as strings and then has [functions](https://www.sqlite.org/lang_datefunc.html) to work with date-like strings. Yes it's weird.

  That's correct. But my issue is about the application level libraries that, i suppose, have better data understanding if see a specific type such as `DATETIME`. 

  I'm writing data with **dataset** i've mentioned. The lib changes its behavior depending on a type. I saw different behavior with types `DATETIME, FLOAT, TEXT`. Dataset, for their part, is built upon Sqlalchemy, you know what it is.

  To be honest, i didn't dive into the details of why the behavior changes, but when i altered manually by other util a type of column to `DATETIME` things got back to normal.

  On the matter, can i achieve it with Sqlite Utils at the moment?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419734229
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Un2zV
created_at: 2023-02-06T20:53:28Z
updated_at: 2023-02-06T21:16:29Z
author_association: NONE
body:
  I think it's not currently possible: sqlite-utils requires that it be one of `integer`, `text`, `float`, `blob` ([see code](https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/cli.py#L2266))

  IMO, this is a bit of friction and it would be nice if it was more permissive. SQLite permits developers to use any data type when creating a table. For example, this is a perfectly cromulent sqlite session that creates a table with columns of type `baz` and `bar`:

  ```
  sqlite> create table foo(column1 baz, column2 bar);
  sqlite> .schema foo
  CREATE TABLE foo(column1 baz, column2 bar);
  sqlite> select * from pragma_table_info('foo');
  cid         name        type        notnull     dflt_value  pk        
  ----------  ----------  ----------  ----------  ----------  ----------
  0           column1     baz         0                       0         
  1           column2     bar         0                       0   
  ```

  The idea is that the application developer will know what meaning to ascribe to those types. For example, I'm working on a plugin to Datasette. Dates are tricky to handle. If you have some existing rows, you can look at the values in them to know how a user is serializing the dates -- as an ISO 8601 string? An RFC 3339 string? With millisecond precision? With timezone offset? But if you don't yet have any rows, you have to guess. If the column is of type `TEXT`, you don't even know that it's meant to hold a date! In this case, my plugin will look to see if the column is of type `DATE` or `DATETIME`, and assume a certain representation when writing.

  Perhaps there is an argument that sqlite-utils is trying to conform to SQLite's strict mode, and that is why it limits the choices. In strict mode, SQLite requires that the data type be one of `INT`, `INTEGER`, `REAL`, `TEXT`, `BLOB`, `ANY`. But that can't be the case -- sqlite-utils supports `FLOAT`, which is not one of the valid types in strict mode, and it rejects `INT`, `REAL` and `ANY`, which _are_ valid.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1419740776
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Un4Zo
created_at: 2023-02-06T20:59:01Z
updated_at: 2023-02-06T20:59:01Z
author_association: NONE
body:
  That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1419916684
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UojWM
created_at: 2023-02-06T23:09:51Z
updated_at: 2023-02-06T23:10:13Z
author_association: OWNER
body:
  The inputs and outputs for this are pretty complex.

  Inputs:

  - `?_next=` from the query string
  - `is_view` - is this for a table or view? If it's a view it uses offset/limit pagination - which could actually work for arbitrary queries too. Also views could have keyset pagination if they are known to be sorted by a particular column.
  - `sort` and `sort_desc` reflecting the current sort order
  - `use_rowid` for if the table is a rowid table with no primary key of its own
  - `pks` - the primary keys for the table
  - `params` - the current set of parameters, I think used just to count their length so new params can be added as `p5` etc without collisions. This could be handled with a `s0`, `s1` etc naming convention instead.

  Outputs:

  - `where_clauses` - a list of where clauses to add to the query
  - `params` - additional parameters to use with the query due to the new where clauses
  - `order_by` - the order by clause

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1419917661
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5Uojld
created_at: 2023-02-06T23:10:51Z
updated_at: 2023-02-06T23:10:51Z
author_association: OWNER
body:
  I should turn `sort` and `sort_desc` into an object representing the sort order earlier in the code.

  I should also create something that bundles together `pks` and `use_rowid` and maybe `is_view` as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1419921228
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UokdM
created_at: 2023-02-06T23:14:15Z
updated_at: 2023-02-06T23:14:15Z
author_association: OWNER
body:
  Crucial utility function: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/utils/__init__.py#L137-L160

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1419928455
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UomOH
created_at: 2023-02-06T23:21:50Z
updated_at: 2023-02-06T23:21:50Z
author_association: OWNER
body:
  Found more logic relating to this:

  https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L684-L732

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1419953256
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UosRo
created_at: 2023-02-06T23:42:56Z
updated_at: 2023-02-06T23:43:10Z
author_association: OWNER
body:
  Relevant issue:
  - https://github.com/simonw/datasette/issues/1773

  Explains this comment: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L697

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1420088670
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UpNVe
created_at: 2023-02-07T02:14:35Z
updated_at: 2023-02-07T02:14:35Z
author_association: OWNER
body:
  Maybe the correct level of abstraction here is that pagination is something that happens to a SQL query that is defined as SQL and params, without an order by or limit. That's then wrapped in a sub-select and those things are added to it, plus the necessary `where` clauses depending on the page.

  Need to check that the query plan for pagination of a subquery isn't slower than the plan for pagination as it works today.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1420094396
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UpOu8
created_at: 2023-02-07T02:18:11Z
updated_at: 2023-02-07T02:19:16Z
author_association: OWNER
body:
  For the SQL underlying this page (the second page in that compound primary key paginated sequence): https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv

  The explain for the default query: https://latest.datasette.io/fixtures?sql=explain+select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0Awhere%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

  The explain for that query rewritten as this:

  ```sql
  explain
  select
    *
  from
    (
      select
        pk1,
        pk2,
        pk3,
        content
      from
        compound_three_primary_keys
    )
  where
    (
      (pk1 > :p0)
      or (
        pk1 = :p0
        and pk2 > :p1
      )
      or (
        pk1 = :p0
        and pk2 = :p1
        and pk3 > :p2
      )
    )
  order by
    pk1,
    pk2,
    pk3
  limit
    101
  ```
  https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28select+%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0A%29%0D%0A++where%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

  Both explains have 31 steps and look pretty much identical.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1420101175
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UpQY3
created_at: 2023-02-07T02:22:11Z
updated_at: 2023-02-07T02:22:11Z
author_association: OWNER
body:
  A more complex example: https://latest.datasette.io/fixtures/sortable?_next=0~2E2650566289400591%2Ca%2Cu&_sort=sortable_with_nulls_2

  SQL:

  ```sql
  select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0)
    or
  (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101
  ```

  https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p0=a&p1=u&p2=0.2650566289400591

  Here's the explain: 49 steps long https://latest.datasette.io/fixtures?sql=explain+select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

  Rewritten with a subselect:

  ```sql
  select * from (
    select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable
  )
  where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0)
    or
  (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101
  ```
  https://latest.datasette.io/fixtures?sql=select+*+from+(%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A)%0D%0Awhere+(sortable_with_nulls_2+%3E+%3Ap2+or+(sortable_with_nulls_2+%3D+%3Ap2+and+((pk1+%3E+%3Ap0)%0D%0A++or%0D%0A(pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1))))+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

  And here's the explain for that - also 49 steps: https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A%29%0D%0Awhere+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1420104254
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UpRI-
created_at: 2023-02-07T02:24:46Z
updated_at: 2023-02-07T02:24:46Z
author_association: OWNER
body:
  Even more complicated: https://latest.datasette.io/fixtures/sortable?sortable_with_nulls__notnull=1&_next=0~2E692704598586882%2Ce%2Cr&_sort=sortable_with_nulls_2

  The rewritten SQL for that is:

  ```sql
  select * from (select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where "sortable_with_nulls" is not null)
    where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0)
    or
  (pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101
  ```
  And it still has the same number of explain steps as the current SQL witohut the subselect.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1420106315
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UpRpL
created_at: 2023-02-07T02:28:03Z
updated_at: 2023-02-07T02:28:36Z
author_association: OWNER
body:
  So I think I can write an abstraction that applies keyset pagination to ANY arbitrary SQL query provided it is given the query, the existing params (so it can pick names for the new params that won't overlap with them), the desired sort order, any existing `_next` token AND the columns that should be used to tie-break any duplicates.

  Those tie breakers will be either the primary key(s) or `rowid` if none are provided.

  What about the case of SQL views, where offset/limit should be used instead? I'm inclined to have that as a separate pagination abstraction entirely, with the calling code deciding which pagination helper to use based on if keyset pagination makes sense or not.

  Might be easier to design a class structure for this starting with `OffsetPaginator`, then using that to inform the design of `KeysetPaginator`.

  Might put these in `datasette.utils.pagination` to start off with, then maybe extract them out to `sqlite-utils` later once they've proven themselves.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1420109153
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5UpSVh
created_at: 2023-02-07T02:32:36Z
updated_at: 2023-02-07T02:32:36Z
author_association: OWNER
body:
  Doing this as a class makes sense to me. There are a few steps:

  - Instantiate the class with the information it needs, which includes sort order, page size, tiebreaker columns and SQL query and parameters
  - Generate the new SQL query that will actually be executed - maybe this takes the optional `_next` parameter? This returns the SQL and params that should be executed, where the SQL now includes pagination logic plus order by and limit
  - The calling code then gets to execute the SQL query to fetch the rows
  - Last step: those rows are passed to a paginator method which returns `(rows, next)` - where `rows` is the rows truncated to the correct length (really just with the last one cut off if it's too long for the length) and `next` is either `None` or a token, depending on if there should be a next page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420496447
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Uqw4_
created_at: 2023-02-07T09:57:38Z
updated_at: 2023-02-07T09:57:38Z
author_association: NONE
body:
  > That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.

  It works, but a column becomes `TEXT`

  ```python
  In [1]: import sqlite_utils
  In [2]: db = sqlite_utils.Database('events.sqlite')
  In [3]: table = db['cards.chunk.get']
  In [4]: table.columns_dict
  Out[4]:
  {'id': int,
   'timestamp': float,
   'data_chunk_number': int,
   'user_id': str,
   'meta_duplication_source_id': int,
   'context_sort_attribute': str,
   'context_sort_order': str}

  In [5]: from datetime import datetime
  In [7]: table.transform(types={'timestamp': datetime})
  In [8]: table.columns_dict
  Out[8]:
  {'id': int,
   'timestamp': str,
   'data_chunk_number': int,
   'user_id': str,
   'meta_duplication_source_id': int,
   'context_sort_attribute': str,
   'context_sort_order': str}
  ```

  ```bash
  ❯ sqlite-utils schema events.sqlite cards.chunk.get
  CREATE TABLE "cards.chunk.get" (
     [id] INTEGER PRIMARY KEY NOT NULL,
     [timestamp] TEXT,
     ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420809773
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Ur9Yt
created_at: 2023-02-07T13:53:01Z
updated_at: 2023-02-07T13:53:01Z
author_association: NONE
body:
  Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178

  I suspect you could overwrite the datetime entry to achieve what you want

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/564#issuecomment-1420941334
issue_url: https://api.github.com/repos/simonw/datasette/issues/564
node_id: IC_kwDOBm6k_c5UsdgW
created_at: 2023-02-07T15:14:10Z
updated_at: 2023-02-07T15:14:10Z
author_association: CONTRIBUTOR
body:
  Is this feature covered by any more recent updates to `datasette`, or via any plugins that you're aware of?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420992261
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Usp8F
created_at: 2023-02-07T15:45:58Z
updated_at: 2023-02-07T15:45:58Z
author_association: NONE
body:
  I'd support that, but I'm not the author of this library.

  One challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421022917
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5UsxbF
created_at: 2023-02-07T16:06:03Z
updated_at: 2023-02-07T16:08:58Z
author_association: NONE
body:
  > Do you see a way to enable it without affecting existing users or bumping the major version number?

  I don't see a clean solution, only extending code with a side variable that tells us we want to apply advanced types instead of basic.

  it could be a similiar command like `tranform-v2 --type column DATETIME` or a cli option `transform --adv-type column DATETIME` along with a dict that contains the advanced types. Then with knowledge that we run an advanced command we take that dictionary somehow, we can wrap the current and new dictionaries by a superdict and work with it everywhere according to the knowledge. This way shouldn't affect users who are using the previous lib versions and it have to be merged in the next major one.

  But this way looks a bad design, too messy.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421033725
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Us0D9
created_at: 2023-02-07T16:12:13Z
updated_at: 2023-02-07T16:12:13Z
author_association: NONE
body:
  I think the bigger issue is that `sqlite-utils` mixes mechanism (it implements the [12-step way to alter SQLite tables](https://www.sqlite.org/lang_altertable.html#otheralter)) and policy (it has an opinionated stance on what column types should be used).

  That might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case.

  It might make sense to extract a separate library that provides just the mechanisms, and then `sqlite-utils` would sit on top of that library with its opinionated set of policies.

  That would be a very big change, though.

  I might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that `sqlite-utils` supports. I wouldn't expect `sqlite-utils` to depend on it.

  Part of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421052195
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Us4kj
created_at: 2023-02-07T16:23:17Z
updated_at: 2023-02-07T16:23:57Z
author_association: NONE
body:
  Isn't your suggestion too fundamental for the utility?

  The bigger flexibility, the bigger complexity. Your idea make sense defenitely, but how often do you make schema changes? And how many people could benefit from it, what do you think?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421055590
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Us5Zm
created_at: 2023-02-07T16:25:31Z
updated_at: 2023-02-07T16:25:31Z
author_association: NONE
body:
  > Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178
  > 
  > I suspect you could overwrite the datetime entry to achieve what you want

  And thank you for pointing me to it. At least, i can make a monkey patch for my need...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421081939
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5Us_1T
created_at: 2023-02-07T16:42:25Z
updated_at: 2023-02-07T16:43:42Z
author_association: NONE
body:
  Ha, yes, I might end up making something very niche. That's OK.

  I'm building a UI for [Datasette](https://datasette.io/) that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421177666
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5UtXNC
created_at: 2023-02-07T17:39:00Z
updated_at: 2023-02-07T17:39:00Z
author_association: NONE
body:
  >  lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed

  I've got your concern. Let's see if we will be replied on it and i'll close the issue some later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1421600789
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5Uu-gV
created_at: 2023-02-07T23:12:40Z
updated_at: 2023-02-07T23:16:20Z
author_association: OWNER
body:
  Most complicated example of a paginated query: https://latest.datasette.io/fixtures?sql=select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++content%2C%0D%0A++sortable%2C%0D%0A++sortable_with_nulls%2C%0D%0A++sortable_with_nulls_2%2C%0D%0A++text%0D%0Afrom%0D%0A++sortable%0D%0Awhere%0D%0A++(%0D%0A++++sortable_with_nulls+is+null%0D%0A++++and+(%0D%0A++++++(pk1+%3E+%3Ap0)%0D%0A++++++or+(%0D%0A++++++++pk1+%3D+%3Ap0%0D%0A++++++++and+pk2+%3E+%3Ap1%0D%0A++++++)%0D%0A++++)%0D%0A++)%0D%0Aorder+by%0D%0A++sortable_with_nulls+desc%2C%0D%0A++pk1%2C%0D%0A++pk2%0D%0Alimit%0D%0A++101&p0=h&p1=r

  ```sql
  select
    pk1,
    pk2,
    content,
    sortable,
    sortable_with_nulls,
    sortable_with_nulls_2,
    text
  from
    sortable
  where
    (
      sortable_with_nulls is null
      and (
        (pk1 > :p0)
        or (
          pk1 = :p0
          and pk2 > :p1
        )
      )
    )
  order by
    sortable_with_nulls desc,
    pk1,
    pk2
  limit
    101
  ```
  Generated by this page: https://latest.datasette.io/fixtures/sortable?_next=%24null%2Ch%2Cr&_sort_desc=sortable_with_nulls

  The `_next=` parameter there decodes as `$null,h,r` - and those components are tilde-encoded, so this can be distinguished from an actual `$null` value which would be represented as `~24null`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2019#issuecomment-1421784930
issue_url: https://api.github.com/repos/simonw/datasette/issues/2019
node_id: IC_kwDOBm6k_c5Uvrdi
created_at: 2023-02-08T01:28:25Z
updated_at: 2023-02-08T01:40:46Z
author_association: OWNER
body:
  Rather than duplicate this rather awful hack:

  https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L694-L714

  I'm tempted to say that the code that calls the new pagination helper needs to ensure that the `sort` or `sort_desc` columns are selected. If it wants to ditch them later (e.g. because they were not included in `?_col=`) it can do that later once the results have come back.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1421988953
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5UwdRZ
created_at: 2023-02-08T04:35:44Z
updated_at: 2023-02-08T05:27:48Z
author_association: OWNER
body:
  Next step: get `?_next=...` working (it is ignored at the moment, even though the returned JSON includes the `"next"` key).

  Then... figure out how to render HTML and other requested formats.

  Then get the tests to pass!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1422681850
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/524
node_id: IC_kwDOCGYnMM5UzGb6
created_at: 2023-02-08T14:25:50Z
updated_at: 2023-02-08T14:29:09Z
author_association: NONE
body:
  I live the patch here for others:

  _original code_
  ```shell
  $ which sqlite-utils | xargs cat
  ```
  ```python
  #!/usr/bin/python3
  # -*- coding: utf-8 -*-
  import re
  import sys
  from sqlite_utils.cli import cli


  if __name__ == '__main__':
      sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
      sys.exit(cli())
  ```

  _patched/sqlite-utils.py_
  ```python
  #!/usr/bin/python3
  # -*- coding: utf-8 -*-
  import re
  import sys
  from sqlite_utils.cli import cli

  # New imports
  from unittest.mock import patch
  from sqlite_utils.cli import VALID_COLUMN_TYPES


  if __name__ == '__main__':
      # Choices of the option `--type`
      cli.commands['transform'].params[2].type.types[1].choices.append('DATETIME')

      # The dicts has to be extended with a new type
      with patch.dict('sqlite_utils.db.COLUMN_TYPE_MAPPING', {'DATETIME': 'DATETIME'}),\
           patch('sqlite_utils.cli.VALID_COLUMN_TYPES', VALID_COLUMN_TYPES + ("DATETIME", )):

          # Command is unchanged
          sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
          sys.exit(cli())
  ```

  And now it's working
  ```bash
  $ sqlite-utils schema events.sqlite cards.chunk.get
  CREATE TABLE "cards.chunk.get" (
     [id] INTEGER PRIMARY KEY NOT NULL,
     [timestamp] TEXT,
  )

  $ python patched/sqlite-utils.py transform events.sqlite cards.chunk.get --type timestamp DATETIME

  $ sqlite-utils schema events.sqlite cards.chunk.get
  CREATE TABLE "cards.chunk.get" (
     [id] INTEGER PRIMARY KEY NOT NULL,
     [timestamp] DATETIME,
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1423067724
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5U0kpM
created_at: 2023-02-08T18:33:32Z
updated_at: 2023-02-08T18:36:48Z
author_association: OWNER
body:
  Just realized that it's useful to be able to tell what parameters were used to generate a page... but reflecting things like `_next` back in the JSON is confusing in the presence of `next`.

  So I'm going to add an extra for that information too.

  Not sure what to call it though:

  - `params` - confusing because in the code that's usually used for params passed to SQL queries
  - `query_string` - wouldn't that be a string, not params as a dictionary?

  I'm going to experiment with a `request` extra that returns some bits of information about the request.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2023#issuecomment-1425974877
issue_url: https://api.github.com/repos/simonw/datasette/issues/2023
node_id: IC_kwDOBm6k_c5U_qZd
created_at: 2023-02-10T15:32:41Z
updated_at: 2023-02-10T15:32:41Z
author_association: CONTRIBUTOR
body:
  I think this feature was removed in Datasette 0.61 and moved to a plugin. People who want hashed URLs can use the [datasette-hashed-urls](https://docs.datasette.io/en/stable/performance.html#performance-hashed-urls) plugin to achieve the same affect.

  It looks like you're trying to disable hashed urls, so I think you can just remove that config setting and things will work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2023#issuecomment-1425988018
issue_url: https://api.github.com/repos/simonw/datasette/issues/2023
node_id: IC_kwDOBm6k_c5U_tmy
created_at: 2023-02-10T15:39:59Z
updated_at: 2023-02-10T15:39:59Z
author_association: NONE
body:
  Thanks for confirming my doubts! I removed it after opening this issue, yup, then had another issue with `default_cache_ttl_hashed` which I assume was removed at the same time. Sorry for not trying that before opening the issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1775#issuecomment-1426158181
issue_url: https://api.github.com/repos/simonw/datasette/issues/1775
node_id: IC_kwDOBm6k_c5VAXJl
created_at: 2023-02-10T18:04:40Z
updated_at: 2023-02-10T18:04:40Z
author_association: NONE
body:
  Is this where we talk about i18n of results? Or is that a separate thread.

  e.g. Having `country_long` show `España` in the Spanish version of the [global power plants demo site](https://global-power-plants.datasettes.com/) instead of `Spain`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1435318713
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/525
node_id: IC_kwDOCGYnMM5VjTm5
created_at: 2023-02-17T21:55:01Z
updated_at: 2023-02-17T21:55:01Z
author_association: CONTRIBUTOR
body:
  Meanwhile, a cheap workaround is to invalidate the registered function cache:
  ``` python
  table.convert(...)
  db._registered_functions = set()
  table.convert(...)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1258#issuecomment-1437671409
issue_url: https://api.github.com/repos/simonw/datasette/issues/1258
node_id: IC_kwDOBm6k_c5VsR_x
created_at: 2023-02-20T23:39:58Z
updated_at: 2023-02-20T23:39:58Z
author_association: CONTRIBUTOR
body:
  This is pretty annoying for FTS because sqlite throws an error instead of just doing something like returning all or no results. This makes users who are unfamiliar with SQL and Datasette think the canned query page is broken and is a frequent source of confusion.

  To anyone dealing with this: My solution is to modify the canned query so that it returns no results which cues people to fill in the blank parameters.

  So instead of `emails_fts match escape_fts(:search))`

  My canned queries now look like this:

  `emails_fts match escape_fts(iif(:search=="", "*", :search))`

  There are no asterisks in my data so the result is always blank.

  Ultimately it would be nice to be able to handle this in the metadata. Either making some named parameters required or setting some default values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2027#issuecomment-1440811364
issue_url: https://api.github.com/repos/simonw/datasette/issues/2027
node_id: IC_kwDOBm6k_c5V4Qlk
created_at: 2023-02-22T21:19:47Z
updated_at: 2023-02-22T21:19:47Z
author_association: NONE
body:
  yes @dmick . How did you make your public IP redirect to your uvicorn server?

  Instead of nginx, I have apache2 on my GCP VM. Any pointers here are helpful too.

  Thanks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2030#issuecomment-1440814680
issue_url: https://api.github.com/repos/simonw/datasette/issues/2030
node_id: IC_kwDOBm6k_c5V4RZY
created_at: 2023-02-22T21:22:42Z
updated_at: 2023-02-22T21:22:42Z
author_association: NONE
body:
  @gk7279, you had asked in a separate bug about how to redirect web servers in general.  The datasette docs actually have pretty good information on this for both nginx and apache2: https://docs.datasette.io/en/stable/deploying.html#running-datasette-behind-a-proxy

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2030#issuecomment-1440854834
issue_url: https://api.github.com/repos/simonw/datasette/issues/2030
node_id: IC_kwDOBm6k_c5V4bMy
created_at: 2023-02-22T21:54:39Z
updated_at: 2023-02-22T21:54:39Z
author_association: NONE
body:
  Thanks @dmick  . I chose to create a firewall rule under my GCP to open the port of interest and datasette works. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1444474487
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/433
node_id: IC_kwDOCGYnMM5WGO53
created_at: 2023-02-24T20:57:43Z
updated_at: 2023-02-24T22:22:18Z
author_association: CONTRIBUTOR
body:
  I think I see what is happening here, although I haven't quite work out a fix yet. Usually:

  * `click.progressbar.render_progress()` renders the cursor invisible on each invocation (update of the bar)
  * When the progress bar goes out of scope, the `__exit()__` method is invoked, which calls `render_finish()` to make the cursor re-appear.

  (See terminal escape sequences `BEFORE_BAR` and `AFTER_BAR` in click).

  However the sqlite-utils `utils.file_progress` context manager wraps `click.progressbar` and yields an instance of a helper class:

  ``` python
  @contextlib.contextmanager     
  def file_progress(file, silent=False, **kwargs):
     ...
          with click.progressbar(length=file_length, **kwargs) as bar:
              yield UpdateWrapper(file, bar.update) 
  ```

  The yielded `UpdateWrapper` goes out of scope quickly and `click.progressbar.__exit__()` is called. The cursor is made un-invisible. Hoewever `bar` is still live and so when the caller iterates on the yielded wrapper this invokes the bar's update method, calling `render_progress()`, each time printing the "make cursor invisible" escape code. The `progressbar.__exit__` function is not called again, so the cursor doesn't re-appear.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1619#issuecomment-1455196849
issue_url: https://api.github.com/repos/simonw/datasette/issues/1619
node_id: IC_kwDOBm6k_c5WvIqx
created_at: 2023-03-05T20:29:55Z
updated_at: 2023-03-05T20:30:14Z
author_association: NONE
body:
  I have this same issue, which is happening with both json links and facets. It is not happening with column sort links in the gear popup menus, but it is happening with the sort arrow that results after you use one of those links. I'm using Apache as a proxy to Datasette; the relevant configs are:

  ```
    ProxyPass /datasette/ http://127.0.0.1:8000/datasette/ nocanon
    ProxyPreserveHost on
  ```

  ```
  {
  	"base_url": "/datasette/"
  }
  ```

  If it would be useful to get a look at the running installation via the Web, Simon, let me know.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2028#issuecomment-1456925875
issue_url: https://api.github.com/repos/simonw/datasette/issues/2028
node_id: IC_kwDOBm6k_c5W1uyz
created_at: 2023-03-06T20:26:53Z
updated_at: 2023-03-06T20:26:53Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/2028?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`0b4a286`)](https://codecov.io/gh/simonw/datasette/commit/0b4a28691468b5c758df74fa1d72a823813c96bf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11% compared to head [(`a8dde13`)](https://codecov.io/gh/simonw/datasette/pull/2028?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2028   +/-   ##
  =======================================
    Coverage   92.11%   92.11%           
  =======================================
    Files          38       38           
    Lines        5555     5555           
  =======================================
    Hits         5117     5117           
    Misses        438      438           
  ```



  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/2028?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2031#issuecomment-1456997425
issue_url: https://api.github.com/repos/simonw/datasette/issues/2031
node_id: IC_kwDOBm6k_c5W2AQx
created_at: 2023-03-06T21:04:27Z
updated_at: 2023-03-06T21:06:34Z
author_association: OWNER
body:
  This is a very neat fix, for something I've been wanting for a while.

  Add a unit test for the row HTML page - I suggest against this page: https://latest.datasette.io/fixtures/foreign_key_references/1 - and I'll land this PR.

  You can model it on this test here: https://github.com/simonw/datasette/blob/a53b893c46453f35decc8c145c138671cee6140c/tests/test_table_html.py#L609-L632

  I think adding it to `test_table_html.py` is OK, even though it's technically for the row page and not the table page.

  Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2031#issuecomment-1457016788
issue_url: https://api.github.com/repos/simonw/datasette/issues/2031
node_id: IC_kwDOBm6k_c5W2E_U
created_at: 2023-03-06T21:18:18Z
updated_at: 2023-03-24T18:29:49Z
author_association: NONE
body:
  ## [Codecov](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`97.72`**% and project coverage change: **`-0.36`** :warning:
  > Comparison is base [(`3feed1f`)](https://codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`c8a2904`)](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11%.

  > :exclamation: Current head c8a2904 differs from pull request most recent head ef25867. Consider uploading reports for the commit ef25867 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2031      +/-   ##
  ==========================================
  - Coverage   92.46%   92.11%   -0.36%     
  ==========================================
    Files          38       38              
    Lines        5750     5556     -194     
  ==========================================
  - Hits         5317     5118     -199     
  - Misses        433      438       +5     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.57% <97.56%> (-3.35%)` | :arrow_down: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.17% <100.00%> (+2.38%)` | :arrow_up: |
  | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.93% <100.00%> (+0.10%)` | :arrow_up: |

  ... and [8 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/2031/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2031#issuecomment-1457243738
issue_url: https://api.github.com/repos/simonw/datasette/issues/2031
node_id: IC_kwDOBm6k_c5W28Za
created_at: 2023-03-07T00:05:25Z
updated_at: 2023-03-07T00:12:09Z
author_association: NONE
body:
  I've implemented the test (thanks for pointing me in the right direction!).

  At [tmcl-it/datasette:0.64.1+row-view-expand-labels](https://github.com/tmcl-it/datasette/tree/0.64.1%2Brow-view-expand-labels) I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2027#issuecomment-1459455356
issue_url: https://api.github.com/repos/simonw/datasette/issues/2027
node_id: IC_kwDOBm6k_c5W_YV8
created_at: 2023-03-08T04:42:22Z
updated_at: 2023-03-08T04:42:22Z
author_association: NONE
body:
  I managed to make it work by using nginx's 'exact match' (=) combined with 'prefix match'; that is, match explicitly on `/`, and redirect to `/<db>/<table>`, and then have the normal ProxyPath for the unadorned (prefix-matching) `/`.

  ```
          location = / {
                  return 302 /<db>/<table>;
          }
          location / {
                  proxy_pass http://127.0.0.1:8001/;
                  proxy_set_header Host $host;
          }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460618433
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD0TB
created_at: 2023-03-08T18:06:34Z
updated_at: 2023-03-08T18:06:34Z
author_association: OWNER
body:
  One way to do this would be to dynamically generate the `where id in (?, ?, ?)` with the correct number of question marks, then feed in a list from `request.args.getlist("id")` - but that would require rewriting the SQL query text to add those question marks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460621871
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD1Iv
created_at: 2023-03-08T18:08:25Z
updated_at: 2023-03-08T18:09:04Z
author_association: OWNER
body:
  My current preferred solution is to lean into SQLite's JSON support.

  What if the query page spotted `?id=11&id=32&id=62` and turned that into  a JSON string called `:id:` with a value of `["11", "32", "62"]`?

  Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where `?id[]=1&id[]=2` would result in an actual PHP array object, which often broke underlying code that had expected `$_GET["id"]` to be a string, not an array.

  So in a query you'd be able to do this:

      where id in (select value from json_each(:id))

  And then call it with `?id=11&id=32&id=62`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460628199
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD2rn
created_at: 2023-03-08T18:11:31Z
updated_at: 2023-03-08T18:11:31Z
author_association: OWNER
body:
  One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first `?id=` parameter version)? Maybe with syntax like this:

      where id in (select value from json_each(:id__list))

  Datasette would automatically pass `{"id": "11", "id__list": '["11", "32", "62"]'}` as arguments to the `db.execute()` method, if the page was called with `?id=11&id=32&id=62`.

  This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about `?id__list=` conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block `?id__list=` style parameters entirely.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460632758
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD3y2
created_at: 2023-03-08T18:13:49Z
updated_at: 2023-03-08T18:13:49Z
author_association: OWNER
body:
  https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the `<select multiple>` default HTML widget produces this exact format of query string:

  ```html
  <form action="https://www.example.com/">
  <select multiple name="id">
    <option>21</option>
    <option>32</option>
    <option>15</option>
    <option>63</option>
  </select>
  <input type="submit">
  </form>
  ```
  Submitting that form with the middle two options selected navigates to: `https://www.example.com/?id=32&id=15`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460637906
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD5DS
created_at: 2023-03-08T18:16:31Z
updated_at: 2023-03-08T18:16:31Z
author_association: OWNER
body:
  I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:

  1. `?id=1&?id=2` results in `:id` in the query being `["1", "2"]` - no additional syntax required
  2. `:id` in the query continues to reference just the first of those parameters - but `:id__list` (or some other custom syntax) instead gets `["1", "2"]` - or, if the URL is `?id=1` - gets `["1"]`

  Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, `?id=1`, resulting in their query breaking.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460639749
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD5gF
created_at: 2023-03-08T18:17:31Z
updated_at: 2023-03-08T18:17:31Z
author_association: OWNER
body:
  Since we are pre-1.0 it's still OK to implement a feature that disallows `?id__list=` in the URL, but allows `:id__list` in SQL queries to reference the JSON list of parameters.

  So I'm going to prototype this as the `:id__list` feature and see how it feels.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460654136
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD9A4
created_at: 2023-03-08T18:25:46Z
updated_at: 2023-03-08T18:25:46Z
author_association: OWNER
body:
  Trickiest part of the implementation here is that it needs to know to output three `id` HTML form fields on the page, such that their values are persisted when the form is submitted a second time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460659382
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD-S2
created_at: 2023-03-08T18:28:00Z
updated_at: 2023-03-08T18:28:00Z
author_association: OWNER
body:
  Also: `datasette-explain` may need to be updated to understand how to handle this:

  `ERROR: conn=<sqlite3.Connection object at 0x102834940>, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460664619
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XD_kr
created_at: 2023-03-08T18:32:29Z
updated_at: 2023-03-08T18:32:29Z
author_association: OWNER
body:
  Got a prototype working:
  ```diff
  diff --git a/datasette/views/database.py b/datasette/views/database.py
  index 8d289105..6f9d8a44 100644
  --- a/datasette/views/database.py
  +++ b/datasette/views/database.py
  @@ -226,6 +226,12 @@ class QueryView(DataView):
       ):
           db = await self.ds.resolve_database(request)
           database = db.name
  +        # Disallow x__list query string parameters
  +        invalid_params = [k for k in request.args if k.endswith("__list")]
  +        if invalid_params:
  +            raise DatasetteError(
  +                "Invalid query string parameters: {}".format(", ".join(invalid_params))
  +            )
           params = {key: request.args.get(key) for key in request.args}
           if "sql" in params:
               params.pop("sql")
  @@ -258,6 +264,11 @@ class QueryView(DataView):
               for named_parameter in named_parameters
               if not named_parameter.startswith("_")
           }
  +        # Handle any __list parameters
  +        for named_parameter in named_parameters:
  +            if named_parameter.endswith("__list"):
  +                list_values = request.args.getlist(named_parameter[:-6])
  +                params[named_parameter] = json.dumps(list_values)
   
           # Set to blank string if missing from params
           for named_parameter in named_parameters:
  ```
  This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the `?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL:

  <img width="1226" alt="image" src="https://user-images.githubusercontent.com/9599/223803758-2a1f90bd-bc20-4a77-bdce-4644afe235ba.png">

  That was this URL: `http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460668431
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XEAgP
created_at: 2023-03-08T18:35:34Z
updated_at: 2023-03-08T18:35:34Z
author_association: OWNER
body:
  To implement this properly need to do the following:
  - Get the page to display multiple `id: [ text input here ]` fields such that re-submission works
  - Figure out how this should work for canned queries and for writable canned queries
  - Tests that cover queries, canned queries, writable canned queries

  And a bonus feature: what if the Datasette UI layer spotted `:id__list` parameters and used them to add a bit of JavaScript that allowed users to click a `+` button next to an `id` form field to add another one?

  Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value.

  Though this has an annoying problem: how to tell the difference between an additional `id` input field that the user chose not to populate, v.s. one that is supposed to represent an empty string?

  Maybe only support multiple `id` fields for users with JavaScript in order to avoid this problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460679434
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XEDMK
created_at: 2023-03-08T18:39:35Z
updated_at: 2023-03-08T18:39:35Z
author_association: OWNER
body:
  I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters

  - `_actor_*`
  - `_header_*`
  - `_cookie_`
  - `_now_epoch`
  - `_now_date_utc`
  - `_now_datetime_utc`
  - `_random_chars_*`

  Should this new `id__list` syntax look more like those magic parameters, or is it OK to use `name__magic` syntax here instead?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460682625
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XED-B
created_at: 2023-03-08T18:40:57Z
updated_at: 2023-03-08T18:40:57Z
author_association: OWNER
body:
  Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460759358
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XEWs-
created_at: 2023-03-08T19:48:13Z
updated_at: 2023-03-20T18:47:12Z
author_association: OWNER
body:
  Breaking this down into smaller steps:

  - [x] Get `?_next=` working
  - [x] Implement extensions - so `.json` is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML
  - [ ] That HTML view should only access JSON data, which can be seen by using `.context` - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context.
  - [ ] Get a bunch of the existing table HTML and JSON tests to pass
  - [ ] Use those tests to refactor the nasty `_next` code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469
  - [ ] Figure out how the [register_output_renderer(datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette) plugin hook should work

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460760116
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XEW40
created_at: 2023-03-08T19:48:52Z
updated_at: 2023-03-08T19:48:52Z
author_association: OWNER
body:
  I'm trying to get `http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v` to return the correct results.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2035#issuecomment-1460808028
issue_url: https://api.github.com/repos/simonw/datasette/issues/2035
node_id: IC_kwDOBm6k_c5XEilc
created_at: 2023-03-08T20:14:47Z
updated_at: 2023-03-08T20:14:47Z
author_association: NONE
body:
  +1, I have been wishing for this feature (also for use with template-sql). It was requested before here #1304.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2036#issuecomment-1460809643
issue_url: https://api.github.com/repos/simonw/datasette/issues/2036
node_id: IC_kwDOBm6k_c5XEi-r
created_at: 2023-03-08T20:16:10Z
updated_at: 2023-03-08T20:16:10Z
author_association: OWNER
body:
  I think the code at fault is here:

  https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182

  That name ends up defaulting to `datasette` - so multiple different projects may end up deploying to the same `image_id`.

  What I think happened in the `datasette.io` bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time.

  And so the image that was pushed to `gcr.io/datasette-222320/datasette:latest` by the `simonw/simonwillisonblog-backup` action was then deployed by the `simonw/datasette.io/` action, which broke the site.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2036#issuecomment-1460810523
issue_url: https://api.github.com/repos/simonw/datasette/issues/2036
node_id: IC_kwDOBm6k_c5XEjMb
created_at: 2023-03-08T20:17:01Z
updated_at: 2023-03-08T20:17:01Z
author_association: OWNER
body:
  I'm going to solve this by using the service name in that `image_id` instead:

  ```python
  image_id = f"gcr.io/{project}/{service_name}"
  ```
  This is a nasty bug, so I'm going to backport it to a `0.64.2` release as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2036#issuecomment-1460816528
issue_url: https://api.github.com/repos/simonw/datasette/issues/2036
node_id: IC_kwDOBm6k_c5XEkqQ
created_at: 2023-03-08T20:22:50Z
updated_at: 2023-03-08T20:23:20Z
author_association: OWNER
body:
  Testing this manually:

  ```
  % datasette publish cloudrun content.db --service new-service
  Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression.
  Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz]
  Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d].
  Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ].
  ---- REMOTE BUILD OUTPUT ----
  starting build "290f41a4-e29a-443c-a1e5-c54513c6143d"

  FETCHSOURCE
  Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483
  Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483...
  / [1 files][  3.9 MiB/  3.9 MiB]                                                
  Operation completed over 1 objects/3.9 MiB.
  BUILD
  Already have image (with digest): gcr.io/cloud-builders/docker
  Sending build context to Docker daemon  14.52MB
  Step 1/9 : FROM python:3.11.0-slim-bullseye
  ...
  Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette
  Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

  [notice] A new release of pip available: 22.3 -> 23.0.1
  [notice] To update, run: pip install --upgrade pip
  Removing intermediate container 8ccebfebebc9
   ---> b972c85b38bb
  ...
  Successfully built 606b7c286d7f
  Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest
  PUSH
  Pushing gcr.io/datasette-222320/datasette-new-service
  The push refers to repository [gcr.io/datasette-222320/datasette-new-service]
  667b1dc69e5e: Preparing
  ...
  d8ddfcff216f: Pushed
  latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004
  DONE
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  ID                                    CREATE_TIME                DURATION  SOURCE                                                                                          IMAGES                                                   STATUS
  290f41a4-e29a-443c-a1e5-c54513c6143d  2023-03-08T20:21:03+00:00  39S       gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz  gcr.io/datasette-222320/datasette-new-service (+1 more)  SUCCESS
  Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1]
  ✓ Deploying new service... Done.                                                                                                                                                      
    ✓ Creating Revision...                                                                                                                                                              
    ✓ Routing traffic...                                                                                                                                                                
    ✓ Setting IAM Policy...                                                                                                                                                             
  Done.                                                                                                                                                                                 
  Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic.
  Service URL: https://new-service-j7hipcg4aq-uc.a.run.app
  ```
  https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2036#issuecomment-1460827178
issue_url: https://api.github.com/repos/simonw/datasette/issues/2036
node_id: IC_kwDOBm6k_c5XEnQq
created_at: 2023-03-08T20:25:10Z
updated_at: 2023-03-08T20:25:10Z
author_association: OWNER
body:
  https://console.cloud.google.com/run/detail/us-central1/new-service/revisions?project=datasette-222320 confirms that the image deployed is:

  <img width="556" alt="image" src="https://user-images.githubusercontent.com/9599/223841493-a104df28-5c9b-47d1-9c56-e7c7f6c61b00.png">

  Compared to https://console.cloud.google.com/run/detail/us-central1/datasette-io/revisions?project=datasette-222320 which shows that `datasette.io` is running:

  <img width="539" alt="image" src="https://user-images.githubusercontent.com/9599/223841616-67a1bb99-b704-4c2c-9dcd-247fb8592766.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2037#issuecomment-1460838797
issue_url: https://api.github.com/repos/simonw/datasette/issues/2037
node_id: IC_kwDOBm6k_c5XEqGN
created_at: 2023-03-08T20:31:15Z
updated_at: 2023-03-08T20:31:15Z
author_association: OWNER
body:
  It's this test here:

  https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/tests/test_cli.py#L181-L189

  Added in:
  - #2033 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2037#issuecomment-1460840620
issue_url: https://api.github.com/repos/simonw/datasette/issues/2037
node_id: IC_kwDOBm6k_c5XEqis
created_at: 2023-03-08T20:33:00Z
updated_at: 2023-03-08T20:33:00Z
author_association: OWNER
body:
  Got the same failure again for a recent commit: https://github.com/simonw/datasette/actions/runs/4368239376/jobs/7640567282

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2036#issuecomment-1460848869
issue_url: https://api.github.com/repos/simonw/datasette/issues/2036
node_id: IC_kwDOBm6k_c5XEsjl
created_at: 2023-03-08T20:40:55Z
updated_at: 2023-03-08T20:40:55Z
author_association: OWNER
body:
  Here's the https://latest.datasette.io/ deployment that just went out, further demonstrating that this change is working correctly:

  <img width="565" alt="image" src="https://user-images.githubusercontent.com/9599/223844709-bc01813f-b4da-4bb9-bdb8-7be839700a58.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XE6X9
created_at: 2023-03-08T21:32:56Z
updated_at: 2023-03-08T21:32:56Z
author_association: OWNER
body:
  I'd really like to extract this ugly logic out into a helper function:

  https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684

  I copied it in and asked ChatGPT to "List all of the variable that are either modified or assigned to by the above code":

  > Modified variables:
  > 
  >     offset
  >     where_clauses
  >     params
  >     order_by
  > 
  > Assigned variables:
  > 
  >     sort_value
  >     next_by_pk_clauses
  >     order_by (when sort or sort_desc is not None)

  Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:

  > My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:
  > 
  >     _next
  >     is_view
  >     sort
  >     sort_desc
  >     use_rowid
  >     pks
  >     order_by

  Note that `use_rowid` is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460906741
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XE6r1
created_at: 2023-03-08T21:34:08Z
updated_at: 2023-03-08T21:34:08Z
author_association: OWNER
body:
  So maybe I can refactor it to look a bit more like this:

  https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604

  One thing that's useful here is that `is_view` is handled early, like this:

  https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472

  So if I omit the `is_view` bit from the extracted function I can simplify more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460907148
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XE6yM
created_at: 2023-03-08T21:34:30Z
updated_at: 2023-03-08T21:34:30Z
author_association: OWNER
body:
  I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460916405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XE9C1
created_at: 2023-03-08T21:43:27Z
updated_at: 2023-03-08T21:43:27Z
author_association: OWNER
body:
  Just noticed that `_json=colname` is not working, and that's because it's handled by the renderer here:

  https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40

  But that's not currently being called by my new code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460943097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFDj5
created_at: 2023-03-08T22:09:24Z
updated_at: 2023-03-08T22:09:47Z
author_association: OWNER
body:
  The ease with which I added that  `?_extra=query` feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.

  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 8d3bb2c930..3e1db9c85f 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -1913,6 +1913,13 @@ async def extra_request():
               "args": request.args._data,
           }
   
  +    async def extra_query():
  +        "Details of the underlying SQL query"
  +        return {
  +            "sql": sql,
  +            "params": params,
  +        }
  +
       async def extra_extras():
           "Available ?_extra= blocks"
           return {
  @@ -1938,6 +1945,7 @@ async def extra_extras():
           extra_primary_keys,
           extra_debug,
           extra_request,
  +        extra_query,
           extra_extras,
       )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460970807
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFKU3
created_at: 2023-03-08T22:31:49Z
updated_at: 2023-03-08T22:33:03Z
author_association: OWNER
body:
  For the HTML version, I need to decide where all of the stuff that happens in `async def extra_template()` is going to live.

  I think it's another one of those extra functions, triggered for `?_extra=context`.

  https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460986533
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFOKl
created_at: 2023-03-08T22:40:28Z
updated_at: 2023-03-08T22:40:28Z
author_association: OWNER
body:
  Figuring out what to do with `display_columns_and_rows()` is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates:

  https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1460988975
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFOwv
created_at: 2023-03-08T22:42:57Z
updated_at: 2023-03-08T22:42:57Z
author_association: OWNER
body:
  Aside idea: it might be interesting if there were "lazy" template variables available in the context: things that are not actually executed unless a template author requests them.

  Imagine if `metadata` was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461002039
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFR83
created_at: 2023-03-08T22:58:16Z
updated_at: 2023-03-08T23:02:09Z
author_association: OWNER
body:
  The reason for that `Row` thing is that it allows custom templates that do things like this:

  https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows

  ```html+jinja
  {% for row in display_rows %}
      <div>
          <h2>{{ row["title"] }}</h2>
          <p>{{ row["description"] }}<lp>
          <p>Category: {{ row.display("category_id") }}</p>
      </div>
  {% endfor %}
  ```
  Is that a good design? the `.display()` thing feels weird - I wonder if anyone has ever actually used that.

  It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates

  > If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use `{{ row.display("column_name") }}`.

  I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display

  It is however useful to have some kind of abstraction layer here that insulates the SQLite `Row` object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461023559
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFXNH
created_at: 2023-03-08T23:23:02Z
updated_at: 2023-03-08T23:23:02Z
author_association: OWNER
body:
  To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461044477
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFcT9
created_at: 2023-03-08T23:47:26Z
updated_at: 2023-03-08T23:47:26Z
author_association: OWNER
body:
  I want to package together all of the extras that are needed for the HTML format. A few options for doing that:

  - Introduce `?_extra=_html` where the leading underscore indicates that this is a "bundle" of extras, then define a bundle that's everything needed for the HTML renderer
  - Have some other mechanism whereby different renderers can request a bundle of extras.

  I'm leaning towards the first option. I'll try that and see what it looks like.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461047607
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFdE3
created_at: 2023-03-08T23:51:46Z
updated_at: 2023-03-08T23:51:46Z
author_association: OWNER
body:
  This feels quite nice:

  <img width="906" alt="image" src="https://user-images.githubusercontent.com/9599/223879235-b5ac71d9-213c-48ff-b17f-077db5b16e90.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461070937
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFixZ
created_at: 2023-03-09T00:18:52Z
updated_at: 2023-03-09T00:19:36Z
author_association: OWNER
body:
  I managed to get HTML view working! I did it by continuing to add more things to the extras and the `_html` bundle until the page loaded for me:

  ```diff
       async def extra_extras():
           "Available ?_extra= blocks"
           return {
  @@ -1981,6 +2053,14 @@ async def extra_extras():
               "query",
               "display_columns",
               "display_rows",
  +            "database",
  +            "table",
  +            "database_color",
  +            "table_actions",
  +            "filters",
  +            "renderers",
  +            "custom_table_templates",
  +            "sorted_facet_results",
           ]
       }
   
  @@ -2006,6 +2086,14 @@ async def extra_extras():
           extra_query,
           extra_metadata,
           extra_extras,
  +        extra_database,
  +        extra_table,
  +        extra_database_color,
  +        extra_table_actions,
  +        extra_filters,
  +        extra_renderers,
  +        extra_custom_table_templates,
  +        extra_sorted_facet_results,
       )
  ```
  I'll probably refactor this into something cleaner, and maybe but a bunch of them in a `"html"` dictionary and update the templates to use `{{ html.filters }}` or similar. Will look at that once the tests are passing.

  <img width="1161" alt="image" src="https://user-images.githubusercontent.com/9599/223882676-942ff28b-1a4b-4ef0-adaa-ccd664877dcd.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461074526
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFjpe
created_at: 2023-03-09T00:23:06Z
updated_at: 2023-03-09T00:23:06Z
author_association: OWNER
body:
      pytest tests/test_table_html.py

  Currently 44 failed, 24 passed in 7.53s

  Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461075648
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XFj7A
created_at: 2023-03-09T00:24:22Z
updated_at: 2023-03-09T00:24:22Z
author_association: OWNER
body:
  `127.0.0.1:8001/fixtures/neighborhood_search` fails because the forwarding to a canned query does not yet work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461148254
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XF1pe
created_at: 2023-03-09T01:53:41Z
updated_at: 2023-03-09T01:53:41Z
author_association: OWNER
body:
  Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503

  The problem is that calling `.data()` on `QueryView` only works here because we expect to ourselves be inside a `.data()` method, with all of the existing magic that knows how to render things that are returned by that.

  So I may need to substantially re-engineer how `QueryView` works in order to get this to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461148579
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XF1uj
created_at: 2023-03-09T01:54:10Z
updated_at: 2023-03-09T01:55:33Z
author_association: OWNER
body:
  Or... I could temporarily build a quick additional `CannedQueryView` subclass that just does the necessary bits to get the existing code to work. I'm going to try that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1461161256
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XF40o
created_at: 2023-03-09T02:10:07Z
updated_at: 2023-03-09T02:10:07Z
author_association: OWNER
body:
  Just ran into a `no such table: columns` error - which I think is because my new view code sometimes bypasses calling this method (currently done in `BaseView.dispatch_request()`):

  https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461226402
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1
node_id: IC_kwDOJHON9s5XGIui
created_at: 2023-03-09T03:47:11Z
updated_at: 2023-03-09T03:47:11Z
author_association: MEMBER
body:
  Me:
  > Write applescript to loop through all of my Apple Notes and output their contents

  ChatGPT:

  ```applescript
  tell application "Notes"
     repeat with eachNote in every note
        set noteTitle to the name of eachNote
        set noteBody to the body of eachNote
        display dialog noteTitle & return & return & noteBody
     end repeat
  end tell
  ```
  Me:
  > How can I turn that into a zsh script which will output the results to my terminal

  ChatGPT:
  ```zsh
  #!/usr/bin/zsh

  osascript -e 'tell application "Notes"
     repeat with eachNote in every note
        set noteTitle to the name of eachNote
        set noteBody to the body of eachNote
        set output to noteTitle & "\n" & noteBody & "\n"
        display dialog output
        log output
     end repeat
  end tell'
  ```
  I had to change that first line to `#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.

  Me:

  > I want to log the notes contents to the terminal, not display it in a dialog

  ChatGPT:

  ```zsh
  #!/bin/zsh

  osascript -e 'tell application "Notes"
     repeat with eachNote in every note
        set noteTitle to the name of eachNote
        set noteBody to the body of eachNote
        set output to noteTitle & "\n" & noteBody & "\n"
        log output
     end repeat
  end tell'
  ```
  This really helps!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230197
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1
node_id: IC_kwDOJHON9s5XGJp1
created_at: 2023-03-09T03:51:36Z
updated_at: 2023-03-09T03:51:36Z
author_association: MEMBER
body:
  After a few more rounds I got to this script, which outputs them to a `/tmp/notes.txt` file:

  ```zsh
  #!/bin/zsh

  osascript -e '
  set notesFile to "/tmp/notes.txt"
  set fileRef to open for access notesFile with write permission
  tell application "Notes"
     repeat with eachNote in every note
        set noteId to the id of eachNote
        set noteTitle to the name of eachNote
        set noteBody to the body of eachNote
        write "------------------------" & "\n" to fileRef
        write noteId & "\n" to fileRef
        write noteTitle & "\n\n" to fileRef
        write noteBody & "\n" to fileRef
     end repeat
  end tell
  close access fileRef'
  ```
  Then I wrote this little Python script to load them into a database:
  ```python
  import sqlite_utils
  split = b"------------------------\n"
  s = open("/tmp/notes.txt", "rb").read()
  notes = [n.decode("mac_roman") for n in s.split(split) if n]

  cleaned_notes = [{
      "id": n.split("\n")[0],
      "title": n.split("\n")[1],
      "body": "\n".join(n.split("\n")[2:]).strip()
  } for n in notes]

  db = sqlite_utils.Database("/tmp/notes.db")
  db["notes"].insert_all(cleaned_notes)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461232709
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2
node_id: IC_kwDOJHON9s5XGKRF
created_at: 2023-03-09T03:54:28Z
updated_at: 2023-03-09T03:54:28Z
author_association: MEMBER
body:
  I think the AppleScript I want to pass to `osascript` looks like this:
  ```applescript
  tell application "Notes"
     repeat with eachNote in every note
        set noteId to the id of eachNote
        set noteTitle to the name of eachNote
        set noteBody to the body of eachNote
        log "------------------------" & "\n"
        log noteId & "\n"
        log noteTitle & "\n\n"
        log noteBody & "\n"
     end repeat
  end tell
  ```
  But there are a few more properties I'd like to get - created and updated date for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234311
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2
node_id: IC_kwDOJHON9s5XGKqH
created_at: 2023-03-09T03:56:24Z
updated_at: 2023-03-09T03:56:24Z
author_association: MEMBER
body:
  I opened the "Script Editor" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:

  <img width="1240" alt="image" src="https://user-images.githubusercontent.com/9599/223912876-09e8660e-f59a-4f04-8992-668e09c8d7d4.png">

  So the notes object has these properties:

  - name (text) : the name of the note (normally the first line of the body)
  - id (text, r/o) : the unique identifier of the note
  - container ([folder](applewebdata://621FA8D9-C995-4081-B3B3-149B0EA04C7F#Notes-Suite.folder), r/o) : the folder of the note
  - body (text) : the HTML content of the note
  - plaintext (text, r/o) : the plaintext content of the note
  - creation date (date, r/o) : the creation date of the note
  - modification date (date, r/o) : the modification date of the note
  - password protected (boolean, r/o) : Is the note password protected?
  - shared (boolean, r/o) : Is the note shared?

  I'm going to ignore the concept of attachments for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234591
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2
node_id: IC_kwDOJHON9s5XGKuf
created_at: 2023-03-09T03:56:45Z
updated_at: 2023-03-09T03:56:45Z
author_association: MEMBER
body:
  My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461259490
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2
node_id: IC_kwDOJHON9s5XGQzi
created_at: 2023-03-09T04:24:27Z
updated_at: 2023-03-09T04:24:27Z
author_association: MEMBER
body:
  Converting AppleScript date strings to ISO format is hard!

  https://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:

      set todayISO to (todayDate as «class isot» as string)

  Not clear to me how timezones work here. I'm going to ignore them for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461260978
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2
node_id: IC_kwDOJHON9s5XGRKy
created_at: 2023-03-09T04:27:18Z
updated_at: 2023-03-09T04:27:18Z
author_association: MEMBER
body:
  Before that conversion:

      Monday, March 6, 2023 at 11:55:15 AM

  After:

      2023-03-06T11:55:15

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462554175
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4
node_id: IC_kwDOJHON9s5XLM4_
created_at: 2023-03-09T18:19:34Z
updated_at: 2023-03-09T18:19:34Z
author_association: MEMBER
body:
  It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by `apple-notes-to-sqlite --dump`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462556829
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4
node_id: IC_kwDOJHON9s5XLNid
created_at: 2023-03-09T18:20:56Z
updated_at: 2023-03-09T18:20:56Z
author_association: MEMBER
body:
  In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.

  You can do a full import again ignoring that logic with `apple-notes-to-sqlite notes.db --full`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462562735
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7
node_id: IC_kwDOJHON9s5XLO-v
created_at: 2023-03-09T18:23:56Z
updated_at: 2023-03-09T18:25:22Z
author_association: MEMBER
body:
  From the Script Editor library docs:

  A note has a:

  > - `container` (folder), r/o) : the folder of the note

  Here's what a folder looks like:

  > folder n : a folder containing notes
  > elements:
  >
  > - contains folders, notes; contained by application, accounts, folders.
  >
  > properties:
  >
  > - `name` (text) : the name of the folder
  > - `id` (text, r/o) : the unique identifier of the folder
  > - `shared` (boolean, r/o) : Is the folder shared?
  > - `container` (account or folder, r/o) : the container of the folder

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462570187
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7
node_id: IC_kwDOJHON9s5XLQzL
created_at: 2023-03-09T18:30:24Z
updated_at: 2023-03-09T18:30:24Z
author_association: MEMBER
body:
  I used ChatGPT to write this:
  ```
  osascript -e 'tell application "Notes"
      set allFolders to folders
      repeat with aFolder in allFolders
          set folderId to id of aFolder
          set folderName to name of aFolder
          set folderContainer to container of aFolder
          set folderContainerName to name of folderContainer
          log "Folder ID: " & folderId
          log "Folder Name: " & folderName
          log "Folder Container: " & folderContainerName
          log " "
          --check for nested folders
          if count of folders of aFolder > 0 then
              set nestedFolders to folders of aFolder
              repeat with aNestedFolder in nestedFolders
                  set nestedFolderId to id of aNestedFolder
                  set nestedFolderName to name of aNestedFolder
                  set nestedFolderContainer to container of aNestedFolder
                  set nestedFolderContainerName to name of nestedFolderContainer
                  log "    Nested Folder ID: " & nestedFolderId
                  log "    Nested Folder Name: " & nestedFolderName
                  log "    Nested Folder Container: " & nestedFolderContainerName
                  log " "
              end repeat
          end if
      end repeat
  end tell
  '
  ```
  Which for my account output this:
  ```
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
  Folder Name: Blog posts
  Folder Container: iCloud
   
      Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
      Nested Folder Name: Nested inside blog posts
      Nested Folder Container: Blog posts
   
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698
  Folder Name: JSK
  Folder Container: iCloud
   
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
  Folder Name: Nested inside blog posts
  Folder Container: Blog posts
   
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526
  Folder Name: New Folder
  Folder Container: iCloud
   
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839
  Folder Name: New Folder 1
  Folder Container: iCloud
   
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2
  Folder Name: Notes
  Folder Container: iCloud
   
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059
  Folder Name: Quick Notes
  Folder Container: iCloud
   
  Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283
  Folder Name: UK Christmas 2022
  Folder Container: iCloud
  ```
  So I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.

  Then when I import notes I can foreign key reference them back to their containing folder.

  I'm tempted to use `rowid` for the foreign keys because the official IDs are pretty long.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462682795
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7
node_id: IC_kwDOJHON9s5XLsSr
created_at: 2023-03-09T19:52:20Z
updated_at: 2023-03-09T19:52:44Z
author_association: MEMBER
body:
  Created through several rounds with ChatGPT (including hints like "rewrite that using setdefault()"):
  ```python
  def topological_sort(nodes):
      children = {}
      for node in nodes:
          parent_id = node["parent"]
          if parent_id is not None:
              children.setdefault(parent_id, []).append(node)

      def traverse(node, result):
          result.append(node)
          if node["id"] in children:
              for child in children[node["id"]]:
                  traverse(child, result)

      sorted_data = []

      for node in nodes:
          if node["parent"] is None:
              traverse(node, sorted_data)

      return sorted_data
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462691466
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7
node_id: IC_kwDOJHON9s5XLuaK
created_at: 2023-03-09T19:59:52Z
updated_at: 2023-03-09T19:59:52Z
author_association: MEMBER
body:
  Improved script:
  ```zsh
  osascript -e 'tell application "Notes"
      set allFolders to folders
      repeat with aFolder in allFolders
          set folderId to id of aFolder
          set folderName to name of aFolder
          set folderContainer to container of aFolder
          if class of folderContainer is folder then
              set folderContainerId to id of folderContainer
          else
              set folderContainerId to ""
          end if
          log "ID: " & folderId
          log "Name: " & folderName
          log "Container: " & folderContainerId
          log " "
      end repeat
  end tell
  '
  ```
  ```
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
  Name: Blog posts
  Container: 
   
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698
  Name: JSK
  Container: 
   
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
  Name: Nested inside blog posts
  Container: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
   
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526
  Name: New Folder
  Container: 
   
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839
  Name: New Folder 1
  Container: 
   
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2
  Name: Notes
  Container: 
   
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059
  Name: Quick Notes
  Container: 
   
  ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283
  Name: UK Christmas 2022
  Container: 
  ```
  I filtered out things where the parent was an account and not a folder using `if class of folderContainer is folder then`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462693867
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7
node_id: IC_kwDOJHON9s5XLu_r
created_at: 2023-03-09T20:01:39Z
updated_at: 2023-03-09T20:02:11Z
author_association: MEMBER
body:
  My `folders` table will have:

  - `id` - rowid
  - `long_id` - that long unique string ID
  - `name` - the name
  - `parent` - foreign key to `id`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2031#issuecomment-1462921890
issue_url: https://api.github.com/repos/simonw/datasette/issues/2031
node_id: IC_kwDOBm6k_c5XMmqi
created_at: 2023-03-09T22:35:30Z
updated_at: 2023-03-09T22:35:30Z
author_association: OWNER
body:
  > I've implemented the test (thanks for pointing me in the right direction!).
  > 
  > At [tmcl-it/datasette:0.64.1+row-view-expand-labels](https://github.com/tmcl-it/datasette/tree/0.64.1%2Brow-view-expand-labels) I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

  Sure, let's merge that one too - it can go out in the next `0.64.x` series release (maybe even a 0.65).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462962682
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11
node_id: IC_kwDOJHON9s5XMwn6
created_at: 2023-03-09T23:20:35Z
updated_at: 2023-03-09T23:22:41Z
author_association: MEMBER
body:
  Here's a query that returns all notes in folder 1, including notes in descendant folders:
  ```sql
  with recursive nested_folders(folder_id, descendant_folder_id) as (
    -- base case: select all immediate children of the root folder
    select id, id from folders where parent is null
    union all
    -- recursive case: select all children of the previous level of nested folders
    select nf.folder_id, f.id from nested_folders nf
    join folders f on nf.descendant_folder_id = f.parent
  )
  -- Find notes within all descendants of folder 1
  select *
  from notes
  where folder in (
    select descendant_folder_id from nested_folders where folder_id = 1
  );
  ```
  With assistance from ChatGPT. Prompts were:

  ```
  SQLite schema:

  CREATE TABLE [folders] (
     [id] INTEGER PRIMARY KEY,
     [long_id] TEXT,
     [name] TEXT,
     [parent] INTEGER,
     FOREIGN KEY([parent]) REFERENCES [folders]([id])
  );

  Write a recursive CTE that returns the following:

  folder_id | descendant_folder_id

  With a row for every nested child of every folder - so the top level folder has lots of rows
  ```
  Then I tweaked it a bit, then ran this:
  ```
  WITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS (
    -- base case: select all immediate children of the root folder
    SELECT id, id FROM folders WHERE parent IS NULL
    UNION ALL
    -- recursive case: select all children of the previous level of nested folders
    SELECT nf.folder_id, f.id FROM nested_folders nf
    JOIN folders f ON nf.descendant_folder_id = f.parent
  )
  -- select all rows from the recursive CTE
  SELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)

  Convert all SQL keywords to lower case, and re-indent
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462965256
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11
node_id: IC_kwDOJHON9s5XMxQI
created_at: 2023-03-09T23:22:12Z
updated_at: 2023-03-09T23:22:12Z
author_association: MEMBER
body:
  Here's what the CTE from that looks like:

  <img width="469" alt="image" src="https://user-images.githubusercontent.com/9599/224182888-50f315eb-417f-4fb1-b624-151e1740ffed.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462968053
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11
node_id: IC_kwDOJHON9s5XMx71
created_at: 2023-03-09T23:24:01Z
updated_at: 2023-03-09T23:24:01Z
author_association: MEMBER
body:
  I improved the readability by removing some unnecessary table aliases:
  ```sql
  with recursive nested_folders(folder_id, descendant_folder_id) as (
    -- base case: select all immediate children of the root folder
    select id, id from folders where parent is null
    union all
    -- recursive case: select all children of the previous level of nested folders
    select nested_folders.folder_id, folders.id from nested_folders
    join folders on nested_folders.descendant_folder_id = folders.parent
  )
  -- Find notes within all descendants of folder 1
  select *
  from notes
  where folder in (
    select descendant_folder_id from nested_folders where folder_id = 1
  );
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1462997800
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XM5Mo
created_at: 2023-03-09T23:39:47Z
updated_at: 2023-03-09T23:39:47Z
author_association: OWNER
body:
  Found a neat trick:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 186f192d..40416713 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -23,7 +23,13 @@ from pathlib import Path
   
   from markupsafe import Markup, escape
   from itsdangerous import URLSafeSerializer
  -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
  +from jinja2 import (
  +    ChoiceLoader,
  +    Environment,
  +    FileSystemLoader,
  +    PrefixLoader,
  +    StrictUndefined,
  +)
   from jinja2.environment import Template
   from jinja2.exceptions import TemplateNotFound
   
  @@ -394,7 +400,10 @@ class Datasette:
               ]
           )
           self.jinja_env = Environment(
  -            loader=template_loader, autoescape=True, enable_async=True
  +            loader=template_loader,
  +            autoescape=True,
  +            enable_async=True,
  +            undefined=StrictUndefined,
           )
           self.jinja_env.filters["escape_css_string"] = escape_css_string
           self.jinja_env.filters["quote_plus"] = urllib.parse.quote_plus
  ```
  This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context.

  It's helping me spot things that are still missing, rather than just relying on failed unit tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1463005744
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XM7Iw
created_at: 2023-03-09T23:52:15Z
updated_at: 2023-03-09T23:52:23Z
author_association: OWNER
body:
  I need to figure out what to do about `extra_context_from_filters` - which was previously passed straight to the HTML context.

  https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1463022397
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XM_M9
created_at: 2023-03-10T00:14:21Z
updated_at: 2023-03-10T00:14:21Z
author_association: OWNER
body:
  Ironically the thing I most need right now is comprehensive documentation of what variables are being passed to the templates!

  One big challenge is that I need to untangle the template context that happens in `BaseView` - I'm hacking that together at the moment, but I need a real answer for how that should work in a world in which view functions aren't using that base class.

  https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L145

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1463023674
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XM_g6
created_at: 2023-03-10T00:16:03Z
updated_at: 2023-03-10T00:16:03Z
author_association: OWNER
body:
  I also need to figure out the `renderers` stuff, so I can link to the right URLs for CSV and JSON and other formats:

  https://github.com/simonw/datasette/blob/6d07a7da1531cd749844fc6827d9a1e57009b2ea/datasette/views/base.py#L474-L518

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1463024951
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XM_03
created_at: 2023-03-10T00:17:58Z
updated_at: 2023-03-10T00:17:58Z
author_association: OWNER
body:
  Renderers have an impact on three different pages: query results, table page and row page.

  The row page feature is incomplete though:

  https://congress-legislators.datasettes.com/legislators/social_media/A000055

  <img width="668" alt="image" src="https://user-images.githubusercontent.com/9599/224190630-b5e07524-94a7-4d8c-98b5-2364f3b0495d.png">

  Why is there no `.csv` link there?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2038#issuecomment-1463110978
issue_url: https://api.github.com/repos/simonw/datasette/issues/2038
node_id: IC_kwDOBm6k_c5XNU1C
created_at: 2023-03-10T02:09:41Z
updated_at: 2023-03-10T02:09:41Z
author_association: OWNER
body:
  I'm torn on this. It's useful for me right now for refactoring, but I feel like it should be a permanent thing, not a setting - or it should default to on and people should turn it off, but who would ever do that?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2038#issuecomment-1463112173
issue_url: https://api.github.com/repos/simonw/datasette/issues/2038
node_id: IC_kwDOBm6k_c5XNVHt
created_at: 2023-03-10T02:11:06Z
updated_at: 2023-03-10T02:11:06Z
author_association: OWNER
body:
  Here's an example of something that would break:

  https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/templates/database.html#L12-L15

  Because `metadata` is an empty dictionary sometimes, so `{{ metadata.title or database }}` would raise an error and need to be replaced by `{{ metadata.get("title") or database }}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1463113856
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5XNViA
created_at: 2023-03-10T02:13:15Z
updated_at: 2023-03-10T02:13:15Z
author_association: OWNER
body:
  Idea for if this change ends up making a bunch of breaking changes to the templates (which I think it should) - I can generate a GitHub diff link between the old and new templates and include that link in the 1.0 upgrade documentation to help people who wrote custom templates see what they might need to change - with minimal effort from myself to document those changes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464786643
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24
node_id: IC_kwDOC8tyDs5XTt7T
created_at: 2023-03-11T02:01:27Z
updated_at: 2023-03-11T02:01:27Z
author_association: NONE
body:
  Thanks for reporting this and providing a solution -- I was puzzled by this error when I revisited my walking data and experienced this issues. I haven't tried the fix yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464796494
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24
node_id: IC_kwDOC8tyDs5XTwVO
created_at: 2023-03-11T02:23:42Z
updated_at: 2023-03-11T02:23:42Z
author_association: NONE
body:
  @simonw - maybe put in some error handling to trap for poorly formed XML (from Apple engineers) so that it suggests that there are problems with export.zip rather than odd looking Python errors :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465038901
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/531
node_id: IC_kwDOCGYnMM5XUrg1
created_at: 2023-03-11T22:29:19Z
updated_at: 2023-04-07T18:19:49Z
author_association: NONE
body:
  ## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00`**% and project coverage change: **`+0.03`** :tada:
  > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`afdf618`)](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.29%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #531      +/-   ##
  ==========================================
  + Coverage   96.25%   96.29%   +0.03%     
  ==========================================
    Files           6        6              
    Lines        2671     2671              
  ==========================================
  + Hits         2571     2572       +1     
  + Misses        100       99       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | |
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.09%)` | :arrow_up: |

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-1465208436
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: IC_kwDOBm6k_c5XVU50
created_at: 2023-03-12T14:04:15Z
updated_at: 2023-03-12T14:04:15Z
author_association: NONE
body:
  I keep coming back to this in search for the related exploration, so I'll just link it now:

  @simonw has meanwhile researched _how to deploy Datasette to AWS Lambda using function URLs and Mangum_ via https://github.com/simonw/public-notes/issues/6 and concluded _that's everything I need to know in order to build a datasette-publish-lambda plugin_.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/531
node_id: IC_kwDOCGYnMM5XVr1H
created_at: 2023-03-12T21:19:13Z
updated_at: 2023-03-12T21:19:13Z
author_association: OWNER
body:
  Aah, I think I see why you wrote it like that.

  The problem is that `init_spatialite()` does other stuff too:

  https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171

  So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run `select InitSpatialMetadata()` to configure the database, if needed.

  So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465302936
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/533
node_id: IC_kwDOCGYnMM5XVr-Y
created_at: 2023-03-12T21:22:09Z
updated_at: 2023-03-12T21:22:09Z
author_association: OWNER
body:
  Could be the same problem as:
  - https://github.com/simonw/datasette/issues/1972

  Which I fixed in https://github.com/simonw/datasette/commit/3af313e165215696af899e772f47bf7c27873ae3

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465303378
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/533
node_id: IC_kwDOCGYnMM5XVsFS
created_at: 2023-03-12T21:24:04Z
updated_at: 2023-03-12T21:24:04Z
author_association: OWNER
body:
  Upgraded to Sphinx 6 locally and got the same error:

  ```
  % just docs
  Cogging README.md
  Cogging docs/changelog.rst
  Cogging docs/cli-reference.rst
  Cogging docs/cli.rst
  Cogging docs/contributing.rst
  Cogging docs/index.rst
  Cogging docs/installation.rst
  Cogging docs/python-api.rst
  Cogging docs/reference.rst
  sphinx-autobuild -a -b html "." "_build"   --watch ../sqlite_utils
  [sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build
  Running Sphinx v6.1.3
  loading pickled environment... failed
  failed: Can't get attribute '_stable_repr_object' on <sphinx.deprecation._ModuleWrapper object at 0x103f155d0>
  building [mo]: all of 0 po files
  writing output... 
  building [html]: all source files
  updating environment: [new config] 8 added, 0 changed, 0 removed
  reading sources... [ 12%] changelog                                                                                                          
  Exception occurred:
    File "/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py", line 103, in role
      title = caption % part
              ~~~~~~~~^~~~~~
  TypeError: not all arguments converted during string formatting
  The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/531
node_id: IC_kwDOCGYnMM5XVvGO
created_at: 2023-03-12T22:21:56Z
updated_at: 2023-03-12T22:21:56Z
author_association: CONTRIBUTOR
body:
  Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with `--load-extension`, I still hit an error on `create-spatial-index`.

  What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why `init_spatialite()` gets called again.

  Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch.

  I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/apple-notes-to-sqlite/issues/8#issuecomment-1468898285
issue_url: https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8
node_id: IC_kwDOJHON9s5XjZvt
created_at: 2023-03-14T22:00:21Z
updated_at: 2023-03-14T22:00:21Z
author_association: NONE
body:
  Well that's embarrassing. I made a fork using macnotesapp and it's actually slower. This is because the Scripting Bridge sometimes fails to return the folder and thus macnotesapp resorts to AppleScript in this situation. The repeated AppleScript calls on a large library are slower than your "slurp it all in" approach.  I've got some ideas about how to improve this--will make another attempt if I can fix the issues.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1474704790
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5X5jWW
created_at: 2023-03-18T04:52:59Z
updated_at: 2023-03-18T04:52:59Z
author_association: OWNER
body:
  Here are the next set of tests to get passing:
  ```
  % pytest tests/test_table_api.py
  ```
  ```
  FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id-expected_facet_results0] - KeyError: 'name'
  FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id&state=MI-expected_facet_results1] - KeyError: 'name'
  FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int-expected_facet_results2] - KeyError: 'name'
  FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int&planet_int=1-expected_facet_results3] - KeyError: 'name'
  FAILED tests/test_table_api.py::test_suggested_facets - KeyError: 'suggested_facets'
  FAILED tests/test_table_api.py::test_allow_facet_off - KeyError: 'suggested_facets'
  FAILED tests/test_table_api.py::test_suggest_facets_off - KeyError: 'suggested_facets'
  FAILED tests/test_table_api.py::test_nofacet[True] - KeyError: 'suggested_facets'
  FAILED tests/test_table_api.py::test_nofacet[False] - KeyError: 'suggested_facets'
  FAILED tests/test_table_api.py::test_nosuggest[True] - KeyError: 'suggested_facets'
  FAILED tests/test_table_api.py::test_nosuggest[False] - KeyError: 'suggested_facets'
  FAILED tests/test_table_api.py::test_nocount[True-None] - KeyError: 'count'
  FAILED tests/test_table_api.py::test_nocount[False-15] - KeyError: 'count'
  FAILED tests/test_table_api.py::test_expand_labels - AssertionError: assert {'13': {'_cit...:00:00', ...}} == {'13': {'_cit...:00:00', ...}}
  FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'forei...l': '1', ...}} == {'1': {'forei...': '1'}, ...}}
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1,...'value': 1.5}] == [{'rowid': 1,...'value': 1.5}]
  FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'foreign_ke...': None, ...}] == [{'foreign_ke...': None, ...}]
  FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'data': "b'..., 'rowid': 3}] == [{'data': {'$..., 'rowid': 3}]
  FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false...title": null}' == '{"rowid": 1,..."data": null}'
  FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns'
  ============================================================================= 38 failed, 73 passed, 1 xfailed in 7.25s =============================================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1475003292
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5X6sOc
created_at: 2023-03-18T21:46:20Z
updated_at: 2023-03-18T21:46:20Z
author_association: OWNER
body:
  Now 25 failures in `test_table_api.py`:
  ```
  FAILED tests/test_table_api.py::test_expand_labels - assert {'2': {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', '_city_id': 1, '_neighborhood': 'Dogpatch', 'tags': '["tag1", "tag3"]', 'complex_array'...
  FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_co...
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control'
  FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1, 'value': inf}, {'rowid': 2, 'value': -inf}, {'rowid': 3, 'value': 1.5}] == [{'rowid': 1, 'value': None}, {'rowid': 2, 'value': None}, {'rowid': 3, 'value'...
  FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_compoun...
  FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'rowid': 1, 'data': "b'\\x15\\x1c\\x02\\xc7\\xad\\x05\\xfe'"}, {'rowid': 2, 'data': "b'\\x15\\x1c\\x03\\xc7\\xad\\x05\\xfe'"}, {'rowid': 3, 'data': None}] == [{'rowid': 1, 'data': {'...
  FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid": 2, "data": {"$base64": true, "encoded": "FRwDx60F/g=="}}\n{"rowid": 3, "data": null}] - assert '{"ok": false, "error": "Object of type bytes is not JSON serializable", "status": 500, "title": null}' == '{"rowid": 1, "data": {"$base64": true, "encoded": "FRwCx60F/g=="}}\n{"rowid"...
  FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21
  FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns'
  FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns'
  ============================================================================= 25 failed, 86 passed, 1 xfailed in 7.18s =============================================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1475016834
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5X6viC
created_at: 2023-03-18T22:30:31Z
updated_at: 2023-03-18T22:30:31Z
author_association: OWNER
body:
  `test_paginate_using_link_header` will be tricky.

  The reason the tests are failing is that `json_renderer()` attempts to populate the `link` header using `data["next_url"]` - but that's not present unless `?_extra=next_url` has been passed:

  https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/renderer.py#L101-L102

  But I can only rely on `next` being present, not `next_url`.

  I thought I could maybe assemble the `link` header using `next`, by turning that into a `next_url` link - but there's some custom logic which kicks in for pagination against SQL views (offset/limit pagination) to calculate the `next_url` which isn't easily replicable at the `json_renderer()` layer, in the `_next_value_and_url()` utility function:

  https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/views/table.py#L2275-L2282

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1475074025
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5X69fp
created_at: 2023-03-19T02:14:28Z
updated_at: 2023-03-19T02:14:51Z
author_association: OWNER
body:
  I had to replicate quite a bit of this logic from `base.py`:

  https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L526-L544

  I should refactor this when I move the canned / arbitrary query views away from that base class too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1475074311
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5X69kH
created_at: 2023-03-19T02:16:08Z
updated_at: 2023-03-19T02:17:33Z
author_association: OWNER
body:
  Next big challenge: fix all the tests in `test_table_html.py`:

  ```
  FAILED tests/test_table_html.py::test_add_filter_redirects - assert 500 == 302
  FAILED tests/test_table_html.py::test_existing_filter_redirects - assert 500 == 302
  FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood-expected_hidden0] - AssertionError: assert {} == {'_facet': '_neighborhood'}
  FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_where=1+=+1&_col=_city_id-expected_hidden1] - AssertionError: assert {} == {'_col': '_ci...ere': '1 = 1'}
  FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_neighborhood__exact=Downtown-expected_hidden2] - AssertionError: assert {} == {'_facet': '_neighborhood'}
  FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_city_id__gt=1-expected_hidden3] - AssertionError: assert {} == {'_facet': '_neighborhood'}
  FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed - assert 500 == 302
  FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table - AssertionError: assert [('_fts_table...ts_pk', 'pk')] == []
  FAILED tests/test_table_html.py::test_sort_by_desc_redirects - assert 500 == 302
  FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...]
  FAILED tests/test_table_html.py::test_facets_persist_through_filter_form - AssertionError: assert [] == [('_facet', '...ray', 'tags')]
  FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field - AssertionError: assert [] == [('_size', '1')]
  FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AttributeError: 'NoneType' object has no attribute 'string'
  FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/s...x', '#export'] == ['/', '', '#export']
  FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/f...x', '#export'] == ['/', '', '#export']
  FAILED tests/test_table_html.py::test_table_html_no_primary_key - AssertionError: assert ['content', 'a', 'b', 'c'] == ['a', 'b', 'c']
  FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AttributeError: 'NoneType' object has no attribute 'string'
  FAILED tests/test_table_html.py::test_table_html_compound_primary_key - AssertionError: assert 'Link' == 'pk1'
  FAILED tests/test_table_html.py::test_table_html_foreign_key_links - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']]
  FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels - assert [['<td class=...str">b</td>']] == [['<td class=...str">b</td>']]
  FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...1</em></td>']]
  FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -'...wid', 'value'] == ['- column -']
  FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -'...', 'content2'] == ['- column -']
  FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -'...2', 'content'] == ['- column -']
  FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -'...nulls_2', ...]
  FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references - assert [['<td class=...ne</a></td>']] == [['<td class=...ne</a></td>']]
  FAILED tests/test_table_html.py::test_view_html - assert None is not None
  FAILED tests/test_table_html.py::test_extra_where_clauses - assert [('_where', "...'_city_id=1')] == []
  FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10-expected_hidden0] - AssertionError: assert [] == [('_size', '10')]
  FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10&_ignore=1&_ignore=2-expected_hidden1] - AssertionError: assert [] == [('_size', '1...ignore', '2')]
  FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated[/fixtures/searchable?_sort=text2&_where=1-expected_hidden3] - AssertionError: assert [] == [('_where', '1')]
  FAILED tests/test_table_html.py::test_binary_data_display_in_table - assert [['<td class=...">\xa0</td>']] == [['<td class=...">\xa0</td>']]
  FAILED tests/test_table_html.py::test_metadata_sort - AttributeError: 'NoneType' object has no attribute 'string'
  FAILED tests/test_table_html.py::test_metadata_sort_desc - AttributeError: 'NoneType' object has no attribute 'string'
  FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood-2-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 2
  FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood&_facet_size=50-5-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 5
  FAILED tests/test_table_html.py::test_facet_total - assert 500 == 200
  ```

  Deduped that's 30 tests:

  ```
  FAILED tests/test_table_html.py::test_add_filter_redirects
  FAILED tests/test_table_html.py::test_binary_data_display_in_table
  FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references
  FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys
  FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed
  FAILED tests/test_table_html.py::test_existing_filter_redirects
  FAILED tests/test_table_html.py::test_extra_where_clauses
  FAILED tests/test_table_html.py::test_facet_more_links
  FAILED tests/test_table_html.py::test_facet_total
  FAILED tests/test_table_html.py::test_facets_persist_through_filter_form
  FAILED tests/test_table_html.py::test_metadata_sort
  FAILED tests/test_table_html.py::test_metadata_sort_desc
  FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field
  FAILED tests/test_table_html.py::test_other_hidden_form_fields
  FAILED tests/test_table_html.py::test_reflected_hidden_form_fields
  FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key
  FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated
  FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table
  FAILED tests/test_table_html.py::test_sort_by_desc_redirects
  FAILED tests/test_table_html.py::test_sort_links
  FAILED tests/test_table_html.py::test_table_csv_json_export_interface
  FAILED tests/test_table_html.py::test_table_html_compound_primary_key
  FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels
  FAILED tests/test_table_html.py::test_table_html_filter_form_column_options
  FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns
  FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column
  FAILED tests/test_table_html.py::test_table_html_foreign_key_links
  FAILED tests/test_table_html.py::test_table_html_no_primary_key
  FAILED tests/test_table_html.py::test_table_html_simple_primary_key
  FAILED tests/test_table_html.py::test_view_html
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1476781401
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YBeVZ
created_at: 2023-03-20T19:01:35Z
updated_at: 2023-03-20T19:01:35Z
author_association: OWNER
body:
  Down to just these failures:
  ```
  FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'attrs': {'class': ['col-Link'], 'scope': 'col', 'data-column': 'Link', 'data-column-type': '', 'data-column-not-null': '0', 'data-is-pk': '0'}, 'a_href': None}, {'attrs': {'class': ['col-pk1'], 'scope': 'col', 'data-column': 'pk1', 'data-column-typ...
  FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AssertionError: assert 'id\xa0▼' == 'id'
  FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/simple_primary_key.json?id__gt=2', '/fixtures/simple_primary_key.testall?id__gt=2', '/fixtures/simple_primary_key.testnone?id__gt=2', '/fixtures/simple_primary_key.testresponse?id__gt=2', '/fixtures/simple_primary_key.csv?id__gt=2&_size=ma...
  FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/facetable.json?_labels=on', '/fixtures/facetable.testall?_labels=on', '/fixtures/facetable.testnone?_labels=on', '/fixtures/facetable.testresponse?_labels=on', '/fixtures/facetable.csv?_labels=on&_size=max', '#export'] == ['/', '', '#export']
  FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AssertionError: assert 'rowid\xa0▼' == 'rowid'
  FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -', 'rowid', 'value'] == ['- column -']
  FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -', 'id', 'content', 'content2'] == ['- column -']
  FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -', 'pk1', 'pk2', 'content'] == ['- column -']
  FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -', 'pk1', 'pk2', 'content', 'sortable_with_nulls', 'sortable_with_nulls_2', 'text', 'sortable']
  FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id', 'name']
  FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk', 'name']
  === 11 failed, 57 passed in 7.59s ===
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1476851525
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YBvdF
created_at: 2023-03-20T19:59:51Z
updated_at: 2023-03-20T20:01:40Z
author_association: OWNER
body:
  Three failures in `test_html.py`:
  ```
  FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: simple_primary_key: 5 rows</title>\n    <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary_key", "ad...
  FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: table/with/slashes.csv: 1 row</title>\n    <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fslashes~2Ec...
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link'
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link'
  ```

  In CI I'm getting this: https://github.com/simonw/datasette/actions/runs/4472331872/jobs/7858360154

      ====== 44 failed, 1222 passed, 1 skipped, 1 xfailed in 171.24s (0:02:51) =======

  Including failures in `tests/test_plugins.py` and `tests/test_routes.py`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1476854645
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YBwN1
created_at: 2023-03-20T20:02:24Z
updated_at: 2023-03-20T20:02:24Z
author_association: OWNER
body:
  ```
      def test_routes(routes, path, expected_class, expected_matches):
          match, view = resolve_routes(routes, path)
          if expected_class is None:
              assert match is None
          else:
  >           assert view.view_class.__name__ == expected_class
  E           AttributeError: 'function' object has no attribute 'view_class'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1476860334
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YBxmu
created_at: 2023-03-20T20:06:56Z
updated_at: 2023-03-20T22:09:03Z
author_association: OWNER
body:
  `pytest -n 8` locally says:

  ```
  FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name_specify_id-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\n});\n</...
  FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name-True] - assert '<input type="hidden" name="csrftoken" value="' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\n});\n</...
  FAILED tests/test_canned_queries.py::test_canned_query_permissions - AssertionError: assert 200 == 500
  FAILED tests/test_canned_queries.py::test_canned_query_with_named_parameter - KeyError: 'rows'
  FAILED tests/test_canned_queries.py::test_canned_write_custom_template - assert 500 == 200
  FAILED tests/test_canned_queries.py::test_canned_write_query_disabled_for_immutable_database - assert 500 == 200
  FAILED tests/test_canned_queries.py::test_custom_params - assert '<input type="text" id="qp3" name="extra" value="foo">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel="stylesheet" href="/-/static/app.css?d59...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\...
  FAILED tests/test_canned_queries.py::test_insert - KeyError: 'ds_csrftoken'
  FAILED tests/test_canned_queries.py::test_insert_error - KeyError: 'ds_csrftoken'
  FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[False-True] - KeyError: 'ds_csrftoken'
  FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[True-True] - KeyError: 'ds_csrftoken'
  FAILED tests/test_canned_queries.py::test_vary_header - KeyError: 'vary'
  FAILED tests/test_csv.py::test_csv_trace - AssertionError: assert 'application/...charset=utf-8' == 'text/html; charset=utf-8'
  FAILED tests/test_csv.py::test_table_csv - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
  FAILED tests/test_csv.py::test_table_csv_blob_columns - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
  FAILED tests/test_csv.py::test_table_csv_download - AssertionError: assert 'application/...charset=utf-8' == 'text/csv; charset=utf-8'
  FAILED tests/test_csv.py::test_table_csv_no_header - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
  FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101
  FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_counts - AttributeError: 'NoneType' object has no attribute 'text'
  FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_facets - AttributeError: 'NoneType' object has no attribute 'text'
  FAILED tests/test_csv.py::test_table_csv_with_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
  FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
  FAILED tests/test_facets.py::test_array_facet_handle_duplicate_tags - KeyError: 'facet_results'
  FAILED tests/test_facets.py::test_conflicting_facet_names_json - AssertionError: assert {'results', 'timed_out'} == {'created', '...gs', 'tags_2'}
  FAILED tests/test_facets.py::test_facet_size - KeyError: 'suggested_facets'
  FAILED tests/test_facets.py::test_json_array_with_blanks_and_nulls - KeyError: 'suggested_facets'
  FAILED tests/test_facets.py::test_other_types_of_facet_in_metadata - assert '<strong>created (date)\n' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: facetable: 15 rows</title>\n    <link rel="stylesheet" href="/-...name": "table", "request_path": "/fixtures/facetable", "added": 15, "columns": null};</script>\n\n\n\n</bo...
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link'
  FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link'
  FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: simple_primary_key: 5 rows</title>\n    <link rel="stylesheet" ...able", "request_path": "/fixtures/simple_primary...
  FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: table/with/slashes.csv: 1 row</title>\n    <link rel="styleshee...quest_path": "/fixtures/table~2Fwith~2Fs...
  FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...}
  FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...}
  FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets'
  FAILED tests/test_plugins.py::test_hook_register_magic_parameters - KeyError: 'ds_csrftoken'
  FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'next': None...', ...}, ...]} == {'1+1': 2, 'c... 0xXXX>', ...}
  FAILED tests/test_plugins.py::test_hook_register_output_renderer_no_parameters - assert b'Hello' == b'{"ok": true... "n": null}]}'
  FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_broken_value - assert 200 == 500
  FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_response - assert {'next': None...', ...}, ...]} == {'this_is': 'json'}
  === 39 failed, 1259 passed, 2 skipped, 1 xfailed in 58.07s ===
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1476898261
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YB63V
created_at: 2023-03-20T20:37:52Z
updated_at: 2023-03-20T20:37:52Z
author_association: OWNER
body:
  Manual testing spotted a bug.

  `/content/repos.json?owner=9599&_facet_array=topics` - does not return a `facet_results` key.

  `/content/repos.json?owner=9599&_facet_array=topics&_facet=owner` does.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1477082852
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YCn7k
created_at: 2023-03-20T23:27:25Z
updated_at: 2023-03-20T23:27:25Z
author_association: OWNER
body:
  Urgh getting CSV to work is going to be _so hard_, because the logic for that currently lives in a huge chunk of code in `BaseView` which depends on the old design of the `data()` method:

  https://github.com/simonw/datasette/blob/4bb49848697e40b8b9a1557be42b8e59eac965b3/datasette/views/base.py#L177-L343

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2041#issuecomment-1477144853
issue_url: https://api.github.com/repos/simonw/datasette/issues/2041
node_id: IC_kwDOBm6k_c5YC3EV
created_at: 2023-03-21T01:02:13Z
updated_at: 2023-03-21T01:02:13Z
author_association: OWNER
body:
  This code here:

  https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/table.py#L117-L169

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2041#issuecomment-1477155602
issue_url: https://api.github.com/repos/simonw/datasette/issues/2041
node_id: IC_kwDOBm6k_c5YC5sS
created_at: 2023-03-21T01:17:43Z
updated_at: 2023-03-21T01:17:43Z
author_association: OWNER
body:
  Removed code in https://github.com/simonw/datasette/commit/538ca9d2e2d41308f5b149c934e2bc43727fd77c - it will merge when I land:
  - #1999

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1480154453
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YOV1V
created_at: 2023-03-22T19:41:39Z
updated_at: 2023-03-22T19:43:04Z
author_association: OWNER
body:
  To replace this code: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L122

  Maybe `datasette.render_template()` should optionally accept a list of templates.

  https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none - turns out it does already:

  > If this is a list of template file names then the first one that exists will be loaded and rendered.

  It doesn't have an easy way to populate that `select_templates` debug template variable though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1480163485
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YOYCd
created_at: 2023-03-22T19:48:00Z
updated_at: 2023-03-22T19:48:00Z
author_association: OWNER
body:
  Getting close now! Only 13 failures left, mostly relating to CSV.
  ```
  FAILED tests/test_csv.py::test_table_csv - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101
  FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...}
  FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets'
  FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text'
  FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...}
  FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...}
  =============== 13 failed, 1287 passed, 2 skipped, 1 xfailed in 61.57s (0:01:01) ================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1480179217
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YOb4R
created_at: 2023-03-22T19:55:31Z
updated_at: 2023-03-22T21:34:02Z
author_association: OWNER
body:
  I rebased from `main`. Now:

  ```
  FAILED tests/test_csv.py::test_table_csv - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200
  FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101
  FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text'
  FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...}
  FAILED tests/test_plugins.py::test_hook_render_cell_demo - AttributeError: 'NoneType' object has no attribute 'string'
  FAILED tests/test_plugins.py::test_hook_render_cell_async[/fixtures/simple_primary_key] - assert b'RENDER_CELL_ASYNC_RESULT' in b'<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel="stylesheet" href="/-/static/app.css?d5...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\n});\n</script>\n\n\n\n\n</b...
  FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...}
  FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets'
  FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...}
  FAILED tests/test_permissions.py::test_permissions_checked[/fixtures/simple_primary_key-permissions3] - assert 500 in (200, 403)
  FAILED tests/test_table_html.py::test_table_csv_json_export_interface - assert 500 == 200
  FAILED tests/test_table_html.py::test_table_metadata - assert 500 == 200
  FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/simple_primary_key-expected_classes3] - assert 500 == 200
  FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert 500 == 200
  FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - AssertionError: assert [] == [{'href': '/'...simple_view'}]
  ERROR tests/test_internals_database.py::test_execute_write_fn_connection_exception - Failed: Timeout >1.0s
  ERROR tests/test_internals_database.py::test_mtime_ns - Failed: Timeout >1.0s
  ERROR tests/test_internals_database.py::test_mtime_ns_is_none_for_memory - Failed: Timeout >1.0s
  ERROR tests/test_internals_database.py::test_is_mutable - Failed: Timeout >1.0s
  ERROR tests/test_internals_database.py::test_database_memory_name - Failed: Timeout >1.0s
  ERROR tests/test_internals_database.py::test_in_memory_databases_forbid_writes - Failed: Timeout >1.0s
  =============== 21 failed, 1275 passed, 2 skipped, 1 xfailed, 6 errors in 59.18s ================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1480299765
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YO5T1
created_at: 2023-03-22T21:44:03Z
updated_at: 2023-03-22T21:45:10Z
author_association: OWNER
body:
  Oh this is a bit tricky.

  I have a failing test because a plugin that uses the `extra_css_urls` hook can't see the columns for the page.

  Turns out that bit comes from here:

  https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/app.py#L1203-L1217

  Which assumes the context has `"columns"` - but that's only now available if `?_extra=columns` was passed.

  Actually I think I can cheat here, since it's still getting the HTML context in order to render the template.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1480339527
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YPDBH
created_at: 2023-03-22T22:28:54Z
updated_at: 2023-03-22T22:28:54Z
author_association: OWNER
body:
  I hacked at the CSV stuff until it worked.

  I need to clean it up though, but I can do that in this separate task:
  - #1101

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1999#issuecomment-1480349156
issue_url: https://api.github.com/repos/simonw/datasette/issues/1999
node_id: IC_kwDOBm6k_c5YPFXk
created_at: 2023-03-22T22:40:58Z
updated_at: 2023-03-22T22:48:57Z
author_association: NONE
body:
  ## [Codecov](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`87.89`**% and project coverage change: **`-4.43`** :warning:
  > Comparison is base [(`56b0758`)](https://codecov.io/gh/simonw/datasette/commit/56b0758a5fbf85d01ff80a40c9b028469d7bb65f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.15% compared to head [(`921faae`)](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 87.73%.

  > :exclamation: Current head 921faae differs from pull request most recent head 69a31cd. Consider uploading reports for the commit 69a31cd to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #1999      +/-   ##
  ==========================================
  - Coverage   92.15%   87.73%   -4.43%     
  ==========================================
    Files          38       38              
    Lines        5560     6066     +506     
  ==========================================
  + Hits         5124     5322     +198     
  - Misses        436      744     +308     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.61% <ø> (ø)` | |
  | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.82% <ø> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `69.11% <86.76%> (-23.46%)` | :arrow_down: |
  | [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `93.33% <90.90%> (-0.87%)` | :arrow_down: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.78% <91.66%> (-2.39%)` | :arrow_down: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.48% <100.00%> (-0.01%)` | :arrow_down: |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `79.93% <100.00%> (ø)` | |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
  | [datasette/publish/cloudrun.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY2xvdWRydW4ucHk=) | `97.29% <100.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.59% <100.00%> (-0.27%)` | :arrow_down: |

  ... and [2 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/1999/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-1480355670
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: IC_kwDOBm6k_c5YPG9W
created_at: 2023-03-22T22:50:30Z
updated_at: 2023-03-22T22:50:30Z
author_association: OWNER
body:
  I just landed this PR so this feature is now in `main`:
  - #1999

  Still needs documentation and maybe some extra tests too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2043#issuecomment-1482859217
issue_url: https://api.github.com/repos/simonw/datasette/issues/2043
node_id: IC_kwDOBm6k_c5YYqLR
created_at: 2023-03-24T14:05:54Z
updated_at: 2023-03-24T14:05:54Z
author_association: NONE
body:
  ## [Codecov](https://codecov.io/gh/simonw/datasette/pull/2043?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage has no change and project coverage change: **`-0.06`** :warning:
  > Comparison is base [(`3feed1f`)](https://codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`85b9e51`)](https://codecov.io/gh/simonw/datasette/pull/2043?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2043      +/-   ##
  ==========================================
  - Coverage   92.46%   92.41%   -0.06%     
  ==========================================
    Files          38       38              
    Lines        5750     5750              
  ==========================================
  - Hits         5317     5314       -3     
  - Misses        433      436       +3     
  ```


  [see 2 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/2043/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  </details>

  [:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/2043?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


