html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/pull/1999#issuecomment-1461047607,https://api.github.com/repos/simonw/datasette/issues/1999,1461047607,IC_kwDOBm6k_c5XFdE3,9599,2023-03-08T23:51:46Z,2023-03-08T23:51:46Z,OWNER,"This feels quite nice:

<img width=""906"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223879235-b5ac71d9-213c-48ff-b17f-077db5b16e90.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1461044477,https://api.github.com/repos/simonw/datasette/issues/1999,1461044477,IC_kwDOBm6k_c5XFcT9,9599,2023-03-08T23:47:26Z,2023-03-08T23:47:26Z,OWNER,"I want to package together all of the extras that are needed for the HTML format. A few options for doing that:

- Introduce `?_extra=_html` where the leading underscore indicates that this is a ""bundle"" of extras, then define a bundle that's everything needed for the HTML renderer
- Have some other mechanism whereby different renderers can request a bundle of extras.

I'm leaning towards the first option. I'll try that and see what it looks like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1461023559,https://api.github.com/repos/simonw/datasette/issues/1999,1461023559,IC_kwDOBm6k_c5XFXNH,9599,2023-03-08T23:23:02Z,2023-03-08T23:23:02Z,OWNER,"To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1461002039,https://api.github.com/repos/simonw/datasette/issues/1999,1461002039,IC_kwDOBm6k_c5XFR83,9599,2023-03-08T22:58:16Z,2023-03-08T23:02:09Z,OWNER,"The reason for that `Row` thing is that it allows custom templates that do things like this:

https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows

```html+jinja
{% for row in display_rows %}
    <div>
        <h2>{{ row[""title""] }}</h2>
        <p>{{ row[""description""] }}<lp>
        <p>Category: {{ row.display(""category_id"") }}</p>
    </div>
{% endfor %}
```
Is that a good design? the `.display()` thing feels weird - I wonder if anyone has ever actually used that.

It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates

> If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use `{{ row.display(""column_name"") }}`.

I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display

It is however useful to have some kind of abstraction layer here that insulates the SQLite `Row` object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460988975,https://api.github.com/repos/simonw/datasette/issues/1999,1460988975,IC_kwDOBm6k_c5XFOwv,9599,2023-03-08T22:42:57Z,2023-03-08T22:42:57Z,OWNER,"Aside idea: it might be interesting if there were ""lazy"" template variables available in the context: things that are not actually executed unless a template author requests them.

Imagine if `metadata` was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460986533,https://api.github.com/repos/simonw/datasette/issues/1999,1460986533,IC_kwDOBm6k_c5XFOKl,9599,2023-03-08T22:40:28Z,2023-03-08T22:40:28Z,OWNER,"Figuring out what to do with `display_columns_and_rows()` is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates:

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460970807,https://api.github.com/repos/simonw/datasette/issues/1999,1460970807,IC_kwDOBm6k_c5XFKU3,9599,2023-03-08T22:31:49Z,2023-03-08T22:33:03Z,OWNER,"For the HTML version, I need to decide where all of the stuff that happens in `async def extra_template()` is going to live.

I think it's another one of those extra functions, triggered for `?_extra=context`.

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460943097,https://api.github.com/repos/simonw/datasette/issues/1999,1460943097,IC_kwDOBm6k_c5XFDj5,9599,2023-03-08T22:09:24Z,2023-03-08T22:09:47Z,OWNER,"The ease with which I added that  `?_extra=query` feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.

```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 8d3bb2c930..3e1db9c85f 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -1913,6 +1913,13 @@ async def extra_request():
             ""args"": request.args._data,
         }
 
+    async def extra_query():
+        ""Details of the underlying SQL query""
+        return {
+            ""sql"": sql,
+            ""params"": params,
+        }
+
     async def extra_extras():
         ""Available ?_extra= blocks""
         return {
@@ -1938,6 +1945,7 @@ async def extra_extras():
         extra_primary_keys,
         extra_debug,
         extra_request,
+        extra_query,
         extra_extras,
     )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460916405,https://api.github.com/repos/simonw/datasette/issues/1999,1460916405,IC_kwDOBm6k_c5XE9C1,9599,2023-03-08T21:43:27Z,2023-03-08T21:43:27Z,OWNER,"Just noticed that `_json=colname` is not working, and that's because it's handled by the renderer here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40

But that's not currently being called by my new code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460907148,https://api.github.com/repos/simonw/datasette/issues/1999,1460907148,IC_kwDOBm6k_c5XE6yM,9599,2023-03-08T21:34:30Z,2023-03-08T21:34:30Z,OWNER,"I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460906741,https://api.github.com/repos/simonw/datasette/issues/1999,1460906741,IC_kwDOBm6k_c5XE6r1,9599,2023-03-08T21:34:08Z,2023-03-08T21:34:08Z,OWNER,"So maybe I can refactor it to look a bit more like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604

One thing that's useful here is that `is_view` is handled early, like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472

So if I omit the `is_view` bit from the extracted function I can simplify more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469,https://api.github.com/repos/simonw/datasette/issues/1999,1460905469,IC_kwDOBm6k_c5XE6X9,9599,2023-03-08T21:32:56Z,2023-03-08T21:32:56Z,OWNER,"I'd really like to extract this ugly logic out into a helper function:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684

I copied it in and asked ChatGPT to ""List all of the variable that are either modified or assigned to by the above code"":

> Modified variables:
> 
>     offset
>     where_clauses
>     params
>     order_by
> 
> Assigned variables:
> 
>     sort_value
>     next_by_pk_clauses
>     order_by (when sort or sort_desc is not None)

Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:

> My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:
> 
>     _next
>     is_view
>     sort
>     sort_desc
>     use_rowid
>     pks
>     order_by

Note that `use_rowid` is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460866243,https://api.github.com/repos/simonw/datasette/issues/2036,1460866243,IC_kwDOBm6k_c5XEwzD,9599,2023-03-08T20:57:34Z,2023-03-08T20:57:34Z,OWNER,This fix is released in 0.64.2 https://docs.datasette.io/en/stable/changelog.html#v0-64-2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460848869,https://api.github.com/repos/simonw/datasette/issues/2036,1460848869,IC_kwDOBm6k_c5XEsjl,9599,2023-03-08T20:40:55Z,2023-03-08T20:40:55Z,OWNER,"Here's the https://latest.datasette.io/ deployment that just went out, further demonstrating that this change is working correctly:

<img width=""565"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223844709-bc01813f-b4da-4bb9-bdb8-7be839700a58.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/issues/2037#issuecomment-1460840620,https://api.github.com/repos/simonw/datasette/issues/2037,1460840620,IC_kwDOBm6k_c5XEqis,9599,2023-03-08T20:33:00Z,2023-03-08T20:33:00Z,OWNER,Got the same failure again for a recent commit: https://github.com/simonw/datasette/actions/runs/4368239376/jobs/7640567282,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,
https://github.com/simonw/datasette/issues/2037#issuecomment-1460838797,https://api.github.com/repos/simonw/datasette/issues/2037,1460838797,IC_kwDOBm6k_c5XEqGN,9599,2023-03-08T20:31:15Z,2023-03-08T20:31:15Z,OWNER,"It's this test here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/tests/test_cli.py#L181-L189

Added in:
- #2033 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,
https://github.com/simonw/datasette/issues/2037#issuecomment-1460838109,https://api.github.com/repos/simonw/datasette/issues/2037,1460838109,IC_kwDOBm6k_c5XEp7d,9599,2023-03-08T20:30:36Z,2023-03-08T20:30:36Z,OWNER,Instead of using `isolated_filesystem()` I could use a `tmpdir` fixture instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460827178,https://api.github.com/repos/simonw/datasette/issues/2036,1460827178,IC_kwDOBm6k_c5XEnQq,9599,2023-03-08T20:25:10Z,2023-03-08T20:25:10Z,OWNER,"https://console.cloud.google.com/run/detail/us-central1/new-service/revisions?project=datasette-222320 confirms that the image deployed is:

<img width=""556"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223841493-a104df28-5c9b-47d1-9c56-e7c7f6c61b00.png"">

Compared to https://console.cloud.google.com/run/detail/us-central1/datasette-io/revisions?project=datasette-222320 which shows that `datasette.io` is running:

<img width=""539"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223841616-67a1bb99-b704-4c2c-9dcd-247fb8592766.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460816528,https://api.github.com/repos/simonw/datasette/issues/2036,1460816528,IC_kwDOBm6k_c5XEkqQ,9599,2023-03-08T20:22:50Z,2023-03-08T20:23:20Z,OWNER,"Testing this manually:

```
% datasette publish cloudrun content.db --service new-service
Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression.
Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz]
Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d].
Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ].
---- REMOTE BUILD OUTPUT ----
starting build ""290f41a4-e29a-443c-a1e5-c54513c6143d""

FETCHSOURCE
Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483
Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483...
/ [1 files][  3.9 MiB/  3.9 MiB]                                                
Operation completed over 1 objects/3.9 MiB.
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
Sending build context to Docker daemon  14.52MB
Step 1/9 : FROM python:3.11.0-slim-bullseye
...
Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3 -> 23.0.1
[notice] To update, run: pip install --upgrade pip
Removing intermediate container 8ccebfebebc9
 ---> b972c85b38bb
...
Successfully built 606b7c286d7f
Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest
PUSH
Pushing gcr.io/datasette-222320/datasette-new-service
The push refers to repository [gcr.io/datasette-222320/datasette-new-service]
667b1dc69e5e: Preparing
...
d8ddfcff216f: Pushed
latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004
DONE
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ID                                    CREATE_TIME                DURATION  SOURCE                                                                                          IMAGES                                                   STATUS
290f41a4-e29a-443c-a1e5-c54513c6143d  2023-03-08T20:21:03+00:00  39S       gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz  gcr.io/datasette-222320/datasette-new-service (+1 more)  SUCCESS
Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1]
✓ Deploying new service... Done.                                                                                                                                                      
  ✓ Creating Revision...                                                                                                                                                              
  ✓ Routing traffic...                                                                                                                                                                
  ✓ Setting IAM Policy...                                                                                                                                                             
Done.                                                                                                                                                                                 
Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic.
Service URL: https://new-service-j7hipcg4aq-uc.a.run.app
```
https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460810523,https://api.github.com/repos/simonw/datasette/issues/2036,1460810523,IC_kwDOBm6k_c5XEjMb,9599,2023-03-08T20:17:01Z,2023-03-08T20:17:01Z,OWNER,"I'm going to solve this by using the service name in that `image_id` instead:

```python
image_id = f""gcr.io/{project}/{service_name}""
```
This is a nasty bug, so I'm going to backport it to a `0.64.2` release as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460809643,https://api.github.com/repos/simonw/datasette/issues/2036,1460809643,IC_kwDOBm6k_c5XEi-r,9599,2023-03-08T20:16:10Z,2023-03-08T20:16:10Z,OWNER,"I think the code at fault is here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182

That name ends up defaulting to `datasette` - so multiple different projects may end up deploying to the same `image_id`.

What I think happened in the `datasette.io` bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time.

And so the image that was pushed to `gcr.io/datasette-222320/datasette:latest` by the `simonw/simonwillisonblog-backup` action was then deployed by the `simonw/datasette.io/` action, which broke the site.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460760116,https://api.github.com/repos/simonw/datasette/issues/1999,1460760116,IC_kwDOBm6k_c5XEW40,9599,2023-03-08T19:48:52Z,2023-03-08T19:48:52Z,OWNER,"I'm trying to get `http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v` to return the correct results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460682625,https://api.github.com/repos/simonw/datasette/issues/2035,1460682625,IC_kwDOBm6k_c5XED-B,9599,2023-03-08T18:40:57Z,2023-03-08T18:40:57Z,OWNER,Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460679434,https://api.github.com/repos/simonw/datasette/issues/2035,1460679434,IC_kwDOBm6k_c5XEDMK,9599,2023-03-08T18:39:35Z,2023-03-08T18:39:35Z,OWNER,"I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters

- `_actor_*`
- `_header_*`
- `_cookie_`
- `_now_epoch`
- `_now_date_utc`
- `_now_datetime_utc`
- `_random_chars_*`

Should this new `id__list` syntax look more like those magic parameters, or is it OK to use `name__magic` syntax here instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460668431,https://api.github.com/repos/simonw/datasette/issues/2035,1460668431,IC_kwDOBm6k_c5XEAgP,9599,2023-03-08T18:35:34Z,2023-03-08T18:35:34Z,OWNER,"To implement this properly need to do the following:
- Get the page to display multiple `id: [ text input here ]` fields such that re-submission works
- Figure out how this should work for canned queries and for writable canned queries
- Tests that cover queries, canned queries, writable canned queries

And a bonus feature: what if the Datasette UI layer spotted `:id__list` parameters and used them to add a bit of JavaScript that allowed users to click a `+` button next to an `id` form field to add another one?

Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value.

Though this has an annoying problem: how to tell the difference between an additional `id` input field that the user chose not to populate, v.s. one that is supposed to represent an empty string?

Maybe only support multiple `id` fields for users with JavaScript in order to avoid this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460664619,https://api.github.com/repos/simonw/datasette/issues/2035,1460664619,IC_kwDOBm6k_c5XD_kr,9599,2023-03-08T18:32:29Z,2023-03-08T18:32:29Z,OWNER,"Got a prototype working:
```diff
diff --git a/datasette/views/database.py b/datasette/views/database.py
index 8d289105..6f9d8a44 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -226,6 +226,12 @@ class QueryView(DataView):
     ):
         db = await self.ds.resolve_database(request)
         database = db.name
+        # Disallow x__list query string parameters
+        invalid_params = [k for k in request.args if k.endswith(""__list"")]
+        if invalid_params:
+            raise DatasetteError(
+                ""Invalid query string parameters: {}"".format("", "".join(invalid_params))
+            )
         params = {key: request.args.get(key) for key in request.args}
         if ""sql"" in params:
             params.pop(""sql"")
@@ -258,6 +264,11 @@ class QueryView(DataView):
             for named_parameter in named_parameters
             if not named_parameter.startswith(""_"")
         }
+        # Handle any __list parameters
+        for named_parameter in named_parameters:
+            if named_parameter.endswith(""__list""):
+                list_values = request.args.getlist(named_parameter[:-6])
+                params[named_parameter] = json.dumps(list_values)
 
         # Set to blank string if missing from params
         for named_parameter in named_parameters:
```
This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the `?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL:

<img width=""1226"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223803758-2a1f90bd-bc20-4a77-bdce-4644afe235ba.png"">

That was this URL: `http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460659382,https://api.github.com/repos/simonw/datasette/issues/2035,1460659382,IC_kwDOBm6k_c5XD-S2,9599,2023-03-08T18:28:00Z,2023-03-08T18:28:00Z,OWNER,"Also: `datasette-explain` may need to be updated to understand how to handle this:

`ERROR: conn=<sqlite3.Connection object at 0x102834940>, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460654136,https://api.github.com/repos/simonw/datasette/issues/2035,1460654136,IC_kwDOBm6k_c5XD9A4,9599,2023-03-08T18:25:46Z,2023-03-08T18:25:46Z,OWNER,"Trickiest part of the implementation here is that it needs to know to output three `id` HTML form fields on the page, such that their values are persisted when the form is submitted a second time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460639749,https://api.github.com/repos/simonw/datasette/issues/2035,1460639749,IC_kwDOBm6k_c5XD5gF,9599,2023-03-08T18:17:31Z,2023-03-08T18:17:31Z,OWNER,"Since we are pre-1.0 it's still OK to implement a feature that disallows `?id__list=` in the URL, but allows `:id__list` in SQL queries to reference the JSON list of parameters.

So I'm going to prototype this as the `:id__list` feature and see how it feels.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460637906,https://api.github.com/repos/simonw/datasette/issues/2035,1460637906,IC_kwDOBm6k_c5XD5DS,9599,2023-03-08T18:16:31Z,2023-03-08T18:16:31Z,OWNER,"I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:

1. `?id=1&?id=2` results in `:id` in the query being `[""1"", ""2""]` - no additional syntax required
2. `:id` in the query continues to reference just the first of those parameters - but `:id__list` (or some other custom syntax) instead gets `[""1"", ""2""]` - or, if the URL is `?id=1` - gets `[""1""]`

Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, `?id=1`, resulting in their query breaking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460632758,https://api.github.com/repos/simonw/datasette/issues/2035,1460632758,IC_kwDOBm6k_c5XD3y2,9599,2023-03-08T18:13:49Z,2023-03-08T18:13:49Z,OWNER,"https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the `<select multiple>` default HTML widget produces this exact format of query string:

```html
<form action=""https://www.example.com/"">
<select multiple name=""id"">
  <option>21</option>
  <option>32</option>
  <option>15</option>
  <option>63</option>
</select>
<input type=""submit"">
</form>
```
Submitting that form with the middle two options selected navigates to: `https://www.example.com/?id=32&id=15`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460628199,https://api.github.com/repos/simonw/datasette/issues/2035,1460628199,IC_kwDOBm6k_c5XD2rn,9599,2023-03-08T18:11:31Z,2023-03-08T18:11:31Z,OWNER,"One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first `?id=` parameter version)? Maybe with syntax like this:

    where id in (select value from json_each(:id__list))

Datasette would automatically pass `{""id"": ""11"", ""id__list"": '[""11"", ""32"", ""62""]'}` as arguments to the `db.execute()` method, if the page was called with `?id=11&id=32&id=62`.

This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about `?id__list=` conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block `?id__list=` style parameters entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460621871,https://api.github.com/repos/simonw/datasette/issues/2035,1460621871,IC_kwDOBm6k_c5XD1Iv,9599,2023-03-08T18:08:25Z,2023-03-08T18:09:04Z,OWNER,"My current preferred solution is to lean into SQLite's JSON support.

What if the query page spotted `?id=11&id=32&id=62` and turned that into  a JSON string called `:id:` with a value of `[""11"", ""32"", ""62""]`?

Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where `?id[]=1&id[]=2` would result in an actual PHP array object, which often broke underlying code that had expected `$_GET[""id""]` to be a string, not an array.

So in a query you'd be able to do this:

    where id in (select value from json_each(:id))

And then call it with `?id=11&id=32&id=62`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460618433,https://api.github.com/repos/simonw/datasette/issues/2035,1460618433,IC_kwDOBm6k_c5XD0TB,9599,2023-03-08T18:06:34Z,2023-03-08T18:06:34Z,OWNER,"One way to do this would be to dynamically generate the `where id in (?, ?, ?)` with the correct number of question marks, then feed in a list from `request.args.getlist(""id"")` - but that would require rewriting the SQL query text to add those question marks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,
https://github.com/simonw/datasette/issues/2033#issuecomment-1457117383,https://api.github.com/repos/simonw/datasette/issues/2033,1457117383,IC_kwDOBm6k_c5W2djH,9599,2023-03-06T22:28:55Z,2023-03-06T22:28:55Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugins.html#installing-plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1612296210,
https://github.com/simonw/datasette/pull/2031#issuecomment-1456997425,https://api.github.com/repos/simonw/datasette/issues/2031,1456997425,IC_kwDOBm6k_c5W2AQx,9599,2023-03-06T21:04:27Z,2023-03-06T21:06:34Z,OWNER,"This is a very neat fix, for something I've been wanting for a while.

Add a unit test for the row HTML page - I suggest against this page: https://latest.datasette.io/fixtures/foreign_key_references/1 - and I'll land this PR.

You can model it on this test here: https://github.com/simonw/datasette/blob/a53b893c46453f35decc8c145c138671cee6140c/tests/test_table_html.py#L609-L632

I think adding it to `test_table_html.py` is OK, even though it's technically for the row page and not the table page.

Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,
https://github.com/simonw/datasette/pull/2028#issuecomment-1456914694,https://api.github.com/repos/simonw/datasette/issues/2028,1456914694,IC_kwDOBm6k_c5W1sEG,9599,2023-03-06T20:19:37Z,2023-03-06T20:19:37Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590839187,
https://github.com/simonw/datasette/issues/2024#issuecomment-1426031395,https://api.github.com/repos/simonw/datasette/issues/2024,1426031395,IC_kwDOBm6k_c5U_4Mj,9599,2023-02-10T16:11:53Z,2023-02-10T16:11:53Z,OWNER,Relevant: https://til.simonwillison.net/sqlite/enabling-wal-mode,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1579973223,
https://github.com/simonw/datasette/issues/262#issuecomment-1423067724,https://api.github.com/repos/simonw/datasette/issues/262,1423067724,IC_kwDOBm6k_c5U0kpM,9599,2023-02-08T18:33:32Z,2023-02-08T18:36:48Z,OWNER,"Just realized that it's useful to be able to tell what parameters were used to generate a page... but reflecting things like `_next` back in the JSON is confusing in the presence of `next`.

So I'm going to add an extra for that information too.

Not sure what to call it though:

- `params` - confusing because in the code that's usually used for params passed to SQL queries
- `query_string` - wouldn't that be a string, not params as a dictionary?

I'm going to experiment with a `request` extra that returns some bits of information about the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/pull/1999#issuecomment-1421988953,https://api.github.com/repos/simonw/datasette/issues/1999,1421988953,IC_kwDOBm6k_c5UwdRZ,9599,2023-02-08T04:35:44Z,2023-02-08T05:27:48Z,OWNER,"Next step: get `?_next=...` working (it is ignored at the moment, even though the returned JSON includes the `""next""` key).

Then... figure out how to render HTML and other requested formats.

Then get the tests to pass!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/2019#issuecomment-1421784930,https://api.github.com/repos/simonw/datasette/issues/2019,1421784930,IC_kwDOBm6k_c5Uvrdi,9599,2023-02-08T01:28:25Z,2023-02-08T01:40:46Z,OWNER,"Rather than duplicate this rather awful hack:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L694-L714

I'm tempted to say that the code that calls the new pagination helper needs to ensure that the `sort` or `sort_desc` columns are selected. If it wants to ditch them later (e.g. because they were not included in `?_col=`) it can do that later once the results have come back.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1421600789,https://api.github.com/repos/simonw/datasette/issues/2019,1421600789,IC_kwDOBm6k_c5Uu-gV,9599,2023-02-07T23:12:40Z,2023-02-07T23:16:20Z,OWNER,"Most complicated example of a paginated query: https://latest.datasette.io/fixtures?sql=select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++content%2C%0D%0A++sortable%2C%0D%0A++sortable_with_nulls%2C%0D%0A++sortable_with_nulls_2%2C%0D%0A++text%0D%0Afrom%0D%0A++sortable%0D%0Awhere%0D%0A++(%0D%0A++++sortable_with_nulls+is+null%0D%0A++++and+(%0D%0A++++++(pk1+%3E+%3Ap0)%0D%0A++++++or+(%0D%0A++++++++pk1+%3D+%3Ap0%0D%0A++++++++and+pk2+%3E+%3Ap1%0D%0A++++++)%0D%0A++++)%0D%0A++)%0D%0Aorder+by%0D%0A++sortable_with_nulls+desc%2C%0D%0A++pk1%2C%0D%0A++pk2%0D%0Alimit%0D%0A++101&p0=h&p1=r

```sql
select
  pk1,
  pk2,
  content,
  sortable,
  sortable_with_nulls,
  sortable_with_nulls_2,
  text
from
  sortable
where
  (
    sortable_with_nulls is null
    and (
      (pk1 > :p0)
      or (
        pk1 = :p0
        and pk2 > :p1
      )
    )
  )
order by
  sortable_with_nulls desc,
  pk1,
  pk2
limit
  101
```
Generated by this page: https://latest.datasette.io/fixtures/sortable?_next=%24null%2Ch%2Cr&_sort_desc=sortable_with_nulls

The `_next=` parameter there decodes as `$null,h,r` - and those components are tilde-encoded, so this can be distinguished from an actual `$null` value which would be represented as `~24null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1421274434,https://api.github.com/repos/simonw/datasette/issues/2019,1421274434,IC_kwDOBm6k_c5Utu1C,9599,2023-02-07T18:42:42Z,2023-02-07T18:42:42Z,OWNER,I'm going to build completely separate tests for this in `test_pagination.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1420109153,https://api.github.com/repos/simonw/datasette/issues/2019,1420109153,IC_kwDOBm6k_c5UpSVh,9599,2023-02-07T02:32:36Z,2023-02-07T02:32:36Z,OWNER,"Doing this as a class makes sense to me. There are a few steps:

- Instantiate the class with the information it needs, which includes sort order, page size, tiebreaker columns and SQL query and parameters
- Generate the new SQL query that will actually be executed - maybe this takes the optional `_next` parameter? This returns the SQL and params that should be executed, where the SQL now includes pagination logic plus order by and limit
- The calling code then gets to execute the SQL query to fetch the rows
- Last step: those rows are passed to a paginator method which returns `(rows, next)` - where `rows` is the rows truncated to the correct length (really just with the last one cut off if it's too long for the length) and `next` is either `None` or a token, depending on if there should be a next page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1420106315,https://api.github.com/repos/simonw/datasette/issues/2019,1420106315,IC_kwDOBm6k_c5UpRpL,9599,2023-02-07T02:28:03Z,2023-02-07T02:28:36Z,OWNER,"So I think I can write an abstraction that applies keyset pagination to ANY arbitrary SQL query provided it is given the query, the existing params (so it can pick names for the new params that won't overlap with them), the desired sort order, any existing `_next` token AND the columns that should be used to tie-break any duplicates.

Those tie breakers will be either the primary key(s) or `rowid` if none are provided.

What about the case of SQL views, where offset/limit should be used instead? I'm inclined to have that as a separate pagination abstraction entirely, with the calling code deciding which pagination helper to use based on if keyset pagination makes sense or not.

Might be easier to design a class structure for this starting with `OffsetPaginator`, then using that to inform the design of `KeysetPaginator`.

Might put these in `datasette.utils.pagination` to start off with, then maybe extract them out to `sqlite-utils` later once they've proven themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1420104254,https://api.github.com/repos/simonw/datasette/issues/2019,1420104254,IC_kwDOBm6k_c5UpRI-,9599,2023-02-07T02:24:46Z,2023-02-07T02:24:46Z,OWNER,"Even more complicated: https://latest.datasette.io/fixtures/sortable?sortable_with_nulls__notnull=1&_next=0~2E692704598586882%2Ce%2Cr&_sort=sortable_with_nulls_2

The rewritten SQL for that is:

```sql
select * from (select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where ""sortable_with_nulls"" is not null)
  where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101
```
And it still has the same number of explain steps as the current SQL witohut the subselect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1420101175,https://api.github.com/repos/simonw/datasette/issues/2019,1420101175,IC_kwDOBm6k_c5UpQY3,9599,2023-02-07T02:22:11Z,2023-02-07T02:22:11Z,OWNER,"A more complex example: https://latest.datasette.io/fixtures/sortable?_next=0~2E2650566289400591%2Ca%2Cu&_sort=sortable_with_nulls_2

SQL:

```sql
select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101
```

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p0=a&p1=u&p2=0.2650566289400591

Here's the explain: 49 steps long https://latest.datasette.io/fixtures?sql=explain+select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+where+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

Rewritten with a subselect:

```sql
select * from (
  select pk1, pk2, content, sortable, sortable_with_nulls, sortable_with_nulls_2, text from sortable
)
where (sortable_with_nulls_2 > :p2 or (sortable_with_nulls_2 = :p2 and ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1)))) order by sortable_with_nulls_2, pk1, pk2 limit 101
```
https://latest.datasette.io/fixtures?sql=select+*+from+(%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A)%0D%0Awhere+(sortable_with_nulls_2+%3E+%3Ap2+or+(sortable_with_nulls_2+%3D+%3Ap2+and+((pk1+%3E+%3Ap0)%0D%0A++or%0D%0A(pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1))))+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u

And here's the explain for that - also 49 steps: https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28%0D%0A++select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable%0D%0A%29%0D%0Awhere+%28sortable_with_nulls_2+%3E+%3Ap2+or+%28sortable_with_nulls_2+%3D+%3Ap2+and+%28%28pk1+%3E+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%29%29%29+order+by+sortable_with_nulls_2%2C+pk1%2C+pk2+limit+101&p2=0.2650566289400591&p0=a&p1=u","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1420094396,https://api.github.com/repos/simonw/datasette/issues/2019,1420094396,IC_kwDOBm6k_c5UpOu8,9599,2023-02-07T02:18:11Z,2023-02-07T02:19:16Z,OWNER,"For the SQL underlying this page (the second page in that compound primary key paginated sequence): https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv

The explain for the default query: https://latest.datasette.io/fixtures?sql=explain+select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0Awhere%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

The explain for that query rewritten as this:

```sql
explain
select
  *
from
  (
    select
      pk1,
      pk2,
      pk3,
      content
    from
      compound_three_primary_keys
  )
where
  (
    (pk1 > :p0)
    or (
      pk1 = :p0
      and pk2 > :p1
    )
    or (
      pk1 = :p0
      and pk2 = :p1
      and pk3 > :p2
    )
  )
order by
  pk1,
  pk2,
  pk3
limit
  101
```
https://latest.datasette.io/fixtures?sql=explain+select+*+from+%28select+%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%2C%0D%0A++content%0D%0Afrom%0D%0A++compound_three_primary_keys%0D%0A%29%0D%0A++where%0D%0A++%28%0D%0A++++%28pk1+%3E+%3Ap0%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3E+%3Ap1%0D%0A++++%29%0D%0A++++or+%28%0D%0A++++++pk1+%3D+%3Ap0%0D%0A++++++and+pk2+%3D+%3Ap1%0D%0A++++++and+pk3+%3E+%3Ap2%0D%0A++++%29%0D%0A++%29%0D%0Aorder+by%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++pk3%0D%0Alimit%0D%0A++101&p0=a&p1=d&p2=v

Both explains have 31 steps and look pretty much identical.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1420088670,https://api.github.com/repos/simonw/datasette/issues/2019,1420088670,IC_kwDOBm6k_c5UpNVe,9599,2023-02-07T02:14:35Z,2023-02-07T02:14:35Z,OWNER,"Maybe the correct level of abstraction here is that pagination is something that happens to a SQL query that is defined as SQL and params, without an order by or limit. That's then wrapped in a sub-select and those things are added to it, plus the necessary `where` clauses depending on the page.

Need to check that the query plan for pagination of a subquery isn't slower than the plan for pagination as it works today.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1419953256,https://api.github.com/repos/simonw/datasette/issues/2019,1419953256,IC_kwDOBm6k_c5UosRo,9599,2023-02-06T23:42:56Z,2023-02-06T23:43:10Z,OWNER,"Relevant issue:
- https://github.com/simonw/datasette/issues/1773

Explains this comment: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L697

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1419928455,https://api.github.com/repos/simonw/datasette/issues/2019,1419928455,IC_kwDOBm6k_c5UomOH,9599,2023-02-06T23:21:50Z,2023-02-06T23:21:50Z,OWNER,"Found more logic relating to this:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L684-L732","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1419921228,https://api.github.com/repos/simonw/datasette/issues/2019,1419921228,IC_kwDOBm6k_c5UokdM,9599,2023-02-06T23:14:15Z,2023-02-06T23:14:15Z,OWNER,Crucial utility function: https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/utils/__init__.py#L137-L160,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1419917661,https://api.github.com/repos/simonw/datasette/issues/2019,1419917661,IC_kwDOBm6k_c5Uojld,9599,2023-02-06T23:10:51Z,2023-02-06T23:10:51Z,OWNER,"I should turn `sort` and `sort_desc` into an object representing the sort order earlier in the code.

I should also create something that bundles together `pks` and `use_rowid` and maybe `is_view` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/issues/2019#issuecomment-1419916684,https://api.github.com/repos/simonw/datasette/issues/2019,1419916684,IC_kwDOBm6k_c5UojWM,9599,2023-02-06T23:09:51Z,2023-02-06T23:10:13Z,OWNER,"The inputs and outputs for this are pretty complex.

Inputs:

- `?_next=` from the query string
- `is_view` - is this for a table or view? If it's a view it uses offset/limit pagination - which could actually work for arbitrary queries too. Also views could have keyset pagination if they are known to be sorted by a particular column.
- `sort` and `sort_desc` reflecting the current sort order
- `use_rowid` for if the table is a rowid table with no primary key of its own
- `pks` - the primary keys for the table
- `params` - the current set of parameters, I think used just to count their length so new params can be added as `p5` etc without collisions. This could be handled with a `s0`, `s1` etc naming convention instead.

Outputs:

- `where_clauses` - a list of where clauses to add to the query
- `params` - additional parameters to use with the query due to the new where clauses
- `order_by` - the order by clause","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,
https://github.com/simonw/datasette/pull/1999#issuecomment-1399343659,https://api.github.com/repos/simonw/datasette/issues/1999,1399343659,IC_kwDOBm6k_c5TaEor,9599,2023-01-21T22:19:20Z,2023-02-06T23:02:12Z,OWNER,"HTML mode needs a list of renderers so it can show links to `.geojson` etc - can do that as a hidden extra (maybe called `renderers`), repeating this code:

https://github.com/simonw/datasette/blob/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d/datasette/views/base.py#L477-L497","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/262#issuecomment-1418288327,https://api.github.com/repos/simonw/datasette/issues/262,1418288327,IC_kwDOBm6k_c5UiVzH,9599,2023-02-05T22:57:58Z,2023-02-06T23:01:15Z,OWNER,"I think that does make sense: `?_extra=table` perhaps, which would add `{""table"": ""...""}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/2016#issuecomment-1418288077,https://api.github.com/repos/simonw/datasette/issues/2016,1418288077,IC_kwDOBm6k_c5UiVvN,9599,2023-02-05T22:56:43Z,2023-02-05T22:56:43Z,OWNER,"This absolutely makes sense. One of the biggest goals for Datasette 1.0 is ""documented template contexts"" - for any default template in Datasette that people might want to over-ride there should be documentation that describes the available context variables, plus tests that ensure they don't accidentally get broken by future changes.

Ensuring description/title/etc are available on the index page feels like it fits well into that bucket.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1571207083,
https://github.com/simonw/datasette/issues/2011#issuecomment-1410827249,https://api.github.com/repos/simonw/datasette/issues/2011,1410827249,IC_kwDOBm6k_c5UF4Px,9599,2023-01-31T17:58:54Z,2023-01-31T17:58:54Z,OWNER,"I think this is the relevant code:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/facets.py#L260-L268","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1564769997,
https://github.com/simonw/datasette/issues/2010#issuecomment-1409406327,https://api.github.com/repos/simonw/datasette/issues/2010,1409406327,IC_kwDOBm6k_c5UAdV3,9599,2023-01-30T21:51:58Z,2023-01-30T21:51:58Z,OWNER,"Here's a quick prototype I knocked up for this:
```diff
diff --git a/datasette/static/app.css b/datasette/static/app.css
index 71437bd4..d763bcff 100644
--- a/datasette/static/app.css
+++ b/datasette/static/app.css
@@ -695,7 +695,48 @@ p.zero-results {
 
 
 
+/* Force table to not be like tables anymore */
+body.row table.rows-and-columns,
+body.row .rows-and-columns thead,
+body.row .rows-and-columns tbody,
+body.row .rows-and-columns th,
+body.row .rows-and-columns td,
+body.row .rows-and-columns tr {
+    display: block;
+}
+
+/* Hide table headers (but not display: none;, for accessibility) */
+body.row .rows-and-columns thead tr {
+    position: absolute;
+    top: -9999px;
+    left: -9999px;
+}
+
+body.row .rows-and-columns tr {
+    border: 1px solid #ccc;
+    margin-bottom: 1em;
+    border-radius: 10px;
+    background-color: white;
+    padding: 0.2rem;
+}
 
+body.row .rows-and-columns td {
+    /* Behave  like a ""row"" */
+    border: none;
+    border-bottom: 1px solid #eee;
+    padding: 0;
+    padding-left: 10%;
+    padding-bottom: 0.3em;
+}
+
+body.row .rows-and-columns td:before {
+    display: block;
+    color: black;
+    padding-bottom: 0.2em;
+    font-size: 0.8em;
+    font-weight: bold;
+    background-color: #f5f5f5;
+}
 
 
 /* Overrides ===============================================================*/
diff --git a/datasette/templates/row.html b/datasette/templates/row.html
index 1d1b0bfd..339eb643 100644
--- a/datasette/templates/row.html
+++ b/datasette/templates/row.html
@@ -5,6 +5,9 @@
 {% block extra_head %}
 {{- super() -}}
 <style>
+{% for column in columns %}
+body.row .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: ""{{ column|escape_css_string }}""; }
+{% endfor %}
 @media only screen and (max-width: 576px) {
 {% for column in columns %}
     .rows-and-columns td:nth-of-type({{ loop.index }}):before { content: ""{{ column|escape_css_string }}""; }
```
Now the row page looks like this at all page widths:

<img width=""776"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/215603142-4d020ac0-cc33-436c-a1c6-17c9bcf9edb1.png"">

I think that's better (could do with a bit of tightening up).

One catch: you can't copy and paste the column labels, since they are added using generated content like this:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/static/app.css#L752-L757

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/templates/row.html#L9-L11

I think the row page should switch to different HTML entirely, rather than continuing to share the `<table>` that's used by the table page.

This will be a breaking change for users who customize Datasette, so I should aim to ship it before 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1563264257,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407733793,https://api.github.com/repos/simonw/datasette/issues/2008,1407733793,IC_kwDOBm6k_c5T6FAh,9599,2023-01-29T18:17:40Z,2023-01-29T18:17:40Z,OWNER,"> We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

Had an interesting conversation about this just now: https://fedi.simonwillison.net/@simon/109773800944614366

There's a risk that different runs will return different results due to the shared resource nature of GitHub Actions runners, but a good fix for that is to run comparative tests where you run the benchmark against e.g. both `main` and the incoming PR branch and report back on any differences.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407568923,https://api.github.com/repos/simonw/datasette/issues/2008,1407568923,IC_kwDOBm6k_c5T5cwb,9599,2023-01-29T05:47:36Z,2023-01-29T05:47:36Z,OWNER,"> I don't know how/if you do automated tests for performance, so I haven't changed any of the tests.

We don't have any performance tests yet - would be a useful thing to add, I've not built anything like that before (at least not in CI, I've always done as-hoc performance testing using something like Locust) so I don't have a great feel for how it could work.

I see not having to change the tests at all for this change as a really positive sign. If you find any behaviour differences between this and the previous that's a sign we should add a mother test or two specifying the behaviour we want.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/pull/2008#issuecomment-1407567753,https://api.github.com/repos/simonw/datasette/issues/2008,1407567753,IC_kwDOBm6k_c5T5ceJ,9599,2023-01-29T05:39:54Z,2023-01-29T05:40:34Z,OWNER,"I absolutely _love_ this performance boost - really nice find.

One concern: this will be the first time Datasette ships a core feature that uses window functions.

Window functions were added to SQLite in [version 3.25.0](https://www.sqlite.org/releaselog/3_25_0.html) on 2018-09-15 - which means it's still very common for Datasette to run on versions that don't yet support them.

So I see two options:
- Detect window function support and switch between the old implementation and this better, new one
- Detect window functions and disable the facet-by-JSON feature entirely if they are missing

I like the first option a bit better.

This also leads to a tricky CI challenge: Datasette needs to be able to run its test suite against more than one SQLite version to confidently test this feature going forward.

I don't yet have a good GitHub Actions recipe for this, but I _really_ need one - for `sqlite-utils` too.

Might be able to use this trick for that: https://til.simonwillison.net/sqlite/ld-preload","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560982210,
https://github.com/simonw/datasette/issues/2006#issuecomment-1405488884,https://api.github.com/repos/simonw/datasette/issues/2006,1405488884,IC_kwDOBm6k_c5Txg70,9599,2023-01-26T19:20:53Z,2023-01-26T19:20:53Z,OWNER,I can run a GitHub code search a week or so before the release to give me time to get in touch with anyone who has actions that look like they might break.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1558644003,
https://github.com/simonw/datasette/issues/2006#issuecomment-1405488523,https://api.github.com/repos/simonw/datasette/issues/2006,1405488523,IC_kwDOBm6k_c5Txg2L,9599,2023-01-26T19:20:32Z,2023-01-26T19:20:32Z,OWNER,"This won't actually help that much if the user's GitHub Actions workflow does the equivalent of this:
```
pip install datasette
datasette publish cloudrun mydb.db ...
```
Since they'll get the 1.0 release too.

Since they'll need to make changes anyway, maybe a better solution is to document this and tell people they should use this instead:

```
datasette publish cloudrun mydb.db ... --branch 0.64.1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1558644003,
https://github.com/simonw/datasette/issues/2005#issuecomment-1404458011,https://api.github.com/repos/simonw/datasette/issues/2005,1404458011,IC_kwDOBm6k_c5TtlQb,9599,2023-01-26T01:41:30Z,2023-01-26T01:41:50Z,OWNER,"My code looked like this:
```python
@hookimpl
def extra_template_vars(datasette, view_name, database):
    async def inner():
        if view_name == ""database"":
            return {""blah"": 1}

    return inner
```
It returns `None` in the case where the `if` condition does not match.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1557507274,
https://github.com/simonw/datasette/issues/262#issuecomment-1404253358,https://api.github.com/repos/simonw/datasette/issues/262,1404253358,IC_kwDOBm6k_c5TszSu,9599,2023-01-25T21:35:32Z,2023-01-25T21:35:32Z,OWNER,"This issue here would benefit from some kid of mechanism for returning just the HTML of the table itself, without any of the surrounding material. I'm not sure if that would make sense as an extra or not:
- https://github.com/simonw/datasette-search-all/issues/17","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/pull/1999#issuecomment-1399341658,https://api.github.com/repos/simonw/datasette/issues/1999,1399341658,IC_kwDOBm6k_c5TaEJa,9599,2023-01-21T22:06:29Z,2023-01-21T22:07:30Z,OWNER,"Relevant:
- #1101
- #1672
- #1062","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/1101#issuecomment-1399341761,https://api.github.com/repos/simonw/datasette/issues/1101,1399341761,IC_kwDOBm6k_c5TaELB,9599,2023-01-21T22:07:19Z,2023-01-21T22:07:19Z,OWNER,"Idea for supporting streaming with the `register_output_renderer` hook:

```python
@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""test"",
        ""render"": render_demo,
        ""can_render"": can_render_demo,
        ""render_stream"": render_demo_stream, # This is new
    }
```
So there's a new `""render_stream""` key which can be returned, which if present means that the output renderer supports streaming.

I'll play around with the design of that function signature in:

- #1999
- #1062 ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,
https://github.com/simonw/datasette/pull/1999#issuecomment-1399341151,https://api.github.com/repos/simonw/datasette/issues/1999,1399341151,IC_kwDOBm6k_c5TaEBf,9599,2023-01-21T22:03:20Z,2023-01-21T22:03:20Z,OWNER,"I think I'm going to have to write a new view function from scratch which completely ignores the existing BaseView/DataView/TableView hierarchy.

Here's what I get on the incoming request:
```
(Pdb) request.url, request.full_path, request.host, request.url_vars
('http://127.0.0.1:8001/content/repos.json', '/content/repos.json', '127.0.0.1:8001',
  {'database': 'content', 'table': 'repos', 'format': 'json'})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,
https://github.com/simonw/datasette/issues/262#issuecomment-1399184642,https://api.github.com/repos/simonw/datasette/issues/262,1399184642,IC_kwDOBm6k_c5TZd0C,9599,2023-01-21T05:36:22Z,2023-01-21T05:41:06Z,OWNER,"Maybe `""rows""` should be a default `?_extra=`... but it should be possible to request `""arrays""` instead which would be a list of arrays, more suitable perhaps for custom renderers such as the CSV one.

This could be quite neat, in that EVERY key in the JSON representation would be defined as an extra - just some would be on by default. There could even be a mechanism for turning them back off again, maybe using `?_extra=-rows`.

In which case maybe `?_extra=` isn't actually the right name for this feature. It could be `?_key=` perhaps, or `?_field=`.

Being able to pass `?_field=count,-rows` to get back just the count (and skip executing the count entirely) would be pretty neat.

Although `?_only=count` would be tidier. So maybe the pair of `?_only=` and `?_extra=` would make sense.

Would `?_only=rows` still return the `""ok""` field so you can always look at that to confirm an error didn't occur?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1399184540,https://api.github.com/repos/simonw/datasette/issues/262,1399184540,IC_kwDOBm6k_c5TZdyc,9599,2023-01-21T05:35:32Z,2023-01-21T05:35:32Z,OWNER,"It's annoying that the https://docs.datasette.io/en/0.64.1/plugin_hooks.html#register-output-renderer-datasette plugin hook passes `rows` as ""list of sqlite3.Row objects"" - I'd prefer it if that plugin hook worked with JSON data, not `sqlite3.Row`.

https://docs.datasette.io/en/0.64.1/plugin_hooks.html#render-cell-row-value-column-table-database-datasette is documented as accepting `Row` but actually gets `CustomRow`, see:

- #1973","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1399178823,https://api.github.com/repos/simonw/datasette/issues/262,1399178823,IC_kwDOBm6k_c5TZcZH,9599,2023-01-21T04:54:49Z,2023-01-21T04:54:49Z,OWNER,"I pushed my prototype so far, going to start a draft PR for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1399178591,https://api.github.com/repos/simonw/datasette/issues/262,1399178591,IC_kwDOBm6k_c5TZcVf,9599,2023-01-21T04:53:15Z,2023-01-21T04:53:15Z,OWNER,"Implementing this to work with the `.json` extension is going to be a lot harder.

The challenge here is that we're working with the whole `BaseView()` v.s. `TableView()` abstraction, which I've been wanting to get rid of for a long time.

`BaseView()` calls `.data()` and expects to get back a `(data, extra_template_data, templates)` tuple - then if a format is in play (`.json` or `.geojson` or similar from a plugin) it hands off `data` to that. If `.csv` is involved it does something special, in order to support streaming responses. And if it's regular HTML it calls `await extra_template_data()` and combines that with `data` and passes it to the template.

I want this to work completely differently: I want the formats (including HTML) to have the option of adding some extra `?_extra=` extras, then I want HTML to be able to render the page entirely from the JSON if necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1399145981,https://api.github.com/repos/simonw/datasette/issues/262,1399145981,IC_kwDOBm6k_c5TZUX9,9599,2023-01-21T01:56:52Z,2023-01-21T01:56:52Z,OWNER,"Got first prototype working using `asyncinject` and it's pretty nice:
```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index ad45ecd3..c8690b22 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -2,6 +2,7 @@ import asyncio
 import itertools
 import json
 
+from asyncinject import Registry
 import markupsafe
 
 from datasette.plugins import pm
@@ -538,57 +539,60 @@ class TableView(DataView):
         # Execute the main query!
         results = await db.execute(sql, params, truncate=True, **extra_args)
 
-        # Calculate the total count for this query
-        count = None
-        if (
-            not db.is_mutable
-            and self.ds.inspect_data
-            and count_sql == f""select count(*) from {table_name} ""
-        ):
-            # We can use a previously cached table row count
-            try:
-                count = self.ds.inspect_data[database_name][""tables""][table_name][
-                    ""count""
-                ]
-            except KeyError:
-                pass
-
-        # Otherwise run a select count(*) ...
-        if count_sql and count is None and not nocount:
-            try:
-                count_rows = list(await db.execute(count_sql, from_sql_params))
-                count = count_rows[0][0]
-            except QueryInterrupted:
-                pass
-
-        # Faceting
-        if not self.ds.setting(""allow_facet"") and any(
-            arg.startswith(""_facet"") for arg in request.args
-        ):
-            raise BadRequest(""_facet= is not allowed"")
+        # Resolve extras
+        extras = _get_extras(request)
+        if request.args.getlist(""_facet""):
+            extras.add(""facet_results"")
 
-        # pylint: disable=no-member
-        facet_classes = list(
-            itertools.chain.from_iterable(pm.hook.register_facet_classes())
-        )
-        facet_results = {}
-        facets_timed_out = []
-        facet_instances = []
-        for klass in facet_classes:
-            facet_instances.append(
-                klass(
-                    self.ds,
-                    request,
-                    database_name,
-                    sql=sql_no_order_no_limit,
-                    params=params,
-                    table=table_name,
-                    metadata=table_metadata,
-                    row_count=count,
-                )
+        async def extra_count():
+            # Calculate the total count for this query
+            count = None
+            if (
+                not db.is_mutable
+                and self.ds.inspect_data
+                and count_sql == f""select count(*) from {table_name} ""
+            ):
+                # We can use a previously cached table row count
+                try:
+                    count = self.ds.inspect_data[database_name][""tables""][table_name][
+                        ""count""
+                    ]
+                except KeyError:
+                    pass
+
+            # Otherwise run a select count(*) ...
+            if count_sql and count is None and not nocount:
+                try:
+                    count_rows = list(await db.execute(count_sql, from_sql_params))
+                    count = count_rows[0][0]
+                except QueryInterrupted:
+                    pass
+            return count
+
+        async def facet_instances(extra_count):
+            facet_instances = []
+            facet_classes = list(
+                itertools.chain.from_iterable(pm.hook.register_facet_classes())
             )
+            for facet_class in facet_classes:
+                facet_instances.append(
+                    facet_class(
+                        self.ds,
+                        request,
+                        database_name,
+                        sql=sql_no_order_no_limit,
+                        params=params,
+                        table=table_name,
+                        metadata=table_metadata,
+                        row_count=extra_count,
+                    )
+                )
+            return facet_instances
+
+        async def extra_facet_results(facet_instances):
+            facet_results = {}
+            facets_timed_out = []
 
-        async def execute_facets():
             if not nofacet:
                 # Run them in parallel
                 facet_awaitables = [facet.facet_results() for facet in facet_instances]
@@ -607,9 +611,13 @@ class TableView(DataView):
                         facet_results[key] = facet_info
                     facets_timed_out.extend(instance_facets_timed_out)
 
-        suggested_facets = []
+            return {
+                ""results"": facet_results,
+                ""timed_out"": facets_timed_out,
+            }
 
-        async def execute_suggested_facets():
+        async def extra_suggested_facets(facet_instances):
+            suggested_facets = []
             # Calculate suggested facets
             if (
                 self.ds.setting(""suggest_facets"")
@@ -624,8 +632,15 @@ class TableView(DataView):
                 ]
                 for suggest_result in await gather(*facet_suggest_awaitables):
                     suggested_facets.extend(suggest_result)
+            return suggested_facets
+
+        # Faceting
+        if not self.ds.setting(""allow_facet"") and any(
+            arg.startswith(""_facet"") for arg in request.args
+        ):
+            raise BadRequest(""_facet= is not allowed"")
 
-        await gather(execute_facets(), execute_suggested_facets())
+        # pylint: disable=no-member
 
         # Figure out columns and rows for the query
         columns = [r[0] for r in results.description]
@@ -732,17 +747,56 @@ class TableView(DataView):
             rows = rows[:page_size]
 
         # human_description_en combines filters AND search, if provided
-        human_description_en = filters.human_description_en(
-            extra=extra_human_descriptions
-        )
+        async def extra_human_description_en():
+            human_description_en = filters.human_description_en(
+                extra=extra_human_descriptions
+            )
+            if sort or sort_desc:
+                human_description_en = "" "".join(
+                    [b for b in [human_description_en, sorted_by] if b]
+                )
+            return human_description_en
 
         if sort or sort_desc:
             sorted_by = ""sorted by {}{}"".format(
                 (sort or sort_desc), "" descending"" if sort_desc else """"
             )
-            human_description_en = "" "".join(
-                [b for b in [human_description_en, sorted_by] if b]
-            )
+
+        async def extra_next_url():
+            return next_url
+
+        async def extra_columns():
+            return columns
+
+        async def extra_primary_keys():
+            return pks
+
+        registry = Registry(
+            extra_count,
+            extra_facet_results,
+            extra_suggested_facets,
+            facet_instances,
+            extra_human_description_en,
+            extra_next_url,
+            extra_columns,
+            extra_primary_keys,
+        )
+
+        results = await registry.resolve_multi(
+            [""extra_{}"".format(extra) for extra in extras]
+        )
+        data = {
+            ""ok"": True,
+            ""rows"": rows[:page_size],
+            ""next"": next_value and str(next_value) or None,
+        }
+        data.update({
+            key.replace(""extra_"", """"): value
+            for key, value in results.items()
+            if key.startswith(""extra_"")
+            and key.replace(""extra_"", """") in extras
+        })
+        return Response.json(data, default=repr)
 
         async def extra_template():
             nonlocal sort
@@ -1334,3 +1388,11 @@ class TableDropView(BaseView):
 
         await db.execute_write_fn(drop_table)
         return Response.json({""ok"": True}, status=200)
+
+
+def _get_extras(request):
+    extra_bits = request.args.getlist(""_extra"")
+    extras = set()
+    for bit in extra_bits:
+        extras.update(bit.split("",""))
+    return extras
```
With that in place, `http://127.0.0.1:8001/content/releases?author=25778&_size=1&_extra=count,primary_keys,columns&_facet=author` returns:
```json
{
  ""ok"": true,
  ""rows"": [
    {
      ""html_url"": ""https://github.com/eyeseast/geocode-sqlite/releases/tag/0.1.2"",
      ""id"": 30926270,
      ""author"": {
        ""value"": 25778,
        ""label"": ""eyeseast""
      },
      ""node_id"": ""MDc6UmVsZWFzZTMwOTI2Mjcw"",
      ""tag_name"": ""0.1.2"",
      ""target_commitish"": ""master"",
      ""name"": ""v0.1.2"",
      ""draft"": 0,
      ""prerelease"": 1,
      ""created_at"": ""2020-09-08T17:48:24Z"",
      ""published_at"": ""2020-09-08T17:50:15Z"",
      ""body"": ""Basic API is in place, with CLI support for Google, Bing, MapQuest and Nominatum (OSM) geocoders."",
      ""repo"": {
        ""value"": 293361514,
        ""label"": ""geocode-sqlite""
      },
      ""reactions"": null,
      ""mentions_count"": null
    }
  ],
  ""next"": ""30926270"",
  ""primary_keys"": [
    ""id""
  ],
  ""columns"": [
    ""html_url"",
    ""id"",
    ""author"",
    ""node_id"",
    ""tag_name"",
    ""target_commitish"",
    ""name"",
    ""draft"",
    ""prerelease"",
    ""created_at"",
    ""published_at"",
    ""body"",
    ""repo"",
    ""reactions"",
    ""mentions_count""
  ],
  ""count"": 25,
  ""facet_results"": {
    ""results"": {
      ""author"": {
        ""name"": ""author"",
        ""type"": ""column"",
        ""hideable"": true,
        ""toggle_url"": ""/content/releases?author=25778&_size=1&_extra=count%2Cprimary_keys%2Ccolumns"",
        ""results"": [
          {
            ""value"": 25778,
            ""label"": ""eyeseast"",
            ""count"": 25,
            ""toggle_url"": ""http://127.0.0.1:8001/content/releases?_size=1&_extra=count%2Cprimary_keys%2Ccolumns&_facet=author"",
            ""selected"": true
          }
        ],
        ""truncated"": false
      }
    },
    ""timed_out"": []
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/1998#issuecomment-1398768399,https://api.github.com/repos/simonw/datasette/issues/1998,1398768399,IC_kwDOBm6k_c5TX4MP,9599,2023-01-20T18:19:06Z,2023-01-20T18:19:06Z,OWNER,"Simplest solution would be to ditch the `version_option` decorator and roll a custom option based on it instead, imitating what this code does:

https://github.com/pallets/click/blob/7586834cab38c5592d9d6de3ee0ebe75d4353bfb/src/click/decorators.py#L413-L524","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551113681,
https://github.com/simonw/datasette/issues/1998#issuecomment-1398767813,https://api.github.com/repos/simonw/datasette/issues/1998,1398767813,IC_kwDOBm6k_c5TX4DF,9599,2023-01-20T18:18:27Z,2023-01-20T18:18:27Z,OWNER,"Fell down a bit of a rabbit hole trying to figure out how to get Click's `version_option()` to evaluate a custom message. Got this far:

```python
class _VersionMessage(UserString):
    @property
    def data(self):
        return ""%(prog)s, version %(version)s (SQLite {})"".format(
            sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0]
        )

    @data.setter
    def data(self, value):
        pass


@click.group(cls=DefaultGroup, default=""serve"", default_if_no_args=True)
@click.version_option(version=__version__, message=_VersionMessage(""""))
def cli():
    """"""
    Datasette is an open source multi-tool for exploring and publishing data

    \b
    About Datasette: https://datasette.io/
    Full documentation: https://docs.datasette.io/
    """"""
```
But now:

```
% datasette --version
%(prog)s, version %(version)s (SQLite 3.40.1)
```

I was trying to avoid running that `select sqlite_version()` thing unless the `--version` option was used.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551113681,
https://github.com/simonw/datasette/issues/262#issuecomment-1397942113,https://api.github.com/repos/simonw/datasette/issues/262,1397942113,IC_kwDOBm6k_c5TUudh,9599,2023-01-20T05:33:00Z,2023-01-20T05:33:00Z,OWNER,"I'm going to write code which parses `?_extra=` in the comma separated or multiple parameter format and then looks up functions in a dictionary.

It will return an error if you ask for an extra that does not exist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1385807684,https://api.github.com/repos/simonw/datasette/issues/262,1385807684,IC_kwDOBm6k_c5Smb9E,9599,2023-01-17T17:51:54Z,2023-01-19T23:20:59Z,OWNER,"In most cases, the `?_extra=xxx` name exactly corresponds to the additional key that is added to the JSON.

`?_facet=...` is one example of a query string argument that causes an extra key - `""facet_results""` - to be added to the JSON even though it wasn't requested by name in a `?_extra=`.

Am I OK with that? I think so.

Related issue:
- #1558 

Actually there's an edge-case here that's worth considering: it's possible to use metadata to set default facets for a table. If you do this for a table, then `.json` for that table will always calculate and return those facets - which may be an expensive and unnecessary operation.

So maybe we don't include `facet_results` in the JSON unless explicitly asked for in that case, but have a rule that `?_facet` implies `?_extra=facet_results`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/1989#issuecomment-1397734424,https://api.github.com/repos/simonw/datasette/issues/1989,1397734424,IC_kwDOBm6k_c5TT7wY,9599,2023-01-19T23:18:45Z,2023-01-19T23:18:45Z,OWNER,"There's a `?_nocol=x` argument you can use to hide a column when you link to a table, but that won't help you if you need to hide the column for privacy reasons: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments

One solution right now is to define a SQL view for the things that you DO want people to be able to see, and then use Datasette's permission system to hide the tables (`'""allow"": false` in metadata for each table) but show the views.

If you want to redact specific columns there's a plugin for doing that: https://datasette.io/plugins/datasette-mask-columns

This does make sense as more of a core Datasette feature though - tagging it as a feature suggestion.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1531991339,
https://github.com/simonw/datasette/issues/262#issuecomment-1385805702,https://api.github.com/repos/simonw/datasette/issues/262,1385805702,IC_kwDOBm6k_c5SmbeG,9599,2023-01-17T17:50:17Z,2023-01-17T17:50:17Z,OWNER,Or maybe have a `permissions` extra which includes `allow_execute_sql` and `private`? Could anything else go in there?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1384752452,https://api.github.com/repos/simonw/datasette/issues/262,1384752452,IC_kwDOBm6k_c5SiaVE,9599,2023-01-17T02:14:41Z,2023-01-17T02:15:58Z,OWNER,"Thinking about `?_extra=` values just for the table JSON. The default shape will look like this:

```json
{
  ""ok"": true,
  ""rows"": [{""id"": 1, ""name"": ""Name""}],
  ""next"": null,
}
```
The table extras could be:

- `count` - adds a `""count""` field with a full `count(*)` for that filtered table
- `next_url` - the full URL to the next page
- `columns` - adds `""columns"": [""id"", ""name""]`
- `expandable_columns` - a list of columns that can be expanded (note that `""expanded_columns"": [...]` shows up automatically if the user passes any `?_label=` options, like on https://latest.datasette.io/fixtures/facetable.json?_label=_city_id ) - I'm tempted to rename this to `label_columns` and have it add both `label_columns` and `label_columns_selected` or similar.
- `primary_keys` - a list of primary keys e.g. `[""id""]` - not sure what to do about `rowid` columns here
- `query` - a `{""sql"": ""select ..."", ""params"": {""p0"": ""1""}}` object
- `units` - the units feature
- `suggested_facets` - suggested facets
- `metadata` - a `{""metadata"": {""source_url"": ""...""}}` etc block - differs from current in that it would be nested in `""metadata"": {...}`.

Stuff currently in https://latest.datasette.io/fixtures/facetable.json that is not yet covered by the above:
```
  ""database"": ""fixtures"",
  ""table"": ""facetable"",
  ""is_view"": false,
  ""human_description_en"": ""where id = 1"",
  ""private"": false,
  ""allow_execute_sql"": true,
  ""query_ms"": 16.749476999393664,
```
I'm tempted to bundle `database`, `table`, `is_view` and `human_description_en` into one (not sure what to call it though, perhaps `display_details`?) - and then drop `allow_execute_sql` entirely and have `private` and `query_ms` as their own named extras.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/782#issuecomment-1368269732,https://api.github.com/repos/simonw/datasette/issues/782,1368269732,IC_kwDOBm6k_c5RjiOk,9599,2022-12-31T19:32:33Z,2023-01-17T02:05:45Z,OWNER,"New thinking on the trimmed-down default. Previously I was going to use `""row""` and `""next_url""` - I now want to do this instead:
```json
{
  ""ok"": true,
  ""rows"": [
    {
      ""pk1"": ""a"",
      ""pk2"": ""a"",
      ""pk3"": ""a"",
      ""content"": ""a-a-a""
    },
    {
      ""pk1"": ""a"",
      ""pk2"": ""a"",
      ""pk3"": ""b"",
      ""content"": ""a-a-b""
    }
  ],
  ""next"": ""a,a,b""
}
```
If there isn't a next page it will return `""next"": null`.

This is even more succinct. I'm OK with people having to request `next_url` if they don't want to construct the new URL themselves.

The `""ok"": true` is there so it can be `false` for errors, consistently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,
https://github.com/simonw/datasette/issues/262#issuecomment-1384743243,https://api.github.com/repos/simonw/datasette/issues/262,1384743243,IC_kwDOBm6k_c5SiYFL,9599,2023-01-17T02:01:26Z,2023-01-17T02:01:26Z,OWNER,"I'm tempted NOT to document the JSON for the `/.json` page, simply because I'm not at all convinced that the current homepage design is the best possible use of that space - and I'd like to reserve the opportunity to redesign that in e.g. Datasette 1.1 without it being a breaking change to the documented JSON API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1384742385,https://api.github.com/repos/simonw/datasette/issues/262,1384742385,IC_kwDOBm6k_c5SiX3x,9599,2023-01-17T02:00:23Z,2023-01-17T02:00:38Z,OWNER,"I'm not actually too happy about how `/fixtures.json` currently entirely changes shape based on whether or not you pass a `?sql=` argument to it.

Maybe I can fix that disparity with extras too?

The list of tables you see on `/fixtures.json` without the `?sql=` could become another extra. The HTML version of that page could know to request that extra by default.

This would also support running a SQL query but also returning a list of tables - which can be useful for building a SQL editor interface which hints at the tables that are available to the user - or even for generating the configuration needed by the CodeMirror editor's SQL completion, added in:
- #1893","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-1384741055,https://api.github.com/repos/simonw/datasette/issues/262,1384741055,IC_kwDOBm6k_c5SiXi_,9599,2023-01-17T01:58:24Z,2023-01-17T01:58:24Z,OWNER,"As suggested in this issue:
- #1721 

There are three parts of the Datasette API that need to support extras:

- Table, e.g. https://latest.datasette.io/fixtures/facetable.json
- Row, e.g. https://latest.datasette.io/fixtures/facetable/1.json
- Query, e.g. https://latest.datasette.io/fixtures/neighborhood_search.json or https://latest.datasette.io/fixtures.json?sql=%0Aselect+_neighborhood%2C+facet_cities.name%2C+state%0Afrom+facetable%0A++++join+facet_cities%0A++++++++on+facetable._city_id+%3D+facet_cities.id%0Awhere+_neighborhood+like+%27%25%27+||+%3Atext+||+%27%25%27%0Aorder+by+_neighborhood%3B%0A&text=

There are two other pages I should consider though:

- https://latest.datasette.io/.json - the JSON version of the https://latest.datasette.io/ homepage
- https://latest.datasette.io/fixtures.json - note that this is different from the same URL with `?sql=...` appended to it. This is the index of tables in a specific database","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/262#issuecomment-712988146,https://api.github.com/repos/simonw/datasette/issues/262,712988146,MDEyOklzc3VlQ29tbWVudDcxMjk4ODE0Ng==,9599,2020-10-20T16:32:02Z,2023-01-17T01:54:13Z,OWNER,"Just realized I added an undocumented `?_extras=` option to the row view years ago and forgot about it - it's not even documented. Added in a30c5b220c15360d575e94b0e67f3255e120b916 - https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

That will need to be made consistent with the new mechanism. I think `?_extra=a&_extra=b` is more consistent with other Datasette features (like `?_facet=col1&_facet=col2`) but potentially quite verbose.

So I could support `?_extra=a,b,c` as an alternative allowed syntax, or I could allow `?_extra=single` and `?_extras=comma,separated`.

I think I prefer allowing commas in `?_extra=`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,
https://github.com/simonw/datasette/issues/1988#issuecomment-1379502161,https://api.github.com/repos/simonw/datasette/issues/1988,1379502161,IC_kwDOBm6k_c5SOYhR,9599,2023-01-11T21:24:24Z,2023-01-11T21:25:05Z,OWNER,Renaming `sql()` to `template_sql()` in https://github.com/simonw/datasette-template-sql and shipping a 2.0 release of that plugin (it is currently at 1.0.2) would be a sensible short-term fix too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,
https://github.com/simonw/datasette/issues/1988#issuecomment-1379500131,https://api.github.com/repos/simonw/datasette/issues/1988,1379500131,IC_kwDOBm6k_c5SOYBj,9599,2023-01-11T21:22:31Z,2023-01-11T21:22:31Z,OWNER,"Relevant code:

https://github.com/simonw/datasette/blob/6a352e99ab988dbf8fd22a100049caa6ad33f1ec/datasette/app.py#L1138-L1149

https://github.com/simonw/datasette/blob/6a352e99ab988dbf8fd22a100049caa6ad33f1ec/datasette/app.py#L1193-L1195","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,
https://github.com/simonw/datasette/issues/1988#issuecomment-1379492499,https://api.github.com/repos/simonw/datasette/issues/1988,1379492499,IC_kwDOBm6k_c5SOWKT,9599,2023-01-11T21:15:01Z,2023-01-11T21:20:14Z,OWNER,"This is tricky. `datasette-template-sql` registers a custom `sql()` function, but @cldellow spotted a situation where this would interfere with an existing `{{ sql }}` variable in a template context!

https://github.com/simonw/datasette-template-sql/blob/70483fd94bb0349761830e37c9ab928e8966841a/datasette_template_sql/__init__.py

```python
@hookimpl
def extra_template_vars(datasette):
    async def execute_sql(sql, args=None, database=None):
        db = datasette.get_database(database)
        return (await db.execute(sql, args)).rows

    return {""sql"": execute_sql}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,
https://github.com/simonw/datasette/issues/1988#issuecomment-1379495504,https://api.github.com/repos/simonw/datasette/issues/1988,1379495504,IC_kwDOBm6k_c5SOW5Q,9599,2023-01-11T21:18:00Z,2023-01-11T21:18:25Z,OWNER,"A few options:
- Ensure that the explicit template context overrides anything that plugins might do - so in this case the `sql(...)` function would not be available on that page. This would break people who have custom templates that use that function though.
- Encourage naming conventions where functions and variables  from plugins are less likely to interfere with existing functionality.
- Completely change how templates work, so you never have a variable called `{{ sql }}` - it is always accessed via some parent object instead, such as `{{ page.sql }}`.

That last option actually fits quite well with my efforts to unify template rendering with JSON (and JSON extras) so it might be the best way to address this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,
https://github.com/simonw/datasette/issues/1987#issuecomment-1379313564,https://api.github.com/repos/simonw/datasette/issues/1987,1379313564,IC_kwDOBm6k_c5SNqec,9599,2023-01-11T18:29:25Z,2023-01-11T18:29:25Z,OWNER,"Fixed:
- https://docs.datasette.io/en/latest/installation.html#using-pip
- https://docs.datasette.io/en/stable/installation.html#using-pip
- https://docs.datasette.io/en/0.64.1/installation.html#using-pip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,
https://github.com/simonw/datasette/issues/1987#issuecomment-1379311498,https://api.github.com/repos/simonw/datasette/issues/1987,1379311498,IC_kwDOBm6k_c5SNp-K,9599,2023-01-11T18:27:30Z,2023-01-11T18:27:46Z,OWNER,"I'm going to leave the bad link in the old versions, because I can't figure out a way to update them using ReadTheDocs - shipping 0.64.1 will at least mean that both `/latest/` and `/stable/` have a good link.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,
https://github.com/simonw/datasette/issues/1987#issuecomment-1379282756,https://api.github.com/repos/simonw/datasette/issues/1987,1379282756,IC_kwDOBm6k_c5SNi9E,9599,2023-01-11T18:04:22Z,2023-01-11T18:04:22Z,OWNER,"That change is now live on https://docs.datasette.io/en/latest/installation.html#using-pip

I may need to ship a new stable dot-release to update the `/en/stable/` page though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,
https://github.com/simonw/datasette/issues/1987#issuecomment-1379276484,https://api.github.com/repos/simonw/datasette/issues/1987,1379276484,IC_kwDOBm6k_c5SNhbE,9599,2023-01-11T18:00:35Z,2023-01-11T18:00:35Z,OWNER,"Next challenge: how to best deploy this.

Ideally I'd like to fix all of the older versions of the docs too - pages like https://docs.datasette.io/en/0.59.4/installation.html#using-pip

Not sure if ReadTheDocs has a mechanism for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,
https://github.com/simonw/datasette/issues/1985#issuecomment-1378253346,https://api.github.com/repos/simonw/datasette/issues/1985,1378253346,IC_kwDOBm6k_c5SJnoi,9599,2023-01-11T05:18:05Z,2023-01-11T05:18:05Z,OWNER,`Datasette(path)` should either work or it should return an easily debugged error message.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1528448642,
https://github.com/simonw/datasette/pull/1984#issuecomment-1376619691,https://api.github.com/repos/simonw/datasette/issues/1984,1376619691,IC_kwDOBm6k_c5SDYyr,9599,2023-01-10T02:02:16Z,2023-01-10T02:02:16Z,OWNER,Preview at https://datasette--1984.org.readthedocs.build/en/1984/ looks good to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1526635374,
https://github.com/simonw/datasette/issues/979#issuecomment-1375801928,https://api.github.com/repos/simonw/datasette/issues/979,1375801928,IC_kwDOBm6k_c5SARJI,9599,2023-01-09T15:32:22Z,2023-01-09T15:32:22Z,OWNER,This will not be part of the new default JSON design. It will be handled by an ?_extra instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710650633,
https://github.com/simonw/datasette/issues/1558#issuecomment-1375800361,https://api.github.com/repos/simonw/datasette/issues/1558,1375800361,IC_kwDOBm6k_c5SAQwp,9599,2023-01-09T15:31:17Z,2023-01-09T15:31:17Z,OWNER,"Related:
- #782 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1082584499,
https://github.com/simonw/datasette/issues/1983#issuecomment-1375797856,https://api.github.com/repos/simonw/datasette/issues/1983,1375797856,IC_kwDOBm6k_c5SAQJg,9599,2023-01-09T15:29:34Z,2023-01-09T15:29:34Z,OWNER,"I had forgotten this had special handling for cursors!

The default representation of a row as a tuple rather than a dictionary might not be the right thing to do based on:
- #782 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,
https://github.com/simonw/datasette/issues/1983#issuecomment-1375795441,https://api.github.com/repos/simonw/datasette/issues/1983,1375795441,IC_kwDOBm6k_c5SAPjx,9599,2023-01-09T15:27:53Z,2023-01-09T15:27:53Z,OWNER,https://github.com/simonw/datasette/blob/7b48664d751e59222c01c97a2739d53d6363bdda/datasette/utils/__init__.py#L163-L178,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,
https://github.com/simonw/datasette/issues/1981#issuecomment-1375142205,https://api.github.com/repos/simonw/datasette/issues/1981,1375142205,IC_kwDOBm6k_c5R9wE9,9599,2023-01-09T06:05:44Z,2023-01-09T06:05:44Z,OWNER,"I want the labels to take as much space as they need, but line up with each other.

If they are more than half the width of the page then the form fields should break to another line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524983536,