html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1519#issuecomment-974562942,https://api.github.com/repos/simonw/datasette/issues/1519,974562942,IC_kwDOBm6k_c46FqZ-,9599,simonw,2021-11-20T00:59:32Z,2021-11-20T00:59:32Z,OWNER,"Ouch a nasty bug crept through there - https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/compound_three_primary_keys says

> 500: name 'ds' is not defined","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974561593,https://api.github.com/repos/simonw/datasette/issues/1519,974561593,IC_kwDOBm6k_c46FqE5,9599,simonw,2021-11-20T00:53:19Z,2021-11-20T00:53:19Z,OWNER,Adding that test found (I hope!) all of the remaining `base_url` bugs. There were a bunch! I think I finally get to close #838 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974559176,https://api.github.com/repos/simonw/datasette/issues/1519,974559176,IC_kwDOBm6k_c46FpfI,9599,simonw,2021-11-20T00:42:08Z,2021-11-20T00:42:08Z,OWNER,"> In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here.

This worked, I managed to get the tests to fail! Here's the change I made:

```diff
diff --git a/tests/test_html.py b/tests/test_html.py
index f24165b..dbdfe59 100644
--- a/tests/test_html.py
+++ b/tests/test_html.py
@@ -1614,12 +1614,19 @@ def test_metadata_sort_desc(app_client):
         ""/fixtures/compound_three_primary_keys/a,a,a"",
         ""/fixtures/paginated_view"",
         ""/fixtures/facetable"",
+        ""/fixtures?sql=select+1"",
     ],
 )
-def test_base_url_config(app_client_base_url_prefix, path):
+@pytest.mark.parametrize(""use_prefix"", (True, False))
+def test_base_url_config(app_client_base_url_prefix, path, use_prefix):
     client = app_client_base_url_prefix
-    response = client.get(""/prefix/"" + path.lstrip(""/""))
+    path_to_get = path
+    if use_prefix:
+        path_to_get = ""/prefix/"" + path.lstrip(""/"")
+    response = client.get(path_to_get)
     soup = Soup(response.body, ""html.parser"")
+    if path == ""/fixtures?sql=select+1"":
+        assert False
     for el in soup.findAll([""a"", ""link"", ""script""]):
         if ""href"" in el.attrs:
             href = el[""href""]
@@ -1642,11 +1649,12 @@ def test_base_url_config(app_client_base_url_prefix, path):
             # If this has been made absolute it may start http://localhost/
             if href.startswith(""http://localhost/""):
                 href = href[len(""http://localost/"") :]
-            assert href.startswith(""/prefix/""), {
+            assert href.startswith(""/prefix/""), json.dumps({
                 ""path"": path,
+                ""path_to_get"":  path_to_get,
                 ""href_or_src"": href,
                 ""element_parent"": str(el.parent),
-            }
+            }, indent=4, default=repr)
 
 
 def test_base_url_affects_metadata_extra_css_urls(app_client_base_url_prefix):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974558267,https://api.github.com/repos/simonw/datasette/issues/1519,974558267,IC_kwDOBm6k_c46FpQ7,9599,simonw,2021-11-20T00:37:57Z,2021-11-20T00:37:57Z,OWNER,Thanks to #1522 I have a live demo that exhibits this bug now: https://apache-proxy-demo.datasette.io/prefix/fixtures/attraction_characteristic,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1522#issuecomment-974558076,https://api.github.com/repos/simonw/datasette/issues/1522,974558076,IC_kwDOBm6k_c46FpN8,9599,simonw,2021-11-20T00:36:56Z,2021-11-20T00:36:56Z,OWNER,That 503 error is _really_ frustrating: I have a deploy running at https://apache-proxy-demo.datasette.io/prefix/ and after a fresh deploy it serves 503 errors for quite a while - then eventually starts working.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1522#issuecomment-974557766,https://api.github.com/repos/simonw/datasette/issues/1522,974557766,IC_kwDOBm6k_c46FpJG,9599,simonw,2021-11-20T00:35:25Z,2021-11-20T00:35:25Z,OWNER,Wrote a TIL about `--build-arg` and Cloud Run: https://til.simonwillison.net/cloudrun/using-build-args-with-cloud-run,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1522#issuecomment-974542348,https://api.github.com/repos/simonw/datasette/issues/1522,974542348,IC_kwDOBm6k_c46FlYM,9599,simonw,2021-11-19T23:41:47Z,2021-11-19T23:44:07Z,OWNER,Do I have to use `cloudbuild.yml` to specify these? https://stackoverflow.com/a/58327340/6083 and https://stackoverflow.com/a/66232670/6083 suggest I do.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1522#issuecomment-974541971,https://api.github.com/repos/simonw/datasette/issues/1522,974541971,IC_kwDOBm6k_c46FlST,9599,simonw,2021-11-19T23:40:32Z,2021-11-19T23:40:32Z,OWNER,"I want to be able to use build arguments to specify which commit version or branch of Datasette to deploy.

This is proving hard to work out. I have this in my Dockerfile now:

```
ARG DATASETTE_REF

RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip
```
Which works locally:

    docker build -t datasette-apache-proxy-demo . \
      --build-arg DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d

But I can't figure out the right incantation to pass to `gcloud build submit`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1522#issuecomment-974523569,https://api.github.com/repos/simonw/datasette/issues/1522,974523569,IC_kwDOBm6k_c46Fgyx,9599,simonw,2021-11-19T22:51:10Z,2021-11-19T22:51:10Z,OWNER,I wan a GitHub Action which I can manually activate to deploy a new version of that demo... and I want it to bake in the latest release of Datasette so I can use it to demonstrate bug fixes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1522#issuecomment-974523297,https://api.github.com/repos/simonw/datasette/issues/1522,974523297,IC_kwDOBm6k_c46Fguh,9599,simonw,2021-11-19T22:50:31Z,2021-11-19T22:50:31Z,OWNER,Demo code is now at: https://github.com/simonw/datasette/tree/main/demos/apache-proxy,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1522#issuecomment-974521687,https://api.github.com/repos/simonw/datasette/issues/1522,974521687,IC_kwDOBm6k_c46FgVX,9599,simonw,2021-11-19T22:46:26Z,2021-11-19T22:46:26Z,OWNER,"Oh weird, it started working: https://datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app/prefix/fixtures/sortable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1522#issuecomment-974506401,https://api.github.com/repos/simonw/datasette/issues/1522,974506401,IC_kwDOBm6k_c46Fcmh,9599,simonw,2021-11-19T22:11:51Z,2021-11-19T22:11:51Z,OWNER,"This is frustrating: I have the following Dockerfile:
```dockerfile
FROM python:3-alpine

RUN apk add --no-cache \
	apache2 \
	apache2-proxy \
	bash

RUN pip install datasette

ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
RUN chmod +x /tini

# Append this to the end of the default httpd.conf file
RUN echo $'ServerName localhost\n\
\n\
<Proxy *>\n\
  Order deny,allow\n\
  Allow from all\n\
</Proxy>\n\
\n\
ProxyPass        /prefix/ http://localhost:8001/\n\
Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf

RUN echo $'<a href=""/prefix/"">Datasette</a>' > /var/www/localhost/htdocs/index.html

WORKDIR /app

ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

RUN echo $'#!/usr/bin/env bash\n\
set -e\n\
\n\
httpd -D FOREGROUND &\n\
datasette fixtures.db --setting base_url ""/prefix/"" -h 0.0.0.0 -p 8001 &\n\
\n\
wait -n' > /app/start.sh

RUN chmod +x /app/start.sh

EXPOSE 80
ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""]
```
It works fine when I run it locally:
```
docker build -t datasette-apache-proxy-demo .
docker run -p 5000:80 datasette-apache-proxy-demo
```
But when I deploy it to Cloud Run with the following script:
```bash
#!/bin/bash
# https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run

NAME=""datasette-apache-proxy-demo""
PROJECT=$(gcloud config get-value project)
IMAGE=""gcr.io/$PROJECT/$NAME""

gcloud builds submit --tag $IMAGE
gcloud run deploy \
    --allow-unauthenticated \
    --platform=managed \
    --image $IMAGE $NAME \
    --port 80
```
It serves the `/` page successfully, but hits to `/prefix/` return the following 503 error:

> Service Unavailable
>
> The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.
>
> Apache/2.4.51 (Unix) Server at datasette-apache-proxy-demo-j7hipcg4aq-uc.a.run.app Port 80

Cloud Run logs:

<img width=""1347"" alt=""Screen Shot 2021-11-19 at 2 10 54 PM"" src=""https://user-images.githubusercontent.com/9599/142698024-4811e228-0373-45ea-9758-24564c46b3e4.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1519#issuecomment-974478126,https://api.github.com/repos/simonw/datasette/issues/1519,974478126,IC_kwDOBm6k_c46FVsu,9599,simonw,2021-11-19T21:16:36Z,2021-11-19T21:16:36Z,OWNER,"In the meantime I can catch these errors by changing the test to run each path twice, once with and once without the prefix. This should accurately simulate how Apache is working here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974477465,https://api.github.com/repos/simonw/datasette/issues/1519,974477465,IC_kwDOBm6k_c46FViZ,9599,simonw,2021-11-19T21:15:30Z,2021-11-19T21:15:30Z,OWNER,"I think what's happening here is Apache is actually making a request to `/fixtures` rather than making a request to `/prefix/fixtures` - and Datasette is replying to requests on both the prefixed and the non-prefixed paths.

This is pretty confusing! I think Datasette should ONLY reply to `/prefix/fixtures` instead and return a 404 for `/fixtures` - this would make things a whole lot easier to debug.

But shipping that change could break existing deployments. Maybe that should be a breaking change for 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974450232,https://api.github.com/repos/simonw/datasette/issues/1519,974450232,IC_kwDOBm6k_c46FO44,9599,simonw,2021-11-19T20:41:53Z,2021-11-19T20:42:19Z,OWNER,https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration says I should use `ProxyPreserveHost on`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974447950,https://api.github.com/repos/simonw/datasette/issues/1519,974447950,IC_kwDOBm6k_c46FOVO,9599,simonw,2021-11-19T20:40:19Z,2021-11-19T20:40:19Z,OWNER,"Figured it out! The test is not an accurate recreation of what is happening, because it doesn't simulate a request with a path of `/fixtures` that has been redirected by the proxy to `/prefix/fixtures`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1522#issuecomment-974435661,https://api.github.com/repos/simonw/datasette/issues/1522,974435661,IC_kwDOBm6k_c46FLVN,9599,simonw,2021-11-19T20:33:42Z,2021-11-19T20:33:42Z,OWNER,"Should just be a case of deploying this `Dockerfile`:

```Dockerfile
FROM python:3-alpine

RUN apk add --no-cache \
	apache2 \
	apache2-proxy \
	bash

RUN pip install datasette

ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
RUN chmod +x /tini

# Append this to the end of the default httpd.conf file
RUN echo $'ServerName localhost\n\
\n\
<Proxy *>\n\
  Order deny,allow\n\
  Allow from all\n\
</Proxy>\n\
\n\
ProxyPass        /foo/bar/ http://localhost:9000/\n\
Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf

RUN echo $'<a href=""/foo/bar/"">Datasette</a>' > /var/www/localhost/htdocs/index.html

WORKDIR /app

ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

RUN echo $'#!/usr/bin/env bash\n\
set -e\n\
\n\
httpd -D FOREGROUND &\n\
datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\
\n\
wait -n' > /app/start.sh

RUN chmod +x /app/start.sh

EXPOSE 80
ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""]
```
I can follow this TIL: https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058896236,Deploy a live instance of demos/apache-proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974433520,https://api.github.com/repos/simonw/datasette/issues/1521,974433520,IC_kwDOBm6k_c46FKzw,9599,simonw,2021-11-19T20:32:29Z,2021-11-19T20:32:29Z,OWNER,This configuration works great.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1519#issuecomment-974433320,https://api.github.com/repos/simonw/datasette/issues/1519,974433320,IC_kwDOBm6k_c46FKwo,9599,simonw,2021-11-19T20:32:04Z,2021-11-19T20:32:04Z,OWNER,Still not clear why the tests pass but the live example fails.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974433206,https://api.github.com/repos/simonw/datasette/issues/1519,974433206,IC_kwDOBm6k_c46FKu2,9599,simonw,2021-11-19T20:31:52Z,2021-11-19T20:31:52Z,OWNER,"Modified my `Dockerfile` to do this:

    RUN pip install https://github.com/simonw/datasette/archive/ff0dd4da38d48c2fa9250ecf336002c9ed724e36.zip

And now the `request` in that debug `?_context=1` looks like this:
```
    ""request"": ""<asgi.Request method=\""GET\"" url=\""http://localhost:9000/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1\"">""
```
That explains the bug - that request doesn't maintain the original path prefix of `http://localhost:5000/foo/bar/fixtures?sql=` (also it's been rewritten to `localhost:9000` instead of `localhost:5000`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974422829,https://api.github.com/repos/simonw/datasette/issues/1519,974422829,IC_kwDOBm6k_c46FIMt,9599,simonw,2021-11-19T20:26:35Z,2021-11-19T20:26:35Z,OWNER,"In the `?_context=` debug view the request looks like this:
```
    ""request"": ""<datasette.utils.asgi.Request object at 0x7faf9fe06200>"",
```
I'm going to add a `repr()` to it such that it's a bit more useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974420619,https://api.github.com/repos/simonw/datasette/issues/1519,974420619,IC_kwDOBm6k_c46FHqL,9599,simonw,2021-11-19T20:25:19Z,2021-11-19T20:25:19Z,OWNER,"The implementations of `path_with_removed_args` and `path_with_format`:

https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L228-L254

https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/utils/__init__.py#L710-L729","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974418496,https://api.github.com/repos/simonw/datasette/issues/1519,974418496,IC_kwDOBm6k_c46FHJA,9599,simonw,2021-11-19T20:24:16Z,2021-11-19T20:24:16Z,OWNER,"Here's the code that generates `edit_sql_url` correctly: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L416-L420

And  here's the code for `show_hide_link`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/database.py#L432-L433

And for `url_csv`: https://github.com/simonw/datasette/blob/85849935292e500ab7a99f8fe0f9546e903baad3/datasette/views/base.py#L600-L602","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974398399,https://api.github.com/repos/simonw/datasette/issues/1519,974398399,IC_kwDOBm6k_c46FCO_,9599,simonw,2021-11-19T20:08:20Z,2021-11-19T20:22:02Z,OWNER,"The relevant test is this one: https://github.com/simonw/datasette/blob/30255055150d7bc0affc8156adc18295495020ff/tests/test_html.py#L1608-L1649

I modified that test to add `""/fixtures/facetable?sql=select+1""` as one of the tested paths, and dropped in an `assert False` to pause it in the debugger:
```
    @pytest.mark.parametrize(
        ""path"",
        [
            ""/"",
            ""/fixtures"",
            ""/fixtures/compound_three_primary_keys"",
            ""/fixtures/compound_three_primary_keys/a,a,a"",
            ""/fixtures/paginated_view"",
            ""/fixtures/facetable"",
            ""/fixtures?sql=select+1"",
        ],
    )
    def test_base_url_config(app_client_base_url_prefix, path):
        client = app_client_base_url_prefix
        response = client.get(""/prefix/"" + path.lstrip(""/""))
        soup = Soup(response.body, ""html.parser"")
        if path == ""/fixtures?sql=select+1"":
>           assert False
E           assert False
```
BUT... in the debugger:
```
(Pdb) print(soup)
...
<p class=""export-links"">This data as
  <a href=""/prefix/fixtures.json?sql=select+1"">json</a>,
  <a href=""/prefix/fixtures.testall?sql=select+1"">testall</a>,
  <a href=""/prefix/fixtures.testnone?sql=select+1"">testnone</a>,
  <a href=""/prefix/fixtures.testresponse?sql=select+1"">testresponse</a>,
  <a href=""/prefix/fixtures.csv?sql=select+1&amp;_size=max"">CSV</a></p>
```
Those all have the correct prefix! But that's not what I'm seeing in my `Dockerfile` reproduction of the issue.

Something very weird is going on here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974405016,https://api.github.com/repos/simonw/datasette/issues/1519,974405016,IC_kwDOBm6k_c46FD2Y,9599,simonw,2021-11-19T20:14:19Z,2021-11-19T20:15:05Z,OWNER,"I added `template_debug` in the Dockerfile:
```
datasette fixtures.db --setting template_debug 1 --setting base_url ""/foo/bar/"" -p 9000 &\n\
```
And then hit `http://localhost:5000/foo/bar/fixtures?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1` to view the template context - and it showed the bug, output edited to just show relevant keys:

```json
{
    ""edit_sql_url"": ""/foo/bar/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1"",
    ""settings"": {
        ""force_https_urls"": false,
        ""template_debug"": true,
        ""trace_debug"": false,
        ""base_url"": ""/foo/bar/""
    },
    ""show_hide_link"": ""/fixtures?sql=select+%2A+from+compound_three_primary_keys+limit+1&_context=1&_hide_sql=1"",
    ""show_hide_text"": ""hide"",
    ""show_hide_hidden"": """",
    ""renderers"": {
        ""json"": ""/fixtures.json?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1""
    },
    ""url_csv"": ""/fixtures.csv?sql=select+*+from+compound_three_primary_keys+limit+1&_context=1&_size=max"",
    ""url_csv_path"": ""/fixtures.csv"",
    ""base_url"": ""/foo/bar/""
}
```
This is so strange. `edit_sql_url` and `base_url` are correct, but `show_hide_link` and `url_csv` and `renderers.json` are not.

And it's _really strange_ that the bug doesn't show up in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974391204,https://api.github.com/repos/simonw/datasette/issues/1519,974391204,IC_kwDOBm6k_c46FAek,9599,simonw,2021-11-19T20:02:41Z,2021-11-19T20:02:41Z,OWNER,"Bug confirmed:

![proxy-bug](https://user-images.githubusercontent.com/9599/142684666-112136bf-9243-4b6e-8202-339fcfe91bcc.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974389472,https://api.github.com/repos/simonw/datasette/issues/1519,974389472,IC_kwDOBm6k_c46FADg,9599,simonw,2021-11-19T20:01:02Z,2021-11-19T20:01:02Z,OWNER,I now have a `Dockerfile` in https://github.com/simonw/datasette/issues/1521#issuecomment-974388295 that I can use to run a local Apache 2 with `mod_proxy` to investigate this class of bugs!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1521#issuecomment-974388295,https://api.github.com/repos/simonw/datasette/issues/1521,974388295,IC_kwDOBm6k_c46E_xH,9599,simonw,2021-11-19T20:00:06Z,2021-11-19T20:00:06Z,OWNER,"And this is the version that proxies to a `base_url` of `/foo/bar/`:

```Dockerfile
FROM python:3-alpine

RUN apk add --no-cache \
	apache2 \
	apache2-proxy \
	bash

RUN pip install datasette

ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
RUN chmod +x /tini

# Append this to the end of the default httpd.conf file
RUN echo $'ServerName localhost\n\
\n\
<Proxy *>\n\
  Order deny,allow\n\
  Allow from all\n\
</Proxy>\n\
\n\
ProxyPass        /foo/bar/ http://localhost:9000/\n\
Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf

RUN echo $'<a href=""/foo/bar/"">Datasette</a>' > /var/www/localhost/htdocs/index.html

WORKDIR /app

ADD https://latest.datasette.io/fixtures.db /app/fixtures.db

RUN echo $'#!/usr/bin/env bash\n\
set -e\n\
\n\
httpd -D FOREGROUND &\n\
datasette fixtures.db --setting base_url ""/foo/bar/"" -p 9000 &\n\
\n\
wait -n' > /app/start.sh

RUN chmod +x /app/start.sh

EXPOSE 80
ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974380798,https://api.github.com/repos/simonw/datasette/issues/1521,974380798,IC_kwDOBm6k_c46E97-,9599,simonw,2021-11-19T19:54:26Z,2021-11-19T19:54:26Z,OWNER,"Got it working! Here's a `Dockerfile` which runs completely stand-alone (thanks to using the `echo $'` trick to write out the config files it needs) and successfully serves Datasette behind Apache and `mod_proxy`:

```Dockerfile
FROM python:3-alpine

RUN apk add --no-cache \
	apache2 \
	apache2-proxy \
	bash

RUN pip install datasette

ENV TINI_VERSION v0.18.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static /tini
RUN chmod +x /tini

# Append this to the end of the default httpd.conf file
RUN echo $'ServerName localhost\n\
\n\
<Proxy *>\n\
  Order deny,allow\n\
  Allow from all\n\
</Proxy>\n\
\n\
ProxyPass        / http://localhost:9000/\n\
ProxyPassReverse / http://localhost:9000/\n\
Header add X-Proxied-By ""Apache2""' >> /etc/apache2/httpd.conf

WORKDIR /app

RUN echo $'#!/usr/bin/env bash\n\
set -e\n\
\n\
httpd -D FOREGROUND &\n\
datasette -p 9000 &\n\
\n\
wait -n' > /app/start.sh

RUN chmod +x /app/start.sh

EXPOSE 80
ENTRYPOINT [""/tini"", ""--"", ""/app/start.sh""]
```

Run it like this:

```
docker build -t datasette-apache2-proxy .   
docker run -p 5000:80 --rm datasette-apache2-proxy
```
Then run this to confirm:
```
~ % curl -i 'http://localhost:5000/-/versions.json'
HTTP/1.1 200 OK
Date: Fri, 19 Nov 2021 19:54:05 GMT
Server: uvicorn
content-type: application/json; charset=utf-8
X-Proxied-By: Apache2
Transfer-Encoding: chunked

{""python"": {""version"": ""3.10.0"", ""full"": ""3.10.0 (default, Nov 13 2021, 03:23:03) [GCC 10.3.1 20210424]""}, ""datasette"": {""version"": ""0.59.2""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.15.0"", ""sqlite"": {""version"": ""3.35.5"", ""fts_versions"": [""FTS5"", ""FTS4"", ""FTS3""], ""extensions"": {""json1"": null}, ""compile_options"": [""COMPILER=gcc-10.3.1 20210424"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_GEOPOLY"", ""ENABLE_JSON1"", ""ENABLE_MATH_FUNCTIONS"", ""ENABLE_RTREE"", ""ENABLE_UNLOCK_NOTIFY"", ""MAX_VARIABLE_NUMBER=250000"", ""SECURE_DELETE"", ""THREADSAFE=1"", ""USE_URI""]}}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974371116,https://api.github.com/repos/simonw/datasette/issues/1521,974371116,IC_kwDOBm6k_c46E7ks,9599,simonw,2021-11-19T19:45:47Z,2021-11-19T19:45:47Z,OWNER,"https://github.com/krallin/tini says:

> *NOTE: If you are using Docker 1.13 or greater, Tini is included in Docker itself. This includes all versions of Docker CE. To enable Tini, just [pass the `--init` flag to `docker run`](https://docs.docker.com/engine/reference/commandline/run/).*","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974336020,https://api.github.com/repos/simonw/datasette/issues/1521,974336020,IC_kwDOBm6k_c46EzAU,9599,simonw,2021-11-19T19:10:48Z,2021-11-19T19:10:48Z,OWNER,"There's a promising looking minimal Apache 2 proxy config here: https://stackoverflow.com/questions/26474476/minimal-configuration-for-apache-reverse-proxy-in-docker-container

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974334278,https://api.github.com/repos/simonw/datasette/issues/1521,974334278,IC_kwDOBm6k_c46EylG,9599,simonw,2021-11-19T19:08:09Z,2021-11-19T19:08:09Z,OWNER,"Stripping comments using this StackOverflow recipe: https://unix.stackexchange.com/a/157619

    docker run -it --entrypoint sh alpine-apache2-sh \
      -c ""cat /etc/apache2/httpd.conf"" | sed '/^[[:blank:]]*#/d;s/#.*//'

Result is here: https://gist.github.com/simonw/0a05090df5fcff8e8b3334621fa17976","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974332787,https://api.github.com/repos/simonw/datasette/issues/1521,974332787,IC_kwDOBm6k_c46EyNz,9599,simonw,2021-11-19T19:05:52Z,2021-11-19T19:05:52Z,OWNER,"Made myself this Dockerfile to let me explore a bit:
```Dockerfile
FROM python:3-alpine

RUN apk add --no-cache \
	apache2

CMD [""sh""]
```
Then:
```
% docker run alpine-apache2-sh
% docker run -it alpine-apache2-sh
/ # ls /etc/apache2/httpd.conf
/etc/apache2/httpd.conf
/ # cat /etc/apache2/httpd.conf
#
# This is the main Apache HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
...
```
Copying that into a GIST like so:
```
docker run -it --entrypoint sh alpine-apache2-sh -c ""cat /etc/apache2/httpd.conf"" | pbcopy
```
Gist here: https://gist.github.com/simonw/5ea0db6049192cb9f761fbd6beb3a84a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974327812,https://api.github.com/repos/simonw/datasette/issues/1521,974327812,IC_kwDOBm6k_c46ExAE,9599,simonw,2021-11-19T18:58:49Z,2021-11-19T18:59:55Z,OWNER,"From this example: https://github.com/tigelane/dockerfiles/blob/06cff2ac8cdc920ebd64f50965115eaa3d0afb84/Alpine-Apache2/Dockerfile#L25-L31 it looks like running `apk add apache2` installs a config file at `/etc/apache2/httpd.conf` - so one approach is to then modify that file.

```
# APACHE - Alpine
#################
RUN apk --update add apache2 php5-apache2 && \
    #apk add openrc --no-cache && \
    rm -rf /var/cache/apk/* && \
    sed -i 's/#ServerName www.example.com:80/ServerName localhost/' /etc/apache2/httpd.conf && \
    mkdir -p /run/apache2/

# Upload our files from folder ""dist"".
COPY dist /var/www/localhost/htdocs

# Manually set up the apache environment variables
ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
ENV APACHE_LOCK_DIR /var/lock/apache2
ENV APACHE_PID_FILE /var/run/apache2.pid

# Execute apache2 on run
########################
EXPOSE 80
ENTRYPOINT [""httpd""]
CMD [""-D"", ""FOREGROUND""]
```

I think I'll create my own separate copy and modify that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974321391,https://api.github.com/repos/simonw/datasette/issues/1521,974321391,IC_kwDOBm6k_c46Evbv,9599,simonw,2021-11-19T18:49:15Z,2021-11-19T18:57:18Z,OWNER,"This pattern looks like it can help: https://ahmet.im/blog/cloud-run-multiple-processes-easy-way/ - see example in https://github.com/ahmetb/multi-process-container-lazy-solution

I got that demo working locally like this:

```bash
cd /tmp
git clone https://github.com/ahmetb/multi-process-container-lazy-solution
cd multi-process-container-lazy-solution
docker build -t multi-process-container-lazy-solution .
docker run -p 5000:8080 --rm multi-process-container-lazy-solution
```

I want to use `apache2` rather than `nginx` though. I found a few relevant examples of Apache in Alpine:

- https://github.com/Hacking-Lab/alpine-apache2-reverse-proxy/blob/master/Dockerfile
- https://www.sentiatechblog.com/running-apache-in-a-docker-container
- https://github.com/search?l=Dockerfile&q=alpine+apache2&type=code

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1521#issuecomment-974322178,https://api.github.com/repos/simonw/datasette/issues/1521,974322178,IC_kwDOBm6k_c46EvoC,9599,simonw,2021-11-19T18:50:22Z,2021-11-19T18:50:22Z,OWNER,"I'll get this working on my laptop first, but then I want to get it up and running on Cloud Run - maybe with a GitHub Actions  workflow in this repo that re-deploys it on manual execution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058815557,Docker configuration for exercising Datasette behind Apache mod_proxy,
https://github.com/simonw/datasette/issues/1519#issuecomment-974310208,https://api.github.com/repos/simonw/datasette/issues/1519,974310208,IC_kwDOBm6k_c46EstA,9599,simonw,2021-11-19T18:32:31Z,2021-11-19T18:32:31Z,OWNER,Having a live demo running on Cloud Run that proxies through Apache and uses `base_url` would be incredibly useful for replicating and debugging this kind of thing. I wonder how hard it is to run Apache and `mod_proxy` in the same Docker container as Datasette?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1519#issuecomment-974309591,https://api.github.com/repos/simonw/datasette/issues/1519,974309591,IC_kwDOBm6k_c46EsjX,9599,simonw,2021-11-19T18:31:32Z,2021-11-19T18:31:32Z,OWNER,"`base_url` has been a source of so many bugs like this! I often find them quite hard to replicate, likely because I haven't made myself a good Apache `mod_proxy` testing environment yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058790545,base_url is omitted in JSON and CSV views,
https://github.com/simonw/datasette/issues/1520#issuecomment-974308215,https://api.github.com/repos/simonw/datasette/issues/1520,974308215,IC_kwDOBm6k_c46EsN3,9599,simonw,2021-11-19T18:29:26Z,2021-11-19T18:29:26Z,OWNER,"The solution that jumps to mind first is that it would be neat if routes could return something that meant ""actually my bad, I can't handle this after all - move to the next one in the list"".

A related idea: it might be useful for custom views like my one here to say ""no actually call the default view for this, but give me back the response so I can modify it in some way"". Kind of like Django or ASGI middleware.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058803238,Pattern for avoiding accidental URL over-rides,
https://github.com/simonw/datasette/issues/1518#issuecomment-974300823,https://api.github.com/repos/simonw/datasette/issues/1518,974300823,IC_kwDOBm6k_c46EqaX,9599,simonw,2021-11-19T18:18:32Z,2021-11-19T18:18:32Z,OWNER,"> This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.

I can definitely support this using pure-JSON - I could make two versions of the row available, one that's an array of cell objects and the other that's an object mapping column names to column raw values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-974285803,https://api.github.com/repos/simonw/datasette/issues/1518,974285803,IC_kwDOBm6k_c46Emvr,9599,simonw,2021-11-19T17:56:48Z,2021-11-19T18:14:30Z,OWNER,"Very confused by this piece of code here: https://github.com/simonw/datasette/blob/1c13e1af0664a4dfb1e69714c56523279cae09e4/datasette/views/table.py#L37-L63

I added it in https://github.com/simonw/datasette/commit/754836eef043676e84626c4fd3cb993eed0d2976 - in the new world that should probably be replaced by pure JSON.

Aha - this comment explains it: https://github.com/simonw/datasette/issues/521#issuecomment-505279560

> I think the trick is to redefine what a ""cell_row"" is. Each row is currently a list of cells:
> 
> https://github.com/simonw/datasette/blob/6341f8cbc7833022012804dea120b838ec1f6558/datasette/views/table.py#L159-L163
> 
> I can redefine the row (the `cells` variable in the above example) as a thing-that-iterates-cells (hence behaving like a list) but that also supports `__getitem__` access for looking up cell values if you know the name of the column.

The goal was to support neater custom templates like this:
```html+jinja
{% for row in display_rows %}
  <h2 class=""scientist"">{{ row[""First_Name""] }} {{ row[""Last_Name""] }}</h2>
  ...
```
This may be an argument for continuing to allow non-JSON-objects through to the HTML templates. Need to think about that a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-974287570,https://api.github.com/repos/simonw/datasette/issues/1518,974287570,IC_kwDOBm6k_c46EnLS,9599,simonw,2021-11-19T17:59:33Z,2021-11-19T17:59:33Z,OWNER,"I'm going to try leaning into the `asyncinject` mechanism a bit here. One method can execute and return the raw rows. Another can turn that into the default minimal JSON representation. Then a third can take that (or take both) and use it to inflate out the JSON that the HTML template needs, with those extras and with the rendered cells from plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/pull/1495#issuecomment-974108455,https://api.github.com/repos/simonw/datasette/issues/1495,974108455,IC_kwDOBm6k_c46D7cn,192568,mroswell,2021-11-19T14:14:35Z,2021-11-19T14:14:35Z,CONTRIBUTOR,A nudge on this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973820125,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973820125,IC_kwDOCGYnMM46C1Dd,9599,simonw,2021-11-19T07:25:55Z,2021-11-19T07:25:55Z,OWNER,"`alter=True` doesn't make sense to support here either, because `.lookup()` already adds missing columns: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2743-L2746","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`,
https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802998,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802998,IC_kwDOCGYnMM46Cw32,9599,simonw,2021-11-19T06:59:22Z,2021-11-19T06:59:32Z,OWNER,"I don't think I need the `DEFAULT` defaults for `.insert()` either, since it just passes through to `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`,
https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802766,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802766,IC_kwDOCGYnMM46Cw0O,9599,simonw,2021-11-19T06:58:45Z,2021-11-19T06:58:45Z,OWNER,"And neither does `hash_id`. On that basis I'm going to specifically list the ones that DO make sense, and hope that I remember to add any new ones in the future. I can add a code comment hint to `.insert()` about that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`,
https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802469,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802469,IC_kwDOCGYnMM46Cwvl,9599,simonw,2021-11-19T06:58:03Z,2021-11-19T06:58:03Z,OWNER,Also: I don't think `ignore=` and `replace=` make sense in the context of `lookup()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`,
https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973802308,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973802308,IC_kwDOCGYnMM46CwtE,9599,simonw,2021-11-19T06:57:37Z,2021-11-19T06:57:37Z,OWNER,"Here's the current full method signature for `.insert()`: https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2462-L2477

I could add a test which uses introspection (`inspect.signature(method).parameters`) to confirm that `.lookup()` has a super-set of the arguments accepted by `.insert()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`,
https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973801650,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973801650,IC_kwDOCGYnMM46Cwiy,9599,simonw,2021-11-19T06:55:56Z,2021-11-19T06:55:56Z,OWNER,"`pk` needs to be an explicit argument to `.lookup()`. The rest could be `**kwargs` passed through to `.insert()`, like this hacked together version (docstring removed for brevity):

```python
    def lookup(
        self,
        lookup_values: Dict[str, Any],
        extra_values: Optional[Dict[str, Any]] = None,
        pk=""id"",
        **insert_kwargs,
    ):
        """"""
        assert isinstance(lookup_values, dict)
        if extra_values is not None:
            assert isinstance(extra_values, dict)
        combined_values = dict(lookup_values)
        if extra_values is not None:
            combined_values.update(extra_values)
        if self.exists():
            self.add_missing_columns([combined_values])
            unique_column_sets = [set(i.columns) for i in self.indexes]
            if set(lookup_values.keys()) not in unique_column_sets:
                self.create_index(lookup_values.keys(), unique=True)
            wheres = [""[{}] = ?"".format(column) for column in lookup_values]
            rows = list(
                self.rows_where(
                    "" and "".join(wheres), [value for _, value in lookup_values.items()]
                )
            )
            try:
                return rows[0][pk]
            except IndexError:
                return self.insert(combined_values, pk=pk, **insert_kwargs).last_pk
        else:
            pk = self.insert(combined_values, pk=pk, **insert_kwargs).last_pk
            self.create_index(lookup_values.keys(), unique=True)
            return pk
```
I think I'll explicitly list the parameters, mainly so they can be typed and covered by automatic documentation.

I do worry that I'll add more keyword arguments to `.insert()` in the future and forget to mirror them to `.lookup()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`,
https://github.com/simonw/sqlite-utils/issues/342#issuecomment-973800795,https://api.github.com/repos/simonw/sqlite-utils/issues/342,973800795,IC_kwDOCGYnMM46CwVb,9599,simonw,2021-11-19T06:54:08Z,2021-11-19T06:54:08Z,OWNER,"Looking at the code for `lookup()` it currently hard-codes `pk` to `""id""` - but it actually only calls `.insert()` in two places, both of which could be passed extra arguments.

https://github.com/simonw/sqlite-utils/blob/3b8abe608796e99e4ffc5f3f4597a85e605c0e9b/sqlite_utils/db.py#L2756-L2763","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058196641,Extra options to `lookup()` which get passed to `insert()`,
https://github.com/simonw/datasette/issues/1518#issuecomment-973700549,https://api.github.com/repos/simonw/datasette/issues/1518,973700549,IC_kwDOBm6k_c46CX3F,9599,simonw,2021-11-19T03:31:20Z,2021-11-19T03:31:26Z,OWNER,"... and while I'm doing all of this I can rewrite the templates to not use those cheating magical functions AND document the template context at the same time, refs:
- #1510.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-973700322,https://api.github.com/repos/simonw/datasette/issues/1518,973700322,IC_kwDOBm6k_c46CXzi,9599,simonw,2021-11-19T03:30:30Z,2021-11-19T03:30:30Z,OWNER,"Right now the HTML version gets to cheat - it passes through objects that are not JSON serializable, including custom functions that can then be called by Jinja.

I'm interested in maybe removing this cheating - if the HTML version could only request JSON-serializable extras those could be exposed in the API as well.

It would also help cleanup the kind-of-nasty pattern I use in the current `BaseView` where everything returns both a bunch of JSON-serializable data AND an awaitable function that then gets to add extra things to the HTML context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-973698917,https://api.github.com/repos/simonw/datasette/issues/1518,973698917,IC_kwDOBm6k_c46CXdl,9599,simonw,2021-11-19T03:26:18Z,2021-11-19T03:29:03Z,OWNER,"A (likely incomplete) list of features on the table page:

- [ ] Display table/database/instance metadata
- [ ] Show count of all results
- [ ] Display table of results
  - [ ] Special table display treatment for URLs, numbers
  - [ ] Allow plugins to modify table cells
  - [ ] Respect `?_col=` and `?_nocol=`
- [ ] Show interface for filtering by columns and operations
- [ ] Show search box, support executing FTS searches
- [ ] Sort table by specified column
- [ ] Paginate table
- [ ] Show facet results
- [ ] Show suggested facets
- [ ] Link to available exports
- [ ] Display schema for table
  - [ ] Maybe it should show the SQL for the query too?
- [ ] Handle various non-obvious querystring options, like `?_where=` and `?_through=`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-973699424,https://api.github.com/repos/simonw/datasette/issues/1518,973699424,IC_kwDOBm6k_c46CXlg,9599,simonw,2021-11-19T03:27:49Z,2021-11-19T03:27:49Z,OWNER,"My goal is to break up a lot of this functionality into separate methods. These methods can be executed in parallel by `asyncinject`, but more importantly they can be used to build a much better JSON representation, where the default representation is lighter and `?_extra=x` options can be used to execute more expensive portions and add them to the response.

So the HTML version itself needs to be re-written to use those JSON extras.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1517#issuecomment-973696604,https://api.github.com/repos/simonw/datasette/issues/1517,973696604,IC_kwDOBm6k_c46CW5c,9599,simonw,2021-11-19T03:20:00Z,2021-11-19T03:20:00Z,OWNER,Confirmed - my test plugin is indeed correctly over-riding the table page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette,
https://github.com/simonw/datasette/issues/1518#issuecomment-973687978,https://api.github.com/repos/simonw/datasette/issues/1518,973687978,IC_kwDOBm6k_c46CUyq,9599,simonw,2021-11-19T03:07:47Z,2021-11-19T03:07:47Z,OWNER,"I was wrong about that, you CAN over-ride default routes already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1517#issuecomment-973686874,https://api.github.com/repos/simonw/datasette/issues/1517,973686874,IC_kwDOBm6k_c46CUha,9599,simonw,2021-11-19T03:06:58Z,2021-11-19T03:06:58Z,OWNER,"I made a mistake: I just wrote a test that proves that plugins CAN over-ride default routes, plus if you look at the code here the plugins get to register themselves first: https://github.com/simonw/datasette/blob/0156c6b5e52d541e93f0d68e9245f20ae83bc933/datasette/app.py#L965-L981","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057996111,Let `register_routes()` over-ride default routes within Datasette,
https://github.com/simonw/datasette/issues/1518#issuecomment-973682389,https://api.github.com/repos/simonw/datasette/issues/1518,973682389,IC_kwDOBm6k_c46CTbV,9599,simonw,2021-11-19T02:57:39Z,2021-11-19T02:57:39Z,OWNER,"Ideally I'd like to execute the existing test suite against the new implementation - that would require me to solve this so I can replace the view with the plugin version though:

- #1517 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/1518#issuecomment-973681970,https://api.github.com/repos/simonw/datasette/issues/1518,973681970,IC_kwDOBm6k_c46CTUy,9599,simonw,2021-11-19T02:56:31Z,2021-11-19T02:56:53Z,OWNER,"Here's where I got to with my hacked-together initial plugin prototype - it managed to render the table page with some rows on it (and a bunch of missing functionality such as filters): https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2

<img width=""899"" alt=""fixtures__roadside_attractions__4_rows_and__11__Liked___Twitter"" src=""https://user-images.githubusercontent.com/9599/142557265-6e10c808-5898-49ed-a8e3-7b5207b2872a.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1058072543,Complete refactor of TableView and table.html template,
https://github.com/simonw/datasette/issues/878#issuecomment-973678931,https://api.github.com/repos/simonw/datasette/issues/878,973678931,IC_kwDOBm6k_c46CSlT,9599,simonw,2021-11-19T02:51:17Z,2021-11-19T02:51:17Z,OWNER,"OK, I managed to get a table to render! Here's the code I used - I had to copy a LOT of stuff. https://gist.github.com/simonw/281eac9c73b062c3469607ad86470eb2

I'm going to move this work into a new, separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-973635157,https://api.github.com/repos/simonw/datasette/issues/878,973635157,IC_kwDOBm6k_c46CH5V,9599,simonw,2021-11-19T01:07:08Z,2021-11-19T01:07:08Z,OWNER,"This exercise is proving so useful in getting my head around how the enormous and complex `TableView` class works again.

Here's where I've got to now - I'm systematically working through the variables that are returned for HTML and for JSON copying across code to get it to work:

```python
from datasette.database import QueryInterrupted
from datasette.utils import escape_sqlite
from datasette.utils.asgi import Response, NotFound, Forbidden
from datasette.views.base import DatasetteError
from datasette import hookimpl
from asyncinject import AsyncInject, inject
from pprint import pformat


class Table(AsyncInject):
    @inject
    async def database(self, request, datasette):
        # TODO: all that nasty hash resolving stuff can go here
        db_name = request.url_vars[""db_name""]
        try:
            db = datasette.databases[db_name]
        except KeyError:
            raise NotFound(f""Database '{db_name}' does not exist"")
        return db

    @inject
    async def table_and_format(self, request, database, datasette):
        table_and_format = request.url_vars[""table_and_format""]
        # TODO: be a lot smarter here
        if ""."" in table_and_format:
            return table_and_format.split(""."", 2)
        else:
            return table_and_format, ""html""

    @inject
    async def main(self, request, database, table_and_format, datasette):
        # TODO: if this is actually a canned query, dispatch to it

        table, format = table_and_format

        is_view = bool(await database.get_view_definition(table))
        table_exists = bool(await database.table_exists(table))
        if not is_view and not table_exists:
            raise NotFound(f""Table not found: {table}"")

        await check_permissions(
            datasette,
            request,
            [
                (""view-table"", (database.name, table)),
                (""view-database"", database.name),
                ""view-instance"",
            ],
        )

        private = not await datasette.permission_allowed(
            None, ""view-table"", (database.name, table), default=True
        )

        pks = await database.primary_keys(table)
        table_columns = await database.table_columns(table)

        specified_columns = await columns_to_select(datasette, database, table, request)
        select_specified_columns = "", "".join(
            escape_sqlite(t) for t in specified_columns
        )
        select_all_columns = "", "".join(escape_sqlite(t) for t in table_columns)

        use_rowid = not pks and not is_view
        if use_rowid:
            select_specified_columns = f""rowid, {select_specified_columns}""
            select_all_columns = f""rowid, {select_all_columns}""
            order_by = ""rowid""
            order_by_pks = ""rowid""
        else:
            order_by_pks = "", "".join([escape_sqlite(pk) for pk in pks])
            order_by = order_by_pks

        if is_view:
            order_by = """"

        nocount = request.args.get(""_nocount"")
        nofacet = request.args.get(""_nofacet"")

        if request.args.get(""_shape"") in (""array"", ""object""):
            nocount = True
            nofacet = True

        # Next, a TON of SQL to build where_params and filters and suchlike
        # skipping that and jumping straight to...
        where_clauses = []
        where_clause = """"
        if where_clauses:
            where_clause = f""where {' and '.join(where_clauses)} ""

        from_sql = ""from {table_name} {where}"".format(
            table_name=escape_sqlite(table),
            where=(""where {} "".format("" and "".join(where_clauses)))
            if where_clauses
            else """",
        )
        from_sql_params ={}
        params = {}
        count_sql = f""select count(*) {from_sql}""
        sql_no_order_no_limit = (
            ""select {select_all_columns} from {table_name} {where}"".format(
                select_all_columns=select_all_columns,
                table_name=escape_sqlite(table),
                where=where_clause,
            )
        )

        page_size = 100
        offset = "" offset 0""

        sql = ""select {select_specified_columns} from {table_name} {where}{order_by} limit {page_size}{offset}"".format(
            select_specified_columns=select_specified_columns,
            table_name=escape_sqlite(table),
            where=where_clause,
            order_by=order_by,
            page_size=page_size + 1,
            offset=offset,
        )

        # Fetch rows
        results = await database.execute(sql, params, truncate=True)
        columns = [r[0] for r in results.description]
        rows = list(results.rows)

        # Fetch count
        filtered_table_rows_count = None
        if count_sql:
            try:
                count_rows = list(await database.execute(count_sql, from_sql_params))
                filtered_table_rows_count = count_rows[0][0]
            except QueryInterrupted:
                pass


        vars = {
            ""json"": {
                # THIS STUFF is from the regular JSON
                ""database"": database.name,
                ""table"": table,
                ""is_view"": is_view,
                # ""human_description_en"": human_description_en,
                ""rows"": rows[:page_size],
                ""truncated"": results.truncated,
                ""filtered_table_rows_count"": filtered_table_rows_count,
                # ""expanded_columns"": expanded_columns,
                # ""expandable_columns"": expandable_columns,
                ""columns"": columns,
                ""primary_keys"": pks,
                # ""units"": units,
                ""query"": {""sql"": sql, ""params"": params},
                # ""facet_results"": facet_results,
                # ""suggested_facets"": suggested_facets,
                # ""next"": next_value and str(next_value) or None,
                # ""next_url"": next_url,
                ""private"": private,
                ""allow_execute_sql"": await datasette.permission_allowed(
                    request.actor, ""execute-sql"", database, default=True
                ),
            },
            ""html"": {
                # ... this is the HTML special stuff
                # ""table_actions"": table_actions,
                # ""supports_search"": bool(fts_table),
                # ""search"": search or """",
                ""use_rowid"": use_rowid,
                # ""filters"": filters,
                # ""display_columns"": display_columns,
                # ""filter_columns"": filter_columns,
                # ""display_rows"": display_rows,
                # ""facets_timed_out"": facets_timed_out,
                # ""sorted_facet_results"": sorted(
                #     facet_results.values(),
                #     key=lambda f: (len(f[""results""]), f[""name""]),
                #     reverse=True,
                # ),
                # ""show_facet_counts"": special_args.get(""_facet_size"") == ""max"",
                # ""extra_wheres_for_ui"": extra_wheres_for_ui,
                # ""form_hidden_args"": form_hidden_args,
                # ""is_sortable"": any(c[""sortable""] for c in display_columns),
                # ""path_with_replaced_args"": path_with_replaced_args,
                # ""path_with_removed_args"": path_with_removed_args,
                # ""append_querystring"": append_querystring,
                ""request"": request,
                # ""sort"": sort,
                # ""sort_desc"": sort_desc,
                ""disable_sort"": is_view,
                # ""custom_table_templates"": [
                #     f""_table-{to_css_class(database)}-{to_css_class(table)}.html"",
                #     f""_table-table-{to_css_class(database)}-{to_css_class(table)}.html"",
                #     ""_table.html"",
                # ],
                # ""metadata"": metadata,
                # ""view_definition"": await db.get_view_definition(table),
                # ""table_definition"": await db.get_table_definition(table),
            },
        }

        # I'm just trying to get HTML to work for the moment
        if format == ""json"":
            return Response.json(dict(vars, locals=locals()), default=repr)
        else:
            return Response.html(repr(vars[""html""]))

    async def view(self, request, datasette):
        return await self.main(request=request, datasette=datasette)


@hookimpl
def register_routes():
    return [
        (r""/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)"", Table().view),
    ]


async def check_permissions(datasette, request, permissions):
    """"""permissions is a list of (action, resource) tuples or 'action' strings""""""
    for permission in permissions:
        if isinstance(permission, str):
            action = permission
            resource = None
        elif isinstance(permission, (tuple, list)) and len(permission) == 2:
            action, resource = permission
        else:
            assert (
                False
            ), ""permission should be string or tuple of two items: {}"".format(
                repr(permission)
            )
        ok = await datasette.permission_allowed(
            request.actor,
            action,
            resource=resource,
            default=None,
        )
        if ok is not None:
            if ok:
                return
            else:
                raise Forbidden(action)


async def columns_to_select(datasette, database, table, request):
    table_columns = await database.table_columns(table)
    pks = await database.primary_keys(table)
    columns = list(table_columns)
    if ""_col"" in request.args:
        columns = list(pks)
        _cols = request.args.getlist(""_col"")
        bad_columns = [column for column in _cols if column not in table_columns]
        if bad_columns:
            raise DatasetteError(
                ""_col={} - invalid columns"".format("", "".join(bad_columns)),
                status=400,
            )
        # De-duplicate maintaining order:
        columns.extend(dict.fromkeys(_cols))
    if ""_nocol"" in request.args:
        # Return all columns EXCEPT these
        bad_columns = [
            column
            for column in request.args.getlist(""_nocol"")
            if (column not in table_columns) or (column in pks)
        ]
        if bad_columns:
            raise DatasetteError(
                ""_nocol={} - invalid columns"".format("", "".join(bad_columns)),
                status=400,
            )
        tmp_columns = [
            column for column in columns if column not in request.args.getlist(""_nocol"")
        ]
        columns = tmp_columns
    return columns
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-973568285,https://api.github.com/repos/simonw/datasette/issues/878,973568285,IC_kwDOBm6k_c46B3kd,9599,simonw,2021-11-19T00:29:20Z,2021-11-19T00:29:20Z,OWNER,"This is working!
```python
from datasette.utils.asgi import Response
from datasette import hookimpl
import html
from asyncinject import AsyncInject, inject


class Table(AsyncInject):
    @inject
    async def database(self, request):
        return request.url_vars[""db_name""]

    @inject
    async def main(self, request, database):
        return Response.html(""Database: {}"".format(
            html.escape(database)
        ))

    async def view(self, request):
        return await self.main(request=request)


@hookimpl
def register_routes():
    return [
        (r""/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)"", Table().view),
    ]
```
This project will definitely show me if I actually like the `asyncinject` patterns or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-973564260,https://api.github.com/repos/simonw/datasette/issues/878,973564260,IC_kwDOBm6k_c46B2lk,9599,simonw,2021-11-19T00:27:06Z,2021-11-19T00:27:06Z,OWNER,"Problem: the fancy `asyncinject` stuff inteferes with the fancy Datasette thing that introspects view functions to look for what parameters they take:
```python
class Table(asyncinject.AsyncInjectAll):
    async def view(self, request):
        return Response.html(""Hello from {}"".format(
            html.escape(repr(request.url_vars))
        ))


@hookimpl
def register_routes():
    return [
        (r""/t/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)"", Table().view),
    ]
```
This failed with error: ""Table.view() takes 1 positional argument but 2 were given""

So I'm going to use `AsyncInject` and have the `view` function NOT use the `@inject` decorator.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-973554024,https://api.github.com/repos/simonw/datasette/issues/878,973554024,IC_kwDOBm6k_c46B0Fo,9599,simonw,2021-11-19T00:21:20Z,2021-11-19T00:21:20Z,OWNER,"That's annoying: it looks like plugins can't use `register_routes()` to over-ride default routes within Datasette itself. This didn't work:
```python
from datasette.utils.asgi import Response
from datasette import hookimpl
import html


async def table(request):
    return Response.html(""Hello from {}"".format(
        html.escape(repr(request.url_vars))
    ))


@hookimpl
def register_routes():
    return [
        (r""/(?P<db_name>[^/]+)/(?P<table_and_format>[^/]+?$)"", table),
    ]
```
I'll use a `/t/` prefix for the moment, but this is probably something I'll fix in Datasette itself later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-973542284,https://api.github.com/repos/simonw/datasette/issues/878,973542284,IC_kwDOBm6k_c46BxOM,9599,simonw,2021-11-19T00:16:44Z,2021-11-19T00:16:44Z,OWNER,"```
Development % cookiecutter gh:simonw/datasette-plugin
You've downloaded /Users/simon/.cookiecutters/datasette-plugin before. Is it okay to delete and re-download it? [yes]: yes
plugin_name []: table-new
description []: New implementation of TableView, see https://github.com/simonw/datasette/issues/878
hyphenated [table-new]: 
underscored [table_new]: 
github_username []: simonw
author_name []: Simon Willison
include_static_directory []:  
include_templates_directory []: 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-973527870,https://api.github.com/repos/simonw/datasette/issues/878,973527870,IC_kwDOBm6k_c46Bts-,9599,simonw,2021-11-19T00:13:43Z,2021-11-19T00:13:43Z,OWNER,"New plan: I'm going to build a brand new implementation of `TableView` starting out as a plugin, using the `register_routes()` plugin hook.

It will reuse the existing HTML template but will be a completely new Python implementation, based on `asyncinject`.

I'm going to start by just getting the table to show up on the page - then I'll add faceting, suggested facets, filters and so-on.

Bonus: I'm going to see if I can get it to work for arbitrary SQL queries too (stretch goal).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/pull/1516#issuecomment-972858458,https://api.github.com/repos/simonw/datasette/issues/1516,972858458,IC_kwDOBm6k_c45_KRa,22429695,codecov[bot],2021-11-18T13:19:01Z,2021-11-18T13:19:01Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1516](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a82c620) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1516/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1516   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4430     4430           
=======================================
  Hits         4068     4068           
  Misses        362      362           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...a82c620](https://codecov.io/gh/simonw/datasette/pull/1516?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1057340779,Bump black from 21.9b0 to 21.11b1,
https://github.com/simonw/datasette/pull/1514#issuecomment-972852184,https://api.github.com/repos/simonw/datasette/issues/1514,972852184,IC_kwDOBm6k_c45_IvY,49699333,dependabot[bot],2021-11-18T13:11:15Z,2021-11-18T13:11:15Z,CONTRIBUTOR,Superseded by #1516.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1056117435,Bump black from 21.9b0 to 21.11b0,
https://github.com/simonw/datasette/pull/1514#issuecomment-971575746,https://api.github.com/repos/simonw/datasette/issues/1514,971575746,IC_kwDOBm6k_c456RHC,22429695,codecov[bot],2021-11-17T13:18:58Z,2021-11-17T13:18:58Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1514](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b02c35a) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1514/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1514   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4430     4430           
=======================================
  Hits         4068     4068           
  Misses        362      362           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...b02c35a](https://codecov.io/gh/simonw/datasette/pull/1514?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1056117435,Bump black from 21.9b0 to 21.11b0,
https://github.com/simonw/datasette/pull/1500#issuecomment-971568829,https://api.github.com/repos/simonw/datasette/issues/1500,971568829,IC_kwDOBm6k_c456Pa9,49699333,dependabot[bot],2021-11-17T13:13:58Z,2021-11-17T13:13:58Z,CONTRIBUTOR,Superseded by #1514.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1041158024,Bump black from 21.9b0 to 21.10b0,
https://github.com/simonw/datasette/issues/878#issuecomment-971209475,https://api.github.com/repos/simonw/datasette/issues/878,971209475,IC_kwDOBm6k_c4543sD,9599,simonw,2021-11-17T05:41:42Z,2021-11-17T05:41:42Z,OWNER,"I'm going to build a brand new implementation of the `TableView` class that doesn't subclass `BaseView` at all, instead using `asyncinject`. If I'm lucky that will clean up the grungiest part of the codebase.

I can maybe even run the tests against old `TableView` and `TableView2` to check that they behave the same.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-971057553,https://api.github.com/repos/simonw/datasette/issues/878,971057553,IC_kwDOBm6k_c454SmR,9599,simonw,2021-11-17T01:40:45Z,2021-11-17T01:40:45Z,OWNER,"I shipped that code as a new library, `asyncinject`: https://pypi.org/project/asyncinject/ - I'll open a new PR to attempt to refactor `TableView` to use it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/pull/1512#issuecomment-971056169,https://api.github.com/repos/simonw/datasette/issues/1512,971056169,IC_kwDOBm6k_c454SQp,9599,simonw,2021-11-17T01:39:44Z,2021-11-17T01:39:44Z,OWNER,Closing this PR because I shipped the code in it as a separate library instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes,
https://github.com/simonw/datasette/pull/1512#issuecomment-971055677,https://api.github.com/repos/simonw/datasette/issues/1512,971055677,IC_kwDOBm6k_c454SI9,9599,simonw,2021-11-17T01:39:25Z,2021-11-17T01:39:25Z,OWNER,https://github.com/simonw/asyncinject version 0.1a0 is now live on PyPI: https://pypi.org/project/asyncinject/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes,
https://github.com/simonw/datasette/pull/1512#issuecomment-971010724,https://api.github.com/repos/simonw/datasette/issues/1512,971010724,IC_kwDOBm6k_c454HKk,9599,simonw,2021-11-17T01:12:22Z,2021-11-17T01:12:22Z,OWNER,I'm going to extract out the `asyncinject` stuff into a separate library.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes,
https://github.com/simonw/datasette/pull/1512#issuecomment-970718652,https://api.github.com/repos/simonw/datasette/issues/1512,970718652,IC_kwDOBm6k_c452_28,22429695,codecov[bot],2021-11-16T22:02:59Z,2021-11-16T23:51:48Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1512](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8f757da) into [main](https://codecov.io/gh/simonw/datasette/commit/0156c6b5e52d541e93f0d68e9245f20ae83bc933?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0156c6b) will **decrease** coverage by `2.10%`.
> The diff coverage is `36.20%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1512/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main    #1512      +/-   ##
==========================================
- Coverage   91.82%   89.72%   -2.11%     
==========================================
  Files          34       36       +2     
  Lines        4430     4604     +174     
==========================================
+ Hits         4068     4131      +63     
- Misses        362      473     +111     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/utils/vendored\_graphlib.py](https://codecov.io/gh/simonw/datasette/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3ZlbmRvcmVkX2dyYXBobGliLnB5) | `0.00% <0.00%> (ø)` | |
| [datasette/utils/asyncdi.py](https://codecov.io/gh/simonw/datasette/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzeW5jZGkucHk=) | `96.92% <96.92%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0156c6b...8f757da](https://codecov.io/gh/simonw/datasette/pull/1512?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes,
https://github.com/simonw/datasette/pull/1512#issuecomment-970861628,https://api.github.com/repos/simonw/datasette/issues/1512,970861628,IC_kwDOBm6k_c453iw8,9599,simonw,2021-11-16T23:46:07Z,2021-11-16T23:46:07Z,OWNER,"I made the changes locally and tested them with Python 3.6 like so:
```
cd /tmp
mkdir v
cd v
pipenv shell --python=python3.6
cd ~/Dropbox/Development/datasette
pip install -e '.[test]'
pytest tests/test_asyncdi.py
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes,
https://github.com/simonw/datasette/pull/1512#issuecomment-970857411,https://api.github.com/repos/simonw/datasette/issues/1512,970857411,IC_kwDOBm6k_c453hvD,9599,simonw,2021-11-16T23:43:21Z,2021-11-16T23:43:21Z,OWNER,"```
E     File ""/home/runner/work/datasette/datasette/datasette/utils/vendored_graphlib.py"", line 56
E       if (result := self._node2info.get(node)) is None:
E                  ^
E   SyntaxError: invalid syntax
```
Oh no - the vendored code I use has `:=` so doesn't work on Python 3.6! Will have to backport it more thoroughly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes,
https://github.com/simonw/datasette/issues/1513#issuecomment-970855084,https://api.github.com/repos/simonw/datasette/issues/1513,970855084,IC_kwDOBm6k_c453hKs,9599,simonw,2021-11-16T23:41:46Z,2021-11-16T23:41:46Z,OWNER,Conclusion: using a giant convoluted CTE and UNION ALL query to attempt to calculate facets at the same time as retrieving rows is a net LOSS for performance! Very surprised to see that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970853917,https://api.github.com/repos/simonw/datasette/issues/1513,970853917,IC_kwDOBm6k_c453g4d,9599,simonw,2021-11-16T23:41:01Z,2021-11-16T23:41:01Z,OWNER,"One very interesting difference between the two: on the single giant query page:

```json
{
  ""request_duration_ms"": 376.4317020000476,
  ""sum_trace_duration_ms"": 370.0828700000329,
  ""num_traces"": 5
}
```
And on the page that uses separate queries:
```json
{
  ""request_duration_ms"": 819.012272000009,
  ""sum_trace_duration_ms"": 201.52852100000018,
  ""num_traces"": 19
}
```
The separate pages page takes 819ms total to render the page, but spends 201ms across 19 SQL queries.

The single big query takes 376ms total to render the page, spending 370ms in 5 queries

<details><summary>Those 5 queries, if you're interested</summary>

```sql
select database_name, schema_version from databases
PRAGMA schema_version
PRAGMA schema_version
explain with cte as (\r\n  select rowid, date, county, state, fips, cases, deaths\r\n  from ny_times_us_counties\r\n),\r\ntruncated as (\r\n  select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n  from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n  select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n  select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n  select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet
with cte as (\r\n  select rowid, date, county, state, fips, cases, deaths\r\n  from ny_times_us_counties\r\n),\r\ntruncated as (\r\n  select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths\r\n  from cte order by date desc limit 4\r\n),\r\nstate_facet as (\r\n  select 'state' as _facet, state as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\nfips_facet as (\r\n  select 'fips' as _facet, fips as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n),\r\ncounty_facet as (\r\n  select 'county' as _facet, county as facet_name, count(*) as facet_count,\r\n  null, null, null, null, null, null, null\r\n  from cte group by facet_name order by facet_count desc limit 3\r\n)\r\nselect * from truncated\r\nunion all select * from state_facet\r\nunion all select * from fips_facet\r\nunion all select * from county_facet
```
</details>

All of that additional non-SQL overhead must be stuff relating to Python and template rendering code running on the page. I'm really surprised at how much overhead that is! This is worth researching separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970845844,https://api.github.com/repos/simonw/datasette/issues/1513,970845844,IC_kwDOBm6k_c453e6U,9599,simonw,2021-11-16T23:35:38Z,2021-11-16T23:35:38Z,OWNER,"I tried adding `cases > 10000` but the SQL query now takes too long - so moving this to my laptop.

```
cd /tmp
wget https://covid-19.datasettes.com/covid.db
datasette covid.db \
  --setting facet_time_limit_ms 10000 \
  --setting sql_time_limit_ms 10000 \
  --setting trace_debug 1
```
`http://127.0.0.1:8006/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&cases__gt=10000` shows in the traces:

```json
[
    {
      ""type"": ""sql"",
      ""start"": 12.693033525,
      ""end"": 12.694056904,
      ""duration_ms"": 1.0233789999993803,
      ""traceback"": [
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n    return await self.view_get(\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n    response_or_template_contexts = await self.data(\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 705, in data\n    results = await db.execute(sql, params, truncate=True, **extra_args)\n""
      ],
      ""database"": ""covid"",
      ""sql"": ""select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 order by rowid limit 3"",
      ""params"": {
        ""p0"": 10000
      }
    },
    {
      ""type"": ""sql"",
      ""start"": 12.694285093,
      ""end"": 12.814936275,
      ""duration_ms"": 120.65118200000136,
      ""traceback"": [
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 262, in get\n    return await self.view_get(\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/base.py\"", line 477, in view_get\n    response_or_template_contexts = await self.data(\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 723, in data\n    count_rows = list(await db.execute(count_sql, from_sql_params))\n""
      ],
      ""database"": ""covid"",
      ""sql"": ""select count(*) from ny_times_us_counties where \""cases\"" > :p0"",
      ""params"": {
        ""p0"": 10000
      }
    },
    {
      ""type"": ""sql"",
      ""start"": 12.818812089,
      ""end"": 12.851172544,
      ""duration_ms"": 32.360455000000954,
      ""traceback"": [
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n    suggested_facets.extend(await facet.suggest())\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n    distinct_values = await self.ds.execute(\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n    return await self.databases[db_name].execute(\n""
      ],
      ""database"": ""covid"",
      ""sql"": ""select county, count(*) as n from (\n                    select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n                ) where county is not null\n                group by county\n                limit 4"",
      ""params"": {
        ""p0"": 10000
      }
    },
    {
      ""type"": ""sql"",
      ""start"": 12.851418868,
      ""end"": 12.871268359,
      ""duration_ms"": 19.84949100000044,
      ""traceback"": [
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n    suggested_facets.extend(await facet.suggest())\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n    distinct_values = await self.ds.execute(\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n    return await self.databases[db_name].execute(\n""
      ],
      ""database"": ""covid"",
      ""sql"": ""select state, count(*) as n from (\n                    select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n                ) where state is not null\n                group by state\n                limit 4"",
      ""params"": {
        ""p0"": 10000
      }
    },
    {
      ""type"": ""sql"",
      ""start"": 12.871497655,
      ""end"": 12.897715027,
      ""duration_ms"": 26.217371999999628,
      ""traceback"": [
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/views/table.py\"", line 856, in data\n    suggested_facets.extend(await facet.suggest())\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/facets.py\"", line 164, in suggest\n    distinct_values = await self.ds.execute(\n"",
        ""  File \""/usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/datasette/app.py\"", line 634, in execute\n    return await self.databases[db_name].execute(\n""
      ],
      ""database"": ""covid"",
      ""sql"": ""select fips, count(*) as n from (\n                    select rowid, date, county, state, fips, cases, deaths from ny_times_us_counties where \""cases\"" > :p0 \n                ) where fips is not null\n                group by fips\n                limit 4"",
      ""params"": {
        ""p0"": 10000
      }
    }
]
```
So that's:
```
fetch rows: 1.0233789999993803 ms
count: 120.65118200000136 ms
facet county: 32.360455000000954 ms
facet state: 19.84949100000044 ms
facet fips: 26.217371999999628 ms
```
= 200.1 ms total

Compared to: `http://127.0.0.1:8006/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1`

Which is 353ms total.

The separate queries ran faster! Really surprising result there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970828568,https://api.github.com/repos/simonw/datasette/issues/1513,970828568,IC_kwDOBm6k_c453asY,9599,simonw,2021-11-16T23:27:11Z,2021-11-16T23:27:11Z,OWNER,One last experiment: I'm going to try running an expensive query in the CTE portion.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970827674,https://api.github.com/repos/simonw/datasette/issues/1513,970827674,IC_kwDOBm6k_c453aea,9599,simonw,2021-11-16T23:26:58Z,2021-11-16T23:26:58Z,OWNER,"With trace.

https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2&_trace=1 shows the following:

```
fetch rows: 0.41762600005768036 ms
facet state: 284.30423800000426 ms
facet county: 273.2565999999679 ms
facet fips: 197.80996999998024 ms
```
= 755.78843400001ms total

It didn't run a count because that's the homepage and the count is cached. So I dropped the count from the query and ran it:

https://covid-19.datasettes.com/covid?sql=with+cte+as+(%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A)%2C%0D%0Atruncated+as+(%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A)%2C%0D%0Astate_facet+as+(%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Afips_facet+as+(%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%2C%0D%0Acounty_facet+as+(%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count(*)+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A)%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet&_trace=1

Shows 649.4359889999259 ms for the query - compared to 755.78843400001ms for the separate. So it saved about 100ms.

Still not a huge difference though!



","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970780866,https://api.github.com/repos/simonw/datasette/issues/1513,970780866,IC_kwDOBm6k_c453PDC,9599,simonw,2021-11-16T23:01:57Z,2021-11-16T23:01:57Z,OWNER,"One disadvantage to this approach: if you have a SQL time limit of 1s and it takes 0.9s to return the rows but then 0.5s to calculate each of the requested facets the entire query will exceed the time limit.

Could work around this by catching that error and then re-running the query just for the rows, but that would result in the user having to wait longer for the results.

Could try to remember if that has happened using an in-memory Python data structure and skip the faceting optimization if it's caused problems in the past? That seems a bit gross.

Maybe this becomes an opt-in optimization you can request in your `metadata.json` setting for that table, which massively increases the time limit? That's a bit weird too - now there are two separate implementations of the faceting logic, which had better have a REALLY big pay-off to be worth maintaining.

What if we kept the query that returns the rows to be displayed on the page separate from the facets, but then executed all of the facets together using this method such that the `cte` only (presumably) has to be calculated once? That would still lead to multiple facets potentially exceeding the SQL time limit when single facets would not have.

Maybe a better optimization would be to move facets to happening via `fetch()` calls from the client, so the user gets to see their rows instantly and the facets then appear as and when they are available (though it would cause page jank).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970766486,https://api.github.com/repos/simonw/datasette/issues/1513,970766486,IC_kwDOBm6k_c453LiW,9599,simonw,2021-11-16T22:52:56Z,2021-11-16T22:56:07Z,OWNER,"https://covid-19.datasettes.com/covid is 805.2MB

https://covid-19.datasettes.com/covid/ny_times_us_counties?_trace=1&_facet_size=3&_size=2

Equivalent SQL:

https://covid-19.datasettes.com/covid?sql=with+cte+as+%28%0D%0A++select+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+ny_times_us_counties%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+date%2C+county%2C+state%2C+fips%2C+cases%2C+deaths%0D%0A++from+cte+order+by+date+desc+limit+4%0D%0A%29%2C%0D%0Astate_facet+as+%28%0D%0A++select+%27state%27+as+_facet%2C+state+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Afips_facet+as+%28%0D%0A++select+%27fips%27+as+_facet%2C+fips+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Acounty_facet+as+%28%0D%0A++select+%27county%27+as+_facet%2C+county+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+state_facet%0D%0Aunion+all+select+*+from+fips_facet%0D%0Aunion+all+select+*+from+county_facet%0D%0Aunion+all+select+*+from+total_count

```sql
with cte as (
  select rowid, date, county, state, fips, cases, deaths
  from ny_times_us_counties
),
truncated as (
  select null as _facet, null as facet_name, null as facet_count, rowid, date, county, state, fips, cases, deaths
  from cte order by date desc limit 4
),
state_facet as (
  select 'state' as _facet, state as facet_name, count(*) as facet_count,
  null, null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
fips_facet as (
  select 'fips' as _facet, fips as facet_name, count(*) as facet_count,
  null, null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
county_facet as (
  select 'county' as _facet, county as facet_name, count(*) as facet_count,
  null, null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
total_count as (
  select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count,
  null, null, null, null, null, null, null
  from cte
)
select * from truncated
union all select * from state_facet
union all select * from fips_facet
union all select * from county_facet
union all select * from total_count
```

_facet | facet_name | facet_count | rowid | date | county | state | fips | cases | deaths
-- | -- | -- | -- | -- | -- | -- | -- | -- | --
  |   |   | 1917344 | 2021-11-15 | Autauga | Alabama | 1001 | 10407 | 154
  |   |   | 1917345 | 2021-11-15 | Baldwin | Alabama | 1003 | 37875 | 581
  |   |   | 1917346 | 2021-11-15 | Barbour | Alabama | 1005 | 3648 | 79
  |   |   | 1917347 | 2021-11-15 | Bibb | Alabama | 1007 | 4317 | 92
state | Texas | 148028 |   |   |   |   |   |   |  
state | Georgia | 96249 |   |   |   |   |   |   |  
state | Virginia | 79315 |   |   |   |   |   |   |  
fips |   | 17580 |   |   |   |   |   |   |  
fips | 53061 | 665 |   |   |   |   |   |   |  
fips | 17031 | 662 |   |   |   |   |   |   |  
county | Washington | 18666 |   |   |   |   |   |   |  
county | Unknown | 15840 |   |   |   |   |   |   |  
county | Jefferson | 15637 |   |   |   |   |   |   |  
COUNT |   | 1920593 |   |   |   |   |   |   |  ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970770304,https://api.github.com/repos/simonw/datasette/issues/1513,970770304,IC_kwDOBm6k_c453MeA,9599,simonw,2021-11-16T22:55:19Z,2021-11-16T22:55:19Z,OWNER,(One thing I really like about this pattern is that it should work exactly the same when used to facet the results of arbitrary SQL queries as it does when faceting results from the table page.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970767952,https://api.github.com/repos/simonw/datasette/issues/1513,970767952,IC_kwDOBm6k_c453L5Q,9599,simonw,2021-11-16T22:53:52Z,2021-11-16T22:53:52Z,OWNER,It's going to take another 15 minutes for the build to finish and deploy the version with `_trace=1`: https://github.com/simonw/covid-19-datasette/actions/runs/1469150112,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970758179,https://api.github.com/repos/simonw/datasette/issues/1513,970758179,IC_kwDOBm6k_c453Jgj,9599,simonw,2021-11-16T22:47:38Z,2021-11-16T22:47:38Z,OWNER,"Trace now enabled: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1&_trace=1

Here are the relevant traces:
```json
[
    {
      ""type"": ""sql"",
      ""start"": 31.214430154,
      ""end"": 31.214817089,
      ""duration_ms"": 0.3869350000016425,
      ""traceback"": [
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 262, in get\n    return await self.view_get(\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/views/base.py\"", line 477, in view_get\n    response_or_template_contexts = await self.data(\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 705, in data\n    results = await db.execute(sql, params, truncate=True, **extra_args)\n""
      ],
      ""database"": ""global-power-plants"",
      ""sql"": ""select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] order by rowid limit 3"",
      ""params"": {}
    },
    {
      ""type"": ""sql"",
      ""start"": 31.215234586,
      ""end"": 31.220110342,
      ""duration_ms"": 4.875756000000564,
      ""traceback"": [
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n    ) = await facet.facet_results()\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n    facet_rows_results = await self.ds.execute(\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n    return await self.databases[db_name].execute(\n""
      ],
      ""database"": ""global-power-plants"",
      ""sql"": ""select country_long as value, count(*) as count from (\n                    select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n                )\n                where country_long is not null\n                group by country_long order by count desc, value limit 4"",
      ""params"": []
    },
    {
      ""type"": ""sql"",
      ""start"": 31.221062485,
      ""end"": 31.228968364,
      ""duration_ms"": 7.905878999999061,
      ""traceback"": [
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n    ) = await facet.facet_results()\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n    facet_rows_results = await self.ds.execute(\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n    return await self.databases[db_name].execute(\n""
      ],
      ""database"": ""global-power-plants"",
      ""sql"": ""select owner as value, count(*) as count from (\n                    select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n                )\n                where owner is not null\n                group by owner order by count desc, value limit 4"",
      ""params"": []
    },
    {
      ""type"": ""sql"",
      ""start"": 31.229809757,
      ""end"": 31.253902162,
      ""duration_ms"": 24.09240499999754,
      ""traceback"": [
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/views/table.py\"", line 760, in data\n    ) = await facet.facet_results()\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 212, in facet_results\n    facet_rows_results = await self.ds.execute(\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n    return await self.databases[db_name].execute(\n""
      ],
      ""database"": ""global-power-plants"",
      ""sql"": ""select primary_fuel as value, count(*) as count from (\n                    select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] \n                )\n                where primary_fuel is not null\n                group by primary_fuel order by count desc, value limit 4"",
      ""params"": []
    },
    {
      ""type"": ""sql"",
      ""start"": 31.255699745,
      ""end"": 31.256243889,
      ""duration_ms"": 0.544143999999136,
      ""traceback"": [
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 145, in suggest\n    row_count = await self.get_row_count()\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/facets.py\"", line 132, in get_row_count\n    await self.ds.execute(\n"",
        ""  File \""/usr/local/lib/python3.8/site-packages/datasette/app.py\"", line 634, in execute\n    return await self.databases[db_name].execute(\n""
      ],
      ""database"": ""global-power-plants"",
      ""sql"": ""select count(*) from (select rowid, country, country_long, name, gppd_idnr, capacity_mw, latitude, longitude, primary_fuel, other_fuel1, other_fuel2, other_fuel3, commissioning_year, owner, source, url, geolocation_source, wepp_id, year_of_capacity_data, generation_gwh_2013, generation_gwh_2014, generation_gwh_2015, generation_gwh_2016, generation_gwh_2017, generation_data_source, estimated_generation_gwh from [global-power-plants] )"",
      ""params"": []
    }
]
```
```
fetch rows: 0.3869350000016425 ms
facet country_long: 4.875756000000564 ms
facet owner: 7.905878999999061 ms
facet primary_fuel: 24.09240499999754 ms
count: 0.544143999999136 ms
```
Total = 37.8ms

I modified the query to include the total count as well: https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Atotal_count+as+%28%0D%0A++select+%27COUNT%27+as+_facet%2C+%27%27+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet%0D%0Aunion+all+select+*+from+total_count&_trace=1

```sql
with cte as (
  select rowid, country, country_long, name, owner, primary_fuel
  from [global-power-plants]
),
truncated as (
  select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel
  from cte order by rowid limit 4
),
country_long_facet as (
  select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count,
  null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
owner_facet as (
  select 'owner' as _facet, owner as facet_name, count(*) as facet_count,
  null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
primary_fuel_facet as (
  select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count,
  null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
total_count as (
  select 'COUNT' as _facet, '' as facet_name, count(*) as facet_count,
  null, null, null, null, null, null
  from cte
)
select * from truncated
union all select * from country_long_facet
union all select * from owner_facet
union all select * from primary_fuel_facet
union all select * from total_count
```
The trace says that query took 34.801436999998714 ms.

To my huge surprise, this convoluted optimization only shaves the sum query time down from 37.8ms to 34.8ms!

That entire database file is just 11.1 MB though. Maybe it would make a meaningful difference on something larger?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970742415,https://api.github.com/repos/simonw/datasette/issues/1513,970742415,IC_kwDOBm6k_c453FqP,9599,simonw,2021-11-16T22:37:14Z,2021-11-16T22:37:14Z,OWNER,"The query takes 42.794ms to run.

Here's the equivalent page using separate queries: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet_size=3&_size=2&_nocount=1

Annoyingly I can't disable facet suggestions but keep facets.

I'm going to turn on tracing so I can see how long the separate queries took.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/issues/1513#issuecomment-970738130,https://api.github.com/repos/simonw/datasette/issues/1513,970738130,IC_kwDOBm6k_c453EnS,9599,simonw,2021-11-16T22:32:19Z,2021-11-16T22:32:19Z,OWNER,"I came up with the following query which seems to work!

```sql
with cte as (
  select rowid, country, country_long, name, owner, primary_fuel
  from [global-power-plants]
),
truncated as (
  select null as _facet, null as facet_name, null as facet_count, rowid, country, country_long, name, owner, primary_fuel
  from cte order by rowid limit 4
),
country_long_facet as (
  select 'country_long' as _facet, country_long as facet_name, count(*) as facet_count,
  null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
owner_facet as (
  select 'owner' as _facet, owner as facet_name, count(*) as facet_count,
  null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
),
primary_fuel_facet as (
  select 'primary_fuel' as _facet, primary_fuel as facet_name, count(*) as facet_count,
  null, null, null, null, null, null
  from cte group by facet_name order by facet_count desc limit 3
)
select * from truncated
union all select * from country_long_facet
union all select * from owner_facet
union all select * from primary_fuel_facet
```
(Limits should be 101, 31, 31, 31 but I reduced size to get a shorter example table).

Results [look like this](https://global-power-plants.datasettes.com/global-power-plants?sql=with+cte+as+%28%0D%0A++select+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Atruncated+as+%28%0D%0A++select+null+as+_facet%2C+null+as+facet_name%2C+null+as+facet_count%2C+rowid%2C+country%2C+country_long%2C+name%2C+owner%2C+primary_fuel%0D%0A++from+cte+order+by+rowid+limit+4%0D%0A%29%2C%0D%0Acountry_long_facet+as+%28%0D%0A++select+%27country_long%27+as+_facet%2C+country_long+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aowner_facet+as+%28%0D%0A++select+%27owner%27+as+_facet%2C+owner+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%2C%0D%0Aprimary_fuel_facet+as+%28%0D%0A++select+%27primary_fuel%27+as+_facet%2C+primary_fuel+as+facet_name%2C+count%28*%29+as+facet_count%2C%0D%0A++null%2C+null%2C+null%2C+null%2C+null%2C+null%0D%0A++from+cte+group+by+facet_name+order+by+facet_count+desc+limit+3%0D%0A%29%0D%0Aselect+*+from+truncated%0D%0Aunion+all+select+*+from+country_long_facet%0D%0Aunion+all+select+*+from+owner_facet%0D%0Aunion+all+select+*+from+primary_fuel_facet):

_facet | facet_name | facet_count | rowid | country | country_long | name | owner | primary_fuel
-- | -- | -- | -- | -- | -- | -- | -- | --
  |   |   | 1 | AFG | Afghanistan | Kajaki Hydroelectric Power Plant Afghanistan |   | Hydro
  |   |   | 2 | AFG | Afghanistan | Kandahar DOG |   | Solar
  |   |   | 3 | AFG | Afghanistan | Kandahar JOL |   | Solar
  |   |   | 4 | AFG | Afghanistan | Mahipar Hydroelectric Power Plant Afghanistan |   | Hydro
country_long | United States of America | 8688 |   |   |   |   |   |  
country_long | China | 4235 |   |   |   |   |   |  
country_long | United Kingdom | 2603 |   |   |   |   |   |  
owner |   | 14112 |   |   |   |   |   |  
owner | Lightsource Renewable Energy | 120 |   |   |   |   |   |  
owner | Cypress Creek Renewables | 109 |   |   |   |   |   |  
primary_fuel | Solar | 9662 |   |   |   |   |   |  
primary_fuel | Hydro | 7155 |   |   |   |   |   |  
primary_fuel | Wind | 5188 |   |   |   |   |   |  

This is a neat proof of concept. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055469073,Research: CTEs and union all to calculate facets AND query at the same time,
https://github.com/simonw/datasette/pull/1512#issuecomment-970718337,https://api.github.com/repos/simonw/datasette/issues/1512,970718337,IC_kwDOBm6k_c452_yB,9599,simonw,2021-11-16T22:02:30Z,2021-11-16T22:02:30Z,OWNER,"I've decided to make the clever `asyncio` dependency injection opt-in - so you can either decorate with `@inject` or you can set `inject_all = True` on the class - for example:
```python
import asyncio
from datasette.utils.asyncdi import AsyncBase, inject


class Simple(AsyncBase):
    def __init__(self):
        self.log = []

    @inject
    async def two(self):
        self.log.append(""two"")

    @inject
    async def one(self, two):
        self.log.append(""one"")
        return self.log

    async def not_inject(self, one, two):
        return one + two


class Complex(AsyncBase):
    inject_all = True

    def __init__(self):
        self.log = []

    async def b(self):
        self.log.append(""b"")

    async def a(self, b):
        self.log.append(""a"")

    async def go(self, a):
        self.log.append(""go"")
        return self.log
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1055402144,New pattern for async view classes,
https://github.com/simonw/datasette/issues/878#issuecomment-970712713,https://api.github.com/repos/simonw/datasette/issues/878,970712713,IC_kwDOBm6k_c452-aJ,9599,simonw,2021-11-16T21:54:33Z,2021-11-16T21:54:33Z,OWNER,I'm going to continue working on this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-970705738,https://api.github.com/repos/simonw/datasette/issues/878,970705738,IC_kwDOBm6k_c4528tK,9599,simonw,2021-11-16T21:44:31Z,2021-11-16T21:44:31Z,OWNER,Wrote a TIL about what I learned using `TopologicalSorter`: https://til.simonwillison.net/python/graphlib-topologicalsorter,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-970673085,https://api.github.com/repos/simonw/datasette/issues/878,970673085,IC_kwDOBm6k_c4520u9,9599,simonw,2021-11-16T20:58:24Z,2021-11-16T20:58:24Z,OWNER,"New test:
```python

class Complex(AsyncBase):
    def __init__(self):
        self.log = []

    async def d(self):
        await asyncio.sleep(random() * 0.1)
        print(""LOG: d"")
        self.log.append(""d"")

    async def c(self):
        await asyncio.sleep(random() * 0.1)
        print(""LOG: c"")
        self.log.append(""c"")

    async def b(self, c, d):
        print(""LOG: b"")
        self.log.append(""b"")

    async def a(self, b, c):
        print(""LOG: a"")
        self.log.append(""a"")

    async def go(self, a):
        print(""LOG: go"")
        self.log.append(""go"")
        return self.log


@pytest.mark.asyncio
async def test_complex():
    result = await Complex().go()
    # 'c' should only be called once
    assert tuple(result) in (
        # c and d could happen in either order
        (""c"", ""d"", ""b"", ""a"", ""go""),
        (""d"", ""c"", ""b"", ""a"", ""go""),
    )
```
And this code passes it:
```python
import asyncio
from functools import wraps
import inspect

try:
    import graphlib
except ImportError:
    from . import vendored_graphlib as graphlib


class AsyncMeta(type):
    def __new__(cls, name, bases, attrs):
        # Decorate any items that are 'async def' methods
        _registry = {}
        new_attrs = {""_registry"": _registry}
        for key, value in attrs.items():
            if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"":
                new_attrs[key] = make_method(value)
                _registry[key] = new_attrs[key]
            else:
                new_attrs[key] = value
        # Gather graph for later dependency resolution
        graph = {
            key: {
                p
                for p in inspect.signature(method).parameters.keys()
                if p != ""self"" and not p.startswith(""_"")
            }
            for key, method in _registry.items()
        }
        new_attrs[""_graph""] = graph
        return super().__new__(cls, name, bases, new_attrs)


def make_method(method):
    parameters = inspect.signature(method).parameters.keys()

    @wraps(method)
    async def inner(self, _results=None, **kwargs):
        print(""\n{}.{}({}) _results={}"".format(self, method.__name__, kwargs, _results))

        # Any parameters not provided by kwargs are resolved from registry
        to_resolve = [p for p in parameters if p not in kwargs and p != ""self""]
        missing = [p for p in to_resolve if p not in self._registry]
        assert (
            not missing
        ), ""The following DI parameters could not be found in the registry: {}"".format(
            missing
        )

        results = {}
        results.update(kwargs)
        if to_resolve:
            resolved_parameters = await self.resolve(to_resolve, _results)
            results.update(resolved_parameters)
        return_value = await method(self, **results)
        if _results is not None:
            _results[method.__name__] = return_value
        return return_value

    return inner


class AsyncBase(metaclass=AsyncMeta):
    async def resolve(self, names, results=None):
        print(""\n  resolve: "", names)
        if results is None:
            results = {}

        # Come up with an execution plan, just for these nodes
        ts = graphlib.TopologicalSorter()
        to_do = set(names)
        done = set()
        while to_do:
            item = to_do.pop()
            dependencies = self._graph[item]
            ts.add(item, *dependencies)
            done.add(item)
            # Add any not-done dependencies to the queue
            to_do.update({k for k in dependencies if k not in done})

        ts.prepare()
        plan = []
        while ts.is_active():
            node_group = ts.get_ready()
            plan.append(node_group)
            ts.done(*node_group)

        print(""plan:"", plan)

        results = {}
        for node_group in plan:
            awaitables = [
                self._registry[name](
                    self,
                    _results=results,
                    **{k: v for k, v in results.items() if k in self._graph[name]},
                )
                for name in node_group
            ]
            print(""    results = "", results)
            print(""    awaitables: "", awaitables)
            awaitable_results = await asyncio.gather(*awaitables)
            results.update(
                {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)}
            )

        print(""  End of resolve(), returning"", results)
        return {key: value for key, value in results.items() if key in names}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-970660299,https://api.github.com/repos/simonw/datasette/issues/878,970660299,IC_kwDOBm6k_c452xnL,9599,simonw,2021-11-16T20:39:43Z,2021-11-16T20:42:27Z,OWNER,"But that does seem to be the plan that `TopographicalSorter` provides:
```python
graph = {""go"": {""a""}, ""a"": {""b"", ""c""}, ""b"": {""c"", ""d""}}

ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
    nodes = ts.get_ready()
    print(nodes)
    ts.done(*nodes)
```
Outputs:
```
('c', 'd')
('b',)
('a',)
('go',)
```
Also:
```python
graph = {""go"": {""d"", ""e"", ""f""}, ""d"": {""b"", ""c""}, ""b"": {""c""}}

ts = TopologicalSorter(graph)
ts.prepare()
while ts.is_active():
    nodes = ts.get_ready()
    print(nodes)
    ts.done(*nodes)
```
Gives:
```
('e', 'f', 'c')
('b',)
('d',)
('go',)
```
I'm confident that `TopologicalSorter` is the way to do this. I think I need to rewrite my code to call it once to get that plan, then `await asyncio.gather(*nodes)` in turn to execute it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-970657874,https://api.github.com/repos/simonw/datasette/issues/878,970657874,IC_kwDOBm6k_c452xBS,9599,simonw,2021-11-16T20:36:01Z,2021-11-16T20:36:01Z,OWNER,"My goal here is to calculate the most efficient way to resolve the different nodes, running them in parallel where possible.

So for this class:

```python
class Complex(AsyncBase):
    async def d(self):
        pass

    async def c(self):
        pass

    async def b(self, c, d):
        pass

    async def a(self, b, c):
        pass

    async def go(self, a):
        pass
```
A call to `go()` should do this:

- `c` and `d` in parallel
- `b`
- `a`
- `go`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-970655927,https://api.github.com/repos/simonw/datasette/issues/878,970655927,IC_kwDOBm6k_c452wi3,9599,simonw,2021-11-16T20:33:11Z,2021-11-16T20:33:11Z,OWNER,"What should be happening here instead is it should resolve the full graph and notice that `c` is depended on by both `b` and `a` - so it should run `c` first, then run the next ones in parallel.

So maybe the algorithm I'm inheriting from https://docs.python.org/3/library/graphlib.html isn't the correct algorithm?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-970655304,https://api.github.com/repos/simonw/datasette/issues/878,970655304,IC_kwDOBm6k_c452wZI,9599,simonw,2021-11-16T20:32:16Z,2021-11-16T20:32:16Z,OWNER,"This code is really fiddly. I just got to this version:
```python
import asyncio
from functools import wraps
import inspect

try:
    import graphlib
except ImportError:
    from . import vendored_graphlib as graphlib


class AsyncMeta(type):
    def __new__(cls, name, bases, attrs):
        # Decorate any items that are 'async def' methods
        _registry = {}
        new_attrs = {""_registry"": _registry}
        for key, value in attrs.items():
            if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"":
                new_attrs[key] = make_method(value)
                _registry[key] = new_attrs[key]
            else:
                new_attrs[key] = value
        # Gather graph for later dependency resolution
        graph = {
            key: {
                p
                for p in inspect.signature(method).parameters.keys()
                if p != ""self"" and not p.startswith(""_"")
            }
            for key, method in _registry.items()
        }
        new_attrs[""_graph""] = graph
        return super().__new__(cls, name, bases, new_attrs)


def make_method(method):
    @wraps(method)
    async def inner(self, _results=None, **kwargs):
        print(""inner - _results="", _results)
        parameters = inspect.signature(method).parameters.keys()
        # Any parameters not provided by kwargs are resolved from registry
        to_resolve = [p for p in parameters if p not in kwargs and p != ""self""]
        missing = [p for p in to_resolve if p not in self._registry]
        assert (
            not missing
        ), ""The following DI parameters could not be found in the registry: {}"".format(
            missing
        )
        results = {}
        results.update(kwargs)
        if to_resolve:
            resolved_parameters = await self.resolve(to_resolve, _results)
            results.update(resolved_parameters)
        return_value = await method(self, **results)
        if _results is not None:
            _results[method.__name__] = return_value
        return return_value

    return inner


class AsyncBase(metaclass=AsyncMeta):
    async def resolve(self, names, results=None):
        print(""\n  resolve: "", names)
        if results is None:
            results = {}

        # Resolve them in the correct order
        ts = graphlib.TopologicalSorter()
        for name in names:
            ts.add(name, *self._graph[name])
        ts.prepare()

        async def resolve_nodes(nodes):
            print(""    resolve_nodes"", nodes)
            print(""    (current results = {})"".format(repr(results)))
            awaitables = [
                self._registry[name](
                    self,
                    _results=results,
                    **{k: v for k, v in results.items() if k in self._graph[name]},
                )
                for name in nodes
                if name not in results
            ]
            print(""    awaitables: "", awaitables)
            awaitable_results = await asyncio.gather(*awaitables)
            results.update(
                {p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)}
            )

        if not ts.is_active():
            # Nothing has dependencies - just resolve directly
            print(""    no dependencies, resolve directly"")
            await resolve_nodes(names)
        else:
            # Resolve in topological order
            while ts.is_active():
                nodes = ts.get_ready()
                print(""    ts.get_ready() returned nodes:"", nodes)
                await resolve_nodes(nodes)
                for node in nodes:
                    ts.done(node)

        print(""  End of resolve(), returning"", results)
        return {key: value for key, value in results.items() if key in names}
```
With this test:
```python
class Complex(AsyncBase):
    def __init__(self):
        self.log = []

    async def c(self):
        print(""LOG: c"")
        self.log.append(""c"")

    async def b(self, c):
        print(""LOG: b"")
        self.log.append(""b"")

    async def a(self, b, c):
        print(""LOG: a"")
        self.log.append(""a"")

    async def go(self, a):
        print(""LOG: go"")
        self.log.append(""go"")
        return self.log


@pytest.mark.asyncio
async def test_complex():
    result = await Complex().go()
    # 'c' should only be called once
    assert result == [""c"", ""b"", ""a"", ""go""]
```
This test sometimes passes, and sometimes fails!

Output for a pass:
```
tests/test_asyncdi.py inner - _results= None

  resolve:  ['a']
    ts.get_ready() returned nodes: ('c', 'b')
    resolve_nodes ('c', 'b')
    (current results = {})
    awaitables:  [<coroutine object Complex.c at 0x1074ac890>, <coroutine object Complex.b at 0x1074ac820>]
inner - _results= {}
LOG: c
inner - _results= {'c': None}

  resolve:  ['c']
    ts.get_ready() returned nodes: ('c',)
    resolve_nodes ('c',)
    (current results = {'c': None})
    awaitables:  []
  End of resolve(), returning {'c': None}
LOG: b
    ts.get_ready() returned nodes: ('a',)
    resolve_nodes ('a',)
    (current results = {'c': None, 'b': None})
    awaitables:  [<coroutine object Complex.a at 0x1074ac7b0>]
inner - _results= {'c': None, 'b': None}
LOG: a
  End of resolve(), returning {'c': None, 'b': None, 'a': None}
LOG: go
```
Output for a fail:
```
tests/test_asyncdi.py inner - _results= None

  resolve:  ['a']
    ts.get_ready() returned nodes: ('b', 'c')
    resolve_nodes ('b', 'c')
    (current results = {})
    awaitables:  [<coroutine object Complex.b at 0x10923c890>, <coroutine object Complex.c at 0x10923c820>]
inner - _results= {}

  resolve:  ['c']
    ts.get_ready() returned nodes: ('c',)
    resolve_nodes ('c',)
    (current results = {})
    awaitables:  [<coroutine object Complex.c at 0x10923c6d0>]
inner - _results= {}
LOG: c
inner - _results= {'c': None}
LOG: c
  End of resolve(), returning {'c': None}
LOG: b
    ts.get_ready() returned nodes: ('a',)
    resolve_nodes ('a',)
    (current results = {'c': None, 'b': None})
    awaitables:  [<coroutine object Complex.a at 0x10923c6d0>]
inner - _results= {'c': None, 'b': None}
LOG: a
  End of resolve(), returning {'c': None, 'b': None, 'a': None}
LOG: go
F

=================================================================================================== FAILURES ===================================================================================================
_________________________________________________________________________________________________ test_complex _________________________________________________________________________________________________

    @pytest.mark.asyncio
    async def test_complex():
        result = await Complex().go()
        # 'c' should only be called once
>       assert result == [""c"", ""b"", ""a"", ""go""]
E       AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go']
E         At index 1 diff: 'c' != 'b'
E         Left contains one more item: 'go'
E         Use -v to get the full diff

tests/test_asyncdi.py:48: AssertionError
================== short test summary info ================================
FAILED tests/test_asyncdi.py::test_complex - AssertionError: assert ['c', 'c', 'b', 'a', 'go'] == ['c', 'b', 'a', 'go']
```
I figured out why this is happening.

`a` requires `b` and `c`

`b` also requires `c`

The code decides to run `b` and `c` in parallel.

If `c` completes first, then when `b` runs it gets to use the already-calculated result for `c` - so it doesn't need to call `c` again.

If `b` gets to that point before `c` does it also needs to call `c`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-970624197,https://api.github.com/repos/simonw/datasette/issues/878,970624197,IC_kwDOBm6k_c452ozF,9599,simonw,2021-11-16T19:49:05Z,2021-11-16T19:49:05Z,OWNER,"Here's the latest version of my weird dependency injection async class:
```python
import inspect

class AsyncMeta(type):
    def __new__(cls, name, bases, attrs):
        # Decorate any items that are 'async def' methods
        _registry = {}
        new_attrs = {""_registry"": _registry}
        for key, value in attrs.items():
            if inspect.iscoroutinefunction(value) and not value.__name__ == ""resolve"":
                new_attrs[key] = make_method(value)
                _registry[key] = new_attrs[key]
            else:
                new_attrs[key] = value

        # Topological sort of _registry by parameter dependencies
        graph = {
            key: {
                p for p in inspect.signature(method).parameters.keys()
                if p != ""self"" and not p.startswith(""_"")
            }
            for key, method in _registry.items()
        }
        new_attrs[""_graph""] = graph
        return super().__new__(cls, name, bases, new_attrs)


def make_method(method):
    @wraps(method)
    async def inner(self, **kwargs):
        parameters = inspect.signature(method).parameters.keys()
        # Any parameters not provided by kwargs are resolved from registry
        to_resolve = [p for p in parameters if p not in kwargs and p != ""self""]
        missing = [p for p in to_resolve if p not in self._registry]
        assert (
            not missing
        ), ""The following DI parameters could not be found in the registry: {}"".format(
            missing
        )
        results = {}
        results.update(kwargs)
        results.update(await self.resolve(to_resolve))
        return await method(self, **results)

    return inner


bad = [0]

class AsyncBase(metaclass=AsyncMeta):
    async def resolve(self, names):
        print("" resolve({})"".format(names))
        results = {}
        # Resolve them in the correct order
        ts = TopologicalSorter()
        ts2 = TopologicalSorter()
        print("" names = "", names)
        print("" self._graph = "", self._graph)
        for name in names:
            if self._graph[name]:
                ts.add(name, *self._graph[name])
                ts2.add(name, *self._graph[name])
        print("" static_order ="", tuple(ts2.static_order()))
        ts.prepare()
        while ts.is_active():
            print(""  is_active, i = "", bad[0])
            bad[0] += 1
            if bad[0] > 20:
                print(""    Infinite loop?"")
                break
            nodes = ts.get_ready()
            print(""  Do nodes:"", nodes)
            awaitables = [self._registry[name](self, **{
                k: v for k, v in results.items() if k in self._graph[name]
            }) for name in nodes]
            print(""  awaitables: "", awaitables)
            awaitable_results = await asyncio.gather(*awaitables)
            results.update({
                p[0].__name__: p[1] for p in zip(awaitables, awaitable_results)
            })
            print(results)
            for node in nodes:
                ts.done(node)

        return results
```
Example usage:
```python
class Foo(AsyncBase):
    async def graa(self, boff):
        print(""graa"")
        return 5
    async def boff(self):
        print(""boff"")
        return 8
    async def other(self, boff, graa):
        print(""other"")
        return 5 + boff + graa

foo = Foo()
await foo.other()
```
Output:
```
 resolve(['boff', 'graa'])
 names =  ['boff', 'graa']
 self._graph =  {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}}
 static_order = ('boff', 'graa')
  is_active, i =  0
  Do nodes: ('boff',)
  awaitables:  [<coroutine object Foo.boff at 0x10bd81a40>]
 resolve([])
 names =  []
 self._graph =  {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}}
 static_order = ()
boff
{'boff': 8}
  is_active, i =  1
  Do nodes: ('graa',)
  awaitables:  [<coroutine object Foo.graa at 0x10d66b340>]
 resolve([])
 names =  []
 self._graph =  {'graa': {'boff'}, 'boff': set(), 'other': {'graa', 'boff'}}
 static_order = ()
graa
{'boff': 8, 'graa': 5}
other
18
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/782#issuecomment-970554697,https://api.github.com/repos/simonw/datasette/issues/782,970554697,IC_kwDOBm6k_c452X1J,9599,simonw,2021-11-16T18:32:03Z,2021-11-16T18:32:03Z,OWNER,"I'm going to take another look at this:
- https://github.com/simonw/datasette/issues/878","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-970553780,https://api.github.com/repos/simonw/datasette/issues/782,970553780,IC_kwDOBm6k_c452Xm0,9599,simonw,2021-11-16T18:30:51Z,2021-11-16T18:30:58Z,OWNER,"OK, I'm ready to start working on this today.

I'm going to go with a default representation that looks like this:

```json
{
    ""rows"": [
        {""id"": 1, ""name"": ""One""},
        {""id"": 2, ""name"": ""Two""}
    ],
    ""next_url"": null
}
```
Note that there's no `count` - all it provides is the current selection of results and an indication as to how the next can be retrieved (`null` if there are no more results).

I'll implement `?_extra=` to provide everything else.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1509#issuecomment-970544733,https://api.github.com/repos/simonw/datasette/issues/1509,970544733,IC_kwDOBm6k_c452VZd,9599,simonw,2021-11-16T18:22:32Z,2021-11-16T18:22:32Z,OWNER,"This is mainly happening here:
- https://github.com/simonw/datasette/issues/782","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054243511,Datasette 1.0 JSON API (and documentation),
https://github.com/simonw/datasette/issues/1012#issuecomment-970266123,https://api.github.com/repos/simonw/datasette/issues/1012,970266123,IC_kwDOBm6k_c451RYL,45380,bollwyvl,2021-11-16T13:18:36Z,2021-11-16T13:18:36Z,CONTRIBUTOR,"Congratulations, looks like it went through! There was a bit of a hold-up
on the JupyterLab ones, but it's semi automated: a dependabot pr to
warehouse and a CI deploy, with a click in between.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1505#issuecomment-970188065,https://api.github.com/repos/simonw/datasette/issues/1505,970188065,IC_kwDOBm6k_c450-Uh,7094907,Segerberg,2021-11-16T11:40:52Z,2021-11-16T11:40:52Z,NONE,A suggestion is to have the option to choose an arbitrary delimiter (and quoting characters ),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052247023,Datasette should have an option to output CSV with semicolons,
https://github.com/simonw/datasette/issues/448#issuecomment-969621662,https://api.github.com/repos/simonw/datasette/issues/448,969621662,IC_kwDOBm6k_c45y0Ce,9599,simonw,2021-11-16T01:32:04Z,2021-11-16T01:32:04Z,OWNER,"Tests are failing and I think it's because the facets come back in different orders, need a tie-breaker. https://github.com/simonw/datasette/runs/4219325197?check_suite_focus=true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/1176#issuecomment-969616626,https://api.github.com/repos/simonw/datasette/issues/1176,969616626,IC_kwDOBm6k_c45yyzy,9599,simonw,2021-11-16T01:29:13Z,2021-11-16T01:29:13Z,OWNER,"I'm inclined to create a Sphinx reference documentation page for this, as I did for `sqlite-utils` here: https://sqlite-utils.datasette.io/en/stable/reference.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1012#issuecomment-969613166,https://api.github.com/repos/simonw/datasette/issues/1012,969613166,IC_kwDOBm6k_c45yx9u,9599,simonw,2021-11-16T01:27:25Z,2021-11-16T01:27:25Z,OWNER,"Requested here:

- https://github.com/pypa/trove-classifiers/pull/85","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1012#issuecomment-969602825,https://api.github.com/repos/simonw/datasette/issues/1012,969602825,IC_kwDOBm6k_c45yvcJ,9599,simonw,2021-11-16T01:21:14Z,2021-11-16T01:21:14Z,OWNER,"I'd been wondering how to get new classifiers into Trove - thanks, I'll give this a go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1511#issuecomment-969600859,https://api.github.com/repos/simonw/datasette/issues/1511,969600859,IC_kwDOBm6k_c45yu9b,9599,simonw,2021-11-16T01:20:13Z,2021-11-16T01:20:13Z,OWNER,"See:
- #830","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054246919,Review plugin hooks for Datasette 1.0,
https://github.com/simonw/datasette/issues/448#issuecomment-969582098,https://api.github.com/repos/simonw/datasette/issues/448,969582098,IC_kwDOBm6k_c45yqYS,9599,simonw,2021-11-16T01:10:28Z,2021-11-16T01:10:28Z,OWNER,"Also note that this demo data is using a SQL view to create the JSON arrays - the view is defined as such:

```sql
CREATE VIEW ads_with_targets as
select
  ads.*,
  json_group_array(targets.name) as target_names
from
  ads
  join ad_targets on ad_targets.ad_id = ads.id
  join targets on ad_targets.target_id = targets.id
group by
  ad_targets.ad_id;
```
So running JSON faceting on top of that view is a pretty big ask!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969578466,https://api.github.com/repos/simonw/datasette/issues/448,969578466,IC_kwDOBm6k_c45ypfi,9599,simonw,2021-11-16T01:08:29Z,2021-11-16T01:08:29Z,OWNER,Actually with the cache warmed up it looks like the facet query is taking 150ms which is good enough.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969572281,https://api.github.com/repos/simonw/datasette/issues/448,969572281,IC_kwDOBm6k_c45yn-5,9599,simonw,2021-11-16T01:05:11Z,2021-11-16T01:05:11Z,OWNER,"I tried this and it seems to work correctly:
```python
        for source_and_config in self.get_configs():
            config = source_and_config[""config""]
            source = source_and_config[""source""]
            column = config.get(""column"") or config[""simple""]
            facet_sql = """"""
                with inner as ({sql}),
                deduped_array_items as (
                    select
                        distinct j.value,
                        inner.*
                    from
                        json_each([inner].{col}) j
                        join inner
                )
                select
                    value as value,
                    count(*) as count
                from
                    deduped_array_items
                group by
                    value
                order by
                    count(*) desc limit {limit}
            """""".format(
                col=escape_sqlite(column), sql=self.sql, limit=facet_size + 1
            )
```
The queries are _very_ slow though - I had to bump up to 2s time limit even against only a view returning 3,499 rows.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969557008,https://api.github.com/repos/simonw/datasette/issues/448,969557008,IC_kwDOBm6k_c45ykQQ,9599,simonw,2021-11-16T00:56:09Z,2021-11-16T00:59:59Z,OWNER,"This looks like it might work:
```sql
with inner as (
  select
    *
  from
    ads_with_targets
  where
    :p0 in (
      select
        value
      from
        json_each([ads_with_targets].[target_names])
    )
),
deduped_array_items as (
  select
    distinct j.value,
    inner.*
  from
    json_each([inner].[target_names]) j
    join inner
)
select
  value,
  count(*)
from
  deduped_array_items
group by
  value
order by
  count(*) desc
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969557972,https://api.github.com/repos/simonw/datasette/issues/448,969557972,IC_kwDOBm6k_c45ykfU,9599,simonw,2021-11-16T00:56:58Z,2021-11-16T00:56:58Z,OWNER,It uses a CTE which were introduced in SQLite 3.8 - and AWS Lambda Python 3.9 still provides 3.7 - but I've checked and I can use `pysqlite3-binary` to work around that there so I'm OK relying on CTEs for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969449772,https://api.github.com/repos/simonw/datasette/issues/448,969449772,IC_kwDOBm6k_c45yKEs,9599,simonw,2021-11-15T23:48:37Z,2021-11-15T23:48:37Z,OWNER,"Given this query: https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++j.value+as+value%2C%0D%0A++count%28*%29+as+count%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++id%2C%0D%0A++++++file%2C%0D%0A++++++clicks%2C%0D%0A++++++impressions%2C%0D%0A++++++text%2C%0D%0A++++++url%2C%0D%0A++++++spend_amount%2C%0D%0A++++++spend_currency%2C%0D%0A++++++created%2C%0D%0A++++++ended%2C%0D%0A++++++target_names%0D%0A++++from%0D%0A++++++ads_with_targets%0D%0A++++where%0D%0A++++++%3Ap0+in+%28%0D%0A++++++++select%0D%0A++++++++++value%0D%0A++++++++from%0D%0A++++++++++json_each%28%5Bads_with_targets%5D.%5Btarget_names%5D%29%0D%0A++++++%29%0D%0A++%29%0D%0A++join+json_each%28target_names%29+j%0D%0Agroup+by%0D%0A++j.value%0D%0Aorder+by%0D%0A++count+desc%2C%0D%0A++value%0D%0Alimit%0D%0A++31&p0=people_who_match%3Ainterests%3AAfrican-American+culture

```sql
select
  j.value as value,
  count(*) as count
from
  (
    select
      id,
      file,
      clicks,
      impressions,
      text,
      url,
      spend_amount,
      spend_currency,
      created,
      ended,
      target_names
    from
      ads_with_targets
    where
      :p0 in (
        select
          value
        from
          json_each([ads_with_targets].[target_names])
      )
  )
  join json_each(target_names) j
group by
  j.value
order by
  count desc,
  value
limit
  31
```
How can I return a count of the number of documents containing each tag, but not the number of total tags that match including duplicates?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969446972,https://api.github.com/repos/simonw/datasette/issues/448,969446972,IC_kwDOBm6k_c45yJY8,9599,simonw,2021-11-15T23:46:13Z,2021-11-15T23:46:13Z,OWNER,"It looks like the problem here is that some of the tags occur more than once in the documents:

<img width=""1146"" alt=""russian-ads__ads_with_targets__172_rows_where_where_target_names_contains__people_who_match_interests_African-American_culture_"" src=""https://user-images.githubusercontent.com/9599/141870312-5ffe7151-826e-42e0-b1cc-36c46de6b618.png"">

So they get counted more than once, hence the 182 count for something that couldn't possibly return more than 172 documents.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969442215,https://api.github.com/repos/simonw/datasette/issues/448,969442215,IC_kwDOBm6k_c45yIOn,9599,simonw,2021-11-15T23:42:03Z,2021-11-15T23:42:03Z,OWNER,I think this code is wrong in the `ArrayFacet` class: https://github.com/simonw/datasette/blob/502c02fa6dde6a8bb840af6c4c8cf858aa1db687/datasette/facets.py#L357-L364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969440918,https://api.github.com/repos/simonw/datasette/issues/448,969440918,IC_kwDOBm6k_c45yH6W,9599,simonw,2021-11-15T23:40:17Z,2021-11-15T23:40:35Z,OWNER,"Applied that fix to the `arraycontains` filter but I'm still getting bad results for the faceting:

<img width=""915"" alt=""russian-ads__ads_with_targets__172_rows_where_where_target_names_contains__people_who_match_interests_African-American_culture__and_datasette_—_pipenv_shell_▸_python_—_80×24"" src=""https://user-images.githubusercontent.com/9599/141869800-0092e0e1-005e-43bc-bac5-bf0e159224a8.png"">

Should never get 182 results on a page that faceting against only 172 items.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/448#issuecomment-969436930,https://api.github.com/repos/simonw/datasette/issues/448,969436930,IC_kwDOBm6k_c45yG8C,9599,simonw,2021-11-15T23:31:58Z,2021-11-15T23:31:58Z,OWNER,"I think this SQL recipe may work instead:
```sql
select
  *
from
  ads_with_targets
where
  'people_who_match:interests:African-American Civil Rights Movement (1954—68)' in (
    select
      value
    from
      json_each(target_names)
  )
  and 'interests:Martin Luther King III' in (
    select
      value
    from
      json_each(target_names)
  )
```
https://json-view-facet-bug-demo-j7hipcg4aq-uc.a.run.app/russian-ads?sql=select%0D%0A++*%0D%0Afrom%0D%0A++ads_with_targets%0D%0Awhere%0D%0A++%27people_who_match%3Ainterests%3AAfrican-American+Civil+Rights+Movement+%281954%E2%80%9468%29%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29%0D%0A++and+%27interests%3AMartin+Luther+King+III%27+in+%28%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each%28target_names%29%0D%0A++%29&interests=&African=&Martin=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",440222719,_facet_array should work against views,
https://github.com/simonw/datasette/issues/519#issuecomment-969433734,https://api.github.com/repos/simonw/datasette/issues/519,969433734,IC_kwDOBm6k_c45yGKG,9599,simonw,2021-11-15T23:26:11Z,2021-11-15T23:26:11Z,OWNER,"I'm happy with this as the goals for 1.0. I'm going to close this issue and create three tracking tickets for the three key themes:

- https://github.com/simonw/datasette/issues/1509
- https://github.com/simonw/datasette/issues/1510
- https://github.com/simonw/datasette/issues/1511","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459590021,Decide what goes into Datasette 1.0,
https://github.com/simonw/datasette/pull/1508#issuecomment-968904414,https://api.github.com/repos/simonw/datasette/issues/1508,968904414,IC_kwDOBm6k_c45wE7e,22429695,codecov[bot],2021-11-15T13:20:49Z,2021-11-15T13:20:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1508](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (299774b) into [main](https://codecov.io/gh/simonw/datasette/commit/502c02fa6dde6a8bb840af6c4c8cf858aa1db687?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (502c02f) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1508/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1508   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4430     4430           
=======================================
  Hits         4068     4068           
  Misses        362      362           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [502c02f...299774b](https://codecov.io/gh/simonw/datasette/pull/1508?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053655062,Update docutils requirement from <0.18 to <0.19,
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968470212,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968470212,IC_kwDOCGYnMM45ua7E,9599,simonw,2021-11-15T02:49:28Z,2021-11-15T02:49:28Z,OWNER,"I was going to replace all of the `validate_column_names()` bits with something that fixed them instead, but I think I have a better idea: I'm only going to apply the fix for the various '.insert()` methods that create the initial tables.

I'll keep the `validate_column_names()` where they are at the moment. Once you've inserted the data and created the tables it will be up to you to use the new, correct column names.

This avoids the whole issue of needing to rewrite parameters, and solves the immediate problem which is consuming CSV files with bad column names.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968458837,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968458837,IC_kwDOCGYnMM45uYJV,9599,simonw,2021-11-15T02:21:15Z,2021-11-15T02:21:15Z,OWNER,"I'm not going to implement a fix that rewrites the `pk` and `column_order` and other parameters - at least not yet. The main thing I'm trying to fix here is what happens when you attempt to import a CSV file with `[ ]` in the column names, which should be unaffected by that second challenge.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968453129,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968453129,IC_kwDOCGYnMM45uWwJ,9599,simonw,2021-11-15T02:07:46Z,2021-11-15T02:07:46Z,OWNER,"If I replace `validate_column_names(row.keys())` with `fix_column_names(row)` I need to decide what to do about things like `pk=` and `column_order=`.

What should the following do?

```python
table.insert({""foo[bar]"": 4}, pk=""foo[bar]"", column_order=[""foo[bar]""])
```
Should it spot the old column names in the `pk=` and `column_order=` parameters and pretend that `foo_bar_` was passed instead?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-968451954,https://api.github.com/repos/simonw/sqlite-utils/issues/329,968451954,IC_kwDOCGYnMM45uWdy,9599,simonw,2021-11-15T02:05:29Z,2021-11-15T02:05:29Z,OWNER,"> I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them.

I'm not going to do this, it's unnecessary extra complexity and it means the function that fixes the column names needs to have access to the current `Database` instance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968450579,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968450579,IC_kwDOCGYnMM45uWIT,9599,simonw,2021-11-15T02:02:34Z,2021-11-15T02:02:34Z,OWNER,Documentation: https://github.com/simonw/sqlite-utils/blob/54a2269e91ce72b059618662ed133a85f3d42e4a/docs/python-api.rst#working-with-lookup-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record,
https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968435041,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968435041,IC_kwDOCGYnMM45uSVh,9599,simonw,2021-11-15T01:44:42Z,2021-11-15T01:44:42Z,OWNER,"`lookup(column_values, extra_values)` is one option.

`column_values` isn't actually a great name for the first parameter any more, since the second parameter also takes column values. The first parameter is now all about the unique lookup values.

Maybe this:

    lookup(lookup_values, extra_values)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record,
https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434594,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968434594,IC_kwDOCGYnMM45uSOi,9599,simonw,2021-11-15T01:43:10Z,2021-11-15T01:43:10Z,OWNER,What should I call this parameter? Django has a similar feature where it calls them `defaults=` (for `get_or_create()`) but I'm not a huge fan of that name.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record,
https://github.com/simonw/sqlite-utils/issues/339#issuecomment-968434425,https://api.github.com/repos/simonw/sqlite-utils/issues/339,968434425,IC_kwDOCGYnMM45uSL5,9599,simonw,2021-11-15T01:42:36Z,2021-11-15T01:42:36Z,OWNER,"Here's the current signature of `table.lookup()`: https://github.com/simonw/sqlite-utils/blob/9cda5b070f885a7995f0c307bcc4f45f0812994a/sqlite_utils/db.py#L2716-L2729

I'm going to add a second positional argument which can provide a dictionary of column->value to use when creating the original table and populating the initial row. If the row already exists, those columns will be ignored entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1053122092,`table.lookup()` option to populate additional columns when creating a record,
https://github.com/simonw/sqlite-utils/pull/322#issuecomment-968401459,https://api.github.com/repos/simonw/sqlite-utils/issues/322,968401459,IC_kwDOCGYnMM45uKIz,9599,simonw,2021-11-15T00:26:42Z,2021-11-15T00:26:42Z,OWNER,"This relates to the fact that dictionaries, lists and tuples get special treatment and are converted to JSON strings, using this code: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L2937-L2947

So the `COLUMN_TYPE_MAPPING` should include those  too - right now it looks like this: https://github.com/simonw/sqlite-utils/blob/e8d958109ee290cfa1b44ef7a39629bb50ab673e/sqlite_utils/db.py#L165-L188","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",979612115,Add dict type to be mapped as TEXT in sqllite,
https://github.com/simonw/sqlite-utils/pull/324#issuecomment-968384988,https://api.github.com/repos/simonw/sqlite-utils/issues/324,968384988,IC_kwDOCGYnMM45uGHc,9599,simonw,2021-11-14T23:25:16Z,2021-11-14T23:25:16Z,OWNER,"Yes this was absolutely the intention! Thanks, I wonder how often I've made that mistake in other projects?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988013247,Use python-dateutil package instead of dateutils,
https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968384005,https://api.github.com/repos/simonw/sqlite-utils/issues/331,968384005,IC_kwDOCGYnMM45uF4F,9599,simonw,2021-11-14T23:19:29Z,2021-11-14T23:20:32Z,OWNER,"Tested it like this, against a freshly built `.tar.gz` package from my development environment:
```
(w) w % mypy .                    
hello.py:1: error: Skipping analyzing ""sqlite_utils"": found module but no type hints or library stubs
hello.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
(w) w % pip install ~/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz
Processing /Users/simon/Dropbox/Development/sqlite-utils/dist/sqlite-utils-3.17.1.tar.gz
...
Successfully installed sqlite-utils-3.17.1
(w) w % mypy .                                                                        
Success: no issues found in 1 source file
```
I tested against the `.whl` too.

My `hello.py` script contained this:
```python
import sqlite_utils
from typing import cast

if __name__ == ""__main__"":
    db = sqlite_utils.Database(memory=True)
    table = cast(sqlite_utils.db.Table, db[""foo""])
    table.insert({""id"": 5})
    print(list(db.query(""select * from foo"")))
```
That `cast()` is necessary because without it you get this error:
```
(w) w % mypy .
hello.py:7: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""insert""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026794056,Mypy error: found module but no type hints or library stubs,
https://github.com/simonw/sqlite-utils/issues/331#issuecomment-968381939,https://api.github.com/repos/simonw/sqlite-utils/issues/331,968381939,IC_kwDOCGYnMM45uFXz,9599,simonw,2021-11-14T23:06:20Z,2021-11-14T23:06:20Z,OWNER,Thanks - I didn't know this was needed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026794056,Mypy error: found module but no type hints or library stubs,
https://github.com/simonw/sqlite-utils/issues/332#issuecomment-968380675,https://api.github.com/repos/simonw/sqlite-utils/issues/332,968380675,IC_kwDOCGYnMM45uFED,9599,simonw,2021-11-14T22:57:56Z,2021-11-14T22:57:56Z,OWNER,This is a great idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1028056713,`sqlite-utils memory --flatten` option to flatten nested JSON,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968380387,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968380387,IC_kwDOCGYnMM45uE_j,9599,simonw,2021-11-14T22:55:56Z,2021-11-14T22:55:56Z,OWNER,"OK, this should fix it.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968371112,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968371112,IC_kwDOCGYnMM45uCuo,9599,simonw,2021-11-14T21:57:43Z,2021-11-14T22:21:31Z,OWNER,"`create_index(..., find_unique_name=)` is good. Default to false. `index_foreign_keys` can set it to true.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361671,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361671,IC_kwDOCGYnMM45uAbH,9599,simonw,2021-11-14T20:54:53Z,2021-11-14T21:01:14Z,OWNER,"I'm leaning towards `table.create_index(columns, ignore_existing_name=True)` now.

Or `resolve_existing_name` - or `skip_existing_name`?

""ignore"" sounds like it might not create the index if the name exists, but we want to still create the index but pick a new name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362285,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968362285,IC_kwDOCGYnMM45uAkt,9599,simonw,2021-11-14T20:59:44Z,2021-11-14T20:59:44Z,OWNER,I think I'll attempt to create the index and re-try if it fails with that error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968362214,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968362214,IC_kwDOCGYnMM45uAjm,9599,simonw,2021-11-14T20:59:15Z,2021-11-14T20:59:15Z,OWNER,"How to figure out if an index name is already in use? `PRAGMA index_list(t)` requires a table name. This does it:

```sql
SELECT name 
FROM sqlite_master 
WHERE type = 'index';
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361409,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361409,IC_kwDOCGYnMM45uAXB,9599,simonw,2021-11-14T20:52:55Z,2021-11-14T20:52:55Z,OWNER,"Looking at the method signature: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1518-L1524

`if_not_exists` just adds a `IF NOT EXISTS` clause here: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1549-L1561","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968361285,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968361285,IC_kwDOCGYnMM45uAVF,9599,simonw,2021-11-14T20:51:57Z,2021-11-14T20:51:57Z,OWNER,"SQLite will happily create multiple identical indexes on a table, using more disk space each time:
```pycon
>>> import sqlite_utils
>>> db = sqlite_utils.Database(""dupes.db"")
>>> db[""t""].insert_all({""id"": i} for i in range(10000))
<Table t (id)>
# dupes.db is 98304 bytes
>>> db[""t""].create_index([""id""])
<Table t (id)>
# dupes.db is 204800 bytes
>>> db[""t""].indexes
[Index(seq=0, name='idx_t_id', unique=0, origin='c', partial=0, columns=['id'])]
>>> db[""t""].create_index([""id""], index_name=""t_idx_t_id_2"")
<Table t (id)>
# 311296 bytes
>>> db[""t""].create_index([""id""], index_name=""t_idx_t_id_3"")
<Table t (id)>
# 417792 bytes
>>> db.vacuum()
# Still 417792 bytes
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360538,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968360538,IC_kwDOCGYnMM45uAJa,9599,simonw,2021-11-14T20:46:56Z,2021-11-14T20:46:56Z,OWNER,"I'm tempted to not provide an opt-out option either: if you call `table.create_index(...)` without specifying an index name I think the tool should create the index for you, quietly picking an index name that works.

But... it feels wasteful to create an index that exactly duplicates an existing index. Would SQLite even let you do that or would it notice and NOT double the amount of disk space used for that index?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968360387,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968360387,IC_kwDOCGYnMM45uAHD,9599,simonw,2021-11-14T20:45:44Z,2021-11-14T20:45:44Z,OWNER,"What would such an option be called? Some options:

- `table.create_index([fk.column], force=True)` - not obvious what `force` means here
- `table.create_index([fk.column], ignore_existing_name=True)` - not obvious what `ignore` means here
- `table.create_index([fk.column], pick_unique_name=True)` - bit verbose

If the user doesn't pass in an explicit name it seems like their intent is ""just create me the index, I don't care what name you use"" - so actually perhaps the default behaviour here should be to pick a new unique name if that name is already in use.

Then maybe there should be an option to disable that - some options there:

- `table.create_index([fk.column], error_on_existing_index_name=True)` - too verbose
- `table.create_index([fk.column], force=False)` - not clear what `force` means

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359868,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968359868,IC_kwDOCGYnMM45t_-8,9599,simonw,2021-11-14T20:41:42Z,2021-11-14T20:41:42Z,OWNER,"The ""index idx_generators_eia860_report_date already exists"" error suggests that the problem here is actually one of an index name collision.

```python
                 table.create_index([fk.column]) 
```
This will derive a name for the index automatically from the name of the table and the name of the passed in columns: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L1536-L1539

So perhaps `.create_index()` should grow an extra option that creates the index even if the name already exists, by finding a new name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/sqlite-utils/issues/335#issuecomment-968359137,https://api.github.com/repos/simonw/sqlite-utils/issues/335,968359137,IC_kwDOCGYnMM45t_zh,9599,simonw,2021-11-14T20:37:00Z,2021-11-14T20:37:00Z,OWNER,This is strange - the code already checks that an index doesn't exist before attempting to create it: https://github.com/simonw/sqlite-utils/blob/92aa5c9c5d26b0889c8c3d97c76a908d5f8af211/sqlite_utils/db.py#L893-L902,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1042569687,sqlite-utils index-foreign-keys fails due to pre-existing index,
https://github.com/simonw/datasette/issues/1507#issuecomment-968210842,https://api.github.com/repos/simonw/datasette/issues/1507,968210842,IC_kwDOBm6k_c45tbma,9599,simonw,2021-11-14T05:41:55Z,2021-11-14T05:41:55Z,OWNER,"Here's the build with that fix: https://readthedocs.org/projects/datasette/builds/15268498/

It passed and published the docs: https://docs.datasette.io/en/latest/changelog.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release,
https://github.com/simonw/datasette/issues/1507#issuecomment-968210222,https://api.github.com/repos/simonw/datasette/issues/1507,968210222,IC_kwDOBm6k_c45tbcu,9599,simonw,2021-11-14T05:34:14Z,2021-11-14T05:34:14Z,OWNER,"Here's the new build using Python 3: https://readthedocs.org/projects/datasette/builds/15268482/

It's still broken. Here's one of many issue threads about it, this one has a workaround fix: https://github.com/readthedocs/readthedocs.org/issues/8616#issuecomment-952034858

> For future readers, the solution for this problem is to pin `docutils<0.18` in your `requirements.txt` file, and have a `.readthedocs.yaml` file with these contents:
> 
> ```
> version: 2
> 
> python:
>    install:
>    - requirements: docs/requirements.txt
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release,
https://github.com/simonw/datasette/issues/1507#issuecomment-968209957,https://api.github.com/repos/simonw/datasette/issues/1507,968209957,IC_kwDOBm6k_c45tbYl,9599,simonw,2021-11-14T05:31:07Z,2021-11-14T05:31:07Z,OWNER,"Looks like ReadTheDocs builds started failing for `latest` a few weeks ago:

<img width=""845"" alt=""Banners_and_Alerts_and_Builds___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/141668921-4dd56f14-fe16-4e7f-82c1-12e6893dddd7.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release,
https://github.com/simonw/datasette/issues/1507#issuecomment-968209731,https://api.github.com/repos/simonw/datasette/issues/1507,968209731,IC_kwDOBm6k_c45tbVD,9599,simonw,2021-11-14T05:28:41Z,2021-11-14T05:28:41Z,OWNER,"I will try adding a `.readthedocs.yml` file: https://docs.readthedocs.io/en/stable/config-file/v2.html#python-version

This might work:

```
version: 2

build:
  os: ubuntu-20.04
  tools:
    python: ""3.9""

sphinx:
   configuration: docs/conf.py
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release,
https://github.com/simonw/datasette/issues/1507#issuecomment-968209616,https://api.github.com/repos/simonw/datasette/issues/1507,968209616,IC_kwDOBm6k_c45tbTQ,9599,simonw,2021-11-14T05:27:22Z,2021-11-14T05:27:22Z,OWNER,https://blog.readthedocs.com/default-python-3/ they started defaulting new projects to Python 3 back in Feb 2019 but clearly my project was created before then.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release,
https://github.com/simonw/datasette/issues/1507#issuecomment-968209560,https://api.github.com/repos/simonw/datasette/issues/1507,968209560,IC_kwDOBm6k_c45tbSY,9599,simonw,2021-11-14T05:26:36Z,2021-11-14T05:26:36Z,OWNER,"It looks like my builds there still run on Python 2!

```
git clone --no-single-branch --depth 50 https://github.com/simonw/datasette .
git checkout --force de1e031713f47fbd51eb7239db3e7e6025fbf81a
git clean -d -f -f
python2.7 -mvirtualenv /home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2
/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir pip setuptools
/home/docs/checkouts/readthedocs.org/user_builds/datasette/envs/0.59.2/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2
cat docs/conf.py 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052851176,ReadTheDocs build failed for 0.59.2 release,
https://github.com/simonw/datasette/issues/1503#issuecomment-968207906,https://api.github.com/repos/simonw/datasette/issues/1503,968207906,IC_kwDOBm6k_c45ta4i,9599,simonw,2021-11-14T05:08:26Z,2021-11-14T05:08:26Z,OWNER,"Error:
```
    def test_table_html_filter_form_column_options(
        path, expected_column_options, app_client
    ):
        response = app_client.get(path)
        assert response.status == 200
        form = Soup(response.body, ""html.parser"").find(""form"")
        column_options = [
            o.attrs.get(""value"") or o.string
            for o in form.select(""select[name=_filter_column] option"")
        ]
>       assert expected_column_options == column_options
E       AssertionError: assert ['- column -'...wid', 'value'] == ['- column -', 'value']
E         At index 1 diff: 'rowid' != 'value'
E         Left contains one more item: 'value'
E         Use -v to get the full diff
```
This is because `rowid` isn't a table column but IS returned by the query used on that page.

My solution: start with the query columns, but then add any table columns that were not already returned by the query to the end of the `filter_columns` list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1050163432,`?_nocol=` removes that column from the filter interface,
https://github.com/simonw/datasette/issues/1506#issuecomment-968192980,https://api.github.com/repos/simonw/datasette/issues/1506,968192980,IC_kwDOBm6k_c45tXPU,9599,simonw,2021-11-14T02:22:40Z,2021-11-14T02:22:40Z,OWNER,"I think the answer is to spot this case and link to `?_item_exact=x` instead of `?_item=x` - it looks like that's already recommended in the documentation here: https://docs.datasette.io/en/stable/json_api.html#column-filter-arguments

> **?column__exact=value** or **?_column=value**
>     Returns rows where the specified column exactly matches the value.

So maybe the facet selection rendering logic needs to spot this and link correctly to it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1052826038,Columns beginning with an underscore do not facet correctly,
https://github.com/simonw/datasette/issues/1380#issuecomment-967801997,https://api.github.com/repos/simonw/datasette/issues/1380,967801997,IC_kwDOBm6k_c45r3yN,7094907,Segerberg,2021-11-13T08:05:37Z,2021-11-13T08:09:11Z,NONE,"@glasnt yeah I guess that could be an option.  I run datasette on large databases > 75gb and the startup time is a bit slow for me even with -i --inspect-file options. Here's a quick sketch for a plugin that will reload db's in a folder that you set for the plugin in metadata.json. If you request /-reload-db new db's will be added. (You probably want to implement some authentication for this =) ) 

https://gist.github.com/Segerberg/b96a0e0a5389dce2396497323cda7042
","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/datasette/issues/1380#issuecomment-967747190,https://api.github.com/repos/simonw/datasette/issues/1380,967747190,IC_kwDOBm6k_c45rqZ2,813732,glasnt,2021-11-13T00:47:26Z,2021-11-13T00:47:26Z,CONTRIBUTOR,"Would it make sense to run datasette with a fswatch/inotifywait on a folder, then? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/datasette/issues/1380#issuecomment-967181828,https://api.github.com/repos/simonw/datasette/issues/1380,967181828,IC_kwDOBm6k_c45pgYE,7094907,Segerberg,2021-11-12T15:00:18Z,2021-11-12T20:02:29Z,NONE,"There is no such option see https://github.com/simonw/datasette/issues/43. But you could write a plugin using the datasette.add_database(db, name=None) https://docs.datasette.io/en/stable/internals.html#add-database-db-name-none ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/sqlite-utils/issues/26#issuecomment-964205475,https://api.github.com/repos/simonw/sqlite-utils/issues/26,964205475,IC_kwDOCGYnMM45eJuj,536941,fgregg,2021-11-09T14:31:29Z,2021-11-09T14:31:29Z,CONTRIBUTOR,i was just reaching for a tool to do this this morning,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286,Mechanism for turning nested JSON into foreign keys / many-to-many,
https://github.com/simonw/sqlite-utils/issues/336#issuecomment-962411119,https://api.github.com/repos/simonw/sqlite-utils/issues/336,962411119,IC_kwDOCGYnMM45XTpv,9599,simonw,2021-11-06T07:21:04Z,2021-11-06T07:21:04Z,OWNER,I've never used `DEFAULT 'CURRENT_TIMESTAMP'` myself so this one should be an interesting bug to explore.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1044267332,"sqlite-util tranform --column-order mangles columns of type ""timestamp""",
https://github.com/simonw/sqlite-utils/pull/337#issuecomment-962259527,https://api.github.com/repos/simonw/sqlite-utils/issues/337,962259527,IC_kwDOCGYnMM45WupH,771193,urbas,2021-11-05T22:33:02Z,2021-11-05T22:33:02Z,NONE,"Smokes, it looks like there was a bug in click 8.0.2 (fixed in 8.0.3: https://github.com/pallets/click/issues/2089). Meaning this PR is not needed. Closing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1046271107,Default values for `--attach` and `--param` options,
https://github.com/simonw/datasette/issues/1284#issuecomment-851567204,https://api.github.com/repos/simonw/datasette/issues/1284,851567204,MDEyOklzc3VlQ29tbWVudDg1MTU2NzIwNA==,192568,mroswell,2021-05-31T15:42:10Z,2021-11-04T03:15:01Z,CONTRIBUTOR,"I very much want to make:
  https://list.SaferDisinfectants.org/disinfectants/listN 
have this URL:
 https://list.SaferDisinfectants.org/
 
I'm using only one table page on the site, with no pagination. I'm not using the home page, though when I tried to move my table to the home page as mentioned above, I failed to figure out how. 

I am using cloudflare, but I haven't figured out a forwarding or HTML re-write method of doing this, either.

Is there any way I can get a prettier list URL? I'm on Vercel.

(I have a wordpress site on the main domain on a separate host.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",845794436,Feature or Documentation Request: Individual table as home page template,
https://github.com/simonw/datasette/pull/1495#issuecomment-960420237,https://api.github.com/repos/simonw/datasette/issues/1495,960420237,IC_kwDOBm6k_c45PtmN,192568,mroswell,2021-11-04T03:12:01Z,2021-11-04T03:12:01Z,CONTRIBUTOR,"This all looks promising! I will need detailed documentation on how to upgrade datasette once it's available, and how to implement. (@fgregg example looks very straightforward on the plugin front.) 
I'll be so excited if I can get:
https://list.saferdisinfectants.org/ 
instead of
https://list.saferdisinfectants.org/disinfectants/listN


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-960295228,https://api.github.com/repos/simonw/sqlite-utils/issues/239,960295228,IC_kwDOCGYnMM45PPE8,350038,tmaier,2021-11-03T23:35:37Z,2021-11-03T23:36:50Z,NONE,"I think I only wonder how I would parse the JSON `value` within such a lambda...

My naive approach would have been `$ sqlite-utils convert demo.db statuses statuses 'return value' --multi`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-960292442,https://api.github.com/repos/simonw/sqlite-utils/issues/239,960292442,IC_kwDOCGYnMM45POZa,350038,tmaier,2021-11-03T23:28:55Z,2021-11-03T23:28:55Z,NONE,"I am super interested in this feature.

After reading the other issues you referenced, I think the right way would be to use the current extract feature and then to use `sqlite-utils convert` to extract the json object into individual columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/datasette/pull/1500#issuecomment-956225475,https://api.github.com/repos/simonw/datasette/issues/1500,956225475,IC_kwDOBm6k_c44_tfD,22429695,codecov[bot],2021-11-01T13:16:36Z,2021-11-01T13:16:36Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1500](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b7f679) into [main](https://codecov.io/gh/simonw/datasette/commit/2c31d1cd9cd3b63458ccbe391866499fa3f44978?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2c31d1c) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1500/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1500   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4426     4426           
=======================================
  Hits         4064     4064           
  Misses        362      362           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2c31d1c...1b7f679](https://codecov.io/gh/simonw/datasette/pull/1500?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1041158024,Bump black from 21.9b0 to 21.10b0,
https://github.com/simonw/sqlite-utils/issues/173#issuecomment-956041692,https://api.github.com/repos/simonw/sqlite-utils/issues/173,956041692,IC_kwDOCGYnMM44_Anc,2118708,Florents-Tselai,2021-11-01T08:42:24Z,2021-11-01T08:42:24Z,NONE,"> I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.
> 
> Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

I was thinking about this, while inserting a stream of ~40M line-delimited json docs. Wouldn't a `--total-expected` flag work ? 

That's [how tqdm does it](https://github.com/tqdm/tqdm/blob/fc69d5dcf578f7c7986fa76841a6b793f813df35/tqdm/std.py#L366)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert,
https://github.com/simonw/datasette/issues/1253#issuecomment-955384545,https://api.github.com/repos/simonw/datasette/issues/1253,955384545,IC_kwDOBm6k_c448gLh,1449512,dufferzafar,2021-10-30T16:00:42Z,2021-10-30T16:00:42Z,NONE,"Yeah, I was pressing Ctrl + Enter as well. Came here to open this issue and found out Shift + Enter works.

@simonw Any way to configure this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826064552,"Capture ""Ctrl + Enter"" or ""⌘ + Enter"" to send SQL query?",
https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955370190,https://api.github.com/repos/simonw/sqlite-utils/issues/206,955370190,IC_kwDOCGYnMM448crO,1449512,dufferzafar,2021-10-30T15:52:16Z,2021-10-30T15:52:16Z,NONE,@simonw That was working fine. It turned out that I had to use `--nl` flag since I was passing in jsonl.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",761915790,sqlite-utils should suggest --csv if JSON parsing fails,
https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955367409,https://api.github.com/repos/simonw/sqlite-utils/issues/206,955367409,IC_kwDOCGYnMM448b_x,9599,simonw,2021-10-30T15:50:39Z,2021-10-30T15:50:39Z,OWNER,"What's the error message?

Sometimes I pipe JSON through `jq` to check if it's valid:

    cat my.json | jq","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",761915790,sqlite-utils should suggest --csv if JSON parsing fails,
https://github.com/simonw/sqlite-utils/issues/206#issuecomment-955365098,https://api.github.com/repos/simonw/sqlite-utils/issues/206,955365098,IC_kwDOCGYnMM448bbq,1449512,dufferzafar,2021-10-30T15:49:19Z,2021-10-30T15:49:19Z,NONE,"@simonw Hey! JSON parsing for me is failing and I'm getting this same error, but I feel that my json is correct. How can I debug this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",761915790,sqlite-utils should suggest --csv if JSON parsing fails,
https://github.com/simonw/datasette/pull/1495#issuecomment-954384496,https://api.github.com/repos/simonw/datasette/issues/1495,954384496,IC_kwDOBm6k_c444sBw,536941,fgregg,2021-10-29T03:07:13Z,2021-10-29T03:07:13Z,CONTRIBUTOR,"okay @simonw, made the requested changes. tests are running locally. i think this is ready for you to look at again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/sqlite-utils/issues/248#issuecomment-954303095,https://api.github.com/repos/simonw/sqlite-utils/issues/248,954303095,IC_kwDOCGYnMM444YJ3,2118708,Florents-Tselai,2021-10-28T23:46:47Z,2021-10-28T23:46:47Z,NONE,@mhalle maybe you can try out #333 ? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836829560,support for Apache Arrow / parquet files I/O,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-953911245,https://api.github.com/repos/simonw/sqlite-utils/issues/242,953911245,IC_kwDOCGYnMM4424fN,25778,eyeseast,2021-10-28T14:37:55Z,2021-10-28T14:37:55Z,CONTRIBUTOR,"I've been thinking about this a bit lately, doing a project that involves moving a lot of data in and out of SQLite files, datasette and GeoJSON. This has me leaning toward the idea that something like [`datasette query`](https://github.com/simonw/datasette/issues/1356) would be a better place to do async queries.

I know there's a lot of overlap in sqlite-utils and datasette, and maybe keeping sqlite-utils synchronous would let datasette be entirely async and give a cleaner separation of implementations.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/datasette/issues/1497#issuecomment-953508979,https://api.github.com/repos/simonw/datasette/issues/1497,953508979,IC_kwDOBm6k_c441WRz,9599,simonw,2021-10-28T05:13:49Z,2021-10-28T05:13:49Z,OWNER,Wrote about this in my weeknotes: https://simonwillison.net/2021/Oct/28/weeknotes-kubernetes-web-components/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1380#issuecomment-953366110,https://api.github.com/repos/simonw/datasette/issues/1380,953366110,IC_kwDOBm6k_c440zZe,813732,glasnt,2021-10-27T22:48:55Z,2021-10-27T22:48:55Z,CONTRIBUTOR,"It looks like if the files argument is a directory, `config_dir` is set, but files in that folder are only loaded into `self.files` at the `Datasette` class initialisation. 

I tried seeing if I could get `--reload` to work, but I'm getting issues trying to use that command when specifying a directory, as the command `serve` ends up in the files list(?): 

```
datasette serve  . --reload
Error: Invalid value for '[FILES]...': Path 'serve' does not exist.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/datasette/issues/1380#issuecomment-953334718,https://api.github.com/repos/simonw/datasette/issues/1380,953334718,IC_kwDOBm6k_c440ru-,813732,glasnt,2021-10-27T21:45:04Z,2021-10-27T21:45:04Z,CONTRIBUTOR,"I am also getting this issue, using the currently most recent version of datasette

```
$ datasette --version
datasette, version 0.59.1
```

If I run `datasette` within just a folder of files, 

```
$ datasette serve .
```

Adding new files while datasette is running shows no new files, and removing files causes datasette to return 500 errors. 


```
home
Error 500
[Errno 2] No such file or directory: 'mydatabase.db'
Powered by Datasette
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924748955,Serve all db files in a folder,
https://github.com/simonw/datasette/issues/878#issuecomment-951740637,https://api.github.com/repos/simonw/datasette/issues/878,951740637,IC_kwDOBm6k_c44umjd,30934,20after4,2021-10-26T09:12:15Z,2021-10-26T09:12:15Z,NONE,"This sounds really ambitious but also really awesome.  I like the idea that basically any piece of a page could be selectively replaced.

It sort of sounds like a python asyncio version of https://github.com/observablehq/runtime","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/pull/1204#issuecomment-951731255,https://api.github.com/repos/simonw/datasette/issues/1204,951731255,IC_kwDOBm6k_c44ukQ3,30934,20after4,2021-10-26T09:01:28Z,2021-10-26T09:01:28Z,NONE,"> Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?

Why not return a data structure instead of just a template name?

I've already done some custom hacking to modify datasette but the plugin mechanism you are building here would be much cleaner than what I've built.  I'd be happy to help with testing this PR and fleshing it out further if you are still considering merging this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes,
https://github.com/simonw/datasette/issues/1497#issuecomment-950417375,https://api.github.com/repos/simonw/datasette/issues/1497,950417375,IC_kwDOBm6k_c44pjff,9599,simonw,2021-10-24T23:36:54Z,2021-10-24T23:36:54Z,OWNER,"Tried fixing this by pushing a new `latest` tag from my laptop:
```
(datasette) datasette % docker pull datasetteproject/datasette:0.59.1
0.59.1: Pulling from datasetteproject/datasette
7d63c13d9b9b: Already exists 
6ad2a11ca37b: Already exists 
e9edbe81a001: Already exists 
36629b83aba2: Already exists 
7338abefe51c: Already exists 
6b825daddc6c: Pull complete 
d7508b065a21: Pull complete 
Digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c
Status: Downloaded newer image for datasetteproject/datasette:0.59.1
docker.io/datasetteproject/datasette:0.59.1
(datasette) datasette % docker tag datasetteproject/datasette:0.59.1 datasetteproject/datasette:latest
(datasette) datasette % docker push datasetteproject/datasette:latest
The push refers to repository [docker.io/datasetteproject/datasette]
d668c99b6ff1: Layer already exists 
aa20c9013575: Layer already exists 
c97eebf2b227: Layer already exists 
284a6c64b82c: Layer already exists 
388eedeb736e: Layer already exists 
2feece0964b8: Layer already exists 
e8b689711f21: Layer already exists 
errors:
denied: requested access to the resource is denied
unauthorized: authentication required
```
So I logged in with `docker login`:
```
(datasette) datasette % docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: datasetteproject
Password: 
```
And ran the push again and it worked:
```
(datasette) datasette % docker push datasetteproject/datasette:latest
The push refers to repository [docker.io/datasetteproject/datasette]
d668c99b6ff1: Layer already exists 
aa20c9013575: Layer already exists 
c97eebf2b227: Layer already exists 
284a6c64b82c: Layer already exists 
388eedeb736e: Layer already exists 
2feece0964b8: Layer already exists 
e8b689711f21: Layer already exists 
latest: digest: sha256:dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c size: 1793
```
https://hub.docker.com/layers/datasetteproject/datasette/latest/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950416802,https://api.github.com/repos/simonw/datasette/issues/1497,950416802,IC_kwDOBm6k_c44pjWi,9599,simonw,2021-10-24T23:32:39Z,2021-10-24T23:32:39Z,OWNER,"That's because the `publish.yml` workflow ends with this, which isn't in the `push_docker_tag.yml` workflow:

https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/publish.yml#L117-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950416682,https://api.github.com/repos/simonw/datasette/issues/1497,950416682,IC_kwDOBm6k_c44pjUq,9599,simonw,2021-10-24T23:31:51Z,2021-10-24T23:31:51Z,OWNER,One catch: the `latest` tag on Docker Hub is still three months old.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950416659,https://api.github.com/repos/simonw/datasette/issues/1497,950416659,IC_kwDOBm6k_c44pjUT,9599,simonw,2021-10-24T23:31:41Z,2021-10-24T23:31:41Z,OWNER,"Published `0.59.1` as well: https://github.com/simonw/datasette/runs/3991214225?check_suite_focus=true

Result: https://hub.docker.com/layers/datasetteproject/datasette/0.59.1/images/sha256-dc134f65bec40ed4ea7049188fe1e3915b8e6c3fd999b17effe8ec24868b979c?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950416460,https://api.github.com/repos/simonw/datasette/issues/1497,950416460,IC_kwDOBm6k_c44pjRM,9599,simonw,2021-10-24T23:30:10Z,2021-10-24T23:30:10Z,OWNER,"Testing that newly published image:
```
% docker run -p 8002:8001 -v `pwd`:/mnt \
    datasetteproject/datasette:0.59 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
Unable to find image 'datasetteproject/datasette:0.59' locally
0.59: Pulling from datasetteproject/datasette
7d63c13d9b9b: Already exists 
6ad2a11ca37b: Already exists 
e9edbe81a001: Already exists 
36629b83aba2: Already exists 
7338abefe51c: Already exists 
6d71b6b88b82: Pull complete 
8c4da3c56bdc: Pull complete 
Digest: sha256:038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed
Status: Downloaded newer image for datasetteproject/datasette:0.59
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
```
and `http://localhost:8002/versions.json` returns:

```json
{
    ""python"": {
        ""version"": ""3.9.7"",
        ""full"": ""3.9.7 (default, Oct 12 2021, 02:43:43) \n[GCC 10.2.1 20210110]""
    },
    ""datasette"": {
        ""version"": ""0.59""
    },
    ""asgi"": ""3.0"",
    ""uvicorn"": ""0.15.0"",
    ""sqlite"": {
        ""version"": ""3.34.1""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950416061,https://api.github.com/repos/simonw/datasette/issues/1497,950416061,IC_kwDOBm6k_c44pjK9,9599,simonw,2021-10-24T23:27:18Z,2021-10-24T23:27:18Z,OWNER,That worked: https://hub.docker.com/layers/datasetteproject/datasette/0.59/images/sha256-038decc28e0ea84b281ecc0058fe8eba7aa99596e5a2177ff714092ad03294ed?context=explore,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950415822,https://api.github.com/repos/simonw/datasette/issues/1497,950415822,IC_kwDOBm6k_c44pjHO,9599,simonw,2021-10-24T23:25:45Z,2021-10-24T23:25:45Z,OWNER,I'm going to attempt to publish `0.59` to Docker Hub using https://github.com/simonw/datasette/blob/2c31d1cd9cd3b63458ccbe391866499fa3f44978/.github/workflows/push_docker_tag.yml - if that works I'll push `0.59.1` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950415129,https://api.github.com/repos/simonw/datasette/issues/1497,950415129,IC_kwDOBm6k_c44pi8Z,9599,simonw,2021-10-24T23:21:33Z,2021-10-24T23:21:33Z,OWNER,That fixed it! Resulting image is 249MB which is a very slight size reduction (I think previous was 259MB (uncompressed).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950413185,https://api.github.com/repos/simonw/datasette/issues/1497,950413185,IC_kwDOBm6k_c44pieB,9599,simonw,2021-10-24T23:16:25Z,2021-10-24T23:18:30Z,OWNER,"Debian stable these days is ""bullseye"" - https://www.debian.org/releases/ - which has the version of SpatiaLite that I was previously pulling in from Sid: https://packages.debian.org/bullseye/libsqlite3-mod-spatialite

So upgrading to the 3.9.7-slim-bullseye base image may help.

https://hub.docker.com/layers/python/library/python/3.9.7-slim-bullseye/images/sha256-67af5f544115124dc6d6da1d9d2815aa9825f6fd4aa6710adb0ec1725280fb89?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950412628,https://api.github.com/repos/simonw/datasette/issues/1497,950412628,IC_kwDOBm6k_c44piVU,9599,simonw,2021-10-24T23:13:20Z,2021-10-24T23:13:27Z,OWNER,"I think the root cause here is that I'm using a Debian Buster base image and then installing SpatiaLite from Debian unstable (sid) - as described in this comment: https://github.com/simonw/datasette/issues/1249#issuecomment-804309510

That's has worked fine in the past, but Sid is unstable - and this seems to be one of those instabilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950411417,https://api.github.com/repos/simonw/datasette/issues/1497,950411417,IC_kwDOBm6k_c44piCZ,9599,simonw,2021-10-24T23:06:45Z,2021-10-24T23:11:14Z,OWNER,"Same errors with `3.9.7`:
```
#5 41.46 /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
#5 41.46 dpkg: error processing package libc6:amd64 (--configure):
#5 41.46  installed libc6:amd64 package post-installation script subprocess returned error exit status 127
#5 41.47 Errors were encountered while processing:
#5 41.47  libc6:amd64
#5 41.50 E: Sub-process /usr/bin/dpkg returned an error code (1)
```
I'm suspicious of this part of the `Dockerfile`: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1-L18

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950411912,https://api.github.com/repos/simonw/datasette/issues/1497,950411912,IC_kwDOBm6k_c44piKI,9599,simonw,2021-10-24T23:09:41Z,2021-10-24T23:09:41Z,OWNER,Here that is in the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950411808,https://api.github.com/repos/simonw/datasette/issues/1497,950411808,IC_kwDOBm6k_c44piIg,9599,simonw,2021-10-24T23:08:59Z,2021-10-24T23:08:59Z,OWNER,"Looks like it's this bug, reported on the Debian mailing list: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1818037.html

No obvious workaround there though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950411320,https://api.github.com/repos/simonw/datasette/issues/1497,950411320,IC_kwDOBm6k_c44piA4,9599,simonw,2021-10-24T23:06:05Z,2021-10-24T23:06:05Z,OWNER,"Right now the base image is: https://github.com/simonw/datasette/blob/e6e44372b34414eac2f36a4c1120af4f755aa423/Dockerfile#L1

I'm going to try `python:3.9.7-slim-buster` instead: https://hub.docker.com/layers/python/library/python/3.9.7-slim-buster/images/sha256-290b95e4b379762a9bd3d72644598e0972f4e2b5442bba60592c018fadcc744d?context=explore","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950410718,https://api.github.com/repos/simonw/datasette/issues/1497,950410718,IC_kwDOBm6k_c44ph3e,9599,simonw,2021-10-24T23:02:30Z,2021-10-24T23:02:30Z,OWNER,I got the same error publishing 0.59: https://github.com/simonw/datasette/actions/runs/1343251945,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/issues/1497#issuecomment-950410554,https://api.github.com/repos/simonw/datasette/issues/1497,950410554,IC_kwDOBm6k_c44ph06,9599,simonw,2021-10-24T23:01:20Z,2021-10-24T23:01:28Z,OWNER,"I can replicate locally by running:

```
docker build -f Dockerfile \
          -t datasetteproject/datasette:0.59.1 \
          --build-arg VERSION=0.59.1 .
```
This gives me the same error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1034535001,"Publish to Docker Hub failing with ""libcrypt.so.1: cannot open shared object file""",
https://github.com/simonw/datasette/pull/1481#issuecomment-938141121,https://api.github.com/repos/simonw/datasette/issues/1481,938141121,IC_kwDOBm6k_c436uXB,22429695,codecov[bot],2021-10-07T20:42:37Z,2021-10-24T22:19:28Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1481](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (77542e7) into [main](https://codecov.io/gh/simonw/datasette/commit/63886178a649586b403966a27a45881709d2b868?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (6388617) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

> :exclamation: Current head 77542e7 differs from pull request most recent head 50005bd. Consider uploading reports for the commit 50005bd to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1481/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main    #1481      +/-   ##
==========================================
- Coverage   91.83%   91.82%   -0.02%     
==========================================
  Files          34       34              
  Lines        4421     4426       +5     
==========================================
+ Hits         4060     4064       +4     
- Misses        361      362       +1     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.36% <0.00%> (-0.14%)` | :arrow_down: |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.41% <0.00%> (ø)` | |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.56% <0.00%> (ø)` | |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (+<0.01%)` | :arrow_up: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1481/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.78% <0.00%> (+0.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [6388617...50005bd](https://codecov.io/gh/simonw/datasette/pull/1481?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1495#issuecomment-950403692,https://api.github.com/repos/simonw/datasette/issues/1495,950403692,IC_kwDOBm6k_c44pgJs,9599,simonw,2021-10-24T22:10:43Z,2021-10-24T22:10:43Z,OWNER,"To land this change we'll need a unit test that demonstrates the new capability - I suggest putting that next to this test: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/tests/test_plugins.py#L648-L659

It will also need documentation, which should be added here: https://github.com/simonw/datasette/blob/15a9d4abfff0c45dee2a9f851326e1d61b1c678c/docs/plugin_hooks.rst#register-routes-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/datasette/pull/1495#issuecomment-950403521,https://api.github.com/repos/simonw/datasette/issues/1495,950403521,IC_kwDOBm6k_c44pgHB,9599,simonw,2021-10-24T22:09:18Z,2021-10-24T22:09:18Z,OWNER,"This is a great idea - I've wanted this myself before, but never spent any time thinking about how to achieve it.

I think your design here is exactly right - an optional third item in the tuple consisting of a dictionary of options to pass to the view function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033678984,Allow routes to have extra options,
https://github.com/simonw/datasette/issues/1482#issuecomment-950402273,https://api.github.com/repos/simonw/datasette/issues/1482,950402273,IC_kwDOBm6k_c44pfzh,9599,simonw,2021-10-24T22:00:29Z,2021-10-24T22:00:29Z,OWNER,Janus 0.6.2 is out now and should have the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10,
https://github.com/simonw/datasette/issues/1401#issuecomment-950150483,https://api.github.com/repos/simonw/datasette/issues/1401,950150483,IC_kwDOBm6k_c44oiVT,418191,jaywgraves,2021-10-23T13:09:10Z,2021-10-23T13:09:10Z,CONTRIBUTOR,"I think it's because of this in `app.css` 

```
ol,
ul {
	list-style: none;
}
```

https://github.com/simonw/datasette/blame/main/datasette/static/app.css#L35-L38

You could probably reinstate that by providing your own CSS.
https://docs.datasette.io/en/0.24/custom_templates.html#custom-css-and-javascript","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",950664971,unordered list is not rendering bullet points in description_html on database page,
https://github.com/simonw/datasette/issues/1496#issuecomment-949912718,https://api.github.com/repos/simonw/datasette/issues/1496,949912718,IC_kwDOBm6k_c44noSO,9599,simonw,2021-10-22T19:38:23Z,2021-10-22T19:38:23Z,OWNER,https://docs.datasette.io/en/latest/sql_queries.html#named-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1033864602,Named parameters docs should include an example of a cast,
https://github.com/simonw/datasette/issues/1284#issuecomment-949604763,https://api.github.com/repos/simonw/datasette/issues/1284,949604763,IC_kwDOBm6k_c44mdGb,536941,fgregg,2021-10-22T12:54:34Z,2021-10-22T12:54:34Z,CONTRIBUTOR,i'm going to take a swing at this today. we'll see.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",845794436,Feature or Documentation Request: Individual table as home page template,
https://github.com/simonw/datasette/issues/1480#issuecomment-947203725,https://api.github.com/repos/simonw/datasette/issues/1480,947203725,IC_kwDOBm6k_c44dS6N,110420,ghing,2021-10-20T00:21:54Z,2021-10-20T00:21:54Z,CONTRIBUTOR,"This StackOverflow post, [sqlite - Cloud Run: Why does my instance need so much RAM?](https://stackoverflow.com/questions/59812405/cloud-run-why-does-my-instance-need-so-much-ram), points to [this section of the Cloud Run docs](https://cloud.google.com/run/docs/troubleshooting) that says:

> Note that the Cloud Run container instances run in an environment where the files written to the local filesystem count towards the available memory. This also includes any log files that are not written to /var/log/* or /dev/log.

Does datasette write any large files when starting? 

Or does the [`COPY` command in the Dockerfile](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L349) count as writing to the local filesystem?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/1480#issuecomment-947196177,https://api.github.com/repos/simonw/datasette/issues/1480,947196177,IC_kwDOBm6k_c44dRER,110420,ghing,2021-10-20T00:05:10Z,2021-10-20T00:05:10Z,CONTRIBUTOR,"I was looking through the Dockerfile-generation code to see if there was anything that would cause memory usage to be a lot during deployment. 

I noticed that the Dockerfile [runs `datasette --inspect`](https://github.com/simonw/datasette/blob/main/datasette/utils/__init__.py#L354). Is it possible that this is using a lot of memory usage?

Or would that come into play when running `gcloud builds submit`, not when it's actually deployed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/942#issuecomment-946493045,https://api.github.com/repos/simonw/datasette/issues/942,946493045,IC_kwDOBm6k_c44alZ1,8451755,kokes,2021-10-19T08:42:39Z,2021-10-19T08:42:39Z,NONE,"@simonw I know this is closed, just found this via the annotated release notes, but I wanted to note this one thing:

Not sure how widely used this is, but I've seen CSVW a couple times in the wild. It is trying to address these metadata challenges in a standardised way.

See e.g.
- https://www.w3.org/TR/tabular-data-primer/#h-documentation-columns
- https://w3c.github.io/csvw/tests/

I'm not suggesting you change the syntax you've implemented, just letting you know of this effort by W3C.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1396#issuecomment-946467547,https://api.github.com/repos/simonw/datasette/issues/1396,946467547,IC_kwDOBm6k_c44afLb,72577720,MichaelTiemannOSC,2021-10-19T08:10:26Z,2021-10-19T08:10:26Z,CONTRIBUTOR,"Now that 0.59 has excellent annotated release notes, you can re-confirm this is fixed by updating the published Docker image and checking that these fixes still work ;-)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1493#issuecomment-946360891,https://api.github.com/repos/simonw/datasette/issues/1493,946360891,IC_kwDOBm6k_c44aFI7,9599,simonw,2021-10-19T04:37:27Z,2021-10-19T04:37:27Z,OWNER,"I renamed `/:memory:` to `/_memory` in version 0.55 - https://docs.datasette.io/en/stable/changelog.html#v0-55

But... in 0.59 I stopped following HTTP redirects by default, which is why this used to work and no longer does!

So the fix is to update the Homebrew regression test to use this instead:

    datasette --get '/_memory.json?sql=select+3*5'

Thanks for catching this!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1028115674,`--get '/:memory:.json?sql=select+3*5'` error with datasette 0.59,
https://github.com/simonw/datasette/issues/1432#issuecomment-946287922,https://api.github.com/repos/simonw/datasette/issues/1432,946287922,IC_kwDOBm6k_c44ZzUy,192568,mroswell,2021-10-19T01:16:41Z,2021-10-19T01:16:41Z,CONTRIBUTOR,"Resolved, with assistance from @ashishdotme (Thank you!)

Updated requirements.txt to include:
```
datasette==0.59
datasette-publish-vercel==0.11
sqlite-utils==3.6
```

Ran:
```
$ pip3 install -r requirements.txt
```
The site is back at work! Yay!
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/issues/1432#issuecomment-946255239,https://api.github.com/repos/simonw/datasette/issues/1432,946255239,IC_kwDOBm6k_c44ZrWH,192568,mroswell,2021-10-18T23:55:25Z,2021-10-18T23:55:25Z,CONTRIBUTOR,"I am getting this when I visit my live Datasette page:
```
This Serverless Function has crashed.
Your connection is working correctly.
Vercel is working correctly.
500: INTERNAL_SERVER_ERROR
Code: FUNCTION_INVOCATION_FAILED
ID: ...
```
And in the server logs, I'm getting

```
[GET] /disinfectants/listN
19:53:14:23
module initialization error: __init__() got an unexpected keyword argument 'config'
module initialization error
__init__() got an unexpected keyword argument 'config'
```
 Which is the same error that @ashishdotme reported above.
 
 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/issues/1470#issuecomment-946097058,https://api.github.com/repos/simonw/datasette/issues/1470,946097058,IC_kwDOBm6k_c44ZEui,9599,simonw,2021-10-18T19:30:15Z,2021-10-18T19:30:15Z,OWNER,https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid is fixed now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error,
https://github.com/simonw/datasette/pull/1494#issuecomment-945763015,https://api.github.com/repos/simonw/datasette/issues/1494,945763015,IC_kwDOBm6k_c44XzLH,22429695,codecov[bot],2021-10-18T13:22:56Z,2021-10-18T13:22:56Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1494](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7a3e3c3) into [main](https://codecov.io/gh/simonw/datasette/commit/ff9ccfb0310501a3b4b4ca24d73246a8eb3e7914?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ff9ccfb) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1494/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1494   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4426     4426           
=======================================
  Hits         4064     4064           
  Misses        362      362           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ff9ccfb...7a3e3c3](https://codecov.io/gh/simonw/datasette/pull/1494?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1029100823,"Update pytest-asyncio requirement from <0.16,>=0.10 to >=0.10,<0.17",
https://github.com/simonw/datasette/issues/1432#issuecomment-945639639,https://api.github.com/repos/simonw/datasette/issues/1432,945639639,IC_kwDOBm6k_c44XVDX,5802411,ashishdotme,2021-10-18T10:44:56Z,2021-10-18T10:44:56Z,NONE,"@simonw I am getting the below issue again now, even after removing branch argument from vercel datasette plugin

module initialization error: init() got an unexpected keyword argument 'config'
module initialization error
init() got an unexpected keyword argument 'config'","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/pull/1467#issuecomment-945037884,https://api.github.com/repos/simonw/datasette/issues/1467,945037884,IC_kwDOBm6k_c44VCI8,3058200,jameslittle230,2021-10-17T02:29:06Z,2021-10-17T02:29:06Z,NONE,Yay! Thank you @simonw!!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set,
https://github.com/simonw/datasette/pull/1481#issuecomment-945020210,https://api.github.com/repos/simonw/datasette/issues/1481,945020210,IC_kwDOBm6k_c44U90y,9599,simonw,2021-10-16T23:19:51Z,2021-10-16T23:19:51Z,OWNER,"Since that Janus PR hasn't been merged yet, one temporary option for a fix would be to entirely vendor the fixed Janus - https://github.com/aio-libs/janus/blob/9e13d3fb74e2c93d7501443b370a455d1b302b1f/janus/__init__.py - since it's only a single module.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-944986367,https://api.github.com/repos/simonw/datasette/issues/1481,944986367,IC_kwDOBm6k_c44U1j_,9599,simonw,2021-10-16T19:07:38Z,2021-10-16T19:09:02Z,OWNER,This is blocking an upgrade for the Homebrew Datasette package: https://github.com/Homebrew/homebrew-core/pull/86932,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/sqlite-utils/issues/310#issuecomment-944918759,https://api.github.com/repos/simonw/sqlite-utils/issues/310,944918759,IC_kwDOCGYnMM44UlDn,22523840,rdtq,2021-10-16T13:54:56Z,2021-10-16T13:54:56Z,NONE,It would be cool if --flatten worked with `sqlite-utils memory` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964400482,`sqlite-utils insert --flatten` option to flatten nested JSON,
https://github.com/simonw/datasette/pull/1467#issuecomment-943632697,https://api.github.com/repos/simonw/datasette/issues/1467,943632697,IC_kwDOBm6k_c44PrE5,9599,simonw,2021-10-14T18:54:18Z,2021-10-14T18:54:18Z,OWNER,The test there failed because it turns out there's a whole bunch of places that set the `Access-Control-Allow-Origin` header. I'm going to close this PR and ship a fix that refactors those places to use the same code.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set,
https://github.com/simonw/datasette/pull/1467#issuecomment-943623246,https://api.github.com/repos/simonw/datasette/issues/1467,943623246,IC_kwDOBm6k_c44PoxO,9599,simonw,2021-10-14T18:42:19Z,2021-10-14T18:42:19Z,OWNER,This looks like a good fix to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991575770,Add Authorization header when CORS flag is set,
https://github.com/simonw/datasette/pull/1458#issuecomment-943620649,https://api.github.com/repos/simonw/datasette/issues/1458,943620649,IC_kwDOBm6k_c44PoIp,9599,simonw,2021-10-14T18:38:58Z,2021-10-14T18:38:58Z,OWNER,"This is a great idea, thanks.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988555009,Rework the `--static` documentation a bit,
https://github.com/simonw/datasette/pull/1489#issuecomment-943594738,https://api.github.com/repos/simonw/datasette/issues/1489,943594738,IC_kwDOBm6k_c44Phzy,49699333,dependabot[bot],2021-10-14T18:04:13Z,2021-10-14T18:04:13Z,CONTRIBUTOR,"OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. You can also ignore all major, minor, or patch releases for a dependency by adding an [`ignore` condition](https://docs.github.com/en/code-security/supply-chain-security/configuration-options-for-dependency-updates#ignore) with the desired `update_types` to your config file.

If you change your mind, just re-open this PR and I'll resolve any conflicts on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0",
https://github.com/simonw/datasette/pull/1489#issuecomment-943594735,https://api.github.com/repos/simonw/datasette/issues/1489,943594735,IC_kwDOBm6k_c44Phzv,49699333,dependabot[bot],2021-10-14T18:04:12Z,2021-10-14T18:04:12Z,CONTRIBUTOR,Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use `@dependabot reopen` if the branch has been deleted).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0",
https://github.com/simonw/datasette/pull/1489#issuecomment-943594712,https://api.github.com/repos/simonw/datasette/issues/1489,943594712,IC_kwDOBm6k_c44PhzY,9599,simonw,2021-10-14T18:04:11Z,2021-10-14T18:04:11Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1026379132,"Update pyyaml requirement from ~=5.3 to >=5.3,<7.0",
https://github.com/simonw/datasette/issues/1488#issuecomment-942782673,https://api.github.com/repos/simonw/datasette/issues/1488,942782673,IC_kwDOBm6k_c44MbjR,9599,simonw,2021-10-13T23:04:54Z,2021-10-13T23:04:54Z,OWNER,"I think this is the change in `httpx` which is causing the bug for me: https://github.com/encode/httpx/commit/ff9813e84dab56f0f3c4ef3a159a4cce8c644a91#diff-0d0cbe9ebcd03cc8c780b0407762540a082f70cc64257f2fcd588cc30f43c15cR96

<img width=""706"" alt=""Transport_API_as_plain_`request_-__response`_method____1840__·_encode_httpx_ff9813e"" src=""https://user-images.githubusercontent.com/9599/137224339-0aed171f-b351-4d67-8d6c-ad22f2301f18.png"">

Previously it was using `path` from `path, _, query = full_path.partition(b""?"")` to populate the `raw_path` key - but it changed to instead using `request.url.raw_path` which presumably implements the logic that includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'),
https://github.com/simonw/datasette/issues/1488#issuecomment-942779926,https://api.github.com/repos/simonw/datasette/issues/1488,942779926,IC_kwDOBm6k_c44Ma4W,9599,simonw,2021-10-13T22:59:05Z,2021-10-13T22:59:05Z,OWNER,This is weird - as far as I can tell `httpx` has included the query string in `raw_path` for well over a year: https://github.com/encode/httpx/commit/8e4a8a1c73f60fe5754f95b308beaa725cb8791d#diff-c9a78eb3b5f5c4fac4e5552165fbdd5320c7e3fadf9eedabcb5461393466c090R235,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'),
https://github.com/simonw/datasette/issues/1488#issuecomment-942778673,https://api.github.com/repos/simonw/datasette/issues/1488,942778673,IC_kwDOBm6k_c44Makx,9599,simonw,2021-10-13T22:55:44Z,2021-10-13T22:55:44Z,OWNER,"```
(Pdb) request.scope['path']
'/_memory.json'
(Pdb) request.scope['raw_path']
b'/_memory.json?sql=select+sqlite_version()'
```
So `raw_path` now includes the query string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'),
https://github.com/simonw/datasette/issues/1488#issuecomment-942778382,https://api.github.com/repos/simonw/datasette/issues/1488,942778382,IC_kwDOBm6k_c44MagO,9599,simonw,2021-10-13T22:55:01Z,2021-10-13T22:55:01Z,OWNER,"I think the issue is in `route_path()`:
```
> /Users/simon/Dropbox/Development/datasette/datasette/app.py(1182)route_path()
-> response = await view(request, send)
(Pdb) path
'/_memory.json?sql=select+sqlite_version()'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'),
https://github.com/simonw/datasette/issues/1488#issuecomment-942777414,https://api.github.com/repos/simonw/datasette/issues/1488,942777414,IC_kwDOBm6k_c44MaRG,9599,simonw,2021-10-13T22:52:40Z,2021-10-13T22:52:40Z,OWNER,"Upgrading to 0.20.0 gives me lots of the following errors:

    '{""ok"": false, ""error"": ""Database not found: .json?_sort=relationships"", ""status"": 404, ""title"": null}'

It looks like the full query string is now being treated as the name of the database.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025754125,Upgrade to httpx 0.20.0 (request() got an unexpected keyword argument 'allow_redirects'),
https://github.com/simonw/sqlite-utils/pull/330#issuecomment-942752844,https://api.github.com/repos/simonw/sqlite-utils/issues/330,942752844,IC_kwDOCGYnMM44MURM,22429695,codecov[bot],2021-10-13T22:00:31Z,2021-10-13T22:11:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#330](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (fc3de90) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/718a8f61bcaed39c04d5d223104056213f8c8516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (718a8f6) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/330/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main     #330   +/-   ##
=======================================
  Coverage   96.59%   96.59%           
=======================================
  Files           5        5           
  Lines        2230     2230           
=======================================
  Hits         2154     2154           
  Misses         76       76           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [718a8f6...fc3de90](https://codecov.io/gh/simonw/sqlite-utils/pull/330?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1025726600,Test against Python 3.10,
https://github.com/simonw/datasette/pull/1463#issuecomment-915229323,https://api.github.com/repos/simonw/datasette/issues/1463,915229323,IC_kwDOBm6k_c42jUqL,22429695,codecov[bot],2021-09-08T13:15:26Z,2021-10-13T21:52:23Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1463](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2dd94c5) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

> :exclamation: Current head 2dd94c5 differs from pull request most recent head 481212c. Consider uploading reports for the commit 481212c to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1463/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main    #1463      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1463/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (-0.01%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e1012e7...481212c](https://codecov.io/gh/simonw/datasette/pull/1463?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991121619,"Update beautifulsoup4 requirement from <4.10.0,>=4.8.1 to >=4.8.1,<4.11.0",
https://github.com/simonw/datasette/issues/1469#issuecomment-942725632,https://api.github.com/repos/simonw/datasette/issues/1469,942725632,IC_kwDOBm6k_c44MNoA,9599,simonw,2021-10-13T21:13:30Z,2021-10-13T21:13:30Z,OWNER,"The core problem here is treating the `?_facet=` query string parameters as the point of truth for which facets are currently enabled.

Instead, I could use a `data-` attribute on the displayed facets.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted",
https://github.com/simonw/datasette/pull/1471#issuecomment-919141156,https://api.github.com/repos/simonw/datasette/issues/1471,919141156,IC_kwDOBm6k_c42yPsk,22429695,codecov[bot],2021-09-14T13:16:29Z,2021-10-13T21:12:25Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1471](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (01b05ed) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

> :exclamation: Current head 01b05ed differs from pull request most recent head 847238a. Consider uploading reports for the commit 847238a to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1471/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main    #1471      +/-   ##
==========================================
- Coverage   91.83%   91.83%   -0.01%     
==========================================
  Files          34       34              
  Lines        4422     4421       -1     
==========================================
- Hits         4061     4060       -1     
  Misses        361      361              
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1471/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <0.00%> (-0.01%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a673a93...847238a](https://codecov.io/gh/simonw/datasette/pull/1471?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",996002181,Bump black from 21.7b0 to 21.9b0,
https://github.com/simonw/datasette/pull/1487#issuecomment-942722595,https://api.github.com/repos/simonw/datasette/issues/1487,942722595,IC_kwDOBm6k_c44MM4j,9599,simonw,2021-10-13T21:08:53Z,2021-10-13T21:08:53Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1023245060,"Added instructions for installing plugins via pipx, #1486",
https://github.com/simonw/datasette/issues/1432#issuecomment-941585767,https://api.github.com/repos/simonw/datasette/issues/1432,941585767,IC_kwDOBm6k_c44H3Vn,5802411,ashishdotme,2021-10-12T21:23:19Z,2021-10-12T21:23:19Z,NONE,"Nevermind, had to remove the branch argument in the workflow to make vercel publish work","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/dogsheep/swarm-to-sqlite/issues/12#issuecomment-941274088,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12,941274088,IC_kwDODD6af844GrPo,33631,fs111,2021-10-12T18:31:57Z,2021-10-12T18:31:57Z,NONE,I am running into the same problem. Is there any workaround?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951817328,403 when getting token,
https://github.com/simonw/datasette/issues/1432#issuecomment-941002127,https://api.github.com/repos/simonw/datasette/issues/1432,941002127,IC_kwDOBm6k_c44Fo2P,5802411,ashishdotme,2021-10-12T13:14:31Z,2021-10-12T13:14:39Z,NONE,"Any workaround for making it work with datasette-publish-vercel. Currently getting below error

module initialization error: __init__() got an unexpected keyword argument 'config'
module initialization error
__init__() got an unexpected keyword argument 'config'","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/pull/1485#issuecomment-940023938,https://api.github.com/repos/simonw/datasette/issues/1485,940023938,IC_kwDOBm6k_c44B6CC,22429695,codecov[bot],2021-10-11T13:18:16Z,2021-10-11T13:18:16Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1485](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (be9aed9) into [main](https://codecov.io/gh/simonw/datasette/commit/0d5cc20aeffa3537cfc9296d01ec24b9c6e23dcf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0d5cc20) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1485/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1485   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4422     4422           
=======================================
  Hits         4061     4061           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0d5cc20...be9aed9](https://codecov.io/gh/simonw/datasette/pull/1485?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1022688960,"Update pytest-timeout requirement from <1.5,>=1.4.2 to >=1.4.2,<2.1",
https://github.com/simonw/datasette/issues/1470#issuecomment-939386591,https://api.github.com/repos/simonw/datasette/issues/1470,939386591,IC_kwDOBm6k_c43_ebf,9599,simonw,2021-10-10T01:17:34Z,2021-10-10T01:17:34Z,OWNER,I'll open a separate issue for removing `_next=` when running a search.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error,
https://github.com/simonw/datasette/issues/1482#issuecomment-939191311,https://api.github.com/repos/simonw/datasette/issues/1482,939191311,IC_kwDOBm6k_c43-uwP,9599,simonw,2021-10-09T00:35:04Z,2021-10-09T00:35:04Z,OWNER,I think that SQLite error message difference was caused by https://github.com/python/cpython/commit/a50e28377bcf37121b55c2de70d95a5386c478f8 or related work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1021550542,Support Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939185319,https://api.github.com/repos/simonw/datasette/issues/1481,939185319,IC_kwDOBm6k_c43-tSn,9599,simonw,2021-10-09T00:04:54Z,2021-10-09T00:04:54Z,OWNER,"I applied my PR against Janus to my local copy of Datasette like so:

    pip uninstall janus
    pip install https://github.com/aio-libs/janus/archive/9e13d3fb74e2c93d7501443b370a455d1b302b1f.zip

Then I ran the Datasette tests and got a much happier pass rate.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939180313,https://api.github.com/repos/simonw/datasette/issues/1481,939180313,IC_kwDOBm6k_c43-sEZ,9599,simonw,2021-10-08T23:41:39Z,2021-10-08T23:41:39Z,OWNER,I submitted a PR to Janus with a workaround for this: https://github.com/aio-libs/janus/pull/359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939100803,https://api.github.com/repos/simonw/datasette/issues/1481,939100803,IC_kwDOBm6k_c43-YqD,9599,simonw,2021-10-08T20:33:42Z,2021-10-08T20:33:42Z,OWNER,"There's a tiny chance this could be a bug in Python 3.10 itself - I filed an issue here: https://bugs.python.org/issue45416 - in which I said:

> In Python 3.10 it is not possible to instantiate an asyncio.Condition that wraps an asyncio.Lock without raising a ""loop argument must agree with lock"" exception.
> 
> This code raises that exception:
> 
>     asyncio.Condition(asyncio.Lock())
> 
> This worked in previous Python versions.
> 
> Note that the error only occurs if an event loop is running. Here's a simple script that replicates the problem:
> 
>     import asyncio
> 
>     # This runs without an exception:
>     print(asyncio.Condition(asyncio.Lock()))
> 
>     # This does not work:
>     async def example():
>         print(asyncio.Condition(asyncio.Lock()))
> 
>     # This raises ""ValueError: loop argument must agree with lock"":
>     asyncio.run(example())","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939079727,https://api.github.com/repos/simonw/datasette/issues/1481,939079727,IC_kwDOBm6k_c43-Tgv,9599,simonw,2021-10-08T19:50:52Z,2021-10-08T19:50:52Z,OWNER,"And here's the relevant Janus code: https://github.com/aio-libs/janus/blob/d7970f8b76bcac2e087067ca4575ac845e481874/janus/__init__.py#L24-L42

```python
class Queue(Generic[T]):
    def __init__(self, maxsize: int = 0) -> None:
        self._loop = current_loop()
        self._maxsize = maxsize

        self._init(maxsize)

        self._unfinished_tasks = 0

        self._sync_mutex = threading.Lock()
        self._sync_not_empty = threading.Condition(self._sync_mutex)
        self._sync_not_full = threading.Condition(self._sync_mutex)
        self._all_tasks_done = threading.Condition(self._sync_mutex)

        self._async_mutex = asyncio.Lock()
        # ""loop argument must agree with lock"" exception is raised here:
        self._async_not_empty = asyncio.Condition(self._async_mutex)
        self._async_not_full = asyncio.Condition(self._async_mutex)
        self._finished = asyncio.Event()
        self._finished.set()
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939078872,https://api.github.com/repos/simonw/datasette/issues/1481,939078872,IC_kwDOBm6k_c43-TTY,9599,simonw,2021-10-08T19:49:08Z,2021-10-08T19:49:08Z,OWNER,"Here's the code that raises that error: https://github.com/python/cpython/blob/bb3e0c240bc60fe08d332ff5955d54197f79751c/Lib/asyncio/locks.py#L219-L234

```python
class Condition(_ContextManagerMixin, mixins._LoopBoundMixin):
    """"""Asynchronous equivalent to threading.Condition.
    This class implements condition variable objects. A condition variable
    allows one or more coroutines to wait until they are notified by another
    coroutine.
    A new Lock object is created and used as the underlying lock.
    """"""

    def __init__(self, lock=None, *, loop=mixins._marker):
        super().__init__(loop=loop)
        if lock is None:
            lock = Lock()
        elif lock._loop is not self._get_loop():
            raise ValueError(""loop argument must agree with lock"")
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939078095,https://api.github.com/repos/simonw/datasette/issues/1481,939078095,IC_kwDOBm6k_c43-THP,9599,simonw,2021-10-08T19:47:29Z,2021-10-08T19:47:29Z,OWNER,"Only mention I can find of that ""loop argument must agree with lock"" error is here - which doesn't have any tips for a workaround yet: https://giters.com/django/channels_redis/issues/278","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939076399,https://api.github.com/repos/simonw/datasette/issues/1481,939076399,IC_kwDOBm6k_c43-Ssv,9599,simonw,2021-10-08T19:43:33Z,2021-10-08T19:43:33Z,OWNER,"So maybe this is an issue with Janus? I'm using https://pypi.org/project/janus/ 0.6.1 which is the latest release, from October 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939075686,https://api.github.com/repos/simonw/datasette/issues/1481,939075686,IC_kwDOBm6k_c43-Shm,9599,simonw,2021-10-08T19:42:00Z,2021-10-08T19:42:00Z,OWNER,"Running `pytest -x --pdb` helped me see this error:

```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 122, in dispatch_request
    await self.ds.refresh_schemas()
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 344, in refresh_schemas
    await self._refresh_schemas()
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 349, in _refresh_schemas
    await init_internal_db(internal_db)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/internal_db.py"", line 5, in init_internal_db
    await db.execute_write(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 102, in execute_write
    return await self.execute_write_fn(_inner, block=block)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 113, in execute_write_fn
    reply_queue = janus.Queue()
  File ""/Users/simon/.local/share/virtualenvs/py310-Z8fTATkJ/lib/python3.10/site-packages/janus/__init__.py"", line 39, in __init__
    self._async_not_empty = asyncio.Condition(self._async_mutex)
  File ""/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/asyncio/locks.py"", line 234, in __init__
    raise ValueError(""loop argument must agree with lock"")
ValueError: loop argument must agree with lock
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/pull/1481#issuecomment-939074818,https://api.github.com/repos/simonw/datasette/issues/1481,939074818,IC_kwDOBm6k_c43-SUC,9599,simonw,2021-10-08T19:40:23Z,2021-10-08T19:40:23Z,OWNER,"Then I created myself a temporary 3.10 environment using `pipenv` like so:

    cd /tmp
    mkdir py310
    cd py310
    pipenv shell --python /Users/simon/.pyenv/versions/3.10.0/bin/python

And used that with my Datasette checkout like so:

    cd ~/.../datasette
    pip install -e '.[test]'
    pytest
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/issues/1480#issuecomment-938171377,https://api.github.com/repos/simonw/datasette/issues/1480,938171377,IC_kwDOBm6k_c4361vx,110420,ghing,2021-10-07T21:33:12Z,2021-10-07T21:33:12Z,CONTRIBUTOR,"Thanks for the reply @simonw. What services have you had better success with than Cloud Run for larger database?

Also, what about my issue description makes you think there may be a workaround?

Is there any instrumentation I could add to see at which point in the deploy the memory usage spikes? Should I be able to see this whether it's running under Docker locally, or do you suspect this is Cloud Run-specific?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/pull/1481#issuecomment-938142436,https://api.github.com/repos/simonw/datasette/issues/1481,938142436,IC_kwDOBm6k_c436urk,9599,simonw,2021-10-07T20:44:43Z,2021-10-07T20:44:43Z,OWNER,"The 3.10 tests failed a lot. Trying to run this locally:

```
/tmp % pyenv install 3.10
python-build: definition not found: 3.10

The following versions contain `3.10' in the name:
  3.10.0a6
  3.10-dev
  miniconda-3.10.1
  miniconda3-3.10.1

See all available versions with `pyenv install --list'.

If the version you need is missing, try upgrading pyenv:

  brew update && brew upgrade pyenv
```
So trying:

    brew update && brew upgrade pyenv

Then did this:

```
/tmp % brew upgrade pyenv 
==> Upgrading 1 outdated package:
pyenv 1.2.24.1 -> 2.1.0
```
This decided to upgrade everything by downloaded everything on the internet. Aah, Homebrew.

But it looks like I have `3.10.0` available to `pyenv` now.

```
/tmp % pyenv install 3.10.0
python-build: use openssl@1.1 from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.0.tar.xz...
-> https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz
Installing Python-3.10.0...
...
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1020436713,Fix compatibility with Python 3.10,
https://github.com/simonw/datasette/issues/1480#issuecomment-938134038,https://api.github.com/repos/simonw/datasette/issues/1480,938134038,IC_kwDOBm6k_c436soW,9599,simonw,2021-10-07T20:31:46Z,2021-10-07T20:31:46Z,OWNER,"I've had this problem too - my solution was to not use Cloud Run for databases larger than about 2GB, but the way you describe it here makes me think that maybe there is a workaround here which could get it to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1015646369,Exceeding Cloud Run memory limits when deploying a 4.8G database,
https://github.com/simonw/datasette/issues/1470#issuecomment-938131806,https://api.github.com/repos/simonw/datasette/issues/1470,938131806,IC_kwDOBm6k_c436sFe,9599,simonw,2021-10-07T20:28:30Z,2021-10-07T20:28:30Z,OWNER,"On further investigation this isn't related to `_search` at all - it happens when you explicitly sort by `_sort=rowid` and apply a `_next`

- https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200 works without an error (currently)
- https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_next=200&_sort=rowid shows that error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error,
https://github.com/simonw/datasette/issues/1470#issuecomment-938124652,https://api.github.com/repos/simonw/datasette/issues/1470,938124652,IC_kwDOBm6k_c436qVs,9599,simonw,2021-10-07T20:17:53Z,2021-10-07T20:18:55Z,OWNER,"Here's the exception:
```
-> params[f""p{len(params)}""] = components[0]
(Pdb) list
603  	
604  	                # Figure out the SQL for next-based-on-primary-key first
605  	                next_by_pk_clauses = []
606  	                if use_rowid:
607  	                    next_by_pk_clauses.append(f""rowid > :p{len(params)}"")
608  ->	                    params[f""p{len(params)}""] = components[0]
609  	                else:
610  	                    # Apply the tie-breaker based on primary keys
611  	                    if len(components) == len(pks):
612  	                        param_len = len(params)
613  	                        next_by_pk_clauses.append(
```
Debugger shows that `components` is an empty array, so `components[0]` cannot be resolved:

```
-> params[f""p{len(params)}""] = components[0]
(Pdb) params
{'search': 'hello'}
(Pdb) components
[]
```

So the bug is in this code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/views/table.py#L604-L617
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",995098231,?_sort=rowid with _next= returns error,
https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934372104,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3,934372104,IC_kwDOD079W843sWMI,41546558,RhetTbull,2021-10-05T12:38:24Z,2021-10-05T12:38:24Z,CONTRIBUTOR,"As dogsheep-photos already uses [osxphotos](https://github.com/RhetTbull/osxphotos) to load photos you can access the EXIF data via osxphotos. Apple Photos imports a small subset of EXIF data at the time the photo is imported and osxphotos provides this via the [exif_info](https://github.com/RhetTbull/osxphotos#exifinfo) property. If you want the full EXIF data, osxphotos also provides a wrapper around [exiftool](https://github.com/RhetTbull/osxphotos#exiftool).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533481,"Import EXIF data into SQLite - lens used, ISO, aperture etc",
https://github.com/dogsheep/dogsheep-photos/issues/3#issuecomment-934207940,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/3,934207940,IC_kwDOD079W843ruHE,1751612,jratike80,2021-10-05T08:57:41Z,2021-10-05T08:57:41Z,NONE,"Maybe the exif-loader from the SpatiaLite project could be useful as a reference even it is written in C and it also saves images as blobs https://www.gaia-gis.it/fossil/spatialite-tools/file?name=exif_loader.c&ci=tip. The tool is also integrated into the spatialite-gui application.

I found some user documentation from the web archive http://web.archive.org/web/20180629041238/https://www.gaia-gis.it/spatialite-2.3.1/spatialite-exif-2.3.1.html.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602533481,"Import EXIF data into SQLite - lens used, ISO, aperture etc",
https://github.com/simonw/datasette/issues/1479#issuecomment-932808216,https://api.github.com/repos/simonw/datasette/issues/1479,932808216,IC_kwDOBm6k_c43mYYY,9599,simonw,2021-10-02T19:25:09Z,2021-10-02T19:25:09Z,OWNER,"Actually no, from that stack trace you provided:
```
  File ""c:\users\grott\anaconda3\lib\site-packages\click\core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py"", line 283, in package
    call(args)
  File ""c:\users\grott\anaconda3\lib\contextlib.py"", line 119, in __exit__
    next(self.gen)
  File ""c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py"", line 451, in temporary_docker_directory
    tmp.cleanup()
```
It looks like the problem occurs here: https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L449-L452","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish",
https://github.com/simonw/datasette/issues/1479#issuecomment-932808043,https://api.github.com/repos/simonw/datasette/issues/1479,932808043,IC_kwDOBm6k_c43mYVr,9599,simonw,2021-10-02T19:23:52Z,2021-10-02T19:23:52Z,OWNER,I suspect the root cause of this may be in this code: https://github.com/simonw/datasette/blob/63886178a649586b403966a27a45881709d2b868/datasette/utils/__init__.py#L673-L677,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish",
https://github.com/simonw/datasette/issues/1479#issuecomment-932807859,https://api.github.com/repos/simonw/datasette/issues/1479,932807859,IC_kwDOBm6k_c43mYSz,9599,simonw,2021-10-02T19:22:35Z,2021-10-02T19:22:35Z,OWNER,"I'm pretty sure this is a Windows issue, not a Fly issue. I imagine it affects other forms of `datasette publish` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish",
https://github.com/simonw/datasette/issues/1479#issuecomment-930071625,https://api.github.com/repos/simonw/datasette/issues/1479,930071625,IC_kwDOBm6k_c43b8RJ,76450761,kirajano,2021-09-29T11:01:30Z,2021-09-29T11:01:30Z,NONE,"Thanks, but this one has a different error type. Unfortunately, still not working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish",
https://github.com/simonw/datasette/issues/1479#issuecomment-929927144,https://api.github.com/repos/simonw/datasette/issues/1479,929927144,IC_kwDOBm6k_c43bY_o,1244799,soobrosa,2021-09-29T07:49:40Z,2021-09-29T07:49:40Z,NONE,"My search yielded these four entries: https://github.com/simonw/datasette/issues?q=PermissionError%3A+%5BWinError+32%5D+

Maybe this is the closet hit?
https://github.com/simonw/datasette/issues/744
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish",
https://github.com/dogsheep/github-to-sqlite/pull/66#issuecomment-929651819,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/66,929651819,IC_kwDODFdgUs43aVxr,30531572,sarcasticadmin,2021-09-28T21:50:31Z,2021-09-28T21:50:31Z,NONE,@simonw any feedback/thoughts? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",975161924,Add --merged-by flag to pull-requests sub command,
https://github.com/simonw/sqlite-utils/issues/98#issuecomment-928790381,https://api.github.com/repos/simonw/sqlite-utils/issues/98,928790381,IC_kwDOCGYnMM43XDdt,36834097,patricktrainer,2021-09-28T04:38:44Z,2021-09-28T04:38:44Z,NONE,"Hi @simonw - wondering if you might be able to shed some light here. I've seemed to reproduce this issue. 
Here's the stacktrace: 

```
...     db[""potholes""].insert(pothole, pk='id', alter=True, replace=True)
... 

Traceback (most recent call last):
  File ""<stdin>"", line 3, in <module>
  File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2481, in insert
    return self.insert_all(
  File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2596, in insert_all
    self.insert_chunk(
  File ""/Users/patricktrainer/.pyenv/versions/3.9.0/lib/python3.9/site-packages/sqlite_utils/db.py"", line 2424, in insert_chunk
    row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0]
IndexError: list index out of range
```

Interesting enough, I found that omitting the `pk` param does not throw the error.

Let me know how I can help out! 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",597671518,"Only set .last_rowid and .last_pk for single update/inserts, not for .insert_all()/.upsert_all() with multiple records",
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-927312650,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,927312650,IC_kwDODEm0Qs43RasK,2182,danp,2021-09-26T14:09:51Z,2021-09-26T14:09:51Z,NONE,"Similar trouble with ageinfo using 0.22. Here's what my ageinfo.js file looks like:

```
window.YTD.ageinfo.part0 = [
  {
    ""ageMeta"" : { }
  }
]
```

Commenting out the registration for ageinfo in archive.py gets my archive to import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926208819,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926208819,IC_kwDOCGYnMM43NNMz,9599,simonw,2021-09-23T22:26:54Z,2021-09-23T22:26:54Z,OWNER,"I could even have those replacement characters be properties of the `Database` class, so uses can sub-class and change them.

```python
class Database:
    left_brace_replace = ""_""
    right_brace_replace = ""_""
    ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207719,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926207719,IC_kwDOCGYnMM43NM7n,9599,simonw,2021-09-23T22:24:09Z,2021-09-23T22:24:09Z,OWNER,"I think I like the underscore option best. I don't like the idea of injecting surprise `( )` parenthesis, and having them vanish entirely could result in things like `item[price]` becoming `itemprice` which feels confusing.

`item_price_` is a little ugly but I think I can live with it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926207246,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926207246,IC_kwDOCGYnMM43NM0O,9599,simonw,2021-09-23T22:23:09Z,2021-09-23T22:23:09Z,OWNER,"What are my options for replacing those characters?

- `[` becomes `(` and `]` becomes `)`
- `[` and `]` are removed entirely
- `[` and `]` both become `_`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206705,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926206705,IC_kwDOCGYnMM43NMrx,9599,simonw,2021-09-23T22:21:58Z,2021-09-23T22:21:58Z,OWNER,"I'm inclined to just fix them and not have an option for opting-out of fixing them, since it adds quite a bit of cruft to the overall API design for an option that maybe no-one will ever use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926206220,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926206220,IC_kwDOCGYnMM43NMkM,9599,simonw,2021-09-23T22:20:55Z,2021-09-23T22:21:19Z,OWNER,"If I add a new parameter for opting in and out of fixing these, what should it be called? A few options:

- `fix_columns=False`
- `rename_bad_columns=False`
- `fix_column_names=False`
- `error_on_invalid_column_names=True`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926205047,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926205047,IC_kwDOCGYnMM43NMR3,9599,simonw,2021-09-23T22:18:34Z,2021-09-23T22:19:38Z,OWNER,"Here's the code where this happens: https://github.com/simonw/sqlite-utils/blob/54191d4dc114d7dc21e849b48a4d5ae4f9e601ca/sqlite_utils/db.py#L2943-L2948

It's called from three different methods in `db.py`: `create_table_sql()`, `update()` and `insert_all()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926204046,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926204046,IC_kwDOCGYnMM43NMCO,9599,simonw,2021-09-23T22:16:18Z,2021-09-23T22:16:18Z,OWNER,"So either I do the automatic replacement, or I let the user request automatic replacement, or a third option: I do automatic replacement but let the user opt to receive an error instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/329#issuecomment-926203501,https://api.github.com/repos/simonw/sqlite-utils/issues/329,926203501,IC_kwDOCGYnMM43NL5t,9599,simonw,2021-09-23T22:15:07Z,2021-09-23T22:15:30Z,OWNER,"Quoting https://github.com/simonw/sqlite-utils/issues/86#issuecomment-586676856 in full:

> I'm not sure what to do about this one.
> 
> I can't fix it: this bug in Python's `sqlite3` module means that even if I write a database out with column names that include `[]` I won't be able to read them back again.
> 
> So... I could do one of the following:
> 
> * Throw an error if a column name includes those characters. That's my preferred option I think.
> * Automatically replace `[` in column names with `(` and `]` with `)`
> * Do the automatic replacement but show a user-visible warning when I do it
> * Throw an error, but give the user an option to run with e.g. `--fix-column-names` which applies that automatic fix.
> 
> 
> Since this is likely to be an incredibly rare edge-case I think I'd rather minimize the amount of code that deals with it, so my preferred option is to just throw that error and stop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1005891028,Rethink approach to [ and ] in column names (currently throws error),
https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925321439,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925321439,IC_kwDOCGYnMM43J0jf,9599,simonw,2021-09-22T20:52:56Z,2021-09-22T20:52:56Z,OWNER,"Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-directly-against-csv-or-json

> If two files have the same name they will be assigned a numeric suffix:
> 
>     $ sqlite-utils memory foo/data.csv bar/data.csv ""select * from data_2""","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name,
https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925303497,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925303497,IC_kwDOCGYnMM43JwLJ,9599,simonw,2021-09-22T20:25:44Z,2021-09-22T20:25:44Z,OWNER,"Here's the relevant code: https://github.com/simonw/sqlite-utils/blob/7427a9137f60de961b6331d0922a3f03da0d1890/sqlite_utils/cli.py#L1289-L1292

I can fix this by checking to see if `csv_table` is already in use and adding a suffix.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name,
https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925301981,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925301981,IC_kwDOCGYnMM43Jvzd,9599,simonw,2021-09-22T20:23:25Z,2021-09-22T20:23:25Z,OWNER,"Oddly I can't replicate this on macOS:

```
(sqlite-utils) sqlite-utils % ls foo/*.csv
foo/bug.csv
(sqlite-utils) sqlite-utils % ls bar/*.csv
bar/bug.csv
(sqlite-utils) sqlite-utils % sqlite-utils memory foo/bug.csv bar/bug.csv --schema
CREATE TABLE ""bug"" (
   [col1] TEXT,
   [col2] TEXT
);
CREATE VIEW t1 AS select * from [bug];
CREATE VIEW t AS select * from [bug];
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name,
https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925300720,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925300720,IC_kwDOCGYnMM43Jvfw,12752,gravis,2021-09-22T20:21:33Z,2021-09-22T20:21:33Z,NONE,"Wow, that was fast! Thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows,
https://github.com/simonw/sqlite-utils/issues/325#issuecomment-925300637,https://api.github.com/repos/simonw/sqlite-utils/issues/325,925300637,IC_kwDOCGYnMM43Jved,9599,simonw,2021-09-22T20:21:26Z,2021-09-22T20:21:26Z,OWNER,"The `t1` and `t2` aliases were meant to handle this case, but the are no good if the tool throws an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990844088,sqlite-utils memory can't deal with multiple files with the same name,
https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925296085,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925296085,IC_kwDOCGYnMM43JuXV,9599,simonw,2021-09-22T20:14:53Z,2021-09-22T20:14:53Z,OWNER,The bug is in this code: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/cli.py#L2205-L2227,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows,
https://github.com/simonw/sqlite-utils/issues/328#issuecomment-925292384,https://api.github.com/repos/simonw/sqlite-utils/issues/328,925292384,IC_kwDOCGYnMM43Jtdg,9599,simonw,2021-09-22T20:09:53Z,2021-09-22T20:09:53Z,OWNER,"Good catch, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1004613267,Invalid JSON output when no rows,
https://github.com/simonw/datasette/issues/111#issuecomment-924437942,https://api.github.com/repos/simonw/datasette/issues/111,924437942,IC_kwDOBm6k_c43Gc22,9599,simonw,2021-09-21T22:32:59Z,2021-09-21T22:47:07Z,OWNER,"Side-note: Django 4.0 [will switch](https://docs.djangoproject.com/en/dev/releases/4.0/#zoneinfo-default-timezone-implementation) from using `pytz` to using the standard library `zoneinfo` module introduced in Python 3.9, which has a backport that works as far back as 3.6: https://github.com/pganssle/zoneinfo (https://pypi.org/project/backports.zoneinfo/)

If I need to handle timezones I'll use that, but I think I can get away without it?

Django does this: https://github.com/django/django/blob/b0ed619303d2fb723330ca9efa3acf23d49f1d19/setup.cfg#L39-L43

```
install_requires =
    asgiref >= 3.3.2
    backports.zoneinfo; python_version<""3.9""
    sqlparse >= 0.2.2
    tzdata; sys_platform == 'win32'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/issues/111#issuecomment-924443089,https://api.github.com/repos/simonw/datasette/issues/111,924443089,IC_kwDOBm6k_c43GeHR,9599,simonw,2021-09-21T22:45:14Z,2021-09-21T22:45:26Z,OWNER,"The audiences I care about here are:

- Producers of this timestamp - generally that will be users who are using `datasette publish` to share their data
- Human consumers of this timestamp - end users who look at a Datasette site and want to know how recent the data is
- Machine consumers of this timestamp - API integrations that might want to check if a Datasette instance has been updated before downloading new data

For producers I think there are going to be two categories. The first is users who run ""publish"" and want the site to reflect when they did so (probably using `--updated=now` when they publish). The second are users who are willing to spend more time thinking about this - for example my various git scraping projects where I want to use a date derived from the git history.

For humans... I'd like to be able to calculate a relative time (3 hours ago) in addition to showing the display time, because that helps avoid confusion over timezones.

For machine consumers, it might be nice to offer the option of a calculated Unix timestamp-since-1970, since those can be easier to work with in some languages than running a full ISO date parser.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/issues/111#issuecomment-924438481,https://api.github.com/repos/simonw/datasette/issues/111,924438481,IC_kwDOBm6k_c43Gc_R,9599,simonw,2021-09-21T22:34:03Z,2021-09-21T22:34:21Z,OWNER,"The simplest possible version of this: it's always represented as a UTC ISO datetime, like this: 

    ""updated"": ""2020-10-31T12:00:00+00:00""

Later versions of Datasette could extend this to handle other timezones or support just the date (though that's a backwards incompatible change so probably better to decide on the date thing right now).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/issues/111#issuecomment-924435971,https://api.github.com/repos/simonw/datasette/issues/111,924435971,IC_kwDOBm6k_c43GcYD,9599,simonw,2021-09-21T22:29:15Z,2021-09-21T22:29:49Z,OWNER,"So this is a metadata key called `updated` which can be applied at the table, database or instance level. It is represented as a `.isoformat()` timestamp.

Question: should I support just the date - `yyyy-mm-dd` - in addition to the datetime?

I think so. I can easily imagine situations where the exact time of day that a change was made hasn't been recorded, but the overall date is known.

But in that case, should the `updated` key sometimes be `yyyy-mm-dd` and sometimes be the full isoformat datetime? Or should there be an `updated_date` key that's used for just the date?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/issues/111#issuecomment-924432643,https://api.github.com/repos/simonw/datasette/issues/111,924432643,IC_kwDOBm6k_c43GbkD,9599,simonw,2021-09-21T22:23:23Z,2021-09-21T22:23:23Z,OWNER,I'm going to use https://github.com/dateutil/dateutil for this - it's been maintained constantly (by an evolving team of contributors) [since 2003](https://github.com/dateutil/dateutil/commit/68ae2757ae15c84bf947d47a82a314b3b975bc9b) and is a very trustworthy dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/dogsheep/twitter-to-sqlite/pull/59#issuecomment-924209583,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/59,924209583,IC_kwDODEm0Qs43FlGv,9599,simonw,2021-09-21T17:37:34Z,2021-09-21T17:37:34Z,MEMBER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",984942782,"Fix for since_id bug, closes #58",
https://github.com/simonw/datasette/pull/1476#issuecomment-923979964,https://api.github.com/repos/simonw/datasette/issues/1476,923979964,IC_kwDOBm6k_c43EtC8,22429695,codecov[bot],2021-09-21T13:18:22Z,2021-09-21T13:18:22Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1476](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2eb01ff) into [main](https://codecov.io/gh/simonw/datasette/commit/b28b6cd2fe97f7e193a235877abeec2c8eb0a821?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b28b6cd) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1476/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1476   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b28b6cd...2eb01ff](https://codecov.io/gh/simonw/datasette/pull/1476?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1002459220,"Update pytest-xdist requirement from <2.4,>=2.2.1 to >=2.2.1,<2.5",
https://github.com/simonw/datasette/issues/111#issuecomment-923106887,https://api.github.com/repos/simonw/datasette/issues/111,923106887,IC_kwDOBm6k_c43BX5H,9599,simonw,2021-09-20T16:58:39Z,2021-09-20T16:58:39Z,OWNER,Still a good idea today too! Would be great for https://cdc-vaccination-history.datasette.io/ for example.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/issues/1473#issuecomment-922394999,https://api.github.com/repos/simonw/datasette/issues/1473,922394999,IC_kwDOBm6k_c42-qF3,192568,mroswell,2021-09-19T00:44:39Z,2021-09-19T00:45:32Z,CONTRIBUTOR,"I replaced:
```
<a class=""text-3xl font-bold leading-none"" href=""https://www.saferortoxic.org"">
<img 
src=""https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp"" 
class=""custom-logo"" 
alt=""Logo: Safer or Toxic?"" 
width=""300px"">
</a>
```
with:
```
<img src=""https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp"" 
alt=""Logo: Safer or Toxic?"" 
style=""width:300px; cursor:pointer;"" 
onclick=""window.location.href='https://saferortoxic.org';return false;"">
```

I'd still love to know what caused this (and how to troubleshoot to figure it out), so I'll leave it open for a bit, but I do have a functional logo linking to the Hugo home page, at least locally. I'll likely push tomorrow.

(Before trying this, I tried to apply a background image to the `a` tag. That didn't work.)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",999902754,base logo link visits `undefined` rather than href url,
https://github.com/simonw/datasette/issues/1473#issuecomment-922363640,https://api.github.com/repos/simonw/datasette/issues/1473,922363640,IC_kwDOBm6k_c42-ib4,192568,mroswell,2021-09-18T19:45:47Z,2021-09-18T19:45:47Z,CONTRIBUTOR,"An update, if I remove the `img` tag and replace it with the text, ""Safer or Toxic?"" it links to the right place.

Also, if I keep things exactly as they are, and it improperly, but consistently goes to the `undefined` page, on THAT 404 page, a click on the image properly clicks through to the www.SaferOrToxic.org page.

Weird stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",999902754,base logo link visits `undefined` rather than href url,
https://github.com/simonw/datasette/issues/236#issuecomment-922075480,https://api.github.com/repos/simonw/datasette/issues/236,922075480,IC_kwDOBm6k_c429cFY,9599,simonw,2021-09-17T20:54:13Z,2021-09-17T20:54:13Z,OWNER,"That's so useful @sethvincent! Really interesting reading your code there, especially clever how you're using the `base_url` config.

I'd be very interested to see what your demo looks like without using serverless - completely agree that the less additional dependencies there are for this the better.

I'm also very interested in figuring out a way to run Datasette in Lambda but with the SQLite database on an EFS volume. Do you have a feel for how hard that would be?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/236#issuecomment-920543967,https://api.github.com/repos/simonw/datasette/issues/236,920543967,IC_kwDOBm6k_c423mLf,164214,sethvincent,2021-09-16T03:19:08Z,2021-09-16T03:19:08Z,NONE,":wave: I just put together a small example using the lambda container image support: https://github.com/sethvincent/datasette-aws-lambda-example

It uses mangum and AWS's [python runtime interface client](https://github.com/aws/aws-lambda-python-runtime-interface-client) to handle the lambda event stuff.

I'd be happy to help with a publish plugin for AWS lambda as I plan to use this for upcoming projects.

The example uses the [serverless](https://www.serverless.com) cli for deployment but there might be a more suitable deployment approach for the plugin. It would be cool if users didn't have to install anything additional other than the aws cli and its associated config/credentials setup.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/pull/1453#issuecomment-919135732,https://api.github.com/repos/simonw/datasette/issues/1453,919135732,IC_kwDOBm6k_c42yOX0,49699333,dependabot[bot],2021-09-14T13:10:38Z,2021-09-14T13:10:38Z,CONTRIBUTOR,Superseded by #1471.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",982780906,Bump black from 21.7b0 to 21.8b0,
https://github.com/simonw/datasette/issues/1464#issuecomment-918621705,https://api.github.com/repos/simonw/datasette/issues/1464,918621705,IC_kwDOBm6k_c42wQ4J,7476523,bobwhitelock,2021-09-13T22:17:17Z,2021-09-13T22:17:17Z,CONTRIBUTOR,"> haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?

Not certain, but I think tests in CI run on Ubuntu and don't appear to install any additional Sqlite-related dependencies, and so my guess is the version of Sqlite installed by default on Ubuntu has the `SQLITE_ENABLE_FTS3_PARENTHESIS` option enabled and so doesn't run into this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/pull/1385#issuecomment-869105782,https://api.github.com/repos/simonw/datasette/issues/1385,869105782,MDEyOklzc3VlQ29tbWVudDg2OTEwNTc4Mg==,22429695,codecov[bot],2021-06-27T05:48:55Z,2021-09-13T17:29:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1385](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (db78094) into [main](https://codecov.io/gh/simonw/datasette/commit/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ea627ba) will **not change** coverage.
> The diff coverage is `n/a`.

> :exclamation: Current head db78094 differs from pull request most recent head 8d78c8c. Consider uploading reports for the commit 8d78c8c to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1385/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1385   +/-   ##
=======================================
  Coverage   91.70%   91.70%           
=======================================
  Files          34       34           
  Lines        4364     4364           
=======================================
  Hits         4002     4002           
  Misses        362      362           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ea627ba...8d78c8c](https://codecov.io/gh/simonw/datasette/pull/1385?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930855052,Fix + improve get_metadata plugin hook docs,
https://github.com/simonw/datasette/issues/1466#issuecomment-917840012,https://api.github.com/repos/simonw/datasette/issues/1466,917840012,IC_kwDOBm6k_c42tSCM,9599,simonw,2021-09-13T04:54:59Z,2021-09-13T04:54:59Z,OWNER,Especially relevant now that 0.2.0 is out which is a much higher quality release. https://github.com/simonw/datasette-app/releases/tag/0.2.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991467558,Add Datasette Desktop to installation documentation,
https://github.com/simonw/datasette/issues/1468#issuecomment-917839801,https://api.github.com/repos/simonw/datasette/issues/1468,917839801,IC_kwDOBm6k_c42tR-5,9599,simonw,2021-09-13T04:54:17Z,2021-09-13T04:54:17Z,OWNER,Here's a already open issue for this: #972,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries,
https://github.com/simonw/datasette/issues/1468#issuecomment-917839507,https://api.github.com/repos/simonw/datasette/issues/1468,917839507,IC_kwDOBm6k_c42tR6T,9599,simonw,2021-09-13T04:53:22Z,2021-09-13T04:53:22Z,OWNER,"At the moment this isn't possible - though there's a workaround which is to define a SQL view for the query, at which point facets will be displayed again.

I did a lot of the work required to support this when I refactored how facets worked a while back - but to finally implement this I need to refactor the table view and the arbitrary query view to share much more logic than they do at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994390593,Faceting for custom SQL queries,
https://github.com/simonw/datasette/issues/1469#issuecomment-917839062,https://api.github.com/repos/simonw/datasette/issues/1469,917839062,IC_kwDOBm6k_c42tRzW,9599,simonw,2021-09-13T04:52:01Z,2021-09-13T04:52:01Z,OWNER,Here's the code at fault: https://github.com/simonw/datasette/blob/b28b6cd2fe97f7e193a235877abeec2c8eb0a821/datasette/static/table.js#L137-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",994450961,"Column cog shows ""facet by this"" when already default faceted",
https://github.com/simonw/datasette/issues/1464#issuecomment-917642487,https://api.github.com/repos/simonw/datasette/issues/1464,917642487,IC_kwDOBm6k_c42shz3,51016,ctb,2021-09-12T14:03:09Z,2021-09-12T14:03:09Z,CONTRIBUTOR,"haven't had time to get back to this, but idle thought that I'm recording for later investigation: how does the continuous integration handle this installation issue? Is it documented there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/sqlite-utils/pull/326#issuecomment-916119657,https://api.github.com/repos/simonw/sqlite-utils/issues/326,916119657,IC_kwDOCGYnMM42muBp,191622,meatcar,2021-09-09T13:54:10Z,2021-09-09T13:54:10Z,CONTRIBUTOR,dupe of #293?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991237645,Test against 3.10-dev,
https://github.com/simonw/datasette/issues/1464#issuecomment-915343886,https://api.github.com/repos/simonw/datasette/issues/1464,915343886,IC_kwDOBm6k_c42jwoO,7476523,bobwhitelock,2021-09-08T15:32:06Z,2021-09-08T15:32:06Z,CONTRIBUTOR,"Thanks, that does look similar!

> Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.

Ah that makes sense, I guess that's why this isn't just always installed already. I wonder if a possible solution to this issue could be doing feature detection on whether this feature is supported by the current Sqlite version, and if not these tests could be disabled locally? But possibly there's a better way to handle this, will see what @simonw thinks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/sqlite-utils/pull/326#issuecomment-915321467,https://api.github.com/repos/simonw/sqlite-utils/issues/326,915321467,IC_kwDOCGYnMM42jrJ7,22429695,codecov[bot],2021-09-08T15:05:53Z,2021-09-08T15:05:53Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#326](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (078a087) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/77c240df56068341561e95e4a412cbfa24dc5bc7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (77c240d) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/326/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main     #326   +/-   ##
=======================================
  Coverage   96.58%   96.58%           
=======================================
  Files           5        5           
  Lines        2223     2223           
=======================================
  Hits         2147     2147           
  Misses         76       76           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [77c240d...078a087](https://codecov.io/gh/simonw/sqlite-utils/pull/326?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991237645,Test against 3.10-dev,
https://github.com/simonw/datasette/issues/1464#issuecomment-915302885,https://api.github.com/repos/simonw/datasette/issues/1464,915302885,IC_kwDOBm6k_c42jmnl,51016,ctb,2021-09-08T14:44:50Z,2021-09-08T14:44:50Z,CONTRIBUTOR,"thanks for the response! full errors attached; excerpt:

```
...


    def test_searchmode(table_metadata, querystring, expected_rows):
        with make_app_client(
            metadata={""databases"": {""fixtures"": {""tables"": {""searchable"": table_metadata}}}}
        ) as client:
            response = client.get(""/fixtures/searchable.json?"" + querystring)
>           assert expected_rows == response.json[""rows""]
E           AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
E             Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
E             Use -v to get the full diff

/Users/t/dev/datasette/tests/test_api.py:1115: AssertionError
```

[errors.txt](https://github.com/simonw/datasette/files/7129719/errors.txt)

A quick scan of #1223 suggests you're right. Unfortunately, pysqlite3-binary isn't available for Mac OS X, so I can't quickly check that that fixes it; will do so later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1464#issuecomment-915299013,https://api.github.com/repos/simonw/datasette/issues/1464,915299013,IC_kwDOBm6k_c42jlrF,7476523,bobwhitelock,2021-09-08T14:40:28Z,2021-09-08T14:40:28Z,CONTRIBUTOR,"What are the full errors you're getting?

This *may* be the same issue as described in https://github.com/simonw/datasette/pull/1223 - essentially the test suite (and corresponding Datasette features I assume) are by default implicitly dependent on your Sqlite installation having been compiled with the `SQLITE_ENABLE_FTS3_PARENTHESIS` option. If this is the same issue then I think this can be fixed either by recompiling with that option or (probably more easily) by running `pip install pysqlite3-binary`, which will be used in preference to your system Sqlite installation and has this option enabled.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1464#issuecomment-915279711,https://api.github.com/repos/simonw/datasette/issues/1464,915279711,IC_kwDOBm6k_c42jg9f,51016,ctb,2021-09-08T14:16:49Z,2021-09-08T14:16:49Z,CONTRIBUTOR,on commit d57ab156b35ec642,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",991191951,clean checkout & clean environment has test failures,
https://github.com/simonw/datasette/issues/1462#issuecomment-914644260,https://api.github.com/repos/simonw/datasette/issues/1462,914644260,IC_kwDOBm6k_c42hF0k,9599,simonw,2021-09-07T21:34:32Z,2021-09-07T21:34:32Z,OWNER,"I think this is a setting. There are two relevant settings at the moment:
```
    ""template_debug"": false,
    ""trace_debug"": false,
```
For consistence then this should be called `something_debug` - but do I want a single setting that exposes the `_internal` database and adds those debug options to the menu, or do I want those as two separate settings?

- `internal_debug` to enable access to that `_internal` database
- `menu_debug` for those menu options?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",990367646,"Separate out ""debug"" options from ""root"" options",
https://github.com/simonw/datasette/issues/1461#issuecomment-914441037,https://api.github.com/repos/simonw/datasette/issues/1461,914441037,IC_kwDOBm6k_c42gUNN,9599,simonw,2021-09-07T16:13:59Z,2021-09-07T16:13:59Z,OWNER,"I don't think I'll adopt it for this project. For example, here:
```diff
     response = Response.redirect(""/"")
-    response.set_cookie(""ds_actor"", datasette.sign({
-        ""a"": {
-            ""id"": ""cleopaws""
-        }
-    }, ""actor""))
+    response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor""))
```
I chose to use the multi-line version to help emphasize the structure - the single-line replacement loses that. I think I'll continue to make my own editorial choices about how the code examples are laid out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs,
https://github.com/simonw/datasette/issues/1461#issuecomment-914440282,https://api.github.com/repos/simonw/datasette/issues/1461,914440282,IC_kwDOBm6k_c42gUBa,9599,simonw,2021-09-07T16:12:57Z,2021-09-07T16:12:57Z,OWNER,"Here's the diff it produced from that first run:
```diff
diff --git a/docs/authentication.rst b/docs/authentication.rst
index 0d98cf8..8008023 100644
--- a/docs/authentication.rst
+++ b/docs/authentication.rst
@@ -381,11 +381,7 @@ Authentication plugins can set signed ``ds_actor`` cookies themselves like so:
 .. code-block:: python
 
     response = Response.redirect(""/"")
-    response.set_cookie(""ds_actor"", datasette.sign({
-        ""a"": {
-            ""id"": ""cleopaws""
-        }
-    }, ""actor""))
+    response.set_cookie(""ds_actor"", datasette.sign({""a"": {""id"": ""cleopaws""}}, ""actor""))
 
 Note that you need to pass ``""actor""`` as the namespace to :ref:`datasette_sign`.
 
@@ -412,12 +408,16 @@ To include an expiry, add a ``""e""`` key to the cookie value containing a `base62
     expires_at = int(time.time()) + (24 * 60 * 60)
 
     response = Response.redirect(""/"")
-    response.set_cookie(""ds_actor"", datasette.sign({
-        ""a"": {
-            ""id"": ""cleopaws""
-        },
-        ""e"": baseconv.base62.encode(expires_at),
-    }, ""actor""))
+    response.set_cookie(
+        ""ds_actor"",
+        datasette.sign(
+            {
+                ""a"": {""id"": ""cleopaws""},
+                ""e"": baseconv.base62.encode(expires_at),
+            },
+            ""actor"",
+        ),
+    )
 
 The resulting cookie will encode data that looks something like this:
 
diff --git a/docs/spatialite.rst b/docs/spatialite.rst
index d1b300b..556bad8 100644
--- a/docs/spatialite.rst
+++ b/docs/spatialite.rst
@@ -58,19 +58,22 @@ Here's a recipe for taking a table with existing latitude and longitude columns,
 .. code-block:: python
 
     import sqlite3
-    conn = sqlite3.connect('museums.db')
+
+    conn = sqlite3.connect(""museums.db"")
     # Lead the spatialite extension:
     conn.enable_load_extension(True)
-    conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
+    conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"")
     # Initialize spatial metadata for this database:
-    conn.execute('select InitSpatialMetadata(1)')
+    conn.execute(""select InitSpatialMetadata(1)"")
     # Add a geometry column called point_geom to our museums table:
     conn.execute(""SELECT AddGeometryColumn('museums', 'point_geom', 4326, 'POINT', 2);"")
     # Now update that geometry column with the lat/lon points
-    conn.execute('''
+    conn.execute(
+        """"""
         UPDATE museums SET
         point_geom = GeomFromText('POINT('||""longitude""||' '||""latitude""||')',4326);
-    ''')
+    """"""
+    )
     # Now add a spatial index to that column
     conn.execute('select CreateSpatialIndex(""museums"", ""point_geom"");')
     # If you don't commit your changes will not be persisted:
@@ -186,13 +189,14 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
 .. code-block:: python
 
     import sqlite3
-    conn = sqlite3.connect('places.db')
+
+    conn = sqlite3.connect(""places.db"")
     # Enable SpatialLite extension
     conn.enable_load_extension(True)
-    conn.load_extension('/usr/local/lib/mod_spatialite.dylib')
+    conn.load_extension(""/usr/local/lib/mod_spatialite.dylib"")
     # Create the masic countries table
-    conn.execute('select InitSpatialMetadata(1)')
-    conn.execute('create table places (id integer primary key, name text);')
+    conn.execute(""select InitSpatialMetadata(1)"")
+    conn.execute(""create table places (id integer primary key, name text);"")
     # Add a MULTIPOLYGON Geometry column
     conn.execute(""SELECT AddGeometryColumn('places', 'geom', 4326, 'MULTIPOLYGON', 2);"")
     # Add a spatial index against the new column
@@ -201,13 +205,17 @@ Here's Python code to create a SQLite database, enable SpatiaLite, create a plac
     from shapely.geometry.multipolygon import MultiPolygon
     from shapely.geometry import shape
     import requests
-    geojson = requests.get('https://data.whosonfirst.org/404/227/475/404227475.geojson').json()
+
+    geojson = requests.get(
+        ""https://data.whosonfirst.org/404/227/475/404227475.geojson""
+    ).json()
     # Convert to ""Well Known Text"" format
-    wkt = shape(geojson['geometry']).wkt
+    wkt = shape(geojson[""geometry""]).wkt
     # Insert and commit the record
-    conn.execute(""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"", (
-       ""Wales"", wkt
-    ))
+    conn.execute(
+        ""INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))"",
+        (""Wales"", wkt),
+    )
     conn.commit()
 
 Querying polygons using within()
diff --git a/docs/writing_plugins.rst b/docs/writing_plugins.rst
index bd60a4b..5af01f6 100644
--- a/docs/writing_plugins.rst
+++ b/docs/writing_plugins.rst
@@ -18,9 +18,10 @@ The quickest way to start writing a plugin is to create a ``my_plugin.py`` file
 
     from datasette import hookimpl
 
+
     @hookimpl
     def prepare_connection(conn):
-        conn.create_function('hello_world', 0, lambda: 'Hello world!')
+        conn.create_function(""hello_world"", 0, lambda: ""Hello world!"")
 
 If you save this in ``plugins/my_plugin.py`` you can then start Datasette like this::
 
@@ -60,22 +61,18 @@ The example consists of two files: a ``setup.py`` file that defines the plugin:
 
     from setuptools import setup
 
-    VERSION = '0.1'
+    VERSION = ""0.1""
 
     setup(
-        name='datasette-plugin-demos',
-        description='Examples of plugins for Datasette',
-        author='Simon Willison',
-        url='https://github.com/simonw/datasette-plugin-demos',
-        license='Apache License, Version 2.0',
+        name=""datasette-plugin-demos"",
+        description=""Examples of plugins for Datasette"",
+        author=""Simon Willison"",
+        url=""https://github.com/simonw/datasette-plugin-demos"",
+        license=""Apache License, Version 2.0"",
         version=VERSION,
-        py_modules=['datasette_plugin_demos'],
-        entry_points={
-            'datasette': [
-                'plugin_demos = datasette_plugin_demos'
-            ]
-        },
-        install_requires=['datasette']
+        py_modules=[""datasette_plugin_demos""],
+        entry_points={""datasette"": [""plugin_demos = datasette_plugin_demos""]},
+        install_requires=[""datasette""],
     )
 
 And a Python module file, ``datasette_plugin_demos.py``, that implements the plugin:
@@ -88,12 +85,12 @@ And a Python module file, ``datasette_plugin_demos.py``, that implements the plu
 
     @hookimpl
     def prepare_jinja2_environment(env):
-        env.filters['uppercase'] = lambda u: u.upper()
+        env.filters[""uppercase""] = lambda u: u.upper()
 
 
     @hookimpl
     def prepare_connection(conn):
-        conn.create_function('random_integer', 2, random.randint)
+        conn.create_function(""random_integer"", 2, random.randint)
 
 
 Having built a plugin in this way you can turn it into an installable package using the following command::
@@ -123,11 +120,13 @@ To bundle the static assets for a plugin in the package that you publish to PyPI
 
 .. code-block:: python
 
-        package_data={
-            'datasette_plugin_name': [
-                'static/plugin.js',
-            ],
-        },
+        package_data = (
+            {
+                ""datasette_plugin_name"": [
+                    ""static/plugin.js"",
+                ],
+            },
+        )
 
 Where ``datasette_plugin_name`` is the name of the plugin package (note that it uses underscores, not hyphens) and ``static/plugin.js`` is the path within that package to the static file.
 
@@ -152,11 +151,13 @@ Templates should be bundled for distribution using the same ``package_data`` mec
 
 .. code-block:: python
 
-        package_data={
-            'datasette_plugin_name': [
-                'templates/my_template.html',
-            ],
-        },
+        package_data = (
+            {
+                ""datasette_plugin_name"": [
+                    ""templates/my_template.html"",
+                ],
+            },
+        )
 
 You can also use wildcards here such as ``templates/*.html``. See `datasette-edit-schema <https://github.com/simonw/datasette-edit-schema>`__ for an example of this pattern.
 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs,
https://github.com/simonw/datasette/issues/1461#issuecomment-914439356,https://api.github.com/repos/simonw/datasette/issues/1461,914439356,IC_kwDOBm6k_c42gTy8,9599,simonw,2021-09-07T16:11:37Z,2021-09-07T16:11:37Z,OWNER,"```
(datasette) datasette % blacken-docs docs/*.rst
docs/authentication.rst: Rewriting...
docs/internals.rst:169: code block parse error Cannot parse: 14:0: <line number missing in source>
docs/plugin_hooks.rst:251: code block parse error Cannot parse: 6:4:     ]
docs/plugin_hooks.rst:312: code block parse error Cannot parse: 38:0: <line number missing in source>
docs/spatialite.rst: Rewriting...
docs/testing_plugins.rst:135: code block parse error Cannot parse: 5:0: <line number missing in source>
docs/writing_plugins.rst: Rewriting...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",989986586,Try blacken-docs,
https://github.com/simonw/datasette/issues/1459#issuecomment-913218494,https://api.github.com/repos/simonw/datasette/issues/1459,913218494,IC_kwDOBm6k_c42bpu-,9599,simonw,2021-09-05T19:58:51Z,2021-09-05T19:59:15Z,OWNER,"This idea makes sense to me.

There's actually an existing option that takes a path, called `--get` - it returns the HTML or JSON for that oath directly to the console, eg `datasette my.db --get /mydb/mytable.json`

So... one option would be to allow combining that with `-o` to open that URL in the browser:

    datasette my.db -o --get /mydb

So some options here are:

- `datasette my.db --open-url /mydb`
- `datasette my.db --open-path /mydb`
- `datasette my.db --open --get /mydb`

I quite like that last combination option, mainly to avoid adding even more command options.


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988556488,suggestion: allow `datasette --open` to take a relative URL,
https://github.com/simonw/datasette/pull/1455#issuecomment-913001416,https://api.github.com/repos/simonw/datasette/issues/1455,913001416,IC_kwDOBm6k_c42a0vI,9599,simonw,2021-09-04T16:32:21Z,2021-09-04T16:32:21Z,OWNER,I'll add researchers too.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups,
https://github.com/simonw/datasette/pull/1455#issuecomment-913001298,https://api.github.com/repos/simonw/datasette/issues/1455,913001298,IC_kwDOBm6k_c42a0tS,9599,simonw,2021-09-04T16:31:32Z,2021-09-04T16:31:32Z,OWNER,Great idea!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups,
https://github.com/simonw/datasette/pull/1455#issuecomment-913001282,https://api.github.com/repos/simonw/datasette/issues/1455,913001282,IC_kwDOBm6k_c42a0tC,51016,ctb,2021-09-04T16:31:24Z,2021-09-04T16:31:24Z,CONTRIBUTOR,I love it! maybe 'researchers' instead? Or 'scientists and researchers'?,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",988325628,Add scientists to target groups,
https://github.com/dogsheep/evernote-to-sqlite/issues/14#issuecomment-911772943,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/14,911772943,IC_kwDOEhK-wc42WI0P,46968,step21,2021-09-02T14:53:11Z,2021-09-02T14:53:11Z,NONE,"Additionally, assuming the line numbers match up with the provided enenx file, the mentioned line plus one before and after is  as follows:
```
<![CDATA[>]]>
</span></div>
<div style=""padding: 0px; font-family: Arial, sans-serif; font-size: 12px; line-height: 16px; white-space: pre-wrap;""><br style="" padding: 0px;""/>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",986829194,xml.etree.ElementTree.Parse Error - mismatched tag,
https://github.com/dogsheep/twitter-to-sqlite/issues/58#issuecomment-910121331,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/58,910121331,IC_kwDODEm0Qs42P1lz,42904,rubenv,2021-09-01T09:49:33Z,2021-09-01T09:49:33Z,CONTRIBUTOR,"Found the cause, it's the other commands. PR #59 submitted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",984939366,"Error: Use either --since or --since_id, not both - still broken",
https://github.com/simonw/datasette/issues/1446#issuecomment-908832938,https://api.github.com/repos/simonw/datasette/issues/1446,908832938,IC_kwDOBm6k_c42K7Cq,9599,simonw,2021-08-31T01:54:59Z,2021-08-31T01:54:59Z,OWNER,I used the sticky footer mechanism in `datasette.app`: https://github.com/simonw/datasette.app/issues/3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer,
https://github.com/simonw/datasette/pull/1453#issuecomment-908337614,https://api.github.com/repos/simonw/datasette/issues/1453,908337614,IC_kwDOBm6k_c42JCHO,22429695,codecov[bot],2021-08-30T13:20:26Z,2021-08-30T13:20:26Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1453](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4f492a7) into [main](https://codecov.io/gh/simonw/datasette/commit/67cbf0ae7243431bf13702e6e3ba466b619c4d6f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (67cbf0a) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1453/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1453   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files          34       34           
  Lines        4421     4421           
=======================================
  Hits         4060     4060           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [67cbf0a...4f492a7](https://codecov.io/gh/simonw/datasette/pull/1453?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",982780906,Bump black from 21.7b0 to 21.8b0,
https://github.com/simonw/datasette/issues/1449#issuecomment-907547624,https://api.github.com/repos/simonw/datasette/issues/1449,907547624,IC_kwDOBm6k_c42GBPo,9599,simonw,2021-08-28T01:44:57Z,2021-08-28T01:58:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907547736,https://api.github.com/repos/simonw/datasette/issues/1449,907547736,IC_kwDOBm6k_c42GBRY,9599,simonw,2021-08-28T01:45:36Z,2021-08-28T01:45:36Z,OWNER,I need to push this out as an alpha so I can release a demo plugin that uses it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907543982,https://api.github.com/repos/simonw/datasette/issues/1449,907543982,IC_kwDOBm6k_c42GAWu,9599,simonw,2021-08-28T01:14:27Z,2021-08-28T01:14:27Z,OWNER,"Writing the test for this is proving difficult, because the `cli` module has already been imported when I attempt to register a new plugin - so it doesn't pick up on the additional command registrations.

Trying to work around that with `importlib.reload(cli)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907542214,https://api.github.com/repos/simonw/datasette/issues/1449,907542214,IC_kwDOBm6k_c42F_7G,9599,simonw,2021-08-28T01:02:38Z,2021-08-28T01:02:38Z,OWNER,"Partial prototype of `datasette-verify`:
```python
from datasette import hookimpl
import click


@hookimpl
def register_commands(cli):
    from datasette.cli import sqlite_extensions
    @cli.command()
    @click.argument(""files"", type=click.Path(exists=True), nargs=-1)
    @sqlite_extensions
    def verify(files, sqlite_extensions):
        ""Verify that files can be opened by Datasette""
        for file in files:
            print(file)
```
I had to move the `from datasette.cli import sqlite_extensions` inside the hook function to avoid a circular import.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907540928,https://api.github.com/repos/simonw/datasette/issues/1449,907540928,IC_kwDOBm6k_c42F_nA,9599,simonw,2021-08-28T00:53:37Z,2021-08-28T00:53:37Z,OWNER,I'll probably have to use this mechanism for the tests then: https://til.simonwillison.net/pytest/registering-plugins-in-tests,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907540790,https://api.github.com/repos/simonw/datasette/issues/1449,907540790,IC_kwDOBm6k_c42F_k2,9599,simonw,2021-08-28T00:52:32Z,2021-08-28T00:52:32Z,OWNER,I don't think I can get this new hook to support the handy [--plugins-dir= mechanism](https://docs.datasette.io/en/stable/plugins.html#one-off-plugins-using-plugins-dir) for loading plugins from Python files as opposed to registering them with setuptools - that mechanism is itself implemented inside of code called by `datasette serve` so I don't have a way of taking advantage of it from outside that command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1450#issuecomment-907540240,https://api.github.com/repos/simonw/datasette/issues/1450,907540240,IC_kwDOBm6k_c42F_cQ,9599,simonw,2021-08-28T00:48:30Z,2021-08-28T00:48:30Z,OWNER,"I'll go with this:
```
% datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette is an open source multi-tool for exploring and publishing data

  About Datasette: https://datasette.io/
  Full documentation: https://docs.datasette.io/

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981681138,"Datasette --help should show something more useful than ""Datasette!""",
https://github.com/simonw/datasette/issues/1449#issuecomment-907539668,https://api.github.com/repos/simonw/datasette/issues/1449,907539668,IC_kwDOBm6k_c42F_TU,9599,simonw,2021-08-28T00:44:16Z,2021-08-28T00:44:16Z,OWNER,"Considering this piece of code: https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L142

I think the hook itself gets called with a single argument, `cli`, which it can then use in the standard Click way to register extra stuff.

I can't pass it `datasette` (like I do with `register_routes()`) because the Datasette object itself is instantiated by the `serve` command, which will not have been called.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907539251,https://api.github.com/repos/simonw/datasette/issues/1449,907539251,IC_kwDOBm6k_c42F_Mz,9599,simonw,2021-08-28T00:41:37Z,2021-08-28T00:41:50Z,OWNER,The first example plugin I'm going to build for this will be `datasette verify file.db file2.db` - it will take one or more paths to SQLite files and verify if they can be opened by Datasette or not.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907539065,https://api.github.com/repos/simonw/datasette/issues/1449,907539065,IC_kwDOBm6k_c42F_J5,9599,simonw,2021-08-28T00:40:24Z,2021-08-28T00:40:24Z,OWNER,I'm going to call the new hook `register_commands` - since it will allow ambitious plugins to register more than one command if they want to. That's also pleasingly similar to `register_routes`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907538940,https://api.github.com/repos/simonw/datasette/issues/1449,907538940,IC_kwDOBm6k_c42F_H8,9599,simonw,2021-08-28T00:39:28Z,2021-08-28T00:39:28Z,OWNER,"Here's the answer to that:
```
~ % datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet...
  uninstall  Uninstall Python packages (e.g.
```
Since it's adding extra things that show up in `--help` under the ""Commands:"" heading, I should call them commands.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907537693,https://api.github.com/repos/simonw/datasette/issues/1449,907537693,IC_kwDOBm6k_c42F-0d,9599,simonw,2021-08-28T00:31:26Z,2021-08-28T00:31:26Z,OWNER,Terminology question: is it correct to call these subcommands or should they be commands? `publish_subcommand()` adds subcommands of the format `datasette publish X` - but are we instead adding commands with this new one?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907537610,https://api.github.com/repos/simonw/datasette/issues/1449,907537610,IC_kwDOBm6k_c42F-zK,9599,simonw,2021-08-28T00:30:51Z,2021-08-28T00:30:51Z,OWNER,"There's also the option for plugins to muck around with existing registered commands - this could get a bit untidy if multiple plugins try to do it, but being able to replace `serve` with a fresh implementation that adds an additional command-line option before calling back to the original might open up some interesting possibilities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/simonw/datasette/issues/1449#issuecomment-907537366,https://api.github.com/repos/simonw/datasette/issues/1449,907537366,IC_kwDOBm6k_c42F-vW,9599,simonw,2021-08-28T00:29:16Z,2021-08-28T00:29:29Z,OWNER,"The closest plugin hook to this right now is [publish_subcommand](https://docs.datasette.io/en/stable/plugin_hooks.html#publish-subcommand-publish) - which looks like this:
```python
@hookimpl
def publish_subcommand(publish):
    @publish.command()
    @add_common_publish_arguments_and_options
    @click.option(
        ""-k"",
        ""--api_key"",
        help=""API key for talking to my hosting provider"",
    )
    def my_hosting_provider(...):
```
But there are also several plugin hooks with `register_` prefixes, which may be a good naming convention to stick to here: `register_output_renderer(datasette)`, `register_routes(datasette)`, `register_facet_classes()`, `register_magic_parameters(datasette)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",981676832,`register_commands()` plugin hook to register extra CLI commands,
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906646452,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,906646452,IC_kwDOEhK-wc42ClO0,9599,simonw,2021-08-26T18:34:34Z,2021-08-26T18:35:20Z,MEMBER,"I tried this ampersand fix: https://regex101.com/r/ojU2H9/1
```python

# https://regex101.com/r/ojU2H9/1
_invalid_ampersand_re = re.compile(r'&(?![a-z0-9]+;)')

def fix_bad_xml(xml):
    # More fixes for things like '&' not as part of an entity
    return _invalid_ampersand_re.sub('&amp;', xml)
```

Even with that I'm still getting total garbage in the `<en-note>` content - it's just HTML, not even trying to be XML.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-906635938,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,906635938,IC_kwDOEhK-wc42Ciqi,9599,simonw,2021-08-26T18:18:27Z,2021-08-26T18:18:27Z,MEMBER,"It looks like I was using the round-trip to dump the `<?xml version=""1.0"" encoding=""UTF-8"" standalone=""no""?>` and `<!DOCTYPE` prefixes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/simonw/datasette/pull/1448#issuecomment-906414811,https://api.github.com/repos/simonw/datasette/issues/1448,906414811,IC_kwDOBm6k_c42Bsrb,22429695,codecov[bot],2021-08-26T13:29:02Z,2021-08-26T13:29:02Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1448](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a211747) into [main](https://codecov.io/gh/simonw/datasette/commit/a1a33bb5822214be1cebd98cd858b2058d91a4aa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a1a33bb) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1448/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1448   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4418     4418           
=======================================
  Hits         4057     4057           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a1a33bb...a211747](https://codecov.io/gh/simonw/datasette/pull/1448?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",980228553,"Update pluggy requirement from ~=0.13.0 to >=0.13,<1.1",
https://github.com/dogsheep/dogsheep-photos/issues/7#issuecomment-906015471,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/7,906015471,IC_kwDOD079W842ALLv,18232,dkam,2021-08-26T02:01:01Z,2021-08-26T02:01:01Z,NONE,Perceptual hashes might be what you're after : http://phash.org,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",602585497,Integrate image content hashing,
https://github.com/simonw/datasette/issues/859#issuecomment-905904540,https://api.github.com/repos/simonw/datasette/issues/859,905904540,IC_kwDOBm6k_c41_wGc,2670795,brandonrobertz,2021-08-25T21:59:14Z,2021-08-25T21:59:55Z,CONTRIBUTOR,"I did two tests: one with 1000 5-30mb DBs and a second with 20 multi gig DBs. For the second, I created them like so:
`for i in {1..20}; do sqlite-generate db$i.db --tables ${i}00 --rows 100,2000 --columns 5,100 --pks 0 --fks 0; done`

This was for deciding whether to use lots of small DBs or to group things into a smaller number of bigger DBs. The second strategy wins.

By simply persisting the `_internal` DB to disk, I was able to avoid most of the performance issues I was experiencing previously. (To do this, I changed the `datasette/internal_db.py:init_internal_db` creates to if not exists, and changed the `_internal` DB instantiation in `datasette/app.py:Datasette.__init__` to a path with `is_mutable=True`.) Super rough, but the pages now load so I can continue testing ideas.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-905900807,https://api.github.com/repos/simonw/datasette/issues/859,905900807,IC_kwDOBm6k_c41_vMH,9599,simonw,2021-08-25T21:51:10Z,2021-08-25T21:51:10Z,OWNER,"10-20 minutes to populate `_internal`! How many databases and tables is that for?

I may have to rethink the `_internal` mechanism entirely. One possible alternative would be for the Datasette homepage to just show a list of available databases (maybe only if there are more than X connected) and then load in their metadata only the first time they are accessed.

I need to get my own stress testing rig setup for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/859#issuecomment-905899177,https://api.github.com/repos/simonw/datasette/issues/859,905899177,IC_kwDOBm6k_c41_uyp,2670795,brandonrobertz,2021-08-25T21:48:00Z,2021-08-25T21:48:00Z,CONTRIBUTOR,"Upon first stab, there's two issues here:
- DB/table/row counts (as discussed above). This isn't too bad if the DBs are actually above the MAX limit check.
- Populating the internal DB. On first load of a giant set of DBs, it can take 10-20 mins to populate. By altering datasette and persisting the internal DB to disk, this problem is vastly improved, but I'm sure this will cause problems elsewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/sqlite-utils/issues/323#issuecomment-905886797,https://api.github.com/repos/simonw/sqlite-utils/issues/323,905886797,IC_kwDOCGYnMM41_rxN,9599,simonw,2021-08-25T21:25:18Z,2021-08-25T21:25:18Z,OWNER,"As far as I can tell the Python `sqlite3` module doesn't actually have a mechanism for de-registering a custom SQL function.

This means that if I implement a mechanism whereby each call to `.convert()` registers a new SQL function with a random suffix (`convert_value_23424()` for example) those functions will stay registered - and if `.convert()` is called a large number of times the number of obsolete custom function registrations will grow without bounds.

For that reason, I'm going to `wontfix` this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",979627285,`table.convert()` method should clean up after itself,
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905206234,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,905206234,IC_kwDOEhK-wc419Fna,9599,simonw,2021-08-25T05:58:42Z,2021-08-25T05:58:42Z,MEMBER,"https://github.com/dogsheep/evernote-to-sqlite/blob/36a466f142e5bad52719851c2fbda0c05cd35b99/evernote_to_sqlite/utils.py#L34-L42

Not sure why I was round-tripping the `content_xml` like that - I will try not doing that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/dogsheep/evernote-to-sqlite/issues/13#issuecomment-905203570,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/13,905203570,IC_kwDOEhK-wc419E9y,9599,simonw,2021-08-25T05:51:22Z,2021-08-25T05:53:27Z,MEMBER,"The debugger showed me that it broke on a string that looked like this:
```xml
<?xml version=""1.0"" encoding=""UTF-8"" standalone=""no""?>
<!DOCTYPE en-note SYSTEM ""http://xml.evernote.com/pub/enml2.dtd"">
<en-note>
  <h1 title=""Q3 2018 Reflection & Development"">
    <span title=Q3 2018 Reflection & Development"">
      Q3 2018 Reflection & Development
    </span>
  </h1>
  ...
```
Yeah that is not valid XML!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978743426,xml.etree.ElementTree.ParseError: not well-formed (invalid token),
https://github.com/simonw/datasette/pull/1447#issuecomment-905097468,https://api.github.com/repos/simonw/datasette/issues/1447,905097468,IC_kwDOBm6k_c418rD8,9599,simonw,2021-08-25T01:28:53Z,2021-08-25T01:28:53Z,OWNER,"Good catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978614898,Remove underscore from search mode parameter name,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905043974,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905043974,IC_kwDOCGYnMM418eAG,9599,simonw,2021-08-24T23:33:44Z,2021-08-24T23:33:44Z,OWNER,Updated documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-data-from-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905022931,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905022931,IC_kwDOCGYnMM418Y3T,22429695,codecov[bot],2021-08-24T22:38:38Z,2021-08-24T23:27:26Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#321](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (db2dd6d) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/9258f4bd8450c951900de998a7bf81ca9b45a014?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9258f4b) will **increase** coverage by `0.16%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/321/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main     #321      +/-   ##
==========================================
+ Coverage   96.41%   96.58%   +0.16%     
==========================================
  Files           5        5              
  Lines        2206     2223      +17     
==========================================
+ Hits         2127     2147      +20     
+ Misses         79       76       -3     
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/321/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.69% <100.00%> (+0.42%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9258f4b...db2dd6d](https://codecov.io/gh/simonw/sqlite-utils/pull/321?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob",
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040902,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040902,IC_kwDOCGYnMM418dQG,9599,simonw,2021-08-24T23:25:03Z,2021-08-24T23:25:03Z,OWNER,I'm going to skip this test on windows.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob",
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905040307,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905040307,IC_kwDOCGYnMM418dGz,9599,simonw,2021-08-24T23:23:36Z,2021-08-24T23:23:36Z,OWNER,"https://discuss.python.org/t/pep-597-use-utf-8-for-default-text-file-encoding/1819 says:

> Currently, `TextIOWrapper` uses `locale.getpreferredencoding(False)` (hereinafter called “locale encoding”) when encoding is not specified.
> ...
> Package authors using macOS or Linux may forget that the default encoding
is not always UTF-8.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob",
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905039576,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905039576,IC_kwDOCGYnMM418c7Y,9599,simonw,2021-08-24T23:21:29Z,2021-08-24T23:21:29Z,OWNER,"Hah, the error here is actually:
```
>           assert result.exit_code == 1, result.output
E           AssertionError: 
E             
E           assert 0 == 1
E            +  where 0 = <Result okay>.exit_code
```
So I was expecting an error, but instead the command worked.

I suspect this is because on Windows the default character set may not be UTF-8?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob",
https://github.com/simonw/sqlite-utils/pull/321#issuecomment-905037323,https://api.github.com/repos/simonw/sqlite-utils/issues/321,905037323,IC_kwDOCGYnMM418cYL,9599,simonw,2021-08-24T23:15:29Z,2021-08-24T23:15:29Z,OWNER,"Huh, tests are failing but only on Windows!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978537855,"Ability to insert file contents as text, in addition to blob",
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905024066,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905024066,IC_kwDOCGYnMM418ZJC,66709385,pjamargh,2021-08-24T22:41:39Z,2021-08-24T22:41:39Z,NONE,"I'm happy with this functionality left the way you describe. In my case the data is homogeneous but other cases would work just by being consistent on the encoding. Thanks a lot, Simon!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021933,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905021933,IC_kwDOCGYnMM418Ynt,9599,simonw,2021-08-24T22:36:04Z,2021-08-24T22:36:04Z,OWNER,"> Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

I thought about supporting those different policies (with something like `--errors ignore`) but I feel like that's getting a little bit too deep into the weeds. Right now if you try to import an invalid file the behaviour is the same as for the `sqlite-utils insert` command (I added the same detailed error message):

```
Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte

The input you provided uses a character encoding other than utf-8.

You can fix this by passing the --encoding= option with the encoding of the file.

If you do not know the encoding, running 'file filename.csv' may tell you.

It's often worth trying: --encoding=latin-1
```
If someone has data that can't be translated to valid text using a known encoding, I'm happy leaving them to have to insert it into a `BLOB` column instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021047,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905021047,IC_kwDOCGYnMM418YZ3,9599,simonw,2021-08-24T22:33:48Z,2021-08-24T22:33:48Z,OWNER,"I had a few doubts about the design just now. Since `content_text` is supported as a special argument, an alternative way of handling the above would be:

    sqlite-utils insert-files /tmp/text.db files *.txt -c path -c content_text -c size

This does exactly the same thing as just using `--text` and not specifying any columns, because the actual implementation of `--text` is as follows:

https://github.com/simonw/sqlite-utils/blob/0c796cd945b146b7395ff5f553861400be503867/sqlite_utils/cli.py#L1851-L1855

But actually I think that's OK - ``--text`` is a useful shorthand that avoids you having to remember how to manually specify those columns with `-c`. So I'm going to leave the design as is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905021010,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905021010,IC_kwDOCGYnMM418YZS,66709385,pjamargh,2021-08-24T22:33:42Z,2021-08-24T22:33:42Z,NONE,"Oh, I misread. Yes some files will not be valid UTF-8, I'd throw a warning and continue (not adding that file) but if you want to get more elaborate you could allow to define a policy on what to do. Not adding the file, index binary content or use a conversion policy like the ones available on Python's decode.

From https://stackoverflow.com/questions/24616678/unicodedecodeerror-in-python-when-reading-a-file-how-to-ignore-the-error-and-ju :
 - 'ignore' ignores errors. Note that ignoring encoding errors can lead to data loss.
 - 'replace' causes a replacement marker (such as '?') to be inserted where there is malformed data.
 - 'surrogateescape' will represent any incorrect bytes as code points in the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These private code points will then be turned back into the same bytes when the surrogateescape error handler is used when writing data. This is useful for processing files in an unknown encoding.
 - 'xmlcharrefreplace' is only supported when writing to a file. Characters not supported by the encoding are replaced with the appropriate XML character reference &#nnn;.
 - 'backslashreplace'  (also only supported when writing) replaces unsupported characters with Python’s backslashed escape sequences.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013183,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905013183,IC_kwDOCGYnMM418We_,9599,simonw,2021-08-24T22:15:34Z,2021-08-24T22:15:34Z,OWNER,"Here's the error message I have working for invalid unicode:
```
sqlite-utils insert-files /tmp/text.db files *.txt --text
  [------------------------------------]    0%
Error: Could not read file '/Users/simon/Dropbox/Development/sqlite-utils/data.txt' as text

'utf-8' codec can't decode byte 0xe3 in position 83: invalid continuation byte
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905013162,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905013162,IC_kwDOCGYnMM418Weq,9599,simonw,2021-08-24T22:15:31Z,2021-08-24T22:15:31Z,OWNER,"I'm going to assume utf-8 but allow `--encoding` to be used to specify something different, since that option is already supported by other commands.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905003381,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905003381,IC_kwDOCGYnMM418UF1,66709385,pjamargh,2021-08-24T21:56:49Z,2021-08-24T21:56:49Z,NONE,"I was thinking that an approach could be making FILE_COLUMNS a generator (_get_file_columns(mode)) or you can just have a different set of columns (is there something else that makes sense to be changed on the text scenario?).

About UTF-8 I was referring to the encoding to use when reading files. This can be difficult to auto-detect but I believe that UTF-8 is pretty much the standard for text files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-905001586,https://api.github.com/repos/simonw/sqlite-utils/issues/319,905001586,IC_kwDOCGYnMM418Tpy,9599,simonw,2021-08-24T21:52:50Z,2021-08-24T21:52:50Z,OWNER,"Will need to re-title this section of the documentation: https://sqlite-utils.datasette.io/en/3.16/cli.html#inserting-binary-data-from-files - ""Inserting binary data from files"" will become ""Inserting data from files""

I'm OK with keeping the default as `BLOB` but I could add a `--text` option which stores the content as text instead.

If the text can't be stored as `utf-8` I'll probably raise an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/sqlite-utils/issues/319#issuecomment-904999850,https://api.github.com/repos/simonw/sqlite-utils/issues/319,904999850,IC_kwDOCGYnMM418TOq,9599,simonw,2021-08-24T21:49:08Z,2021-08-24T21:49:08Z,OWNER,This is a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976399638,[Enhancement] Please allow 'insert-files' to insert content as text.,
https://github.com/simonw/datasette/issues/859#issuecomment-904982056,https://api.github.com/repos/simonw/datasette/issues/859,904982056,IC_kwDOBm6k_c418O4o,2670795,brandonrobertz,2021-08-24T21:15:04Z,2021-08-24T21:15:30Z,CONTRIBUTOR,"I'm running into issues with this as well. All other pages seem to work with lots of DBs except the home page, which absolutely tanks. Would be willing to put some work into this, if there's been any kind of progress on concepts on how this ought to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/1446#issuecomment-904954530,https://api.github.com/repos/simonw/datasette/issues/1446,904954530,IC_kwDOBm6k_c418IKi,9599,simonw,2021-08-24T20:32:47Z,2021-08-24T20:32:47Z,OWNER,"Pasting that CSS into the styles editor in the developer tools on https://latest.datasette.io/ has the desired effect: footer at the bottom of the window unless the page is too long, in which case the footer is at the bottom of the scroll.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer,
https://github.com/simonw/datasette/issues/1446#issuecomment-904866495,https://api.github.com/repos/simonw/datasette/issues/1446,904866495,IC_kwDOBm6k_c417yq_,9599,simonw,2021-08-24T18:13:49Z,2021-08-24T18:13:49Z,OWNER,"OK, now the following optional CSS gives us a sticky footer:
```css
html, body {
  height: 100%;
}
body {
  display: flex;
  flex-direction: column;
}
.not-footer {
  flex: 1 0 auto;
}
footer {
  flex-shrink: 0;
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978357984,Modify base.html template to support optional sticky footer,
https://github.com/dogsheep/healthkit-to-sqlite/pull/13#issuecomment-904642396,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13,904642396,IC_kwDOC8tyDs41679c,32016596,FabianHertwig,2021-08-24T13:27:40Z,2021-08-24T13:28:26Z,NONE,This would fix #21 and make #22 obsolete.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743071410,SQLite does not have case sensitive columns,
https://github.com/dogsheep/healthkit-to-sqlite/pull/22#issuecomment-904641261,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/22,904641261,IC_kwDOC8tyDs4167rt,32016596,FabianHertwig,2021-08-24T13:26:20Z,2021-08-24T13:26:20Z,NONE,Did not see that #13 fixes the same issue in a similar way. You can decide which one to merge ;),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",978086284,Make sure that case-insensitive column names are unique,
https://github.com/simonw/datasette/issues/1445#issuecomment-904037087,https://api.github.com/repos/simonw/datasette/issues/1445,904037087,IC_kwDOBm6k_c414oLf,9599,simonw,2021-08-23T19:10:17Z,2021-08-23T19:10:17Z,OWNER,"Rather than trying to run that monstrosity in a single `union all` query, a better approach may be to use `fetch()` requests as seen in https://datasette.io/plugins/datasette-search-all","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table,
https://github.com/simonw/datasette/issues/1445#issuecomment-904036200,https://api.github.com/repos/simonw/datasette/issues/1445,904036200,IC_kwDOBm6k_c414n9o,9599,simonw,2021-08-23T19:08:54Z,2021-08-23T19:08:54Z,OWNER,"Figured out a query for searching across every column in every table! https://til.simonwillison.net/datasette/search-all-columns-trick#user-content-same-trick-for-the-entire-database

```sql
with tables as (
  select
    name as table_name
  from
    sqlite_master
  where
    type = 'table'
),
queries as (
  select
    'select ''' || tables.table_name || ''' as _table, rowid from ""' || tables.table_name || '"" where ' || group_concat(
      '""' || name || '"" like ''%'' || :search || ''%''',
      ' or '
    ) as query
  from
    pragma_table_info(tables.table_name),
    tables
  group by
    tables.table_name
)
select
  group_concat(query, ' union all ')
from
  queries
```
The SQL query this generates for larger databases is _extremely_ long - but it does seem to work for smaller databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table,
https://github.com/simonw/datasette/issues/1445#issuecomment-904027166,https://api.github.com/repos/simonw/datasette/issues/1445,904027166,IC_kwDOBm6k_c414lwe,9599,simonw,2021-08-23T18:56:20Z,2021-08-23T18:56:20Z,OWNER,A related but potentially even more useful ability would be running a search across every column of every table in a whole database. For anything less than a few 100MB this could be incredibly useful.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table,
https://github.com/simonw/datasette/issues/1445#issuecomment-904026253,https://api.github.com/repos/simonw/datasette/issues/1445,904026253,IC_kwDOBm6k_c414liN,9599,simonw,2021-08-23T18:54:49Z,2021-08-23T18:54:49Z,OWNER,"The bigger problem here is UI design. This feels like a pretty niche requirement to me, so adding a prominent search box to the table page (which already has the filters interface, plus the full-text search box for tables that have FTS configured) feels untidy.

I could tuck it away in the table cog menu, but that's a weird place for something like this to live.

Maybe add it as a new type of filter? Filters apply to specific columns though, so this would be the first filter that applied to _all_ columns - which doesn't really fit the existing filter interface very well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table,
https://github.com/simonw/datasette/issues/1445#issuecomment-904024939,https://api.github.com/repos/simonw/datasette/issues/1445,904024939,IC_kwDOBm6k_c414lNr,9599,simonw,2021-08-23T18:52:35Z,2021-08-23T18:52:35Z,OWNER,"The downside of the current implementation of this trick is that it only works for exact LIKE partial matches in a specific table - if you search for `dog cat` and `dog` appears in `title` but `cat` appears in `description` you won't get back that result.

I think that's fine though. If you want more advanced search there are other mechanisms you can use. This is meant to be a very quick and dirty starting point for exploring a brand new table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977323133,Ability to search for text across all columns in a table,
https://github.com/dogsheep/healthkit-to-sqlite/issues/21#issuecomment-903950096,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/21,903950096,IC_kwDOC8tyDs414S8Q,32016596,FabianHertwig,2021-08-23T17:00:59Z,2021-08-23T17:00:59Z,NONE,"I think the issue is that I have records like these:

```xml
 <Record type=""HKQuantityTypeIdentifierDietaryCholesterol"" sourceName=""MyFitnessPal"" sourceVersion=""35120"" unit=""mg"" creationDate=""2021-07-04 20:55:27 +0200"" startDate=""2021-07-04 20:55:00 +0200"" endDate=""2021-07-04 20:55:00 +0200"" value=""124"">
  <MetadataEntry key=""meal"" value=""Dinner""/>
  <MetadataEntry key=""Meal"" value=""Dinner""/>
 </Record>
```

And if sqlite is case insensitive, then `metadata_meal` and `metadata_Meal` result in the same column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",977128935,Duplicate Column,
https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288691,https://api.github.com/repos/simonw/sqlite-utils/issues/320,903288691,IC_kwDOCGYnMM411xdz,9599,simonw,2021-08-22T15:46:56Z,2021-08-22T15:46:56Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#schema-analyze-dump-and-save,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976405225,sqlite-utils memory --analyze option,
https://github.com/simonw/sqlite-utils/issues/320#issuecomment-903288430,https://api.github.com/repos/simonw/sqlite-utils/issues/320,903288430,IC_kwDOCGYnMM411xZu,9599,simonw,2021-08-22T15:44:55Z,2021-08-22T15:45:52Z,OWNER,"```
curl 'https://api.github.com/users/dogsheep/repos' | sqlite-utils memory - --analyze
```
```
stdin.id: (1/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.node_id: (2/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.name: (3/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.full_name: (4/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.private: (5/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 1

  Most common:
    13: 0

stdin.owner: (6/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 1

  Most common:
    13: {""login"": ""dogsheep"", ""id"": 53015001, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjUzMDE1MD...

stdin.html_url: (7/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.description: (8/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.fork: (9/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 1

  Most common:
    13: 0

stdin.url: (10/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.forks_url: (11/73)

  Total rows: 13
  Null rows: 0
  Blank rows: 0

  Distinct values: 13

stdin.keys_url: (12/73)

  Total rows: 13
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",976405225,sqlite-utils memory --analyze option,
https://github.com/simonw/datasette/issues/894#issuecomment-902375388,https://api.github.com/repos/simonw/datasette/issues/894,902375388,IC_kwDOBm6k_c41ySfc,9599,simonw,2021-08-20T02:07:53Z,2021-08-20T02:07:53Z,OWNER,I could add these sorting links to the cog menu for any `TEXT` columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/simonw/datasette/issues/894#issuecomment-902375088,https://api.github.com/repos/simonw/datasette/issues/894,902375088,IC_kwDOBm6k_c41ySaw,9599,simonw,2021-08-20T02:07:13Z,2021-08-20T02:07:26Z,OWNER,Maybe `?_sort_numeric=col` and `?_sort_numeric_desc=col` would be better here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by  by column cast to real,
https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902356871,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20,902356871,IC_kwDOC8tyDs41yN-H,9599,simonw,2021-08-20T01:12:48Z,2021-08-20T01:12:48Z,MEMBER,Also on `workout_points.workout_id` to speed up queries to show all points in a specific workout.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",975166271,Add index on workout_points.date,
https://github.com/dogsheep/healthkit-to-sqlite/issues/20#issuecomment-902355471,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/20,902355471,IC_kwDOC8tyDs41yNoP,9599,simonw,2021-08-20T01:09:07Z,2021-08-20T01:09:07Z,MEMBER,"Workaround:

    sqlite-utils create-index healthkit.db workout_points -- -date

See https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",975166271,Add index on workout_points.date,
https://github.com/dogsheep/twitter-to-sqlite/pull/49#issuecomment-902330301,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/49,902330301,IC_kwDODEm0Qs41yHe9,9599,simonw,2021-08-20T00:01:56Z,2021-08-20T00:01:56Z,MEMBER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681575714,"Document the use of --stop_after with favorites, refs #20",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329884,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902329884,IC_kwDODEm0Qs41yHYc,9599,simonw,2021-08-20T00:01:05Z,2021-08-20T00:01:05Z,MEMBER,Maybe Click changed something which meant that this broke things when it didn't used to?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902329455,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902329455,IC_kwDODEm0Qs41yHRv,9599,simonw,2021-08-19T23:59:56Z,2021-08-19T23:59:56Z,MEMBER,"This looks like the bug to me:

https://github.com/dogsheep/twitter-to-sqlite/blob/197e69cec40052c423a5ed071feb5f7cccea41b9/twitter_to_sqlite/cli.py#L239-L241

`type=str, default=False`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328760,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902328760,IC_kwDODEm0Qs41yHG4,9599,simonw,2021-08-19T23:57:41Z,2021-08-19T23:57:41Z,MEMBER,"Weird, added debug code and got this: `{'screen_name': 'simonw', 'count': 200, 'since_id': 'False', 'tweet_mode': 'extended'}` -  so maybe it's a `twitter-to-sqlite` bug where somehow the string `False` is being passed somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902328369,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902328369,IC_kwDODEm0Qs41yHAx,9599,simonw,2021-08-19T23:56:26Z,2021-08-19T23:56:26Z,MEMBER,"https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-user_timeline says the API has been replaced by the new v2 one, but it should still work - and the `since_id` parameter is still documented on that page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-902327457,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,902327457,IC_kwDODEm0Qs41yGyh,9599,simonw,2021-08-19T23:53:25Z,2021-08-19T23:53:25Z,MEMBER,I'm getting this too. Looking into it now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/simonw/datasette/issues/1426#issuecomment-902263367,https://api.github.com/repos/simonw/datasette/issues/1426,902263367,IC_kwDOBm6k_c41x3JH,9599,simonw,2021-08-19T21:33:51Z,2021-08-19T21:36:28Z,OWNER,"I was worried about if it's possible to allow access to `/fixtures` but deny access to `/fixtures?sql=...`

From various answers on Stack Overflow it looks like this should handle that:

```
User-agent: *
Disallow: /fixtures?
```
I could use this for tables too - it may well be OK to access table index pages while still avoiding pagination, facets etc. I think this should block both query strings and row pages while allowing the table page itself:
```
User-agent: *
Disallow: /fixtures/searchable?
Disallow: /fixtures/searchable/*
```
Could even accompany that with a `sitemap.xml` that explicitly lists all of the tables - which would mean adding sitemaps to Datasette core too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/simonw/datasette/issues/1426#issuecomment-902260338,https://api.github.com/repos/simonw/datasette/issues/1426,902260338,IC_kwDOBm6k_c41x2Zy,9599,simonw,2021-08-19T21:28:25Z,2021-08-19T21:29:40Z,OWNER,"Actually it looks like you can send a `sitemap.xml` to Google using an unauthenticated GET request to:

    https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP

According to https://developers.google.com/search/docs/advanced/sitemaps/build-sitemap","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/simonw/datasette/issues/1426#issuecomment-902260799,https://api.github.com/repos/simonw/datasette/issues/1426,902260799,IC_kwDOBm6k_c41x2g_,9599,simonw,2021-08-19T21:29:13Z,2021-08-19T21:29:13Z,OWNER,"Bing's equivalent is: https://www.bing.com/webmasters/help/Sitemaps-3b5cf6ed

    http://www.bing.com/ping?sitemap=FULL_URL_OF_SITEMAP","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/simonw/datasette/issues/1443#issuecomment-902258509,https://api.github.com/repos/simonw/datasette/issues/1443,902258509,IC_kwDOBm6k_c41x19N,9599,simonw,2021-08-19T21:25:07Z,2021-08-19T21:25:07Z,OWNER,https://docs.datasette.io/en/latest/internals.html#databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974995592,datasette.databases should be a documented property,
https://github.com/simonw/datasette/pull/1434#issuecomment-902254712,https://api.github.com/repos/simonw/datasette/issues/1434,902254712,IC_kwDOBm6k_c41x1B4,9599,simonw,2021-08-19T21:18:31Z,2021-08-19T21:18:57Z,OWNER,"I deployed a demo to https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app using the mechanism from #1442.

e.g. demo here: https://datasette-latest-query-info-j7hipcg4aq-uc.a.run.app/fixtures?sql=select+*+from+searchable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970463436,Enrich arbitrary query results with foreign key links and column descriptions,
https://github.com/simonw/datasette/issues/1415#issuecomment-902251316,https://api.github.com/repos/simonw/datasette/issues/1415,902251316,IC_kwDOBm6k_c41x0M0,9599,simonw,2021-08-19T21:14:15Z,2021-08-19T21:14:15Z,OWNER,"https://github.com/ahmetb/cloud-run-faq#how-do-i-continuously-deploy-to-cloud-run suggests the following:

> - `roles/run.admin` to deploy applications
> - `roles/iam.serviceAccountUser` on the service account that your app will use

It also links to https://cloud.google.com/run/docs/reference/iam/roles","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun,
https://github.com/simonw/datasette/issues/1415#issuecomment-902250361,https://api.github.com/repos/simonw/datasette/issues/1415,902250361,IC_kwDOBm6k_c41xz95,9599,simonw,2021-08-19T21:12:28Z,2021-08-19T21:12:28Z,OWNER,I would love to know this too! I always find figuring out minimal permissions to be really difficult.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun,
https://github.com/simonw/datasette/issues/1442#issuecomment-902243498,https://api.github.com/repos/simonw/datasette/issues/1442,902243498,IC_kwDOBm6k_c41xySq,9599,simonw,2021-08-19T21:04:01Z,2021-08-19T21:04:01Z,OWNER,That successfully deployed to https://datasette-latest-deploy-this-branch-j7hipcg4aq-uc.a.run.app/ even though the tests failed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos,
https://github.com/simonw/datasette/issues/1442#issuecomment-902239215,https://api.github.com/repos/simonw/datasette/issues/1442,902239215,IC_kwDOBm6k_c41xxPv,9599,simonw,2021-08-19T20:56:46Z,2021-08-19T20:56:46Z,OWNER,"I'm going to only run the tests if it's a push to `main` - that way I can ship demo branches really quickly, even if they don't yet have passing tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos,
https://github.com/simonw/datasette/issues/1442#issuecomment-902235714,https://api.github.com/repos/simonw/datasette/issues/1442,902235714,IC_kwDOBm6k_c41xwZC,9599,simonw,2021-08-19T20:50:38Z,2021-08-19T20:50:38Z,OWNER,"Would this allow anyone to push a PR to this repo that would result in their code being deployed against my Cloud Run account? I'm reasonably confident that it would not, since the secrets would not be visible to their PR branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos,
https://github.com/simonw/datasette/issues/1442#issuecomment-902231018,https://api.github.com/repos/simonw/datasette/issues/1442,902231018,IC_kwDOBm6k_c41xvPq,9599,simonw,2021-08-19T20:42:08Z,2021-08-19T20:42:08Z,OWNER,If I get this working I should document it on https://docs.datasette.io/en/stable/contributing.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos,
https://github.com/simonw/datasette/issues/1442#issuecomment-902217726,https://api.github.com/repos/simonw/datasette/issues/1442,902217726,IC_kwDOBm6k_c41xr_-,9599,simonw,2021-08-19T20:21:47Z,2021-08-19T20:21:47Z,OWNER,I think the neatest way to implement this would be for the `on -> push -> branches` list to be the list of branches that should be deployed in this way. The rest of the code can react to that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos,
https://github.com/simonw/datasette/issues/1442#issuecomment-902191150,https://api.github.com/repos/simonw/datasette/issues/1442,902191150,IC_kwDOBm6k_c41xlgu,9599,simonw,2021-08-19T19:43:05Z,2021-08-19T19:43:59Z,OWNER,"Maybe as simple as teaching https://github.com/simonw/datasette/blob/main/.github/workflows/deploy-latest.yml to run on pushes to ALL branches:

https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L3-L6

And then quit early if the branch is not in some allow-list.

If it IS in the allow-list, use the name of the branch to dynamically construct the name of the Cloud Run service here: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/.github/workflows/deploy-latest.yml#L60

Need to skip the documentation build and deployment stuff for other branches though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974987856,Mechanism to cause specific branches to deploy their own demos,
https://github.com/simonw/datasette/issues/1293#issuecomment-901475812,https://api.github.com/repos/simonw/datasette/issues/1293,901475812,IC_kwDOBm6k_c41u23k,9599,simonw,2021-08-18T22:41:19Z,2021-08-18T22:41:19Z,OWNER,"> Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?

I'm going to do this, and call the Python library `sqlite-explain`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-901452199,https://api.github.com/repos/simonw/sqlite-utils/issues/37,901452199,IC_kwDOCGYnMM41uxGn,9599,simonw,2021-08-18T21:48:57Z,2021-08-18T21:48:57Z,OWNER,"I did a bunch of work on this in #266. The library is now pretty thoroughly typed, and I even found a couple of bugs using `mypy` along the way: #313 and #315.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440752,https://api.github.com/repos/simonw/sqlite-utils/issues/318,901440752,IC_kwDOCGYnMM41uuTw,9599,simonw,2021-08-18T21:25:30Z,2021-08-18T21:25:30Z,OWNER,"Some questions:

- Should this support compression formats other than gzip?
- Should `memory` learn to auto-detect gzipped data?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974067156,Research: handle gzipped CSV directly,
https://github.com/simonw/sqlite-utils/issues/318#issuecomment-901440207,https://api.github.com/repos/simonw/sqlite-utils/issues/318,901440207,IC_kwDOCGYnMM41uuLP,9599,simonw,2021-08-18T21:24:28Z,2021-08-18T21:24:49Z,OWNER,"Something like this then:

    sqlite-utils file.db ""select * from t"" --csv --gz > t.csv.gz

Maybe add a `-o t.csv.gz` option too so you don't have to use a `>`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",974067156,Research: handle gzipped CSV directly,
https://github.com/simonw/sqlite-utils/issues/295#issuecomment-901403298,https://api.github.com/repos/simonw/sqlite-utils/issues/295,901403298,IC_kwDOCGYnMM41ulKi,9599,simonw,2021-08-18T20:19:04Z,2021-08-18T20:19:04Z,OWNER,"Thanks, this was a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",934123448,Insert with --tsv and --no-headers give error about --nl arguments,
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901399139,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901399139,IC_kwDOCGYnMM41ukJj,9599,simonw,2021-08-18T20:12:34Z,2021-08-18T20:13:12Z,OWNER,"Documentation for `table.search(..., quote=True)`: https://sqlite-utils.datasette.io/en/latest/python-api.html#searching-with-table-search

In the API reference: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.search

And for the CLI `--quote` option: https://sqlite-utils.datasette.io/en/latest/cli.html#executing-searches","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option",
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901398216,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901398216,IC_kwDOCGYnMM41uj7I,9599,simonw,2021-08-18T20:11:01Z,2021-08-18T20:11:01Z,OWNER,"```
% sqlite-utils search fixtures.db searchable 'dog""'
Error: malformed MATCH expression: [dog""]

Try running this again with the --quote option
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option",
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901390635,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901390635,IC_kwDOCGYnMM41uiEr,9599,simonw,2021-08-18T19:58:53Z,2021-08-18T19:58:53Z,OWNER,"```
sqlite-utils search fixtures.db searchable 'dog""'
Error: malformed MATCH expression: [dog""]
```
This error message could suggest retrying with `--quote`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option",
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901379930,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901379930,IC_kwDOCGYnMM41ufda,9599,simonw,2021-08-18T19:40:38Z,2021-08-18T19:40:38Z,OWNER,Also add `sqlite-utils search ... --quote` option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option",
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901353345,https://api.github.com/repos/simonw/sqlite-utils/issues/246,901353345,IC_kwDOCGYnMM41uY-B,9599,simonw,2021-08-18T18:57:13Z,2021-08-18T18:57:13Z,OWNER,More documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-characters-for-use-in-search,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings,
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338841,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901338841,IC_kwDOCGYnMM41uVbZ,9599,simonw,2021-08-18T18:33:26Z,2021-08-18T18:45:12Z,OWNER,"I think I'll do this as an optional `table.search(..., escape=True)` parameter.

Actually I'll do `quote=True` for consistency with the new `db.quote_fts()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option",
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-901345800,https://api.github.com/repos/simonw/sqlite-utils/issues/246,901345800,IC_kwDOCGYnMM41uXII,9599,simonw,2021-08-18T18:44:48Z,2021-08-18T18:44:48Z,OWNER,"The `db.quote_fts(value)` method from #247 can now be used for this - documentation here: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.quote_fts

I'll be adding further improvements relating to this (a `table.search(q, quote=True)` parameter) in #296.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings,
https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901344634,https://api.github.com/repos/simonw/sqlite-utils/issues/247,901344634,IC_kwDOCGYnMM41uW16,22429695,codecov[bot],2021-08-18T18:42:54Z,2021-08-18T18:42:54Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#247](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (af989af) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/1fe73c898b44695052f1a9ca832818d50cecf662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1fe73c8) will **decrease** coverage by `0.03%`.
> The diff coverage is `85.71%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/247/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main     #247      +/-   ##
==========================================
- Coverage   96.28%   96.24%   -0.04%     
==========================================
  Files           5        5              
  Lines        2179     2186       +7     
==========================================
+ Hits         2098     2104       +6     
- Misses         81       82       +1     
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/247/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.84% <85.71%> (-0.08%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1fe73c8...af989af](https://codecov.io/gh/simonw/sqlite-utils/pull/247?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832687563,FTS quote functionality from datasette,
https://github.com/simonw/sqlite-utils/pull/247#issuecomment-901338988,https://api.github.com/repos/simonw/sqlite-utils/issues/247,901338988,IC_kwDOCGYnMM41uVds,9599,simonw,2021-08-18T18:33:39Z,2021-08-18T18:33:39Z,OWNER,This was also requested in #296.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832687563,FTS quote functionality from datasette,
https://github.com/simonw/sqlite-utils/issues/296#issuecomment-901338356,https://api.github.com/repos/simonw/sqlite-utils/issues/296,901338356,IC_kwDOCGYnMM41uVT0,9599,simonw,2021-08-18T18:32:39Z,2021-08-18T18:32:39Z,OWNER,This is a good call. I have a fix for this in Datasette but it's not in `sqlite-utils` yet: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L824-L835,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944326512,"`table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option",
https://github.com/simonw/sqlite-utils/issues/317#issuecomment-901337305,https://api.github.com/repos/simonw/sqlite-utils/issues/317,901337305,IC_kwDOCGYnMM41uVDZ,9599,simonw,2021-08-18T18:30:59Z,2021-08-18T18:30:59Z,OWNER,"I'm just going to remove this - I added it when the library was mostly undocumented, but it has comprehensive documentation now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972827346,Link to a better example on docs index,
https://github.com/simonw/datasette/issues/1439#issuecomment-900715375,https://api.github.com/repos/simonw/datasette/issues/1439,900715375,IC_kwDOBm6k_c41r9Nv,9599,simonw,2021-08-18T00:15:28Z,2021-08-18T00:15:28Z,OWNER,"Maybe I should use `-/` to encode forward slashes too, to defend against any ASGI servers that might not implement `raw_path` correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0,
https://github.com/simonw/datasette/issues/1439#issuecomment-900714630,https://api.github.com/repos/simonw/datasette/issues/1439,900714630,IC_kwDOBm6k_c41r9CG,9599,simonw,2021-08-18T00:13:33Z,2021-08-18T00:13:33Z,OWNER,"The documentation should definitely cover how table names become URLs, in case any third party code needs to be able to calculate this themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0,
https://github.com/simonw/datasette/issues/1439#issuecomment-900712981,https://api.github.com/repos/simonw/datasette/issues/1439,900712981,IC_kwDOBm6k_c41r8oV,9599,simonw,2021-08-18T00:09:59Z,2021-08-18T00:12:32Z,OWNER,"So given the original examples, a table called `table.csv` would have the following URLs:

- `/db/table-.csv` - the HTML version
- `/db/table-.csv.csv` - the CSV version
- `/db/table-.csv.json` - the JSON version

And if for some horific reason you had a table with the name `/db/table-.csv.csv` (so `/db/` was the first part of the actual table name in SQLite) the URLs would look like this:

- `/db/%2Fdb%2Ftable---.csv-.csv` - the HTML version
- `/db/%2Fdb%2Ftable---.csv-.csv.csv` - the CSV version
- `/db/%2Fdb%2Ftable---.csv-.csv.json` - the JSON version","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0,
https://github.com/simonw/datasette/issues/1439#issuecomment-900711967,https://api.github.com/repos/simonw/datasette/issues/1439,900711967,IC_kwDOBm6k_c41r8Yf,9599,simonw,2021-08-18T00:08:09Z,2021-08-18T00:08:09Z,OWNER,"Here's an alternative I just made up which I'm calling ""dot dash"" encoding:

```python
def dot_dash_encode(s):
    return s.replace(""-"", ""--"").replace(""."", ""-."")

def dot_dash_decode(s):
    return s.replace(""-."", ""."").replace(""--"", ""-"")
```
And some examples:
```python
for example in (
  ""hello"",
  ""hello.csv"",
  ""hello-and-so-on.csv"",
  ""hello-.csv"",
  ""hello--and--so--on-.csv"",
  ""hello.csv."",
  ""hello.csv.-"",
  ""hello.csv.--"",
):
    print(example)
    print(dot_dash_encode(example))
    print(example == dot_dash_decode(dot_dash_encode(example)))
    print()
```
Outputs:
```
hello
hello
True

hello.csv
hello-.csv
True

hello-and-so-on.csv
hello--and--so--on-.csv
True

hello-.csv
hello---.csv
True

hello--and--so--on-.csv
hello----and----so----on---.csv
True

hello.csv.
hello-.csv-.
True

hello.csv.-
hello-.csv-.--
True

hello.csv.--
hello-.csv-.----
True
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0,
https://github.com/simonw/datasette/issues/1439#issuecomment-900709703,https://api.github.com/repos/simonw/datasette/issues/1439,900709703,IC_kwDOBm6k_c41r71H,9599,simonw,2021-08-18T00:03:09Z,2021-08-18T00:03:09Z,OWNER,"But... what if I invent my own escaping scheme?

I actually did this once before, in https://github.com/simonw/datasette/commit/9fdb47ca952b93b7b60adddb965ea6642b1ff523 - while I was working on porting Datasette to ASGI in https://github.com/simonw/datasette/issues/272#issuecomment-494192779 because ASGI didn't yet have the `raw_path` mechanism.

I could bring that back - it looked like this:

```
    ""table/and/slashes"" => ""tableU+002FandU+002Fslashes""
    ""~table"" => ""U+007Etable""
    ""+bobcats!"" => ""U+002Bbobcats!""
    ""U+007Etable"" => ""UU+002B007Etable""
```
But I didn't particularly like it - it was quite verbose.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0,
https://github.com/simonw/datasette/issues/1439#issuecomment-900705226,https://api.github.com/repos/simonw/datasette/issues/1439,900705226,IC_kwDOBm6k_c41r6vK,9599,simonw,2021-08-17T23:50:32Z,2021-08-17T23:50:47Z,OWNER,"An alternative solution would be to use some form of escaping for the characters that form the name of the table.

The obvious way to do this would be URL-encoding - but it doesn't hold for `.` characters. The hex for that is `%2E` but watch what happens with that in a URL:

```
# Against Cloud Run:
curl -s 'https://datasette.io/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path
 'path': '/-/asgi-scope/foo/bar/baz.',
 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz.',
 'root_path': '',
# Against Vercel:
curl -s 'https://til.simonwillison.net/-/asgi-scope/foo/bar%2Fbaz%2E' | rg path
 'path': '/-/asgi-scope/foo/bar%2Fbaz%2E',
 'raw_path': b'/-/asgi-scope/foo/bar%2Fbaz%2E',
 'root_path': '',
```
Surprisingly in this case Vercel DOES keep it intact, but Cloud Run does not.

It's still no good though: I need a solution that works on Vercel, Cloud Run and every other potential hosting provider too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0,
https://github.com/simonw/datasette/issues/1439#issuecomment-900699670,https://api.github.com/repos/simonw/datasette/issues/1439,900699670,IC_kwDOBm6k_c41r5YW,9599,simonw,2021-08-17T23:34:23Z,2021-08-17T23:34:23Z,OWNER,"The challenge comes down to telling the difference between the following:

- `/db/table` - an HTML table page
- `/db/table.csv` - the CSV version of `/db/table`
- `/db/table.csv` - no this one is actually a database table called `table.csv`
- `/db/table.csv.csv` - the CSV version of `/db/table.csv`
- `/db/table.csv.csv.csv` and so on...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",973139047,Rethink how .ext formats (v.s. ?_format=) works before 1.0,
https://github.com/simonw/datasette/issues/1438#issuecomment-900690998,https://api.github.com/repos/simonw/datasette/issues/1438,900690998,IC_kwDOBm6k_c41r3Q2,9599,simonw,2021-08-17T23:11:16Z,2021-08-17T23:12:25Z,OWNER,"I have completely failed to replicate this initial bug - but it's still there on the `thesession.vercel.app` deployment (even though my own deployments to Vercel do not exhibit it). Here's a one-liner to replicate it against that deployment:

`curl -s 'https://thesession.vercel.app/thesession?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22' | rg '.csv'`

Whit outputs this:

`<p class=""export-links"">This data as <a href=""/thesession.json?sql=select * from tunes where name like &#34;%wise maid%&#34;"">json</a>, <a href=""/thesession.csv?sql=select * from tunes where name like &#34;%wise maid%&#34;&amp;_size=max"">CSV</a></p>`

It looks like, rather than being URL-encoded, the original query string is somehow making it through to Jinja and then being auto-escaped there.

The weird thing is that the equivalent query executed against my `til.simonwillison.net` Vercel instance does this:

`curl -s 'https://til.simonwillison.net/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25a%25%22' | rg '.csv'`

`<p class=""export-links"">This data as <a href=""/fixtures.json?sql=select%20*%20from%20searchable%20where%20text1%20like%20%22%25a%25%22"">json</a>, <a href=""/fixtures.csv?sql=select%20*%20from%20searchable%20where%20text1%20like%20%22%25a%25%22&amp;_size=max"">CSV</a></p>`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,
https://github.com/simonw/datasette/issues/1438#issuecomment-900681413,https://api.github.com/repos/simonw/datasette/issues/1438,900681413,IC_kwDOBm6k_c41r07F,9599,simonw,2021-08-17T22:47:44Z,2021-08-17T22:47:44Z,OWNER,I deployed another copy of `fixtures.db` on Vercel at https://til.simonwillison.net/fixtures so I can compare it with `fixtures.db` on Cloud Run at https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,
https://github.com/simonw/datasette/issues/1438#issuecomment-900518343,https://api.github.com/repos/simonw/datasette/issues/1438,900518343,IC_kwDOBm6k_c41rNHH,9599,simonw,2021-08-17T18:04:42Z,2021-08-17T18:04:42Z,OWNER,Here's how `request.query_string` works: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/asgi.py#L86-L88,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,
https://github.com/simonw/datasette/issues/1438#issuecomment-900516826,https://api.github.com/repos/simonw/datasette/issues/1438,900516826,IC_kwDOBm6k_c41rMva,9599,simonw,2021-08-17T18:02:27Z,2021-08-17T18:02:27Z,OWNER,"The key difference I can spot between Vercel and Cloud Run is that `+` in a query string gets converted to `%20` by Vercel before it gets to my app, but does not for Cloud Run:
```
# Vercel
~ % curl -s 'https://til.simonwillison.net/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'sql=select%20*%20from%20tunes%20where%20name%20like%20%22%25'
                 b'wise%20maid%25%22%0D%0A',
 'raw_path': b'/-/asgi-scope',

# Cloud Run
~ % curl -s 'https://latest-with-plugins.datasette.io/-/asgi-scope?sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%22%0D%0A' | rg 'query_string' -C 2
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'sql=select+*+from+tunes+where+name+like+%22%25wise+maid%25%2'
                 b'2%0D%0A',
 'raw_path': b'/-/asgi-scope',
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,
https://github.com/simonw/datasette/issues/1438#issuecomment-900513267,https://api.github.com/repos/simonw/datasette/issues/1438,900513267,IC_kwDOBm6k_c41rL3z,9599,simonw,2021-08-17T17:57:05Z,2021-08-17T17:57:05Z,OWNER,"I'm having trouble replicating this bug outside of Vercel. Against Cloud Run: view-source:https://latest.datasette.io/fixtures?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22

The HTML here is:

```html
<p class=""export-links"">This data as
  <a href=""/fixtures.json?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22"">json</a>, 
 ...
  <a href=""/fixtures.csv?sql=select+*+from+searchable+where+text1+like+%22%25cat%25%22&amp;_size=max"">CSV</a>
</p>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,
https://github.com/simonw/datasette/issues/1438#issuecomment-900502364,https://api.github.com/repos/simonw/datasette/issues/1438,900502364,IC_kwDOBm6k_c41rJNc,9599,simonw,2021-08-17T17:40:41Z,2021-08-17T17:40:41Z,OWNER,Bug is likely in `path_with_format` itself: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/utils/__init__.py#L710-L729,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,
https://github.com/simonw/datasette/issues/1438#issuecomment-900500824,https://api.github.com/repos/simonw/datasette/issues/1438,900500824,IC_kwDOBm6k_c41rI1Y,9599,simonw,2021-08-17T17:38:16Z,2021-08-17T17:38:16Z,OWNER,"Relevant template code: https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/templates/query.html#L71

`renderers` comes from here: https://github.com/simonw/datasette/blob/2883098770fc66e50183b2b231edbde20848d4d6/datasette/views/base.py#L593-L608","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",972918533,Query page .csv and .json links are not correctly URL-encoded on Vercel under unknown specific conditions,
https://github.com/simonw/datasette/issues/1293#issuecomment-899915829,https://api.github.com/repos/simonw/datasette/issues/1293,899915829,IC_kwDOBm6k_c41o6A1,9599,simonw,2021-08-17T01:02:35Z,2021-08-17T01:02:35Z,OWNER,"New approach: this time I'm building a simplified executor for the bytecode operations themselves.
```python
def execute_operations(operations, max_iterations = 100, trace=None):
    trace = trace or (lambda *args: None)
    registers: Dict[int, Any] = {}
    cursors: Dict[int, Tuple[str, Dict]] = {}
    instruction_pointer = 0
    iterations = 0
    result_row = None
    while True:
        iterations += 1
        if iterations > max_iterations:
            break
        operation = operations[instruction_pointer]
        trace(instruction_pointer, dict(operation))
        opcode = operation[""opcode""]
        if opcode == ""Init"":
            if operation[""p2""] != 0:
                instruction_pointer = operation[""p2""]
                continue
            else:
                instruction_pointer += 1
                continue
        elif opcode == ""Goto"":
            instruction_pointer = operation[""p2""]
            continue
        elif opcode == ""Halt"":
            break
        elif opcode == ""OpenRead"":
            cursors[operation[""p1""]] = (""database_table"", {
                ""rootpage"": operation[""p2""],
                ""connection"": operation[""p3""],
            })
        elif opcode == ""OpenEphemeral"":
            cursors[operation[""p1""]] = (""ephemeral"", {
                ""num_columns"": operation[""p2""],
                ""index_keys"": [],
            })
        elif opcode == ""MakeRecord"":
            registers[operation[""p3""]] = (""MakeRecord"", {
                ""registers"": list(range(operation[""p1""] + operation[""p2""]))
            })
        elif opcode == ""IdxInsert"":
            record = registers[operation[""p2""]]
            cursors[operation[""p1""]][1][""index_keys""].append(record)
        elif opcode == ""Rowid"":
            registers[operation[""p2""]] = (""rowid"", {
                ""table"": operation[""p1""]
            })
        elif opcode == ""Sequence"":
            registers[operation[""p2""]] = (""sequence"", {
                ""next_from_cursor"": operation[""p1""]
            })
        elif opcode == ""Column"":
            registers[operation[""p3""]] = (""column"", {
                ""cursor"": operation[""p1""],
                ""column_offset"": operation[""p2""]
            })
        elif opcode == ""ResultRow"":
            p1 = operation[""p1""]
            p2 = operation[""p2""]
            trace(""ResultRow: "", list(range(p1, p1 + p2)), registers)
            result_row = [registers.get(i) for i in range(p1, p1 + p2)]
        elif opcode == ""Integer"":
            registers[operation[""p2""]] = (""Integer"", operation[""p1""])
        elif opcode == ""String8"":
            registers[operation[""p2""]] = (""String"", operation[""p4""])
        instruction_pointer += 1
    return {""registers"": registers, ""cursors"": cursors, ""result_row"": result_row}
```
Results are promising!
```
execute_operations(db.execute(""explain select 'hello', 55, rowid, * from searchable"").fetchall())

{'registers': {1: ('String', 'hello'),
  2: ('Integer', 55),
  3: ('rowid', {'table': 0}),
  4: ('rowid', {'table': 0}),
  5: ('column', {'cursor': 0, 'column_offset': 1}),
  6: ('column', {'cursor': 0, 'column_offset': 2}),
  7: ('column', {'cursor': 0, 'column_offset': 3})},
 'cursors': {0: ('database_table', {'rootpage': 32, 'connection': 0})},
 'result_row': [('String', 'hello'),
  ('Integer', 55),
  ('rowid', {'table': 0}),
  ('rowid', {'table': 0}),
  ('column', {'cursor': 0, 'column_offset': 1}),
  ('column', {'cursor': 0, 'column_offset': 2}),
  ('column', {'cursor': 0, 'column_offset': 3})]}
```
Here's what happens with a union across three tables:
```
execute_operations(db.execute(f""""""
explain select data as content from binary_data
union
select pk as content from complex_foreign_keys
union
select name as content from facet_cities
""""""}).fetchall())

{'registers': {1: ('column', {'cursor': 4, 'column_offset': 0}),
  2: ('MakeRecord', {'registers': [0, 1, 2, 3]}),
  3: ('column', {'cursor': 0, 'column_offset': 1}),
  4: ('column', {'cursor': 3, 'column_offset': 0})},
 'cursors': {3: ('ephemeral',
   {'num_columns': 1,
    'index_keys': [('MakeRecord', {'registers': [0, 1]}),
     ('MakeRecord', {'registers': [0, 1]}),
     ('MakeRecord', {'registers': [0, 1, 2, 3]})]}),
  2: ('database_table', {'rootpage': 44, 'connection': 0}),
  4: ('database_table', {'rootpage': 24, 'connection': 0}),
  0: ('database_table', {'rootpage': 42, 'connection': 0})},
 'result_row': [('column', {'cursor': 3, 'column_offset': 0})]}
```
Note how the result_row refers to cursor 3, which is an ephemeral table which had three different sets of `MakeRecord` index keys assigned to it - indicating that the output column is NOT from the same underlying table source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1423#issuecomment-899749881,https://api.github.com/repos/simonw/datasette/issues/1423,899749881,IC_kwDOBm6k_c41oRf5,9599,simonw,2021-08-16T19:07:02Z,2021-08-16T19:07:02Z,OWNER,"Demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2

<img width=""686"" alt=""fixtures__compound_three_primary_keys__1_001_rows"" src=""https://user-images.githubusercontent.com/9599/129616596-cc51b668-7cb8-482f-9e20-e0d8ca4b71be.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1423#issuecomment-899744109,https://api.github.com/repos/simonw/datasette/issues/1423,899744109,IC_kwDOBm6k_c41oQFt,9599,simonw,2021-08-16T18:58:29Z,2021-08-16T18:58:29Z,OWNER,"I didn't bother with the tooltip, just the visible display if `?_facet_size=max`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1293#issuecomment-898961535,https://api.github.com/repos/simonw/datasette/issues/1293,898961535,IC_kwDOBm6k_c41lRB_,9599,simonw,2021-08-14T21:37:24Z,2021-08-14T21:37:24Z,OWNER,Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898936068,https://api.github.com/repos/simonw/datasette/issues/1293,898936068,IC_kwDOBm6k_c41lK0E,9599,simonw,2021-08-14T17:44:54Z,2021-08-14T17:44:54Z,OWNER,"Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29

That one shows `VColumn` instead of `Column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898933865,https://api.github.com/repos/simonw/datasette/issues/1293,898933865,IC_kwDOBm6k_c41lKRp,9599,simonw,2021-08-14T17:27:16Z,2021-08-14T17:28:29Z,OWNER,"Maybe I split this out into a separate Python library that gets tested against *every* SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?

Datasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out.

I could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898913629,https://api.github.com/repos/simonw/datasette/issues/1293,898913629,IC_kwDOBm6k_c41lFVd,9599,simonw,2021-08-14T16:14:12Z,2021-08-14T16:14:12Z,OWNER,I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898913554,https://api.github.com/repos/simonw/datasette/issues/1293,898913554,IC_kwDOBm6k_c41lFUS,9599,simonw,2021-08-14T16:13:40Z,2021-08-14T16:13:40Z,OWNER,"I think I need to care about the following:

- `ResultRow` and `Column` for the final result
- `OpenRead` for opening tables
- `OpenEphemeral` then `MakeRecord` and `IdxInsert` for writing records into ephemeral tables

`Column` may reference either a table (from `OpenRead`) or an ephemeral table (from `OpenEphemeral`).

That *might* be enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/sqlite-utils/issues/316#issuecomment-898824020,https://api.github.com/repos/simonw/sqlite-utils/issues/316,898824020,IC_kwDOCGYnMM41kvdU,9599,simonw,2021-08-14T05:12:23Z,2021-08-14T05:12:23Z,OWNER,No visible backticks on https://sqlite-utils.datasette.io/en/latest/reference.html any more.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970320615,Fix visible backticks on reference page,
https://github.com/simonw/datasette/issues/1293#issuecomment-898788262,https://api.github.com/repos/simonw/datasette/issues/1293,898788262,IC_kwDOBm6k_c41kmum,9599,simonw,2021-08-14T01:22:26Z,2021-08-14T01:51:08Z,OWNER,"Tried a more complicated query:
```sql
explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101
```
Here's the explain:
```
sqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101
   ...> ;
addr  opcode         p1    p2    p3    p4             p5  comment      
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     41    0                    00  Start at 41  
1     OpenEphemeral  2     6     0     k(1,-B)        00  nColumn=6    
2     Integer        101   1     0                    00  r[1]=101; LIMIT counter
3     OpenRead       0     32    0     4              00  root=32 iDb=0; searchable
4     Integer        16    3     0                    00  r[3]=16; return address
5     Once           0     16    0                    00               
6     OpenEphemeral  3     1     0     k(1,)          00  nColumn=1; Result of SELECT 1
7     VOpen          1     0     0     vtab:7FCBCA72BE80  00               
8     Function0      1     7     6     unknown(-1)    01  r[6]=func(r[7])
9     Integer        5     4     0                    00  r[4]=5       
10    Integer        1     5     0                    00  r[5]=1       
11    VFilter        1     16    4                    00  iplan=r[4] zplan=''
12      Rowid          1     8     0                    00  r[8]=rowid   
13      MakeRecord     8     1     9     C              00  r[9]=mkrec(r[8])
14      IdxInsert      3     9     8     1              00  key=r[9]     
15    VNext          1     12    0                    00               
16    Return         3     0     0                    00               
17    Rewind         3     33    0                    00               
18      Column         3     0     2                    00  r[2]=        
19      IsNull         2     32    0                    00  if r[2]==NULL goto 32
20      SeekRowid      0     32    2                    00  intkey=r[2]  
21      Column         0     1     10                   00  r[10]=searchable.text1
22      Sequence       2     11    0                    00  r[11]=cursor[2].ctr++
23      IfNotZero      1     27    0                    00  if r[1]!=0 then r[1]--, goto 27
24      Last           2     0     0                    00               
25      IdxLE          2     32    10    1              00  key=r[10]    
26      Delete         2     0     0                    00               
27      Rowid          0     12    0                    00  r[12]=rowid  
28      Column         0     2     13                   00  r[13]=searchable.text2
29      Column         0     3     14                   00  r[14]=searchable.name with . and spaces
30      MakeRecord     10    5     16                   00  r[16]=mkrec(r[10..14])
31      IdxInsert      2     16    10    5              00  key=r[16]    
32    Next           3     18    0                    00               
33    Sort           2     40    0                    00               
34      Column         2     4     15                   00  r[15]=[name with . and spaces]
35      Column         2     3     14                   00  r[14]=text2  
36      Column         2     0     13                   00  r[13]=text1  
37      Column         2     2     12                   00  r[12]=pk     
38      ResultRow      12    4     0                    00  output=r[12..15]
39    Next           2     34    0                    00               
40    Halt           0     0     0                    00               
41    Transaction    0     0     35    0              01  usesStmtJournal=0
42    Variable       1     7     0     :search        00  r[7]=parameter(1,:search)
43    Goto           0     1     0                    00               
```
Here the `ResultRow` is for registers `12..15` - but those all refer to `Column` records in `2` - where `2` is the first `OpenEphemeral` declared right at the start. I'm having enormous trouble figuring out how that ephemeral table gets populated by the other operations in a way that would let me derive which columns end up in the `ResultRow`.

Frustratingly SQLite seems to be able to figure that out just fine, see the column of comments on the right hand side - but I only get those in the `sqlite3` CLI shell, they're not available to me with SQLite when called as a library from Python.

Maybe the key to that is this section:
```
27      Rowid          0     12    0                    00  r[12]=rowid  
28      Column         0     2     13                   00  r[13]=searchable.text2
29      Column         0     3     14                   00  r[14]=searchable.name with . and spaces
30      MakeRecord     10    5     16                   00  r[16]=mkrec(r[10..14])
31      IdxInsert      2     16    10    5              00  key=r[16]    
```
MakeRecord:

> Convert P2 registers beginning with P1 into the record format use as a data record in a database table or as a key in an index. The Column opcode can decode the record later.
> 
> P4 may be a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th field of the index key.
> 
> The mapping from character to affinity is given by the SQLITE_AFF_ macros defined in sqliteInt.h.
> 
> If P4 is NULL then all index fields have the affinity BLOB.
> 
> The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM compile-time option is enabled:
> 
> * If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index of the right-most table that can be null-trimmed.
> 
> * If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to accept no-change records with serial_type 10. This value is only used inside an assert() and does not affect the end result.

IdxInsert:
> Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the index P1. Data for the entry is nil.
> 
> If P4 is not zero, then it is the number of values in the unpacked key of reg(P2). In that case, P3 is the index of the first register for the unpacked key. The availability of the unpacked key can sometimes be an optimization.
> 
> If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer that this insert is likely to be an append.
> 
> If P5 has the OPFLAG_NCHANGE bit set, then the change counter is incremented by this instruction. If the OPFLAG_NCHANGE bit is clear, then the change counter is unchanged.
> 
> If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might run faster by avoiding an unnecessary seek on cursor P1. However, the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior seeks on the cursor or if the most recent seek used a key equivalent to P2.
>
> This instruction only works for indices. The equivalent instruction for tables is Insert.

IdxLE:
> The P4 register values beginning with P3 form an unpacked index key that omits the PRIMARY KEY or ROWID. Compare this key value against the index that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID on the P1 index.
>
> If the P1 index entry is less than or equal to the key value then jump to P2. Otherwise fall through to the next instruction.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898760808,https://api.github.com/repos/simonw/datasette/issues/1293,898760808,IC_kwDOBm6k_c41kgBo,9599,simonw,2021-08-13T23:03:01Z,2021-08-13T23:03:01Z,OWNER,Another idea: strip out any `order by` clause to try and keep this simpler. I doubt that's going to cope with complex nested queries though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898760020,https://api.github.com/repos/simonw/datasette/issues/1293,898760020,IC_kwDOBm6k_c41kf1U,9599,simonw,2021-08-13T23:00:28Z,2021-08-13T23:01:27Z,OWNER,"New theory: this is all about `SorterOpen` and `SorterInsert`. Consider the following with extra annotations at the end of the lines after the `--`:
```
addr  opcode         p1    p2    p3    p4             p5  comment      
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     25    0                    00  Start at 25  
1     SorterOpen     2     5     0     k(1,B)         00               -- New SORTER in r2 with 5 slots
2     OpenRead       0     43    0     7              00  root=43 iDb=0; facetable
3     OpenRead       1     42    0     2              00  root=42 iDb=0; facet_cities
4     Rewind         0     16    0                    00               
5       Column         0     6     3                    00  r[3]=facetable.neighborhood
6       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
7       IfNot          1     15    1                    00               
8       Column         0     5     4                    00  r[4]=facetable.city_id
9       SeekRowid      1     15    4                    00  intkey=r[4]  
10      Column         1     1     6                    00  r[6]=facet_cities.name
11      Column         0     4     7                    00  r[7]=facetable.state
12      Column         0     6     5                    00  r[5]=facetable.neighborhood
13      MakeRecord     5     3     9                    00  r[9]=mkrec(r[5..7])
14      SorterInsert   2     9     5     3              00  key=r[9]-- WRITES record from r9 (line above) into sorter in r2
15    Next           0     5     0                    01               
16    OpenPseudo     3     10    5                    00  5 columns in r[10]
17    SorterSort     2     24    0                    00 -- runs the sort, not relevant to my goal
18      SorterData     2     10    3                    00  r[10]=data -- ""Write into register P2 (r10) the current sorter data for sorter cursor P1 (sorter 2)""
19      Column         3     2     8                    00  r[8]=state   
20      Column         3     1     7                    00  r[7]=facet_cities.name
21      Column         3     0     6                    00  r[6]=neighborhood
22      ResultRow      6     3     0                    00  output=r[6..8]
23    SorterNext     2     18    0                    00               
24    Halt           0     0     0                    00               
25    Transaction    0     0     35    0              01  usesStmtJournal=0
26    String8        0     2     0     %bob%          00  r[2]='%bob%' 
27    Goto           0     1     0                    00               
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898576097,https://api.github.com/repos/simonw/datasette/issues/1293,898576097,IC_kwDOBm6k_c41jy7h,9599,simonw,2021-08-13T16:19:57Z,2021-08-13T16:19:57Z,OWNER,"I think I need to look out for `OpenPseudo` and, when that occurs, take a look at the most recent `SorterInsert` and use that to find the `MakeRecord` and then use the `MakeRecord` to figure out the columns that went into it.

After all of that I'll be able to resolve that ""table 3"" reference.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898572065,https://api.github.com/repos/simonw/datasette/issues/1293,898572065,IC_kwDOBm6k_c41jx8h,9599,simonw,2021-08-13T16:13:16Z,2021-08-13T16:13:16Z,OWNER,"Aha! That `MakeRecord` line says `r[5..7]` - and r5 = neighborhood, r6 = facet_cities.name, r7 = facetable.state

So if the `MakeRecord` defines what goes into that pseudo-table column 2 of that pseudo-table would be `state` - which is what we want.

This is really convoluted. I'm no longer confident I can get this to work in a sensible way, especially since I've not started exploring what complex nested tables with CTEs and sub-selects do yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898569319,https://api.github.com/repos/simonw/datasette/issues/1293,898569319,IC_kwDOBm6k_c41jxRn,9599,simonw,2021-08-13T16:09:01Z,2021-08-13T16:10:48Z,OWNER,"Need to figure out what column 2 of that pseudo-table is.

I think the answer is here:

```
4     Rewind         0     16    0                    00               
5       Column         0     6     3                    00  r[3]=facetable.neighborhood
6       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
7       IfNot          1     15    1                    00               
8       Column         0     5     4                    00  r[4]=facetable.city_id
9       SeekRowid      1     15    4                    00  intkey=r[4]  
10      Column         1     1     6                    00  r[6]=facet_cities.name
11      Column         0     4     7                    00  r[7]=facetable.state
12      Column         0     6     5                    00  r[5]=facetable.neighborhood
13      MakeRecord     5     3     9                    00  r[9]=mkrec(r[5..7])
14      SorterInsert   2     9     5     3              00  key=r[9]     
15    Next           0     5     0                    01               
16    OpenPseudo     3     10    5                    00  5 columns in r[10]
```
I think the `OpenPseduo` line puts five columns in `r[10]` - and those five columns are the five from the previous block - maybe the five leading up to the `MakeRecord` call on line 13.

In which case column 2 would be `facet_cities.name` - assuming we start counting from 0.

But the debug code said ""r[8]=state"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898567974,https://api.github.com/repos/simonw/datasette/issues/1293,898567974,IC_kwDOBm6k_c41jw8m,9599,simonw,2021-08-13T16:07:00Z,2021-08-13T16:07:00Z,OWNER,"So this line:
```
19      Column         3     2     8                    00  r[8]=state
```
Means ""Take column 2 of table 3 (the pseudo-table) and store it in register 8""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898564705,https://api.github.com/repos/simonw/datasette/issues/1293,898564705,IC_kwDOBm6k_c41jwJh,9599,simonw,2021-08-13T16:02:12Z,2021-08-13T16:04:06Z,OWNER,"More debug output:
```
table_rootpage_by_register={0: 43, 1: 42}
names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')}
table_id=0 cid=6 column_register=3
table_id=0 cid=5 column_register=4
table_id=1 cid=1 column_register=6
table_id=0 cid=4 column_register=7
table_id=0 cid=6 column_register=5
table_id=3 cid=2 column_register=8
table_id=3 cid=2 column_register=8
  KeyError
    3
    table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
    names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3
    columns_by_column_register[column_register] = (table, cid)
    column_register=8 = (table='facetable', cid=2)
table_id=3 cid=1 column_register=7
  KeyError
    3
    table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
    names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3
    columns_by_column_register[column_register] = (table, cid)
    column_register=7 = (table='facetable', cid=1)
table_id=3 cid=0 column_register=6
  KeyError
    3
    table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
    names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')} table_rootpage_by_register={0: 43, 1: 42} table_id=3
    columns_by_column_register[column_register] = (table, cid)
    column_register=6 = (table='facetable', cid=0)
result_registers=[6, 7, 8]
columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)}
all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'}
```
Those `KeyError` are happening here because of a lookup in `table_rootpage_by_register` for `table_id=3` - but `table_rootpage_by_register` only has keys 0 and 1.

It looks like that `3` actually corresponds to the `OpenPseudo` table from here:

```
16    OpenPseudo     3     10    5                    00  5 columns in r[10]
17    SorterSort     2     24    0                    00               
18      SorterData     2     10    3                    00  r[10]=data   
19      Column         3     2     8                    00  r[8]=state   
20      Column         3     1     7                    00  r[7]=facet_cities.name
21      Column         3     0     6                    00  r[6]=neighborhood
22      ResultRow      6     3     0                    00  output=r[6..8]
```

Python code:

```python
def columns_for_query(conn, sql, params=None):
    """"""
    Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of
    ``(table_name, column_name)`` pairs corresponding to the columns that would be
    returned by that SQL query.

    Each pair indicates the source table and column for the returned column, or
    ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"")
    """"""
    if sql.lower().strip().startswith(""explain""):
        return []
    opcodes = conn.execute(""explain "" + sql, params).fetchall()
    table_rootpage_by_register = {
        r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead""
    }
    print(f""{table_rootpage_by_register=}"")
    names_and_types_by_rootpage = dict(
        [(r[0], (r[1], r[2])) for r in conn.execute(
            ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format(
                "", "".join(map(str, table_rootpage_by_register.values()))
            )
        )]
    )
    print(f""{names_and_types_by_rootpage=}"")
    columns_by_column_register = {}
    for opcode_row in opcodes:
        if opcode_row[""opcode""] in (""Rowid"", ""Column""):
            addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode_row
            print(f""{table_id=} {cid=} {column_register=}"")
            try:
                table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
                columns_by_column_register[column_register] = (table, cid)
            except KeyError as e:
                print(""  KeyError"")
                print(""   "", e)
                print(""    table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]"")
                print(f""    {names_and_types_by_rootpage=} {table_rootpage_by_register=} {table_id=}"")
                print(""    columns_by_column_register[column_register] = (table, cid)"")
                print(f""    {column_register=} = ({table=}, {cid=})"")
                pass
    result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0]
    result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""]))
    print(f""{result_registers=}"")
    print(f""{columns_by_column_register=}"")
    all_column_names = {}
    for (table, _) in names_and_types_by_rootpage.values():
        table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall()
        for column_info in table_xinfo:
            all_column_names[(table, column_info[""cid""])] = column_info[""name""]
    print(f""{all_column_names=}"")
    final_output = []
    for register in result_registers:
        try:
            table, cid = columns_by_column_register[register]
            final_output.append((table, all_column_names[table, cid]))
        except KeyError:
            final_output.append((None, None))
    return final_output

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898554859,https://api.github.com/repos/simonw/datasette/issues/1293,898554859,IC_kwDOBm6k_c41jtvr,9599,simonw,2021-08-13T15:46:18Z,2021-08-13T15:46:18Z,OWNER,So it looks like the bug is in the code that populates `columns_by_column_register`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898554427,https://api.github.com/repos/simonw/datasette/issues/1293,898554427,IC_kwDOBm6k_c41jto7,9599,simonw,2021-08-13T15:45:32Z,2021-08-13T15:45:32Z,OWNER,"Some useful debug output:
```
table_rootpage_by_register={0: 43, 1: 42}
names_and_types_by_rootpage={42: ('facet_cities', 'table'), 43: ('facetable', 'table')}
result_registers=[6, 7, 8]
columns_by_column_register={3: ('facetable', 6), 4: ('facetable', 5), 6: ('facet_cities', 1), 7: ('facetable', 4), 5: ('facetable', 6)}
all_column_names={('facet_cities', 0): 'id', ('facet_cities', 1): 'name', ('facetable', 0): 'pk', ('facetable', 1): 'created', ('facetable', 2): 'planet_int', ('facetable', 3): 'on_earth', ('facetable', 4): 'state', ('facetable', 5): 'city_id', ('facetable', 6): 'neighborhood', ('facetable', 7): 'tags', ('facetable', 8): 'complex_array', ('facetable', 9): 'distinct_some_null'}
```
The `result_registers` should each correspond to the correct entry in `columns_by_column_register` but they do not.

Python code:
```python
def columns_for_query(conn, sql, params=None):
    """"""
    Given a SQLite connection ``conn`` and a SQL query ``sql``, returns a list of
    ``(table_name, column_name)`` pairs corresponding to the columns that would be
    returned by that SQL query.

    Each pair indicates the source table and column for the returned column, or
    ``(None, None)`` if no table and column could be derived (e.g. for ""select 1"")
    """"""
    if sql.lower().strip().startswith(""explain""):
        return []
    opcodes = conn.execute(""explain "" + sql, params).fetchall()
    table_rootpage_by_register = {
        r[""p1""]: r[""p2""] for r in opcodes if r[""opcode""] == ""OpenRead""
    }
    print(f""{table_rootpage_by_register=}"")
    names_and_types_by_rootpage = dict(
        [(r[0], (r[1], r[2])) for r in conn.execute(
            ""select rootpage, name, type from sqlite_master where rootpage in ({})"".format(
                "", "".join(map(str, table_rootpage_by_register.values()))
            )
        )]
    )
    print(f""{names_and_types_by_rootpage=}"")
    columns_by_column_register = {}
    for opcode in opcodes:
        if opcode[""opcode""] in (""Rowid"", ""Column""):
            addr, opcode, table_id, cid, column_register, p4, p5, comment = opcode
            try:
                table = names_and_types_by_rootpage[table_rootpage_by_register[table_id]][0]
                columns_by_column_register[column_register] = (table, cid)
            except KeyError:
                pass
    result_row = [dict(r) for r in opcodes if r[""opcode""] == ""ResultRow""][0]
    result_registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""]))
    print(f""{result_registers=}"")
    print(f""{columns_by_column_register=}"")
    all_column_names = {}
    for (table, _) in names_and_types_by_rootpage.values():
        table_xinfo = conn.execute(""pragma table_xinfo({})"".format(table)).fetchall()
        for column_info in table_xinfo:
            all_column_names[(table, column_info[""cid""])] = column_info[""name""]
    print(f""{all_column_names=}"")
    final_output = []
    for register in result_registers:
        try:
            table, cid = columns_by_column_register[register]
            final_output.append((table, all_column_names[table, cid]))
        except KeyError:
            final_output.append((None, None))
    return final_output
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898545815,https://api.github.com/repos/simonw/datasette/issues/1293,898545815,IC_kwDOBm6k_c41jriX,9599,simonw,2021-08-13T15:31:53Z,2021-08-13T15:31:53Z,OWNER,"My hunch here is that registers or columns are being reused in a way that makes my code break - my code is pretty dumb, there are places in it where maybe the first mention of a register wins instead of the last one?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898541972,https://api.github.com/repos/simonw/datasette/issues/1293,898541972,IC_kwDOBm6k_c41jqmU,9599,simonw,2021-08-13T15:26:06Z,2021-08-13T15:29:06Z,OWNER,"ResultRow:
> The registers P1 through P1+P2-1 contain a single row of results. This opcode causes the sqlite3_step() call to terminate with an SQLITE_ROW return code and it sets up the sqlite3_stmt structure to provide access to the r(P1)..r(P1+P2-1) values as the result row.

Column:
> Interpret the data that cursor P1 points to as a structure built using the MakeRecord instruction. (See the MakeRecord opcode for additional information about the format of the data.) Extract the P2-th column from this record. If there are less that (P2+1) values in the record, extract a NULL.
>
> The value extracted is stored in register P3.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898541543,https://api.github.com/repos/simonw/datasette/issues/1293,898541543,IC_kwDOBm6k_c41jqfn,9599,simonw,2021-08-13T15:25:26Z,2021-08-13T15:25:26Z,OWNER,"But the debug output here seems to be saying what we want it to say:
```
17    SorterSort     2     24    0                    00               
18      SorterData     2     10    3                    00  r[10]=data   
19      Column         3     2     8                    00  r[8]=state   
20      Column         3     1     7                    00  r[7]=facet_cities.name
21      Column         3     0     6                    00  r[6]=neighborhood
22      ResultRow      6     3     0                    00  output=r[6..8]
```
We want to get back `neighborhood`, `facet_cities.name`, `state`.

Why then are we seeing `[('facet_cities', 'name'), ('facetable', 'state'), (None, None)]`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898540260,https://api.github.com/repos/simonw/datasette/issues/1293,898540260,IC_kwDOBm6k_c41jqLk,9599,simonw,2021-08-13T15:23:28Z,2021-08-13T15:23:28Z,OWNER,"SorterInsert:
> Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the sorter P1. Data for the entry is nil.

SorterData:
> Write into register P2 the current sorter data for sorter cursor P1. Then clear the column header cache on cursor P3.
>
> This opcode is normally use to move a record out of the sorter and into a register that is the source for a pseudo-table cursor created using OpenPseudo. That pseudo-table cursor is the one that is identified by parameter P3. Clearing the P3 column cache as part of this opcode saves us from having to issue a separate NullRow instruction to clear that cache.

OpenPseudo:
> Open a new cursor that points to a fake table that contains a single row of data. The content of that one row is the content of memory register P2. In other words, cursor P1 becomes an alias for the MEM_Blob content contained in register P2.
>
> A pseudo-table created by this opcode is used to hold a single row output from the sorter so that the row can be decomposed into individual columns using the Column opcode. The Column opcode is the only cursor opcode that works with a pseudo-table.
>
> P3 is the number of fields in the records that will be stored by the pseudo-table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898536181,https://api.github.com/repos/simonw/datasette/issues/1293,898536181,IC_kwDOBm6k_c41jpL1,9599,simonw,2021-08-13T15:17:20Z,2021-08-13T15:20:33Z,OWNER,"Documentation for `MakeRecord`: https://www.sqlite.org/opcode.html#MakeRecord

Running `explain` inside `sqlite3` provides extra comments and indentation which make it easier to understand:
```
sqlite> explain select neighborhood, facet_cities.name, state
   ...> from facetable
   ...>     join facet_cities
   ...>         on facetable.city_id = facet_cities.id
   ...> where neighborhood like '%bob%';
addr  opcode         p1    p2    p3    p4             p5  comment      
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     15    0                    00  Start at 15  
1     OpenRead       0     43    0     7              00  root=43 iDb=0; facetable
2     OpenRead       1     42    0     2              00  root=42 iDb=0; facet_cities
3     Rewind         0     14    0                    00               
4       Column         0     6     3                    00  r[3]=facetable.neighborhood
5       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
6       IfNot          1     13    1                    00               
7       Column         0     5     4                    00  r[4]=facetable.city_id
8       SeekRowid      1     13    4                    00  intkey=r[4]  
9       Column         0     6     5                    00  r[5]=facetable.neighborhood
10      Column         1     1     6                    00  r[6]=facet_cities.name
11      Column         0     4     7                    00  r[7]=facetable.state
12      ResultRow      5     3     0                    00  output=r[5..7]
13    Next           0     4     0                    01               
14    Halt           0     0     0                    00               
15    Transaction    0     0     35    0              01  usesStmtJournal=0
16    String8        0     2     0     %bob%          00  r[2]='%bob%' 
17    Goto           0     1     0                    00               
```
Compared with:
```
sqlite> explain select neighborhood, facet_cities.name, state
   ...> from facetable
   ...>     join facet_cities
   ...>         on facetable.city_id = facet_cities.id
   ...> where neighborhood like '%bob%' order by neighborhood
   ...> ;
addr  opcode         p1    p2    p3    p4             p5  comment      
----  -------------  ----  ----  ----  -------------  --  -------------
0     Init           0     25    0                    00  Start at 25  
1     SorterOpen     2     5     0     k(1,B)         00               
2     OpenRead       0     43    0     7              00  root=43 iDb=0; facetable
3     OpenRead       1     42    0     2              00  root=42 iDb=0; facet_cities
4     Rewind         0     16    0                    00               
5       Column         0     6     3                    00  r[3]=facetable.neighborhood
6       Function0      1     2     1     like(2)        02  r[1]=func(r[2..3])
7       IfNot          1     15    1                    00               
8       Column         0     5     4                    00  r[4]=facetable.city_id
9       SeekRowid      1     15    4                    00  intkey=r[4]  
10      Column         1     1     6                    00  r[6]=facet_cities.name
11      Column         0     4     7                    00  r[7]=facetable.state
12      Column         0     6     5                    00  r[5]=facetable.neighborhood
13      MakeRecord     5     3     9                    00  r[9]=mkrec(r[5..7])
14      SorterInsert   2     9     5     3              00  key=r[9]     
15    Next           0     5     0                    01               
16    OpenPseudo     3     10    5                    00  5 columns in r[10]
17    SorterSort     2     24    0                    00               
18      SorterData     2     10    3                    00  r[10]=data   
19      Column         3     2     8                    00  r[8]=state   
20      Column         3     1     7                    00  r[7]=facet_cities.name
21      Column         3     0     6                    00  r[6]=neighborhood
22      ResultRow      6     3     0                    00  output=r[6..8]
23    SorterNext     2     18    0                    00               
24    Halt           0     0     0                    00               
25    Transaction    0     0     35    0              01  usesStmtJournal=0
26    String8        0     2     0     %bob%          00  r[2]='%bob%' 
27    Goto           0     1     0                    00               
```
So actually it looks like the `SorterSort` may be key to understanding this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898527525,https://api.github.com/repos/simonw/datasette/issues/1293,898527525,IC_kwDOBm6k_c41jnEl,9599,simonw,2021-08-13T15:08:03Z,2021-08-13T15:08:03Z,OWNER,Am I going to need to look at the `ResultRow` and its columns but then wind back to that earlier `MakeRecord` and its columns?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898524057,https://api.github.com/repos/simonw/datasette/issues/1293,898524057,IC_kwDOBm6k_c41jmOZ,9599,simonw,2021-08-13T15:06:37Z,2021-08-13T15:06:37Z,OWNER,"Comparing the `explain` for the two versions of that query - one with the order by and one without:

<img width=""1031"" alt=""fixtures__explain_select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_and_fixtures__explain_select_neighborh"" src=""https://user-images.githubusercontent.com/9599/129377790-52af28ab-5110-470f-bb1b-a400455e6717.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898519924,https://api.github.com/repos/simonw/datasette/issues/1293,898519924,IC_kwDOBm6k_c41jlN0,9599,simonw,2021-08-13T15:03:36Z,2021-08-13T15:03:36Z,OWNER,"Weird edge-case: adding an `order by` changes the order of the columns with respect to the information I am deriving about them.

Without order by this gets it right:

<img width=""702"" alt=""fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________"" src=""https://user-images.githubusercontent.com/9599/129377247-ec1f67fd-5fc5-46a2-92ef-629276446621.png"">

With order by:

<img width=""708"" alt=""fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood"" src=""https://user-images.githubusercontent.com/9599/129377339-5b338432-6db8-43ac-9408-48a87c03e5e9.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898517872,https://api.github.com/repos/simonw/datasette/issues/1293,898517872,IC_kwDOBm6k_c41jktw,9599,simonw,2021-08-13T15:00:50Z,2021-08-13T15:00:50Z,OWNER,"The primary key column (or `rowid`) often resolves to an `index` record in the `sqlite_master` table, e.g. the second row in this: 

type | name | tbl_name | rootpage | sql
-- | -- | -- | -- | --
table | simple_primary_key | simple_primary_key | 2 | CREATE TABLE simple_primary_key (   id varchar(30) primary key,   content text )
index | sqlite_autoindex_simple_primary_key_1 | simple_primary_key | 3 |  

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898506647,https://api.github.com/repos/simonw/datasette/issues/1293,898506647,IC_kwDOBm6k_c41jh-X,9599,simonw,2021-08-13T14:43:19Z,2021-08-13T14:43:19Z,OWNER,Work will continue in PR #1434.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/pull/1433#issuecomment-898450402,https://api.github.com/repos/simonw/datasette/issues/1433,898450402,IC_kwDOBm6k_c41jUPi,22429695,codecov[bot],2021-08-13T13:15:55Z,2021-08-13T13:15:55Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1433](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ddba6cc) into [main](https://codecov.io/gh/simonw/datasette/commit/2883098770fc66e50183b2b231edbde20848d4d6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (2883098) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1433/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1433   +/-   ##
=======================================
  Coverage   91.82%   91.82%           
=======================================
  Files          34       34           
  Lines        4418     4418           
=======================================
  Hits         4057     4057           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [2883098...ddba6cc](https://codecov.io/gh/simonw/datasette/pull/1433?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",970386262,"Update trustme requirement from <0.9,>=0.7 to >=0.7,<0.10",
https://github.com/simonw/datasette/issues/1429#issuecomment-898185944,https://api.github.com/repos/simonw/datasette/issues/1429,898185944,IC_kwDOBm6k_c41iTrY,9599,simonw,2021-08-13T04:37:41Z,2021-08-13T04:37:41Z,OWNER,"If a count is available and the count is less than 1,000 it could say ""Show all"" instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page,
https://github.com/simonw/datasette/issues/1432#issuecomment-898084675,https://api.github.com/repos/simonw/datasette/issues/1432,898084675,IC_kwDOBm6k_c41h69D,9599,simonw,2021-08-13T01:11:30Z,2021-08-13T01:11:30Z,OWNER,It's only `datasette-publish-vercel` that will break the actual functionality - the others will have broken tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/issues/1432#issuecomment-898079507,https://api.github.com/repos/simonw/datasette/issues/1432,898079507,IC_kwDOBm6k_c41h5sT,9599,simonw,2021-08-13T01:08:42Z,2021-08-13T01:09:41Z,OWNER,"This is going to break some plugins: https://ripgrep.datasette.io/-/ripgrep?pattern=config%3D&literal=on&glob=%21datasette%2F**

> ### datasette-cluster-map/tests/test_cluster_map.py
> 
> @pytest.mark.asyncio
> 
> async def test_respects_base_url():
> ds = Datasette([], memory=True, config={""base_url"": ""/foo/""})
> response = await ds.client.get(""/:memory:?sql=select+1+as+latitude,+2+as+longitude"")
> assert (
> 
> ### datasette-export-notebook/tests/test_export_notebook.py
> 
> @pytest.mark.asyncio
> 
> async def test_notebook_no_csv(db_path):
> datasette = Datasette([db_path], config={""allow_csv_stream"": False})
> response = await datasette.client.get(""/db/big.Notebook"")
> assert "".csv"" not in response.text
> 
> ### datasette-publish-vercel/tests/test_publish_vercel.py
> metadata=metadata,
> cors=True,
> config={""default_page_size"": 10, ""sql_time_limit_ms"": 2000}
> ).app()
> """"""
> 
> ### datasette-publish-vercel/datasette_publish_vercel/__init__.py
> metadata=metadata{extras},
> cors=True,
> config={settings}
> 
> ).app()
> 
> """""".strip()
> 
> ### datasette-search-all/tests/test_search_all.py
> 
> async def test_base_url(db_path, path):
> sqlite_utils.Database(db_path)[""creatures""].enable_fts([""name"", ""description""])
> datasette = Datasette([db_path], config={""base_url"": ""/foo/""})
> response = await datasette.client.get(path)
> assert response.status_code == 200

I should fix those as soon as this goes out in a release. I won't close this issue until then.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/issues/1432#issuecomment-898074849,https://api.github.com/repos/simonw/datasette/issues/1432,898074849,IC_kwDOBm6k_c41h4jh,9599,simonw,2021-08-13T01:03:40Z,2021-08-13T01:03:40Z,OWNER,"Also this method: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/app.py#L422-L424

And the places that use it:

https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/base.py#L617

https://github.com/simonw/datasette/blob/fc4846850fffd54561bc125332dfe97bb41ff42e/datasette/views/database.py#L459

Which is used in this template: https://github.com/simonw/datasette/blob/77f46297a88ac7e49dad2139410b01ee56d5f99c/datasette/templates/table.html#L204

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969855774,Rename Datasette.__init__(config=) parameter to settings=,
https://github.com/simonw/datasette/issues/1431#issuecomment-898072940,https://api.github.com/repos/simonw/datasette/issues/1431,898072940,IC_kwDOBm6k_c41h4Fs,9599,simonw,2021-08-13T00:58:40Z,2021-08-13T00:58:40Z,OWNER,"While I'm doing this I should rename this internal variable to avoid confusion in the future:

https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969840302,`--help-config` should be called `--help-settings`,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134386,https://api.github.com/repos/simonw/datasette/issues/1293,813134386,MDEyOklzc3VlQ29tbWVudDgxMzEzNDM4Ng==,9599,simonw,2021-04-05T01:20:28Z,2021-08-13T00:42:30Z,OWNER,"... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the `Variable` opcodes.

[UPDATE: it did indeed do that, see #1421]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898066466,https://api.github.com/repos/simonw/datasette/issues/1293,898066466,IC_kwDOBm6k_c41h2gi,9599,simonw,2021-08-13T00:40:24Z,2021-08-13T00:40:24Z,OWNER,"It figures out renamed columns too:

<img width=""694"" alt=""fixtures__select_created__state_as_the_state_from_facetable"" src=""https://user-images.githubusercontent.com/9599/129287208-1347fe80-f62e-4ed2-80c6-06a223cbe749.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898065948,https://api.github.com/repos/simonw/datasette/issues/1293,898065948,IC_kwDOBm6k_c41h2Yc,9599,simonw,2021-08-13T00:38:58Z,2021-08-13T00:38:58Z,OWNER,"Trying to run `explain select * from facetable` fails with an error in my prototype, because it tries to execute `explain explain select * from facetable` - so I need to spot that error and ignore it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898065011,https://api.github.com/repos/simonw/datasette/issues/1293,898065011,IC_kwDOBm6k_c41h2Jz,9599,simonw,2021-08-13T00:36:30Z,2021-08-13T00:36:30Z,OWNER,"> https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:
> 
> ```sql
> CREATE VIEW paginated_view AS
>     SELECT
>         content,
>         '- ' || content || ' -' AS content_extra
>     FROM no_primary_key;
> ```
> 
> So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.

Here's what it does for that:

<img width=""748"" alt=""fixtures__select___from_paginated_view"" src=""https://user-images.githubusercontent.com/9599/129286962-426bfa56-3946-447a-996d-668b4d80f5c1.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898063815,https://api.github.com/repos/simonw/datasette/issues/1293,898063815,IC_kwDOBm6k_c41h13H,9599,simonw,2021-08-13T00:33:17Z,2021-08-13T00:33:17Z,OWNER,"Improved version of that function:
```python
def columns_for_query(conn, sql):
    """"""
    Given a SQLite connection ``conn`` and a SQL query ``sql``,
    returns a list of ``(table_name, column_name)`` pairs, one
    per returned column. ``(None, None)`` if no table and column
    could be derived.
    """"""
    rows = conn.execute('explain ' + sql).fetchall()
    table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
    names_by_rootpage = dict(
        conn.execute(
            'select rootpage, name from sqlite_master where rootpage in ({})'.format(
                ', '.join(map(str, table_rootpage_by_register.values()))
            )
        )
    )
    columns_by_column_register = {}
    for row in rows:
        if row['opcode'] in ('Rowid', 'Column'):
            addr, opcode, table_id, cid, column_register, p4, p5, comment = row
            table = names_by_rootpage[table_rootpage_by_register[table_id]]
            columns_by_column_register[column_register] = (table, cid)
    result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
    registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""]))
    all_column_names = {}
    for table in names_by_rootpage.values():
        table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
        for row in table_xinfo:
            all_column_names[(table, row[""cid""])] = row[""name""]
    final_output = []
    for r in registers:
        try:
            table, cid = columns_by_column_register[r]
            final_output.append((table, all_column_names[table, cid]))
        except KeyError:
            final_output.append((None, None))
    return final_output
```
It works!

<img width=""1440"" alt=""Banners_and_Alerts_and_fixtures__select_attraction_id__roadside_attractions_name__characteristic_id__attraction_characteristic_name_as_characteristic_from_roadside_attraction_characteristics_join_roadside_attractions_on_roadside_attractions"" src=""https://user-images.githubusercontent.com/9599/129286745-be5bb268-c31c-413b-9530-c327ae7fd481.png"">

```diff
diff --git a/datasette/templates/query.html b/datasette/templates/query.html
index 75f7f1b..9fe1d4f 100644
--- a/datasette/templates/query.html
+++ b/datasette/templates/query.html
@@ -67,6 +67,8 @@
     </p>
 </form>
 
+extra_column_info: {{ extra_column_info }}
+
 {% if display_rows %}
 <p class=""export-links"">This data as {% for name, url in renderers.items() %}<a href=""{{ url }}"">{{ name }}</a>{{ "", "" if not loop.last }}{% endfor %}, <a href=""{{ url_csv }}"">CSV</a></p>
 <div class=""table-wrapper""><table class=""rows-and-columns"">
diff --git a/datasette/views/database.py b/datasette/views/database.py
index 7c36034..02f8039 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -10,6 +10,7 @@ import markupsafe
 from datasette.utils import (
     await_me_maybe,
     check_visibility,
+    columns_for_query,
     derive_named_parameters,
     to_css_class,
     validate_sql_select,
@@ -248,6 +249,8 @@ class QueryView(DataView):
 
         query_error = None
 
+        extra_column_info = None
+
         # Execute query - as write or as read
         if write:
             if request.method == ""POST"":
@@ -334,6 +337,10 @@ class QueryView(DataView):
                     database, sql, params_for_query, truncate=True, **extra_args
                 )
                 columns = [r[0] for r in results.description]
+
+                # Try to figure out extra column information
+                db = self.ds.get_database(database)
+                extra_column_info = await db.execute_fn(lambda conn: columns_for_query(conn, sql))
             except sqlite3.DatabaseError as e:
                 query_error = e
                 results = None
@@ -462,6 +469,7 @@ class QueryView(DataView):
                 ""show_hide_text"": show_hide_text,
                 ""show_hide_hidden"": markupsafe.Markup(show_hide_hidden),
                 ""hide_sql"": hide_sql,
+                ""extra_column_info"": extra_column_info,
             }
 
         return (
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/1293#issuecomment-898056013,https://api.github.com/repos/simonw/datasette/issues/1293,898056013,IC_kwDOBm6k_c41hz9N,9599,simonw,2021-08-13T00:12:09Z,2021-08-13T00:12:09Z,OWNER,"Having added column metadata in #1430 (ref #942) I could also include a definition list at the top of the query results page exposing the column descriptions for any columns, using the same EXPLAIN mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,Show column metadata plus links for foreign keys on arbitrary query results,
https://github.com/simonw/datasette/issues/942#issuecomment-898051645,https://api.github.com/repos/simonw/datasette/issues/942,898051645,IC_kwDOBm6k_c41hy49,9599,simonw,2021-08-13T00:02:25Z,2021-08-13T00:02:25Z,OWNER,"And on mobile:

![5FAF8D73-7199-4BB7-A5B8-9E46DCB4A985](https://user-images.githubusercontent.com/9599/129284817-dc13cbf4-144e-4f4c-8fb7-470602e2eea0.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-898050457,https://api.github.com/repos/simonw/datasette/issues/942,898050457,IC_kwDOBm6k_c41hymZ,9599,simonw,2021-08-12T23:59:53Z,2021-08-12T23:59:53Z,OWNER,"Documentation: https://docs.datasette.io/en/latest/metadata.html#column-descriptions

Live demo: https://latest.datasette.io/fixtures/roadside_attractions

<img width=""1045"" alt=""fixtures__roadside_attractions__4_rows"" src=""https://user-images.githubusercontent.com/9599/129284675-ef2fd6ed-54ba-4f6a-add5-7a9253921279.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/pull/1430#issuecomment-898043575,https://api.github.com/repos/simonw/datasette/issues/1430,898043575,IC_kwDOBm6k_c41hw63,22429695,codecov[bot],2021-08-12T23:39:36Z,2021-08-12T23:49:51Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1430](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9419947) into [main](https://codecov.io/gh/simonw/datasette/commit/b1fed48a95516ae84c0f020582303ab50ab817e2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b1fed48) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1430/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1430   +/-   ##
=======================================
  Coverage   91.71%   91.71%           
=======================================
  Files          34       34           
  Lines        4417     4418    +1     
=======================================
+ Hits         4051     4052    +1     
  Misses        366      366           
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1430/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.00% <100.00%> (+<0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [b1fed48...9419947](https://codecov.io/gh/simonw/datasette/pull/1430?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969758038,Column metadata,
https://github.com/simonw/datasette/issues/942#issuecomment-898037650,https://api.github.com/repos/simonw/datasette/issues/942,898037650,IC_kwDOBm6k_c41hveS,9599,simonw,2021-08-12T23:23:54Z,2021-08-12T23:23:54Z,OWNER,I like this enough that I'm going to ship it as an alpha and try it out on a couple of live projects.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-898037456,https://api.github.com/repos/simonw/datasette/issues/942,898037456,IC_kwDOBm6k_c41hvbQ,9599,simonw,2021-08-12T23:23:34Z,2021-08-12T23:23:34Z,OWNER,"Prototype with a `<dl>`:

<img width=""721"" alt=""fixtures__sortable__201_rows"" src=""https://user-images.githubusercontent.com/9599/129282175-c94dcf74-beb3-4f0a-9106-f75d19369fb0.png"">

```diff
diff --git a/datasette/static/app.css b/datasette/static/app.css
index c6be1e9..bf068fd 100644
--- a/datasette/static/app.css
+++ b/datasette/static/app.css
@@ -836,6 +841,16 @@ svg.dropdown-menu-icon {
     background-repeat: no-repeat;
 }
 
+dl.column-descriptions dt {
+    font-weight: bold;
+}
+dl.column-descriptions dd {
+    padding-left: 1.5em;
+    white-space: pre-wrap;
+    line-height: 1.1em;
+    color: #666;
+}
+
 .anim-scale-in {
     animation-name: scale-in;
     animation-duration: 0.15s;
diff --git a/datasette/templates/table.html b/datasette/templates/table.html
index 211352b..466e8a4 100644
--- a/datasette/templates/table.html
+++ b/datasette/templates/table.html
@@ -51,6 +51,14 @@
 
 {% block description_source_license %}{% include ""_description_source_license.html"" %}{% endblock %}
 
+{% if metadata.columns %}
+<dl class=""column-descriptions"">
+    {% for column_name, column_description in metadata.columns.items() %}
+        <dt>{{ column_name }}</dt><dd>{{ column_description }}</dd>
+    {% endfor %}
+</dl>
+{% endif %}
+
 {% if filtered_table_rows_count or human_description_en %}
     <h3>{% if filtered_table_rows_count or filtered_table_rows_count == 0 %}{{ ""{:,}"".format(filtered_table_rows_count) }} row{% if filtered_table_rows_count == 1 %}{% else %}s{% endif %}{% endif %}
         {% if human_description_en %}{{ human_description_en }}{% endif %}
 ```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-898032118,https://api.github.com/repos/simonw/datasette/issues/942,898032118,IC_kwDOBm6k_c41huH2,596279,zaneselvans,2021-08-12T23:12:00Z,2021-08-12T23:12:00Z,NONE,"This looks awesome. We'll definitely make extensive use of this feature!

On Thu, Aug 12, 2021 at 5:52 PM Simon Willison ***@***.***>
wrote:

> I like this. Need to solve for mobile though where the cog menu isn't
> visible - I think I'll do that with a definition list at the top of the
> page.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/simonw/datasette/issues/942#issuecomment-898022235>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAERSNYEF6QRZO2HRJGRIWDT4RGDFANCNFSM4QEC6ATA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
> .
>


-- 
Zane A. Selvans, PhD
Chief Data Wrangler
Catalyst Cooperative
https://catalyst.coop
***@***.***
Signal/WhatsApp/SMS: +1 720 443 1363
Twitter: @ZaneSelvans <https://twitter.com/ZaneSelvans>
PGP <https://www.gnupg.org/>: 0x64F7B56F3A127B04
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-898022235,https://api.github.com/repos/simonw/datasette/issues/942,898022235,IC_kwDOBm6k_c41hrtb,9599,simonw,2021-08-12T22:52:23Z,2021-08-12T22:52:23Z,OWNER,I like this. Need to solve for mobile though where the cog menu isn't visible - I think I'll do that with a definition list at the top of the page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-898021895,https://api.github.com/repos/simonw/datasette/issues/942,898021895,IC_kwDOBm6k_c41hroH,9599,simonw,2021-08-12T22:51:36Z,2021-08-12T22:51:36Z,OWNER,"Prototype:

<img width=""650"" alt=""fixtures__sortable__201_rows"" src=""https://user-images.githubusercontent.com/9599/129279808-08d56b2f-a4d6-4147-b3a6-542c62b566c1.png"">

```diff
diff --git a/datasette/static/app.css b/datasette/static/app.css
index c6be1e9..5ca64cb 100644
--- a/datasette/static/app.css
+++ b/datasette/static/app.css
@@ -784,9 +784,14 @@ svg.dropdown-menu-icon {
     font-size: 0.7em;
     color: #666;
     margin: 0;
-    padding: 0;
     padding: 4px 8px 4px 8px;
 }
+.dropdown-menu .dropdown-column-description {
+    margin: 0;
+    color: #666;
+    padding: 4px 8px 4px 8px;
+    max-width: 20em;
+}
 .dropdown-menu li {
     border-bottom: 1px solid #ccc;
 }
diff --git a/datasette/static/table.js b/datasette/static/table.js
index 991346d..a903112 100644
--- a/datasette/static/table.js
+++ b/datasette/static/table.js
@@ -9,6 +9,7 @@ var DROPDOWN_HTML = `<div class=""dropdown-menu"">
   <li><a class=""dropdown-not-blank"" href=""#"">Show not-blank rows</a></li>
 </ul>
 <p class=""dropdown-column-type""></p>
+<p class=""dropdown-column-description""></p>
 </div>`;
 
 var DROPDOWN_ICON_SVG = `<svg xmlns=""http://www.w3.org/2000/svg"" width=""14"" height=""14"" viewBox=""0 0 24 24"" fill=""none"" stroke=""currentColor"" stroke-width=""2"" stroke-linecap=""round"" stroke-linejoin=""round"">
@@ -166,6 +167,14 @@ var DROPDOWN_ICON_SVG = `<svg xmlns=""http://www.w3.org/2000/svg"" width=""14"" heig
     } else {
       columnTypeP.style.display = ""none"";
     }
+
+    var columnDescriptionP = menu.querySelector("".dropdown-column-description"");
+    if (th.dataset.columnDescription) {
+      columnDescriptionP.innerText = th.dataset.columnDescription;
+      columnDescriptionP.style.display = 'block';
+    } else {
+      columnDescriptionP.style.display = 'none';
+    }
     menu.style.position = ""absolute"";
     menu.style.top = menuTop + 6 + ""px"";
     menu.style.left = menuLeft + ""px"";
diff --git a/datasette/templates/_table.html b/datasette/templates/_table.html
index d765937..649f517 100644
--- a/datasette/templates/_table.html
+++ b/datasette/templates/_table.html
@@ -4,7 +4,7 @@
         <thead>
             <tr>
                 {% for column in display_columns %}
-                    <th class=""col-{{ column.name|to_css_class }}"" scope=""col"" data-column=""{{ column.name }}"" data-column-type=""{{ column.type }}"" data-column-not-null=""{{ column.notnull }}"" data-is-pk=""{% if column.is_pk %}1{% else %}0{% endif %}"">
+                    <th {% if column.description %}data-column-description=""{{ column.description }}"" {% endif %}class=""col-{{ column.name|to_css_class }}"" scope=""col"" data-column=""{{ column.name }}"" data-column-type=""{{ column.type }}"" data-column-not-null=""{{ column.notnull }}"" data-is-pk=""{% if column.is_pk %}1{% else %}0{% endif %}"">
                         {% if not column.sortable %}
                             {{ column.name }}
                         {% else %}
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 456d806..486a613 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -125,6 +125,7 @@ class RowTableShared(DataView):
         """"""Returns columns, rows for specified table - including fancy foreign key treatment""""""
         db = self.ds.databases[database]
         table_metadata = self.ds.table_metadata(database, table)
+        column_descriptions = table_metadata.get(""columns"") or {}
         column_details = {col.name: col for col in await db.table_column_details(table)}
         sortable_columns = await self.sortable_columns_for_table(database, table, True)
         pks = await db.primary_keys(table)
@@ -147,6 +148,7 @@ class RowTableShared(DataView):
                     ""is_pk"": r[0] in pks_for_display,
                     ""type"": type_,
                     ""notnull"": notnull,
+                    ""description"": column_descriptions.get(r[0]),
                 }
             )

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-897996296,https://api.github.com/repos/simonw/datasette/issues/942,897996296,IC_kwDOBm6k_c41hlYI,9599,simonw,2021-08-12T22:01:36Z,2021-08-12T22:01:36Z,OWNER,"I'm going with `""columns"": {""name-of-column"": ""description-of-column""}`.

If I decide to make `""col""` and `""nocol""` available in metadata I'll use those as the keys in the metadata, for consistency with the existing query string parameters.

I'm OK with having both `""columns"": ...` and `""col"": ...` keys in the metadata, even though they could be a tiny bit confusing without the documentation.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1429#issuecomment-897960049,https://api.github.com/repos/simonw/datasette/issues/1429,897960049,IC_kwDOBm6k_c41hchx,9599,simonw,2021-08-12T20:53:04Z,2021-08-12T20:53:04Z,OWNER,"Maybe something like this:

> [Next page](#) - 100 per page ([show 1,000 per page](#))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",969548935,UI for setting `?_size=max` on table page,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897600677,https://api.github.com/repos/simonw/sqlite-utils/issues/186,897600677,IC_kwDOCGYnMM41gEyl,9308268,rayvoelker,2021-08-12T12:32:14Z,2021-08-12T12:32:14Z,NONE,"Actually, I forgot to include the `bib_pub_year` in the extract ... 

But also, I tried again with empty string values instead of `NULL` values and it seems to place the foreign key properly / correctly... 

```python3
sql = """"""\
INSERT INTO ""circulation_info"" (""item_id"", ""bib_title"", ""bib_creator"", ""bib_format"", ""bib_pub_year"", ""checkout_date"")
VALUES
(1, ""title one"", ""creator one"", ""Book"", 2018, ""2021-08-12 00:01""),
(2, ""title two"", ""creator one"", ""Book"", 2019, ""2021-08-12 00:02""),
(3, ""title three"", """", ""DVD"", 2020, ""2021-08-12 00:03""),
(4, ""title four"", """", ""DVD"", """", ""2021-08-12 00:04""),
(5, ""title five"", """", ""DVD"", """", ""2021-08-12 00:05"")
""""""

with sqlite3.connect('test_bib_2.db') as con:
    con.execute(sql)
```

```python3
db[""circulation_info""].extract(
    [
        ""bib_title"",
        ""bib_creator"",
        ""bib_format"",
        ""bib_pub_year""
    ],
    table=""bib_info"", 
    fk_column=""bib_info_id""
)
```

```
{'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}
{'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}
{'id': 3, 'item_id': 3, 'bib_info_id': 3, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}
{'id': 4, 'item_id': 4, 'bib_info_id': 4, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:04'}
{'id': 5, 'item_id': 5, 'bib_info_id': 5, 'bib_pub_year': '', 'checkout_date': '2021-08-12 00:05'}

---

{'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 3, 'bib_title': 'title three', 'bib_creator': '', 'bib_format': 'DVD'}
{'id': 4, 'bib_title': 'title four', 'bib_creator': '', 'bib_format': 'DVD'}
{'id': 5, 'bib_title': 'title five', 'bib_creator': '', 'bib_format': 'DVD'}
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/186#issuecomment-897588624,https://api.github.com/repos/simonw/sqlite-utils/issues/186,897588624,IC_kwDOCGYnMM41gB2Q,9308268,rayvoelker,2021-08-12T12:13:25Z,2021-08-12T12:13:25Z,NONE,"I think I ran into an issue that's perhaps related with `extract()`

I have a case where I want to create a lookup table for all the related title data where there are possibly multiple null values in the related columns ....
```python3
sql = """"""\
INSERT INTO ""circulation_info"" (""item_id"", ""bib_title"", ""bib_creator"", ""bib_format"", ""bib_pub_year"", ""checkout_date"")
VALUES
(1, ""title one"", ""creator one"", ""Book"", 2018, ""2021-08-12 00:01""),
(2, ""title two"", ""creator one"", ""Book"", 2019, ""2021-08-12 00:02""),
(3, ""title three"", NULL, ""DVD"", 2020, ""2021-08-12 00:03""),
(4, ""title four"", NULL, ""DVD"", NULL, ""2021-08-12 00:04""),
(5, ""title five"", NULL, ""DVD"", NULL, ""2021-08-12 00:05"")
""""""

with sqlite3.connect('test_bib.db') as con:
    con.execute(sql)
```

when I run the `extract()` method ... 

```python3
db[""circulation_info""].extract(
    [
        ""bib_title"",
        ""bib_creator"",
        ""bib_format""        
    ],
    table=""bib_info"", 
    fk_column=""bib_info_id""
)

db = sqlite_utils.Database(""test_bib.db"")

for row in db[""circulation_info""].rows:
    print(row)

print(""\n---\n"")

for row in db[""bib_info""].rows:
    print(row)
```

results in this .. 
```
{'id': 1, 'item_id': 1, 'bib_info_id': 1, 'bib_pub_year': 2018, 'checkout_date': '2021-08-12 00:01'}
{'id': 2, 'item_id': 2, 'bib_info_id': 2, 'bib_pub_year': 2019, 'checkout_date': '2021-08-12 00:02'}
{'id': 3, 'item_id': 3, 'bib_info_id': None, 'bib_pub_year': 2020, 'checkout_date': '2021-08-12 00:03'}
{'id': 4, 'item_id': 4, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:04'}
{'id': 5, 'item_id': 5, 'bib_info_id': None, 'bib_pub_year': None, 'checkout_date': '2021-08-12 00:05'}

---

{'id': 1, 'bib_title': 'title one', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 2, 'bib_title': 'title two', 'bib_creator': 'creator one', 'bib_format': 'Book'}
{'id': 3, 'bib_title': 'title three', 'bib_creator': None, 'bib_format': 'DVD'}
{'id': 4, 'bib_title': 'title four', 'bib_creator': None, 'bib_format': 'DVD'}
{'id': 5, 'bib_title': 'title five', 'bib_creator': None, 'bib_format': 'DVD'}
```

Seems like it's correctly generating the row data for those lookups, but it's not correctly updating the foreign key back to the primary table? Looks like it just results in a `NULL` value in that original table.

Any ideas on why? Thanks again!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values,
https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896381184,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896381184,IC_kwDOCGYnMM41bbEA,9599,simonw,2021-08-10T23:33:33Z,2021-08-10T23:33:33Z,OWNER,"Now live at https://sqlite-utils.datasette.io/en/latest/reference.html

TIL from what I learned today here: https://til.simonwillison.net/sphinx/sphinx-autodoc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-896378525,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,896378525,IC_kwDODFE5qs41baad,28565,maxhawkins,2021-08-10T23:28:45Z,2021-08-10T23:28:45Z,NONE,"I added parsing of text/html emails using BeautifulSoup.

Around half of the emails in my archive don't include a text/plain payload so adding html parsing makes a good chunk of them searchable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896162082,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896162082,IC_kwDOCGYnMM41alki,22429695,codecov[bot],2021-08-10T17:10:39Z,2021-08-10T23:07:35Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#312](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (43bc064) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/ee469e3122d6f5973ec2584c1580d930daca2e7c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ee469e3) will **decrease** coverage by `0.02%`.
> The diff coverage is `96.84%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/312/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main     #312      +/-   ##
==========================================
- Coverage   96.30%   96.28%   -0.03%     
==========================================
  Files           5        5              
  Lines        2168     2179      +11     
==========================================
+ Hits         2088     2098      +10     
- Misses         80       81       +1     
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/312/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.91% <96.84%> (-0.08%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ee469e3...43bc064](https://codecov.io/gh/simonw/sqlite-utils/pull/312?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896369551,https://api.github.com/repos/simonw/sqlite-utils/issues/314,896369551,IC_kwDOCGYnMM41bYOP,9599,simonw,2021-08-10T23:06:41Z,2021-08-10T23:06:41Z,OWNER,"I took a big bite out of this when I annotated the ``.insert()`` method - but there are a bunch of other places that still need doing: https://github.com/simonw/sqlite-utils/blob/43bc06481783c3cfcee70c0cb541a686e8894adb/sqlite_utils/db.py#L2382-L2397

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965210966,Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__`,
https://github.com/simonw/sqlite-utils/issues/314#issuecomment-896344833,https://api.github.com/repos/simonw/sqlite-utils/issues/314,896344833,IC_kwDOCGYnMM41bSMB,9599,simonw,2021-08-10T22:07:34Z,2021-08-10T22:07:34Z,OWNER,"Also the `.insert()` family of methods - they look pretty ugly in Sphinx right now:

<img width=""688"" alt=""API_Reference_—_sqlite-utils_3_15-16-gf51b712_documentation"" src=""https://user-images.githubusercontent.com/9599/128941267-51b290cb-6c3d-44b7-a584-e3ae3a819e8b.png"">

I should probably define reusable types for things like `pk=`, which have complex type signatures (a string or a list/tuple of strings) and show up in multiple places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965210966,Type signatures for `.create_table()` and `.create_table_sql()` and `.create()` and `Table.__init__`,
https://github.com/simonw/sqlite-utils/issues/315#issuecomment-896339144,https://api.github.com/repos/simonw/sqlite-utils/issues/315,896339144,IC_kwDOCGYnMM41bQzI,9599,simonw,2021-08-10T21:55:41Z,2021-08-10T21:55:41Z,OWNER,Or should we raise an error if you attempt to call `.delete_where()` on a table that doesn't exist?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965440017,`.delete_where()` returns `[]` when it should return self,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896284722,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896284722,IC_kwDOCGYnMM41bDgy,9599,simonw,2021-08-10T20:08:03Z,2021-08-10T20:08:21Z,OWNER,"Spotted a rogue backtick:

![A0147E27-7506-49B0-BEFB-20D99BBFEBAD](https://user-images.githubusercontent.com/9599/128927930-b3333dee-a385-409b-a945-f108e6ea40df.jpeg)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896200682,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896200682,IC_kwDOCGYnMM41au_q,9599,simonw,2021-08-10T18:03:40Z,2021-08-10T18:03:40Z,OWNER,"Adding type signatures to `create_table()` and `.create_table_sql()` is a bit too involved, I'll do that in a separate issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896186025,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896186025,IC_kwDOCGYnMM41arap,9599,simonw,2021-08-10T17:42:51Z,2021-08-10T17:42:51Z,OWNER,That worked! https://sqlite-utils.datasette.io/en/autodoc/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896182934,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896182934,IC_kwDOCGYnMM41aqqW,9599,simonw,2021-08-10T17:38:44Z,2021-08-10T17:38:44Z,OWNER,"From https://docs.readthedocs.io/en/stable/config-file/v2.html#packages it looks like I can tell Read The Docs to run `pip install -e .` using a `.readthedocs.yaml` configuration:

```yaml
version: 2

sphinx:
   configuration: docs/conf.py

python:
  version: ""3.9""
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896180956,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896180956,IC_kwDOCGYnMM41aqLc,9599,simonw,2021-08-10T17:35:51Z,2021-08-10T17:35:51Z,OWNER,Reading the rest of https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration it suggests using a `requirements.txt` file to install dependencies - but I use `setup.py` for that so I need to figure out a different pattern here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896175438,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896175438,IC_kwDOCGYnMM41ao1O,9599,simonw,2021-08-10T17:28:19Z,2021-08-10T17:28:19Z,OWNER,"https://sphinx-rtd-tutorial.readthedocs.io/en/latest/sphinx-config.html#autodoc-configuration says do something like this at the top of `conf.py`:

```python
import os
import sys
sys.path.insert(0, os.path.abspath('../../simpleble/'))
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896174456,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896174456,IC_kwDOCGYnMM41aol4,9599,simonw,2021-08-10T17:27:01Z,2021-08-10T17:27:01Z,OWNER,"Docs are now building at https://sqlite-utils.datasette.io/en/autodoc/reference.html

But there's a problem! The page is semi-blank:

<img width=""663"" alt=""Reference_—_sqlite-utils_3_15-6-gc11ff89_documentation"" src=""https://user-images.githubusercontent.com/9599/128906373-7463e0c9-3b08-4b15-a73a-84b8ba6ab342.png"">

I need to teach Read The Docs how to ensure `sqlite_utils` is available for introspection.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896156971,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896156971,IC_kwDOCGYnMM41akUr,9599,simonw,2021-08-10T17:04:22Z,2021-08-10T17:05:59Z,OWNER,"I'm going to get Read The Docs to build the docs for this branch too - on https://readthedocs.org/projects/sqlite-utils/versions/ I am clicking this button:

<img width=""881"" alt=""Versions___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/128903015-0f2649e7-34a8-4dfd-984c-b1b5bd8a98be.png"">

I then set it to ""active"" (so pushes to the branch will build it) and ""hidden"" (so it wouldn't show up in search or in the navigation menu). https://docs.readthedocs.io/en/stable/versions.html#version-states

<img width=""846"" alt=""autodoc_-_sqlite-utils___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/128903217-efe827c6-4f54-42dd-943e-ddf9ab3a05d8.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/pull/312#issuecomment-896154028,https://api.github.com/repos/simonw/sqlite-utils/issues/312,896154028,IC_kwDOCGYnMM41ajms,9599,simonw,2021-08-10T17:01:06Z,2021-08-10T17:01:06Z,OWNER,"On Python 3.6:

```
sqlite_utils/db.py:366: in Database
    def tables(self) -> List[Table]:
E   NameError: name 'Table' is not defined
```
Python 3.7 can fix this with `from __future__ import annotations` but since we still support 3.6 I'll have to use a string instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965143346,Add reference page to documentation using Sphinx autodoc,
https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896152812,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896152812,IC_kwDOCGYnMM41ajTs,9599,simonw,2021-08-10T16:59:34Z,2021-08-10T16:59:34Z,OWNER,Work will continue in PR #312.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings,
https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896149590,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896149590,IC_kwDOCGYnMM41aihW,9599,simonw,2021-08-10T16:55:36Z,2021-08-10T16:55:36Z,OWNER,"I'm going to use this as an excuse to add a bunch more type signatures too, refs #266.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings,
https://github.com/simonw/sqlite-utils/issues/311#issuecomment-896131902,https://api.github.com/repos/simonw/sqlite-utils/issues/311,896131902,IC_kwDOCGYnMM41aeM-,9599,simonw,2021-08-10T16:31:51Z,2021-08-10T16:31:51Z,OWNER,"`make livehtml` wasn't picking up changes I made to the docstrings `.py` files.

Fix was to change it to this:
```
sphinx-autobuild -a -b html ""$(SOURCEDIR)"" ""$(BUILDDIR)"" $(SPHINXOPTS) $(0) --watch ../sqlite_utils
```
See https://github.com/executablebooks/sphinx-autobuild#relevant-sphinx-bugs - though that suggested `-a` but didn't suggest `--watch`, which is a tip I got from https://github.com/executablebooks/sphinx-autobuild#working-on-a-sphinx-html-theme
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",965102534,Add reference documentation generated from docstrings,
https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895622908,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895622908,IC_kwDOCGYnMM41Yh78,9599,simonw,2021-08-09T23:40:29Z,2021-08-09T23:40:29Z,OWNER,TIL about how the stack inspection works: https://til.simonwillison.net/python/find-local-variables-in-exception-traceback,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible",
https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895581038,IC_kwDOCGYnMM41YXtu,9599,simonw,2021-08-09T22:03:54Z,2021-08-09T23:39:53Z,OWNER,"Steps to reproduce:

    echo '{""v"": 34223049823094832094802398430298048240}' | sqlite-utils insert /tmp/blah.db row -","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible",
https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895592507,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895592507,IC_kwDOCGYnMM41Yag7,9599,simonw,2021-08-09T22:26:28Z,2021-08-09T22:33:48Z,OWNER,"Demo:
```
$ echo '{""v"": 34223049823094832094802398430298048240}' | sqlite-utils insert /tmp/blah.db row -                   
Error: Python int too large to convert to SQLite INTEGER

sql = INSERT INTO [row] ([v]) VALUES (?);
parameters = [34223049823094832094802398430298048240]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible",
https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895587441,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895587441,IC_kwDOCGYnMM41YZRx,9599,simonw,2021-08-09T22:15:45Z,2021-08-09T22:15:45Z,OWNER,"```
OverflowError: Python int too large to convert to SQLite INTEGER
>>> import sys
>>> def find_variables(tb, vars):
        to_find = list(vars)
        found = {}
        for var in to_find:
            if var in tb.tb_frame.f_locals:
                vars.remove(var)
                found[var] = tb.tb_frame.f_locals[var]
        if vars and tb.tb_next:
            found.update(find_variables(tb.tb_next, vars))
        return found
... 
>>> find_variables(sys.last_traceback, [""sql"", ""params""])
{'params': [34223049823094832094802398430298048240], 'sql': 'INSERT INTO [row] ([v]) VALUES (?);'}
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible",
https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895587282,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895587282,IC_kwDOCGYnMM41YZPS,9599,simonw,2021-08-09T22:15:25Z,2021-08-09T22:15:25Z,OWNER,"I'm going to use a bit of a dirty trick for this one: I'm going to recursively inspect the stack on an error and try to find the `sql` and `params` variables.

That way I can handle this all at the CLI layer without changing the exceptions that are being raised by the Python library.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible",
https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895577012,https://api.github.com/repos/simonw/sqlite-utils/issues/309,895577012,IC_kwDOCGYnMM41YWu0,9599,simonw,2021-08-09T21:55:52Z,2021-08-09T21:59:03Z,OWNER,"Yeah this error message could certainly be more helpful.

I thought `OverflowError` might be one of the SQLite exceptions: https://docs.python.org/3/library/sqlite3.html#exceptions - but it turns out it's actually reusing the Python built-in `OverflowError` class:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
caught = []
try:
    db.execute(""create table foo (number integer)"")
    db.execute(""insert into foo (number) values (?)"", [34223049823094832094802398430298048240])
except Exception as e:
    print(e)
    caught.append(e)
isinstance(caught[0], OverflowError)
```
Here's where that happens in the Python `sqlite3` module code: https://github.com/python/cpython/blob/058fb35b57ca8c5063d16ec818e668b3babfea65/Modules/_sqlite/util.c#L123-L124","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963897111,"sqlite-utils insert errors should show SQL and parameters, if possible",
https://github.com/simonw/sqlite-utils/issues/310#issuecomment-895572309,https://api.github.com/repos/simonw/sqlite-utils/issues/310,895572309,IC_kwDOCGYnMM41YVlV,9599,simonw,2021-08-09T21:46:15Z,2021-08-09T21:46:15Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#flattening-nested-json-objects,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964400482,`sqlite-utils insert --flatten` option to flatten nested JSON,
https://github.com/simonw/sqlite-utils/issues/310#issuecomment-895571420,https://api.github.com/repos/simonw/sqlite-utils/issues/310,895571420,IC_kwDOCGYnMM41YVXc,9599,simonw,2021-08-09T21:44:38Z,2021-08-09T21:44:38Z,OWNER,When I ship this I should update the TILs at https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs and https://til.simonwillison.net/jq/flatten-nested-json-objects-jq to reference it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964400482,`sqlite-utils insert --flatten` option to flatten nested JSON,
https://github.com/simonw/datasette/issues/1426#issuecomment-895522818,https://api.github.com/repos/simonw/datasette/issues/1426,895522818,IC_kwDOBm6k_c41YJgC,9599,simonw,2021-08-09T20:34:10Z,2021-08-09T20:34:10Z,OWNER,At the very least Datasette should serve a blank `/robots.txt` by default - I'm seeing a ton of 404s for it in the logs.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/simonw/datasette/issues/1426#issuecomment-895510773,https://api.github.com/repos/simonw/datasette/issues/1426,895510773,IC_kwDOBm6k_c41YGj1,9599,simonw,2021-08-09T20:14:50Z,2021-08-09T20:19:22Z,OWNER,"https://twitter.com/mal/status/1424825895139876870

> True pinging google should be part of the build process on a static site :)

That's another aspect of this: if you DO want your site crawled, teaching the `datasette publish` command how to ping Google when a deploy has gone out could be a nice improvement.

Annoyingly it looks like you need to configure an auth token of some sort in order to use their API though, which is likely too much hassle to be worth building into Datasette itself: https://developers.google.com/search/apis/indexing-api/v3/using-api

```
curl -X POST https://indexing.googleapis.com/v3/urlNotifications:publish -d '{
  ""url"": ""https://careers.google.com/jobs/google/technical-writer"",
  ""type"": ""URL_UPDATED""
}' -H ""Content-Type: application/json""

{
  ""error"": {
    ""code"": 401,
    ""message"": ""Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."",
    ""status"": ""UNAUTHENTICATED""
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/simonw/datasette/issues/1426#issuecomment-895509536,https://api.github.com/repos/simonw/datasette/issues/1426,895509536,IC_kwDOBm6k_c41YGQg,9599,simonw,2021-08-09T20:12:57Z,2021-08-09T20:12:57Z,OWNER,I could try out the `X-Robots` HTTP header too: https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#xrobotstag,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/simonw/datasette/issues/1426#issuecomment-895500565,https://api.github.com/repos/simonw/datasette/issues/1426,895500565,IC_kwDOBm6k_c41YEEV,9599,simonw,2021-08-09T20:00:04Z,2021-08-09T20:00:04Z,OWNER,"A few options for how this would work:

- `datasette ... --robots allow`
- `datasette ... --setting robots allow`

Options could be:

- `allow` - allow all crawling
- `deny` - deny all crawling
- `limited` - allow access to the homepage and the index pages for each database and each table, but disallow crawling any further than that

The ""limited"" mode is particularly interesting. Could even make it the default, but I think that may be a bit too confusing. Idea would be to get the key pages indexed but use `nofollow` to discourage crawlers from indexing individual row pages or deep pages like `https://datasette.io/content/repos?_facet=owner&_facet=language&_facet_array=topics&topics__arraycontains=sqlite#facet-owner`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",964322136,"Manage /robots.txt in Datasette core, block robots by default",
https://github.com/simonw/datasette/issues/1425#issuecomment-895003796,https://api.github.com/repos/simonw/datasette/issues/1425,895003796,IC_kwDOBm6k_c41WKyU,3243482,abdusco,2021-08-09T07:14:35Z,2021-08-09T07:14:35Z,CONTRIBUTOR,"I believe this also provides a workaround for the problem I face in https://github.com/simonw/datasette/issues/1300. 

Now I should be able to get table PKs and generate a row URL. I'll test this out and report my findings.


```py
from datasette.utils import path_from_row_pks

pks = await db.primary_keys(table)
url = self.ds.urls.row_blob(
    database,
    table,
    path_from_row_pks(row, pks, not pks),
    column,
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1421#issuecomment-894930013,https://api.github.com/repos/simonw/datasette/issues/1421,894930013,IC_kwDOBm6k_c41V4xd,9599,simonw,2021-08-09T03:38:06Z,2021-08-09T03:38:06Z,OWNER,"Amusing edge-case: if you run this against a `explain ...` query it falls back to using regular expressions, because `explain explain select ...` is invalid SQL. https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894929769,https://api.github.com/repos/simonw/datasette/issues/1421,894929769,IC_kwDOBm6k_c41V4tp,9599,simonw,2021-08-09T03:36:49Z,2021-08-09T03:36:49Z,OWNER,"SQLite carries a warning about using `EXPLAIN` like this: https://www.sqlite.org/lang_explain.html

> The output from EXPLAIN and EXPLAIN QUERY PLAN is intended for interactive analysis and troubleshooting only. The details of the output format are subject to change from one release of SQLite to the next. Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

I think that's OK here, because of the regular expression fallback. If the format changes in the future in a way that breaks the query the error should be caught and the regex-captured parameters should be returned instead.

Hmmm... actually that's not entirely true:

https://github.com/simonw/datasette/blob/b1fed48a95516ae84c0f020582303ab50ab817e2/datasette/utils/__init__.py#L1084-L1091

If the format changes such that the same columns are returned but the `[row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""]` list comprehension returns an empty array it will break Datasette!

I'm going to take that risk for the moment, but I'll actively watch out for problems in the future. If this does turn out to be bad I can always go back to the pure regular expression mechanism.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894929080,https://api.github.com/repos/simonw/datasette/issues/1421,894929080,IC_kwDOBm6k_c41V4i4,9599,simonw,2021-08-09T03:33:02Z,2021-08-09T03:33:02Z,OWNER,"Fixed! Fantastic, this one has been bothering me for *years*.

https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27

<img width=""707"" alt=""fixtures__select___from_facetable_where_state____state_and_on_earth____on_earth_and_neighborhood_not_like__00_04__and_pyinfra_pip_py_at_current_·_Fizzadar_pyinfra"" src=""https://user-images.githubusercontent.com/9599/128657807-80ac818d-6fd9-4f70-ad26-900cec6a7482.png"">


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894927185,https://api.github.com/repos/simonw/datasette/issues/1421,894927185,IC_kwDOBm6k_c41V4FR,9599,simonw,2021-08-09T03:25:01Z,2021-08-09T03:25:01Z,OWNER,"One catch with this approach: if the SQL query is invalid, the parameters will not be extracted and shown as form fields.

Maybe that's completely fine? Why display a form if it's going to break when the user actually runs the query?

But it does bother me. I worry that someone who is still iterating on and editing their query before actually starting to use it might find the behaviour confusing.

So maybe if the query raises an exception it could fall back on the regular expression results?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894925914,https://api.github.com/repos/simonw/datasette/issues/1421,894925914,IC_kwDOBm6k_c41V3xa,9599,simonw,2021-08-09T03:20:42Z,2021-08-09T03:20:42Z,OWNER,"I think this works!

```python
_re_named_parameter = re.compile("":([a-zA-Z0-9_]+)"")

async def derive_named_parameters(db, sql):
    explain = 'explain {}'.format(sql.strip().rstrip("";""))
    possible_params = _re_named_parameter.findall(sql)
    try:
        results = await db.execute(explain, {p: None for p in possible_params})
        return [row[""p4""].lstrip("":"") for row in results if row[""opcode""] == ""Variable""]
    except sqlite3.DatabaseError:
        return []
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894925437,https://api.github.com/repos/simonw/datasette/issues/1421,894925437,IC_kwDOBm6k_c41V3p9,9599,simonw,2021-08-09T03:19:00Z,2021-08-09T03:19:00Z,OWNER,"This may not work:

> `ERROR: sql = 'explain select 1 + :one + :two', params = None: You did not supply a value for binding 1.`

The `explain` queries themselves want me to pass them parameters.

I could try using the regex to pull out candidates and passing `None` for each of those, including incorrect ones like `:31`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894922703,https://api.github.com/repos/simonw/datasette/issues/1421,894922703,IC_kwDOBm6k_c41V2_P,9599,simonw,2021-08-09T03:09:29Z,2021-08-09T03:09:29Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/ad90a72afa21b737b162e2bbdddc301a97d575cd/datasette/views/database.py#L225-L231,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894922145,https://api.github.com/repos/simonw/datasette/issues/1421,894922145,IC_kwDOBm6k_c41V22h,9599,simonw,2021-08-09T03:07:38Z,2021-08-09T03:07:38Z,OWNER,"I hoped this would work:
```sql
with foo as (
  explain select * from facetable
  where state = :state
  and on_earth = :on_earth
  and neighborhood not like '00:04'
)
select p4 from foo where opcode = 'Variable'
```
 But sadly [it returns an error](https://latest.datasette.io/fixtures?sql=with+foo+as+%28%0D%0A++explain+select+*+from+facetable%0D%0A++where+state+%3D+%3Astate%0D%0A++and+on_earth+%3D+%3Aon_earth%0D%0A++and+neighborhood+not+like+%2700%3A04%27%0D%0A%29%0D%0Aselect+p4+from+foo+where+opcode+%3D+%27Variable%27&state=&on_earth=&04=):

> near ""explain"": syntax error","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894921512,https://api.github.com/repos/simonw/datasette/issues/1421,894921512,IC_kwDOBm6k_c41V2so,9599,simonw,2021-08-09T03:05:26Z,2021-08-09T03:05:26Z,OWNER,"I may have a way to work around this, using `explain`. Consider this query:

```sql
select * from facetable
where state = :state
and on_earth = :on_earth
and neighborhood not like '00:04'
```
Datasette currently gets confused and shows three form fields: https://latest.datasette.io/fixtures?sql=select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=

<img width=""698"" alt=""fixtures__select___from_facetable_where_state____state_and_on_earth____on_earth_and_neighborhood_not_like__00_04__and_pyinfra_pip_py_at_current_·_Fizzadar_pyinfra"" src=""https://user-images.githubusercontent.com/9599/128656369-8af860bf-f7c1-4c9c-beba-1eb6887a8336.png"">

But... if I run `explain` [against that](https://latest.datasette.io/fixtures?sql=explain+select+*+from+facetable%0D%0Awhere+state+%3D+%3Astate%0D%0Aand+on_earth+%3D+%3Aon_earth%0D%0Aand+neighborhood+not+like+%2700%3A04%27&state=&on_earth=&04=) I get this (truncated):

addr | opcode | p1 | p2 | p3 | p4 | p5 | comment
-- | -- | -- | -- | -- | -- | -- | --
20 | ResultRow | 6 | 10 | 0 |   | 0 |  
21 | Next | 0 | 3 | 0 |   | 1 |  
22 | Halt | 0 | 0 | 0 |   | 0 |  
23 | Transaction | 0 | 0 | 35 | 0 | 1 |  
24 | Variable | 1 | 2 | 0 | :state | 0 |  
25 | Variable | 2 | 3 | 0 | :on_earth | 0 |  
26 | String8 | 0 | 4 | 0 | 00:04 | 0 |  
27 | Goto | 0 | 1 | 0 |   | 0 |  

Could it be as simple as pulling out those `Variable` rows to figure out the names of the variables in the query?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1425#issuecomment-894900267,https://api.github.com/repos/simonw/datasette/issues/1425,894900267,IC_kwDOBm6k_c41Vxgr,9599,simonw,2021-08-09T01:31:22Z,2021-08-09T01:31:22Z,OWNER,"I used this to build a new plugin: https://github.com/simonw/datasette-query-links

Demo here: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++%27select+*+from+[facetable]%27+as+query%0D%0Aunion%0D%0Aselect%0D%0A++%27select+sqlite_version()%27%0D%0Aunion%0D%0Aselect%0D%0A++%27select+this+is+invalid+SQL+so+will+not+be+linked%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894893319,https://api.github.com/repos/simonw/datasette/issues/1425,894893319,IC_kwDOBm6k_c41Vv0H,9599,simonw,2021-08-09T01:08:56Z,2021-08-09T01:09:12Z,OWNER,Demo: https://latest.datasette.io/fixtures/simple_primary_key shows `RENDER_CELL_ASYNC_RESULT` where the CSV version shows `RENDER_CELL_ASYNC`: https://latest.datasette.io/fixtures/simple_primary_key.csv - because of this test plugin code: https://github.com/simonw/datasette/blob/a390bdf9cef01d8723d025fc3348e81345ff4856/tests/plugins/my_plugin.py#L98-L122,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894884874,https://api.github.com/repos/simonw/datasette/issues/1425,894884874,IC_kwDOBm6k_c41VtwK,9599,simonw,2021-08-09T00:38:20Z,2021-08-09T00:38:20Z,OWNER,I'm trying the version where I remove `firstresult=True`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894883664,https://api.github.com/repos/simonw/datasette/issues/1425,894883664,IC_kwDOBm6k_c41VtdQ,9599,simonw,2021-08-09T00:33:56Z,2021-08-09T00:33:56Z,OWNER,"I could extract that code out and write my own function which implements the equivalent of calling `pm.hook.render_cell(...)` but runs `await_me_maybe()` before checking if `res is not None`.

That's pretty nasty.

Could I instead call the plugin hook normally, but then have additional logic which says ""if I await it and it returns `None` then try calling the hook again but skip this one"" - not sure if there's a way to do that either.

I could remove the `firstresult=True` from the hookspec - which would cause it to call and return ALL hooks - but then in my own code use only the first one. This is slightly less efficient (since it calls all the hooks and then discards all-but-one value) but it's the least unpleasant in terms of the code I would have to write - plus I don't think it's going to be THAT common for someone to have multiple expensive `render_cell()` hooks installed at once (they are usually pretty cheap).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894882642,https://api.github.com/repos/simonw/datasette/issues/1425,894882642,IC_kwDOBm6k_c41VtNS,9599,simonw,2021-08-09T00:29:57Z,2021-08-09T00:29:57Z,OWNER,"Here's the code in `pluggy` that implements this: https://github.com/pytest-dev/pluggy/blob/0a064fe275060dbdb1fe6e10c888e72bc400fb33/src/pluggy/callers.py#L31-L43

```python
                if hook_impl.hookwrapper:
                    try:
                        gen = hook_impl.function(*args)
                        next(gen)  # first yield
                        teardowns.append(gen)
                    except StopIteration:
                        _raise_wrapfail(gen, ""did not yield"")
                else:
                    res = hook_impl.function(*args)
                    if res is not None:
                        results.append(res)
                        if firstresult:  # halt further impl calls
                            break
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894882123,https://api.github.com/repos/simonw/datasette/issues/1425,894882123,IC_kwDOBm6k_c41VtFL,9599,simonw,2021-08-09T00:27:43Z,2021-08-09T00:27:43Z,OWNER,"Good news: `render_cell()` is the only hook to use `firstresult=True`:

https://github.com/simonw/datasette/blob/f3c9edb376a13c09b5ecf97c7390f4e49efaadf2/datasette/hookspecs.py#L62-L64

https://pluggy.readthedocs.io/en/latest/#first-result-only","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894881448,https://api.github.com/repos/simonw/datasette/issues/1425,894881448,IC_kwDOBm6k_c41Vs6o,9599,simonw,2021-08-09T00:24:25Z,2021-08-09T00:24:39Z,OWNER,"My hunch is that the ""skip this `render_cell()` result if it returns `None`"" logic isn't working correctly, ever since I added the `await_me_maybe` line.

Could that be because Pluggy handles the ""do the next if `None` is returned"" logic itself, but I'm no-longer returning `None`, I'm returning an awaitable which when awaited returns `None`.

This would suggest that all of the `await_me_maybe()` plugin hooks have the same bug. That's definitely possible - it may well be that no-one has yet stumbled across a bug caused by a plugin returning an awaitable and hence not being skipped, because plugin hooks that return awaitable are rare enough that no-one has tried two plugins which both use that trick.

Still don't see why it would pass on my laptop but fail in CI though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894881016,https://api.github.com/repos/simonw/datasette/issues/1425,894881016,IC_kwDOBm6k_c41Vsz4,9599,simonw,2021-08-09T00:21:53Z,2021-08-09T00:21:53Z,OWNER,"Still one test failure:
```
    def test_hook_render_cell_link_from_json(app_client):
        sql = """"""
            select '{""href"": ""http://example.com/"", ""label"":""Example""}'
        """""".strip()
        path = ""/fixtures?"" + urllib.parse.urlencode({""sql"": sql})
        response = app_client.get(path)
        td = Soup(response.body, ""html.parser"").find(""table"").find(""tbody"").find(""td"")
        a = td.find(""a"")
>       assert a is not None, str(a)
E       AssertionError: None
E       assert None is not None
```
The weird thing about this one is that I can't replicate it on my laptop - but it happens in CI every time, including when I shell in and try to run that single test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894869692,https://api.github.com/repos/simonw/datasette/issues/1425,894869692,IC_kwDOBm6k_c41VqC8,9599,simonw,2021-08-08T23:08:29Z,2021-08-08T23:08:29Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-value-column-table-database-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1425#issuecomment-894865323,https://api.github.com/repos/simonw/datasette/issues/1425,894865323,IC_kwDOBm6k_c41Vo-r,9599,simonw,2021-08-08T22:33:19Z,2021-08-08T22:33:19Z,OWNER,"I can do this with the `await_me_maybe()` function, as seen here: https://github.com/simonw/datasette/blob/a21853c9dade240734abc6b4f750fae09a3e840a/datasette/app.py#L864-L873","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963528457,render_cell() hook should support returning an awaitable,
https://github.com/simonw/datasette/issues/1424#issuecomment-894864744,https://api.github.com/repos/simonw/datasette/issues/1424,894864744,IC_kwDOBm6k_c41Vo1o,9599,simonw,2021-08-08T22:27:31Z,2021-08-08T22:27:31Z,OWNER,https://docs.python.org/3/library/sqlite3.html#exceptions is useful - it looks like `sqlite3.DatabaseError` is the super-class of all of the other exceptions that we might see.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends,
https://github.com/simonw/datasette/issues/1424#issuecomment-894864682,https://api.github.com/repos/simonw/datasette/issues/1424,894864682,IC_kwDOBm6k_c41Vo0q,9599,simonw,2021-08-08T22:26:46Z,2021-08-08T22:26:46Z,OWNER,"Note that the `sqlite3` exceptions are in `sqlite3` if using the Python standard library but are in `pysqlite3` if that module is being used instead.

So maybe encourage people to use them from `datasette.sqlite.sqlite3` instead, which will point to the correct package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends,
https://github.com/simonw/datasette/issues/1424#issuecomment-894864616,https://api.github.com/repos/simonw/datasette/issues/1424,894864616,IC_kwDOBm6k_c41Vozo,9599,simonw,2021-08-08T22:26:08Z,2021-08-08T22:26:08Z,OWNER,"- `datasette.database.QueryInterrupted` for queries that were interrupted
- `sqlite3.OperationalError`
- `sqlite3.DatabaseError` and more","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends,
https://github.com/simonw/datasette/issues/1424#issuecomment-894864404,https://api.github.com/repos/simonw/datasette/issues/1424,894864404,IC_kwDOBm6k_c41VowU,9599,simonw,2021-08-08T22:24:06Z,2021-08-08T22:24:06Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/de5ce2e56339ad8966f417a4758f7c210c017dec/datasette/database.py#L176-L200,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",963527045,Document exceptions that can be raised by db.execute() and friends,
https://github.com/simonw/datasette/issues/1422#issuecomment-894607989,https://api.github.com/repos/simonw/datasette/issues/1422,894607989,IC_kwDOBm6k_c41UqJ1,9599,simonw,2021-08-07T05:31:57Z,2021-08-07T05:31:57Z,OWNER,"Demo: https://latest.datasette.io/fixtures/neighborhood_search

Documentation: https://docs.datasette.io/en/latest/sql_queries.html#additional-canned-query-options","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query,
https://github.com/simonw/datasette/issues/1421#issuecomment-894606843,https://api.github.com/repos/simonw/datasette/issues/1421,894606843,IC_kwDOBm6k_c41Up37,9599,simonw,2021-08-07T05:17:12Z,2021-08-07T05:17:12Z,OWNER,Marking this blocked because I don't have a way around the needing-a-SQLite-SQL-parser problem at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1421#issuecomment-894606796,https://api.github.com/repos/simonw/datasette/issues/1421,894606796,IC_kwDOBm6k_c41Up3M,9599,simonw,2021-08-07T05:16:39Z,2021-08-07T05:16:39Z,OWNER,"Urgh, yeah I've seen this one before. Fixing it pretty much requires writing a full SQLite SQL syntax parser in Python, which is frustratingly complicated for solving this issue!

You can work around this for a canned query by using the optional `params:` argument documented here: https://docs.datasette.io/en/stable/sql_queries.html#canned-query-parameters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959999095,"""Query parameters"" form shows wrong input fields if query contains ""03:31"" style times",
https://github.com/simonw/datasette/issues/1422#issuecomment-894589140,https://api.github.com/repos/simonw/datasette/issues/1422,894589140,IC_kwDOBm6k_c41UljU,9599,simonw,2021-08-07T01:58:16Z,2021-08-07T01:58:24Z,OWNER,Also need to consider this hidden field - it should pass the `_hide_sql` or `_show_sql` parameters depending on the same logic: https://github.com/simonw/datasette/blob/acc22436622ff8476c30acf45ed60f54b4aaa5d9/datasette/templates/query.html#L47-L49,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-894581223,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,894581223,IC_kwDODFE5qs41Ujnn,28565,maxhawkins,2021-08-07T00:57:48Z,2021-08-07T00:57:48Z,NONE,"Just added two more fixes:

* Added parsing for rfc 2047 encoded unicode headers
* Body is now stored as TEXT rather than a BLOB regardless of what order the messages are parsed in.

I was able to run this on my Takeout export and everything seems to work fine. @simonw let me know if this looks good to merge.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/simonw/datasette/issues/1423#issuecomment-894454644,https://api.github.com/repos/simonw/datasette/issues/1423,894454644,IC_kwDOBm6k_c41UEt0,9599,simonw,2021-08-06T18:52:49Z,2021-08-06T18:52:49Z,OWNER,"This means that the counts would be unavailable to users who cannot see tooltips (e.g. mobile users) on pages that did not have any facets that broke the 30 limit and hence displayed that ""..."" link.

I think I'm OK with that, for the moment. May revisit in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1423#issuecomment-894454087,https://api.github.com/repos/simonw/datasette/issues/1423,894454087,IC_kwDOBm6k_c41UElH,9599,simonw,2021-08-06T18:51:42Z,2021-08-06T18:51:42Z,OWNER,"The invisible tooltip could say ""Showing 30 items, more available"" (helping save you from counting up to 20 if you know about the secret feature). The numbers could then be fully displayed on the ""..."" page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1423#issuecomment-894453520,https://api.github.com/repos/simonw/datasette/issues/1423,894453520,IC_kwDOBm6k_c41UEcQ,9599,simonw,2021-08-06T18:50:40Z,2021-08-06T18:50:40Z,OWNER,"Point of confusion: if only 30 options are shown, but there's a `...` at the end, what would the number be? It can't be the total number of facets because we haven't counted them all - but if it's just the number of displayed facets that's like to be confusing.

So the original idea of showing the counts only if `_facet_size=max` is a good one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1423#issuecomment-894452990,https://api.github.com/repos/simonw/datasette/issues/1423,894452990,IC_kwDOBm6k_c41UET-,9599,simonw,2021-08-06T18:49:37Z,2021-08-06T18:49:37Z,OWNER,"Could display them always, like this:

<img width=""1299"" alt=""fixtures__compound_three_primary_keys__1_001_rows_and_School_enrollment_analysis_Colorado_-_Google_Docs"" src=""https://user-images.githubusercontent.com/9599/128557997-6bedb52b-45f1-4ab1-9a79-ddf6c01f6036.png"">

That's with `<span style=""font-size: 0.8em;color: #666;"">23</span>`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1423#issuecomment-893996604,https://api.github.com/repos/simonw/datasette/issues/1423,893996604,IC_kwDOBm6k_c41SU48,9599,simonw,2021-08-06T04:43:07Z,2021-08-06T04:43:37Z,OWNER,"Problem: on a page which doesn't have quite enough facet values to trigger the display of the ""..."" link that links to `?_facet_size=max` the user would still have to manually count the values - up to 30 by default.

So maybe the count should always be shown, perhaps as a non-bold light colored number?

I could even hide it in a non-discoverable tooltip.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",962391325,Show count of facet values if ?_facet_size=max,
https://github.com/simonw/datasette/issues/1419#issuecomment-893133496,https://api.github.com/repos/simonw/datasette/issues/1419,893133496,IC_kwDOBm6k_c41PCK4,9599,simonw,2021-08-05T03:22:44Z,2021-08-05T03:22:44Z,OWNER,"I ran into this exact same problem today! I only just learned how to use filter on aggregates: https://til.simonwillison.net/sqlite/sqlite-aggregate-filter-clauses

A workaround I used is to add this to the deploy command:

    datasette publish cloudrun ... --install=pysqlite3-binary

This will install the https://pypi.org/project/pysqlite3-binary for package which bundles a more recent SQLite version.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,`publish cloudrun` should deploy a more recent SQLite version,
https://github.com/simonw/datasette/issues/1422#issuecomment-893131703,https://api.github.com/repos/simonw/datasette/issues/1422,893131703,IC_kwDOBm6k_c41PBu3,9599,simonw,2021-08-05T03:16:46Z,2021-08-05T03:16:46Z,OWNER,"The logic for this is a little bit fiddly, due to the need to switch to using `?_show_sql=1` on the link depending on the context.

- If metadata says hide and there's no query string, hide and link to `?_show_sql=1`
- If metadata says hide but query string says `?_show_sql=1`, show and have hide link linking to URL without `?_show_sql=1`
- Otherwise, show and link to `?_hide_sql=1`
- ... or if that query string is there then hide and link to URL without `?_hide_sql=1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query,
https://github.com/simonw/datasette/issues/1422#issuecomment-893122356,https://api.github.com/repos/simonw/datasette/issues/1422,893122356,IC_kwDOBm6k_c41O_c0,9599,simonw,2021-08-05T02:52:31Z,2021-08-05T02:52:44Z,OWNER,"If you do this it should still be possible to view the SQL - which means we need a new parameter. I propose `?_show_sql=1` to over-ride the hidden default.

I think the configuration should use `hide_sql: true` - looking like this:
```yaml
databases:
  fixtures:
    queries:
      neighborhood_search:
        hide_sql: true
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable
            join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%'
          order by neighborhood
        title: Search neighborhoods
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961367843,Ability to default to hiding the SQL for a canned query,
https://github.com/simonw/datasette/issues/1419#issuecomment-893114612,https://api.github.com/repos/simonw/datasette/issues/1419,893114612,IC_kwDOBm6k_c41O9j0,536941,fgregg,2021-08-05T02:29:06Z,2021-08-05T02:29:06Z,CONTRIBUTOR,"there's a lot of complexity here, that's probably not worth addressing. i got what i needed by patching the dockerfile that cloudrun uses to install a newer version of sqlite.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,`publish cloudrun` should deploy a more recent SQLite version,
https://github.com/simonw/datasette/issues/1420#issuecomment-893079520,https://api.github.com/repos/simonw/datasette/issues/1420,893079520,IC_kwDOBm6k_c41O0_g,9599,simonw,2021-08-05T00:54:59Z,2021-08-05T00:54:59Z,OWNER,"Just saw this error:

`ERROR: (gcloud.run.deploy) The `--cpu` flag is not supported on the fully managed version of Cloud Run. Specify `--platform gke` or run `gcloud config set run/platform gke` to work with Cloud Run for Anthos deployed on Google Cloud.`

Which is weird because I managed to run this successfully the other day. Maybe a region difference thing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option,
https://github.com/simonw/sqlite-utils/issues/308#issuecomment-892988609,https://api.github.com/repos/simonw/sqlite-utils/issues/308,892988609,IC_kwDOCGYnMM41OezB,9599,simonw,2021-08-04T21:30:59Z,2021-08-04T21:30:59Z,OWNER,"OK, this works great as a proof-of-concept (though maybe I should remove the output from the cells so people have to run it themselves in the Binder environment?)

Next challenge: how extensive should it be? What features should I cover? Should I do a basic tutorial and then several advanced tutorials?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961008507,Add an interactive tutorial as a Jupyter notebook,
https://github.com/simonw/sqlite-utils/issues/308#issuecomment-892957281,https://api.github.com/repos/simonw/sqlite-utils/issues/308,892957281,IC_kwDOCGYnMM41OXJh,9599,simonw,2021-08-04T20:37:00Z,2021-08-04T20:37:00Z,OWNER,https://mybinder.org/v2/gh/simonw/sqlite-utils/main?filepath=docs%2Ftutorial.ipynb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",961008507,Add an interactive tutorial as a Jupyter notebook,
https://github.com/simonw/datasette/issues/1420#issuecomment-892376353,https://api.github.com/repos/simonw/datasette/issues/1420,892376353,IC_kwDOBm6k_c41MJUh,9599,simonw,2021-08-04T05:33:12Z,2021-08-04T05:33:12Z,OWNER,"In the Cloud Run console (before I deleted these services) when I click ""Edit and deploy new revision"" I see this for the default one:

<img width=""551"" alt=""Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform"" src=""https://user-images.githubusercontent.com/9599/128127309-7ec2064f-b107-4d99-b630-ee58734c4a72.png"">

And this for the big one:

<img width=""546"" alt=""Deploy_new_revision_–_Cloud_Run_–_datasette_–_Google_Cloud_Platform"" src=""https://user-images.githubusercontent.com/9599/128127366-7cdcb245-2e83-4b88-9d4f-0c595a486139.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option,
https://github.com/simonw/datasette/issues/1420#issuecomment-892374253,https://api.github.com/repos/simonw/datasette/issues/1420,892374253,IC_kwDOBm6k_c41MIzt,9599,simonw,2021-08-04T05:27:21Z,2021-08-04T05:29:59Z,OWNER,"I'll delete these services shortly afterwards. Right now:

https://fixtures-over-provisioned-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil (deployed with `--memory 8G --cpu 4`) returns:
```
process.memory_info()
  pmem(rss=60456960, vms=518930432, shared=0, text=0, lib=0, data=0, dirty=0)

...

psutil.cpu_times(True)
  scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
  scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
  scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
  scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)

psutil.virtual_memory()
  svmem(total=2147483648, available=2092531712, percent=2.6, used=33103872, free=2092531712, active=44130304, inactive=10792960, buffers=0, cached=21848064, shared=262144, slab=0)
```
https://fixtures-default-issue-1420-j7hipcg4aq-uc.a.run.app/-/psutil returns:
```
process.memory_info()
  pmem(rss=49324032, vms=140595200, shared=0, text=0, lib=0, data=0, dirty=0)

...

psutil.cpu_times(True)
  scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)
  scputimes(user=0.0, nice=0.0, system=0.0, idle=0.0, iowait=0.0, irq=0.0, softirq=0.0, steal=0.0, guest=0.0, guest_nice=0.0)

psutil.virtual_memory()
  svmem(total=2147483648, available=2091188224, percent=2.6, used=40071168, free=2091188224, active=41586688, inactive=7983104, buffers=0, cached=16224256, shared=262144, slab=0)
```
These numbers are different enough that I assume this works as advertised. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option,
https://github.com/simonw/datasette/issues/1420#issuecomment-892372509,https://api.github.com/repos/simonw/datasette/issues/1420,892372509,IC_kwDOBm6k_c41MIYd,9599,simonw,2021-08-04T05:22:29Z,2021-08-04T05:22:29Z,OWNER,"Testing this manually with:

    datasette publish cloudrun fixtures.db --memory 8G --cpu 4 \
      --service fixtures-over-provisioned-issue-1420 --install datasette-psutil

And for comparison:

    datasette publish cloudrun fixtures.db --service fixtures-default-issue-1420 \
      --install datasette-psutil
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option,
https://github.com/simonw/datasette/issues/1420#issuecomment-892365639,https://api.github.com/repos/simonw/datasette/issues/1420,892365639,IC_kwDOBm6k_c41MGtH,9599,simonw,2021-08-04T05:05:07Z,2021-08-04T05:05:07Z,OWNER,https://github.com/simonw/datasette/blob/cd8b7bee8fb5c1cdce7c8dbfeb0166011abc72c6/datasette/publish/cloudrun.py#L153-L158,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959898166,`datasette publish cloudrun --cpu X` option,
https://github.com/simonw/datasette/issues/1419#issuecomment-892276385,https://api.github.com/repos/simonw/datasette/issues/1419,892276385,IC_kwDOBm6k_c41Lw6h,536941,fgregg,2021-08-04T00:58:49Z,2021-08-04T00:58:49Z,CONTRIBUTOR,"yes, [filter clause on aggregate queries were added to sqlite3 in 3.30](https://www.sqlite.org/releaselog/3_30_1.html)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959710008,`publish cloudrun` should deploy a more recent SQLite version,
https://github.com/simonw/datasette/pull/1418#issuecomment-891987129,https://api.github.com/repos/simonw/datasette/issues/1418,891987129,IC_kwDOBm6k_c41KqS5,22429695,codecov[bot],2021-08-03T16:26:01Z,2021-08-03T16:32:20Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1418](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (532170f) into [main](https://codecov.io/gh/simonw/datasette/commit/54b6e96ee8aa553b6671e341a1944f93f3fb89c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (54b6e96) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1418/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1418   +/-   ##
=======================================
  Coverage   91.64%   91.64%           
=======================================
  Files          34       34           
  Lines        4382     4382           
=======================================
  Hits         4016     4016           
  Misses        366      366           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [54b6e96...532170f](https://codecov.io/gh/simonw/datasette/pull/1418?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959284434,Spelling corrections plus CI job for codespell,
https://github.com/simonw/datasette/pull/1418#issuecomment-891984359,https://api.github.com/repos/simonw/datasette/issues/1418,891984359,IC_kwDOBm6k_c41Kpnn,9599,simonw,2021-08-03T16:21:56Z,2021-08-03T16:21:56Z,OWNER,"Failed with:
```
docs/authentication.rst:63: perfom ==> perform
docs/authentication.rst:76: perfom ==> perform
docs/changelog.rst:429: repsonse ==> response
docs/changelog.rst:503: permissons ==> permissions
docs/changelog.rst:717: compatibilty ==> compatibility
docs/changelog.rst:1172: browseable ==> browsable
docs/deploying.rst:191: similiar ==> similar
docs/internals.rst:434: Respons ==> Response, respond
docs/internals.rst:440: Respons ==> Response, respond
docs/internals.rst:717: tha ==> than, that, the
docs/performance.rst:42: databse ==> database
docs/plugin_hooks.rst:667: utilites ==> utilities
docs/publish.rst:168: countainer ==> container
docs/settings.rst:352: inalid ==> invalid
docs/sql_queries.rst:406: preceeded ==> preceded, proceeded
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959284434,Spelling corrections plus CI job for codespell,
https://github.com/simonw/datasette/issues/1417#issuecomment-891979858,https://api.github.com/repos/simonw/datasette/issues/1417,891979858,IC_kwDOBm6k_c41KohS,9599,simonw,2021-08-03T16:15:30Z,2021-08-03T16:15:30Z,OWNER,"Docs: https://pypi.org/project/codespell/

There's a `codespell --ignore-words=FILE` option for ignoring words. I don't have any that need ignoring yet but I'm going to add that file anyway, that way I can have codespell be a failing test but still provide a way to work around it if it incorrectly flags a correct word.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959278472,Use codespell in CI to spot spelling errors,
https://github.com/simonw/sqlite-utils/issues/298#issuecomment-891583131,https://api.github.com/repos/simonw/sqlite-utils/issues/298,891583131,IC_kwDOCGYnMM41JHqb,2172260,qqilihq,2021-08-03T06:50:47Z,2021-08-03T06:50:47Z,NONE,@simonw Awesome; thanks so much!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951581763,Read lines with JSON object,
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-891380382,https://api.github.com/repos/simonw/sqlite-utils/issues/251,891380382,IC_kwDOCGYnMM41IWKe,9599,simonw,2021-08-02T22:39:46Z,2021-08-02T22:39:46Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/stable/cli.html#converting-data-in-columns,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/298#issuecomment-891359751,https://api.github.com/repos/simonw/sqlite-utils/issues/298,891359751,IC_kwDOCGYnMM41IRIH,9599,simonw,2021-08-02T21:55:16Z,2021-08-02T21:55:16Z,OWNER,"This is a feature already! You can do this:

    sqlite-utils insert nl-demo.db mytable data.ndjson --nl

See https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951581763,Read lines with JSON object,
https://github.com/simonw/sqlite-utils/issues/300#issuecomment-891356906,https://api.github.com/repos/simonw/sqlite-utils/issues/300,891356906,IC_kwDOCGYnMM41IQbq,9599,simonw,2021-08-02T21:49:10Z,2021-08-02T21:49:10Z,OWNER,"I'm nervous to set this by default in the Python library because it's a global setting, so I can't isolate it to just code that is being called by a `sqlite_utils` method.

The new `sqlite-utils convert` command from #251 DOES turn it on - I think having it on by default for a CLI tool that uses user-defined functions makes sense: https://github.com/simonw/sqlite-utils/blob/e83aef951bd3e8c179511faddb607239a5fa8682/sqlite_utils/cli.py#L2006

I like the suggestion to include this in the documentation. I'll do that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956832836,Returning underlying cause for User Defined Functions ,
https://github.com/simonw/sqlite-utils/issues/306#issuecomment-891352587,https://api.github.com/repos/simonw/sqlite-utils/issues/306,891352587,IC_kwDOCGYnMM41IPYL,9599,simonw,2021-08-02T21:39:34Z,2021-08-02T21:39:34Z,OWNER,This older TIL would have saved me some time! https://til.simonwillison.net/sphinx/sphinx-ext-extlinks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",958516743,Configure sphinx.ext.extlinks for issues,
https://github.com/simonw/datasette/issues/1227#issuecomment-891352132,https://api.github.com/repos/simonw/datasette/issues/1227,891352132,IC_kwDOBm6k_c41IPRE,9599,simonw,2021-08-02T21:38:39Z,2021-08-02T21:38:39Z,OWNER,Relevant TIL: https://til.simonwillison.net/vscode/vs-code-regular-expressions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810394616,Configure sphinx.ext.extlinks for issues,
https://github.com/simonw/sqlite-utils/issues/306#issuecomment-891352073,https://api.github.com/repos/simonw/sqlite-utils/issues/306,891352073,IC_kwDOCGYnMM41IPQJ,9599,simonw,2021-08-02T21:38:32Z,2021-08-02T21:38:32Z,OWNER,TIL about this: https://til.simonwillison.net/vscode/vs-code-regular-expressions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",958516743,Configure sphinx.ext.extlinks for issues,
https://github.com/simonw/sqlite-utils/issues/304#issuecomment-891257730,https://api.github.com/repos/simonw/sqlite-utils/issues/304,891257730,IC_kwDOCGYnMM41H4OC,9599,simonw,2021-08-02T19:00:00Z,2021-08-02T19:00:00Z,OWNER,"Documented here:

- https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns
- https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957731178,"`table.convert(..., where=)` and `sqlite-utils convert ... --where=`",
https://github.com/simonw/sqlite-utils/issues/305#issuecomment-890713185,https://api.github.com/repos/simonw/sqlite-utils/issues/305,890713185,IC_kwDOCGYnMM41FzRh,9599,simonw,2021-08-02T04:53:53Z,2021-08-02T04:53:53Z,OWNER,I'm going to rename `.execute_count()` to `.count_where()` - but I'll keep `.execute_count()` in the codebase since it's a documented API at https://sqlite-utils.datasette.io/en/3.13/python-api.html#count and I don't want to do a major version bump.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957741820,Python: need a way to execute a count with an extra where clause,
https://github.com/simonw/sqlite-utils/issues/304#issuecomment-890711924,https://api.github.com/repos/simonw/sqlite-utils/issues/304,890711924,IC_kwDOCGYnMM41Fy90,9599,simonw,2021-08-02T04:50:13Z,2021-08-02T04:50:13Z,OWNER,"For the Python function, the args will be `where=None, where_args=None`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957731178,"`table.convert(..., where=)` and `sqlite-utils convert ... --where=`",
https://github.com/simonw/sqlite-utils/pull/303#issuecomment-890553014,https://api.github.com/repos/simonw/sqlite-utils/issues/303,890553014,IC_kwDOCGYnMM41FMK2,22429695,codecov[bot],2021-08-01T16:53:35Z,2021-08-02T04:45:19Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#303](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4c3bf97) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/c7e8d72be9fe8fe0811f685a18eebc637662d41b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c7e8d72) will **increase** coverage by `0.24%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/303/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main     #303      +/-   ##
==========================================
+ Coverage   96.04%   96.28%   +0.24%     
==========================================
  Files           4        5       +1     
  Lines        1998     2129     +131     
==========================================
+ Hits         1919     2050     +131     
  Misses         79       79              
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.91% <100.00%> (+0.36%)` | :arrow_up: |
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.97% <100.00%> (+0.10%)` | :arrow_up: |
| [sqlite\_utils/recipes.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3JlY2lwZXMucHk=) | `100.00% <100.00%> (ø)` | |
| [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/303/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `92.39% <100.00%> (+0.48%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c7e8d72...4c3bf97](https://codecov.io/gh/simonw/sqlite-utils/pull/303?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957536983,sqlite-utils convert command and db[table].convert(...) method,
https://github.com/simonw/sqlite-utils/issues/304#issuecomment-890704624,https://api.github.com/repos/simonw/sqlite-utils/issues/304,890704624,IC_kwDOCGYnMM41FxLw,9599,simonw,2021-08-02T04:28:42Z,2021-08-02T04:28:42Z,OWNER,For the command-line version this can duplicate the `--param` option to allow named parameters in the where clause: https://github.com/simonw/sqlite-utils/blob/c7e8d72be9fe8fe0811f685a18eebc637662d41b/sqlite_utils/cli.py#L1096-L1102,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957731178,"`table.convert(..., where=)` and `sqlite-utils convert ... --where=`",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890553783,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890553783,IC_kwDOCGYnMM41FMW3,9599,simonw,2021-08-01T16:59:09Z,2021-08-01T16:59:09Z,OWNER,I'm going with `recipes.jsonsplit()` rather than `recipe.jsonsplit()` because the Python module containing the recipes will be called `recipes`. I'll set up a `r.jsonsplit()` shortcut too as a convenience.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890552827,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890552827,IC_kwDOCGYnMM41FMH7,9599,simonw,2021-08-01T16:52:00Z,2021-08-01T16:52:00Z,OWNER,I'll finish the work on this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/302#issuecomment-890548009,https://api.github.com/repos/simonw/sqlite-utils/issues/302,890548009,IC_kwDOCGYnMM41FK8p,9599,simonw,2021-08-01T16:18:13Z,2021-08-01T16:18:13Z,OWNER,"Basic API design:

    db[table].convert(""headline"", lambda v: v.upper())

You can pass a list of columns instead of a single game column string to apply it to multiple columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957529248,Python library version of `sqlite-utils convert`,
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448623,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890448623,IC_kwDOCGYnMM41Eyrv,9599,simonw,2021-08-01T04:33:30Z,2021-08-01T04:33:30Z,OWNER,"I've started an implementation in the `convert` branch - no documentation yet, and I've not implemented the recipes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448119,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890448119,IC_kwDOCGYnMM41Eyj3,9599,simonw,2021-08-01T04:28:05Z,2021-08-01T04:30:28Z,OWNER,"In which case I think `--code` should be a positional argument instead:
```
sqlite-utils convert mydb.db mytable col 'recipe.parsedatetime(value, dayfirst=True)'
sqlite-utils convert mydb.db mytable col 'recipe.jsonsplit(value, delimiter="":"")'
sqlite-utils convert mydb.db mytable col 'recipe.jsonsplit(value, delimiter="":"")'
sqlite-utils convert mydb.db mytable col '{""lower"": value.lower(), ""upper"": value.upper()}' --multi
```
One problem with this: we already accept one or more columns. I think that's OK though since the code is now a required argument, so it means we have to treat everything between the table and the final code argument as a column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890447102,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890447102,IC_kwDOCGYnMM41EyT-,9599,simonw,2021-08-01T04:20:18Z,2021-08-01T04:29:26Z,OWNER,"I could stick them in a `recipe` namespace so you do this:
```
sqlite-utils convert mydb.db mytable col --code 'recipe.parsedatetime(value, dayfirst=True)'
sqlite-utils convert mydb.db mytable col --code 'recipe.jsonsplit(value, delimiter="":"")'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890448190,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890448190,IC_kwDOCGYnMM41Eyk-,9599,simonw,2021-08-01T04:28:49Z,2021-08-01T04:28:49Z,OWNER,"Would make sense to accept code from standard input too:

    echo 'value.upper()' | sqlite-utils convert my.db mytable col -","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446808,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446808,IC_kwDOCGYnMM41EyPY,9599,simonw,2021-08-01T04:18:18Z,2021-08-01T04:28:18Z,OWNER,"Or.... how about making the `parsedate()` and `parsedatetime()` and `jsonsplit()` functions available within the namespace that is configured for the `--code` block?

Then you could do something like this:

```
sqlite-utils convert mydb.db mytable col --code 'parsedatetime(value, dayfirst=True)'
sqlite-utils convert mydb.db mytable col --code 'jsonsplit(value, delimiter="":"")'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446943,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446943,IC_kwDOCGYnMM41EyRf,9599,simonw,2021-08-01T04:19:09Z,2021-08-01T04:19:09Z,OWNER,"That's a pretty neat fix, though it's a bit more challenging on the documentation front - maybe the help text for `sqlite-utils convert --help` gets a fair bit longer?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446506,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446506,IC_kwDOCGYnMM41EyKq,9599,simonw,2021-08-01T04:16:36Z,2021-08-01T04:16:36Z,OWNER,"Back to the design board then. One way to handle this would be the long-form:

```
sqlite-utils convert jsonsplit mydb.db mytable mycolumn
sqlite-utils convert parsedatetime mydb.db mytable mycolumn
sqlite-utils convert parsedate mydb.db mytable mycolumn
sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'
```
I like the idea that `lambda` is the default action, but in this form it's required that the second argument (the word after `convert`) be the name of the recipe that is being applied to avoid any potential confusion with the database filename.

An ugly solution would be to make all four of those options available on `sqlite-utils convert` - and return an error if you try and use one of those without specifying the accompanying recipe. That's a bit gross though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890446166,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890446166,IC_kwDOCGYnMM41EyFW,9599,simonw,2021-08-01T04:14:26Z,2021-08-01T04:14:26Z,OWNER,"Problem with the `-r/--recipe` idea: the `parsedate` and `parsedatetime` and `jsonsplit` recipes in the current `sqlite-transform` tool all take additional options.

For `sqlite-transform parsedate` and `parsedatetime`:

```python
@click.option(
    ""--dayfirst"",
    is_flag=True,
    help=""Assume day comes first in ambiguous dates, e.g. 03/04/05"",
)
@click.option(
    ""--yearfirst"",
    is_flag=True,
    help=""Assume year comes first in ambiguous dates, e.g. 03/04/05"",
)
```
For `jsonsplit`:
```python
@click.option(""--delimiter"", default="","", help=""Delimiter to split on"")
@click.option(
    ""--type"",
    type=click.Choice((""int"", ""float"")),
    help=""Type to use for values - int or float (defaults to string)"",
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-890443079,https://api.github.com/repos/simonw/sqlite-utils/issues/251,890443079,IC_kwDOCGYnMM41ExVH,9599,simonw,2021-08-01T03:46:43Z,2021-08-01T03:46:43Z,OWNER,"Note that there's already a concept of `conversions` which might be confused with `convert`? https://sqlite-utils.datasette.io/en/stable/python-api.html#converting-column-values-using-sql-functions

```python
db[""example""].insert({
    ""name"": ""The Bigfoot Discovery Museum""
}, conversions={""name"": ""upper(?)""})
```
I think that's OK though - that's a Python library feature, `sqlite-utils convert` is a CLI thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/datasette/issues/1411#issuecomment-890441844,https://api.github.com/repos/simonw/datasette/issues/1411,890441844,IC_kwDOBm6k_c41ExB0,9599,simonw,2021-08-01T03:27:30Z,2021-08-01T03:27:30Z,OWNER,Confirmed: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 no longer exhibits the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957345476,Canned query ?sql= is pointlessly echoed in query string starting from hidden mode,
https://github.com/simonw/datasette/issues/1409#issuecomment-890400425,https://api.github.com/repos/simonw/datasette/issues/1409,890400425,IC_kwDOBm6k_c41Em6p,9599,simonw,2021-07-31T20:25:16Z,2021-07-31T20:26:25Z,OWNER,"If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.

So maybe there's a future task where I introduce those as both permissions and metadata `""allow_x""` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`.

If I was going to do that I should get it in before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890400121,https://api.github.com/repos/simonw/datasette/issues/1409,890400121,IC_kwDOBm6k_c41Em15,9599,simonw,2021-07-31T20:22:21Z,2021-07-31T20:23:34Z,OWNER,"I think `default_allow_sql` is more consistent with the current naming conventions, because both `allow` and `default` are used as prefixes at the moment but neither of them are ever used as a suffix.

Plus `default_allow_sql off` makes sense to me but `allow_default_sql off` does not - what is ""default SQL""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890400059,https://api.github.com/repos/simonw/datasette/issues/1409,890400059,IC_kwDOBm6k_c41Em07,9599,simonw,2021-07-31T20:21:51Z,2021-07-31T20:21:51Z,OWNER,"One of these two options:

- `--setting default_allow_sql off`
- `--setting allow_sql_default off`

Existing settings from https://docs.datasette.io/en/0.58.1/settings.html with similar names that I need to be consistent with:

- `default_page_size`
- `allow_facet`
- `default_facet_size`
- `allow_download`
- `default_cache_ttl`
- `default_cache_ttl_hashed`
- `allow_csv_stream`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890399806,https://api.github.com/repos/simonw/datasette/issues/1409,890399806,IC_kwDOBm6k_c41Emw-,9599,simonw,2021-07-31T20:18:46Z,2021-07-31T20:18:46Z,OWNER,"My rationale for removing it: https://github.com/simonw/datasette/issues/813#issuecomment-640916290

> Naming problem: Datasette already has a config option with this name:
>
>     $ datasette serve data.db --config allow_sql:1
>
> https://datasette.readthedocs.io/en/stable/config.html#allow-sql
>
> It's confusing to have two things called `allow_sql` that do slightly different things.
>
> I could retire the `--config allow_sql:0` option entirely, since the new `metadata.json` mechanism can be used to achieve the exact same thing.
> 
> I'm going to do that.

This is true. The `""allow_sql""` permissions block in `metadata.json` does indeed have a name that is easily confused with `--setting allow_sql off`.

So I definitely need to pick a different name from the setting. `--setting default_allow_sql off` is a good option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890397753,https://api.github.com/repos/simonw/datasette/issues/1409,890397753,IC_kwDOBm6k_c41EmQ5,9599,simonw,2021-07-31T19:57:56Z,2021-07-31T19:57:56Z,OWNER,"I think the correct solution is for the default permissions logic to take the `allow_sql` setting into account, and to return `False` if that setting is set to `off` AND the current actor fails the `actor_matches_allow` checks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890397652,https://api.github.com/repos/simonw/datasette/issues/1409,890397652,IC_kwDOBm6k_c41EmPU,9599,simonw,2021-07-31T19:56:48Z,2021-07-31T19:56:48Z,OWNER,"The other option would be to use the setting to pick the `default=` argument when calling `self.ds.permission_allowed( request.actor, ""execute-sql"", resource=database, default=True)`.

The problem with that is that there are actually a few different places which perform that check, so changing all of them raises the risk of missing one in the future:

https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L436-L444

https://github.com/simonw/datasette/blob/a6c8e7fa4cffdeff84e9e755dcff4788fd6154b8/datasette/views/table.py#L964-L966

https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L220-L221

https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L343-L345

https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/database.py#L134-L136
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890397261,https://api.github.com/repos/simonw/datasette/issues/1409,890397261,IC_kwDOBm6k_c41EmJN,9599,simonw,2021-07-31T19:52:25Z,2021-07-31T19:52:25Z,OWNER,I think I can make this modification by teaching the default permissions code here to take the `allow_sql` setting into account: https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/datasette/default_permissions.py#L38-L45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890397169,https://api.github.com/repos/simonw/datasette/issues/1409,890397169,IC_kwDOBm6k_c41EmHx,9599,simonw,2021-07-31T19:51:35Z,2021-07-31T19:51:35Z,OWNER,I'm going to stick with `--setting allow_sql off`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1409#issuecomment-890397124,https://api.github.com/repos/simonw/datasette/issues/1409,890397124,IC_kwDOBm6k_c41EmHE,9599,simonw,2021-07-31T19:51:10Z,2021-07-31T19:51:10Z,OWNER,"I think I may like `disable_sql` better. Some options:

- `--setting allow_sql off` (consistent with `allow_facet` and `allow_download` and `allow_csv_stream` - all which default to `on` already)
- `--setting disable_sql on`
- `--setting disable_custom_sql on`

The existence of three `allow_*` settings does make a strong argument for staying consistent with that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957310278,`default_allow_sql` setting (a re-imagining of the old `allow_sql` setting),
https://github.com/simonw/datasette/issues/1408#issuecomment-890390845,https://api.github.com/repos/simonw/datasette/issues/1408,890390845,IC_kwDOBm6k_c41Ekk9,9599,simonw,2021-07-31T19:00:32Z,2021-07-31T19:00:32Z,OWNER,"When I revisit this I can also look at dropping the `@pytest.mark.serial` hack, and maybe the `restore_working_directory()` fixture hack too:

https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/pytest.ini#L9-L10

https://github.com/simonw/datasette/blob/ff253f5242e4b0b5d85d29d38b8461feb5ea997a/tests/conftest.py#L62-L75","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,"Review places in codebase that use os.chdir(), in particularly relating to tests",
https://github.com/simonw/datasette/issues/1408#issuecomment-890390495,https://api.github.com/repos/simonw/datasette/issues/1408,890390495,IC_kwDOBm6k_c41Ekff,9599,simonw,2021-07-31T18:57:39Z,2021-07-31T18:57:39Z,OWNER,Opening this issue as an optional follow-up to the work I did in #1406.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957302085,"Review places in codebase that use os.chdir(), in particularly relating to tests",
https://github.com/simonw/datasette/issues/1406#issuecomment-890390342,https://api.github.com/repos/simonw/datasette/issues/1406,890390342,IC_kwDOBm6k_c41EkdG,9599,simonw,2021-07-31T18:56:35Z,2021-07-31T18:56:35Z,OWNER,"But... I've lost enough time to this already, and removing `runner.isolated_filesystem()` has the tests passing again. So I'm not going to work on this any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1406#issuecomment-890390198,https://api.github.com/repos/simonw/datasette/issues/1406,890390198,IC_kwDOBm6k_c41Eka2,9599,simonw,2021-07-31T18:55:33Z,2021-07-31T18:55:33Z,OWNER,"To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted.

`runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them.

Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1407#issuecomment-890388656,https://api.github.com/repos/simonw/datasette/issues/1407,890388656,IC_kwDOBm6k_c41EkCw,9599,simonw,2021-07-31T18:42:41Z,2021-07-31T18:42:41Z,OWNER,I'll try `tempfile.gettempdir()` - on macOS it returns something like `'/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T'` which is still long but hopefully not too long.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,OSError: AF_UNIX path too long in ds_unix_domain_socket_server,
https://github.com/simonw/datasette/issues/1407#issuecomment-890388200,https://api.github.com/repos/simonw/datasette/issues/1407,890388200,IC_kwDOBm6k_c41Ej7o,9599,simonw,2021-07-31T18:38:41Z,2021-07-31T18:38:41Z,OWNER,"The `path` variable there looked like this:

`/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-696/popen-gw0/uds0/datasette.sock`

I think what's happening here is that `pytest-xdist` causes `tmp_path_factory.mktemp(""uds"")` to create significantly longer paths, which in this case is breaking some limit.

So for this code to work with `pytest-xdist` I need to make sure the random path to `datasette.sock` is shorter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",957298475,OSError: AF_UNIX path too long in ds_unix_domain_socket_server,
https://github.com/simonw/datasette/issues/1406#issuecomment-890259755,https://api.github.com/repos/simonw/datasette/issues/1406,890259755,IC_kwDOBm6k_c41EEkr,9599,simonw,2021-07-31T00:04:54Z,2021-07-31T00:04:54Z,OWNER,"STILL failing. I'm going to try removing all instances of `isolated_filesystem()` in favour of a different pattern using pytest temporary files, then see if I can get that to work without the serial hack. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1398#issuecomment-889599513,https://api.github.com/repos/simonw/datasette/issues/1398,889599513,IC_kwDOBm6k_c41BjYZ,192984,aitoehigie,2021-07-30T03:21:49Z,2021-07-30T03:21:49Z,NONE,Does the library support this now?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",947044667,Documentation on using Datasette as a library,
https://github.com/simonw/datasette/issues/1406#issuecomment-889555977,https://api.github.com/repos/simonw/datasette/issues/1406,889555977,IC_kwDOBm6k_c41BYwJ,9599,simonw,2021-07-30T01:06:57Z,2021-07-30T01:06:57Z,OWNER,"Looking at the source code in Click for `isolated_filesystem()`: https://github.com/pallets/click/blob/9da166957f5848b641231d485467f6140bca2bc0/src/click/testing.py#L450-L468

```python
    @contextlib.contextmanager
    def isolated_filesystem(
        self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
    ) -> t.Iterator[str]:
        """"""A context manager that creates a temporary directory and
        changes the current working directory to it. This isolates tests
        that affect the contents of the CWD to prevent them from
        interfering with each other.
        :param temp_dir: Create the temporary directory under this
            directory. If given, the created directory is not removed
            when exiting.
        .. versionchanged:: 8.0
            Added the ``temp_dir`` parameter.
        """"""
        cwd = os.getcwd()
        t = tempfile.mkdtemp(dir=temp_dir)
        os.chdir(t)
```
How about if I pass in that optional `temp_dir` as a temp directory created using the `pytest-xdist` aware pytest mechanisms: https://docs.pytest.org/en/6.2.x/tmpdir.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1406#issuecomment-889553052,https://api.github.com/repos/simonw/datasette/issues/1406,889553052,IC_kwDOBm6k_c41BYCc,9599,simonw,2021-07-30T00:58:43Z,2021-07-30T00:58:43Z,OWNER,Tests are still failing in the job that calculates coverage.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1406#issuecomment-889550391,https://api.github.com/repos/simonw/datasette/issues/1406,889550391,IC_kwDOBm6k_c41BXY3,9599,simonw,2021-07-30T00:49:31Z,2021-07-30T00:49:31Z,OWNER,That fixed it. My hunch is that Click's `runner.isolated_filesystem()` mechanism doesn't play well with `pytest-xdist`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1406#issuecomment-889548536,https://api.github.com/repos/simonw/datasette/issues/1406,889548536,IC_kwDOBm6k_c41BW74,9599,simonw,2021-07-30T00:43:47Z,2021-07-30T00:43:47Z,OWNER,"Still couldn't replicate on my laptop. On a hunch, I'm going to add `@pytest.mark.serial` to every test that uses `runner.isolated_filesystem()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1406#issuecomment-889547142,https://api.github.com/repos/simonw/datasette/issues/1406,889547142,IC_kwDOBm6k_c41BWmG,9599,simonw,2021-07-30T00:39:49Z,2021-07-30T00:39:49Z,OWNER,It happens in CI but not on my laptop. I think I need to run the tests on my laptop like this: https://github.com/simonw/datasette/blob/121e10c29c5b412fddf0326939f1fe46c3ad9d4a/.github/workflows/test.yml#L27-L30,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",956303470,Tests failing with FileNotFoundError in runner.isolated_filesystem,
https://github.com/simonw/datasette/issues/1241#issuecomment-889539227,https://api.github.com/repos/simonw/datasette/issues/1241,889539227,IC_kwDOBm6k_c41BUqb,9599,simonw,2021-07-30T00:15:26Z,2021-07-30T00:15:26Z,OWNER,"One possible treatment:

<img width=""442"" alt=""fixtures__sortable__201_rows"" src=""https://user-images.githubusercontent.com/9599/127581105-dca1193c-1bdd-491e-89d1-ac3136c05c3b.png"">

```html
<style>
a.action-button {
    display: inline-block;
    border: 1.5px solid #666;
    border-radius: 0.5em;
    background-color: #ffffff;
    color: #666;
    padding: 0.1em 0.8em;
    text-decoration: none;
    margin-right: 0.3em;
    font-size: 0.85em;
}
</style>
<p>
    {% if query.sql and allow_execute_sql %}
        <a class=""action-button""
            title=""{{ query.sql }}""
            href=""{{ urls.database(database) }}?{{ {'sql': query.sql}|urlencode|safe }}{% if query.params %}&amp;{{ query.params|urlencode|safe }}{% endif %}"">
            View and edit SQL
        </a>
    {% endif %}
    <a href=""#"" class=""action-button"">Copy and share link</a>
</p>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1405#issuecomment-889525741,https://api.github.com/repos/simonw/datasette/issues/1405,889525741,IC_kwDOBm6k_c41BRXt,9599,simonw,2021-07-29T23:33:30Z,2021-07-29T23:33:30Z,OWNER,New documentation section for `datasette.utils` is here: https://docs.datasette.io/en/latest/internals.html#the-datasette-utils-module,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function,
https://github.com/simonw/datasette/issues/1405#issuecomment-888694261,https://api.github.com/repos/simonw/datasette/issues/1405,888694261,IC_kwDOBm6k_c40-GX1,9599,simonw,2021-07-28T23:52:21Z,2021-07-28T23:52:21Z,OWNER,Document that it can raise a `BadMetadataError` exception.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function,
https://github.com/simonw/datasette/issues/1405#issuecomment-888694144,https://api.github.com/repos/simonw/datasette/issues/1405,888694144,IC_kwDOBm6k_c40-GWA,9599,simonw,2021-07-28T23:51:59Z,2021-07-28T23:51:59Z,OWNER,https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/datasette/utils/__init__.py#L918-L926,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",955316250,utils.parse_metadata() should be a documented internal function,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-888075098,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,888075098,IC_kwDODFE5qs407vNa,28565,maxhawkins,2021-07-28T07:18:56Z,2021-07-28T07:18:56Z,NONE,"> I'm not sure why but my most recent import, when displayed in Datasette, looks like this:
> 
> <img alt=""mbox__mbox_emails__753_446_rows"" width=""574"" src=""https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png"">

I did some investigation into this issue and made a fix [here](https://github.com/dogsheep/google-takeout-to-sqlite/pull/8/commits/8ee555c2889a38ff42b95664ee074b4a01a82f06). The problem was that some messages (like gchat logs) don't have a `Message-Id` and we need to use `X-GM-THRID` as the pkey instead.

@simonw While looking into this I found something unexpected about how sqlite_utils handles upserts if the pkey column is `None`. When the pkey is NULL I'd expect the function to either use rowid or throw an exception. Instead, it seems upsert_all creates a row where all columns are NULL instead of using the values provided as parameters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/issues/1404#issuecomment-887095569,https://api.github.com/repos/simonw/datasette/issues/1404,887095569,IC_kwDOBm6k_c404AER,9599,simonw,2021-07-26T23:27:07Z,2021-07-26T23:27:07Z,OWNER,Updated documentation: https://github.com/simonw/datasette/blob/eccfeb0871dd4bc27870faf64f80ac68e5b6bc0d/docs/plugin_hooks.rst#register-routes-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",953352015,`register_routes()` hook should take `datasette` argument,
https://github.com/simonw/datasette/issues/1402#issuecomment-886969541,https://api.github.com/repos/simonw/datasette/issues/1402,886969541,IC_kwDOBm6k_c403hTF,9599,simonw,2021-07-26T19:31:40Z,2021-07-26T19:31:40Z,OWNER,"Datasette could do a pretty good job of this by default, using `twitter:card` and `og:url` tags - like on https://til.simonwillison.net/jq/extracting-objects-recursively

I could also provide a mechanism to customize these - in particular to add images of some sort.

It feels like something that should tie in to the metadata mechanism.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",951185411,feature request: social meta tags,
https://github.com/simonw/datasette/issues/1402#issuecomment-886968648,https://api.github.com/repos/simonw/datasette/issues/1402,886968648,IC_kwDOBm6k_c403hFI,9599,simonw,2021-07-26T19:30:14Z,2021-07-26T19:30:14Z,OWNER,"I really like this idea. I was thinking it might make a good plugin, but there's not a great mechanism for plugins to inject extra `<head>` content at the moment - plus this actually feels like a reasonable feature for Datasette core itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",951185411,feature request: social meta tags,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886241674,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3,886241674,IC_kwDODtX3eM400vmK,9599,simonw,2021-07-25T18:41:17Z,2021-07-25T18:41:17Z,MEMBER,Got a TIL out of this: https://til.simonwillison.net/jq/extracting-objects-recursively,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886237834,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3,886237834,IC_kwDODtX3eM400uqK,9599,simonw,2021-07-25T18:05:32Z,2021-07-25T18:05:32Z,MEMBER,"If you hit the endpoint for a comment that's part of a thread you get that comment and its recursive children: https://hn.algolia.com/api/v1/items/27941552

You can tell that it's not the top-level because the `parent_id` isn't `null`. You can use `story_id` to figure out what the top-level item is.

```json
{
  ""id"": 27941552,
  ""created_at"": ""2021-07-24T15:08:39.000Z"",
  ""created_at_i"": 1627139319,
  ""type"": ""comment"",
  ""author"": ""nine_k"",
  ""title"": null,
  ""url"": null,
  ""text"": ""<p>I wish ..."",
  ""points"": null,
  ""parent_id"": 27941108,
  ""story_id"": 27941108
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/3#issuecomment-886142671,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3,886142671,IC_kwDODtX3eM400XbP,9599,simonw,2021-07-25T03:51:05Z,2021-07-25T03:51:05Z,MEMBER,"Prototype:

    curl 'https://hn.algolia.com/api/v1/items/27941108' \
      | jq '[recurse(.children[]) | del(.children)]' \
      | sqlite-utils insert hn.db items - --pk id
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952189173,Use HN algolia endpoint to retrieve trees,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886140431,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886140431,IC_kwDODtX3eM400W4P,9599,simonw,2021-07-25T03:12:57Z,2021-07-25T03:12:57Z,MEMBER,"I'm going to build a general-purpose `hacker-new-to-sqlite search ...` command, where one of the options is to search within the URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886136224,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886136224,IC_kwDODtX3eM400V2g,9599,simonw,2021-07-25T02:08:29Z,2021-07-25T02:08:29Z,MEMBER,"Prototype:

    curl ""https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000"" | \
      jq .hits | sqlite-utils insert hn.db items - --pk objectID --alter","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135922,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886135922,IC_kwDODtX3eM400Vxy,9599,simonw,2021-07-25T02:06:20Z,2021-07-25T02:06:20Z,MEMBER,"https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url looks like it does what I want.

https://hn.algolia.com/api/v1/search_by_date?query=simonwillison.net&restrictSearchableAttributes=url&hitsPerPage=1000 - returns 1000 at once.

Otherwise you have to paginate using `&page=2` etc - up to `nbPages` pages.

https://www.algolia.com/doc/api-reference/api-parameters/hitsPerPage/ says 1000 is the maximum.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/dogsheep/hacker-news-to-sqlite/issues/2#issuecomment-886135562,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2,886135562,IC_kwDODtX3eM400VsK,9599,simonw,2021-07-25T02:01:11Z,2021-07-25T02:01:11Z,MEMBER,"That page doesn't have an API but does look easy to scrape.

The other option here is the HN Search API powered by Algolia, documented at https://hn.algolia.com/api","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952179830,Command for fetching Hacker News threads from the search API,
https://github.com/simonw/sqlite-utils/issues/251#issuecomment-886122696,https://api.github.com/repos/simonw/sqlite-utils/issues/251,886122696,IC_kwDOCGYnMM400SjI,9599,simonw,2021-07-24T23:21:32Z,2021-07-24T23:21:32Z,OWNER,"> ```
> sqlite-utils convert jsonsplit mydb.db mytable mycolumn
> sqlite-utils convert parsedatetime mydb.db mytable mycolumn
> sqlite-utils convert parsedate mydb.db mytable mycolumn
> sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'
> ```

This is a bit verbose - and having added `--multi` and `--output` the `lambda` command keeps getting more and more flexible compared to the others.

New idea: ditch the sub-sub-commands and move the `jsonsplit` and `parsedate` recipes to be options of `convert` - maybe like this:

    sqlite-utils convert my.db mytable col1 --jsonsplit

or:

    sqlite-utils convert my.db mytable col1 --recipe jsonsplit

or `-r jsonsplit` for short.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",841377702,"""sqlite-utils convert"" command to replace the separate ""sqlite-transform"" tool",
https://github.com/simonw/sqlite-utils/issues/299#issuecomment-886117120,https://api.github.com/repos/simonw/sqlite-utils/issues/299,886117120,IC_kwDOCGYnMM400RMA,9599,simonw,2021-07-24T22:12:01Z,2021-07-24T22:12:01Z,OWNER,Documentation here: https://sqlite-utils.datasette.io/en/latest/cli.html#showing-the-schema,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",952154468,Ability to see just specific table schemas with `sqlite-utils schema`,
https://github.com/dogsheep/github-to-sqlite/pull/65#issuecomment-885964242,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65,885964242,IC_kwDODFdgUs40zr3S,231498,khimaros,2021-07-23T23:45:35Z,2021-07-23T23:45:35Z,NONE,@simonw is this PR of interest to you?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923270900,basic support for events,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885098025,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885098025,IC_kwDODFE5qs40wYYp,306240,UtahDave,2021-07-22T17:47:50Z,2021-07-22T17:47:50Z,NONE,"Hi @maxhawkins , I'm sorry, I haven't had any time to work on this. I'll have some time tomorrow to test your commits. I think they look great.  I'm great with your commits superseding my initial attempt here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885094284,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885094284,IC_kwDODFE5qs40wXeM,28565,maxhawkins,2021-07-22T17:41:32Z,2021-07-22T17:41:32Z,NONE,I added a follow-up commit that deals with emails that don't have a `Date` header: https://github.com/maxhawkins/google-takeout-to-sqlite/commit/4bc70103582c10802c85a523ef1e99a8a2154aa9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-885022230,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,885022230,IC_kwDODFE5qs40wF4W,28565,maxhawkins,2021-07-22T15:51:46Z,2021-07-22T15:51:46Z,NONE,One thing I noticed is this importer doesn't save attachments along with the body of the emails. It would be nice if those got stored as blobs in a separate attachments table so attachments can be included while fetching search results.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-884672647,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,884672647,IC_kwDODFE5qs40uwiH,28565,maxhawkins,2021-07-22T05:56:31Z,2021-07-22T14:03:08Z,NONE,"How does this commit look? https://github.com/maxhawkins/google-takeout-to-sqlite/commit/72802a83fee282eb5d02d388567731ba4301050d

It seems that Takeout's mbox format is pretty simple, so we can get away with just splitting the file on lines begining with `From `. My commit just splits the file every time a line starts with `From ` and uses `email.message_from_bytes` to parse each chunk.

I was able to load a 12GB takeout mbox without the program using more than a couple hundred MB of memory during the import process. It does make us lose the progress bar, but maybe I can add that back in a later commit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/issues/1401#issuecomment-884910320,https://api.github.com/repos/simonw/datasette/issues/1401,884910320,IC_kwDOBm6k_c40vqjw,536941,fgregg,2021-07-22T13:26:01Z,2021-07-22T13:26:01Z,CONTRIBUTOR,"ordered lists didn't work either, btw","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",950664971,unordered list is not rendering bullet points in description_html on database page,
https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-884688833,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32,884688833,IC_kwDOD079W840u0fB,10793464,aaronyih1,2021-07-22T06:40:25Z,2021-07-22T06:40:25Z,NONE,The solution here is to upload an image to the bucket first. It is caused because it does not properly handle the case when there are no images in the bucket.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803333769,KeyError: 'Contents' on running upload,
https://github.com/simonw/datasette/pull/1296#issuecomment-817403642,https://api.github.com/repos/simonw/datasette/issues/1296,817403642,MDEyOklzc3VlQ29tbWVudDgxNzQwMzY0Mg==,22429695,codecov[bot],2021-04-12T00:29:05Z,2021-07-20T08:52:12Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1296](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (527a056) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **decrease** coverage by `0.11%`.
> The diff coverage is `n/a`.

> :exclamation: Current head 527a056 differs from pull request most recent head 8f00c31. Consider uploading reports for the commit 8f00c31 to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1296/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main    #1296      +/-   ##
==========================================
- Coverage   91.62%   91.51%   -0.12%     
==========================================
  Files          34       34              
  Lines        4371     4255     -116     
==========================================
- Hits         4005     3894     -111     
+ Misses        366      361       -5     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `81.60% <0.00%> (-1.35%)` | :arrow_down: |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <0.00%> (-0.42%)` | :arrow_down: |
| [datasette/facets.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZhY2V0cy5weQ==) | `89.04% <0.00%> (-0.41%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.13% <0.00%> (-0.21%)` | :arrow_down: |
| [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.02% <0.00%> (-0.18%)` | :arrow_down: |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.19% <0.00%> (-0.10%)` | :arrow_down: |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.88% <0.00%> (-0.07%)` | :arrow_down: |
| [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.36% <0.00%> (-0.07%)` | :arrow_down: |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <0.00%> (ø)` | |
| [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.38% <0.00%> (ø)` | |
| ... and [5 more](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c73af5d...8f00c31](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855446829,Dockerfile: use Ubuntu 20.10 as base,
https://github.com/simonw/datasette/pull/1400#issuecomment-882542519,https://api.github.com/repos/simonw/datasette/issues/1400,882542519,IC_kwDOBm6k_c40moe3,22429695,codecov[bot],2021-07-19T13:20:52Z,2021-07-19T13:20:52Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1400](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e95c685) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1400/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1400   +/-   ##
=======================================
  Coverage   91.62%   91.62%           
=======================================
  Files          34       34           
  Lines        4371     4371           
=======================================
  Hits         4005     4005           
  Misses        366      366           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c73af5d...e95c685](https://codecov.io/gh/simonw/datasette/pull/1400?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",947640902,Bump black from 21.6b0 to 21.7b0,
https://github.com/simonw/datasette/issues/123#issuecomment-882138084,https://api.github.com/repos/simonw/datasette/issues/123,882138084,IC_kwDOBm6k_c40lFvk,9599,simonw,2021-07-19T00:04:31Z,2021-07-19T00:04:31Z,OWNER,"I've been thinking more about this one today too. An extension of this (touched on in #417, Datasette Library) would be to support pointing Datasette at a directory and having it automatically load any CSV files it finds anywhere in that folder or its descendants - either loading them fully, or providing a UI that allows users to select a file to open it in Datasette.

For larger files I think the right thing to do is import them into an on-disk SQLite database, which is limited only by available disk space. For smaller files loading them into an in-memory database should work fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/123#issuecomment-882096402,https://api.github.com/repos/simonw/datasette/issues/123,882096402,IC_kwDOBm6k_c40k7kS,921217,RayBB,2021-07-18T18:07:29Z,2021-07-18T18:07:29Z,NONE,"I also love the idea for this feature and wonder if it could work without having to download the whole database into memory at once if it's a rather large db. Obviously this could be slower but could have many use cases.

My comment is partially inspired by this post about streaming sqlite dbs from github pages or such
https://news.ycombinator.com/item?id=27016630
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-882091516,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32,882091516,IC_kwDOD079W840k6X8,10793464,aaronyih1,2021-07-18T17:29:39Z,2021-07-18T17:33:02Z,NONE,Same here for US West (N. California) us-west-1. Running on Catalina.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803333769,KeyError: 'Contents' on running upload,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-882052852,https://api.github.com/repos/simonw/sqlite-utils/issues/297,882052852,IC_kwDOCGYnMM40kw70,9599,simonw,2021-07-18T12:59:20Z,2021-07-18T12:59:20Z,OWNER,I'm not too worried about `sqlite-utils memory` because if your data is large enough that you can benefit from this optimization you probably should use a real file as opposed to a disposable memory database when analyzing it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/datasette/issues/1199#issuecomment-881932880,https://api.github.com/repos/simonw/datasette/issues/1199,881932880,IC_kwDOBm6k_c40kTpQ,9599,simonw,2021-07-17T17:39:17Z,2021-07-17T17:39:17Z,OWNER,"I asked about optimizing performance on the SQLite forum and this came up as a suggestion: https://sqlite.org/forum/forumpost/9a6b9ae8e2048c8b?t=c

I can start by trying this:

    PRAGMA mmap_size=268435456;","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792652391,Experiment with PRAGMA mmap_size=N,
https://github.com/simonw/datasette/issues/1396#issuecomment-881686662,https://api.github.com/repos/simonw/datasette/issues/1396,881686662,IC_kwDOBm6k_c40jXiG,9599,simonw,2021-07-16T20:02:44Z,2021-07-16T20:02:44Z,OWNER,Confirmed fixed: 0.58.1 was successfully published to Docker Hub in https://github.com/simonw/datasette/runs/3089447346 and the `latest` tag on https://hub.docker.com/r/datasetteproject/datasette/tags was updated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1231#issuecomment-881677620,https://api.github.com/repos/simonw/datasette/issues/1231,881677620,IC_kwDOBm6k_c40jVU0,9599,simonw,2021-07-16T19:44:12Z,2021-07-16T19:44:12Z,OWNER,"That fixed the race condition in the `datasette-graphql` tests, which is the only place that I've been able to successfully replicate this. I'm going to land this change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881674857,https://api.github.com/repos/simonw/datasette/issues/1231,881674857,IC_kwDOBm6k_c40jUpp,9599,simonw,2021-07-16T19:38:39Z,2021-07-16T19:38:39Z,OWNER,I can't replicate the race condition locally with or without this patch. I'm going to push the commit and then test the CI run from `datasette-graphql` that was failing against it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881671706,https://api.github.com/repos/simonw/datasette/issues/1231,881671706,IC_kwDOBm6k_c40jT4a,9599,simonw,2021-07-16T19:32:05Z,2021-07-16T19:32:05Z,OWNER,The test suite passes with that change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881668759,https://api.github.com/repos/simonw/datasette/issues/1231,881668759,IC_kwDOBm6k_c40jTKX,9599,simonw,2021-07-16T19:27:46Z,2021-07-16T19:27:46Z,OWNER,"Second attempt at this:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 5976d8b..5f348cb 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -224,6 +224,7 @@ class Datasette:
         self.inspect_data = inspect_data
         self.immutables = set(immutables or [])
         self.databases = collections.OrderedDict()
+        self._refresh_schemas_lock = asyncio.Lock()
         self.crossdb = crossdb
         if memory or crossdb or not self.files:
             self.add_database(Database(self, is_memory=True), name=""_memory"")
@@ -332,6 +333,12 @@ class Datasette:
         self.client = DatasetteClient(self)
 
     async def refresh_schemas(self):
+        if self._refresh_schemas_lock.locked():
+            return
+        async with self._refresh_schemas_lock:
+            await self._refresh_schemas()
+
+    async def _refresh_schemas(self):
         internal_db = self.databases[""_internal""]
         if not self.internal_db_created:
             await init_internal_db(internal_db)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881665383,https://api.github.com/repos/simonw/datasette/issues/1231,881665383,IC_kwDOBm6k_c40jSVn,9599,simonw,2021-07-16T19:21:35Z,2021-07-16T19:21:35Z,OWNER,"https://stackoverflow.com/a/25799871/6083 has a good example of using `asyncio.Lock()`:

```python
stuff_lock = asyncio.Lock()

async def get_stuff(url):
    async with stuff_lock:
        if url in cache:
            return cache[url]
        stuff = await aiohttp.request('GET', url)
        cache[url] = stuff
        return stuff
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881664408,https://api.github.com/repos/simonw/datasette/issues/1231,881664408,IC_kwDOBm6k_c40jSGY,9599,simonw,2021-07-16T19:19:35Z,2021-07-16T19:19:35Z,OWNER,"The only place that calls `refresh_schemas()` is here: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/views/base.py#L120-L124

Ideally only one call to `refresh_schemas()` would be running at any one time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881663968,https://api.github.com/repos/simonw/datasette/issues/1231,881663968,IC_kwDOBm6k_c40jR_g,9599,simonw,2021-07-16T19:18:42Z,2021-07-16T19:18:42Z,OWNER,The race condition happens inside this method - initially with the call to `await init_internal_db()`: https://github.com/simonw/datasette/blob/dd5ee8e66882c94343cd3f71920878c6cfd0da41/datasette/app.py#L334-L359,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881204782,https://api.github.com/repos/simonw/datasette/issues/1231,881204782,IC_kwDOBm6k_c40hh4u,9599,simonw,2021-07-16T06:14:12Z,2021-07-16T06:14:12Z,OWNER,"Here's the traceback I got from `datasette-graphql` (annoyingly only running the tests in GitHub Actions CI - I've not been able to replicate on my laptop yet):

```
tests/test_utils.py .                                                    [100%]

=================================== FAILURES ===================================
_________________________ test_graphql_examples[path0] _________________________

ds = <datasette.app.Datasette object at 0x7f6b8b6f8fd0>
path = PosixPath('/home/runner/work/datasette-graphql/datasette-graphql/examples/filters.md')

    @pytest.mark.asyncio
    @pytest.mark.parametrize(
        ""path"", (pathlib.Path(__file__).parent.parent / ""examples"").glob(""*.md"")
    )
    async def test_graphql_examples(ds, path):
        content = path.read_text()
        query = graphql_re.search(content)[1]
        try:
            variables = variables_re.search(content)[1]
        except TypeError:
            variables = ""{}""
        expected = json.loads(json_re.search(content)[1])
        response = await ds.client.post(
            ""/graphql"",
            json={
                ""query"": query,
                ""variables"": json.loads(variables),
            },
        )
>       assert response.status_code == 200, response.json()
E       AssertionError: {'data': {'repos_arraycontains': None, 'users_contains': None, 'users_date': None, 'users_endswith': None, ...}, 'erro..."", 'path': ['users_gt']}, {'locations': [{'column': 5, 'line': 34}], 'message': ""'rows'"", 'path': ['users_gte']}, ...]}
E       assert 500 == 200
E        +  where 500 = <Response [500 Internal Server Error]>.status_code

tests/test_graphql.py:142: AssertionError
----------------------------- Captured stderr call -----------------------------
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
table databases already exists
Traceback (most recent call last):
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 1171, in route_path
    response = await view(request, send)
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 151, in view
    request, **request.scope[""url_route""][""kwargs""]
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/views/base.py"", line 123, in dispatch_request
    await self.ds.refresh_schemas()
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/app.py"", line 338, in refresh_schemas
    await init_internal_db(internal_db)
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/utils/internal_db.py"", line 16, in init_internal_db
    block=True,
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 102, in execute_write
    return await self.execute_write_fn(_inner, block=block)
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 118, in execute_write_fn
    raise result
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 139, in _execute_writes
    result = task.fn(conn)
  File ""/opt/hostedtoolcache/Python/3.7.11/x64/lib/python3.7/site-packages/datasette/database.py"", line 100, in _inner
    return conn.execute(sql, params or [])
sqlite3.OperationalError: table databases already exists
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-881204343,https://api.github.com/repos/simonw/datasette/issues/1231,881204343,IC_kwDOBm6k_c40hhx3,9599,simonw,2021-07-16T06:13:11Z,2021-07-16T06:13:11Z,OWNER,This just broke the `datasette-graphql` test suite: https://github.com/simonw/datasette-graphql/issues/77 - I need to figure out a solution here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1394#issuecomment-881129149,https://api.github.com/repos/simonw/datasette/issues/1394,881129149,IC_kwDOBm6k_c40hPa9,9599,simonw,2021-07-16T02:23:32Z,2021-07-16T02:23:32Z,OWNER,Wrote about this in the annotated release notes for 0.58: https://simonwillison.net/2021/Jul/16/datasette-058/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/datasette/issues/759#issuecomment-881125124,https://api.github.com/repos/simonw/datasette/issues/759,881125124,IC_kwDOBm6k_c40hOcE,9599,simonw,2021-07-16T02:11:48Z,2021-07-16T02:11:54Z,OWNER,"I added `""searchmode"": ""raw""` as a supported option for table metadata in #1389 and released that in Datasette 0.58.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612673948,fts search on a column doesn't work anymore due to escape_fts,
https://github.com/simonw/datasette/issues/1396#issuecomment-880967052,https://api.github.com/repos/simonw/datasette/issues/1396,880967052,MDEyOklzc3VlQ29tbWVudDg4MDk2NzA1Mg==,9599,simonw,2021-07-15T19:47:25Z,2021-07-15T19:47:25Z,OWNER,Actually I'm going to close this now and re-open it if the problem occurs again in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1394#issuecomment-880900534,https://api.github.com/repos/simonw/datasette/issues/1394,880900534,MDEyOklzc3VlQ29tbWVudDg4MDkwMDUzNA==,9599,simonw,2021-07-15T17:58:03Z,2021-07-15T17:58:03Z,OWNER,Started a conversation about this on the SQLite forum: https://sqlite.org/forum/forumpost/2d76f2bcf65d256a?t=h,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/datasette/issues/1396#issuecomment-880374156,https://api.github.com/repos/simonw/datasette/issues/1396,880374156,MDEyOklzc3VlQ29tbWVudDg4MDM3NDE1Ng==,9599,simonw,2021-07-15T04:03:18Z,2021-07-15T04:03:18Z,OWNER,"I fixed  `datasette:latest` by running the following on my laptop:
```
docker pull datasetteproject/datasette:0.58
docker tag datasetteproject/datasette:0.58 datasetteproject/datasette:latest
docker login -u datasetteproject -p ...
docker push datasetteproject/datasette:latest
```
Confirmed on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated that `datasette:latest` and `datasette:0.58` both now have the same digest of `3b5ba478040e`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880372149,https://api.github.com/repos/simonw/datasette/issues/1396,880372149,MDEyOklzc3VlQ29tbWVudDg4MDM3MjE0OQ==,9599,simonw,2021-07-15T03:56:49Z,2021-07-15T03:56:49Z,OWNER,I'm going to leave this open until I next successfully publish a new version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880326049,https://api.github.com/repos/simonw/datasette/issues/1396,880326049,MDEyOklzc3VlQ29tbWVudDg4MDMyNjA0OQ==,9599,simonw,2021-07-15T01:50:05Z,2021-07-15T01:50:05Z,OWNER,"I think I made a mistake in this commit: https://github.com/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659

<img width=""770"" alt=""Explicitly_push_version_tag__refs__1281_·_simonw_datasette_0486303"" src=""https://user-images.githubusercontent.com/9599/125715241-dc96d147-191a-4a69-bdf0-1cb092d97330.png"">

It looks like I copied `$VERSION_TAG` from here - but it's not available in the `publish.yml` flow: https://github.com/simonw/datasette/blob/0486303b60ce2784fd2e2ecdbecf304b7d6e6659/.github/workflows/push_docker_tag.yml#L18-L25","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880325362,https://api.github.com/repos/simonw/datasette/issues/1396,880325362,MDEyOklzc3VlQ29tbWVudDg4MDMyNTM2Mg==,9599,simonw,2021-07-15T01:48:11Z,2021-07-15T01:48:11Z,OWNER,In particular these three lines: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L117-L119,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880325004,https://api.github.com/repos/simonw/datasette/issues/1396,880325004,MDEyOklzc3VlQ29tbWVudDg4MDMyNTAwNA==,9599,simonw,2021-07-15T01:47:17Z,2021-07-15T01:47:17Z,OWNER,"This is the part of the publish workflow that failed and threw the ""invalid reference format"" error: https://github.com/simonw/datasette/blob/084cfe1e00e1a4c0515390a513aca286eeea20c2/.github/workflows/publish.yml#L100-L119","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1396#issuecomment-880324637,https://api.github.com/repos/simonw/datasette/issues/1396,880324637,MDEyOklzc3VlQ29tbWVudDg4MDMyNDYzNw==,9599,simonw,2021-07-15T01:46:26Z,2021-07-15T01:46:26Z,OWNER,"I manually published the Docker image using https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml https://github.com/simonw/datasette/runs/3072505126

The 0.58 release shows up on https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now - BUT the `latest` tag still points to a version from a month ago.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944903881,"""invalid reference format"" publishing Docker image",
https://github.com/simonw/datasette/issues/1394#issuecomment-880287483,https://api.github.com/repos/simonw/datasette/issues/1394,880287483,MDEyOklzc3VlQ29tbWVudDg4MDI4NzQ4Mw==,9599,simonw,2021-07-15T00:01:47Z,2021-07-15T00:01:47Z,OWNER,"I wrote this code:
```python
_order_by_re = re.compile(r""(^.*) order by [a-zA-Z_][a-zA-Z0-9_]+( desc)?$"", re.DOTALL)
_order_by_braces_re = re.compile(r""(^.*) order by \[[^\]]+\]( desc)?$"", re.DOTALL)


def strip_order_by(sql):
    for regex in (_order_by_re, _order_by_braces_re):
        match = regex.match(sql)
        if match is not None:
            return match.group(1)
    return sql

@pytest.mark.parametrize(
    ""sql,expected"",
    [
        (""blah"", ""blah""),
        (""select * from foo"", ""select * from foo""),
        (""select * from foo order by bah"", ""select * from foo""),
        (""select * from foo order by bah desc"", ""select * from foo""),
        (""select * from foo order by [select]"", ""select * from foo""),
        (""select * from foo order by [select] desc"", ""select * from foo""),
    ],
)
def test_strip_order_by(sql, expected):
    assert strip_order_by(sql) == expected
```
But it turns out I don't need it! The SQL that is passed to the facet class is created by this code: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L677-L684

And the only place that uses that `sql_no_limit` variable is here: https://github.com/simonw/datasette/blob/ba11ef27edd6981eeb26d7ecf5aa236707f5f8ce/datasette/views/table.py#L733-L745

So I can change that to `sql_no_limit_no_order` and fix the bug that way instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/datasette/issues/1394#issuecomment-880278256,https://api.github.com/repos/simonw/datasette/issues/1394,880278256,MDEyOklzc3VlQ29tbWVudDg4MDI3ODI1Ng==,9599,simonw,2021-07-14T23:35:18Z,2021-07-14T23:35:18Z,OWNER,"The challenge here is that faceting doesn't currently modify the inner SQL at all - it wraps it so that it can work against any SQL statement (though Datasette itself does not yet take advantage of that ability, only offering faceting on table pages).

So just removing the order by wouldn't be appropriate if the inner query looked something like this:

```sql
select * from items order by created desc limit 100
```
Since the intent there would be to return facet counts against only the most recent 100 items.

In SQLite the `limit` has to come after the `order by` though, so the fix here could be as easy as using a regular expression to identify queries that end with `order by COLUMN (desc)?` and stripping off that clause.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944870799,Big performance boost on faceting: skip the inner order by,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880259255,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880259255,MDEyOklzc3VlQ29tbWVudDg4MDI1OTI1NQ==,9599,simonw,2021-07-14T22:48:41Z,2021-07-14T22:48:41Z,OWNER,Should also take advantage of `.mode tabs` to support `sqlite-utils insert blah.db blah blah.csv --tsv --fast`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880257587,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880257587,MDEyOklzc3VlQ29tbWVudDg4MDI1NzU4Nw==,9599,simonw,2021-07-14T22:44:05Z,2021-07-14T22:44:05Z,OWNER,"https://unix.stackexchange.com/a/642364 suggests you can also use this to import from stdin, like so:

    sqlite3 -csv $database_file_name "".import '|cat -' $table_name""

Here the `sqlite3 -csv` is an alternative to using `.mode csv`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256865,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256865,MDEyOklzc3VlQ29tbWVudDg4MDI1Njg2NQ==,9599,simonw,2021-07-14T22:42:11Z,2021-07-14T22:42:11Z,OWNER,"Potential workaround for missing `--skip` implementation is that the filename can be a command instead, so maybe it could shell out to `tail -n +1 filename`:

> The source argument is the name of a file to be read or, if it begins with a ""|"" character, specifies a command which will be run to produce the input CSV data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-880256058,https://api.github.com/repos/simonw/sqlite-utils/issues/297,880256058,MDEyOklzc3VlQ29tbWVudDg4MDI1NjA1OA==,9599,simonw,2021-07-14T22:40:01Z,2021-07-14T22:40:47Z,OWNER,"Full docs here: https://www.sqlite.org/draft/cli.html#csv

One catch: how this works has changed in recent SQLite versions: https://www.sqlite.org/changes.html

- 2020-12-01 (3.34.0) - ""Table name quoting works correctly for the .import dot-command""
- 2020-05-22 (3.32.0) - ""Add options to the .import command: --csv, --ascii, --skip""
- 2017-08-01 (3.20.0) - "" The "".import"" command ignores an initial UTF-8 BOM.""

The ""skip"" feature is particularly important to understand. https://www.sqlite.org/draft/cli.html#csv says:

> There are two cases to consider: (1) Table ""tab1"" does not previously exist and (2) table ""tab1"" does already exist.
>
> In the first case, when the table does not previously exist, the table is automatically created and the content of the first row of the input CSV file is used to determine the name of all the columns in the table. In other words, if the table does not previously exist, the first row of the CSV file is interpreted to be column names and the actual data starts on the second row of the CSV file.
>
> For the second case, when the table already exists, every row of the CSV file, including the first row, is assumed to be actual content. If the CSV file contains an initial row of column labels, you can cause the .import command to skip that initial row using the ""--skip 1"" option.

But the `--skip 1` option is only available in 3.32.0 and higher.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/datasette/issues/268#issuecomment-880153069,https://api.github.com/repos/simonw/datasette/issues/268,880153069,MDEyOklzc3VlQ29tbWVudDg4MDE1MzA2OQ==,9599,simonw,2021-07-14T19:31:00Z,2021-07-14T19:31:00Z,OWNER,"... though interestingly I can't replicate that error on `latest.datasette.io` - https://latest.datasette.io/fixtures/searchable?_search=park.&_searchmode=raw

That's running https://latest.datasette.io/-/versions SQLite 3.35.4 whereas https://www.niche-museums.com/-/versions is running 3.27.2 (the most recent version available with Vercel) - but there's nothing in the SQLite changelog between those two versions that suggests changes to how the FTS5 parser works. https://www.sqlite.org/changes.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/268#issuecomment-880150755,https://api.github.com/repos/simonw/datasette/issues/268,880150755,MDEyOklzc3VlQ29tbWVudDg4MDE1MDc1NQ==,9599,simonw,2021-07-14T19:26:47Z,2021-07-14T19:29:08Z,OWNER,"> What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... ""to ensure they do not cause any confusion for users who are not aware of them"", but I'm not sure what those could be.

Mainly that it's possible to generate SQL queries that crash with an error. This was the example that convinced me to default to escaping:

- https://www.niche-museums.com/browse/museums?_search=park.&_searchmode=raw (returns `fts5: syntax error near "".""`)

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/651#issuecomment-579675357,https://api.github.com/repos/simonw/datasette/issues/651,579675357,MDEyOklzc3VlQ29tbWVudDU3OTY3NTM1Nw==,2181410,clausjuhl,2020-01-29T09:45:00Z,2021-07-14T19:26:06Z,NONE,"Hi Simon

Thank you for adding the escape_function, but it does not work on my datasette-installation (0.33). I've added the following file to my datasette-dir: `/plugins/sql_functions.py`:

```python
from datasette import hookimpl

def escape_fts_query(query):
    bits = query.split()
    return ' '.join('""{}""'.format(bit.replace('""', '')) for bit in bits)

@hookimpl
def prepare_connection(conn):
    conn.create_function(""escape_fts_query"", 1, escape_fts_query)`
```
It has no effect on the standard queries to the tables though, as they still produce errors when including any characters like '-', '/', '+' or '?'

Does the function only work when using costum queries, where I can include the escape_fts-function explicitly in the sql-query?

PS. I'm calling datasette with --plugins=plugins, and my other plugins work just fine.
PPS. The fts5 virtual table is created with 'sqlite3' like so:

`CREATE VIRTUAL TABLE ""cases_fts"" USING FTS5(
  title,
  subtitle,
  resume,
  suggestion,
  presentation,
  detail = full,
  content_rowid = 'id',
  content = 'cases',
  tokenize='unicode61', 'remove_diacritics 2', 'tokenchars ""-_""'
);`

Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",539590148,fts5 syntax error when using punctuation,
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-879477586,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,879477586,MDEyOklzc3VlQ29tbWVudDg3OTQ3NzU4Ng==,9599,simonw,2021-07-13T23:50:06Z,2021-07-13T23:50:06Z,MEMBER,"Unfortunately I don't think updating the database is practical, because the export doesn't include unique identifiers which can be used to update existing records and create new ones. Recreating from scratch works around that limitation.

I've not explored workouts with SpatiaLite but that's a really good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/simonw/datasette/pull/1393#issuecomment-879309636,https://api.github.com/repos/simonw/datasette/issues/1393,879309636,MDEyOklzc3VlQ29tbWVudDg3OTMwOTYzNg==,9599,simonw,2021-07-13T18:32:25Z,2021-07-13T18:32:25Z,OWNER,Thanks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941412189,Update deploying.rst,
https://github.com/simonw/datasette/pull/1392#issuecomment-879277953,https://api.github.com/repos/simonw/datasette/issues/1392,879277953,MDEyOklzc3VlQ29tbWVudDg3OTI3Nzk1Mw==,9599,simonw,2021-07-13T17:42:31Z,2021-07-13T17:42:31Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941403676,Update deploying.rst,
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877874117,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,877874117,MDEyOklzc3VlQ29tbWVudDg3Nzg3NDExNw==,956433,Mjboothaus,2021-07-11T23:03:37Z,2021-07-11T23:03:37Z,NONE,P.s. wondering if you have explored using the spatialite functionality with the location data in workouts?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/simonw/datasette/issues/511#issuecomment-877835171,https://api.github.com/repos/simonw/datasette/issues/511,877835171,MDEyOklzc3VlQ29tbWVudDg3NzgzNTE3MQ==,9599,simonw,2021-07-11T17:23:05Z,2021-07-11T17:23:05Z,OWNER,"    == 87 failed, 819 passed, 7 skipped, 29 errors in 2584.85s (0:43:04) ==

https://github.com/simonw/datasette/runs/3038188870?check_suite_focus=true

Full copy of log here: https://gist.github.com/simonw/4b1fdd24496b989fca56bc757be345ad","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/dogsheep/healthkit-to-sqlite/issues/12#issuecomment-877805513,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12,877805513,MDEyOklzc3VlQ29tbWVudDg3NzgwNTUxMw==,956433,Mjboothaus,2021-07-11T14:03:01Z,2021-07-11T14:03:01Z,NONE,"Hi Simon -- just experimenting with your excellent software!

Up to this point in time I have been using the (paid) [HealthFit App](https://apps.apple.com/au/app/healthfit/id1202650514) to export my workouts from my Apple Watch, one walk at the time into either .GPX or .FIT format and then using another library to suck it into Python and eventually here to my ""Emmaus Walking"" app:

https://share.streamlit.io/mjboothaus/emmaus_walking/emmaus_walking/app.py

I just used `healthkit-to-sqlite` to convert my export.zip file and it all ""just worked"".

I did notice the issue with various numeric fields being stored in the SQLite db as TEXT for now and just thought I'd flag it - but you're already self-reported this issue.

Keep up the great work!

I was curious if you have any thoughts about periodically exporting ""export.zip"" and how to just update the SQLite file instead of re-creating it each time. Hopefully Apple will give some thought to managing this data in a more sensible fashion as it grows over time.  Ideally one could pull it from iCloud (where it is allegedly being backed up).


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727848625,"Some workout columns should be float, not text",
https://github.com/simonw/datasette/issues/511#issuecomment-877726495,https://api.github.com/repos/simonw/datasette/issues/511,877726495,MDEyOklzc3VlQ29tbWVudDg3NzcyNjQ5NQ==,9599,simonw,2021-07-11T01:32:27Z,2021-07-11T01:32:27Z,OWNER,"I'm using `pytest-xdist` and this:

    pytest -n auto -m ""not serial""

I'll try not using the `-n auto` bit on Windows and see if that helps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877726288,https://api.github.com/repos/simonw/datasette/issues/511,877726288,MDEyOklzc3VlQ29tbWVudDg3NzcyNjI4OA==,9599,simonw,2021-07-11T01:29:41Z,2021-07-11T01:29:41Z,OWNER,"Lots of errors that look like this:
```
2021-07-11T00:40:32.1189321Z E           NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpdr41pgwg\\data.db'
2021-07-11T00:40:32.1190083Z 
2021-07-11T00:40:32.1191128Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:596: NotADirectoryError
2021-07-11T00:40:32.1191999Z ___________________ ERROR at teardown of test_insert_error ____________________
2021-07-11T00:40:32.1192842Z [gw1] win32 -- Python 3.8.10 c:\hostedtoolcache\windows\python\3.8.10\x64\python.exe
2021-07-11T00:40:32.1193387Z 
2021-07-11T00:40:32.1193930Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_'
2021-07-11T00:40:32.1194876Z onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000291FCEA93A0>
2021-07-11T00:40:32.1195480Z 
2021-07-11T00:40:32.1195927Z     def _rmtree_unsafe(path, onerror):
2021-07-11T00:40:32.1196435Z         try:
2021-07-11T00:40:32.1196910Z             with os.scandir(path) as scandir_it:
2021-07-11T00:40:32.1197504Z                 entries = list(scandir_it)
2021-07-11T00:40:32.1198002Z         except OSError:
2021-07-11T00:40:32.1198607Z             onerror(os.scandir, path, sys.exc_info())
2021-07-11T00:40:32.1199137Z             entries = []
2021-07-11T00:40:32.1199637Z         for entry in entries:
2021-07-11T00:40:32.1200184Z             fullname = entry.path
2021-07-11T00:40:32.1200692Z             if _rmtree_isdir(entry):
2021-07-11T00:40:32.1201198Z                 try:
2021-07-11T00:40:32.1201643Z                     if entry.is_symlink():
2021-07-11T00:40:32.1202280Z                         # This can only happen if someone replaces
2021-07-11T00:40:32.1202944Z                         # a directory with a symlink after the call to
2021-07-11T00:40:32.1203623Z                         # os.scandir or entry.is_dir above.
2021-07-11T00:40:32.1204303Z                         raise OSError(""Cannot call rmtree on a symbolic link"")
2021-07-11T00:40:32.1204942Z                 except OSError:
2021-07-11T00:40:32.1206416Z                     onerror(os.path.islink, fullname, sys.exc_info())
2021-07-11T00:40:32.1207022Z                     continue
2021-07-11T00:40:32.1207584Z                 _rmtree_unsafe(fullname, onerror)
2021-07-11T00:40:32.1208074Z             else:
2021-07-11T00:40:32.1208496Z                 try:
2021-07-11T00:40:32.1208926Z >                   os.unlink(fullname)
2021-07-11T00:40:32.1210053Z E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1210974Z 
2021-07-11T00:40:32.1211638Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:616: PermissionError
2021-07-11T00:40:32.1212211Z 
2021-07-11T00:40:32.1212846Z During handling of the above exception, another exception occurred:
2021-07-11T00:40:32.1213320Z 
2021-07-11T00:40:32.1213797Z func = <built-in function unlink>
2021-07-11T00:40:32.1214529Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1215763Z exc_info = (<class 'PermissionError'>, PermissionError(13, 'The process cannot access the file because it is being used by another process'), <traceback object at 0x00000291FB4D7040>)
2021-07-11T00:40:32.1217263Z 
2021-07-11T00:40:32.1217777Z     def onerror(func, path, exc_info):
2021-07-11T00:40:32.1218421Z         if issubclass(exc_info[0], PermissionError):
2021-07-11T00:40:32.1219079Z             def resetperms(path):
2021-07-11T00:40:32.1219518Z                 try:
2021-07-11T00:40:32.1219992Z                     _os.chflags(path, 0)
2021-07-11T00:40:32.1220535Z                 except AttributeError:
2021-07-11T00:40:32.1221110Z                     pass
2021-07-11T00:40:32.1221545Z                 _os.chmod(path, 0o700)
2021-07-11T00:40:32.1221984Z     
2021-07-11T00:40:32.1222330Z             try:
2021-07-11T00:40:32.1222768Z                 if path != name:
2021-07-11T00:40:32.1223332Z                     resetperms(_os.path.dirname(path))
2021-07-11T00:40:32.1223963Z                 resetperms(path)
2021-07-11T00:40:32.1224408Z     
2021-07-11T00:40:32.1224749Z                 try:
2021-07-11T00:40:32.1225954Z >                   _os.unlink(path)
2021-07-11T00:40:32.1227032Z E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1227927Z 
2021-07-11T00:40:32.1228646Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:802: PermissionError
2021-07-11T00:40:32.1229200Z 
2021-07-11T00:40:32.1229842Z During handling of the above exception, another exception occurred:
2021-07-11T00:40:32.1230355Z 
2021-07-11T00:40:32.1230783Z     @pytest.fixture
2021-07-11T00:40:32.1231322Z     def canned_write_client():
2021-07-11T00:40:32.1231805Z         with make_app_client(
2021-07-11T00:40:32.1232467Z             extra_databases={""data.db"": ""create table names (name text)""},
2021-07-11T00:40:32.1233104Z             metadata={
2021-07-11T00:40:32.1233535Z                 ""databases"": {
2021-07-11T00:40:32.1233989Z                     ""data"": {
2021-07-11T00:40:32.1234416Z                         ""queries"": {
2021-07-11T00:40:32.1235001Z                             ""canned_read"": {""sql"": ""select * from names""},
2021-07-11T00:40:32.1235527Z                             ""add_name"": {
2021-07-11T00:40:32.1236117Z                                 ""sql"": ""insert into names (name) values (:name)"",
2021-07-11T00:40:32.1236686Z                                 ""write"": True,
2021-07-11T00:40:32.1237317Z                                 ""on_success_redirect"": ""/data/add_name?success"",
2021-07-11T00:40:32.1237882Z                             },
2021-07-11T00:40:32.1238331Z                             ""add_name_specify_id"": {
2021-07-11T00:40:32.1239009Z                                 ""sql"": ""insert into names (rowid, name) values (:rowid, :name)"",
2021-07-11T00:40:32.1239610Z                                 ""write"": True,
2021-07-11T00:40:32.1240259Z                                 ""on_error_redirect"": ""/data/add_name_specify_id?error"",
2021-07-11T00:40:32.1240839Z                             },
2021-07-11T00:40:32.1241320Z                             ""delete_name"": {
2021-07-11T00:40:32.1242504Z                                 ""sql"": ""delete from names where rowid = :rowid"",
2021-07-11T00:40:32.1243127Z                                 ""write"": True,
2021-07-11T00:40:32.1243721Z                                 ""on_success_message"": ""Name deleted"",
2021-07-11T00:40:32.1244282Z                                 ""allow"": {""id"": ""root""},
2021-07-11T00:40:32.1244749Z                             },
2021-07-11T00:40:32.1245959Z                             ""update_name"": {
2021-07-11T00:40:32.1246614Z                                 ""sql"": ""update names set name = :name where rowid = :rowid"",
2021-07-11T00:40:32.1247267Z                                 ""params"": [""rowid"", ""name"", ""extra""],
2021-07-11T00:40:32.1247828Z                                 ""write"": True,
2021-07-11T00:40:32.1248247Z                             },
2021-07-11T00:40:32.1248653Z                         }
2021-07-11T00:40:32.1249166Z                     }
2021-07-11T00:40:32.1249577Z                 }
2021-07-11T00:40:32.1249962Z             },
2021-07-11T00:40:32.1250333Z         ) as client:
2021-07-11T00:40:32.1250822Z >           yield client
2021-07-11T00:40:32.1251078Z 
2021-07-11T00:40:32.1251678Z D:\a\datasette\datasette\tests\test_canned_queries.py:43: 
2021-07-11T00:40:32.1252347Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-07-11T00:40:32.1253040Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\contextlib.py:120: in __exit__
2021-07-11T00:40:32.1253759Z     next(self.gen)
2021-07-11T00:40:32.1254398Z D:\a\datasette\datasette\tests\fixtures.py:156: in make_app_client
2021-07-11T00:40:32.1255098Z     yield TestClient(ds)
2021-07-11T00:40:32.1255796Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:827: in __exit__
2021-07-11T00:40:32.1256510Z     self.cleanup()
2021-07-11T00:40:32.1257200Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:831: in cleanup
2021-07-11T00:40:32.1257961Z     self._rmtree(self.name)
2021-07-11T00:40:32.1258712Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
2021-07-11T00:40:32.1259487Z     _shutil.rmtree(name, onerror=onerror)
2021-07-11T00:40:32.1260280Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
2021-07-11T00:40:32.1261039Z     return _rmtree_unsafe(path, onerror)
2021-07-11T00:40:32.1261843Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:618: in _rmtree_unsafe
2021-07-11T00:40:32.1262633Z     onerror(os.unlink, fullname, sys.exc_info())
2021-07-11T00:40:32.1263456Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:805: in onerror
2021-07-11T00:40:32.1264175Z     cls._rmtree(path)
2021-07-11T00:40:32.1264848Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\tempfile.py:813: in _rmtree
2021-07-11T00:40:32.1266329Z     _shutil.rmtree(name, onerror=onerror)
2021-07-11T00:40:32.1267082Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:740: in rmtree
2021-07-11T00:40:32.1267858Z     return _rmtree_unsafe(path, onerror)
2021-07-11T00:40:32.1268615Z c:\hostedtoolcache\windows\python\3.8.10\x64\lib\shutil.py:599: in _rmtree_unsafe
2021-07-11T00:40:32.1269440Z     onerror(os.scandir, path, sys.exc_info())
2021-07-11T00:40:32.1269979Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2021-07-11T00:40:32.1270287Z 
2021-07-11T00:40:32.1270947Z path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1273356Z onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000291FCF40E50>
2021-07-11T00:40:32.1273999Z 
2021-07-11T00:40:32.1274493Z     def _rmtree_unsafe(path, onerror):
2021-07-11T00:40:32.1274953Z         try:
2021-07-11T00:40:32.1275461Z >           with os.scandir(path) as scandir_it:
2021-07-11T00:40:32.1276459Z E           NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpry729pq_\\data.db'
2021-07-11T00:40:32.1277220Z 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877725742,https://api.github.com/repos/simonw/datasette/issues/511,877725742,MDEyOklzc3VlQ29tbWVudDg3NzcyNTc0Mg==,9599,simonw,2021-07-11T01:25:01Z,2021-07-11T01:26:38Z,OWNER,"That's weird. https://github.com/simonw/datasette/runs/3037862798 finished running and came up green - but actually a TON of the tests failed on Windows. Not sure why that didn't fail the whole test suite:

<img width=""1203"" alt=""Ran_Black__refs__1388_·_simonw_datasette_518cb06"" src=""https://user-images.githubusercontent.com/9599/125180192-12257000-e1ac-11eb-8657-d46b7bcdc1b2.png"">

Also the test suite took 50 minutes on Windows!

Here's a copy of the full log file for the tests on Python 3.8 on Windows: https://gist.github.com/simonw/2900ef33693c1bbda09188eb31c8212d","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/1388#issuecomment-877725193,https://api.github.com/repos/simonw/datasette/issues/1388,877725193,MDEyOklzc3VlQ29tbWVudDg3NzcyNTE5Mw==,9599,simonw,2021-07-11T01:18:38Z,2021-07-11T01:18:38Z,OWNER,Wrote up a TIL: https://til.simonwillison.net/nginx/proxy-domain-sockets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877721003,https://api.github.com/repos/simonw/datasette/issues/1388,877721003,MDEyOklzc3VlQ29tbWVudDg3NzcyMTAwMw==,9599,simonw,2021-07-11T00:21:19Z,2021-07-11T00:21:19Z,OWNER,Documentation: https://docs.datasette.io/en/latest/deploying.html#nginx-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/511#issuecomment-877718364,https://api.github.com/repos/simonw/datasette/issues/511,877718364,MDEyOklzc3VlQ29tbWVudDg3NzcxODM2NA==,9599,simonw,2021-07-10T23:54:37Z,2021-07-10T23:54:37Z,OWNER,"Looks like it's not even 10% of the way through, and already a bunch of errors:

<img width=""777"" alt=""Ran_Black__refs__1388_·_simonw_datasette_518cb06"" src=""https://user-images.githubusercontent.com/9599/125179059-81e12e00-e19f-11eb-94d9-0f2d9ce8afad.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877718286,https://api.github.com/repos/simonw/datasette/issues/511,877718286,MDEyOklzc3VlQ29tbWVudDg3NzcxODI4Ng==,9599,simonw,2021-07-10T23:53:29Z,2021-07-10T23:53:29Z,OWNER,"Test suite on Windows seems to run a lot slower:

<img width=""391"" alt=""Ran_Black__refs__1388_·_simonw_datasette_518cb06"" src=""https://user-images.githubusercontent.com/9599/125179033-4e9e9f00-e19f-11eb-8c2e-8df09fafa795.png"">

From https://github.com/simonw/datasette/actions/runs/1018938850 which is still going.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/issues/511#issuecomment-877717791,https://api.github.com/repos/simonw/datasette/issues/511,877717791,MDEyOklzc3VlQ29tbWVudDg3NzcxNzc5MQ==,9599,simonw,2021-07-10T23:45:35Z,2021-07-10T23:45:35Z,OWNER,"> Trying to run on Windows today, I get an error from the utils/asgi.py module.
> 
> It's trying `from os import EX_CANTCREAT` which is Unix-only. I commented this line out, and (so far) it's working.

Good news: that line was removed in #1094.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/simonw/datasette/pull/868#issuecomment-650340914,https://api.github.com/repos/simonw/datasette/issues/868,650340914,MDEyOklzc3VlQ29tbWVudDY1MDM0MDkxNA==,22429695,codecov[bot],2020-06-26T18:53:02Z,2021-07-10T23:41:42Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#868](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (b452fcb) into [master](https://codecov.io/gh/simonw/datasette/commit/000528192eaf891118932250141dabe7a1561ece?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0005281) will **increase** coverage by `0.49%`.
> The diff coverage is `96.19%`.

> :exclamation: Current head b452fcb differs from pull request most recent head c99caba. Consider uploading reports for the commit c99caba to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/868/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##           master     #868      +/-   ##
==========================================
+ Coverage   82.91%   83.40%   +0.49%     
==========================================
  Files          26       27       +1     
  Lines        3547     3634      +87     
==========================================
+ Hits         2941     3031      +90     
+ Misses        606      603       -3     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
| [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <91.17%> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.99% <97.91%> (+1.32%)` | :arrow_up: |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.93% <100.00%> (+0.08%)` | :arrow_up: |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.32% <100.00%> (+0.41%)` | :arrow_up: |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.39% <100.00%> (-0.01%)` | :arrow_down: |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.37% <100.00%> (-1.96%)` | :arrow_down: |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.67% <0.00%> (-0.03%)` | :arrow_down: |
| ... and [6 more](https://codecov.io/gh/simonw/datasette/pull/868/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [180c7a5...c99caba](https://codecov.io/gh/simonw/datasette/pull/868?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",646448486,initial windows ci setup,
https://github.com/simonw/datasette/pull/557#issuecomment-877717392,https://api.github.com/repos/simonw/datasette/issues/557,877717392,MDEyOklzc3VlQ29tbWVudDg3NzcxNzM5Mg==,9599,simonw,2021-07-10T23:39:48Z,2021-07-10T23:39:48Z,OWNER,Abandoning this - need to switch to using GitHub Actions for this instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",466996584,Get tests running on Windows using Travis CI,
https://github.com/simonw/datasette/issues/1388#issuecomment-877717262,https://api.github.com/repos/simonw/datasette/issues/1388,877717262,MDEyOklzc3VlQ29tbWVudDg3NzcxNzI2Mg==,9599,simonw,2021-07-10T23:37:54Z,2021-07-10T23:37:54Z,OWNER,"> I wonder if `--fd` is worth supporting too?

I'm going to hold off on implementing this until someone asks for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716993,https://api.github.com/repos/simonw/datasette/issues/1388,877716993,MDEyOklzc3VlQ29tbWVudDg3NzcxNjk5Mw==,9599,simonw,2021-07-10T23:34:02Z,2021-07-10T23:34:02Z,OWNER,"Figured out an example nginx configuration. This in `nginx.conf`:

    daemon off;
    events {
      worker_connections  1024;
    }
    http {
      server {
        listen 8092;
        location / {
          proxy_pass              http://datasette;
          proxy_set_header        X-Real-IP $remote_addr;
          proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        }
      }
      upstream datasette {
        server unix:/tmp/datasette.sock;
      }
    }

Then run `datasette --uds /tmp/datasette.sock`

Then run nginx like this:

    nginx -c ./nginx.conf

Then hits to `http://localhost:8092/` will be proxied to Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716359,https://api.github.com/repos/simonw/datasette/issues/1388,877716359,MDEyOklzc3VlQ29tbWVudDg3NzcxNjM1OQ==,9599,simonw,2021-07-10T23:24:58Z,2021-07-10T23:24:58Z,OWNER,"Apparently Windows 10 has Unix domain socket support: https://bugs.python.org/issue33408

> Unix socket (AF_UNIX) is now avalible in Windows 10 (April 2018 Update). Please add Python support for it.
> More details about it on https://blogs.msdn.microsoft.com/commandline/2017/12/19/af_unix-comes-to-windows/

But it's not clear if this is going to work. That same issue thread (the issue is still open) suggests using `hasattr(socket, 'AF_UNIX'))` to detect support in tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877716156,https://api.github.com/repos/simonw/datasette/issues/1388,877716156,MDEyOklzc3VlQ29tbWVudDg3NzcxNjE1Ng==,9599,simonw,2021-07-10T23:22:21Z,2021-07-10T23:22:21Z,OWNER,"I don't have the Datasette test suite running on Windows yet, but I'd like it to run there some day - so ideally this test would be skipped if Unix domain sockets are not supported by the underlying operating system.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877715654,https://api.github.com/repos/simonw/datasette/issues/1388,877715654,MDEyOklzc3VlQ29tbWVudDg3NzcxNTY1NA==,9599,simonw,2021-07-10T23:15:06Z,2021-07-10T23:15:06Z,OWNER,"I can run tests against it using `httpx`: https://www.python-httpx.org/advanced/#usage_1

> ```pycon
> >>> import httpx
> >>> # Connect to the Docker API via a Unix Socket.
> >>> transport = httpx.HTTPTransport(uds=""/var/run/docker.sock"")
> >>> client = httpx.Client(transport=transport)
> >>> response = client.get(""http://docker/info"")
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-877714698,https://api.github.com/repos/simonw/datasette/issues/1388,877714698,MDEyOklzc3VlQ29tbWVudDg3NzcxNDY5OA==,9599,simonw,2021-07-10T23:01:37Z,2021-07-10T23:01:37Z,OWNER,"Can test this with:
```
curl --unix-socket ${socket} -i ""http://localhost/"" 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1391#issuecomment-877691558,https://api.github.com/repos/simonw/datasette/issues/1391,877691558,MDEyOklzc3VlQ29tbWVudDg3NzY5MTU1OA==,9599,simonw,2021-07-10T19:26:57Z,2021-07-10T19:26:57Z,OWNER,"The `https://latest.datasette.io/fixtures.db` file no longer includes generated columns, which will help avoid confusion such as seen in #1376.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877691427,https://api.github.com/repos/simonw/datasette/issues/1391,877691427,MDEyOklzc3VlQ29tbWVudDg3NzY5MTQyNw==,9599,simonw,2021-07-10T19:26:00Z,2021-07-10T19:26:00Z,OWNER,I had to run the tests locally on my macOS laptop using `pysqlite3` to get a version that supported generated columns - wrote up a TIL about that here: https://til.simonwillison.net/sqlite/pysqlite3-on-macos,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877687196,https://api.github.com/repos/simonw/datasette/issues/1391,877687196,MDEyOklzc3VlQ29tbWVudDg3NzY4NzE5Ng==,9599,simonw,2021-07-10T18:58:40Z,2021-07-10T18:58:40Z,OWNER,I can use the `extra_databases` mechanism as demonstrated here: https://github.com/simonw/datasette/blob/9552414e1f968c6fc704031cec349c05e6bc2371/tests/test_canned_queries.py#L8-L12,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877686784,https://api.github.com/repos/simonw/datasette/issues/1391,877686784,MDEyOklzc3VlQ29tbWVudDg3NzY4Njc4NA==,9599,simonw,2021-07-10T18:56:03Z,2021-07-10T18:56:03Z,OWNER,Here's the SQL used to generate the table for the test: https://github.com/simonw/datasette/blob/02b19c7a9afd328f22040ab33b5c1911cd904c7c/tests/fixtures.py#L723-L733,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1391#issuecomment-877682533,https://api.github.com/repos/simonw/datasette/issues/1391,877682533,MDEyOklzc3VlQ29tbWVudDg3NzY4MjUzMw==,9599,simonw,2021-07-10T18:28:05Z,2021-07-10T18:28:05Z,OWNER,"Here's the test in question: https://github.com/simonw/datasette/blob/a6c55afe8c82ead8deb32f90c9324022fd422324/tests/test_api.py#L2033-L2046

Various other places in the test code also need changing - anything that calls `supports_generated_columns()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",941300946,Stop using generated columns in fixtures.db,
https://github.com/simonw/datasette/issues/1389#issuecomment-877681031,https://api.github.com/repos/simonw/datasette/issues/1389,877681031,MDEyOklzc3VlQ29tbWVudDg3NzY4MTAzMQ==,9599,simonw,2021-07-10T18:17:29Z,2021-07-10T18:17:29Z,OWNER,"I don't like `?_searchmode=default` because it suggests ""use the default"" - but it actually over-rides the default that was specified by `""searchmode"": ""raw""` in `metadata.json`.

I'm going with `?_searchmode=escaped` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1390#issuecomment-877310125,https://api.github.com/repos/simonw/datasette/issues/1390,877310125,MDEyOklzc3VlQ29tbWVudDg3NzMxMDEyNQ==,9599,simonw,2021-07-09T16:32:57Z,2021-07-09T16:32:57Z,OWNER,https://docs.datasette.io/en/latest/deploying.html#running-datasette-using-systemd,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation,
https://github.com/simonw/datasette/issues/1390#issuecomment-877308310,https://api.github.com/repos/simonw/datasette/issues/1390,877308310,MDEyOklzc3VlQ29tbWVudDg3NzMwODMxMA==,9599,simonw,2021-07-09T16:29:48Z,2021-07-09T16:29:48Z,OWNER,    sudo systemctl restart datasette.service,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940891698,Mention restarting systemd in documentation,
https://github.com/simonw/datasette/issues/268#issuecomment-876721585,https://api.github.com/repos/simonw/datasette/issues/268,876721585,MDEyOklzc3VlQ29tbWVudDg3NjcyMTU4NQ==,9308268,rayvoelker,2021-07-08T20:22:17Z,2021-07-08T20:22:17Z,NONE,"I do like the idea of there being a option for turning that on by default so that you could use those terms in the default ""Search"" bar presented when you browse to a table where FTS has been enabled. Maybe even a small inline pop up with a short bit explaining the FTS feature and the keywords (e.g. case matters). What are the side-effects of turning that on in the query string, or even by default as you suggested? I see that you stated in the docs... ""to ensure they do not cause any confusion for users who are not aware of them"", but I'm not sure what those could be.

Isn't it the case that those keywords are only picked up by sqlite in where you're using the MATCH clause?

Seems like a really powerful feature (even though there are a lot of hurdles around setting it up in the sqlite db ... sqlite-utils makes that so simple by the way!)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/1389#issuecomment-876620095,https://api.github.com/repos/simonw/datasette/issues/1389,876620095,MDEyOklzc3VlQ29tbWVudDg3NjYyMDA5NQ==,9599,simonw,2021-07-08T17:35:09Z,2021-07-08T17:35:09Z,OWNER,Also came up here: https://github.com/simonw/datasette/issues/268#issuecomment-876616414,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876619531,https://api.github.com/repos/simonw/datasette/issues/1389,876619531,MDEyOklzc3VlQ29tbWVudDg3NjYxOTUzMQ==,9599,simonw,2021-07-08T17:34:16Z,2021-07-08T17:34:16Z,OWNER,If I implement this I'll also set it up so `?_searchmode=default` can be used to over-ride that and go back to the default behaviour.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876619271,https://api.github.com/repos/simonw/datasette/issues/1389,876619271,MDEyOklzc3VlQ29tbWVudDg3NjYxOTI3MQ==,9599,simonw,2021-07-08T17:33:49Z,2021-07-08T17:33:49Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L491-L498,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876618847,https://api.github.com/repos/simonw/datasette/issues/1389,876618847,MDEyOklzc3VlQ29tbWVudDg3NjYxODg0Nw==,9599,simonw,2021-07-08T17:33:08Z,2021-07-08T17:33:08Z,OWNER,Relevant documentation: https://docs.datasette.io/en/0.57.1/full_text_search.html#advanced-sqlite-search-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/1389#issuecomment-876618582,https://api.github.com/repos/simonw/datasette/issues/1389,876618582,MDEyOklzc3VlQ29tbWVudDg3NjYxODU4Mg==,9599,simonw,2021-07-08T17:32:40Z,2021-07-08T17:32:40Z,OWNER,This makes sense to me since other useful querystring arguments like this can be set as defaults in the `metadata.json` configuration.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",940077168,"""searchmode"": ""raw"" in table metadata",
https://github.com/simonw/datasette/issues/268#issuecomment-876616414,https://api.github.com/repos/simonw/datasette/issues/268,876616414,MDEyOklzc3VlQ29tbWVudDg3NjYxNjQxNA==,9599,simonw,2021-07-08T17:29:04Z,2021-07-08T17:29:04Z,OWNER,"> I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function

That's a deliberate feature (albeit controversial, see #759) - part of the main problem here is that it's easy to construct a SQLite full-text search string which results in a database error. This is a bad user-experience!

You can opt-in to raw SQL queries by appending `?_searchmode=raw` to the page, see https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries

But maybe there should be an option for turning that on by default without needing the query string?

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/268#issuecomment-876428348,https://api.github.com/repos/simonw/datasette/issues/268,876428348,MDEyOklzc3VlQ29tbWVudDg3NjQyODM0OA==,9308268,rayvoelker,2021-07-08T13:13:12Z,2021-07-08T13:13:12Z,NONE,"I had setup a full text search on my instance of Datasette for title data for our public library, and was noticing that some of the features of the SQLite FTS weren't working as expected ... and maybe the issue is in the `escape_fts()` function

![image](https://user-images.githubusercontent.com/9308268/124925900-f1ea8b00-dfca-11eb-895e-59cc083d6524.png)
vs removing the function...
![image](https://user-images.githubusercontent.com/9308268/124925971-0464c480-dfcb-11eb-8fbf-8e9b5d6e0861.png)

Also, on the issue of sorting by rank by default .. perhaps something like this could work for the baked-in default SQL query for Datasette?
![image](https://user-images.githubusercontent.com/9308268/124927191-5a863780-dfcc-11eb-9908-3f63577d5ff5.png)

[link to the above search in my instance of Datasette](https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-87a9011?sql=with+fts_search+as+%28%0D%0A++select%0D%0A++rowid%2C%0D%0A++rank%0D%0A++++from%0D%0A++++++bib_fts%0D%0A++++where%0D%0A++++++bib_fts+match+%3Asearch%0D%0A%29%0D%0A%0D%0Aselect%0D%0A++%0D%0A++bib_record_num%2C%0D%0A++creation_date%2C%0D%0A++record_last_updated%2C%0D%0A++isbn%2C%0D%0A++best_author%2C%0D%0A++best_title%2C%0D%0A++publisher%2C%0D%0A++publish_year%2C%0D%0A++bib_level_callnumber%2C%0D%0A++indexed_subjects%0D%0Afrom%0D%0A++fts_search%0D%0A++join+bib+on+bib.rowid+%3D+fts_search.rowid%0D%0A++%0D%0Aorder+by%0D%0Arank%0D%0A&search=black+death+NOT+fiction)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/1388#issuecomment-876213177,https://api.github.com/repos/simonw/datasette/issues/1388,876213177,MDEyOklzc3VlQ29tbWVudDg3NjIxMzE3Nw==,80737,aslakr,2021-07-08T07:47:17Z,2021-07-08T07:47:17Z,CONTRIBUTOR,"> This sounds like a valuable feature for people running Datasette behind a proxy.

Yes, in some cases it is easer to use e.g. Apache's [ProxyPass Directive](https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass) with Unix Domain Socket like  `unix:/home/www.socket|http://localhost/whatever/`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875742910,https://api.github.com/repos/simonw/datasette/issues/1388,875742910,MDEyOklzc3VlQ29tbWVudDg3NTc0MjkxMA==,9599,simonw,2021-07-07T16:20:50Z,2021-07-07T16:23:02Z,OWNER,"I wonder if `--fd` is worth supporting too? Uvicorn documentation says that's useful for running under process managers, I'd want to understand exactly how to use that (and test it) before adding the feature though.

https://www.uvicorn.org/settings/

Docs on how to use a process manager: https://www.uvicorn.org/deployment/#supervisor","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875741410,https://api.github.com/repos/simonw/datasette/issues/1388,875741410,MDEyOklzc3VlQ29tbWVudDg3NTc0MTQxMA==,9599,simonw,2021-07-07T16:18:50Z,2021-07-07T16:18:50Z,OWNER,"You could actually run Datasette like this today without modifications by running a thin Python script that imports from `datasette.app`, instantiates the ASGI app and passes that to `uvicorn.run` - but I like this as a supported feature too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875740085,https://api.github.com/repos/simonw/datasette/issues/1388,875740085,MDEyOklzc3VlQ29tbWVudDg3NTc0MDA4NQ==,9599,simonw,2021-07-07T16:17:08Z,2021-07-07T16:17:08Z,OWNER,"Looks pretty easy to implement - here's a hint from Uvicorn source code: https://github.com/encode/uvicorn/blob/b5af1049e63c059dc750a450c807b9768f91e906/uvicorn/main.py#L368

Need to work out a simple pattern for testing this too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1388#issuecomment-875738149,https://api.github.com/repos/simonw/datasette/issues/1388,875738149,MDEyOklzc3VlQ29tbWVudDg3NTczODE0OQ==,9599,simonw,2021-07-07T16:14:29Z,2021-07-07T16:14:29Z,OWNER,This sounds like a valuable feature for people running Datasette behind a proxy.,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",939051549,Serve using UNIX domain socket,
https://github.com/simonw/datasette/issues/1387#issuecomment-873166836,https://api.github.com/repos/simonw/datasette/issues/1387,873166836,MDEyOklzc3VlQ29tbWVudDg3MzE2NjgzNg==,9308268,rayvoelker,2021-07-02T17:58:23Z,2021-07-02T17:58:23Z,NONE,"Thanks Simon for nailing that one down! It does seem a little confusing that the ProxyPreservehost option is set to Off By default, but this config totally did the trick and fixed the issue

```
<Location /collection-analysis/>
   ProxyPass http://127.0.0.1:8010/collection-analysis/
   ProxyPreservehost On
</Location>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873156408,https://api.github.com/repos/simonw/datasette/issues/1387,873156408,MDEyOklzc3VlQ29tbWVudDg3MzE1NjQwOA==,9599,simonw,2021-07-02T17:37:30Z,2021-07-02T17:37:30Z,OWNER,Updated documentation is here: https://docs.datasette.io/en/latest/deploying.html#apache-proxy-configuration,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873141222,https://api.github.com/repos/simonw/datasette/issues/1387,873141222,MDEyOklzc3VlQ29tbWVudDg3MzE0MTIyMg==,9599,simonw,2021-07-02T17:09:32Z,2021-07-02T17:09:32Z,OWNER,I'm going to add this to the suggested Apache configuration at https://docs.datasette.io/en/stable/deploying.html#apache-proxy-configuration,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873140742,https://api.github.com/repos/simonw/datasette/issues/1387,873140742,MDEyOklzc3VlQ29tbWVudDg3MzE0MDc0Mg==,9599,simonw,2021-07-02T17:08:40Z,2021-07-02T17:08:40Z,OWNER,`ProxyPreserveHost On` is the Apache setting - it defaults to `Off`: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypreservehost ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873139138,https://api.github.com/repos/simonw/datasette/issues/1387,873139138,MDEyOklzc3VlQ29tbWVudDg3MzEzOTEzOA==,9599,simonw,2021-07-02T17:05:47Z,2021-07-02T17:05:47Z,OWNER,"In this case the proxy is Apache. So there are a couple of potential fixes:

- Configure Apache to pass the original HTTP request `Host:` header through to the proxied application. This should then be documented.
- Add a new optional feature to Datasette called something like `base_host` which, if set, is always used in place of the host in `request.url` when constructing new URLs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873137935,https://api.github.com/repos/simonw/datasette/issues/1387,873137935,MDEyOklzc3VlQ29tbWVudDg3MzEzNzkzNQ==,9599,simonw,2021-07-02T17:03:36Z,2021-07-02T17:03:36Z,OWNER,"And the links to apply a facet value are broken too! https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber

```json
        {
          ""value"": ""g l fiction"",
          ""label"": ""g l fiction"",
          ""count"": 212,
          ""toggle_url"": ""https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_facet=bib_level_callnumber&bib_level_callnumber=g+l+fiction"",
          ""selected"": false
        }
```
Same problem: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/facets.py#L251-L261","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873136440,https://api.github.com/repos/simonw/datasette/issues/1387,873136440,MDEyOklzc3VlQ29tbWVudDg3MzEzNjQ0MA==,9599,simonw,2021-07-02T17:01:48Z,2021-07-02T17:01:48Z,OWNER,"Here's what's happening: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/views/table.py#L827-L829

This is being run through `absolute_url()` - defined here: https://github.com/simonw/datasette/blob/d23a2671386187f61872b9f6b58e0f80ac61f8fe/datasette/app.py#L633-L637

That's because the `next_url` in the JSON needs to be a full URL that a client can retrieve - as  opposed to the other links on that page which are all relative links that start with `/`: https://github.com/simonw/datasette/blob/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f/datasette/templates/_table.html#L11-L15

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1387#issuecomment-873134866,https://api.github.com/repos/simonw/datasette/issues/1387,873134866,MDEyOklzc3VlQ29tbWVudDg3MzEzNDg2Ng==,9599,simonw,2021-07-02T16:58:52Z,2021-07-02T16:58:52Z,OWNER,What's weird here is that the URL itself is correct - it starts with `/collection-analysis/` as expected - but the hostname is wrong.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",935930820,absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs,
https://github.com/simonw/datasette/issues/1101#issuecomment-869812567,https://api.github.com/repos/simonw/datasette/issues/1101,869812567,MDEyOklzc3VlQ29tbWVudDg2OTgxMjU2Nw==,9599,simonw,2021-06-28T16:06:57Z,2021-06-28T16:07:24Z,OWNER,"Relevant blog post: https://simonwillison.net/2021/Jun/25/streaming-large-api-responses/ - including notes on efficiently streaming formats with some kind of separator in between the records (regular JSON).

> Some export formats are friendlier for streaming than others. CSV and TSV are pretty easy to stream, as is newline-delimited JSON.
> 
> Regular JSON requires a bit more thought: you can output a `[` character, then output each row in a stream with a comma suffix, then skip the comma for the last row and output a `]`. Doing that requires peeking ahead (looping two at a time) to verify that you haven't yet reached the end.
> 
> Or... Martin De Wulf [pointed out](https://twitter.com/madewulf/status/1405559088994467844) that you can output the first row, then output every other row with a preceeding comma---which avoids the whole ""iterate two at a time"" problem entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/pull/1386#issuecomment-869677851,https://api.github.com/repos/simonw/datasette/issues/1386,869677851,MDEyOklzc3VlQ29tbWVudDg2OTY3Nzg1MQ==,22429695,codecov[bot],2021-06-28T13:18:50Z,2021-06-28T13:18:50Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1386](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e974ed1) into [main](https://codecov.io/gh/simonw/datasette/commit/ea627baccf980d7d8ebc9e1ffff1fe34d556e56f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ea627ba) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1386/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1386   +/-   ##
=======================================
  Coverage   91.70%   91.70%           
=======================================
  Files          34       34           
  Lines        4364     4364           
=======================================
  Hits         4002     4002           
  Misses        362      362           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [ea627ba...e974ed1](https://codecov.io/gh/simonw/datasette/pull/1386?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",931557895,"Update asgiref requirement from <3.4.0,>=3.2.10 to >=3.2.10,<3.5.0",
https://github.com/simonw/datasette/issues/1101#issuecomment-869191854,https://api.github.com/repos/simonw/datasette/issues/1101,869191854,MDEyOklzc3VlQ29tbWVudDg2OTE5MTg1NA==,25778,eyeseast,2021-06-27T16:42:14Z,2021-06-27T16:42:14Z,CONTRIBUTOR,This would really help with this issue: https://github.com/eyeseast/datasette-geojson/issues/7,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1168#issuecomment-869076254,https://api.github.com/repos/simonw/datasette/issues/1168,869076254,MDEyOklzc3VlQ29tbWVudDg2OTA3NjI1NA==,2670795,brandonrobertz,2021-06-27T00:03:16Z,2021-06-27T00:05:51Z,CONTRIBUTOR,"> Related: Here's an implementation of a `get_metadata()` plugin hook by @brandonrobertz [next-LI@3fd8ce9](https://github.com/next-LI/datasette/commit/3fd8ce91f3108c82227bf65ff033929426c60437)

Here's a plugin that implements metadata-within-DBs: [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config)

How it works: If a database has a `__metadata` table, then it gets parsed and included in the global metadata. It also implements a database-action hook with a UI for managing config.

More context: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L109-L140","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1384#issuecomment-869075395,https://api.github.com/repos/simonw/datasette/issues/1384,869075395,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM5NQ==,9599,simonw,2021-06-26T23:54:21Z,2021-06-26T23:59:21Z,OWNER,(It may well be that implementing #1168 involves a switch to async metadata),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869075368,https://api.github.com/repos/simonw/datasette/issues/1384,869075368,MDEyOklzc3VlQ29tbWVudDg2OTA3NTM2OA==,9599,simonw,2021-06-26T23:53:55Z,2021-06-26T23:53:55Z,OWNER,"Great, let's drop fallback then.

My instinct at the moment is to ship this plugin hook as-is but with a warning that it may change before Datasette 1.0 - then before 1.0 either figure out an async variant or finish the database-backed metadata concept from #1168 and recommend that as an alternative.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869074701,https://api.github.com/repos/simonw/datasette/issues/1384,869074701,MDEyOklzc3VlQ29tbWVudDg2OTA3NDcwMQ==,2670795,brandonrobertz,2021-06-26T23:45:18Z,2021-06-26T23:45:37Z,CONTRIBUTOR,"> Here's where the plugin hook is called, demonstrating the `fallback=` argument:
> 
> https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472
> 
> I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?

I think you're right. I can't think of a reason why the plugin would care about the `fallback` parameter since plugins are currently mandated to return a full, global metadata dict.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869074182,https://api.github.com/repos/simonw/datasette/issues/1384,869074182,MDEyOklzc3VlQ29tbWVudDg2OTA3NDE4Mg==,2670795,brandonrobertz,2021-06-26T23:37:42Z,2021-06-26T23:37:42Z,CONTRIBUTOR,"> > Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
> > @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?
> 
> Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160
> 
> That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.
> 
> This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.
> 
> But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.

_Ideally_ this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)

In my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as `Datasette.metadata()` is a critical core function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071790,https://api.github.com/repos/simonw/datasette/issues/1384,869071790,MDEyOklzc3VlQ29tbWVudDg2OTA3MTc5MA==,9599,simonw,2021-06-26T23:04:12Z,2021-06-26T23:04:12Z,OWNER,"> Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
> 
> @brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his `datasette-live-config` plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in `sqlite-utils`) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses `await db.execute(...)` - but here's an example where users are encouraged to use blocking calls instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071435,https://api.github.com/repos/simonw/datasette/issues/1384,869071435,MDEyOklzc3VlQ29tbWVudDg2OTA3MTQzNQ==,9599,simonw,2021-06-26T22:59:26Z,2021-06-26T22:59:26Z,OWNER,"The other alternative is to finish the work to build a `_metadata` internal table, see #1168. The idea there was that if we want to support efficient pagination and search across the metadata for thousands of attached tables powering it with a plugin hook doesn't work well - we don't want to call the hook once for every one of 1,000+ tables just to implement the homepage.

So instead, all metadata for all attached databases would be loaded into an in-memory database called `_metadata`. Plugins that want to modify stored metadata could then do so by directly writing to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-869071236,https://api.github.com/repos/simonw/datasette/issues/860,869071236,MDEyOklzc3VlQ29tbWVudDg2OTA3MTIzNg==,9599,simonw,2021-06-26T22:56:28Z,2021-06-26T22:56:28Z,OWNER,This work is continuing in #1384.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869071167,https://api.github.com/repos/simonw/datasette/issues/1384,869071167,MDEyOklzc3VlQ29tbWVudDg2OTA3MTE2Nw==,9599,simonw,2021-06-26T22:55:36Z,2021-06-26T22:55:36Z,OWNER,"Just realized I already have an issue open for this, at #860. I'm going to close that and continue work on this in this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070941,https://api.github.com/repos/simonw/datasette/issues/1384,869070941,MDEyOklzc3VlQ29tbWVudDg2OTA3MDk0MQ==,9599,simonw,2021-06-26T22:53:34Z,2021-06-26T22:53:34Z,OWNER,"The `await` thing is worrying me a lot - it feels like this plugin hook is massively less useful if it can't make it's own DB queries and generally do asynchronous stuff - but I'd also like not to break every existing plugin that calls `datasette.metadata(...)`.

One solution that could work: introduce a new method, maybe `await datasette.get_metadata(...)`, which uses this plugin hook - and keep the existing `datasette.metadata()` method (which doesn't call the hook) around. This would ensure existing plugins keep on working.

Then, upgrade those plugins separately - with the goal of deprecating and removing `.metadata()` entirely in Datasette 1.0 - having upgraded the plugins in the meantime.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070348,https://api.github.com/repos/simonw/datasette/issues/1384,869070348,MDEyOklzc3VlQ29tbWVudDg2OTA3MDM0OA==,9599,simonw,2021-06-26T22:46:18Z,2021-06-26T22:46:18Z,OWNER,"Here's where the plugin hook is called, demonstrating the `fallback=` argument: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

I'm not convinced of the use-case for passing `fallback=` to the hook here - is there a reason a plugin might care whether fallback is `True` or `False`, seeing as the `metadata()` method already respects that fallback logic on line 459?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869070076,https://api.github.com/repos/simonw/datasette/issues/1384,869070076,MDEyOklzc3VlQ29tbWVudDg2OTA3MDA3Ng==,9599,simonw,2021-06-26T22:42:21Z,2021-06-26T22:42:21Z,OWNER,"Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.

@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069926,https://api.github.com/repos/simonw/datasette/issues/1384,869069926,MDEyOklzc3VlQ29tbWVudDg2OTA2OTkyNg==,9599,simonw,2021-06-26T22:40:15Z,2021-06-26T22:40:53Z,OWNER,"The documentation says:

> **datasette**:     You can use this to access plugin configuration options via `datasette.plugin_config(your_plugin_name)`, or to execute SQL queries.

That's not accurate: since the plugin hook is a regular function, not an awaitable, you can't use it to run `await db.execute(...)` so you can't execute SQL queries.

I can fix this with the await-me-maybe pattern, used for other plugin hooks: https://simonwillison.net/2020/Sep/2/await-me-maybe/

BUT... that requires changing the `ds.metadata()` function to be awaitable, which will affect every existing plugn that uses that documented internal method!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069768,https://api.github.com/repos/simonw/datasette/issues/1384,869069768,MDEyOklzc3VlQ29tbWVudDg2OTA2OTc2OA==,9599,simonw,2021-06-26T22:37:53Z,2021-06-26T22:37:53Z,OWNER,The documentation doesn't describe the ``fallback`` argument at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/issues/1384#issuecomment-869069655,https://api.github.com/repos/simonw/datasette/issues/1384,869069655,MDEyOklzc3VlQ29tbWVudDg2OTA2OTY1NQ==,9599,simonw,2021-06-26T22:36:14Z,2021-06-26T22:37:37Z,OWNER,"Documentation for the new hook is now live at https://docs.datasette.io/en/latest/plugin_hooks.html#get-metadata-datasette-key-database-table-fallback

Link to the current snapshot of that documentation: https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/docs/plugin_hooks.rst#get-metadata-datasette-key-database-table-fallback","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",930807135,Plugin hook for dynamic metadata,
https://github.com/simonw/datasette/pull/1368#issuecomment-869068554,https://api.github.com/repos/simonw/datasette/issues/1368,869068554,MDEyOklzc3VlQ29tbWVudDg2OTA2ODU1NA==,9599,simonw,2021-06-26T22:23:57Z,2021-06-26T22:23:57Z,OWNER,The only test failure is Black. I'm going to merge this and then reformat.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881190,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881190,MDEyOklzc3VlQ29tbWVudDg2ODg4MTE5MA==,9599,simonw,2021-06-25T23:24:28Z,2021-06-25T23:24:28Z,OWNER,Maybe I could release a separate Python package `types-sqlite-utils-numpy` which adds an over-ridden type definition that includes the `numpy` types?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-868881033,https://api.github.com/repos/simonw/sqlite-utils/issues/37,868881033,MDEyOklzc3VlQ29tbWVudDg2ODg4MTAzMw==,9599,simonw,2021-06-25T23:23:49Z,2021-06-25T23:23:49Z,OWNER,"Twitter conversation about how to add types to the `.create_table(columns=)` parameter: https://twitter.com/simonw/status/1408532867592818693

> Anyone know how to write a mypy type definition for this?
>
> {""id"": int, ""name"": str, ""image"": bytes, ""weight"": float}
>
> It's a dict where keys are strings and values are one of int/str/bytes/float (weird API design I know, but I designed this long before I was thinking about mypy)

Looks like this could work:

```python
    def create_table(
        self,
        name,
        columns: Dict[str, Union[Type[int], Type[bytes], Type[str], Type[float]]],
        pk=None,
        foreign_keys=None,
        column_order=None,
        not_null=None,
        defaults=None,
        hash_id=None,
        extracts=None,
    ):
```
Except... that method can optionally also accept `numpy` types if `numpy` is installed. I don't know if it's possible to dynamically change a signature based on an import, since `mypy` is a static type analyzer and doesn't ever execute the code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868728092,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868728092,MDEyOklzc3VlQ29tbWVudDg2ODcyODA5Mg==,9599,simonw,2021-06-25T17:39:35Z,2021-06-25T17:39:35Z,OWNER,Here's more about this problem: https://github.com/numpy/numpy/issues/15947,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868134040,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868134040,MDEyOklzc3VlQ29tbWVudDg2ODEzNDA0MA==,9599,simonw,2021-06-25T01:49:44Z,2021-06-25T01:50:33Z,OWNER,"Test failed on 3.10 with `numpy` on macOS:
```
sqlite_utils/__init__.py:1: in <module>
11
    from .db import Database
12
sqlite_utils/db.py:48: in <module>
13
    import numpy as np  # type: ignore
14
../../../hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/numpy/__init__.py:391: in <module>
15
    raise RuntimeError(msg)
16
E   RuntimeError: Polyfit sanity test emitted a warning, most likely due to using a buggy Accelerate backend. If you compiled yourself, more information is available at https://numpy.org/doc/stable/user/building.html#accelerated-blas-lapack-libraries Otherwise report this to the vendor that provided NumPy.
17
E   RankWarning: Polyfit may be poorly conditioned
18
Error: Process completed with exit code 4.
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/pull/293#issuecomment-868125750,https://api.github.com/repos/simonw/sqlite-utils/issues/293,868125750,MDEyOklzc3VlQ29tbWVudDg2ODEyNTc1MA==,22429695,codecov[bot],2021-06-25T01:42:43Z,2021-06-25T01:42:43Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#293](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (ae0f46a) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/747be6057d09a4e5d9d726e29d5cf99b10c59dea?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (747be60) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/293/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main     #293   +/-   ##
=======================================
  Coverage   96.03%   96.03%           
=======================================
  Files           4        4           
  Lines        1994     1994           
=======================================
  Hits         1915     1915           
  Misses         79       79           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [747be60...ae0f46a](https://codecov.io/gh/simonw/sqlite-utils/pull/293?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",929748885,Test against Python 3.10-dev,
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-868021624,https://api.github.com/repos/simonw/sqlite-utils/issues/290,868021624,MDEyOklzc3VlQ29tbWVudDg2ODAyMTYyNA==,9599,simonw,2021-06-24T23:17:38Z,2021-06-24T23:17:38Z,OWNER,The new documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#executing-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/datasette/issues/1377#issuecomment-867209791,https://api.github.com/repos/simonw/datasette/issues/1377,867209791,MDEyOklzc3VlQ29tbWVudDg2NzIwOTc5MQ==,9599,simonw,2021-06-23T22:51:32Z,2021-06-23T22:51:32Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#skip-csrf-datasette-scope,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/simonw/datasette/pull/1368#issuecomment-867102944,https://api.github.com/repos/simonw/datasette/issues/1368,867102944,MDEyOklzc3VlQ29tbWVudDg2NzEwMjk0NA==,9599,simonw,2021-06-23T19:32:01Z,2021-06-23T19:32:01Z,OWNER,"Yes, let's move ahead with getting this into an alpha.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866466388,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866466388,MDEyOklzc3VlQ29tbWVudDg2NjQ2NjM4OA==,9599,simonw,2021-06-23T02:10:24Z,2021-06-23T02:10:24Z,OWNER,This already helped me spot a bug in a test: https://github.com/simonw/sqlite-utils/commit/90e211e3e2f36d2ff911ecf1afe4470ff45c7c0d#diff-4e8715c7a425ee52e74b7df4d34efd32e8c92f3e60bd51bc2e1ad5943b82032eL1160-L1161,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8,
https://github.com/simonw/sqlite-utils/issues/291#issuecomment-866461926,https://api.github.com/repos/simonw/sqlite-utils/issues/291,866461926,MDEyOklzc3VlQ29tbWVudDg2NjQ2MTkyNg==,9599,simonw,2021-06-23T01:59:57Z,2021-06-23T01:59:57Z,OWNER,"That shouldn't be failing: it's complaining about these: https://github.com/simonw/sqlite-utils/blob/02898bf7af4a4e484ecc8ec852d5fee98463277b/tests/test_register_function.py#L56-L67

But I added `# noqa: F811` to them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",927766296,Adopt flake8,
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866241836,https://api.github.com/repos/simonw/sqlite-utils/issues/289,866241836,MDEyOklzc3VlQ29tbWVudDg2NjI0MTgzNg==,857609,adamchainz,2021-06-22T18:44:36Z,2021-06-22T18:44:36Z,NONE,Great!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865511810,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865511810,MDEyOklzc3VlQ29tbWVudDg2NTUxMTgxMA==,9599,simonw,2021-06-22T04:07:34Z,2021-06-22T18:26:21Z,OWNER,"That documentation section is pretty weak at the moment - here's the whole thing:

> ### Executing queries
>
> The `db.execute()` and `db.executescript()` methods provide wrappers around `.execute()` and `.executescript()` on the underlying SQLite connection. These wrappers log to the tracer function if one has been registered.
> ```python
> db = Database(memory=True)
> db[""dogs""].insert({""name"": ""Cleo""})
> db.execute(""update dogs set name = 'Cleopaws'"")
> ```
> You can pass parameters as an optional second argument, using either a list or a dictionary. These will be correctly quoted and escaped.
> ```python
> # Using ? and a list:
> db.execute(""update dogs set name = ?"", [""Cleopaws""])
> # Or using :name and a dictionary:
> db.execute(""update dogs set name = :name"", {""name"": ""Cleopaws""})
> ```

- Talks about `.execute()` - I want to talk about `.query()` instead
- Doesn't clarify that `.execute()` returns a `Cursor` - and assumes you know what to do with one
- Doesn't show an example of a `select` query at all
- The ""tracer function"" bit is confusing (should at least link to docs further down)
- For `UPDATE` should show how to access the number of rows modified (probably using `.execute()` there)

It does at least cover the two types of parameters, though that could be bulked out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/37#issuecomment-509681590,https://api.github.com/repos/simonw/sqlite-utils/issues/37,509681590,MDEyOklzc3VlQ29tbWVudDUwOTY4MTU5MA==,9599,simonw,2019-07-09T15:07:12Z,2021-06-22T18:17:53Z,OWNER,"Here's a magic incantation for generating types detected through running the tests with https://github.com/Instagram/MonkeyType

```
pip install pytest-monkeytype
pytest --monkeytype-output=./monkeytype.sqlite3
monkeytype list-modules
monkeytype apply sqlite_utils.utils
monkeytype apply sqlite_utils.cli
monkeytype apply sqlite_utils.db
```

Here's the result: https://github.com/simonw/sqlite-utils/commit/d18c694fc25b7dd3d76e250c77ddf56d10ddf935","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",465815372,Experiment with type hints,
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-866219755,https://api.github.com/repos/simonw/sqlite-utils/issues/289,866219755,MDEyOklzc3VlQ29tbWVudDg2NjIxOTc1NQ==,9599,simonw,2021-06-22T18:13:26Z,2021-06-22T18:13:26Z,OWNER,Thanks @adamchainz - `mypy` now has a foothold on this project (and runs in CI).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866184260,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866184260,MDEyOklzc3VlQ29tbWVudDg2NjE4NDI2MA==,9599,simonw,2021-06-22T17:26:18Z,2021-06-22T17:27:27Z,OWNER,"If an`.update()` method doesn't work because it collides with an existing dictionary method a `.pk` property could still be nice:
```python
for row in db[""sometable""].rows:
    db[""sometable""].update(row.pk, {""modified"": 1})
```
    
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-866182655,https://api.github.com/repos/simonw/sqlite-utils/issues/267,866182655,MDEyOklzc3VlQ29tbWVudDg2NjE4MjY1NQ==,9599,simonw,2021-06-22T17:24:03Z,2021-06-22T17:24:03Z,OWNER,"I'm re-opening this as a research task because it may be possible to cleanly implement this using a `dict` subclass - some notes on that here: https://treyhunner.com/2019/04/why-you-shouldnt-inherit-from-list-and-dict-in-python/

Since this would just be for adding methods (and maybe a property for returning the primary keys for a row) the usual disadvantages of subclassing `dict` described in that article shouldn't apply.

One catch: dictionaries already have a `.update()` method! So would have to pick another name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865510796,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865510796,MDEyOklzc3VlQ29tbWVudDg2NTUxMDc5Ng==,9599,simonw,2021-06-22T04:04:40Z,2021-06-22T04:04:48Z,OWNER,"Still needs documentation, which will involve rewriting the whole [Executing queries](https://sqlite-utils.datasette.io/en/3.11/python-api.html#executing-queries) section.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865497846,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865497846,MDEyOklzc3VlQ29tbWVudDg2NTQ5Nzg0Ng==,9599,simonw,2021-06-22T03:21:38Z,2021-06-22T03:21:38Z,OWNER,"The Python docs say: https://docs.python.org/3/library/sqlite3.html

> To retrieve data after executing a SELECT statement, you can either treat the cursor as an iterator, call the cursor’s `fetchone()` method to retrieve a single matching row, or call `fetchall()` to get a list of the matching rows.

Looking at the C source code, both `fetchmany()` and `fetchall()` work under the hood by assembling a Python list: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L907-L972 - see calls to `PyList_Append()`

So it looks like the most efficient way to iterate over a cursor may well be `for row in cursor:` - which I think calls this C function: https://github.com/python/cpython/blob/be1cb3214d09d4bf0288bc45f3c1f167f67e4514/Modules/_sqlite/cursor.c#L813-L876","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865495370,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865495370,MDEyOklzc3VlQ29tbWVudDg2NTQ5NTM3MA==,9599,simonw,2021-06-22T03:14:30Z,2021-06-22T03:14:30Z,OWNER,"One small problem with the existing method:
https://github.com/simonw/sqlite-utils/blob/8cedc6a8b29180e68326f6b76f249d5e39e4b591/sqlite_utils/db.py#L362-L365

It returns a full list, but what if the user would rather have a generator they can iterate over without loading the results into memory in one go?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/sqlite-utils/issues/290#issuecomment-865491922,https://api.github.com/repos/simonw/sqlite-utils/issues/290,865491922,MDEyOklzc3VlQ29tbWVudDg2NTQ5MTkyMg==,9599,simonw,2021-06-22T03:05:35Z,2021-06-22T03:05:35Z,OWNER,"Potential names:

- `db.query(sql)` - it's weird to have both this and `db.execute()` but it is at least short and memorable
- `db.sql(sql)`
- `db.execute_d(sql)` - ugly
- `db.execute_dicts(sql)` - confusing
- `db.execute_sql(sql)` - easily confused with `db.execute(sql)`

I think `db.query(sql)` may be the best option here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",926777310,`db.query()` method (renamed `db.execute_returning_dicts()`),
https://github.com/simonw/datasette/pull/1368#issuecomment-865204472,https://api.github.com/repos/simonw/datasette/issues/1368,865204472,MDEyOklzc3VlQ29tbWVudDg2NTIwNDQ3Mg==,2670795,brandonrobertz,2021-06-21T17:11:37Z,2021-06-21T17:11:37Z,CONTRIBUTOR,If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/datasette/pull/1368#issuecomment-865160132,https://api.github.com/repos/simonw/datasette/issues/1368,865160132,MDEyOklzc3VlQ29tbWVudDg2NTE2MDEzMg==,9599,simonw,2021-06-21T16:07:06Z,2021-06-21T16:08:48Z,OWNER,"A few tests failed - Black, the test that checks the docs mention the new hook - the most interesting failing test looks like this one:
```
            updated_metadata[""databases""][""fixtures""][""queries""][""magic_parameters""][
                ""allow""
            ] = (allow if ""query"" in permissions else deny)
>           cascade_app_client.ds._metadata = updated_metadata
E           AttributeError: can't set attribute
```
From https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_permissions.py#L439-L467

This test is directly manipulating `_metadata` purely for the purposes of simulating different permissions - I think updating it to manipulate `_local_metadata` instead would fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864621099,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864621099,MDEyOklzc3VlQ29tbWVudDg2NDYyMTA5OQ==,601708,mcint,2021-06-20T22:39:57Z,2021-06-20T22:39:57Z,CONTRIBUTOR,"Fair. I looked into it, it looks like it could be done, but it would be _a bit ugly_. I can upload and link a gist of my exploration. **Click** can parse a first argument while still recognizing it as a sub-command keyword. From there, the program could:
1. ignore it preemptively if it matches a sub-command
2. and/or check if a (db) file exists at the path.

It would then also need to set a shared db argument variable.

Click also makes it easy to parse arguments from environment variables.  If you're amenable, I may submit a patch for only that, which would update each sub-command to check for a DB/SQLITE_UTILS_DB environment variable. The goal would be usage that looks like: `DB=./convenient.db sqlite-utils [operation] [args]`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/289#issuecomment-864609271,https://api.github.com/repos/simonw/sqlite-utils/issues/289,864609271,MDEyOklzc3VlQ29tbWVudDg2NDYwOTI3MQ==,9599,simonw,2021-06-20T20:42:07Z,2021-06-20T20:42:07Z,OWNER,"Wow, thank you! I didn't know about `typing.cast()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925677191,Mypy fixes for rows_from_file(),
https://github.com/simonw/sqlite-utils/issues/286#issuecomment-864594956,https://api.github.com/repos/simonw/sqlite-utils/issues/286,864594956,MDEyOklzc3VlQ29tbWVudDg2NDU5NDk1Ng==,9599,simonw,2021-06-20T18:38:05Z,2021-06-20T18:38:05Z,OWNER,3.10 is out in Homebrew now (they turn that around so fast): https://formulae.brew.sh/formula/sqlite-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925487946,Add installation instructions,
https://github.com/simonw/datasette/issues/1382#issuecomment-864480051,https://api.github.com/repos/simonw/datasette/issues/1382,864480051,MDEyOklzc3VlQ29tbWVudDg2NDQ4MDA1MQ==,9599,simonw,2021-06-20T00:20:06Z,2021-06-20T00:21:02Z,OWNER,"Yes you can - thanks for pointing this out, I've added a comment to the `install.sh` script in the `datasette-csvs` Glitch project:

```bash
pip3 install -U --no-cache-dir -r requirements.txt --user && \
  mkdir -p .data && \
  rm .data/data.db || true && \
  for f in *.csv
    do
      # Add --encoding=latin-1 to the following if your CSVs use a different encoding:
      sqlite-utils insert .data/data.db ${f%.*} $f --csv
    done
```

So if you edit that file in your own project and change the line to this:

      sqlite-utils insert .data/data.db ${f%.*} $f --csv --encoding=iso-8859-1

It should fix this for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925406964,Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864476167,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864476167,MDEyOklzc3VlQ29tbWVudDg2NDQ3NjE2Nw==,9599,simonw,2021-06-19T23:36:48Z,2021-06-19T23:36:48Z,OWNER,Wrote this up on my blog here: https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/ - with a video demo here: https://www.youtube.com/watch?v=OUjd0rkc678,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864419283,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864419283,MDEyOklzc3VlQ29tbWVudDg2NDQxOTI4Mw==,9599,simonw,2021-06-19T15:15:34Z,2021-06-19T15:15:34Z,OWNER,"I think this code is at fault: https://github.com/simonw/sqlite-utils/blob/5b257949d996fe43dc5d218d4308b88796a90740/sqlite_utils/db.py#L1017-L1023

It's using `.pks` which adds `rowid` if it's missing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418795,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418795,MDEyOklzc3VlQ29tbWVudDg2NDQxODc5NQ==,9599,simonw,2021-06-19T15:11:05Z,2021-06-19T15:11:14Z,OWNER,"Actually I'm going to go with `use_rowid` instead - because the table doesn't inherently use a rowid itself, but you should use one if you want to query it in a way that gives you back a primary key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864418188,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864418188,MDEyOklzc3VlQ29tbWVudDg2NDQxODE4OA==,9599,simonw,2021-06-19T15:05:53Z,2021-06-19T15:05:53Z,OWNER,"```python
    @property
    def uses_rowid(self):
        return not any(column for column in self.columns if column.is_pk)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417808,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417808,MDEyOklzc3VlQ29tbWVudDg2NDQxNzgwOA==,9599,simonw,2021-06-19T15:03:00Z,2021-06-19T15:03:00Z,OWNER,I think I like `table.uses_rowid` best - it reads well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417765,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417765,MDEyOklzc3VlQ29tbWVudDg2NDQxNzc2NQ==,9599,simonw,2021-06-19T15:02:42Z,2021-06-19T15:02:42Z,OWNER,"Some options:

- `table.rowid_only`
- `table.rowid_as_pk`
- `table.no_pks`
- `table.no_pk`
- `table.uses_rowid`
- `table.use_rowid`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417493,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417493,MDEyOklzc3VlQ29tbWVudDg2NDQxNzQ5Mw==,9599,simonw,2021-06-19T15:00:43Z,2021-06-19T15:00:43Z,OWNER,"I have to be careful about the language I use here. Here's the official definition: https://www.sqlite.org/rowidtable.html

> A ""rowid table"" is any table in an SQLite schema that
> 
> -   is *not* a [virtual table](https://www.sqlite.org/vtab.html), and
> -   is *not* a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table.
> 
> Most tables in a typical SQLite database schema are rowid tables.
> 
> Rowid tables are distinguished by the fact that they all have a unique, non-NULL, signed 64-bit integer [rowid](https://www.sqlite.org/lang_createtable.html#rowid) that is used as the access key for the data in the underlying [B-tree](https://www.sqlite.org/fileformat2.html#btree) storage engine.

So it's not correct to call a table a ""rowid table"" only if it is missing its own primary keys.

Maybe `table.has_rowid` is the right language to use here? No, that's no good - because tables with their own primary keys usually also have a rowid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417133,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417133,MDEyOklzc3VlQ29tbWVudDg2NDQxNzEzMw==,9599,simonw,2021-06-19T14:57:36Z,2021-06-19T14:57:36Z,OWNER,"So the logic is:
```python
[column.name for column in self.columns if column.is_pk]
```
I need to decide on a property name. Existing names are documented here: https://sqlite-utils.datasette.io/en/stable/python-api.html#introspection","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/285#issuecomment-864417031,https://api.github.com/repos/simonw/sqlite-utils/issues/285,864417031,MDEyOklzc3VlQ29tbWVudDg2NDQxNzAzMQ==,9599,simonw,2021-06-19T14:56:45Z,2021-06-19T14:56:45Z,OWNER,"```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""rowid_table""].insert({""name"": ""Cleo""})
<Table rowid_table (name)>
>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
<Table regular_table (id, name)>
>>> db[""rowid_table""].pks
['rowid']
>>> db[""regular_table""].pks
['id']
```
But that's because the `.pks` property hides the difference: https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925410305,Introspection property for telling if a table is a rowid table,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416911,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416911,MDEyOklzc3VlQ29tbWVudDg2NDQxNjkxMQ==,9599,simonw,2021-06-19T14:55:45Z,2021-06-19T14:55:45Z,OWNER,"https://github.com/simonw/sqlite-utils/blob/dc94f4bb8cfe922bb2f9c89f8f0f29092ea63133/sqlite_utils/db.py#L805-L810

So I can indeed detect a `rowid` table by looking for no `is_pk` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864416785,MDEyOklzc3VlQ29tbWVudDg2NDQxNjc4NQ==,9599,simonw,2021-06-19T14:54:41Z,2021-06-19T14:54:41Z,OWNER,"```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""rowid_table""].insert({""name"": ""Cleo""})
<Table rowid_table (name)>
>>> db[""regular_table""].insert({""id"": 1, ""name"": ""Cleo""}, pk=""id"")
<Table regular_table (id, name)>
>>> db[""rowid_table""].pks
['rowid']
>>> db[""regular_table""].pks
['id']
```
I think I need an introspection property for working out if a table is a `rowid` table or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/283#issuecomment-864416086,https://api.github.com/repos/simonw/sqlite-utils/issues/283,864416086,MDEyOklzc3VlQ29tbWVudDg2NDQxNjA4Ng==,9599,simonw,2021-06-19T14:49:06Z,2021-06-19T14:49:13Z,OWNER,"Once again, this is difficult because of the use of a generator here - `rows_from_file()` only yields rows, so there is no obvious mechanism for it to communicate back to the wrapping code that the detected format was CSV or TSV as opposed to JSON.

I'm going to change `rows_from_file()` to return a `(generator, detected_format)` tuple.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925319214,memory: Shouldn't detect types for JSON,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358951,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358951,MDEyOklzc3VlQ29tbWVudDg2NDM1ODk1MQ==,9599,simonw,2021-06-19T05:30:00Z,2021-06-19T05:30:00Z,OWNER,If this can be fixed it will be in the `transform_sql()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864358680,https://api.github.com/repos/simonw/sqlite-utils/issues/284,864358680,MDEyOklzc3VlQ29tbWVudDg2NDM1ODY4MA==,9599,simonw,2021-06-19T05:27:13Z,2021-06-19T05:27:13Z,OWNER,How easy is it to detect a `rowid` table? Is it as simple as `.pks` returning `None`? If so the documentation should mention that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925320167,.transform(types=) turns rowid into a concrete column,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864354627,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864354627,MDEyOklzc3VlQ29tbWVudDg2NDM1NDYyNw==,9599,simonw,2021-06-19T04:42:03Z,2021-06-19T04:42:03Z,OWNER,"Demo:

    curl -s 'https://api.github.com/users/simonw/repos?per_page=100' | \
      sqlite-utils memory - 'select sum(size), sum(stargazers_count) from stdin limit 1'
    [{""sum(size)"": 2042547, ""sum(stargazers_count)"": 6769}]
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864350407,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864350407,MDEyOklzc3VlQ29tbWVudDg2NDM1MDQwNw==,9599,simonw,2021-06-19T03:52:20Z,2021-06-19T03:52:20Z,OWNER,I'll have an environment variable for `--detect-types` so users who really want that as the default option can turn it on.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864349123,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864349123,MDEyOklzc3VlQ29tbWVudDg2NDM0OTEyMw==,9599,simonw,2021-06-19T03:36:54Z,2021-06-19T03:36:54Z,OWNER,"I may change the default for `sqlite-utils insert` to detect types if I release `sqlite-utils` 4.0, as a backwards-incompatible change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/179#issuecomment-864349066,https://api.github.com/repos/simonw/sqlite-utils/issues/179,864349066,MDEyOklzc3VlQ29tbWVudDg2NDM0OTA2Ng==,9599,simonw,2021-06-19T03:36:04Z,2021-06-19T03:36:04Z,OWNER,This work is going to happen in #282.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types,
https://github.com/simonw/sqlite-utils/issues/282#issuecomment-864348954,https://api.github.com/repos/simonw/sqlite-utils/issues/282,864348954,MDEyOklzc3VlQ29tbWVudDg2NDM0ODk1NA==,9599,simonw,2021-06-19T03:34:42Z,2021-06-19T03:35:46Z,OWNER,"I built some prototype code here for something which looks at every row in a CSV import and records the likely types: https://gist.github.com/simonw/465f9356f175d1cf86957947dff501d4

This could be used by the command-line tools to figure out what `table.transform(types=...)` method to use at the end.

This is a different approach to the pure SQL version I tried building in https://github.com/simonw/sqlite-utils/issues/179 - I think this is a better approach though, it's less prone to weird idiosyncrasies of SQLite types, and it's also easy for us to add on to the existing CSV import code in a way that won't require scanning the data twice.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",925305186,Automatic type detection for CSV data,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864330508,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864330508,MDEyOklzc3VlQ29tbWVudDg2NDMzMDUwOA==,9599,simonw,2021-06-19T00:34:24Z,2021-06-19T00:34:24Z,OWNER,"Got this working:

    % curl 'https://api.github.com/repos/simonw/datasette/issues' | sqlite-utils memory - 'select id from stdin' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864328927,MDEyOklzc3VlQ29tbWVudDg2NDMyODkyNw==,9599,simonw,2021-06-19T00:25:08Z,2021-06-19T00:25:17Z,OWNER,"I tried writing this function with type hints, but eventually gave up:
```python
def rows_from_file(
    fp: BinaryIO,
    format: Optional[Format] = None,
    dialect: Optional[Type[csv.Dialect]] = None,
    encoding: Optional[str] = None,
) -> Generator[dict, None, None]:
    if format == Format.JSON:
        decoded = json.load(fp)
        if isinstance(decoded, dict):
            decoded = [decoded]
        if not isinstance(decoded, list):
            raise RowsFromFileBadJSON(""JSON must be a list or a dictionary"")
        yield from decoded
    elif format == Format.CSV:
        decoded_fp = io.TextIOWrapper(fp, encoding=encoding or ""utf-8-sig"")
        yield from csv.DictReader(decoded_fp)
    elif format == Format.TSV:
        yield from rows_from_file(
            fp, format=Format.CSV, dialect=csv.excel_tab, encoding=encoding
        )
    elif format is None:
        # Detect the format, then call this recursively
        buffered = io.BufferedReader(fp, buffer_size=4096)
        first_bytes = buffered.peek(2048).strip()
        if first_bytes[0] in (b""["", b""{""):
            # TODO: Detect newline-JSON
            yield from rows_from_file(fp, format=Format.JSON)
        else:
            dialect = csv.Sniffer().sniff(first_bytes.decode(encoding, ""ignore""))
            yield from rows_from_file(
                fp, format=Format.CSV, dialect=dialect, encoding=encoding
            )
    else:
        raise RowsFromFileError(""Bad format"")
```
mypy said:
```
sqlite_utils/utils.py:157: error: Argument 1 to ""BufferedReader"" has incompatible type ""BinaryIO""; expected ""RawIOBase""
sqlite_utils/utils.py:163: error: Argument 1 to ""decode"" of ""bytes"" has incompatible type ""Optional[str]""; expected ""str""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/281#issuecomment-864323438,https://api.github.com/repos/simonw/sqlite-utils/issues/281,864323438,MDEyOklzc3VlQ29tbWVudDg2NDMyMzQzOA==,9599,simonw,2021-06-18T23:55:06Z,2021-06-18T23:55:06Z,OWNER,"The `-:json` idea is flawed: Click thinks that's the syntax for an option called `:json`.

I'm going to do `stdin:json` - which means you can't open a file called `stdin` - but you could use `cat stdin | sqlite-utils memory stdin:json ...`  instead which is an OK workaround.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924992318,Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864208476,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864208476,MDEyOklzc3VlQ29tbWVudDg2NDIwODQ3Ng==,9599,simonw,2021-06-18T18:30:08Z,2021-06-18T23:30:19Z,OWNER,"So maybe this is a function which can either be told the format or, if none is provided, it detects one for itself.
```python
def rows_from_file(fp, format=None):
    # ...
    yield from rows
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864207841,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864207841,MDEyOklzc3VlQ29tbWVudDg2NDIwNzg0MQ==,9599,simonw,2021-06-18T18:28:40Z,2021-06-18T18:28:46Z,OWNER,"```python
def detect_format(fp):
    # ...
    return ""csv"", fp, dialect
    # or
    return ""json"", fp, parsed_data
    # or
    return ""json-nl"", fp, docs
```
The mixed return types here are ugly. In all of these cases what we really want is to return a generator of `{...}` objects. So maybe it returns that instead.
```python
def filepointer_to_documents(fp):
    # ...
    yield from documents
```
I can refactor `sqlite-utils insert` to use this new code too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864206308,MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA==,9599,simonw,2021-06-18T18:25:04Z,2021-06-18T18:25:04Z,OWNER,"Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it.

I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too?

This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway.

So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864129273,MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw==,9599,simonw,2021-06-18T15:47:47Z,2021-06-18T15:47:47Z,OWNER,"Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure.

One way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864128489,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864128489,MDEyOklzc3VlQ29tbWVudDg2NDEyODQ4OQ==,9599,simonw,2021-06-18T15:46:24Z,2021-06-18T15:46:24Z,OWNER,A workaround could be to define a bash or zsh alias of some sort.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864126781,https://api.github.com/repos/simonw/sqlite-utils/issues/278,864126781,MDEyOklzc3VlQ29tbWVudDg2NDEyNjc4MQ==,9599,simonw,2021-06-18T15:43:19Z,2021-06-18T15:43:19Z,OWNER,"I don't think it's possible to do this without breaking backwards compatibility, unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923697888,"Support db as first parameter before subcommand, or as environment variable",
https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005,https://api.github.com/repos/simonw/sqlite-utils/issues/279,864103005,MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ==,9599,simonw,2021-06-18T15:04:15Z,2021-06-18T15:04:15Z,OWNER,"To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",924990677,sqlite-utils memory should handle TSV and JSON in addition to CSV,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-864101267,https://api.github.com/repos/simonw/sqlite-utils/issues/272,864101267,MDEyOklzc3VlQ29tbWVudDg2NDEwMTI2Nw==,9599,simonw,2021-06-18T15:01:41Z,2021-06-18T15:01:41Z,OWNER,I'll split the remaining work out into separate issues.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-864099764,https://api.github.com/repos/simonw/sqlite-utils/issues/273,864099764,MDEyOklzc3VlQ29tbWVudDg2NDA5OTc2NA==,9599,simonw,2021-06-18T14:59:27Z,2021-06-18T14:59:27Z,OWNER,I'm going to merge this as-is and work on the JSON/TSV support in a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/277#issuecomment-864092515,https://api.github.com/repos/simonw/sqlite-utils/issues/277,864092515,MDEyOklzc3VlQ29tbWVudDg2NDA5MjUxNQ==,9599,simonw,2021-06-18T14:47:57Z,2021-06-18T14:47:57Z,OWNER,This is a neat improvement.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276,
https://github.com/simonw/datasette/pull/1378#issuecomment-863230355,https://api.github.com/repos/simonw/datasette/issues/1378,863230355,MDEyOklzc3VlQ29tbWVudDg2MzIzMDM1NQ==,22429695,codecov[bot],2021-06-17T13:16:17Z,2021-06-17T13:16:17Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1378](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0c132d1) into [main](https://codecov.io/gh/simonw/datasette/commit/83e9c8bc7585dcc62f200e37c2daefcd669ee05e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (83e9c8b) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1378/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1378   +/-   ##
=======================================
  Coverage   91.68%   91.68%           
=======================================
  Files          34       34           
  Lines        4340     4340           
=======================================
  Hits         3979     3979           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [83e9c8b...0c132d1](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923910375,"Update pytest-xdist requirement from <2.3,>=2.2.1 to >=2.2.1,<2.4",
https://github.com/simonw/sqlite-utils/pull/277#issuecomment-863205049,https://api.github.com/repos/simonw/sqlite-utils/issues/277,863205049,MDEyOklzc3VlQ29tbWVudDg2MzIwNTA0OQ==,22429695,codecov[bot],2021-06-17T12:40:49Z,2021-06-17T12:40:49Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#277](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (abbd324) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a19ce1a4d0048d389411cfe11a5dbe4c503720e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a19ce1a) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/277/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main     #277   +/-   ##
=======================================
  Coverage   96.06%   96.06%           
=======================================
  Files           4        4           
  Lines        1828     1829    +1     
=======================================
+ Hits         1756     1757    +1     
  Misses         72       72           
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/277/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.03% <100.00%> (+<0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a19ce1a...abbd324](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",923612361,add -h support closes #276,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862817185,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862817185,MDEyOklzc3VlQ29tbWVudDg2MjgxNzE4NQ==,22429695,codecov[bot],2021-06-17T00:15:34Z,2021-06-17T00:15:34Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> :exclamation: No coverage uploaded for pull request base (`main@78aebb6`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#section-missing-base-commit).
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/273/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main     #273   +/-   ##
=======================================
  Coverage        ?   96.10%           
=======================================
  Files           ?        4           
  Lines           ?     1873           
  Branches        ?        0           
=======================================
  Hits            ?     1800           
  Misses          ?       73           
  Partials        ?        0           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [78aebb6...df7a37b](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/275#issuecomment-862617165,https://api.github.com/repos/simonw/sqlite-utils/issues/275,862617165,MDEyOklzc3VlQ29tbWVudDg2MjYxNzE2NQ==,9599,simonw,2021-06-16T18:34:51Z,2021-06-16T18:34:51Z,OWNER,"Also use this: https://github.com/simonw/datasette/blob/83e9c8bc7585dcc62f200e37c2daefcd669ee05e/codecov.yml

And add a badge, as seen on https://github.com/simonw/asgi-csrf","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922955697,Enable code coverage,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862605436,MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg==,9599,simonw,2021-06-16T18:19:05Z,2021-06-16T18:19:05Z,OWNER,`--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862574390,https://api.github.com/repos/simonw/sqlite-utils/issues/270,862574390,MDEyOklzc3VlQ29tbWVudDg2MjU3NDM5MA==,4068,frafra,2021-06-16T17:34:49Z,2021-06-16T17:34:49Z,NONE,"Sorry, I got confused because SQLite has a JSON column type, even if it is treated as TEXT, and I though automatic facets were available for JSON arrays stored as JSON only :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862494864,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862494864,MDEyOklzc3VlQ29tbWVudDg2MjQ5NDg2NA==,9599,simonw,2021-06-16T15:51:28Z,2021-06-16T16:26:15Z,OWNER,"I did add a slightly clumsy mechanism recently to help a bit here though: the `pks_and_rows_where()` method: https://sqlite-utils.datasette.io/en/stable/python-api.html#listing-rows-with-their-primary-keys

More details in the issue for that feature: #240

The idea here is that if you want to call update you need the primary key for the row - so you can do this:

```python
for pk, row in db[""sometable""].pks_and_rows_where():
    db[""sometable""].update(pk, {""modified"": 1}"")
```
The `pk` may end up as a single value or a tuple depending on if the table has a compound primary key - but you don't need to worry about that if you use this method as it will return the correct primary key value for you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-862495803,https://api.github.com/repos/simonw/sqlite-utils/issues/131,862495803,MDEyOklzc3VlQ29tbWVudDg2MjQ5NTgwMw==,9599,simonw,2021-06-16T15:52:33Z,2021-06-16T15:52:33Z,OWNER,I like `-t` or `--type` for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862493179,https://api.github.com/repos/simonw/sqlite-utils/issues/267,862493179,MDEyOklzc3VlQ29tbWVudDg2MjQ5MzE3OQ==,9599,simonw,2021-06-16T15:49:13Z,2021-06-16T15:49:13Z,OWNER,"The big challenge here is that the rows returned  by this library aren't objects, they are Python dictionaries - so adding methods to them isn't possible without changing the type that is returned by these methods.

Part of the philosophy of the library is that it should make it as easy as possible to round-trip between Python dictionaries and SQLite table data, so I don't think adding methods like this is going to fit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915421499,row.update() or row.pk,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862491721,https://api.github.com/repos/simonw/sqlite-utils/issues/270,862491721,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTcyMQ==,9599,simonw,2021-06-16T15:47:06Z,2021-06-16T15:47:06Z,OWNER,"SQLite doesn't have a JSON column type - it has JSON processing functions, but they operate against TEXT columns - so there's nothing I can do here unfortunately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862491016,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862491016,MDEyOklzc3VlQ29tbWVudDg2MjQ5MTAxNg==,9599,simonw,2021-06-16T15:46:13Z,2021-06-16T15:46:13Z,OWNER,"Columns from data imported from CSV in this way is currently treated as `TEXT`, which means numeric sorts and suchlike won't work as people might expect. It would be good to do automatic type detection here, see #179.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862485408,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862485408,MDEyOklzc3VlQ29tbWVudDg2MjQ4NTQwOA==,9599,simonw,2021-06-16T15:38:58Z,2021-06-16T15:39:28Z,OWNER,"Also `sqlite-utils memory` reflects the existing `sqlite-utils :memory:` mechanism, which is a point in its favour.

And it helps emphasize that the file you are querying will be loaded into memory, so probably don't try this against a 1GB CSV file.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862484557,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862484557,MDEyOklzc3VlQ29tbWVudDg2MjQ4NDU1Nw==,9599,simonw,2021-06-16T15:37:51Z,2021-06-16T15:38:34Z,OWNER,"I wonder if there's a better name for this than `sqlite-utils memory`?

- `sqlite-utils memory hello.csv ""select * from hello""`
- `sqlite-utils mem hello.csv ""select * from hello""`
- `sqlite-utils temp hello.csv ""select * from hello""`
- `sqlite-utils adhoc hello.csv ""select * from hello""`
- `sqlite-utils scratch hello.csv ""select * from hello""`

I think `memory` is best. I don't like the others, except for `scratch` which is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862479704,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862479704,MDEyOklzc3VlQ29tbWVudDg2MjQ3OTcwNA==,9599,simonw,2021-06-16T15:31:31Z,2021-06-16T15:31:31Z,OWNER,"Plus, could I make this change to `sqlite-utils query` without breaking backwards compatibility? Adding a new `sqlite-utils memory` command is completely safe from that perspective.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862478881,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862478881,MDEyOklzc3VlQ29tbWVudDg2MjQ3ODg4MQ==,9599,simonw,2021-06-16T15:30:24Z,2021-06-16T15:30:24Z,OWNER,"But... `sqlite-utils my.csv ""select * from my""` is a much more compelling initial experience than `sqlite-utils memory my.csv ""select * from my""`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862475685,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862475685,MDEyOklzc3VlQ29tbWVudDg2MjQ3NTY4NQ==,9599,simonw,2021-06-16T15:26:19Z,2021-06-16T15:29:38Z,OWNER,"Here's a radical idea: what if I combined `sqlite-utils memory` into `sqlite-utils query`?

The trick here would be to detect if the arguments passed on the command-line refer to SQLite databases or if they refer to CSV/JSON data that should be imported into temporary tables.

Detecting a SQLite database file is actually really easy - they all start with the same binary string:
```pycon
>>> open(""my.db"", ""rb"").read(100)
b'SQLite format 3\x00...
```
(Need to carefully check that a CSV file with`SQLite format 3` as the first column name doesn't accidentally get interpreted as a SQLite DB though).

So then what would the semantics of `sqlite-utils query` (which is also the default command) be?

- `sqlite-utils mydb.db ""select * from x""`
- `sqlite-utils my.csv ""select * from my""`
- `sqlite-utils mydb.db my.csv ""select * from mydb.x join my on ...""` - this is where it gets weird. We can't import the CSV data directly into `mpdb.db` - it's suppose to go into the in-memory database - so now we need to start using database aliases like `mydb.x` because we passed at least one other file?

The complexity here is definitely in the handling of a combination of SQLite database files and CSV filenames. Also, `sqlite-utils query` doesn't accept multiple filenames at the moment, so that will change.

I'm not 100% sold on this as being better than having a separate `sqlite-utils memory` command, as seen in #273.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862046009,MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ==,9599,simonw,2021-06-16T05:15:38Z,2021-06-16T05:15:38Z,OWNER,"I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045639,MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ==,9599,simonw,2021-06-16T05:14:38Z,2021-06-16T05:14:38Z,OWNER,"Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862045438,MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA==,9599,simonw,2021-06-16T05:14:00Z,2021-06-16T05:14:00Z,OWNER,I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862043974,MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA==,9599,simonw,2021-06-16T05:10:12Z,2021-06-16T05:10:12Z,OWNER,"I can stop promoting `:memory:` here and promote `memory` instead:

https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110,https://api.github.com/repos/simonw/sqlite-utils/issues/273,862042110,MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA==,9599,simonw,2021-06-16T05:05:51Z,2021-06-16T05:06:11Z,OWNER,"Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data

It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",922099793,sqlite-utils memory command for directly querying CSV/JSON data,
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040971,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040971,MDEyOklzc3VlQ29tbWVudDg2MjA0MDk3MQ==,9599,simonw,2021-06-16T05:02:56Z,2021-06-16T05:02:56Z,OWNER,Moving this to a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040906,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862040906,MDEyOklzc3VlQ29tbWVudDg2MjA0MDkwNg==,9599,simonw,2021-06-16T05:02:47Z,2021-06-16T05:02:47Z,OWNER,"Got a prototype working!
```
 % curl -s 'https://fivethirtyeight.datasettes.com/polls/president_approval_polls.csv?_size=max&_stream=1' | sqlite-utils memory - 'select * from t limit 5' --nl 
{""rowid"": ""1"", ""question_id"": ""139304"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1500"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""42.0"", ""no"": ""53.0""}
{""rowid"": ""2"", ""question_id"": ""139305"", ""poll_id"": ""74225"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""568"", ""pollster"": ""YouGov"", ""sponsor_ids"": ""352"", ""sponsors"": ""Economist"", ""display_name"": ""YouGov"", ""pollster_rating_id"": ""391"", ""pollster_rating_name"": ""YouGov"", ""fte_grade"": ""B"", ""sample_size"": ""1155"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Online"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf"", ""source"": ""538"", ""yes"": ""44.0"", ""no"": ""55.0""}
{""rowid"": ""3"", ""question_id"": ""139306"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""1100"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""30.0"", ""no"": ""66.0""}
{""rowid"": ""4"", ""question_id"": ""139307"", ""poll_id"": ""74226"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""23"", ""pollster"": ""American Research Group"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""American Research Group"", ""pollster_rating_id"": ""9"", ""pollster_rating_name"": ""American Research Group"", ""fte_grade"": ""B"", ""sample_size"": ""990"", ""population"": ""rv"", ""population_full"": ""rv"", ""methodology"": ""Live Phone"", ""start_date"": ""1/16/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/20/21 10:18"", ""notes"": """", ""url"": ""https://americanresearchgroup.com/economy/"", ""source"": ""538"", ""yes"": ""29.0"", ""no"": ""67.0""}
{""rowid"": ""5"", ""question_id"": ""139298"", ""poll_id"": ""74224"", ""state"": """", ""politician_id"": ""11"", ""politician"": ""Donald Trump"", ""pollster_id"": ""1528"", ""pollster"": ""AtlasIntel"", ""sponsor_ids"": """", ""sponsors"": """", ""display_name"": ""AtlasIntel"", ""pollster_rating_id"": ""546"", ""pollster_rating_name"": ""AtlasIntel"", ""fte_grade"": ""B/C"", ""sample_size"": ""5188"", ""population"": ""a"", ""population_full"": ""a"", ""methodology"": ""Online"", ""start_date"": ""1/15/21"", ""end_date"": ""1/19/21"", ""sponsor_candidate"": """", ""tracking"": """", ""created_at"": ""1/19/21 21:52"", ""notes"": """", ""url"": ""https://projects.fivethirtyeight.com/polls/20210119_US_Atlas2.pdf"", ""source"": ""538"", ""yes"": ""44.6"", ""no"": ""53.9""}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937,https://api.github.com/repos/simonw/sqlite-utils/issues/272,862018937,MDEyOklzc3VlQ29tbWVudDg2MjAxODkzNw==,9599,simonw,2021-06-16T03:59:28Z,2021-06-16T04:00:05Z,OWNER,"Mainly for debugging purposes it would be useful to be able to save the created in-memory database back to a file again later. This could be done with:

    sqlite-utils memory blah.csv --save saved.db

Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump

Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861989987,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861989987,MDEyOklzc3VlQ29tbWVudDg2MTk4OTk4Nw==,9599,simonw,2021-06-16T02:34:21Z,2021-06-16T02:34:21Z,OWNER,"The documentation already covers this
```
$ sqlite-utils :memory: ""select sqlite_version()""
[{""sqlite_version()"": ""3.29.0""}]
```
https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-and-returning-json

`sqlite-utils memory ""select sqlite_version()""` is a little bit more intuitive than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861987651,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861987651,MDEyOklzc3VlQ29tbWVudDg2MTk4NzY1MQ==,9599,simonw,2021-06-16T02:27:20Z,2021-06-16T02:27:20Z,OWNER,Solution: `sqlite-utils memory -` attempts to detect the input based on if it starts with a `{` or `[` (likely JSON) or if it doesn't use the `csv.Sniffer()` mechanism. Or you can use `sqlite-utils memory -:csv` to specifically indicate the type of input.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861985944,MDEyOklzc3VlQ29tbWVudDg2MTk4NTk0NA==,9599,simonw,2021-06-16T02:22:52Z,2021-06-16T02:22:52Z,OWNER,"Another option: allow an optional `:suffix` specifying the type of the file. If this is missing we detect based on the filename.

    sqlite-utils memory somefile:csv ""select * from somefile""

One catch: how to treat `-` for standard input?

    cat blah.csv | sqlite-utils memory - ""select * from stdin""

That's fine for CSV, but what about TSV or JSON or nl-JSON? Maybe this:

    cat blah.csv | sqlite-utils memory -:json ""select * from stdin""

Bit weird though. The alternative would be to support this:

    cat blah.csv | sqlite-utils memory --load-csv -

But that's verbose compared to the version without the long `--load-x` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861984707,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861984707,MDEyOklzc3VlQ29tbWVudDg2MTk4NDcwNw==,9599,simonw,2021-06-16T02:19:48Z,2021-06-16T02:19:48Z,OWNER,"This is going to need to be a separate command, for relatively non-obvious reasons.

    sqlite-utils blah.db ""select * from x""

Is equivalent to this, because `query` is the default sub-command:

    sqlite-utils query blah.db ""select * from x""

But... this means that making the filename optional doesn't actually work - because then this is ambiguous:

    sqlite-utils --load-csv blah.csv ""select * from blah""

So instead, I'm going to add a new sub-command. I'm currently thinking `memory` to reflect that this command operates on an in-memory database:

    sqlite-utils memory --load-csv blah.csv ""select * from blah""

I still think I need to use `--load-csv` rather than `--csv` because one interesting use-case for this is loading in CSV and converting it to JSON, or vice-versa.

Another option: allow multiple arguments which are filenames, and use the extension (or sniff the content) to decide what to do with them:

    sqlite-utils memory blah.csv foo.csv ""select * from foo join blah on ...""

This would require the last positional argument to always be a SQL query, and would treat all other positional arguments as files that should be imported into memory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861944202,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861944202,MDEyOklzc3VlQ29tbWVudDg2MTk0NDIwMg==,25778,eyeseast,2021-06-16T01:41:03Z,2021-06-16T01:41:03Z,CONTRIBUTOR,"So, I do things like this a lot, too. I like the idea of piping in from stdin. Something like this would be nice to do in a makefile:

```sh
cat file.csv | sqlite-utils --csv --table data - 'SELECT * FROM data WHERE col=""whatever""' > filtered.csv
```

If you assumed that you're always piping out the same format you're piping in, the option names don't have to change. Depends how much you want to change formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891835,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891835,MDEyOklzc3VlQ29tbWVudDg2MTg5MTgzNQ==,9599,simonw,2021-06-15T23:09:31Z,2021-06-15T23:09:31Z,OWNER,`--load-csv` and `--load-json` and `--load-nl` and `--load-tsv` are unambiguous.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891693,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891693,MDEyOklzc3VlQ29tbWVudDg2MTg5MTY5Mw==,9599,simonw,2021-06-15T23:09:08Z,2021-06-15T23:09:08Z,OWNER,Problem: `--csv` and `--json` and `--nl` are already options for `sqlite-utils query` - need new non-conflicting names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891272,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891272,MDEyOklzc3VlQ29tbWVudDg2MTg5MTI3Mg==,9599,simonw,2021-06-15T23:08:02Z,2021-06-15T23:08:02Z,OWNER,"`--csv -` should work though, for reading from stdin. The table can be called `stdin`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891110,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861891110,MDEyOklzc3VlQ29tbWVudDg2MTg5MTExMA==,9599,simonw,2021-06-15T23:07:38Z,2021-06-15T23:07:38Z,OWNER,`--csvt` seems unnecessary to me: if people want to load different CSV files with the same filename (but in different directories) they will get an error unless they rename the files first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861890689,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861890689,MDEyOklzc3VlQ29tbWVudDg2MTg5MDY4OQ==,9599,simonw,2021-06-15T23:06:37Z,2021-06-15T23:06:37Z,OWNER,"How about `--json` and `--nl` and `--tsv` too? Imitating the format options for `sqlite-utils insert`.

And what happens if you provide a filename too? I'm tempted to say that the `--csv` stuff still gets loaded into an in-memory database but it's given a name and can then be joined against using SQLite `memory.blah` syntax.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861889437,https://api.github.com/repos/simonw/sqlite-utils/issues/272,861889437,MDEyOklzc3VlQ29tbWVudDg2MTg4OTQzNw==,9599,simonw,2021-06-15T23:03:26Z,2021-06-15T23:03:26Z,OWNER,Maybe also support `--csvt` as an alternative option which takes two arguments: the CSV path and the name of the table that should be created from it (rather than auto-detecting from the filename).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",921878733,"Idea: import CSV to memory, run SQL, export in a single command",
https://github.com/simonw/datasette/pull/1130#issuecomment-861497548,https://api.github.com/repos/simonw/datasette/issues/1130,861497548,MDEyOklzc3VlQ29tbWVudDg2MTQ5NzU0OA==,3243482,abdusco,2021-06-15T13:27:48Z,2021-06-15T13:27:48Z,CONTRIBUTOR,"There's a workaround: https://css-tricks.com/css-fix-for-100vh-in-mobile-webkit/

and a future fix: https://css-tricks.com/safari-15-new-ui-theme-colors-and-a-css-tricks-cameo/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756876238,Fix footer not sticking to bottom in short pages,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-861103967,https://api.github.com/repos/simonw/sqlite-utils/issues/269,861103967,MDEyOklzc3VlQ29tbWVudDg2MTEwMzk2Nw==,9599,simonw,2021-06-15T01:34:10Z,2021-06-15T01:34:10Z,OWNER,"SQLite doesn't have the concept of a boolean column, so there's not much I can do here: https://www.sqlite.org/datatype3.html#boolean_datatype","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-861103684,https://api.github.com/repos/simonw/sqlite-utils/issues/266,861103684,MDEyOklzc3VlQ29tbWVudDg2MTEwMzY4NA==,9599,simonw,2021-06-15T01:33:13Z,2021-06-15T01:33:13Z,OWNER,Dupe of #37,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1377#issuecomment-861089794,https://api.github.com/repos/simonw/datasette/issues/1377,861089794,MDEyOklzc3VlQ29tbWVudDg2MTA4OTc5NA==,9599,simonw,2021-06-15T00:53:29Z,2021-06-15T00:53:29Z,OWNER,"Potential hook names:

- `skip_csrf(scope, datasette)`
- ... I can't think of any other ones I would tolerate to be honest","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/simonw/datasette/issues/1377#issuecomment-861087949,https://api.github.com/repos/simonw/datasette/issues/1377,861087949,MDEyOklzc3VlQ29tbWVudDg2MTA4Nzk0OQ==,9599,simonw,2021-06-15T00:49:19Z,2021-06-15T00:49:19Z,OWNER,"The new `skip_if_scope` mechanism in `asgi-csrf` https://github.com/simonw/asgi-csrf/issues/20 is designed to help here.

Now I need to design a plugin hook that allows plugins to have an opinion on whether a specific `scope` should have CSRF skipped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920884085,Mechanism for plugins to exclude certain paths from CSRF checks,
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861087651,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861087651,MDEyOklzc3VlQ29tbWVudDg2MTA4NzY1MQ==,231498,khimaros,2021-06-15T00:48:37Z,2021-06-15T00:48:37Z,NONE,"@simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user.

found by poking around in the ""Explorer"" tab in https://docs.github.com/en/graphql/overview/explorer

note: pagination is still required via `first` and `last` but it seems to allow unlimited history.

```
query MyQuery {
  __typename
  user(login: ""<user>"") {
    id
    pinnedItems(first: 100) {
      edges {
        node
      }
    }
    pullRequests(first: 100) {
      nodes {
        body
        title
        state
        createdAt
      }
    }
    createdAt
    issues(first: 100) {
      pageInfo {
        endCursor
        startCursor
      }
      nodes {
        title
        url
        createdAt
        body
      }
    }
    issueComments(first: 100) {
      edges {
        node {
          id
          updatedAt
          url
          body
        }
      }
    }
    repositories(first: 100) {
      nodes {
        createdAt
        description
        parent {
          name
        }
        pinnedIssues(first: 100) {
          edges {
            node {
              id
            }
          }
        }
        pinnedDiscussions(first: 100) {
          edges {
            node {
              id
            }
          }
        }
      }
    }
    starredRepositories(first: 100) {
      edges {
        node {
          id
        }
      }
    }
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861042050,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861042050,MDEyOklzc3VlQ29tbWVudDg2MTA0MjA1MA==,9599,simonw,2021-06-14T22:45:42Z,2021-06-14T22:45:42Z,MEMBER,I'm definitely interested in supporting events in this tool - see #14.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861041597,MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw==,9599,simonw,2021-06-14T22:44:54Z,2021-06-14T22:44:54Z,MEMBER,Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861035862,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,861035862,MDEyOklzc3VlQ29tbWVudDg2MTAzNTg2Mg==,231498,khimaros,2021-06-14T22:29:20Z,2021-06-14T22:29:20Z,NONE,"it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub.

this is significant change, but may be worth considering in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-860895838,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64,860895838,MDEyOklzc3VlQ29tbWVudDg2MDg5NTgzOA==,231498,khimaros,2021-06-14T18:23:21Z,2021-06-14T21:37:35Z,NONE,"i have a basic working version at https://github.com/khimaros/github-to-sqlite

this can be tested with `github-to-sqlite events.db khimaros/events`

caveat: the GitHub API doesn't seem to provide a complete history of events.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",920636216,"feature: support ""events""",
https://github.com/simonw/datasette/issues/1375#issuecomment-860548546,https://api.github.com/repos/simonw/datasette/issues/1375,860548546,MDEyOklzc3VlQ29tbWVudDg2MDU0ODU0Ng==,4068,frafra,2021-06-14T09:41:59Z,2021-06-14T09:41:59Z,NONE,"> There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
> Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

Thanks :)
 
> I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.

If a developer is not sure if the JSON fields are valid, but then retrieves and parse them, it should handle errors too. Handling inconsistent data is necessary due to the nature of SQLite. A global or dataset option to render the data as they have been defined (JSON, boolean, etc.) when requesting JSON could allow the user to download a regular JSON from the browser without having to rely on APIs. I would guess someone could just make a custom template with an extra JSON-parsed download button otherwise :)","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919508498,JSON export dumps JSON fields as TEXT,
https://github.com/simonw/datasette/issues/1376#issuecomment-860230663,https://api.github.com/repos/simonw/datasette/issues/1376,860230663,MDEyOklzc3VlQ29tbWVudDg2MDIzMDY2Mw==,9599,simonw,2021-06-13T15:39:37Z,2021-06-13T15:39:37Z,OWNER,Actually it looks like there is a PR open already that addresses this: #1296 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/datasette/issues/1375#issuecomment-860230385,https://api.github.com/repos/simonw/datasette/issues/1375,860230385,MDEyOklzc3VlQ29tbWVudDg2MDIzMDM4NQ==,9599,simonw,2021-06-13T15:37:49Z,2021-06-13T15:37:49Z,OWNER,"There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning  string instead of the expected dictionary and breaking your code.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919508498,JSON export dumps JSON fields as TEXT,
https://github.com/simonw/datasette/pull/1373#issuecomment-857684605,https://api.github.com/repos/simonw/datasette/issues/1373,857684605,MDEyOklzc3VlQ29tbWVudDg1NzY4NDYwNQ==,22429695,codecov[bot],2021-06-09T13:15:31Z,2021-06-13T15:34:07Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1373](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (d117ba7) into [main](https://codecov.io/gh/simonw/datasette/commit/03418ee037057aa85204f5a3feb2066cbb6a9b3e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (03418ee) will **increase** coverage by `7.65%`.
> The diff coverage is `93.29%`.

> :exclamation: Current head d117ba7 differs from pull request most recent head 51ff366. Consider uploading reports for the commit 51ff366 to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1373/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main    #1373      +/-   ##
==========================================
+ Coverage   84.02%   91.68%   +7.65%     
==========================================
  Files          28       34       +6     
  Lines        3774     4340     +566     
==========================================
+ Hits         3171     3979     +808     
+ Misses        603      361     -242     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <ø> (ø)` | |
| [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
| [datasette/publish/common.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY29tbW9uLnB5) | `94.73% <ø> (ø)` | |
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `76.33% <70.64%> (+1.19%)` | :arrow_up: |
| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <77.77%> (ø)` | |
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <89.28%> (ø)` | |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `90.98% <92.00%> (-0.95%)` | :arrow_down: |
| [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.73% <94.44%> (+1.13%)` | :arrow_up: |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.73% <94.52%> (-0.27%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.36% <94.53%> (+0.55%)` | :arrow_up: |
| ... and [29 more](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e797565...51ff366](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",916183914,"Update trustme requirement from <0.8,>=0.7 to >=0.7,<0.9",
https://github.com/simonw/datasette/issues/1376#issuecomment-860229397,https://api.github.com/repos/simonw/datasette/issues/1376,860229397,MDEyOklzc3VlQ29tbWVudDg2MDIyOTM5Nw==,9599,simonw,2021-06-13T15:31:02Z,2021-06-13T15:31:02Z,OWNER,Alternative fix would be to update that section of the documentation - if the container upgrade proves tricky I can fall back on that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/datasette/issues/1376#issuecomment-860229226,https://api.github.com/repos/simonw/datasette/issues/1376,860229226,MDEyOklzc3VlQ29tbWVudDg2MDIyOTIyNg==,9599,simonw,2021-06-13T15:29:45Z,2021-06-13T15:29:45Z,OWNER,"Oh good catch - this is a SQLite version issue.

The `fixtures.db` file used on https://latest.datasette.io/ includes a generated column (for testing purposes) which is a feature added in SQLite 3.31.0 on 2020-01-22.

https://latest.datasette.io/-/versions

But... it looks like the packaged Datasette Docker container doesn't have that SQLite version!

I should fix that. I'm renaming this issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919822817,Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns),
https://github.com/simonw/sqlite-utils/issues/271#issuecomment-860142489,https://api.github.com/repos/simonw/sqlite-utils/issues/271,860142489,MDEyOklzc3VlQ29tbWVudDg2MDE0MjQ4OQ==,9599,simonw,2021-06-13T02:53:06Z,2021-06-13T02:53:06Z,OWNER,"Looks like this is the problem: https://github.com/simonw/sqlite-utils/blob/b0f9d1e494c9891ce407e27b0f5c6deeea361d30/sqlite_utils/db.py#L1724-L1742

Note how `set_cols = [col for col in all_columns if col not in pks] ` can potentially return an empty list if ALL of the columns are primary keys - but the next line of code that assigns `sql2` continues regardless, when it should instead be skipped if there are no columns in `set_cols`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919702451,table.upsert_all() fails if input has a single column that should be a primary key,
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-860063190,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57,860063190,MDEyOklzc3VlQ29tbWVudDg2MDA2MzE5MA==,232237,stiles,2021-06-12T14:46:44Z,2021-06-12T14:46:44Z,NONE,I'm having the same issue (same versions of python and twitter-to-sqlite). It's the `user-timeline` command. Other commands are working. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907645813,"Error: Use either --since or --since_id, not both",
https://github.com/simonw/datasette/issues/1286#issuecomment-860047794,https://api.github.com/repos/simonw/datasette/issues/1286,860047794,MDEyOklzc3VlQ29tbWVudDg2MDA0Nzc5NA==,4068,frafra,2021-06-12T12:36:15Z,2021-06-12T12:36:15Z,NONE,"@mroswell That is a very nice solution. I wonder if custom classes, like `col-columnName-value` could be automatically added to cells when facets on such column are enabled, to allow custom styling without having to modify templates or add custom JavaScript code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849220154,Better default display of arrays of items,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-860031217,https://api.github.com/repos/simonw/sqlite-utils/issues/269,860031217,MDEyOklzc3VlQ29tbWVudDg2MDAzMTIxNw==,4068,frafra,2021-06-12T10:01:53Z,2021-06-12T10:01:53Z,NONE,"`sqlite-utils transform` does not allow setting the column type to boolean:
```
Error: Invalid value for '--type': 'bool' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-860031071,https://api.github.com/repos/simonw/sqlite-utils/issues/270,860031071,MDEyOklzc3VlQ29tbWVudDg2MDAzMTA3MQ==,4068,frafra,2021-06-12T10:00:24Z,2021-06-12T10:00:24Z,NONE,"Sure, I am sorry if my message hasn't been clear enough. I am also a new user :)

At the beginning, I just call `sqlite-utils insert ""$db"" ""$table"" ""$jsonfile""` to create the database. sqlite-utils convert JSON values into `TEXT`, when it tries to determine the schema automatically. I then try to transform the table to set `JSON` as type:

```
sqlite-utils transform species.sqlite species --type criteria json
Usage: sqlite-utils transform [OPTIONS] PATH TABLE
Try 'sqlite-utils transform --help' for help.

Error: Invalid value for '--type': 'json' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-859986489,https://api.github.com/repos/simonw/sqlite-utils/issues/270,859986489,MDEyOklzc3VlQ29tbWVudDg1OTk4NjQ4OQ==,9599,simonw,2021-06-12T02:47:12Z,2021-06-12T02:47:12Z,OWNER,"Can you expand on what you'd like to change here? The library and CLI tool already allow JSON data to be stored in columns:

- https://sqlite-utils.datasette.io/en/stable/cli.html#nested-json-values
- https://sqlite-utils.datasette.io/en/stable/python-api.html#storing-json","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919314806,Cannot set type JSON,
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-859940977,https://api.github.com/repos/simonw/sqlite-utils/issues/269,859940977,MDEyOklzc3VlQ29tbWVudDg1OTk0MDk3Nw==,4068,frafra,2021-06-11T22:33:08Z,2021-06-11T22:33:08Z,NONE,"`true` and `false` json values are cast to integer, which is not optimal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919250621,bool type not supported,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859898736,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859898736,MDEyOklzc3VlQ29tbWVudDg1OTg5ODczNg==,9599,simonw,2021-06-11T20:37:44Z,2021-06-11T20:37:44Z,OWNER,"From the prototype:
```
% sqlite-utils schema 24ways.db
CREATE TABLE [articles] (
               [title] TEXT  ,
   [contents] TEXT  ,
   [year] TEXT  ,
   [author] TEXT  ,
   [author_slug] TEXT  ,
   [published] TEXT  ,
   [url] TEXT  ,
   [topic] TEXT  
        );
CREATE VIRTUAL TABLE ""articles_fts"" USING FTS5 (
                title, author, contents,
                content=""articles""
            );
CREATE TABLE 'articles_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'articles_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE 'articles_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE 'articles_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
% sqlite-utils schema 24ways.db | sqlite3 /tmp/boo.db
Error: near line 15: table 'articles_fts_data' already exists
Error: near line 16: table 'articles_fts_idx' already exists
Error: near line 17: table 'articles_fts_docsize' already exists
Error: near line 18: table 'articles_fts_config' already exists
```
The problem here is that the `CREATE VIRTUAL TABLE ""articles_fts""...` line causes those next four tables to be created - but that means that piping the output of this command into `sqlite3` in order to re-create those tables throws errors.

I don't think this matters. I see this tool as more for introspection than for recreating table structures.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859895540,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859895540,MDEyOklzc3VlQ29tbWVudDg1OTg5NTU0MA==,9599,simonw,2021-06-11T20:30:34Z,2021-06-11T20:30:34Z,OWNER,"You can currently see the `sql` on the CLI using:

    % sqlite-utils rows fixtures.db sqlite_master -c name -c sql
    name                                                            sql
    --------------------------------------------------------------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------
    simple_primary_key                                              CREATE TABLE simple_primary_key (
                                                                      id varchar(30) primary key,
                                                                      content text
                                                                    )
    sqlite_autoindex_simple_primary_key_1
    primary_key_multiple_columns                                    CREATE TABLE primary_key_multiple_columns (
                                                                      id varchar(30) primary key,
                                                                      content text,
                                                                      content2 text
                                                                    )
    sqlite_autoindex_primary_key_multiple_columns_1
    primary_key_multiple_columns_explicit_label                     CREATE TABLE primary_key_multiple_columns_explicit_label (
                                                                      id varchar(30) primary key,
                                                                      content text,
                                                                      content2 text
                                                                    )
    sqlite_autoindex_primary_key_multiple_columns_explicit_label_1
    compound_primary_key                                            CREATE TABLE compound_primary_key (
                                                                      pk1 varchar(30),
                                                                      pk2 varchar(30),
                                                                      content text,
                                                                      PRIMARY KEY (pk1, pk2)
                                                                    )
    sqlite_autoindex_compound_primary_key_1
    compound_three_primary_keys                                     CREATE TABLE compound_three_primary_keys (
                                                                      pk1 varchar(30),
                                                                      pk2 varchar(30),
                                                                      pk3 varchar(30),
                                                                      content text,
                                                                      PRIMARY KEY (pk1, pk2, pk3)
                                                                    )
    sqlite_autoindex_compound_three_primary_keys_1
    foreign_key_references                                          CREATE TABLE foreign_key_references (
                                                                      pk varchar(30) primary key,
                                                                      foreign_key_with_label varchar(30),
                                                                      foreign_key_with_no_label varchar(30),
                                                                      FOREIGN KEY (foreign_key_with_label) REFERENCES simple_primary_key(id),
                                                                      FOREIGN KEY (foreign_key_with_no_label) REFERENCES primary_key_multiple_columns(id)
                                                                    )
    sqlite_autoindex_foreign_key_references_1
    sortable                                                        CREATE TABLE sortable (
                                                                      pk1 varchar(30),
                                                                      pk2 varchar(30),
                                                                      content text,
                                                                      sortable integer,
                                                                      sortable_with_nulls real,
                                                                      sortable_with_nulls_2 real,
                                                                      text text,
                                                                      PRIMARY KEY (pk1, pk2)
                                                                    )
    sqlite_autoindex_sortable_1
    no_primary_key                                                  CREATE TABLE no_primary_key (
                                                                      content text,
                                                                      a text,
                                                                      b text,
                                                                      c text
                                                                    )
    123_starts_with_digits                                          CREATE TABLE [123_starts_with_digits] (
                                                                      content text
                                                                    )
    paginated_view                                                  CREATE VIEW paginated_view AS
                                                                        SELECT
                                                                            content,
                                                                            '- ' || content || ' -' AS content_extra
                                                                        FROM no_primary_key
    Table With Space In Name                                        CREATE TABLE ""Table With Space In Name"" (
                                                                      pk varchar(30) primary key,
                                                                      content text
                                                                    )
    sqlite_autoindex_Table With Space In Name_1
    table/with/slashes.csv                                          CREATE TABLE ""table/with/slashes.csv"" (
                                                                      pk varchar(30) primary key,
                                                                      content text
                                                                    )
    sqlite_autoindex_table/with/slashes.csv_1
    complex_foreign_keys                                            CREATE TABLE ""complex_foreign_keys"" (
                                                                      pk varchar(30) primary key,
                                                                      f1 text,
                                                                      f2 text,
                                                                      f3 text,
                                                                      FOREIGN KEY (""f1"") REFERENCES [simple_primary_key](id),
                                                                      FOREIGN KEY (""f2"") REFERENCES [simple_primary_key](id),
                                                                      FOREIGN KEY (""f3"") REFERENCES [simple_primary_key](id)
                                                                    )
    sqlite_autoindex_complex_foreign_keys_1
    custom_foreign_key_label                                        CREATE TABLE ""custom_foreign_key_label"" (
                                                                      pk varchar(30) primary key,
                                                                      foreign_key_with_custom_label text,
                                                                      FOREIGN KEY (""foreign_key_with_custom_label"") REFERENCES [primary_key_multiple_columns_explicit_label](id)
                                                                    )
    sqlite_autoindex_custom_foreign_key_label_1
    units                                                           CREATE TABLE units (
                                                                      pk integer primary key,
                                                                      distance int,
                                                                      frequency int
                                                                    )
    searchable                                                      CREATE TABLE searchable (
                                                                      pk integer primary key,
                                                                      text1 text,
                                                                      text2 text,
                                                                      [name with . and spaces] text
                                                                    )
    searchable_fts                                                  CREATE VIRTUAL TABLE ""searchable_fts""
                                                                        USING FTS3 (text1, text2, [name with . and spaces], content=""searchable"")
    searchable_fts_content                                          CREATE TABLE 'searchable_fts_content'(docid INTEGER PRIMARY KEY, 'c0text1', 'c1text2', 'c2name with . and spaces', 'c3content')
    searchable_fts_segments                                         CREATE TABLE 'searchable_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)
    searchable_fts_segdir                                           CREATE TABLE 'searchable_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))
    sqlite_autoindex_searchable_fts_segdir_1
    select                                                          CREATE TABLE [select] (
                                                                      [group] text,
                                                                      [having] text,
                                                                      [and] text
                                                                    )
    facet_cities                                                    CREATE TABLE facet_cities (
                                                                        id integer primary key,
                                                                        name text
                                                                    )
    simple_view                                                     CREATE VIEW simple_view AS
                                                                        SELECT content, upper(content) AS upper_content FROM simple_primary_key
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859894105,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859894105,MDEyOklzc3VlQ29tbWVudDg1OTg5NDEwNQ==,9599,simonw,2021-06-11T20:28:52Z,2021-06-11T20:28:52Z,OWNER,"Out of interest, here are the rows from that table where `sql` is `null`: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++sqlite_master%0D%0Awhere%0D%0A++sql+is+null

```csv
type,name,tbl_name,rootpage,sql
index,sqlite_autoindex_simple_primary_key_1,simple_primary_key,3,
index,sqlite_autoindex_primary_key_multiple_columns_1,primary_key_multiple_columns,5,
index,sqlite_autoindex_primary_key_multiple_columns_explicit_label_1,primary_key_multiple_columns_explicit_label,7,
index,sqlite_autoindex_compound_primary_key_1,compound_primary_key,9,
index,sqlite_autoindex_compound_three_primary_keys_1,compound_three_primary_keys,11,
index,sqlite_autoindex_foreign_key_references_1,foreign_key_references,14,
index,sqlite_autoindex_sortable_1,sortable,16,
index,sqlite_autoindex_Table With Space In Name_1,Table With Space In Name,20,
index,sqlite_autoindex_table/with/slashes.csv_1,table/with/slashes.csv,22,
index,sqlite_autoindex_complex_foreign_keys_1,complex_foreign_keys,24,
index,sqlite_autoindex_custom_foreign_key_label_1,custom_foreign_key_label,26,
index,sqlite_autoindex_tags_1,tags,31,
index,sqlite_autoindex_searchable_tags_1,searchable_tags,34,
index,sqlite_autoindex_searchable_fts_segdir_1,searchable_fts_segdir,37,
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859888469,https://api.github.com/repos/simonw/sqlite-utils/issues/268,859888469,MDEyOklzc3VlQ29tbWVudDg1OTg4ODQ2OQ==,9599,simonw,2021-06-11T20:26:20Z,2021-06-11T20:26:20Z,OWNER,`sqlite-utils schema data.db` could output the same thing to the console.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",919181559,db.schema property and sqlite-utils schema command,
https://github.com/simonw/datasette/pull/1374#issuecomment-859572791,https://api.github.com/repos/simonw/datasette/issues/1374,859572791,MDEyOklzc3VlQ29tbWVudDg1OTU3Mjc5MQ==,22429695,codecov[bot],2021-06-11T13:12:58Z,2021-06-11T13:12:58Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1374](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0ef0dd5) into [main](https://codecov.io/gh/simonw/datasette/commit/cd7678fde65319d7b6955ce9f4678ba4b9e64b66?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cd7678f) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1374/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1374   +/-   ##
=======================================
  Coverage   91.68%   91.68%           
=======================================
  Files          34       34           
  Lines        4340     4340           
=======================================
  Hits         3979     3979           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [cd7678f...0ef0dd5](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",918730335,Bump black from 21.5b2 to 21.6b0,
https://github.com/simonw/datasette/issues/858#issuecomment-858831895,https://api.github.com/repos/simonw/datasette/issues/858,858831895,MDEyOklzc3VlQ29tbWVudDg1ODgzMTg5NQ==,56045588,rachelmarconi,2021-06-10T17:44:09Z,2021-06-10T17:44:09Z,NONE,"any fixes for that recursive issue with temp file? I get it using both heroku and cloudrun, although it seems to still publish and deploy fine","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/issues/858#issuecomment-858813675,https://api.github.com/repos/simonw/datasette/issues/858,858813675,MDEyOklzc3VlQ29tbWVudDg1ODgxMzY3NQ==,56045588,rachelmarconi,2021-06-10T17:27:46Z,2021-06-10T17:27:46Z,NONE,shell=True is added to line 56 (I guess it used to be 54) of heroku.py as detailed in the original issue. (for posterity),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/issues/1371#issuecomment-858099514,https://api.github.com/repos/simonw/datasette/issues/1371,858099514,MDEyOklzc3VlQ29tbWVudDg1ODA5OTUxNA==,9599,simonw,2021-06-09T21:03:49Z,2021-06-09T21:03:49Z,OWNER,I'll release these as an alpha straight away - it makes sense to have plugin hook changes available for people to test as alpha dependencies ASAP.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",915455228,Menu plugin hooks should include the request,
https://github.com/simonw/datasette/pull/1370#issuecomment-857298526,https://api.github.com/repos/simonw/datasette/issues/1370,857298526,MDEyOklzc3VlQ29tbWVudDg1NzI5ODUyNg==,25778,eyeseast,2021-06-09T01:18:59Z,2021-06-09T01:18:59Z,CONTRIBUTOR,"I'm happy to grab some or all of these in this PR, if you want. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",914130834,Ensure db.path is a string before trying to insert into internal database,
https://github.com/simonw/datasette/pull/1370#issuecomment-857139881,https://api.github.com/repos/simonw/datasette/issues/1370,857139881,MDEyOklzc3VlQ29tbWVudDg1NzEzOTg4MQ==,9599,simonw,2021-06-08T20:58:41Z,2021-06-08T20:58:41Z,OWNER,We can remove a bunch of unnecessary `str(path)` calls too - this search finds a bunch of possible candidates: https://ripgrep.datasette.io/-/ripgrep?pattern=str%5C%28.*%28db%7Cpath%29&glob=datasette%2F**%2F*.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",914130834,Ensure db.path is a string before trying to insert into internal database,
https://github.com/simonw/datasette/pull/1368#issuecomment-856182547,https://api.github.com/repos/simonw/datasette/issues/1368,856182547,MDEyOklzc3VlQ29tbWVudDg1NjE4MjU0Nw==,2670795,brandonrobertz,2021-06-07T18:59:47Z,2021-06-07T23:04:25Z,CONTRIBUTOR,"Note that if we went with a ""update_metadata"" hook, the hook signature would look something like this (it would return nothing):

```
update_metadata(
  datasette=self, metadata=metadata, key=key, database=database, table=table,
  fallback=fallback
)
```

The Datasette function `_metadata_recursive_update(self, orig, updated)` would disappear into the plugins. Doing this, though, we'd lose the easy ability to make the local metadata.yaml immutable (since we'd no longer have the recursive update).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913865304,DRAFT: A new plugin hook for dynamic metadata,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-856231119,https://api.github.com/repos/simonw/sqlite-utils/issues/266,856231119,MDEyOklzc3VlQ29tbWVudDg1NjIzMTExOQ==,9599,simonw,2021-06-07T20:26:05Z,2021-06-07T20:26:05Z,OWNER,"https://github.com/python/cpython/blob/2ab27c4af4ddf7528e1375e77c787c7fbb09b5e6/Lib/typing.py#L2173-L2195

In Python 3.6 or higher can do this:

```python
class Employee(NamedTuple):
    name: str
    id: int
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1365#issuecomment-856212136,https://api.github.com/repos/simonw/datasette/issues/1365,856212136,MDEyOklzc3VlQ29tbWVudDg1NjIxMjEzNg==,9599,simonw,2021-06-07T19:54:04Z,2021-06-07T19:54:04Z,OWNER,"I've hit this one too. I agree, fixing this in Datasette itself is better than fixing it in the tests across multiple other projects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913017577,pathlib.Path breaks internal schema,
https://github.com/simonw/datasette/issues/1369#issuecomment-856208637,https://api.github.com/repos/simonw/datasette/issues/1369,856208637,MDEyOklzc3VlQ29tbWVudDg1NjIwODYzNw==,9599,simonw,2021-06-07T19:47:23Z,2021-06-07T19:47:23Z,OWNER,No point in showing the IDs twice if the blue label doesn't differ from the gray ID,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913900374,Don't show foreign key IDs twice if no label,
https://github.com/simonw/datasette/issues/1367#issuecomment-856160770,https://api.github.com/repos/simonw/datasette/issues/1367,856160770,MDEyOklzc3VlQ29tbWVudDg1NjE2MDc3MA==,9599,simonw,2021-06-07T18:22:33Z,2021-06-07T18:22:33Z,OWNER,Here's why: https://github.com/simonw/datasette/blob/03ec71193b9545536898a4bc7493274fec48bdd7/datasette/static/app.css#L455-L458,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913823889,Navigation menu display bug,
https://github.com/simonw/datasette/issues/1366#issuecomment-856147969,https://api.github.com/repos/simonw/datasette/issues/1366,856147969,MDEyOklzc3VlQ29tbWVudDg1NjE0Nzk2OQ==,9599,simonw,2021-06-07T18:03:03Z,2021-06-07T18:03:03Z,OWNER,"Here's an example of a test that uses it. It's necessary because sometimes fixtures that create temporary directories break in unexpected ways:

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_plugins.py#L658-L666","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913809802,Get rid of this `restore_working_directory` hack entirely,
https://github.com/simonw/datasette/issues/1366#issuecomment-856147450,https://api.github.com/repos/simonw/datasette/issues/1366,856147450,MDEyOklzc3VlQ29tbWVudDg1NjE0NzQ1MA==,9599,simonw,2021-06-07T18:02:13Z,2021-06-07T18:02:13Z,OWNER,"The hack in question is this fixture, which I've been using in an ad-hoc manner to work around errors while running the tests: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/tests/conftest.py#L62-L75

I don't understand the underlying issue well enough to know how to get rid of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913809802,Get rid of this `restore_working_directory` hack entirely,
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-855611939,https://api.github.com/repos/simonw/sqlite-utils/issues/266,855611939,MDEyOklzc3VlQ29tbWVudDg1NTYxMTkzOQ==,9599,simonw,2021-06-07T06:07:41Z,2021-06-07T06:07:41Z,OWNER,"Looks like this is the way to do this:
```python
Point = typing.NamedTuple(
    ""Point"", [('x', int), ('y', int)]
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",913135723,"Add some types, enforce with mypy",
https://github.com/simonw/datasette/issues/1362#issuecomment-855430317,https://api.github.com/repos/simonw/datasette/issues/1362,855430317,MDEyOklzc3VlQ29tbWVudDg1NTQzMDMxNw==,9599,simonw,2021-06-06T17:07:48Z,2021-06-06T17:07:48Z,OWNER,"I guess I can offer a `disable_csp` setting so that people with complex custom templates aren't completely blocked from using them with Datasette, but maybe it would be better not to offer that? Or to offer it as a `datasette-insecure-csp` plugin instead?

I like the idea of very actively encouraging CSP across all Datasette projects, but I'm nervous about making the software unusable for certain edge cases.

Maybe require CSP and wait for someone to complain?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855429111,https://api.github.com/repos/simonw/datasette/issues/1362,855429111,MDEyOklzc3VlQ29tbWVudDg1NTQyOTExMQ==,9599,simonw,2021-06-06T16:59:05Z,2021-06-06T17:00:15Z,OWNER,"Twitter conversation: https://twitter.com/simonw/status/1401565566045806594

@dracos provided some really useful code examples there:

> We generate it here: https://github.com/mysociety/fixmystreet/blob/e9fec4e567e7148ed128816e5770c2963be51af6/perllib/FixMyStreet/Cobrand/Default.pm#L89-L90
And use it e.g. https://github.com/mysociety/fixmystreet/blob/ba6788cd25d8f471a4e3308403607627b4d2f4f6/templates/web/base/common_header_tags.html or https://github.com/mysociety/fixmystreet/blob/cb4f2b96364d151988b5c664888468b25cc62240/templates/web/fixmystreet.com/header/css.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855428601,https://api.github.com/repos/simonw/datasette/issues/1362,855428601,MDEyOklzc3VlQ29tbWVudDg1NTQyODYwMQ==,9599,simonw,2021-06-06T16:55:33Z,2021-06-06T16:55:33Z,OWNER,"> No, because Vary header is about _request_ headers that cause the response to vary, not response headers.

Hah, of course! Thanks for the correction. So the nonce mechanism would actually be pretty great here, especially for the `extra_body_script()` hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855428296,https://api.github.com/repos/simonw/datasette/issues/1362,855428296,MDEyOklzc3VlQ29tbWVudDg1NTQyODI5Ng==,154364,dracos,2021-06-06T16:53:20Z,2021-06-06T16:53:20Z,NONE,"> Presumably this would also require adding Content-Security-Policy to the Vary header though, which will have a nasty effect on Cloudflare and Fastly and such like.

No, because Vary header is about *request* headers that cause the response to vary, not response headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855427396,https://api.github.com/repos/simonw/datasette/issues/1362,855427396,MDEyOklzc3VlQ29tbWVudDg1NTQyNzM5Ng==,9599,simonw,2021-06-06T16:46:17Z,2021-06-06T16:46:17Z,OWNER,"Mind you, since that plugin hook looks like this:

```python
@hookimpl
def extra_body_script():
    return {
        ""module"": True,
        ""script"": ""console.log('Your JavaScript goes here...')""
    }
```
Having it calculate a sha256 hash wouldn't be difficult.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426750,https://api.github.com/repos/simonw/datasette/issues/1362,855426750,MDEyOklzc3VlQ29tbWVudDg1NTQyNjc1MA==,9599,simonw,2021-06-06T16:41:30Z,2021-06-06T16:44:49Z,OWNER,"This is from the current `base.html` template: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/base.html#L62-L66

Which includes this: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/_close_open_menus.html#L1-L16

The `body_scripts` bit is for this `extra_body_script` plugin hook, which is the thing that will be the most affected by implementing CSP: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426516,https://api.github.com/repos/simonw/datasette/issues/1362,855426516,MDEyOklzc3VlQ29tbWVudDg1NTQyNjUxNg==,9599,simonw,2021-06-06T16:39:34Z,2021-06-06T16:39:34Z,OWNER,The reason Datasette uses small inline scripts right now is to avoid the overhead of an extra HTTP request for a JavaScript file - but these are both inherently cachable and perform much better under HTTP/2 so that's likely a false optimization.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855426314,https://api.github.com/repos/simonw/datasette/issues/1362,855426314,MDEyOklzc3VlQ29tbWVudDg1NTQyNjMxNA==,9599,simonw,2021-06-06T16:38:04Z,2021-06-06T16:38:04Z,OWNER,"The other option for inline scripts is the CSP nonce:

    Content-Security-Policy: script-src 'nonce-2726c7f26c'

Then:

    <script nonce=""2726c7f26c"">
      var inline = 1;
    </script>

Since an attacker can't guess what the nonce will be it prevents them from injecting their own script block - this seems easier to make available to plugins than a full hashing mechanism, just make `{{ csp_nonce() }}` available to the template.

That template function can then be smart enough to set a flag which Datasette uses to decide if the `script-src 'nonce-2726c7f26c'` policy should be sent or not.

Presumably this would also require adding `Content-Security-Policy` to the `Vary` header though, which will have a nasty effect on Cloudflare and Fastly and such like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418899,https://api.github.com/repos/simonw/datasette/issues/1362,855418899,MDEyOklzc3VlQ29tbWVudDg1NTQxODg5OQ==,9599,simonw,2021-06-06T15:42:55Z,2021-06-06T15:42:55Z,OWNER,Another consideration: testing that this works correctly could require adoption of a real browser test environment (probably Cypress or maybe Playwright) to execute tests that will fail if CSP is violated.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418698,https://api.github.com/repos/simonw/datasette/issues/1362,855418698,MDEyOklzc3VlQ29tbWVudDg1NTQxODY5OA==,9599,simonw,2021-06-06T15:41:24Z,2021-06-06T15:41:24Z,OWNER,"I think the best way to answer these questions is with some prototyping - of both Datasette and some of the existing JavaScript plugins.

I can start with a `datasette-experimental-csp` plugin that sets the header (and could even run an optional report URI mechanism).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418401,https://api.github.com/repos/simonw/datasette/issues/1362,855418401,MDEyOklzc3VlQ29tbWVudDg1NTQxODQwMQ==,9599,simonw,2021-06-06T15:39:38Z,2021-06-06T15:39:38Z,OWNER,"The security benefit of forcing all JavaScript plugins to be written as CSP-friendly external scripts is very compelling though.

Other plugin-heavy ecosystems such as WordPress have suffered greatly from insecurely written plugins - could this be a huge security win for the Datasette ecosystem generally?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/1362#issuecomment-855418065,https://api.github.com/repos/simonw/datasette/issues/1362,855418065,MDEyOklzc3VlQ29tbWVudDg1NTQxODA2NQ==,9599,simonw,2021-06-06T15:37:11Z,2021-06-06T15:37:11Z,OWNER,"The easiest way to apply CSP is to remove all inline `<script>` blocks (Datasette has a few) and instead serve JavaScript as separate linked files.

It's possible to keep inline script blocks by calculating a hash of their content and adding a `Content-Security-Policy: script-src 'sha256-B2yPHKaXnvFWtRChIbabYmUBFZdVfKKXHbWtWidDVF8='` to the policy. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src

This could be achieved with some Django template tricks, but it feels very risky - and done carelessly could end up calculating a hash of a reflected XSS attack!

The biggest challenge I see around here involves plugins and custom templates. Adopting CSP would require plugins to avoid using any inline scripts, instead keeping their entire implementations in `.js` files.

That's maybe not a bad thing, but it represents a big commitment. It would need to be adopted before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912864936,Consider using CSP to protect against future XSS,
https://github.com/simonw/datasette/issues/283#issuecomment-855369819,https://api.github.com/repos/simonw/datasette/issues/283,855369819,MDEyOklzc3VlQ29tbWVudDg1NTM2OTgxOQ==,9599,simonw,2021-06-06T09:40:18Z,2021-06-06T09:40:18Z,OWNER,"> One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`.
> 
> I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue.

That issue is fixed in #1276.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1361#issuecomment-855308811,https://api.github.com/repos/simonw/datasette/issues/1361,855308811,MDEyOklzc3VlQ29tbWVudDg1NTMwODgxMQ==,9599,simonw,2021-06-05T23:16:21Z,2021-06-05T23:16:21Z,OWNER,That seems to have fixed it. I'd love to get rid of this `restore_working_directory` hack entirely.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855307292,https://api.github.com/repos/simonw/datasette/issues/1361,855307292,MDEyOklzc3VlQ29tbWVudDg1NTMwNzI5Mg==,9599,simonw,2021-06-05T22:59:35Z,2021-06-05T22:59:35Z,OWNER,"That broke things. Here's how `pytest-cov` fixed a similar issue: https://github.com/pytest-dev/pytest-cov/commit/7ccb1783bf8290447df58deeb41eae74296a6d9b

See also https://github.com/nedbat/coveragepy/issues/881 and https://github.com/pytest-dev/pytest-cov/issues/306 and https://github.com/nedbat/coveragepy/issues/824","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855306497,https://api.github.com/repos/simonw/datasette/issues/1361,855306497,MDEyOklzc3VlQ29tbWVudDg1NTMwNjQ5Nw==,9599,simonw,2021-06-05T22:51:01Z,2021-06-05T22:51:01Z,OWNER,I'm going to try removing that `restore_working_directory` fixture entirely.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1361#issuecomment-855306347,https://api.github.com/repos/simonw/datasette/issues/1361,855306347,MDEyOklzc3VlQ29tbWVudDg1NTMwNjM0Nw==,9599,simonw,2021-06-05T22:49:30Z,2021-06-05T22:49:30Z,OWNER,"Stack Overflow: https://stackoverflow.com/a/49367679/6083

> The answer was that `os.chdir()` had been set to the deleted directory by accident. The directory was missing, but the error happened (it seems) at the attempt to get it with `os.getcwd()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912485040,Intermittent CI failure: restore_working_directory FileNotFoundError,
https://github.com/simonw/datasette/issues/1360#issuecomment-855303776,https://api.github.com/repos/simonw/datasette/issues/1360,855303776,MDEyOklzc3VlQ29tbWVudDg1NTMwMzc3Ng==,9599,simonw,2021-06-05T22:23:23Z,2021-06-05T22:23:23Z,OWNER,"Worth noting that I found this issue myself, and to my knowledge it has not been uncovered by anyone else prior to the patch being released.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912464443,"Security flaw, to be fixed in 0.56.1 and 0.57",
https://github.com/simonw/datasette/issues/1360#issuecomment-855303649,https://api.github.com/repos/simonw/datasette/issues/1360,855303649,MDEyOklzc3VlQ29tbWVudDg1NTMwMzY0OQ==,9599,simonw,2021-06-05T22:22:06Z,2021-06-05T22:22:06Z,OWNER,I've released fixes in both 0.56.1 and 0.57.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912464443,"Security flaw, to be fixed in 0.56.1 and 0.57",
https://github.com/simonw/datasette/issues/1358#issuecomment-855302339,https://api.github.com/repos/simonw/datasette/issues/1358,855302339,MDEyOklzc3VlQ29tbWVudDg1NTMwMjMzOQ==,9599,simonw,2021-06-05T22:08:16Z,2021-06-05T22:08:16Z,OWNER,"Release notes are in, pushing the release now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/issues/1358#issuecomment-855302320,https://api.github.com/repos/simonw/datasette/issues/1358,855302320,MDEyOklzc3VlQ29tbWVudDg1NTMwMjMyMA==,9599,simonw,2021-06-05T22:08:06Z,2021-06-05T22:08:06Z,OWNER,See #1360 for the security fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/issues/1358#issuecomment-855288228,https://api.github.com/repos/simonw/datasette/issues/1358,855288228,MDEyOklzc3VlQ29tbWVudDg1NTI4ODIyOA==,9599,simonw,2021-06-05T19:57:18Z,2021-06-05T19:57:18Z,OWNER,"There's also a security fix I need to make, so I'm not going to block this on wrapping up the work on the new Docker image testing from #1344 before putting out this release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912418094,Release Datasette 0.57,
https://github.com/simonw/datasette/pull/1291#issuecomment-855287200,https://api.github.com/repos/simonw/datasette/issues/1291,855287200,MDEyOklzc3VlQ29tbWVudDg1NTI4NzIwMA==,9599,simonw,2021-06-05T19:48:36Z,2021-06-05T19:48:36Z,OWNER,"This is great, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849582643,Update docs: explain allow_download setting,
https://github.com/simonw/datasette/pull/1291#issuecomment-812815358,https://api.github.com/repos/simonw/datasette/issues/1291,812815358,MDEyOklzc3VlQ29tbWVudDgxMjgxNTM1OA==,22429695,codecov[bot],2021-04-03T05:32:50Z,2021-06-05T19:48:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1291](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8789157) into [main](https://codecov.io/gh/simonw/datasette/commit/0a7621f96f8ad14da17e7172e8a7bce24ef78966?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0a7621f) will **not change** coverage.
> The diff coverage is `n/a`.

> :exclamation: Current head 8789157 differs from pull request most recent head 9bf089f. Consider uploading reports for the commit 9bf089f to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1291/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1291   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0a7621f...9bf089f](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849582643,Update docs: explain allow_download setting,
https://github.com/simonw/datasette/issues/1356#issuecomment-855282466,https://api.github.com/repos/simonw/datasette/issues/1356,855282466,MDEyOklzc3VlQ29tbWVudDg1NTI4MjQ2Ng==,9599,simonw,2021-06-05T19:05:06Z,2021-06-05T19:05:06Z,OWNER,Yeah that's a good point. I avoided making them sub-commands because `datasette serve` already supports the multitude of other arguments they also need... but actually that was just me being lazy - I can easily share arguments between multiple functions like I do in `sqlite-utils` itself: https://github.com/simonw/sqlite-utils/blob/d1a372b3006e6cf7d2017b3ddc484bf5c033e45d/sqlite_utils/cli.py#L46,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/pull/1357#issuecomment-855281774,https://api.github.com/repos/simonw/datasette/issues/1357,855281774,MDEyOklzc3VlQ29tbWVudDg1NTI4MTc3NA==,22429695,codecov[bot],2021-06-05T18:59:07Z,2021-06-05T18:59:07Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1357](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b27643) into [main](https://codecov.io/gh/simonw/datasette/commit/6e9b07be92905011211d8df7a872fb7c1f2737b2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (6e9b07b) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1357/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1357   +/-   ##
=======================================
  Coverage   91.71%   91.72%           
=======================================
  Files          34       34           
  Lines        4333     4336    +3     
=======================================
+ Hits         3974     3977    +3     
  Misses        359      359           
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1357/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.38% <ø> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1357/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.71% <100.00%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [6e9b07b...1b27643](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",912394511,Make custom pages compatible with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855278998,https://api.github.com/repos/simonw/datasette/issues/1238,855278998,MDEyOklzc3VlQ29tbWVudDg1NTI3ODk5OA==,9599,simonw,2021-06-05T18:37:16Z,2021-06-05T18:37:16Z,OWNER,"Alternative idea: populate `request.scope` with a new `route_path` which is the base-url-stripped version, which we then use for other routing operations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855278540,https://api.github.com/repos/simonw/datasette/issues/1238,855278540,MDEyOklzc3VlQ29tbWVudDg1NTI3ODU0MA==,9599,simonw,2021-06-05T18:33:25Z,2021-06-05T18:33:25Z,OWNER,"Got the test to pass by ensuring the tests don't accidentally double-rewrite the path.

Ran into a new problem:
```
    @pytest.mark.asyncio
    @pytest.mark.parametrize(
        ""prefix,expected_path"", [(None, ""/asgi-scope""), (""/prefix/"", ""/prefix/asgi-scope"")]
    )
    async def test_client_path(datasette, prefix, expected_path):
        original_base_url = datasette._settings[""base_url""]
        try:
            if prefix is not None:
                datasette._settings[""base_url""] = prefix
            response = await datasette.client.get(""/asgi-scope"")
            path = response.json()[""path""]
>           assert path == expected_path
E           AssertionError: assert '/asgi-scope' == '/prefix/asgi-scope'
E             - /prefix/asgi-scope
E             ? -------
E             + /asgi-scope
```
That test confirms that messing around with the `base_url` doesn't modify the ASGI scope... but the fix I'm using for this issue DOES modify the ASGI scope.

The question raised here is: should the ASGI scope stay unmodified when `base_url` is used?

I think it should. It doesn't make sense to obscure the ""real"" path just to get custom pages to work properly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855272693,https://api.github.com/repos/simonw/datasette/issues/1238,855272693,MDEyOklzc3VlQ29tbWVudDg1NTI3MjY5Mw==,9599,simonw,2021-06-05T17:45:42Z,2021-06-05T17:45:42Z,OWNER,"Applying this fix worked when I manually tested it:
```diff
         base_url = self.ds.setting(""base_url"")
         if base_url != ""/"" and path.startswith(base_url):
             path = ""/"" + path[len(base_url) :]
+            scope = dict(scope, path=path, raw_path=path.encode(""utf-8""))
         request = Request(scope, receive)
```
But... the test I wrote still failed. My hunch is that this is because deep within the test framework requests go through `ds.client` which may be applying its own changes relevant to `base_url`:

https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/utils/testing.py#L139","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-855270917,https://api.github.com/repos/simonw/datasette/issues/1238,855270917,MDEyOklzc3VlQ29tbWVudDg1NTI3MDkxNw==,9599,simonw,2021-06-05T17:32:29Z,2021-06-05T17:32:29Z,OWNER,"This looks like the cause: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/app.py#L1087-L1092

Note how `path` is modified... but then we create a new `Request()` that uses the old scope, which has unmodified `scope[""path""]` - and then the code later on looks at `request.scope[""path""]` when deciding if the request matches:

https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1154-L1155","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1356#issuecomment-853895159,https://api.github.com/repos/simonw/datasette/issues/1356,853895159,MDEyOklzc3VlQ29tbWVudDg1Mzg5NTE1OQ==,25778,eyeseast,2021-06-03T14:03:59Z,2021-06-03T14:03:59Z,CONTRIBUTOR,"(Putting thoughts here to keep the conversation in one place.)

I think using datasette for this use-case is the right approach. I usually have both datasette and sqlite-utils installed in the same project, and that's where I'm trying out queries, so it probably makes the most sense to have datasette also manage the output (and maybe the input, too).

It seems like both `--get` and `--query` could work better as subcommands, rather than options, if you're looking at building out a full CLI experience in datasette. It would give a cleaner separation in what you're trying to do and let each have its own dedicated options. So something like this:

```sh
# run an arbitrary query
datasette query covid.db ""select * from ny_times_us_counties limit 1"" --format yaml

# run a canned query
datasette get covid.db some-canned-query --format yaml
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853567861,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853567861,MDEyOklzc3VlQ29tbWVudDg1MzU2Nzg2MQ==,9599,simonw,2021-06-03T05:12:21Z,2021-06-03T05:12:21Z,OWNER,I think this is more likely to happen in Datasette than in sqlite-utils - see https://github.com/simonw/datasette/issues/1356 for thoughts on this.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/datasette/issues/526#issuecomment-853567413,https://api.github.com/repos/simonw/datasette/issues/526,853567413,MDEyOklzc3VlQ29tbWVudDg1MzU2NzQxMw==,9599,simonw,2021-06-03T05:11:27Z,2021-06-03T05:11:27Z,OWNER,"Another potential way to implement this would be to hold the SQLite connection open and execute the full query there.

I've avoided this in the past due to concerns of resource exhaustion - if multiple requests attempt this at the same time all of the connections in the pool will become tied up and the site will be unable to respond to further requests.

But... now that Datasette has authentication there's the possibility of making this feature only available to specific authenticated users - the `--root` user for example. Which avoids the danger while unlocking a super-useful feature.

Not to mention people who are running Datasette privately on their own laptop, or the proposed `--query` CLI feature in #1356.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries,
https://github.com/simonw/datasette/issues/1356#issuecomment-853566337,https://api.github.com/repos/simonw/datasette/issues/1356,853566337,MDEyOklzc3VlQ29tbWVudDg1MzU2NjMzNw==,9599,simonw,2021-06-03T05:08:32Z,2021-06-03T05:08:32Z,OWNER,Also relevant: CSV streaming for canned queries in #526 - even better if we could stream ANY SQL query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1062#issuecomment-853566011,https://api.github.com/repos/simonw/datasette/issues/1062,853566011,MDEyOklzc3VlQ29tbWVudDg1MzU2NjAxMQ==,9599,simonw,2021-06-03T05:07:42Z,2021-06-03T05:07:42Z,OWNER,Implementing this would make #1356 a whole lot more interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows,
https://github.com/simonw/datasette/issues/1356#issuecomment-853565850,https://api.github.com/repos/simonw/datasette/issues/1356,853565850,MDEyOklzc3VlQ29tbWVudDg1MzU2NTg1MA==,9599,simonw,2021-06-03T05:07:21Z,2021-06-03T05:07:21Z,OWNER,"A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177.

So there's not much point implementing this unless we first make plugins able to add custom streaming formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853562891,https://api.github.com/repos/simonw/datasette/issues/1356,853562891,MDEyOklzc3VlQ29tbWVudDg1MzU2Mjg5MQ==,9599,simonw,2021-06-03T04:59:40Z,2021-06-03T04:59:40Z,OWNER,"It's weird that `--get` is documented on this page right now: https://docs.datasette.io/en/stable/getting_started.html#datasette-get

If I implement this I should build a new ""Datasette on the command-line"" documentation page to cover both `--get` and `--query`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560870,https://api.github.com/repos/simonw/datasette/issues/1356,853560870,MDEyOklzc3VlQ29tbWVudDg1MzU2MDg3MA==,9599,simonw,2021-06-03T04:53:47Z,2021-06-03T04:53:56Z,OWNER,"This is also interesting when used in conjunction with the proposed `datasette insert` command from #1163 - Datasette would become a plugin-driven CLI tool for both ingesting and outputting data, as a side-effect of its web features.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560389,https://api.github.com/repos/simonw/datasette/issues/1356,853560389,MDEyOklzc3VlQ29tbWVudDg1MzU2MDM4OQ==,9599,simonw,2021-06-03T04:52:13Z,2021-06-03T04:52:13Z,OWNER,I should implement #1355 for more efficient `--csv` streaming as part of this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853560237,https://api.github.com/repos/simonw/datasette/issues/1356,853560237,MDEyOklzc3VlQ29tbWVudDg1MzU2MDIzNw==,9599,simonw,2021-06-03T04:51:49Z,2021-06-03T04:51:49Z,OWNER,This feels like a relatively simple feature to implement that unlocks a whole new set of possible uses for Datasette - as described by @eyeseast in https://github.com/simonw/sqlite-utils/issues/264#issue-907642546.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/datasette/issues/1356#issuecomment-853559915,https://api.github.com/repos/simonw/datasette/issues/1356,853559915,MDEyOklzc3VlQ29tbWVudDg1MzU1OTkxNQ==,9599,simonw,2021-06-03T04:50:52Z,2021-06-03T04:50:52Z,OWNER,"What happens if you pass multiple databases? The `--query` would be executed against the first one. And if you pass `--crossdb` it would be executed against the `/_memory` database and would support cross-database joins.

Key thing here is that output plugins are supported (also plugins that add new SQL functions), making many Datasette plugins usable from the command-line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910092577,"Research: syntactic sugar for using --get with SQL queries, maybe ""datasette query""",
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853558741,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853558741,MDEyOklzc3VlQ29tbWVudDg1MzU1ODc0MQ==,9599,simonw,2021-06-03T04:47:19Z,2021-06-03T04:47:19Z,OWNER,"This inspired me to re-examine how `--get` works, hence this issue: https://github.com/simonw/datasette/issues/1355","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/datasette/issues/1355#issuecomment-853557439,https://api.github.com/repos/simonw/datasette/issues/1355,853557439,MDEyOklzc3VlQ29tbWVudDg1MzU1NzQzOQ==,9599,simonw,2021-06-03T04:43:14Z,2021-06-03T04:43:14Z,OWNER,"It's using `TestClient` at the moment which is a wrapper around `httpx` (as of ) that uses the `@async_to_sync` decorator to hide the async nature.

https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/utils/testing.py#L102-L156

Maybe the fix here is to switch the `--get` implementation to using `httpx` directly with https://www.python-httpx.org/async/#streaming-responses","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",910088936,datasette --get should efficiently handle streaming CSV,
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853553754,https://api.github.com/repos/simonw/sqlite-utils/issues/264,853553754,MDEyOklzc3VlQ29tbWVudDg1MzU1Mzc1NA==,9599,simonw,2021-06-03T04:32:42Z,2021-06-03T04:36:36Z,OWNER,"This is a really interesting thought. I've so far resisted the temptation to add plugins to `sqlite-utils`, partly to avoid overlap with Datasette - but I'm open to discussing it.

There's actually a way for you to do what you're describing using `datasette` on the command-line, though it's a little obscure - also Datasette doesn't yet have a GeoJSON output extension, though it really should have one.

Here's an example using [datasette-yaml](https://datasette.io/plugins/datasette-yaml):

```
datasette /tmp/covid.db --get='/covid/ny_times_us_counties.yaml'
- rowid: 1
  date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
- rowid: 2
  date: '2020-01-22'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
```
It even works with arbitrary SQL queries, though you might have to apply URL encoding to the `--get` string (this seems to work though):
```
datasette /tmp/covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 2' 
- date: '2020-01-21'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
- date: '2020-01-22'
  county: Snohomish
  state: Washington
  fips: 53061
  cases: 1
  deaths: 0
```
Here's the documentation for `--get`: https://docs.datasette.io/en/latest/getting_started.html#datasette-get","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907642546,"Supporting additional output formats, like GeoJSON",
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853554550,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853554550,MDEyOklzc3VlQ29tbWVudDg1MzU1NDU1MA==,9599,simonw,2021-06-03T04:34:38Z,2021-06-03T04:34:38Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-indexes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853548442,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853548442,MDEyOklzc3VlQ29tbWVudDg1MzU0ODQ0Mg==,9599,simonw,2021-06-03T04:16:49Z,2021-06-03T04:16:49Z,OWNER,Needs to show the table each index applies to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853547681,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853547681,MDEyOklzc3VlQ29tbWVudDg1MzU0NzY4MQ==,9599,simonw,2021-06-03T04:14:28Z,2021-06-03T04:14:28Z,OWNER,"This looks good:
```
% sqlite-utils indexes /tmp/covid.db -t      
index_name                                                seqno    cid  name                 desc  coll      key
------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1

% sqlite-utils indexes /tmp/covid.db -t --aux
index_name                                                seqno    cid  name                 desc  coll      key
------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
idx_johns_hopkins_csse_daily_reports_combined_key             1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_country_or_region        1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
idx_johns_hopkins_csse_daily_reports_province_or_state        1     -1                          0  BINARY      0
idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
idx_johns_hopkins_csse_daily_reports_day                      1     -1                          0  BINARY      0
idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
idx_ny_times_us_counties_date                                 1     -1                          0  BINARY      0
idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
idx_ny_times_us_counties_fips                                 1     -1                          0  BINARY      0
idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
idx_ny_times_us_counties_county                               1     -1                          0  BINARY      0
idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1
idx_ny_times_us_counties_state                                1     -1                          0  BINARY      0
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853546818,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853546818,MDEyOklzc3VlQ29tbWVudDg1MzU0NjgxOA==,9599,simonw,2021-06-03T04:11:46Z,2021-06-03T04:11:46Z,OWNER,"By default I won't return auxiliary columns, but I'll offer a `--aux` option to return them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853545743,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853545743,MDEyOklzc3VlQ29tbWVudDg1MzU0NTc0Mw==,9599,simonw,2021-06-03T04:08:04Z,2021-06-03T04:08:04Z,OWNER,"Figuring out the right queries:

https://covid-19.datasettes.com/covid?sql=select+sqlite_master.name%2C+i.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Awhere+type+%3D+%27table%27

This query shows all columns across all indexes across all tables:

```sql
select i.name as index_name, xinfo.* from sqlite_master
join pragma_index_list(sqlite_master.name) i
join pragma_index_xinfo(index_name) xinfo
where sqlite_master.type = 'table'
```
https://covid-19.datasettes.com/covid?sql=select+i.name+as+index_name%2C+xinfo.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Ajoin+pragma_index_xinfo%28index_name%29+xinfo%0D%0Awhere+sqlite_master.type+%3D+%27table%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853544493,https://api.github.com/repos/simonw/sqlite-utils/issues/263,853544493,MDEyOklzc3VlQ29tbWVudDg1MzU0NDQ5Mw==,9599,simonw,2021-06-03T04:03:59Z,2021-06-03T04:03:59Z,OWNER,"Here's how `sqlite-utils triggers` works: https://github.com/simonw/sqlite-utils/blob/9c67cb925253cd5ef54a1fe0496e0ff9caeacfd6/sqlite_utils/cli.py#L1266-L1277

Running it from a SQL query makes it easy to support modifiers like `--csv` and `-t`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906356331,`sqlite-utils indexes` command,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853541869,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853541869,MDEyOklzc3VlQ29tbWVudDg1MzU0MTg2OQ==,9599,simonw,2021-06-03T03:54:14Z,2021-06-03T03:54:14Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#xindexes,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853535559,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853535559,MDEyOklzc3VlQ29tbWVudDg1MzUzNTU1OQ==,9599,simonw,2021-06-03T03:32:47Z,2021-06-03T03:32:47Z,OWNER,"New design:
```python
def test_xindexes(fresh_db):
    fresh_db.executescript(
        """"""
        create table Gosh (c1 text, c2 text, c3 text);
        create index Gosh_c1 on Gosh(c1);
        create index Gosh_c2c3 on Gosh(c2, c3 desc);
    """"""
    )
    assert fresh_db[""Gosh""].xindexes == [
        XIndex(
            name=""Gosh_c2c3"",
            columns=[
                XIndexColumn(seqno=0, cid=1, name=""c2"", desc=0, coll=""BINARY"", key=1),
                XIndexColumn(seqno=1, cid=2, name=""c3"", desc=1, coll=""BINARY"", key=1),
                XIndexColumn(seqno=2, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
        XIndex(
            name=""Gosh_c1"",
            columns=[
                XIndexColumn(seqno=0, cid=0, name=""c1"", desc=0, coll=""BINARY"", key=1),
                XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853534732,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853534732,MDEyOklzc3VlQ29tbWVudDg1MzUzNDczMg==,9599,simonw,2021-06-03T03:30:10Z,2021-06-03T03:30:10Z,OWNER,"I'm going to return `XIndex(name, columns)`  - where `columns` is a list of `XIndexColumn`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853530348,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853530348,MDEyOklzc3VlQ29tbWVudDg1MzUzMDM0OA==,9599,simonw,2021-06-03T03:16:33Z,2021-06-03T03:16:33Z,OWNER,"In prototyping this out I realize that I actually want to get back the name of each index, then for each of them the detailed list of index columns. Here's the test from my initial prototype:
```python
def test_xindexes(fresh_db):
    fresh_db.executescript(
        """"""
        create table Gosh (c1 text, c2 text, c3 text);
        create index Gosh_c1 on Gosh(c1);
        create index Gosh_c2c3 on Gosh(c2, c3 desc);
    """"""
    )
    assert fresh_db[""Gosh""].xindexes == [
        (
            ""Gosh_c2c3"",
            [
                XIndex(seqno=0, cid=1, name=""c2"", desc=0, coll=""BINARY"", key=1),
                XIndex(seqno=1, cid=2, name=""c3"", desc=1, coll=""BINARY"", key=1),
                XIndex(seqno=2, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
        (
            ""Gosh_c1"",
            [
                XIndex(seqno=0, cid=0, name=""c1"", desc=0, coll=""BINARY"", key=1),
                XIndex(seqno=1, cid=-1, name=None, desc=0, coll=""BINARY"", key=0),
            ],
        ),
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853525036,https://api.github.com/repos/simonw/sqlite-utils/issues/261,853525036,MDEyOklzc3VlQ29tbWVudDg1MzUyNTAzNg==,9599,simonw,2021-06-03T03:02:22Z,2021-06-03T03:02:22Z,OWNER,"This would be a breaking change - and the fact that it returns auxiliary columns isn't particularly useful for most cases - ""Auxiliary columns are additional columns needed to locate the table entry that corresponds to each index entry"".

Instead, I'm going to add a new property `table.xindexes` which exposes this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906345899,`table.xindexes` using `PRAGMA index_xinfo(table)`,
https://github.com/simonw/datasette/issues/893#issuecomment-852712469,https://api.github.com/repos/simonw/datasette/issues/893,852712469,MDEyOklzc3VlQ29tbWVudDg1MjcxMjQ2OQ==,9599,simonw,2021-06-02T04:29:56Z,2021-06-02T04:29:56Z,OWNER,"I think I fixed this a while ago, though I can't find the commit now. Please re-open if the bug still happens!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",656959584,pip3 install datasette not serving static on linuxbrew.,
https://github.com/simonw/datasette/issues/1127#issuecomment-852712106,https://api.github.com/repos/simonw/datasette/issues/1127,852712106,MDEyOklzc3VlQ29tbWVudDg1MjcxMjEwNg==,9599,simonw,2021-06-02T04:28:55Z,2021-06-02T04:28:55Z,OWNER,This became resizable in #1236.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756818250,Make the custom SQL query text box larger or resizable,
https://github.com/simonw/datasette/pull/1306#issuecomment-852710117,https://api.github.com/repos/simonw/datasette/issues/1306,852710117,MDEyOklzc3VlQ29tbWVudDg1MjcxMDExNw==,9599,simonw,2021-06-02T04:24:33Z,2021-06-02T04:24:33Z,OWNER,I have a test - I'll merge this and then add that test.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/pull/1306#issuecomment-852708574,https://api.github.com/repos/simonw/datasette/issues/1306,852708574,MDEyOklzc3VlQ29tbWVudDg1MjcwODU3NA==,9599,simonw,2021-06-02T04:20:18Z,2021-06-02T04:20:18Z,OWNER,https://github.com/simonw/datasette/blob/115332ce76c0e867d9936406aaf4bcee6b1ef3cb/datasette/views/index.py#L77 helps here - looks like we can pass `?_sort=relationships` to trigger the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/pull/1306#issuecomment-852707676,https://api.github.com/repos/simonw/datasette/issues/1306,852707676,MDEyOklzc3VlQ29tbWVudDg1MjcwNzY3Ng==,9599,simonw,2021-06-02T04:17:41Z,2021-06-02T04:17:41Z,OWNER,We need to add a test that illustrates the bug in #1305 so we can prove that this definitely fixes it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",864979486,Avoid error sorting by relationships if related tables are not allowed,
https://github.com/simonw/datasette/issues/619#issuecomment-852706970,https://api.github.com/repos/simonw/datasette/issues/619,852706970,MDEyOklzc3VlQ29tbWVudDg1MjcwNjk3MA==,9599,simonw,2021-06-02T04:15:54Z,2021-06-02T04:15:54Z,OWNER,"<img width=""1009"" alt=""fixtures__select___from__foo__and_Facets_should_not_execute_for__shape_array_object_·_Issue__263_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/120423073-8a338700-c31e-11eb-8c8f-affc24708b32.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-852703357,https://api.github.com/repos/simonw/datasette/issues/619,852703357,MDEyOklzc3VlQ29tbWVudDg1MjcwMzM1Nw==,9599,simonw,2021-06-02T04:08:03Z,2021-06-02T04:08:03Z,OWNER,"A SQL error now looks like this: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

<img width=""1023"" alt=""fixtures__select___from__foo_"" src=""https://user-images.githubusercontent.com/9599/120422579-691e6680-c31d-11eb-99ea-655bdc2a401d.png"">

I'm going to get rid of that ""0 results"" message if an error is shown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/pull/1346#issuecomment-852693854,https://api.github.com/repos/simonw/datasette/issues/1346,852693854,MDEyOklzc3VlQ29tbWVudDg1MjY5Mzg1NA==,22429695,codecov[bot],2021-06-02T03:44:35Z,2021-06-02T03:44:35Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1346](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (3bffc35) into [main](https://codecov.io/gh/simonw/datasette/commit/7b106e106000713bbee31b34d694b3dadbd4818c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7b106e1) will **increase** coverage by `0.14%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1346/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@            Coverage Diff             @@
##             main    #1346      +/-   ##
==========================================
+ Coverage   91.56%   91.71%   +0.14%     
==========================================
  Files          34       34              
  Lines        4282     4332      +50     
==========================================
+ Hits         3921     3973      +52     
+ Misses        361      359       -2     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.69% <ø> (-0.15%)` | :arrow_down: |
| [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.20% <100.00%> (+0.17%)` | :arrow_up: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.36% <100.00%> (+0.05%)` | :arrow_up: |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.41% <100.00%> (+0.39%)` | :arrow_up: |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.28% <100.00%> (+0.09%)` | :arrow_up: |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.90% <100.00%> (+0.07%)` | :arrow_up: |
| [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `85.05% <0.00%> (+3.44%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7b106e1...3bffc35](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904537568,Re-display user's query with an error message if an error occurs,
https://github.com/simonw/datasette/issues/1257#issuecomment-852686827,https://api.github.com/repos/simonw/datasette/issues/1257,852686827,MDEyOklzc3VlQ29tbWVudDg1MjY4NjgyNw==,9599,simonw,2021-06-02T03:26:27Z,2021-06-02T03:26:27Z,OWNER,I tried and failed to get this fix working for tables with double quotes in their name - I couldn't figure out what the double-quote-in-a-table-name version of this code would look like: https://github.com/simonw/datasette/blob/807de378d08752a0f05bb1b980a0a62620a70520/datasette/utils/__init__.py#L538-L548,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828811618,Table names containing single quotes break things,
https://github.com/simonw/datasette/issues/1257#issuecomment-852681622,https://api.github.com/repos/simonw/datasette/issues/1257,852681622,MDEyOklzc3VlQ29tbWVudDg1MjY4MTYyMg==,9599,simonw,2021-06-02T03:12:18Z,2021-06-02T03:12:18Z,OWNER,"Created a test database like this:
```
% sqlite-utils create-table quote-in-name.db ""this'hasquoteinname"" id integer name text --pk id
% datasette quote-in-name.db -p 8025 --pdb
INFO:     Started server process [86046]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8025 (Press CTRL+C to quit)
> /Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py(530)detect_fts()
-> rows = conn.execute(detect_fts_sql(table)).fetchall()
(Pdb) c
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 1124, in route_path
    response = await view(request, send)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 147, in view
    return await self.dispatch_request(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/base.py"", line 122, in dispatch_request
    return await handler(request, *args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/index.py"", line 72, in get
    ""fts_table"": await db.fts_table(table),
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 279, in fts_table
    return await self.execute_fn(lambda conn: detect_fts(conn, table))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 155, in execute_fn
    return await asyncio.get_event_loop().run_in_executor(
  File ""/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/concurrent/futures/thread.py"", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 153, in in_thread
    return fn(conn)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/database.py"", line 279, in <lambda>
    return await self.execute_fn(lambda conn: detect_fts(conn, table))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 530, in detect_fts
    rows = conn.execute(detect_fts_sql(table)).fetchall()
sqlite3.OperationalError: near ""hasquoteinname"": syntax error
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828811618,Table names containing single quotes break things,
https://github.com/simonw/datasette/pull/1352#issuecomment-852673695,https://api.github.com/repos/simonw/datasette/issues/1352,852673695,MDEyOklzc3VlQ29tbWVudDg1MjY3MzY5NQ==,9599,simonw,2021-06-02T02:52:26Z,2021-06-02T02:52:26Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908276134,Bump black from 21.5b1 to 21.5b2,
https://github.com/simonw/datasette/issues/263#issuecomment-852256784,https://api.github.com/repos/simonw/datasette/issues/263,852256784,MDEyOklzc3VlQ29tbWVudDg1MjI1Njc4NA==,9599,simonw,2021-06-01T16:20:09Z,2021-06-01T16:20:09Z,OWNER,This is a lot easier to implement now that we have a `?_nofacet=1` option from #1350.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323671577,Facets should not execute for ?shape=array|object,
https://github.com/simonw/datasette/issues/1350#issuecomment-852256454,https://api.github.com/repos/simonw/datasette/issues/1350,852256454,MDEyOklzc3VlQ29tbWVudDg1MjI1NjQ1NA==,9599,simonw,2021-06-01T16:19:38Z,2021-06-01T16:19:38Z,OWNER,I renamed this to `?_nofacet=1` in #1353.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906977719,?_nofacets=1 query string argument for disabling facets and suggested facets,
https://github.com/simonw/datasette/issues/1353#issuecomment-852255951,https://api.github.com/repos/simonw/datasette/issues/1353,852255951,MDEyOklzc3VlQ29tbWVudDg1MjI1NTk1MQ==,9599,simonw,2021-06-01T16:18:54Z,2021-06-01T16:18:54Z,OWNER,Documented here: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908446997,?_nocount=1 for opting out of table counts,
https://github.com/simonw/datasette/issues/1354#issuecomment-852254298,https://api.github.com/repos/simonw/datasette/issues/1354,852254298,MDEyOklzc3VlQ29tbWVudDg1MjI1NDI5OA==,9599,simonw,2021-06-01T16:16:32Z,2021-06-01T16:16:32Z,OWNER,Running `python update-docs-help.py` helps fix this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908465747,Update help in tests for latest Click,
https://github.com/simonw/datasette/issues/1353#issuecomment-852238201,https://api.github.com/repos/simonw/datasette/issues/1353,852238201,MDEyOklzc3VlQ29tbWVudDg1MjIzODIwMQ==,9599,simonw,2021-06-01T15:53:57Z,2021-06-01T15:53:57Z,OWNER,I'm going to rename `?_nofacets=1` to `?_nofacet=1` to keep it consistent with the new `?_nocount=1` option (and because I don't like `?_nocounts=1`).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",908446997,?_nocount=1 for opting out of table counts,
https://github.com/simonw/datasette/issues/1349#issuecomment-852237347,https://api.github.com/repos/simonw/datasette/issues/1349,852237347,MDEyOklzc3VlQ29tbWVudDg1MjIzNzM0Nw==,9599,simonw,2021-06-01T15:52:50Z,2021-06-01T15:52:50Z,OWNER,Fixed in https://github.com/simonw/datasette/commit/d1d06ace49606da790a765689b4fbffa4c6deecb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1351#issuecomment-851133471,https://api.github.com/repos/simonw/datasette/issues/1351,851133471,MDEyOklzc3VlQ29tbWVudDg1MTEzMzQ3MQ==,9599,simonw,2021-05-31T03:02:59Z,2021-05-31T03:02:59Z,OWNER,"Since traces only work with `text/html` and JSON at the moment, the easiest way to do this will be to wrap generated CSV in a HTML page in a textarea if the user specified `?_trace=1`:

https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906993731,Get `?_trace=1` working with CSV and streaming CSVs,
https://github.com/simonw/datasette/issues/1349#issuecomment-851133125,https://api.github.com/repos/simonw/datasette/issues/1349,851133125,MDEyOklzc3VlQ29tbWVudDg1MTEzMzEyNQ==,9599,simonw,2021-05-31T03:01:48Z,2021-05-31T03:01:48Z,OWNER,I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851131999,https://api.github.com/repos/simonw/datasette/issues/1349,851131999,MDEyOklzc3VlQ29tbWVudDg1MTEzMTk5OQ==,9599,simonw,2021-05-31T02:57:25Z,2021-05-31T02:57:25Z,OWNER,"I'm having a really hard time figuring out how to unit test this - ideally I'd monitor which SQL queries are executed using the tracing mechanism, but that's not set up to work with anything other than HTML or JSON outputs: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851129464,https://api.github.com/repos/simonw/datasette/issues/1349,851129464,MDEyOklzc3VlQ29tbWVudDg1MTEyOTQ2NA==,9599,simonw,2021-05-31T02:48:06Z,2021-05-31T02:48:06Z,OWNER,"Actually there is precedent for swapping out `request.scope` for a new scope, as seen here in the routing code: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/app.py#L1117-L1122","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851129019,https://api.github.com/repos/simonw/datasette/issues/1349,851129019,MDEyOklzc3VlQ29tbWVudDg1MTEyOTAxOQ==,9599,simonw,2021-05-31T02:46:38Z,2021-05-31T02:46:38Z,OWNER,I think the right way to do this is to construct a new `Request` with a modified ASGI scope: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/utils/asgi.py#L88-L99,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-851128524,https://api.github.com/repos/simonw/datasette/issues/1349,851128524,MDEyOklzc3VlQ29tbWVudDg1MTEyODUyNA==,9599,simonw,2021-05-31T02:44:44Z,2021-05-31T02:44:44Z,OWNER,"Now that I have `?_nofacets=1` I can use that to fix this. The challenge is that in this block of code I need to modify the incoming request's query string arguments, which isn't something I've done before: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L270
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1345#issuecomment-851127314,https://api.github.com/repos/simonw/datasette/issues/1345,851127314,MDEyOklzc3VlQ29tbWVudDg1MTEyNzMxNA==,9599,simonw,2021-05-31T02:40:26Z,2021-05-31T02:40:44Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state - the state column is not selected but facet by state still works: 

<img width=""1134"" alt=""fixtures__facetable__15_rows_and_Why_Russians_do_not_smile"" src=""https://user-images.githubusercontent.com/9599/120132273-110a2780-c198-11eb-91a5-cafb77a78c97.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1350#issuecomment-851122934,https://api.github.com/repos/simonw/datasette/issues/1350,851122934,MDEyOklzc3VlQ29tbWVudDg1MTEyMjkzNA==,9599,simonw,2021-05-31T02:23:47Z,2021-05-31T02:23:47Z,OWNER,"I think `?_facets_off=1` is a good design for this.

I considered `?_facet=` with an empty string but that's a bit weird, and it's not clear what that would do if used like `?_facet=state&_facet=`.

So it definitely needs to be a separate named option.

Actually, since I have `?_nocol=` I'm going to call this `?_nofacets=1` instead as that's a little more consistent.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906977719,?_nofacets=1 query string argument for disabling facets and suggested facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851120306,https://api.github.com/repos/simonw/datasette/issues/1345,851120306,MDEyOklzc3VlQ29tbWVudDg1MTEyMDMwNg==,9599,simonw,2021-05-31T02:14:36Z,2021-05-31T02:14:36Z,OWNER,"Yes! This was easier than I thought. I'm going with that solution - where facets are calculated against all columns, ignoring `?_col=` and `?_nocol=` entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851117233,https://api.github.com/repos/simonw/datasette/issues/1345,851117233,MDEyOklzc3VlQ29tbWVudDg1MTExNzIzMw==,9599,simonw,2021-05-31T02:04:35Z,2021-05-31T02:04:35Z,OWNER,"That long-term solution may not be too difficult. The facets are calculated against `sql_no_limit` which is constructed here:

https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L659-L665

And used here:

https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L706-L718

Crucially, `sql_no_limit` is ONLY used for faceting - nothing else uses it anywhere. So constructing it before constructing `sql` and taking `?_col=` and `?_nocol=` into account may not be a complex change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851116105,https://api.github.com/repos/simonw/datasette/issues/1345,851116105,MDEyOklzc3VlQ29tbWVudDg1MTExNjEwNQ==,9599,simonw,2021-05-31T02:00:44Z,2021-05-31T02:00:44Z,OWNER,"Maybe there's a short-term and longer-term solution for this - where the long-term solution is to use different columns in the faceting selects, while the short-term solution is to disable ""Hide this column"" for certain things.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851115805,https://api.github.com/repos/simonw/datasette/issues/1345,851115805,MDEyOklzc3VlQ29tbWVudDg1MTExNTgwNQ==,9599,simonw,2021-05-31T01:59:39Z,2021-05-31T01:59:39Z,OWNER,"Maybe there's a concept here of the columns that are required by a selected facet? Those can then be included as `data-` attributes on the page, which will then impact which ""Hide this column"" options are available.

I can also use them to provide a better error message than ""no such column: state"" - I can verify that `?_nocol` and `?_col` have not been used to disable the required columns.

There is one other option here: I could still include the columns that are known to be needed for faceting in the faceting SQL queries, but leave them out of the query that is used to return the results! That's actually a pretty tempting (albeit more complex) option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851114806,https://api.github.com/repos/simonw/datasette/issues/1345,851114806,MDEyOklzc3VlQ29tbWVudDg1MTExNDgwNg==,9599,simonw,2021-05-31T01:55:56Z,2021-05-31T01:55:56Z,OWNER,"Two options here:

- Don't provide users with options that will lead to this situation - so no ""Hide this column"" option on pages that are already faceted by that column
- Ignore facet selections for columns which are no longer visible

I think I like the first option more.

I could partially implement that in the `table.js` JavaScript by looking at the `?_facet=` parameters... but that won't cover the case where the facet is happening because of default facets configured in `metadata.yml`.

Instead the JavaScript should look for evidence in the DOM that specific facets are enabled. This could also help me cover other types of faceting, such as `?_facet_array=` or even custom facets provided by plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1345#issuecomment-851113886,https://api.github.com/repos/simonw/datasette/issues/1345,851113886,MDEyOklzc3VlQ29tbWVudDg1MTExMzg4Ng==,9599,simonw,2021-05-31T01:52:27Z,2021-05-31T01:52:27Z,OWNER,"Related issue: visit https://latest.datasette.io/fixtures/facetable?_facet=state and click ""Hide this column"" on the ""state"" cog menu and you get https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state which shows an error:

> ### Invalid SQL
> no such column: state
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904071938,?_nocol= does not interact well with default facets,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778311,https://api.github.com/repos/simonw/datasette/issues/1349,850778311,MDEyOklzc3VlQ29tbWVudDg1MDc3ODMxMQ==,9599,simonw,2021-05-29T06:12:21Z,2021-05-29T06:12:21Z,OWNER,"It's not just facets, I think it's trying to execute suggested facets too!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778273,https://api.github.com/repos/simonw/datasette/issues/1349,850778273,MDEyOklzc3VlQ29tbWVudDg1MDc3ODI3Mw==,9599,simonw,2021-05-29T06:11:59Z,2021-05-29T06:11:59Z,OWNER,"Related issue: https://github.com/simonw/datasette/issues/263 - ""Facets should not execute for ?shape=array|object""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/datasette/issues/1349#issuecomment-850778230,https://api.github.com/repos/simonw/datasette/issues/1349,850778230,MDEyOklzc3VlQ29tbWVudDg1MDc3ODIzMA==,9599,simonw,2021-05-29T06:11:28Z,2021-05-29T06:11:28Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L315,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906385991,CSV ?_stream=on redundantly calculates facets for every page,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850772428,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850772428,MDEyOklzc3VlQ29tbWVudDg1MDc3MjQyOA==,9599,simonw,2021-05-29T05:16:02Z,2021-05-29T05:28:57Z,OWNER,"I needed to find some CSV files on my computer with a BOM at the beginning - I figured out this recipe:
```
% rg -U -E none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .
```
TIL here: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771264,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850771264,MDEyOklzc3VlQ29tbWVudDg1MDc3MTI2NA==,9599,simonw,2021-05-29T05:06:13Z,2021-05-29T05:06:13Z,OWNER,The other option is to check if the file starts with `codecs.BOM_UTF8` - which is `b'\xef\xbb\xbf'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771017,https://api.github.com/repos/simonw/sqlite-utils/issues/250,850771017,MDEyOklzc3VlQ29tbWVudDg1MDc3MTAxNw==,9599,simonw,2021-05-29T05:04:28Z,2021-05-29T05:04:28Z,OWNER,"https://stackoverflow.com/a/44573867/6083 says:

> There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",838148087,Handle byte order marks (BOMs) in CSV files,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850769067,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850769067,MDEyOklzc3VlQ29tbWVudDg1MDc2OTA2Nw==,9599,simonw,2021-05-29T04:48:10Z,2021-05-29T04:48:10Z,OWNER,"I confirmed and it's possible to have a SQLite column with a hyphen at the start, confirmed using:
```
% sqlite-utils create-table demo.db demo -- id integer name text -blah integer
% sqlite-utils tables --schema demo.db -t
table    schema
-------  ---------------------
demo     CREATE TABLE [demo] (
            [id] INTEGER,
            [name] TEXT,
            [-blah] INTEGER
         )
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850768315,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850768315,MDEyOklzc3VlQ29tbWVudDg1MDc2ODMxNQ==,9599,simonw,2021-05-29T04:39:33Z,2021-05-29T04:39:33Z,OWNER,"This doesn't work:
```
sqlite-utils create-index mydb.db dogs ""-age"" name
```
But this does:
```
sqlite-utils create-index mydb.db dogs -- -age name
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850767210,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850767210,MDEyOklzc3VlQ29tbWVudDg1MDc2NzIxMA==,9599,simonw,2021-05-29T04:26:26Z,2021-05-29T04:28:31Z,OWNER,It's weird having to use `Database.DescIndex` - I'm going to put `DescIndex` in `sqlite_utils.db` directly and let people import it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850766552,MDEyOklzc3VlQ29tbWVudDg1MDc2NjU1Mg==,9599,simonw,2021-05-29T04:20:40Z,2021-05-29T04:24:01Z,OWNER,"`PRAGMA index_xinfo(table)` DOES return that data:
```
(Pdb) [c[0] for c in fresh_db.execute(""PRAGMA index_xinfo('idx_dogs_age_name')"").description]
['seqno', 'cid', 'name', 'desc', 'coll', 'key']
(Pdb) fresh_db.execute(""PRAGMA index_xinfo('idx_dogs_age_name')"").fetchall()
[(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)]
```
See https://sqlite.org/pragma.html#pragma_index_xinfo

Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766335,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850766335,MDEyOklzc3VlQ29tbWVudDg1MDc2NjMzNQ==,9599,simonw,2021-05-29T04:18:19Z,2021-05-29T04:18:19Z,OWNER,"Annoyingly the `table.indexes` property won't indicate if an index is in regular or reverse order - because the SQLite `PRAGMA index_info(table)` statement doesn't indicate that either. You have to look at the `sqlite_master` index definition to tell if any of the columns are in reverse order:
```
(Pdb) fresh_db.execute(""select * from sqlite_master where type = 'index'"").fetchall()
[('index', 'idx_dogs_age_name', 'dogs', 3, 'CREATE INDEX [idx_dogs_age_name]\n    ON [dogs] ([age] desc, [name])')]
(Pdb) fresh_db.execute(""PRAGMA index_info('idx_dogs_age_name')"").fetchall()
[(0, 2, 'age'), (1, 0, 'name')]
(Pdb) fresh_db.execute(""PRAGMA index_info('idx_dogs_age_name')"").description
(('seqno', None, None, None, None, None, None), ('cid', None, None, None, None, None, None), ('name', None, None, None, None, None, None))
(Pdb) dogs.indexes
[Index(seq=0, name='idx_dogs_age_name', unique=0, origin='c', partial=0, columns=['age', 'name'])]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765450,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765450,MDEyOklzc3VlQ29tbWVudDg1MDc2NTQ1MA==,9599,simonw,2021-05-29T04:09:13Z,2021-05-29T04:09:13Z,OWNER,"Decisions: for the Python API I'm going with `db.DescIndex(""column"")` as the way to do this.

For the CLI I'm going to do the ""-age"" thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765291,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765291,MDEyOklzc3VlQ29tbWVudDg1MDc2NTI5MQ==,9599,simonw,2021-05-29T04:07:48Z,2021-05-29T04:08:21Z,OWNER,"For the CLI version I could say that you can use a `-` prefix to specify reverse direction:
```sh
sqlite-utils create-index mydb.db dogs -age name
```
No, that doesn't work - it could get confused with a command-line flag. I guess you could do this:
```
sqlite-utils create-index mydb.db dogs ""-age"" name
```
This does mean that if any of your column names begin with a hyphen you can't use the CLI to add indexes to them. Is that an acceptable limitation? Users can always use `sqlite-utils mydb.db ""create index ...""` in that case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765050,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850765050,MDEyOklzc3VlQ29tbWVudDg1MDc2NTA1MA==,9599,simonw,2021-05-29T04:05:24Z,2021-05-29T04:05:40Z,OWNER,"Need to solve this for the CLI tool too. Currently that works like this: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes
```sh
sqlite-utils create-index mydb.db mytable col1 [col2...]
```
Even harder to decide how to add a descending option to this. Maybe like this?
```sh
sqlite-utils create-index mydb.db mytable --direction col1 asc --direction col2 desc
```
It's a bit gross though! We're saying here that if a single one of the columns you are creating an index for is in reverse direction you have to use `--direction` to specify each end every other index.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764700,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764700,MDEyOklzc3VlQ29tbWVudDg1MDc2NDcwMA==,9599,simonw,2021-05-29T04:02:10Z,2021-05-29T04:02:10Z,OWNER,"I could use `db.desc_index(""age"")` to match SQLite SQL syntax, which uses `desc` and not `descending`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764655,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764655,MDEyOklzc3VlQ29tbWVudDg1MDc2NDY1NQ==,9599,simonw,2021-05-29T04:01:41Z,2021-05-29T04:01:41Z,OWNER,"Maybe:
```python
db[""dogs""].create_index([db.descending_index(""age""), ""name""])
```
It's a little verbose but it's for a relatively rare activity and it does make it very clear what is going on.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764594,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764594,MDEyOklzc3VlQ29tbWVudDg1MDc2NDU5NA==,9599,simonw,2021-05-29T04:00:54Z,2021-05-29T04:00:54Z,OWNER,"A few options:
```python
db[""dogs""].create_index([(""age"", ""desc""), ""name""])

db[""dogs""].create_index([desc(""age""), ""name""])

db[""dogs""].create_index([db.desc(""age""), ""name""])
```
The first option uses an optional tuple. The second two use a `desc()` function - the question is where should that live? 

`sqlite_utils.desc(column)` or `db.desc(column)`  are both options.

I don't like using the term `desc()` for ""descending index"" though - it feels like it should mean something more broad.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764253,https://api.github.com/repos/simonw/sqlite-utils/issues/260,850764253,MDEyOklzc3VlQ29tbWVudDg1MDc2NDI1Mw==,9599,simonw,2021-05-29T03:57:54Z,2021-05-29T03:57:54Z,OWNER,"The problem here is differentiating between a column with the name `date desc` and wanting to create a descending index on a column called `date`.

This won't work:
```python
db[""ny_times_us_counties""].create_index([""date desc""], desc=True)
```
Because we need to be able to create compound indexes with columns with different directions - for example:
```sql
create index idx_age_desc_name on dogs (age desc, name)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",906330187,Support creating descending order indexes,
https://github.com/simonw/datasette/pull/1296#issuecomment-850583584,https://api.github.com/repos/simonw/datasette/issues/1296,850583584,MDEyOklzc3VlQ29tbWVudDg1MDU4MzU4NA==,9599,simonw,2021-05-28T18:06:11Z,2021-05-28T18:06:11Z,OWNER,"> As a bonus, the Docker image becomes smaller

That's a huge surprise to me! And most welcome.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",855446829,Dockerfile: use Ubuntu 20.10 as base,
https://github.com/simonw/datasette/pull/1348#issuecomment-850077484,https://api.github.com/repos/simonw/datasette/issues/1348,850077484,MDEyOklzc3VlQ29tbWVudDg1MDA3NzQ4NA==,22429695,codecov[bot],2021-05-28T03:06:16Z,2021-05-28T03:06:16Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
> Merging [#1348](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (56cba8f) into [main](https://codecov.io/gh/simonw/datasette/commit/7b106e106000713bbee31b34d694b3dadbd4818c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7b106e1) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1348/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

```diff
@@           Coverage Diff           @@
##             main    #1348   +/-   ##
=======================================
  Coverage   91.56%   91.56%           
=======================================
  Files          34       34           
  Lines        4282     4282           
=======================================
  Hits         3921     3921           
  Misses        361      361           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7b106e1...56cba8f](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904598267,DRAFT: add test and scan for docker images,
https://github.com/simonw/datasette/pull/1348#issuecomment-850077261,https://api.github.com/repos/simonw/datasette/issues/1348,850077261,MDEyOklzc3VlQ29tbWVudDg1MDA3NzI2MQ==,10801138,blairdrummond,2021-05-28T03:05:38Z,2021-05-28T03:05:38Z,CONTRIBUTOR,"Note, the CVEs are probably resolvable with this https://github.com/simonw/datasette/pull/1296 . My experience is that Ubuntu seems to manage these better? Though that is surprising :/ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904598267,DRAFT: add test and scan for docker images,
https://github.com/simonw/datasette/pull/1346#issuecomment-850059328,https://api.github.com/repos/simonw/datasette/issues/1346,850059328,MDEyOklzc3VlQ29tbWVudDg1MDA1OTMyOA==,9599,simonw,2021-05-28T02:08:10Z,2021-05-28T02:09:25Z,OWNER,"This is the current test failure, but it actually another problem that we don't have tests in place for errors with different formats and shapes:

```
    def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_client):
        response = magic_parameters_client.get(
            ""/data.json?sql=select+:_header_host&_shape=array""
        )
        assert 400 == response.status
>       assert ""You did not supply a value for binding 1."" == response.json[""error""]
E       TypeError: list indices must be integers or slices, not str
```

The test fails because `response.json` here is the empty list `[]`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904537568,Re-display user's query with an error message if an error occurs,
https://github.com/simonw/datasette/issues/619#issuecomment-850059607,https://api.github.com/repos/simonw/datasette/issues/619,850059607,MDEyOklzc3VlQ29tbWVudDg1MDA1OTYwNw==,9599,simonw,2021-05-28T02:08:58Z,2021-05-28T02:08:58Z,OWNER,Remaining work will happen in #1346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/pull/1346#issuecomment-850058851,https://api.github.com/repos/simonw/datasette/issues/1346,850058851,MDEyOklzc3VlQ29tbWVudDg1MDA1ODg1MQ==,9599,simonw,2021-05-28T02:06:46Z,2021-05-28T02:06:46Z,OWNER,"The one test failure here actually illustrates a larger problem: if the user specifies `.json?_shape=array` but an error occurs, what should we do?

Prior to this change we return the following JSON:
```json
{
    ""ok"": false,
    ""error"": ""You did not supply a value for binding 1."",
    ""status"": 500,
    ""title"": null
}
```
But this comes from the `handle_500` higher level code here: https://github.com/simonw/datasette/blob/eae3084b46e2c3931db12cdef79093ad0e644bce/datasette/app.py#L1251-L1263","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",904537568,Re-display user's query with an error message if an error occurs,
https://github.com/simonw/datasette/issues/619#issuecomment-850057694,https://api.github.com/repos/simonw/datasette/issues/619,850057694,MDEyOklzc3VlQ29tbWVudDg1MDA1NzY5NA==,9599,simonw,2021-05-28T02:03:05Z,2021-05-28T02:03:05Z,OWNER,"I nearly got this working, but I ran into one last problem: the code path for when an error is raised but the user specified `?_shape=array`. I'll open a draft PR with where I've got to so far.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-850025106,https://api.github.com/repos/simonw/datasette/issues/619,850025106,MDEyOklzc3VlQ29tbWVudDg1MDAyNTEwNg==,9599,simonw,2021-05-28T00:18:30Z,2021-05-28T00:18:30Z,OWNER,"I'm going to return a 400 HTTP status code for ""bad request"", under the assumption that the client sent bad SQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/619#issuecomment-850019486,https://api.github.com/repos/simonw/datasette/issues/619,850019486,MDEyOklzc3VlQ29tbWVudDg1MDAxOTQ4Ng==,9599,simonw,2021-05-28T00:05:50Z,2021-05-28T00:05:50Z,OWNER,Came up on discussions here: https://github.com/simonw/datasette/discussions/1334,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/issues/1344#issuecomment-849820019,https://api.github.com/repos/simonw/datasette/issues/1344,849820019,MDEyOklzc3VlQ29tbWVudDg0OTgyMDAxOQ==,9599,simonw,2021-05-27T17:44:39Z,2021-05-27T17:52:58Z,OWNER,"This pattern appears to work, executed at the root of a checkout of `datasette`:
```
docker run -it -v `pwd`:/mnt --platform linux/amd64 \
  datasetteproject/datasette:0.57a1 bash -c '
    pip install ""pytest>=5.2.2,<6.3.0"" \
      ""pytest-xdist>=2.2.1,<2.3"" \
      ""pytest-asyncio>=0.10,<0.16"" \
      ""beautifulsoup4>=4.8.1,<4.10.0"" \
      ""black==21.5b1"" \
      ""pytest-timeout>=1.4.2,<1.5"" \
      ""trustme>=0.7,<0.8"" \
    && cd /mnt && pytest'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849815361,https://api.github.com/repos/simonw/datasette/issues/1344,849815361,MDEyOklzc3VlQ29tbWVudDg0OTgxNTM2MQ==,9599,simonw,2021-05-27T17:36:51Z,2021-05-27T17:36:51Z,OWNER,"To run the test suite... need to install a few more dependencies: https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/setup.py#L71-L79

And then mount the `tests/` folder into the container and run `pytest`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849806581,https://api.github.com/repos/simonw/datasette/issues/1344,849806581,MDEyOklzc3VlQ29tbWVudDg0OTgwNjU4MQ==,9599,simonw,2021-05-27T17:22:37Z,2021-05-27T17:34:27Z,OWNER,"```
~ % docker run -it --platform linux/arm/v7 -v `pwd`:/mnt -p 8001:8001 datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
ee0a2cc24f29: Pull complete 
ca231153300d: Pull complete 
6073fc53d406: Pull complete 
bb2a6a6421dd: Pull complete 
14c15f441034: Pull complete 
28d113f72ba5: Pull complete 
40516c2c3785: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='c2e1a05732ed', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='armv7l')
```
Shorter version:
```
~ % docker run -it --platform linux/ppc64le datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
c840eb5e9aed: Pull complete 
fc17d8b577ec: Pull complete 
712a0ad995f8: Pull complete 
56101149914c: Pull complete 
e6241dfe9d38: Pull complete 
41ed85039888: Pull complete 
26beeabe9213: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='0fe85887fa30', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='ppc64le')
~ % 
```

OK! Now I need to figure out how to run the full test suite on each one, then I can call this done.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849805778,https://api.github.com/repos/simonw/datasette/issues/1344,849805778,MDEyOklzc3VlQ29tbWVudDg0OTgwNTc3OA==,9599,simonw,2021-05-27T17:21:11Z,2021-05-27T17:21:22Z,OWNER,"This helps:
```
~ % docker run -it --platform linux/s390x -v `pwd`:/mnt \
  -p 8001:8001 datasetteproject/datasette:0.57a1 \
  python -c 'import platform; print(platform.uname())'
```
`uname_result(system='Linux', node='d14916ca91df', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='s390x')`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849804684,https://api.github.com/repos/simonw/datasette/issues/1344,849804684,MDEyOklzc3VlQ29tbWVudDg0OTgwNDY4NA==,9599,simonw,2021-05-27T17:19:16Z,2021-05-27T17:19:16Z,OWNER,"Having pushed that to Docker Hub the following _seems_ to work on my laptop:

    docker run --platform linux/s390x \
      -v `pwd`:/mnt -p 8001:8001 \
      datasetteproject/datasette:0.57a1 datasette -p 8001 -h 0.0.0.0

I'd like to confirm that this is correctly emulating the architecture and running the correct published image though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849801969,https://api.github.com/repos/simonw/datasette/issues/1344,849801969,MDEyOklzc3VlQ29tbWVudDg0OTgwMTk2OQ==,9599,simonw,2021-05-27T17:14:49Z,2021-05-27T17:14:49Z,OWNER,"Job completed in 13m38s - results are here: https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated&name=0.57a1

<img width=""1286"" alt=""datasetteproject_datasette_Tags"" src=""https://user-images.githubusercontent.com/9599/119868791-5c8caf00-bed4-11eb-9e9f-ebca557621c0.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849797767,https://api.github.com/repos/simonw/datasette/issues/1344,849797767,MDEyOklzc3VlQ29tbWVudDg0OTc5Nzc2Nw==,9599,simonw,2021-05-27T17:08:13Z,2021-05-27T17:08:13Z,OWNER,Related issue: #1272 - being able to run the test suite inside Docker inside GitHub Actions would be incredibly helpful here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849793312,https://api.github.com/repos/simonw/datasette/issues/1344,849793312,MDEyOklzc3VlQ29tbWVudDg0OTc5MzMxMg==,9599,simonw,2021-05-27T17:01:16Z,2021-05-27T17:01:16Z,OWNER,https://github.com/simonw/datasette/runs/2687196350 is a run of https://github.com/simonw/datasette/blob/main/.github/workflows/push_docker_tag.yml to publish the new `0.57a1` alpha to Docker Hub.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849788412,https://api.github.com/repos/simonw/datasette/issues/1344,849788412,MDEyOklzc3VlQ29tbWVudDg0OTc4ODQxMg==,9599,simonw,2021-05-27T16:53:28Z,2021-05-27T16:53:28Z,OWNER,(Should also update https://docs.datasette.io/en/stable/contributing.html#release-process with notes on how this works),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/issues/1344#issuecomment-849788159,https://api.github.com/repos/simonw/datasette/issues/1344,849788159,MDEyOklzc3VlQ29tbWVudDg0OTc4ODE1OQ==,9599,simonw,2021-05-27T16:53:04Z,2021-05-27T16:53:04Z,OWNER,I'm going to release a `0.57a1` alpha to Docker Hub using the new buildx mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903986178,Test Datasette Docker images built for different architectures,
https://github.com/simonw/datasette/pull/1319#issuecomment-849785098,https://api.github.com/repos/simonw/datasette/issues/1319,849785098,MDEyOklzc3VlQ29tbWVudDg0OTc4NTA5OA==,9599,simonw,2021-05-27T16:48:21Z,2021-05-27T16:48:21Z,OWNER,"OK, since I have a mechanism for deploying alpha versions now I'm going to merge this and use it to publish 0.57a1.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1343#issuecomment-849784005,https://api.github.com/repos/simonw/datasette/issues/1343,849784005,MDEyOklzc3VlQ29tbWVudDg0OTc4NDAwNQ==,9599,simonw,2021-05-27T16:46:37Z,2021-05-27T16:46:37Z,OWNER,"That worked:
```
 % docker run -it datasetteproject/datasette:0.57a0
Unable to find image 'datasetteproject/datasette:0.57a0' locally
0.57a0: Pulling from datasetteproject/datasette
75646c2fb410: Pull complete 
edf8b70d3cc4: Pull complete 
229daa3ebd94: Pull complete 
65d21ac50a56: Pull complete 
e083ad7cc3ca: Pull complete 
ed4de1431432: Pull complete 
1da5e8b27cca: Pull complete 
Digest: sha256:4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db
Status: Downloaded newer image for datasetteproject/datasette:0.57a0
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849783398,https://api.github.com/repos/simonw/datasette/issues/1343,849783398,MDEyOklzc3VlQ29tbWVudDg0OTc4MzM5OA==,9599,simonw,2021-05-27T16:45:41Z,2021-05-27T16:45:41Z,OWNER,https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now shows this at the top: https://hub.docker.com/layers/datasetteproject/datasette/0.57a0/images/sha256-4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db?context=explore,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849782098,https://api.github.com/repos/simonw/datasette/issues/1343,849782098,MDEyOklzc3VlQ29tbWVudDg0OTc4MjA5OA==,9599,simonw,2021-05-27T16:43:40Z,2021-05-27T16:43:55Z,OWNER,"https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml

<img width=""944"" alt=""Actions_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/119864935-01f15400-bed0-11eb-8979-0a5cc1b6bcb4.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/issues/1343#issuecomment-849781747,https://api.github.com/repos/simonw/datasette/issues/1343,849781747,MDEyOklzc3VlQ29tbWVudDg0OTc4MTc0Nw==,9599,simonw,2021-05-27T16:43:04Z,2021-05-27T16:43:04Z,OWNER,I'm going to try pushing an alpha of https://github.com/simonw/datasette/releases/tag/0.57a0 using this workflow I built a few months ago: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/push_docker_tag.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903978133,Figure out how to publish alpha/beta releases to Docker Hub,
https://github.com/simonw/datasette/pull/1319#issuecomment-849780481,https://api.github.com/repos/simonw/datasette/issues/1319,849780481,MDEyOklzc3VlQ29tbWVudDg0OTc4MDQ4MQ==,9599,simonw,2021-05-27T16:41:03Z,2021-05-27T16:41:03Z,OWNER,It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the `latest` tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/pull/1319#issuecomment-849778373,https://api.github.com/repos/simonw/datasette/issues/1319,849778373,MDEyOklzc3VlQ29tbWVudDg0OTc3ODM3Mw==,9599,simonw,2021-05-27T16:37:50Z,2021-05-27T16:37:50Z,OWNER,Started [a conversation](https://twitter.com/simonw/status/1397953338407223303) about this on Twitter. Ideally I'd like to push an alpha release to Docker Hub so people can start testing these out before they make it into an official non-alpha release.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",881219362,Add Docker multi-arch support with Buildx,
https://github.com/simonw/datasette/issues/1337#issuecomment-849762735,https://api.github.com/repos/simonw/datasette/issues/1337,849762735,MDEyOklzc3VlQ29tbWVudDg0OTc2MjczNQ==,9599,simonw,2021-05-27T16:14:14Z,2021-05-27T16:14:14Z,OWNER,"Here's a demo showing that once you get above 1,000 results you still get the ""..."" but it no longer links to anything: https://fivethirtyeight.datasettes.com/fivethirtyeight/births%2FUS_births_2000-2014_SSA?_facet=births&_facet_size=max","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1332#issuecomment-849761894,https://api.github.com/repos/simonw/datasette/issues/1332,849761894,MDEyOklzc3VlQ29tbWVudDg0OTc2MTg5NA==,9599,simonw,2021-05-27T16:13:07Z,2021-05-27T16:13:07Z,OWNER,Added `?_facet_size=max` in #1337 and made the `...` truncated note link to that - here's a demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",893890496,?_facet_size=X to increase number of facets results on the page,
https://github.com/simonw/datasette/issues/1337#issuecomment-849761508,https://api.github.com/repos/simonw/datasette/issues/1337,849761508,MDEyOklzc3VlQ29tbWVudDg0OTc2MTUwOA==,9599,simonw,2021-05-27T16:12:31Z,2021-05-27T16:12:31Z,OWNER,More exciting demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1337#issuecomment-849759440,https://api.github.com/repos/simonw/datasette/issues/1337,849759440,MDEyOklzc3VlQ29tbWVudDg0OTc1OTQ0MA==,9599,simonw,2021-05-27T16:09:34Z,2021-05-27T16:09:34Z,OWNER,"Demo: https://latest.datasette.io/fixtures/facetable?_facet_size=2&_facet=state&_facet=city_id

![demo-max](https://user-images.githubusercontent.com/9599/119860507-3f071780-becb-11eb-8698-64c8c6112983.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1337#issuecomment-849725291,https://api.github.com/repos/simonw/datasette/issues/1337,849725291,MDEyOklzc3VlQ29tbWVudDg0OTcyNTI5MQ==,9599,simonw,2021-05-27T15:23:33Z,2021-05-27T15:23:33Z,OWNER,Turns out `path_with_replaced_args()` is already exposed to the `table.html` template context: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",898904402,"""More"" link for facets that shows _facet_size=max results",
https://github.com/simonw/datasette/issues/1342#issuecomment-849724915,https://api.github.com/repos/simonw/datasette/issues/1342,849724915,MDEyOklzc3VlQ29tbWVudDg0OTcyNDkxNQ==,9599,simonw,2021-05-27T15:23:01Z,2021-05-27T15:23:01Z,OWNER,Since they are already exposed (but undocumented) this isn't actually a blocker on #1337 like I thought it was. Should still clean this up in the future though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1342#issuecomment-849724108,https://api.github.com/repos/simonw/datasette/issues/1342,849724108,MDEyOklzc3VlQ29tbWVudDg0OTcyNDEwOA==,9599,simonw,2021-05-27T15:21:54Z,2021-05-27T15:21:54Z,OWNER,"Turns out some of these are already being exposed to the template context on an ad-hoc basis.

In `QueryView`: https://github.com/simonw/datasette/blob/2bd9d54b2762c991e11950c22c88c0336158d49b/datasette/views/database.py#L295-L304

In `TableView`: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,
https://github.com/simonw/datasette/issues/1342#issuecomment-849722218,https://api.github.com/repos/simonw/datasette/issues/1342,849722218,MDEyOklzc3VlQ29tbWVudDg0OTcyMjIxOA==,9599,simonw,2021-05-27T15:19:23Z,2021-05-27T15:19:23Z,OWNER,Current implementations: https://github.com/simonw/datasette/blob/51d788114035458d8f860d9ea6d74078e6c0ea0d/datasette/utils/__init__.py#L214-L272,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",903902495,Improve `path_with_replaced_args()` and friends and document them,