html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/datasette/issues/1153#issuecomment-805047117,https://api.github.com/repos/simonw/datasette/issues/1153,805047117,MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==,9599,simonw,2021-03-23T16:30:15Z,2021-03-23T16:46:06Z,OWNER,"https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.

https://github.com/nodeca/js-yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1270#issuecomment-805058241,https://api.github.com/repos/simonw/datasette/issues/1270,805058241,MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==,9599,simonw,2021-03-23T16:45:39Z,2021-03-23T16:45:39Z,OWNER,"I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1153#issuecomment-805056806,https://api.github.com/repos/simonw/datasette/issues/1153,805056806,MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==,9599,simonw,2021-03-23T16:43:38Z,2021-03-23T16:43:38Z,OWNER,"I used this code to get that:
```javascript
var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
div.querySelector('.highlight pre').innerText = jsonVersion;
div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805055291,https://api.github.com/repos/simonw/datasette/issues/1153,805055291,MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==,9599,simonw,2021-03-23T16:41:31Z,2021-03-23T16:41:31Z,OWNER,"One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:

<img width=""655"" alt=""Metadata_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/112183909-ef9b2700-8bbb-11eb-882b-aeb387c29e61.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805050163,https://api.github.com/repos/simonw/datasette/issues/1153,805050163,MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==,9599,simonw,2021-03-23T16:34:35Z,2021-03-23T16:35:32Z,OWNER,"https://docs.datasette.io/en/stable/metadata.html has this example:
```yaml
title: Demonstrating Metadata from YAML
description_html: |-
  <p>This description includes a long HTML string</p>
  <ul>
    <li>YAML is better for embedding HTML strings than JSON!</li>
  </ul>
license: ODbL
license_url: https://opendatacommons.org/licenses/odbl/
databases:
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search
```

I ran this in the browser dev tools:
```javascript
var s = document.createElement('script')
s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
document.head.appendChild(s)
var yamlExample = document.querySelector('.highlight-yaml').textContent);
console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))
```
And got:
```json
{
    ""title"": ""Demonstrating Metadata from YAML"",
    ""description_html"": ""<p>This description includes a long HTML string</p>\n<ul>\n  <li>YAML is better for embedding HTML strings than JSON!</li>\n</ul>"",
    ""license"": ""ODbL"",
    ""license_url"": ""https://opendatacommons.org/licenses/odbl/"",
    ""databases"": {
        ""fixtures"": {
            ""tables"": {
                ""no_primary_key"": {
                    ""hidden"": true
                }
            },
            ""queries"": {
                ""neighborhood_search"": {
                    ""sql"": ""select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;"",
                    ""title"": ""Search neighborhoods"",
                    ""description_html"": ""<p>This demonstrates <em>basic</em> LIKE search""
                }
            }
        }
    }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805042880,https://api.github.com/repos/simonw/datasette/issues/1153,805042880,MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==,9599,simonw,2021-03-23T16:24:32Z,2021-03-23T16:24:32Z,OWNER,... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-805041522,https://api.github.com/repos/simonw/datasette/issues/1153,805041522,MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==,9599,simonw,2021-03-23T16:22:46Z,2021-03-23T16:22:46Z,OWNER,"That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1249#issuecomment-805033155,https://api.github.com/repos/simonw/datasette/issues/1249,805033155,MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==,9599,simonw,2021-03-23T16:12:13Z,2021-03-23T16:12:13Z,OWNER,"Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source

> The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).

See also #1273","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1159#issuecomment-804698315,https://api.github.com/repos/simonw/datasette/issues/1159,804698315,MDEyOklzc3VlQ29tbWVudDgwNDY5ODMxNQ==,552629,lovasoa,2021-03-23T07:58:28Z,2021-03-23T07:58:38Z,NONE,@mroswell Did you try it with more columns ? The display is flexible and  columns get closer  as new ones are added.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/datasette/issues/1153#issuecomment-804640440,https://api.github.com/repos/simonw/datasette/issues/1153,804640440,MDEyOklzc3VlQ29tbWVudDgwNDY0MDQ0MA==,192568,mroswell,2021-03-23T05:58:20Z,2021-03-23T05:58:20Z,CONTRIBUTOR,Could there be a little widget that offers conversion from one to the other? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/pull/1159#issuecomment-804639427,https://api.github.com/repos/simonw/datasette/issues/1159,804639427,MDEyOklzc3VlQ29tbWVudDgwNDYzOTQyNw==,192568,mroswell,2021-03-23T05:56:02Z,2021-03-23T05:56:02Z,CONTRIBUTOR,"With just three facets, I like it, but it does take more horizontal space. Would be nice to have a switch somewhere, enabling either original compact option or this proposed more-readable option. Also some control over word wrap (width setting) and facet spacing. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/datasette/issues/164#issuecomment-804541064,https://api.github.com/repos/simonw/datasette/issues/164,804541064,MDEyOklzc3VlQ29tbWVudDgwNDU0MTA2NA==,192568,mroswell,2021-03-23T02:45:12Z,2021-03-23T02:45:12Z,CONTRIBUTOR,"""datasette skeleton"" feature removed #476","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",280013907,datasette skeleton command for kick-starting database and table metadata,
https://github.com/simonw/datasette/issues/163#issuecomment-804540869,https://api.github.com/repos/simonw/datasette/issues/163,804540869,MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ==,9599,simonw,2021-03-23T02:44:33Z,2021-03-23T02:44:33Z,OWNER,Comments welcome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,Document the querystring argument for setting a different time limit,
https://github.com/simonw/datasette/issues/163#issuecomment-804539729,https://api.github.com/repos/simonw/datasette/issues/163,804539729,MDEyOklzc3VlQ29tbWVudDgwNDUzOTcyOQ==,192568,mroswell,2021-03-23T02:41:14Z,2021-03-23T02:41:14Z,CONTRIBUTOR,"I'm visiting old issues for context while learning datasette. Let me know if okay to make the occasional comment like this one.
querystring argument now located at:
https://docs.datasette.io/en/latest/settings.html#sql-time-limit-ms","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",279547886,Document the querystring argument for setting a different time limit,
https://github.com/simonw/datasette/issues/88#issuecomment-804471733,https://api.github.com/repos/simonw/datasette/issues/88,804471733,MDEyOklzc3VlQ29tbWVudDgwNDQ3MTczMw==,192568,mroswell,2021-03-22T23:46:36Z,2021-03-22T23:46:36Z,CONTRIBUTOR,Google Map API limits seem to prevent https://nhs-england-map.netlify.com from being a working demo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273775212,Add NHS England Hospitals example to wiki,
https://github.com/simonw/datasette/pull/1271#issuecomment-804299406,https://api.github.com/repos/simonw/datasette/issues/1271,804299406,MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==,9599,simonw,2021-03-22T18:36:14Z,2021-03-22T21:49:27Z,OWNER,"This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.

Need to figure out how to communicate with that thread properly.

Also a test that fails in this particular case would be a good idea!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1149#issuecomment-804415619,https://api.github.com/repos/simonw/datasette/issues/1149,804415619,MDEyOklzc3VlQ29tbWVudDgwNDQxNTYxOQ==,192568,mroswell,2021-03-22T21:43:16Z,2021-03-22T21:43:16Z,CONTRIBUTOR,Sounds like a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769520939,Make it easier to theme Datasette with CSS,
https://github.com/simonw/datasette/issues/1249#issuecomment-804406675,https://api.github.com/repos/simonw/datasette/issues/1249,804406675,MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ==,9599,simonw,2021-03-22T21:26:27Z,2021-03-22T21:26:27Z,OWNER,(Without the `apt-get update ...` SpatiaLite line it's 125MB),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804404544,https://api.github.com/repos/simonw/datasette/issues/1249,804404544,MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==,9599,simonw,2021-03-22T21:22:56Z,2021-03-22T21:24:24Z,OWNER,"Final version of Dockerfile which installs the specified version from GitHub:

    docker build . -t datasette-spatialite --build-arg VERSION=0.55

```dockerfile
FROM python:3.9.2-slim-buster as build

# Version of Datasette to install, e.g. 0.55
#   docker build . -t datasette --build-arg VERSION=0.55
ARG VERSION

# software-properties-common provides add-apt-repository
# which we need in order to install a more recent release
# of libsqlite3-mod-spatialite from the sid distribution
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt-get remove -y software-properties-common && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD [""datasette""]
```
Run against 0.55 this produces an image of 262MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804384196,https://api.github.com/repos/simonw/datasette/issues/1249,804384196,MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==,9599,simonw,2021-03-22T20:48:46Z,2021-03-22T20:48:46Z,OWNER,I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804380181,https://api.github.com/repos/simonw/datasette/issues/1249,804380181,MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==,9599,simonw,2021-03-22T20:42:16Z,2021-03-22T20:42:16Z,OWNER,"Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804379644,https://api.github.com/repos/simonw/datasette/issues/1249,804379644,MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==,9599,simonw,2021-03-22T20:41:23Z,2021-03-22T20:41:23Z,OWNER,I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804372977,https://api.github.com/repos/simonw/datasette/issues/1249,804372977,MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==,9599,simonw,2021-03-22T20:30:37Z,2021-03-22T20:30:37Z,OWNER,"I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:
```
root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
	linux-vdso.so.1 (0x00007ffd021f4000)
	libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
	libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
	librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
	libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
	libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
	libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
	libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
	libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
	libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
	libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
	libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
	libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
	libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
	libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
	libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
	libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
	libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
	libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
	libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
	librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
	libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
	libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
	libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
	libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
	libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
	libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
	libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804368372,https://api.github.com/repos/simonw/datasette/issues/1249,804368372,MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==,9599,simonw,2021-03-22T20:22:43Z,2021-03-22T20:22:43Z,OWNER,"```dockerfile
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD [""datasette""]
```
262 MB","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804363687,https://api.github.com/repos/simonw/datasette/issues/1249,804363687,MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==,9599,simonw,2021-03-22T20:15:00Z,2021-03-22T20:15:00Z,OWNER,"```
RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r
```
That dropped it to 265MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804360701,https://api.github.com/repos/simonw/datasette/issues/1249,804360701,MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ==,9599,simonw,2021-03-22T20:10:07Z,2021-03-22T20:10:07Z,OWNER,Adding `--no-install-recommends` dropped it to 275MB,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804347152,https://api.github.com/repos/simonw/datasette/issues/1249,804347152,MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==,9599,simonw,2021-03-22T19:47:56Z,2021-03-22T19:48:03Z,OWNER,I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804344553,https://api.github.com/repos/simonw/datasette/issues/1249,804344553,MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw==,9599,simonw,2021-03-22T19:43:25Z,2021-03-22T19:43:25Z,OWNER,Does `--no-install-recommends` make a difference?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804338678,https://api.github.com/repos/simonw/datasette/issues/1249,804338678,MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==,9599,simonw,2021-03-22T19:33:43Z,2021-03-22T19:33:43Z,OWNER,"Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with 
```
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg
```
Got the size down to 305MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804318314,https://api.github.com/repos/simonw/datasette/issues/1249,804318314,MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA==,9599,simonw,2021-03-22T19:04:30Z,2021-03-22T19:04:30Z,OWNER,Considering the image on Docker Hub right now is `383MB` this is actually an improvement.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804317545,https://api.github.com/repos/simonw/datasette/issues/1249,804317545,MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==,9599,simonw,2021-03-22T19:03:22Z,2021-03-22T19:03:22Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y install software-properties-common && \
    add-apt-repository ""deb http://httpredir.debian.org/debian sid main"" && \
    apt-get update && \
    apt-get -t sid install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

EXPOSE 8001
CMD [""datasette""]
```
Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804310353,https://api.github.com/repos/simonw/datasette/issues/1249,804310353,MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==,9599,simonw,2021-03-22T18:52:12Z,2021-03-22T18:52:12Z,OWNER,"This Dockerfile:
```dockerfile
FROM python:3.9.2-slim-buster as build

# Setup build dependencies
RUN apt update \
    && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
    libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
    && apt clean

RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
    CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
    && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
    && cd freexl-1.0.6 && ./configure && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install

RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install

# Add local code to the image instead of fetching from pypi.
#COPY . /datasette
#RUN pip install /datasette
RUN pip install datasette

FROM python:3.9.2-slim-buster

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
Produced a 448MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-804309510,https://api.github.com/repos/simonw/datasette/issues/1249,804309510,MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==,9599,simonw,2021-03-22T18:50:50Z,2021-03-22T18:50:50Z,OWNER,"Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7

This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/pull/1271#issuecomment-804265042,https://api.github.com/repos/simonw/datasette/issues/1271,804265042,MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==,9599,simonw,2021-03-22T17:45:45Z,2021-03-22T17:45:45Z,OWNER,"I can remove this code too: 
https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1249#issuecomment-804263434,https://api.github.com/repos/simonw/datasette/issues/1249,804263434,MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==,9599,simonw,2021-03-22T17:43:25Z,2021-03-22T17:43:25Z,OWNER,I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1268#issuecomment-804261915,https://api.github.com/repos/simonw/datasette/issues/1268,804261915,MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ==,9599,simonw,2021-03-22T17:41:12Z,2021-03-22T17:41:12Z,OWNER,"Closing this because I've figured out the root of the problem now, and I have a potential solution.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1269#issuecomment-804261610,https://api.github.com/repos/simonw/datasette/issues/1269,804261610,MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==,9599,simonw,2021-03-22T17:40:41Z,2021-03-22T17:40:41Z,OWNER,"#1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/pull/1271#issuecomment-804261103,https://api.github.com/repos/simonw/datasette/issues/1271,804261103,MDEyOklzc3VlQ29tbWVudDgwNDI2MTEwMw==,22429695,codecov[bot],2021-03-22T17:39:57Z,2021-03-22T17:39:57Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=h1) Report
> Merging [#1271](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=desc) (fb2ad7a) into [main](https://codecov.io/gh/simonw/datasette/commit/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b?el=desc) (c4f1ec7) will **decrease** coverage by `0.28%`.
> The diff coverage is `94.28%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1271/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main    #1271      +/-   ##
==========================================
- Coverage   91.51%   91.22%   -0.29%     
==========================================
  Files          34       34              
  Lines        4255     4263       +8     
==========================================
- Hits         3894     3889       -5     
- Misses        361      374      +13     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.41% <94.28%> (-0.52%)` | :arrow_down: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `92.24% <0.00%> (-1.90%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=footer). Last update [c4f1ec7...fb2ad7a](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837956424,Use SQLite conn.interrupt() instead of sqlite_timelimit(),
https://github.com/simonw/datasette/issues/1270#issuecomment-804255633,https://api.github.com/repos/simonw/datasette/issues/1270,804255633,MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw==,9599,simonw,2021-03-22T17:32:02Z,2021-03-22T17:32:08Z,OWNER,Confirmed that the `interrupt()` based cancellation mechanism fixes the SpatiaLite issue in #1268!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1270#issuecomment-803834784,https://api.github.com/repos/simonw/datasette/issues/1270,803834784,MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==,9599,simonw,2021-03-22T07:31:57Z,2021-03-22T16:22:19Z,OWNER,"I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157

I figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837350092,Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler(),
https://github.com/simonw/datasette/issues/1268#issuecomment-803802957,https://api.github.com/repos/simonw/datasette/issues/1268,803802957,MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==,9599,simonw,2021-03-22T06:38:14Z,2021-03-22T06:38:14Z,OWNER,"Also worth trying is to change this code:
```python
     n = 1000 
     if ms < 50: 
         n = 1 
```
What happens with `n = 10` instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1269#issuecomment-803785808,https://api.github.com/repos/simonw/datasette/issues/1269,803785808,MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA==,9599,simonw,2021-03-22T06:00:53Z,2021-03-22T06:00:53Z,OWNER,This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837348479,Don't attempt to run count(*) against virtual tables,
https://github.com/simonw/datasette/issues/1268#issuecomment-803784902,https://api.github.com/repos/simonw/datasette/issues/1268,803784902,MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==,9599,simonw,2021-03-22T05:59:06Z,2021-03-22T05:59:06Z,OWNER,"Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.

That `interrupt` timeout mechanism is worth digging into further.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803782705,https://api.github.com/repos/simonw/datasette/issues/1268,803782705,MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==,9599,simonw,2021-03-22T05:54:19Z,2021-03-22T05:54:19Z,OWNER,"Got two new TILs out of this:

* [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)
* [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803777724,https://api.github.com/repos/simonw/datasette/issues/1268,803777724,MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==,9599,simonw,2021-03-22T05:42:50Z,2021-03-22T05:43:23Z,OWNER,"<img width=""1129"" alt=""tuscany_housenumbers__select___from_sqlite_master_where_sql_like__create_virtual_table__"" src=""https://user-images.githubusercontent.com/9599/111945249-99809380-8a96-11eb-871b-9a1509360bcf.png"">

If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.

The safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.

This would catch things like ` CREATE       virtual      TABLE` which might be missed by a SQL `like` query.   ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803775121,https://api.github.com/repos/simonw/datasette/issues/1268,803775121,MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ==,9599,simonw,2021-03-22T05:36:26Z,2021-03-22T05:36:26Z,OWNER,So one fix could be to avoid running counts for anything that turns out to be a virtual table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774926,https://api.github.com/repos/simonw/datasette/issues/1268,803774926,MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==,9599,simonw,2021-03-22T05:35:56Z,2021-03-22T05:35:56Z,OWNER,That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803774518,https://api.github.com/repos/simonw/datasette/issues/1268,803774518,MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==,9599,simonw,2021-03-22T05:34:57Z,2021-03-22T05:34:57Z,OWNER,"... and sure enough, adding this code fixed the problem:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 3579cce..b466b12 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -224,6 +226,9 @@ class Database:
         # Try to get counts for each table, $limit timeout for each count
         counts = {}
         for table in await self.table_names():
+            if table == ""SpatialIndex"":
+                counts[table] = 0
+                continue
             try:
                 table_count = (
                     await self.execute(
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803773484,https://api.github.com/repos/simonw/datasette/issues/1268,803773484,MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==,9599,simonw,2021-03-22T05:32:29Z,2021-03-22T05:32:29Z,OWNER,"To figure out which SQL query triggers the problem I added this code to write to a log file:
```python
            with sqlite_timelimit(conn, time_limit_ms):
                try:
                    cursor = conn.cursor()
                    with open(""/tmp/sql.log"", ""ab"", buffering=0) as fp:
                        fp.write((""{}: {}\n"".format(sql, params)).encode(""utf-8""))
                    cursor.execute(sql, params if params is not None else {})
```
I had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.

With the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:

```
select count(*) from [idx_civici_geom_parent]: None
select count(*) from [sqlite_stat1]: None
select count(*) from [sqlite_stat3]: None
select count(*) from [SpatialIndex]: None
```
So attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764919,https://api.github.com/repos/simonw/datasette/issues/1268,803764919,MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==,9599,simonw,2021-03-22T05:11:11Z,2021-03-22T05:11:11Z,OWNER,"Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?

https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803764200,https://api.github.com/repos/simonw/datasette/issues/1268,803764200,MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==,9599,simonw,2021-03-22T05:09:13Z,2021-03-22T05:09:13Z,OWNER,"I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762969,https://api.github.com/repos/simonw/datasette/issues/1268,803762969,MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==,9599,simonw,2021-03-22T05:05:51Z,2021-03-22T05:05:51Z,OWNER,I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803762609,https://api.github.com/repos/simonw/datasette/issues/1268,803762609,MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==,9599,simonw,2021-03-22T05:05:00Z,2021-03-22T05:05:00Z,OWNER,"Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a ""Could not attach to process"" error.
```
docker exec --privileged -it 16197781a7b5 bash
# apt-get install gdb python3-dbg
# gdb /usr/bin/python3 -p 20
```
This paused the process. I tried running this:
```
(gdb) py-bt
Traceback (most recent call first):
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 1845, in _run_once
    if handle._cancelled:
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 570, in run_forever
    self._run_once()
  File ""/usr/lib/python3.8/asyncio/base_events.py"", line 603, in run_until_complete
    self.run_forever()
  File ""/usr/local/lib/python3.8/dist-packages/uvicorn/server.py"", line 49, in run
    loop.run_until_complete(self.serve(sockets=sockets))
  File ""/usr/local/lib/python3.8/dist-packages/uvicorn/main.py"", line 386, in run
    server.run()
  File ""/usr/local/lib/python3.8/dist-packages/datasette/cli.py"", line 575, in serve
    uvicorn.run(ds.app(), **uvicorn_kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/usr/local/lib/python3.8/dist-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/usr/local/bin/datasette"", line 8, in <module>
    sys.exit(cli())
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File ""/usr/lib/python3.8/trace.py"", line 450, in runctx
    exec(cmd, globals, locals)
  File ""/usr/lib/python3.8/trace.py"", line 6632, in main
  File ""/usr/lib/python3.8/trace.py"", line 756, in <module>
    main()
  <built-in method exec of module object at remote 0x7f0981a280e0>
  File ""/usr/lib/python3.8/runpy.py"", line 343, in _run_code
  File ""/usr/lib/python3.8/runpy.py"", line 450, in _run_module_as_main
```
Not sure if that's useful or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803759051,https://api.github.com/repos/simonw/datasette/issues/1268,803759051,MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ==,9599,simonw,2021-03-22T04:55:22Z,2021-03-22T04:55:22Z,OWNER,So I think there's a bug in the way the `set_progress_handler()` mechanism works when used in conjunction with SpatiaLite 5.0 on Linux.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758793,https://api.github.com/repos/simonw/datasette/issues/1268,803758793,MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw==,9599,simonw,2021-03-22T04:54:32Z,2021-03-22T04:54:32Z,OWNER,"Hitting http://localhost:8001/tuscany_housenumbers triggers the bug. It gets stuck in a loop that looks like this:

<img width=""381"" alt=""datasette_—_root_16197781a7b5____—_com_docker_cli_◂_docker_run_-it_-p_8001_8001_-v___Dropbox_Development_datasette__mnt_datasette-spatialite_latest_bash_—_195×48_and_getIncidentsGit_—_-zsh_—_162×60"" src=""https://user-images.githubusercontent.com/9599/111942115-e0b75600-8a8f-11eb-8b6d-5c37240ce2f8.png"">

Which looks to me like this code: https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L139-L158","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803758182,https://api.github.com/repos/simonw/datasette/issues/1268,803758182,MDEyOklzc3VlQ29tbWVudDgwMzc1ODE4Mg==,9599,simonw,2021-03-22T04:52:15Z,2021-03-22T04:52:15Z,OWNER,Hitting http://localhost:8001/ successfully shows the homepage (after a lot more scrolling).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803757746,https://api.github.com/repos/simonw/datasette/issues/1268,803757746,MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng==,9599,simonw,2021-03-22T04:50:40Z,2021-03-22T04:51:52Z,OWNER,"Here's a fun debugging trick:

    docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
    root@16197781a7b5:/# python3 -m trace --trace $(which datasette) \
      -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite \
      --load-extension=spatialite

A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python.

After about a minute it's finished starting and gets to this point:

```
selectors.py(452):             if timeout is None:
selectors.py(454):             elif timeout <= 0:
selectors.py(459):                 timeout = math.ceil(timeout * 1e3) * 1e-3
selectors.py(464):             max_ev = max(len(self._fd_to_key), 1)
selectors.py(466):             ready = []
selectors.py(467):             try:
selectors.py(468):                 fd_event_list = self._selector.poll(timeout, max_ev)
```
Now  I can make some HTTP requests against it.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1268#issuecomment-803756495,https://api.github.com/repos/simonw/datasette/issues/1268,803756495,MDEyOklzc3VlQ29tbWVudDgwMzc1NjQ5NQ==,9599,simonw,2021-03-22T04:46:04Z,2021-03-22T04:46:04Z,OWNER,`gdb` may be able to help debug this: https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837308703,Figure out why SpatiaLite 5.0 hangs the database page on Linux,
https://github.com/simonw/datasette/issues/1249#issuecomment-803755698,https://api.github.com/repos/simonw/datasette/issues/1249,803755698,MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA==,9599,simonw,2021-03-22T04:43:02Z,2021-03-22T04:43:02Z,OWNER,I'll spin off a separate ticket to investigate the hang.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1267#issuecomment-803754226,https://api.github.com/repos/simonw/datasette/issues/1267,803754226,MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg==,9599,simonw,2021-03-22T04:37:26Z,2021-03-22T04:37:26Z,OWNER,"Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there.

This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins).

It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",837208901,Update Datasette alternativeto listening with details,
https://github.com/simonw/datasette/issues/1249#issuecomment-803753388,https://api.github.com/repos/simonw/datasette/issues/1249,803753388,MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA==,9599,simonw,2021-03-22T04:34:20Z,2021-03-22T04:35:10Z,OWNER,"Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
(Which produced a 550MB image)

And when I ran Datasette I got that same error where the database listing page hangs!
```
docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803751068,https://api.github.com/repos/simonw/datasette/issues/1249,803751068,MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA==,9599,simonw,2021-03-22T04:26:45Z,2021-03-22T04:26:45Z,OWNER,"Here's why:
```
datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash                                                      
root@3430352ff378:/# datasette
bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750617,https://api.github.com/repos/simonw/datasette/issues/1249,803750617,MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw==,9599,simonw,2021-03-22T04:25:14Z,2021-03-22T04:25:14Z,OWNER,"Got this error attempting to run Datasette (with or without SpatiaLite):
```
standard_init_linux.go:219: exec user process caused: no such file or directory
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803750399,https://api.github.com/repos/simonw/datasette/issues/1249,803750399,MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ==,9599,simonw,2021-03-22T04:24:25Z,2021-03-22T04:24:25Z,OWNER,"I'll try using `ubuntu:20.10` for everything:
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM ubuntu:20.10 as build

RUN apt update && \
    apt install -y python3-pip && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM ubuntu:20.10

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803749831,https://api.github.com/repos/simonw/datasette/issues/1249,803749831,MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ==,9599,simonw,2021-03-22T04:22:35Z,2021-03-22T04:22:35Z,OWNER,"I tried copying just the `mod_spatialite.so` file:
```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
But when I ran Datasette with `--load-extension=spatialite` I got this:
```
  File ""/usr/local/lib/python3.9/site-packages/datasette/database.py"", line 151, in in_thread
    self.ds._prepare_connection(conn, self.name)
  File ""/usr/local/lib/python3.9/site-packages/datasette/app.py"", line 502, in _prepare_connection
    conn.execute(f""SELECT load_extension('{extension}')"")
sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748469,https://api.github.com/repos/simonw/datasette/issues/1249,803748469,MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ==,9599,simonw,2021-03-22T04:17:51Z,2021-03-22T04:17:51Z,OWNER,"... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:

```
datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  File ""/usr/local/lib/python3.9/sqlite3/dbapi2.py"", line 27, in <module>
    from _sqlite3 import *
ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803748158,https://api.github.com/repos/simonw/datasette/issues/1249,803748158,MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA==,9599,simonw,2021-03-22T04:16:57Z,2021-03-22T04:16:57Z,OWNER,"Which is great, because the image on Docker Hub right now is 383MB.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803747701,https://api.github.com/repos/simonw/datasette/issues/1249,803747701,MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ==,9599,simonw,2021-03-22T04:15:40Z,2021-03-22T04:15:40Z,OWNER,"Here's a trick: install SpatiaLite in `ubuntu:20.10` and then copy it into the final `python:3.9.2-slim` image.

```dockerfile
FROM ubuntu:20.10 as install_spatialite

RUN apt update && \
    apt install -y libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/

FROM python:3.9.2-slim as build

RUN pip install datasette

#COPY . /datasette
#RUN pip install /datasette

FROM python:3.9.2-slim

# Copy python dependencies and spatialite libraries
COPY --from=build /usr/local/lib/ /usr/local/lib/
# Copy executables
COPY --from=build /usr/local/bin /usr/local/bin
# Copy spatial extensions
COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

ENV LD_LIBRARY_PATH=/usr/local/lib

EXPOSE 8001
CMD [""datasette""]
```
That produced a 265MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700940,https://api.github.com/repos/simonw/datasette/issues/1249,803700940,MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA==,9599,simonw,2021-03-22T01:14:24Z,2021-03-22T01:14:24Z,OWNER,I tried that with just `python3-pip` (removing `libsqlite3-mod-spatialite`) and got 435MB.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803700626,https://api.github.com/repos/simonw/datasette/issues/1249,803700626,MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg==,9599,simonw,2021-03-22T01:13:04Z,2021-03-22T01:13:04Z,OWNER,"Building a Dockerfile containing just `FROM ubuntu:20.10` gave me `79.5MB`.

Building this one:
```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && \
    apt install -y python3-pip libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/{apt,dpkg,cache,log}/
```
Resulted in a 515MB image.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698983,https://api.github.com/repos/simonw/datasette/issues/1249,803698983,MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw==,9599,simonw,2021-03-22T01:05:36Z,2021-03-22T01:06:23Z,OWNER,"It's pretty big though. I tried this version which avoids copying junk from my laptop in:

```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

RUN pip install datasette

EXPOSE 8001
CMD [""datasette""]
```
And got this:
```
datasette % docker images datasette-spatialite    
REPOSITORY             TAG       IMAGE ID       CREATED         SIZE
datasette-spatialite   latest    0796950653c2   2 seconds ago   528MB
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803698168,https://api.github.com/repos/simonw/datasette/issues/1249,803698168,MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA==,9599,simonw,2021-03-22T01:02:02Z,2021-03-22T01:02:30Z,OWNER,"This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:
```dockerfile
FROM ubuntu:20.10

# Setup build dependencies
RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

# Add local code to the image instead of fetching from pypi.
COPY . /datasette

RUN pip install /datasette

RUN rm -rf /datasette

EXPOSE 8001
CMD [""datasette""]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697546,https://api.github.com/repos/simonw/datasette/issues/1249,803697546,MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng==,9599,simonw,2021-03-22T00:59:47Z,2021-03-22T00:59:47Z,OWNER,"To debug I'm running:

    docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash

This gets me a shell I can use.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803697211,https://api.github.com/repos/simonw/datasette/issues/1249,803697211,MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ==,9599,simonw,2021-03-22T00:58:01Z,2021-03-22T00:58:01Z,OWNER,"I'm messing around with the `Dockerfile` and after each change I'm running:

    docker build . -t datasette-spatialite

And then:

    docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694661,https://api.github.com/repos/simonw/datasette/issues/1249,803694661,MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ==,9599,simonw,2021-03-22T00:46:49Z,2021-03-22T00:46:49Z,OWNER,Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694436,https://api.github.com/repos/simonw/datasette/issues/1249,803694436,MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg==,9599,simonw,2021-03-22T00:46:00Z,2021-03-22T00:46:00Z,OWNER,So I'm going to try `20.10` and see where that gets me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803694359,https://api.github.com/repos/simonw/datasette/issues/1249,803694359,MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ==,9599,simonw,2021-03-22T00:45:47Z,2021-03-22T00:45:47Z,OWNER,https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using `python:3.9-slim-buster` or `ubuntu:20.04` - but 20.04 is focal which still has SpatiaLite `4.3.0a-6build1` - It's `20.10` that has 5.0: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803693181,https://api.github.com/repos/simonw/datasette/issues/1249,803693181,MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ==,9599,simonw,2021-03-22T00:41:02Z,2021-03-22T00:41:02Z,OWNER,Debian sid has it too: https://packages.debian.org/sid/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803692673,https://api.github.com/repos/simonw/datasette/issues/1249,803692673,MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw==,9599,simonw,2021-03-22T00:38:42Z,2021-03-22T00:38:42Z,OWNER,Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-803691236,https://api.github.com/repos/simonw/datasette/issues/1249,803691236,MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg==,9599,simonw,2021-03-22T00:32:03Z,2021-03-22T00:32:03Z,OWNER,"Here's something odd: when I run `datasette tuscany_housenumbers.sqlite --load-extension=spatialite` on macOS against SpatiaLite installed using Homebrew (which reports `""spatialite"": ""5.0.0""` on the `/-/versions` page) I don't get any weird errors at all, everything works fine.

But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages.

This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1259#issuecomment-803674728,https://api.github.com/repos/simonw/datasette/issues/1259,803674728,MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA==,9599,simonw,2021-03-21T22:55:31Z,2021-03-21T22:55:31Z,OWNER,CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/647#issuecomment-803673225,https://api.github.com/repos/simonw/datasette/issues/647,803673225,MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ==,9599,simonw,2021-03-21T22:44:19Z,2021-03-21T22:44:19Z,OWNER,"Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature.

I'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy.

I may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531755959,Move hashed URL mode out to a plugin,
https://github.com/simonw/datasette/issues/942#issuecomment-803631102,https://api.github.com/repos/simonw/datasette/issues/942,803631102,MDEyOklzc3VlQ29tbWVudDgwMzYzMTEwMg==,192568,mroswell,2021-03-21T17:48:42Z,2021-03-21T17:48:42Z,CONTRIBUTOR,"I like this idea. Though it might be nice to have some kind of automated system from database to file, so that developers could easily track diffs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803502424,https://api.github.com/repos/simonw/sqlite-utils/issues/249,803502424,MDEyOklzc3VlQ29tbWVudDgwMzUwMjQyNA==,36287,prabhur,2021-03-21T02:43:32Z,2021-03-21T02:43:32Z,NONE,"> Did you run `enable-fts` before you inserted the data?
> 
> If so you'll need to run `populate-fts` after the insert to populate the FTS index.
> 
> A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.

Wow. Wasn't expecting a response this quick, especially during a weekend. :-) Sincerely appreciate it.
I tried the `populate-fts` and that did the trick. My bad for not consulting the docs again. I think I forgot to add that step when I automated the workflow.
Thanks for the suggestion. I'll close this issue. Have a great weekend and many many thanks for creating these suite of tools around sqlite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836963850,Full text search possibly broken?,
https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756,https://api.github.com/repos/simonw/sqlite-utils/issues/249,803501756,MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng==,9599,simonw,2021-03-21T02:33:45Z,2021-03-21T02:33:45Z,OWNER,"Did you run `enable-fts` before you inserted the data?

If so you'll need to run `populate-fts` after the insert to populate the FTS index.

A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836963850,Full text search possibly broken?,
https://github.com/simonw/datasette/issues/1261#issuecomment-803499509,https://api.github.com/repos/simonw/datasette/issues/1261,803499509,MDEyOklzc3VlQ29tbWVudDgwMzQ5OTUwOQ==,812795,brimstone,2021-03-21T02:06:43Z,2021-03-21T02:06:43Z,NONE,I can confirm 0.9.2 fixes the problem. Thanks for the fast response!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/878#issuecomment-803473015,https://api.github.com/repos/simonw/datasette/issues/878,803473015,MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ==,9599,simonw,2021-03-20T22:33:05Z,2021-03-20T22:33:05Z,OWNER,"Things this mechanism needs to be able to support:

- Returning a default JSON representation
- Defining ""extra"" JSON representations blocks, which can be requested using `?_extra=`
- Returning rendered HTML, based on the default JSON + one or more extras + a template
- Using Datasette output renderers to return e.g. CSV data
- Potentially also supporting streaming output renderers for streaming CSV/TSV/JSON-nl etc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472595,https://api.github.com/repos/simonw/datasette/issues/878,803472595,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ==,9599,simonw,2021-03-20T22:28:12Z,2021-03-20T22:28:12Z,OWNER,"Another idea I had: a view is a class that takes the `datasette` instance in its constructor, and defines a `__call__` method that accepts a request and returns a response. Except `await __call__` looks like it might be a bit messy, discussion in https://github.com/encode/starlette/issues/886","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803472278,https://api.github.com/repos/simonw/datasette/issues/878,803472278,MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA==,9599,simonw,2021-03-20T22:25:04Z,2021-03-20T22:25:04Z,OWNER,"I came up with a slightly wild idea for this that would involve pytest-style dependency injection.

Prototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7

Copying from my private notes:

> Use the lazy evaluated DI mechanism to break up table view into different pieces eg for faceting
> 
> Use that to solve JSON vs HTML views
> 
> Oh here's an idea: what if the various components of the table view were each defined  as async functions.... and then executed using asyncio.gather in order to run the SQL queries in parallel? Then try benchmarking with different numbers of threads?
> 
> The async_call_with_arguments function could do this automatically for any awaitable dependencies
> 
> This would give me massively parallel dependency injection
> 
> (I could build an entire framework around this and call it c64)
> 
> Idea: arguments called eg ""count"" are executed and the result passed to the function. If called count_fn then a reference to the not-yet-called function is passed instead 
> 
> I'm not going to completely combine the views mechanism and the render hooks. Instead, the core view will define a bunch of functions used to compose the page and the render hook will have conditional access to those functions - which will otherwise be asyncio.gather executed directly by the HTML page version
> 
> Using asyncio.gather to execute facets and suggest facets in parallel would be VERY interesting 
> 
> suggest facets should be VERY cachable - doesn't matter if it's wrong unlike actual facets themselves
> 
> What if all Datasette views were defined in terms of dependency injection - and those dependency functions could themselves depend on others just like pytest fixtures. Everything would become composable and async stuff could execute in parallel
> 
> FURTHER IDEA: use this for the ?_extra= mechanism as well.
> 
> Any view in Datasette can be defined as a collection of named keys. Each of those keys maps to a function or an async function that accepts as input other named keys, using DI to handle them.
> 
> The HTML view is a defined function. So are the other outputs.
> 
> Default original inputs include “request” and “datasette”.
> 
> So… maybe a view function is a class methods that use DI. One of those methods as an .html() method used for the default page.
> 
> Output formats are a bit more complicated because they are supposed to be defined separately in plugins. They are unified across query, row and table though.
> 
> I’m going to try breaking up the TableView to see what happens.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/878#issuecomment-803471917,https://api.github.com/repos/simonw/datasette/issues/878,803471917,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw==,9599,simonw,2021-03-20T22:21:33Z,2021-03-20T22:21:33Z,OWNER,"This has been blocking things for too long.

If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",648435885,"New pattern for views that return either JSON or HTML, available for plugins",
https://github.com/simonw/datasette/issues/1258#issuecomment-803471702,https://api.github.com/repos/simonw/datasette/issues/1258,803471702,MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg==,9599,simonw,2021-03-20T22:19:39Z,2021-03-20T22:19:39Z,OWNER,"This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between `?sql=...&param=` and `?sql=` with no `param` specified at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/datasette/issues/782#issuecomment-803469623,https://api.github.com/repos/simonw/datasette/issues/782,803469623,MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw==,9599,simonw,2021-03-20T22:01:23Z,2021-03-20T22:01:23Z,OWNER,"I'm going to keep `?_shape=array` working on the assumption that many existing uses of the Datasette API are already using that option, so it would be nice not to break them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1261#issuecomment-803468314,https://api.github.com/repos/simonw/datasette/issues/1261,803468314,MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA==,9599,simonw,2021-03-20T21:48:48Z,2021-03-20T21:48:48Z,OWNER,That's fixed in this release of `datasette-publish-vercel`: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.9.2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1261#issuecomment-803466868,https://api.github.com/repos/simonw/datasette/issues/1261,803466868,MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA==,9599,simonw,2021-03-20T21:36:06Z,2021-03-20T21:36:06Z,OWNER,"This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28

I'm looking at a fix for that now, so watch that issue for updates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",832092321,Some links aren't properly URL encoded.,
https://github.com/simonw/datasette/issues/1266#issuecomment-803466730,https://api.github.com/repos/simonw/datasette/issues/1266,803466730,MDEyOklzc3VlQ29tbWVudDgwMzQ2NjczMA==,9599,simonw,2021-03-20T21:35:00Z,2021-03-20T21:35:00Z,OWNER,https://docs.datasette.io/en/latest/internals.html#returning-a-response-with-asgi-send-send,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836273891,Documentation for Response.asgi_send(send) method,
https://github.com/simonw/datasette/issues/1265#issuecomment-803160804,https://api.github.com/repos/simonw/datasette/issues/1265,803160804,MDEyOklzc3VlQ29tbWVudDgwMzE2MDgwNA==,468612,yunzheng,2021-03-19T22:05:12Z,2021-03-19T22:05:12Z,NONE,Wow that was fast! Thanks for this very cool project and quick update! 👍 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1265#issuecomment-803130332,https://api.github.com/repos/simonw/datasette/issues/1265,803130332,MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg==,9599,simonw,2021-03-19T21:03:09Z,2021-03-19T21:03:09Z,OWNER,This is now available in `datasette-auth-passwords` 0.4! https://github.com/simonw/datasette-auth-passwords/releases/tag/0.4,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1265#issuecomment-802923254,https://api.github.com/repos/simonw/datasette/issues/1265,802923254,MDEyOklzc3VlQ29tbWVudDgwMjkyMzI1NA==,7476523,bobwhitelock,2021-03-19T15:39:15Z,2021-03-19T15:39:15Z,CONTRIBUTOR,"It doesn't use basic auth, but you can put a whole datasette instance, or parts of this, behind a username/password prompt using https://github.com/simonw/datasette-auth-passwords","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",836123030,Support for HTTP Basic Authentication,
https://github.com/simonw/datasette/issues/1262#issuecomment-802164134,https://api.github.com/repos/simonw/datasette/issues/1262,802164134,MDEyOklzc3VlQ29tbWVudDgwMjE2NDEzNA==,19328961,henry501,2021-03-18T17:55:00Z,2021-03-18T17:55:00Z,NONE,"Thanks for the comments. I'll take a look at the documentation to familiarize myself, as I haven't tried to write any plugins yet. With some luck I might be ready to write it when the hook is implemented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/datasette/issues/1262#issuecomment-802099264,https://api.github.com/repos/simonw/datasette/issues/1262,802099264,MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA==,9599,simonw,2021-03-18T16:43:09Z,2021-03-18T16:43:09Z,OWNER,"I often find myself wanting this too, when I'm exploring a new dataset.

i agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way.

I'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables `order by random()` could enable a lot of other useful plugin features too.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/datasette/issues/1262#issuecomment-802095132,https://api.github.com/repos/simonw/datasette/issues/1262,802095132,MDEyOklzc3VlQ29tbWVudDgwMjA5NTEzMg==,7476523,bobwhitelock,2021-03-18T16:37:45Z,2021-03-18T16:37:45Z,CONTRIBUTOR,"This sounds like a good use case for a plugin, since this will only be useful for a subset of Datasette users. It shouldn't be too difficult to add a button to do this with the available plugin hooks - have you taken a look at https://docs.datasette.io/en/latest/writing_plugins.html?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",834602299,Plugin hook that could support 'order by random()' for table view,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-802032152,https://api.github.com/repos/simonw/sqlite-utils/issues/159,802032152,MDEyOklzc3VlQ29tbWVudDgwMjAzMjE1Mg==,1025224,limar,2021-03-18T15:42:52Z,2021-03-18T15:42:52Z,NONE,"I confirm the bug. Happens for me in version 3.6. I use the call to delete all the records:
`table.delete_where()`
This does not delete anything. 

I see that `delete()` method DOES use context manager `with self.db.conn:` which should help. You may want to align the code of both methods.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-801816980,https://api.github.com/repos/simonw/sqlite-utils/issues/246,801816980,MDEyOklzc3VlQ29tbWVudDgwMTgxNjk4MA==,37962604,polyrand,2021-03-18T10:40:32Z,2021-03-18T10:43:04Z,NONE,"I have found a similar problem, but I only when using that type of query (with `*` for doing a prefix search). I'm also building something on top of FTS5/sqlite-utils, and the way I decided to handle it was creating a specific function for prefixes. According to [the docs](https://www2.sqlite.org/fts5.html#fts5_prefix_queries), the query can be done in this 2 ways:

```sql
... MATCH '""one two thr"" * '
... MATCH 'one + two + thr*'
```

I thought I could build a query like the first one using this function:

```python
def prefix(query: str):
    return f'""{query}"" *'
```

And then I use the output of that function as the query parameter for the standard `.search()` method in sqlite-utils.

However, my use case is different because I'm the one ""deciding"" when to use a prefix search, not the end user. I also haven't done many tests, but maybe you found that useful. One thing I could think of is checking if the query has an `*` at the end, remove it and build the prefix query using the function above.

This is just for prefix queries, I think having the escaping function is still useful for other use cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings,
https://github.com/simonw/sqlite-utils/issues/246#issuecomment-799479175,https://api.github.com/repos/simonw/sqlite-utils/issues/246,799479175,MDEyOklzc3VlQ29tbWVudDc5OTQ3OTE3NQ==,9599,simonw,2021-03-15T14:47:31Z,2021-03-15T14:47:31Z,OWNER,"This is a smart feature. I have something that does this in Datasette, extracting it out to `sqlite-utils` makes a lot of sense.

https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L818-L829","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831751367,Escaping FTS search strings,
https://github.com/simonw/datasette/issues/236#issuecomment-799066252,https://api.github.com/repos/simonw/datasette/issues/236,799066252,MDEyOklzc3VlQ29tbWVudDc5OTA2NjI1Mg==,9599,simonw,2021-03-15T03:34:52Z,2021-03-15T03:34:52Z,OWNER,"Yeah the Lambda Docker stuff is pretty odd - you still don't get to speak HTTP, you have to speak their custom event protocol instead.

https://github.com/glassechidna/serverlessish looks interesting here - it adds a proxy inside the container which allows your existing HTTP Docker image to run within Docker-on-Lambda. I've not tried it out yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/236#issuecomment-799003172,https://api.github.com/repos/simonw/datasette/issues/236,799003172,MDEyOklzc3VlQ29tbWVudDc5OTAwMzE3Mg==,21148,jacobian,2021-03-14T23:42:57Z,2021-03-14T23:42:57Z,CONTRIBUTOR,"Oh, and the container image can be up to 10GB, so the EFS step might not be needed except for pretty big stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/issues/236#issuecomment-799002993,https://api.github.com/repos/simonw/datasette/issues/236,799002993,MDEyOklzc3VlQ29tbWVudDc5OTAwMjk5Mw==,21148,jacobian,2021-03-14T23:41:51Z,2021-03-14T23:41:51Z,CONTRIBUTOR,"Now that [Lambda supports Docker](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/), this probably is a bit easier and may be able to build on top of the existing package command.

There are weirdnesses in how the command actually gets invoked; the [aws-lambda-python image](https://hub.docker.com/r/amazon/aws-lambda-python) shows a bit of that. So Datasette would probably need some sort of Lambda-specific entry point to make this work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/simonw/datasette/pull/1260#issuecomment-798913090,https://api.github.com/repos/simonw/datasette/issues/1260,798913090,MDEyOklzc3VlQ29tbWVudDc5ODkxMzA5MA==,22429695,codecov[bot],2021-03-14T14:01:30Z,2021-03-14T14:01:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=h1) Report
> Merging [#1260](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=desc) (90f5fb6) into [main](https://codecov.io/gh/simonw/datasette/commit/8e18c7943181f228ce5ebcea48deb59ce50bee1f?el=desc) (8e18c79) will **not change** coverage.
> The diff coverage is `83.33%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1260/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1260   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <0.00%> (ø)` | |
| [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <50.00%> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (ø)` | |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <100.00%> (ø)` | |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.88% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=footer). Last update [8e18c79...90f5fb6](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",831163537,Fix: code quality issues,
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798468572,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,798468572,MDEyOklzc3VlQ29tbWVudDc5ODQ2ODU3Mg==,1234956,n8henrie,2021-03-13T14:47:31Z,2021-03-13T14:47:31Z,NONE,"Ok, new PR works. I'm not `git` enough so I just force-pushed over the old one.

I still end up with a lot of activities that are missing an `id` and therefore skipped (since this is used as the primary key). For example:

```
{'workoutActivityType': 'HKWorkoutActivityTypeRunning', 'duration': '35.31666666666667', 'durationUnit': 'min', 'totalDistance': '4.010870267636999', 'totalDistanceUnit': 'mi', 'totalEnergyBurned': '660.3516235351562', 'totalEnergyBurnedUnit': 'Cal', 'sourceName': 'Strava', 'sourceVersion': '22810', 'creationDate': '2020-07-16 13:38:26 -0700', 'startDate': '2020-07-16 06:38:26 -0700', 'endDate': '2020-07-16 07:13:45 -0700'}
```

I also end up with some unhappy characters (in the skipped events), such as: `'sourceName': 'Nathan’s Apple\xa0Watch',`.

But it's successfully making it through the file, and the resulting db opens in datasette, so I'd call that progress.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798436026,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,798436026,MDEyOklzc3VlQ29tbWVudDc5ODQzNjAyNg==,1234956,n8henrie,2021-03-13T14:23:16Z,2021-03-13T14:23:16Z,NONE,"This PR allows my import to succeed.

It looks like some events don't have an `id`, but do have `HKExternalUUID` (which gets turned into `metadata_HKExternalUUID`), so I use this as a fallback.

If a record has neither of these, I changed it to just print the record (for debugging) and `return`.

For some odd reason this ran fine at first, and now (after removing the generated db and trying again) I'm getting a different error (duplicate column name).

Looks like it may have run when I had two successive runs without remembering to delete the db in between. Will try to refactor.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/simonw/datasette/issues/1259#issuecomment-797827038,https://api.github.com/repos/simonw/datasette/issues/1259,797827038,MDEyOklzc3VlQ29tbWVudDc5NzgyNzAzOA==,9599,simonw,2021-03-13T00:15:40Z,2021-03-13T00:15:40Z,OWNER,"If all of the facets were being calculated in a single query, I'd be willing to bump the facet time limit up to something a lot higher, maybe even a full second. There's a chance that could work amazingly well with a materialized CTE.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1259#issuecomment-797804869,https://api.github.com/repos/simonw/datasette/issues/1259,797804869,MDEyOklzc3VlQ29tbWVudDc5NzgwNDg2OQ==,9599,simonw,2021-03-12T23:05:05Z,2021-03-12T23:05:05Z,OWNER,"I wonder if I could optimize facet suggestion in the same way?

One challenge: the query time limit will apply to the full CTE query, not to the individual columns.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1259#issuecomment-797801075,https://api.github.com/repos/simonw/datasette/issues/1259,797801075,MDEyOklzc3VlQ29tbWVudDc5NzgwMTA3NQ==,9599,simonw,2021-03-12T22:53:56Z,2021-03-12T22:55:16Z,OWNER,"OK, a better comparison:

https://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aowner+as+%28%0D%0Aselect%0D%0A++%27owner%27+as+col%2C+owner+as+value%2C+count%28*%29+as+c+from+data+group+by+owner%0D%0A++order+by+c+desc+limit+31%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long%0D%0Aunion+select+*+from+owner+order+by+col%2C+c+desc calculates facets against three columns. It takes **78.5ms** (and 34.5ms when I refreshed it, presumably after warming some SQLite caches of some sort).

https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=country_long&_facet=primary_fuel&_trace=1&_size=0 shows those facets with size=0 on the SQL query - and shows a SQL trace at the bottom of the page.

The country_long facet query takes 45.36ms, owner takes 38.45ms, primary_fuel takes 49.04ms - so a total of 132.85ms

That's against https://global-power-plants.datasettes.com/-/versions says SQLite 3.27.3 - so even on a SQLite version that doesn't materialize the CTEs there's a significant performance boost to doing all three facets in a single CTE query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1259#issuecomment-797790017,https://api.github.com/repos/simonw/datasette/issues/1259,797790017,MDEyOklzc3VlQ29tbWVudDc5Nzc5MDAxNw==,9599,simonw,2021-03-12T22:22:12Z,2021-03-12T22:22:12Z,OWNER,"https://sqlite.org/lang_with.html

>  Prior to SQLite 3.35.0, all CTEs where treated as if the NOT MATERIALIZED phrase was present

It looks like this optimization is completely unavailable on SQLite prior to 3.35.0 (released 12th March 2021). But I could still rewrite the faceting to work in this way, using the exact same SQL - it would just be significantly faster on 3.35.0+ (assuming it's actually faster in practice - would need to benchmark).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",830567275,Research using CTEs for faster facet counts,
https://github.com/simonw/datasette/issues/1193#issuecomment-797159434,https://api.github.com/repos/simonw/datasette/issues/1193,797159434,MDEyOklzc3VlQ29tbWVudDc5NzE1OTQzNA==,9599,simonw,2021-03-12T01:01:54Z,2021-03-12T01:01:54Z,OWNER,"DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python

```python
import duckdb
con = duckdb.connect(database=""/tmp/blah.db"", read_only=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787173276,Research plugin hook for alternative database backends,
https://github.com/simonw/datasette/issues/1250#issuecomment-797159221,https://api.github.com/repos/simonw/datasette/issues/1250,797159221,MDEyOklzc3VlQ29tbWVudDc5NzE1OTIyMQ==,9599,simonw,2021-03-12T01:01:17Z,2021-03-12T01:01:17Z,OWNER,This is a duplicate of #1193.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824067604,Research: Plugin hook for alternative database connections,
https://github.com/simonw/datasette/issues/670#issuecomment-797158641,https://api.github.com/repos/simonw/datasette/issues/670,797158641,MDEyOklzc3VlQ29tbWVudDc5NzE1ODY0MQ==,9599,simonw,2021-03-12T00:59:49Z,2021-03-12T00:59:49Z,OWNER,"> Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials?

It looks like the answer to this is yes - I'll need users to setup read-only credentials. Here's a TIL about that: https://til.simonwillison.net/postgresql/read-only-postgresql-user","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL,
https://github.com/simonw/datasette/pull/1211#issuecomment-796854370,https://api.github.com/repos/simonw/datasette/issues/1211,796854370,MDEyOklzc3VlQ29tbWVudDc5Njg1NDM3MA==,9599,simonw,2021-03-11T16:15:29Z,2021-03-11T16:15:29Z,OWNER,Thanks very much for this - it's really comprehensive. I need to bake some of these patterns into my coding habits better!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797649915,Use context manager instead of plain open,
https://github.com/simonw/datasette/issues/838#issuecomment-795950636,https://api.github.com/repos/simonw/datasette/issues/838,795950636,MDEyOklzc3VlQ29tbWVudDc5NTk1MDYzNg==,79913,tsibley,2021-03-10T19:24:13Z,2021-03-10T19:24:13Z,NONE,"I think this could be solved by one of:

1. Stop generating absolute URLs, e.g. ones that include an origin. Relative URLs with absolute paths are fine, as long as they take `base_url` into account (as they do now, yay!).
2. Extend `base_url` to include the expected frontend origin, and then use that information when generating absolute URLs.
3. Document which HTTP headers the reverse proxy should set (e.g. the `X-Forwarded-*` family of conventional headers) to pass the frontend origin information to Datasette, and then use that information when generating absolute URLs.

Option 1 seems like the easiest to me, if you can get away with never having to generate an absolute URL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/838#issuecomment-795939998,https://api.github.com/repos/simonw/datasette/issues/838,795939998,MDEyOklzc3VlQ29tbWVudDc5NTkzOTk5OA==,79913,tsibley,2021-03-10T19:16:55Z,2021-03-10T19:16:55Z,NONE,"Nod. The problem with the tests is that they're ignoring the origin (hostname, port) of links. In a reverse proxy situation, the frontend request origin is different than the backend request origin. The problem is Datasette generates links with the backend request origin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/838#issuecomment-795918377,https://api.github.com/repos/simonw/datasette/issues/838,795918377,MDEyOklzc3VlQ29tbWVudDc5NTkxODM3Nw==,9599,simonw,2021-03-10T19:01:48Z,2021-03-10T19:01:48Z,OWNER,The biggest challenge here I think is to replicate the exact situation here this happens in a Python unit test. The fix should be easy once we have a test in place.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/838#issuecomment-795895436,https://api.github.com/repos/simonw/datasette/issues/838,795895436,MDEyOklzc3VlQ29tbWVudDc5NTg5NTQzNg==,9599,simonw,2021-03-10T18:44:46Z,2021-03-10T18:44:57Z,OWNER,Let's reopen this.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/issues/838#issuecomment-795893813,https://api.github.com/repos/simonw/datasette/issues/838,795893813,MDEyOklzc3VlQ29tbWVudDc5NTg5MzgxMw==,79913,tsibley,2021-03-10T18:43:39Z,2021-03-10T18:43:39Z,NONE,"@simonw Unfortunately this issue as I reported it is not actually solved in version 0.55.

Every link which is returned by the `Datasette.absolute_url` method is still wrong, because it uses the request URL as the base. This still includes the suggested facet links and pagination links.

What I wrote originally still stands:

> Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include `base_url` or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at `http://localhost:8001`.
> 
> I looked into this a little in the source code, and it seems to be an issue anywhere `request.url` or `request.path` is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the `path_with_…` family of utility functions and the `Datasette.absolute_url` method.

 Would you prefer to re-open this issue or have me create a new one?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set,
https://github.com/simonw/datasette/pull/1254#issuecomment-795870524,https://api.github.com/repos/simonw/datasette/issues/1254,795870524,MDEyOklzc3VlQ29tbWVudDc5NTg3MDUyNA==,9599,simonw,2021-03-10T18:27:45Z,2021-03-10T18:27:45Z,OWNER,What other breaks did you spot?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1256#issuecomment-795869144,https://api.github.com/repos/simonw/datasette/issues/1256,795869144,MDEyOklzc3VlQ29tbWVudDc5NTg2OTE0NA==,9599,simonw,2021-03-10T18:26:46Z,2021-03-10T18:26:46Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",827341657,Minor type in IP adress,
https://github.com/simonw/datasette/pull/1256#issuecomment-795112935,https://api.github.com/repos/simonw/datasette/issues/1256,795112935,MDEyOklzc3VlQ29tbWVudDc5NTExMjkzNQ==,6371750,JBPressac,2021-03-10T08:59:45Z,2021-03-10T08:59:45Z,CONTRIBUTOR,"Sorry, I meant ""minor typo"" not ""minor type"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",827341657,Minor type in IP adress,
https://github.com/simonw/datasette/pull/1256#issuecomment-795085921,https://api.github.com/repos/simonw/datasette/issues/1256,795085921,MDEyOklzc3VlQ29tbWVudDc5NTA4NTkyMQ==,22429695,codecov[bot],2021-03-10T08:35:17Z,2021-03-10T08:35:17Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=h1) Report
> Merging [#1256](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=desc) (4eef524) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1256/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1256   +/-   ##
=======================================
  Coverage   91.56%   91.56%           
=======================================
  Files          34       34           
  Lines        4244     4244           
=======================================
  Hits         3886     3886           
  Misses        358      358           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=footer). Last update [d0fd833...4eef524](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",827341657,Minor type in IP adress,
https://github.com/simonw/datasette/pull/1254#issuecomment-794518438,https://api.github.com/repos/simonw/datasette/issues/1254,794518438,MDEyOklzc3VlQ29tbWVudDc5NDUxODQzOA==,3200608,durkie,2021-03-09T22:04:23Z,2021-03-09T22:04:23Z,NONE,"Dang, you're absolutely right. Spatialite 5.0 had been working fine for a plugin I was developing, but it apparently is broken in several other ways.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1254#issuecomment-794441034,https://api.github.com/repos/simonw/datasette/issues/1254,794441034,MDEyOklzc3VlQ29tbWVudDc5NDQ0MTAzNA==,22429695,codecov[bot],2021-03-09T20:54:18Z,2021-03-09T21:12:15Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=h1) Report
> Merging [#1254](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=desc) (b103204) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **decrease** coverage by `0.04%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1254/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main    #1254      +/-   ##
==========================================
- Coverage   91.56%   91.51%   -0.05%     
==========================================
  Files          34       34              
  Lines        4244     4244              
==========================================
- Hits         3886     3884       -2     
- Misses        358      360       +2     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1254/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=footer). Last update [d0fd833...b103204](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1254#issuecomment-794443710,https://api.github.com/repos/simonw/datasette/issues/1254,794443710,MDEyOklzc3VlQ29tbWVudDc5NDQ0MzcxMA==,3200608,durkie,2021-03-09T20:56:45Z,2021-03-09T20:56:45Z,NONE,Oh wow I didn't even see that you had opened an issue about this so recently. I'll check on `/dbname` and report back.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1254#issuecomment-794439632,https://api.github.com/repos/simonw/datasette/issues/1254,794439632,MDEyOklzc3VlQ29tbWVudDc5NDQzOTYzMg==,9599,simonw,2021-03-09T20:53:02Z,2021-03-09T20:53:02Z,OWNER,Thanks for catching that documentation update!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1254#issuecomment-794437715,https://api.github.com/repos/simonw/datasette/issues/1254,794437715,MDEyOklzc3VlQ29tbWVudDc5NDQzNzcxNQ==,9599,simonw,2021-03-09T20:51:19Z,2021-03-09T20:51:19Z,OWNER,Did you see my note on https://github.com/simonw/datasette/issues/1249#issuecomment-792384382 about a weird issue I was having with the `/dbname` page hanging the server? Have you seen anything like that in your work here?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",826613352,Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions,
https://github.com/simonw/datasette/pull/1252#issuecomment-793308483,https://api.github.com/repos/simonw/datasette/issues/1252,793308483,MDEyOklzc3VlQ29tbWVudDc5MzMwODQ4Mw==,22429695,codecov[bot],2021-03-09T03:06:10Z,2021-03-09T03:06:10Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=h1) Report
> Merging [#1252](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=desc) (d22aa32) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **decrease** coverage by `0.04%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1252/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main    #1252      +/-   ##
==========================================
- Coverage   91.56%   91.51%   -0.05%     
==========================================
  Files          34       34              
  Lines        4244     4244              
==========================================
- Hits         3886     3884       -2     
- Misses        358      360       +2     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1252/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=footer). Last update [d0fd833...d22aa32](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",825217564,Add back styling to lists within table cells (fixes #1141),
https://github.com/simonw/datasette/issues/1250#issuecomment-792386484,https://api.github.com/repos/simonw/datasette/issues/1250,792386484,MDEyOklzc3VlQ29tbWVudDc5MjM4NjQ4NA==,9599,simonw,2021-03-08T00:29:06Z,2021-03-08T00:29:06Z,OWNER,"DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python

```python
import duckdb
con = duckdb.connect(database=""/tmp/blah.db"", read_only=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824067604,Research: Plugin hook for alternative database connections,
https://github.com/simonw/datasette/issues/1248#issuecomment-792385274,https://api.github.com/repos/simonw/datasette/issues/1248,792385274,MDEyOklzc3VlQ29tbWVudDc5MjM4NTI3NA==,9599,simonw,2021-03-08T00:25:10Z,2021-03-08T00:25:10Z,OWNER,"It's not possible yet, unfortunately. This came up on the forums recently: https://github.com/simonw/datasette/discussions/968

I'm leaning further towards making the database connection layer itself work via a plugin hook, which would open up the possibility of supporting DuckDB and other databases as well. I've not committed to doing this yet though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",823035080,duckdb database (very low performance in SQLite),
https://github.com/simonw/datasette/issues/1249#issuecomment-792384854,https://api.github.com/repos/simonw/datasette/issues/1249,792384854,MDEyOklzc3VlQ29tbWVudDc5MjM4NDg1NA==,9599,simonw,2021-03-08T00:23:38Z,2021-03-08T00:23:38Z,OWNER,One reason to prioritize this issue: Homebrew upgraded to SpatiaLite 5.0 recently https://formulae.brew.sh/formula/spatialite-tools and as a result SpatiaLite database created on my laptop don't appear to be compatible with Datasette when published using `datasette publish`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-792384382,https://api.github.com/repos/simonw/datasette/issues/1249,792384382,MDEyOklzc3VlQ29tbWVudDc5MjM4NDM4Mg==,9599,simonw,2021-03-08T00:22:02Z,2021-03-08T00:22:02Z,OWNER,"I tried this patch against `Dockerfile`:
```diff
diff --git a/Dockerfile b/Dockerfile
index f4b1414..dd659e1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,25 +1,26 @@
-FROM python:3.7.10-slim-stretch as build
+FROM python:3.9.2-slim-buster as build
 
 # Setup build dependencies
 RUN apt update \
-&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
- && apt clean
+  && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
+  libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
+  && apt clean
 
-
-RUN wget ""https://www.sqlite.org/2020/sqlite-autoconf-3310100.tar.gz"" && tar xzf sqlite-autoconf-3310100.tar.gz \
-    && cd sqlite-autoconf-3310100 && ./configure --disable-static --enable-fts5 --enable-json1 CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
+RUN wget ""https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz"" && tar xzf sqlite-autoconf-3340100.tar.gz \
+    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
+    CFLAGS=""-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1"" \
     && make && make install
 
-RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-sources/freexl-1.0.5.tar.gz"" && tar zxf freexl-1.0.5.tar.gz \
-    && cd freexl-1.0.5 && ./configure && make && make install
+RUN wget ""http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz"" && tar zxf freexl-1.0.6.tar.gz \
+    && cd freexl-1.0.6 && ./configure && make && make install
 
-RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-4.4.0-RC0.tar.gz"" && tar zxf libspatialite-4.4.0-RC0.tar.gz \
-    && cd libspatialite-4.4.0-RC0 && ./configure && make && make install
+RUN wget ""http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz"" && tar zxf libspatialite-5.0.1.tar.gz \
+    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install
 
 RUN wget ""http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz"" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install
 
-RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/spatialite-tools-4.4.0-RC0.tar.gz"" && tar zxf spatialite-tools-4.4.0-RC0.tar.gz \
-    && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install
+RUN wget ""http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz"" && tar zxf spatialite-tools-5.0.0.tar.gz \
+    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install
 
 
 # Add local code to the image instead of fetching from pypi.
@@ -27,7 +28,7 @@ COPY . /datasette
 
 RUN pip install /datasette
 
-FROM python:3.7.10-slim-stretch
+FROM python:3.9.2-slim-buster
 
 # Copy python dependencies and spatialite libraries
 COPY --from=build /usr/local/lib/ /usr/local/lib/
```

I had to use `--disable-rttopo` from the tip in https://github.com/OSGeo/gdal/pull/3443 and also needed to install `libminizip-dev`.

This works, sort of... I'm getting a weird issue where the `/dbname` page is hanging some of the time instead of loading correctly. Other than that it seems to work, but a hanging page is bad!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/1249#issuecomment-792383956,https://api.github.com/repos/simonw/datasette/issues/1249,792383956,MDEyOklzc3VlQ29tbWVudDc5MjM4Mzk1Ng==,9599,simonw,2021-03-08T00:20:09Z,2021-03-08T00:20:09Z,OWNER,"Worth noting that the Docker image used by `datasette publish cloudrun` doesn't actually use that Datasette docker image - it does this:

https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L349-L353

Where the apt extras for SpatiaLite are: https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L344-L345

`libsqlite3-mod-spatialite` against that official `python:3.8` image doesn't appear to install SpatiaLite 5.0.


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",824064069,Updated Dockerfile with SpatiaLite version 5.0,
https://github.com/simonw/datasette/issues/858#issuecomment-792308036,https://api.github.com/repos/simonw/datasette/issues/858,792308036,MDEyOklzc3VlQ29tbWVudDc5MjMwODAzNg==,1219001,robroc,2021-03-07T16:41:54Z,2021-03-07T16:41:54Z,NONE,"Apologies if I sound dense but I don't see where you would pass
'shell=True'. I'm using the CLI installed via pip.

On Sun., Mar. 7, 2021, 2:15 a.m. David Smith, <notifications@github.com>
wrote:

> To get it to work I had to:
>
>    -
>
>    add shell=true to the various commands in datasette
>    -
>
>    use the name argument of the publish command. (
>    https://docs.datasette.io/en/stable/publish.html)
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/simonw/datasette/issues/858#issuecomment-792230560>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAJJTOMZMGYSCGUU4J3AVSDTCMRX5ANCNFSM4ODNEDYA>
> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/pull/1223#issuecomment-792233255,https://api.github.com/repos/simonw/datasette/issues/1223,792233255,MDEyOklzc3VlQ29tbWVudDc5MjIzMzI1NQ==,9599,simonw,2021-03-07T07:41:01Z,2021-03-07T07:41:01Z,OWNER,"This is fantastic, thanks so much for tracking this down.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",806918878,Add compile option to Dockerfile to fix failing test (fixes #696),
https://github.com/simonw/datasette/issues/858#issuecomment-792230560,https://api.github.com/repos/simonw/datasette/issues/858,792230560,MDEyOklzc3VlQ29tbWVudDc5MjIzMDU2MA==,39445562,smithdc1,2021-03-07T07:14:58Z,2021-03-07T07:14:58Z,NONE,"To get it to work I had to:

- add `shell=true` to the various commands in datasette 

- use the name argument of the publish command. (https://docs.datasette.io/en/stable/publish.html)


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/issues/858#issuecomment-792129022,https://api.github.com/repos/simonw/datasette/issues/858,792129022,MDEyOklzc3VlQ29tbWVudDc5MjEyOTAyMg==,1219001,robroc,2021-03-07T00:23:34Z,2021-03-07T00:23:34Z,NONE,@smithdc1 Can you tell us what you did to get it to publish in Windows? What commands did you pass?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10,
https://github.com/simonw/datasette/issues/766#issuecomment-791509910,https://api.github.com/repos/simonw/datasette/issues/766,791509910,MDEyOklzc3VlQ29tbWVudDc5MTUwOTkxMA==,6371750,JBPressac,2021-03-05T15:57:35Z,2021-03-05T16:35:21Z,CONTRIBUTOR,"Hello, 
I have the same wildcards search problems with an instance of Datasette. http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwerz&_sort=rowid is OK but http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwe* is not (FTS is activated on ""Reference"" ""IntituleAnalyse"" ""NomDuProducteur"" ""PresentationDuContenu"" ""Notes""). 

Notice that a SQL query as below launched directly from SQLite in the server's shell, retrieves results.

`select * from fonds_auguste_dupouy_1872_1967_fts where IntituleAnalyse MATCH ""gwe*"";`

Thanks,","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",617323873,Enable wildcard-searches by default,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791530093,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,791530093,MDEyOklzc3VlQ29tbWVudDc5MTUzMDA5Mw==,306240,UtahDave,2021-03-05T16:28:07Z,2021-03-05T16:28:07Z,NONE,"> I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.
> 
> Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?

@maxhawkins a limitation of the python mbox module is it loads the entire mbox into memory.  I did find another approach to this problem that didn't use the builtin python mbox module and created a generator so that it didn't have to load the whole mbox into memory.  I was hoping to use standard library modules, but this might be a good reason to investigate that approach a bit more. My worry is making sure a custom processor handles all the ins and outs of the mbox format correctly.

Hm. As I'm writing this, I thought of something.  I think I can parse each message one at a time, and then use an mbox function to load each message using the python mbox module. That way the mbox module can still deal with the specifics of the mbox format, but I can use a generator.

I'll give that a try.  Thanks for the feedback @maxhawkins and @simonw. I'll give that a try.

@simonw can we hold off on merging this until I can test this new approach?","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791089881,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,791089881,MDEyOklzc3VlQ29tbWVudDc5MTA4OTg4MQ==,28565,maxhawkins,2021-03-05T02:03:19Z,2021-03-05T02:03:19Z,NONE,"I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.

Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/dogsheep-photos/issues/32#issuecomment-791053721,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32,791053721,MDEyOklzc3VlQ29tbWVudDc5MTA1MzcyMQ==,6213,dsisnero,2021-03-05T00:31:27Z,2021-03-05T00:31:27Z,NONE,I am getting the same thing for US West (N. California) us-west-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803333769,KeyError: 'Contents' on running upload,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790934616,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,790934616,MDEyOklzc3VlQ29tbWVudDc5MDkzNDYxNg==,203343,Btibert3,2021-03-04T20:54:44Z,2021-03-04T20:54:44Z,NONE,"Sorry for the delay, I got sidetracked after class last night.  I am getting the following error:

```
/content# google-takeout-to-sqlite mbox takeout.db Takeout/Mail/gmail.mbox 
Usage: google-takeout-to-sqlite [OPTIONS] COMMAND [ARGS]...Try 'google-takeout-to-sqlite --help' for help.

Error: No such command 'mbox'.
```

On the box, I installed with pip after cloning: https://github.com/UtahDave/google-takeout-to-sqlite.git","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/simonw/datasette/issues/1238#issuecomment-790857004,https://api.github.com/repos/simonw/datasette/issues/1238,790857004,MDEyOklzc3VlQ29tbWVudDc5MDg1NzAwNA==,79913,tsibley,2021-03-04T19:06:55Z,2021-03-04T19:06:55Z,NONE,"@rgieseke Ah, that's super helpful. Thank you for the workaround for now!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790695126,MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg==,9599,simonw,2021-03-04T15:20:42Z,2021-03-04T15:20:42Z,MEMBER,"I'm not sure why but my most recent import, when displayed in Datasette, looks like this:

<img width=""574"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png"">

Sorting by `id` in the opposite order gives me the data I would expect - so it looks like a bunch of null/blank messages are being imported at some point and showing up first due to ID ordering.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790693674,MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA==,9599,simonw,2021-03-04T15:18:36Z,2021-03-04T15:18:36Z,MEMBER,"I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790669767,MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw==,9599,simonw,2021-03-04T14:46:06Z,2021-03-04T14:46:06Z,MEMBER,"Solution could be to pre-process that string by splitting on `(` and dropping everything afterwards, assuming that the `(...)` bit isn't necessary for correctly parsing the date.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790668263,MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw==,9599,simonw,2021-03-04T14:43:58Z,2021-03-04T14:43:58Z,MEMBER,"I added this code to output a message ID on errors:
```diff
             print(""Errors: {}"".format(num_errors))
             print(traceback.format_exc())
+            print(""Message-Id: {}"".format(email.get(""Message-Id"", ""None"")))
             continue
```
Having found a message ID that had an error, I ran this command to see the context:

    rg --text --context 20 '44F289B0.000001.02100@SCHWARZE-DWFXMI' ~/gmail.mbox

This was for the following error:
```
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 102, in get_mbox
    message[""date""] = get_message_date(email.get(""Date""), email.get_from())
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 178, in get_message_date
    datetime_tuple = email.utils.parsedate_tz(mail_date)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 50, in parsedate_tz
    res = _parsedate_tz(data)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 69, in _parsedate_tz
    data = data.split()
AttributeError: 'Header' object has no attribute 'split'
```
Here's what I spotted in the `ripgrep` output:
```
177133570:Message-Id: <44F289B0.000001.02100@SCHWARZE-DWFXMI>
177133571-Date: Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)
177133572-X-Mailer: IncrediMail (5002253)
```
So it could it be that `_parsedate_tz` is having trouble with that `Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)` string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790391711,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790391711,MDEyOklzc3VlQ29tbWVudDc5MDM5MTcxMQ==,306240,UtahDave,2021-03-04T07:36:24Z,2021-03-04T07:36:24Z,NONE,"> Looks like you're doing this:
> 
> ```python
>     elif message.get_content_type() == ""text/plain"":
>         body = message.get_payload(decode=True)
> ```
> 
> So presumably that decodes to a unicode string?
> 
> I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.

Ah, that's good to know. I think explicitly creating the tables will be a great improvement. I'll add that.

Also, I noticed after I opened this PR that the `message.get_payload()` is being deprecated in favor of `message.get_content()` or something like that.  I'll see if that handles the decoding better, too.

Thanks for the feedback. I should have time tomorrow to put together some improvements.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790389335,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790389335,MDEyOklzc3VlQ29tbWVudDc5MDM4OTMzNQ==,306240,UtahDave,2021-03-04T07:32:04Z,2021-03-04T07:32:04Z,NONE,"> The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:
> 
> https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67
> 
> I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.

The wait is from python loading the mbox file. This happens regardless if you're getting the length of the mbox.  The mbox module is on the slow side. It is possible to do one's own parsing of the mbox, but I kind of wanted to avoid doing that.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/6#issuecomment-790384087,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6,790384087,MDEyOklzc3VlQ29tbWVudDc5MDM4NDA4Nw==,9599,simonw,2021-03-04T07:22:51Z,2021-03-04T07:22:51Z,MEMBER,#3 also mentions the conflicting version with other tools.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",821841046,Upgrade to latest sqlite-utils,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790380839,MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ==,9599,simonw,2021-03-04T07:17:05Z,2021-03-04T07:17:05Z,MEMBER,"Looks like you're doing this:
```python
    elif message.get_content_type() == ""text/plain"":
        body = message.get_payload(decode=True)
```
So presumably that decodes to a unicode string?

I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790379629,MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ==,9599,simonw,2021-03-04T07:14:41Z,2021-03-04T07:14:41Z,MEMBER,"Confirmed: removing the `len()` call does not speed things up, so it's reading through the entire file for some other purpose too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790378658,MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA==,9599,simonw,2021-03-04T07:12:48Z,2021-03-04T07:12:48Z,MEMBER,"It looks like the `body` is being loaded into a BLOB column - so in Datasette default it looks like this:

<img width=""1650"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109924808-b4b96980-7c75-11eb-8c9e-307f2ae32d5a.png"">

If I `datasette install datasette-render-binary` and then try again I get this:

<img width=""1487"" alt=""mbox__mbox_emails__753_446_rows"" src=""https://user-images.githubusercontent.com/9599/109924944-ea5e5280-7c75-11eb-9a32-404f3d68455f.png"">

It would be great if we could store the `body` as unicode text instead. May have to do something clever to decode it based on some kind of charset header?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790373024,MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA==,9599,simonw,2021-03-04T07:01:58Z,2021-03-04T07:04:06Z,MEMBER,"I got 9 warnings that look like this:
```
Errors: 1
Traceback (most recent call last):
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 103, in get_mbox
    message[""date""] = get_message_date(email.get(""Date""), email.get_from())
  File ""/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py"", line 167, in get_message_date
    datetime_tuple = email.utils.parsedate_tz(mail_date)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 50, in parsedate_tz
    res = _parsedate_tz(data)
  File ""/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py"", line 69, in _parsedate_tz
    data = data.split()
AttributeError: 'Header' object has no attribute 'split'
```
It would be useful if those warnings told me the message ID (or similar) of the affected message so I could grep for it in the `mbox` and see what was going on.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790372621,MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ==,9599,simonw,2021-03-04T07:01:18Z,2021-03-04T07:01:18Z,MEMBER,"I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in `healthkit-to-sqlite` - you set the progress bar size to the size of the file in bytes, then update a counter as you read the file.

https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the `progress_callback()` bit) is where that happens.

It can be a bit of a convoluted pattern, and I'm not at all sure it would work for `mbox` files since it looks like that library has other reasons it needs to do a file scan rather than streaming it through one chunk of bytes at a time. So I imagine this would not work here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790370485,MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ==,9599,simonw,2021-03-04T06:57:25Z,2021-03-04T06:57:48Z,MEMBER,"The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:

https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67

I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790369076,MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng==,9599,simonw,2021-03-04T06:54:46Z,2021-03-04T06:54:46Z,MEMBER,"The Rich-powered progress bar is pretty:

![rich](https://user-images.githubusercontent.com/9599/109923307-71f69200-7c73-11eb-9ee2-8f0a240f3994.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790312268,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,790312268,MDEyOklzc3VlQ29tbWVudDc5MDMxMjI2OA==,9599,simonw,2021-03-04T05:48:16Z,2021-03-04T05:48:16Z,MEMBER,"Wow, my mbox is a 10.35 GB download!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/pull/1243#issuecomment-790311215,https://api.github.com/repos/simonw/datasette/issues/1243,790311215,MDEyOklzc3VlQ29tbWVudDc5MDMxMTIxNQ==,9599,simonw,2021-03-04T05:45:57Z,2021-03-04T05:45:57Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815955014,fix small typo,
https://github.com/simonw/datasette/issues/268#issuecomment-790257263,https://api.github.com/repos/simonw/datasette/issues/268,790257263,MDEyOklzc3VlQ29tbWVudDc5MDI1NzI2Mw==,649467,mhalle,2021-03-04T03:20:23Z,2021-03-04T03:20:23Z,NONE,"It's kind of an ugly hack, but you can try out what using the fts5 table as an actual datasette-accessible table looks like without changing any datasette code by creating yet another view on top of the fts5 table:

`create view proxyview as select *, rank, table_fts as fts from table_fts;`

That's now visible from datasette, just like any other view, but you can use `fts match escape_fts(search_string) order by rank`.

This is only good as a proof of concept because you're inefficiently going from view -> fts5 external content table -> view -> data table. However, it does show it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790198930,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,790198930,MDEyOklzc3VlQ29tbWVudDc5MDE5ODkzMA==,203343,Btibert3,2021-03-04T00:58:40Z,2021-03-04T00:58:40Z,NONE,"I am just seeing this sorry, yes!  I will kick the tires later on tonight.  My apologies for the delay.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/simonw/datasette/issues/283#issuecomment-789680230,https://api.github.com/repos/simonw/datasette/issues/283,789680230,MDEyOklzc3VlQ29tbWVudDc4OTY4MDIzMA==,605492,justinpinkney,2021-03-03T12:28:42Z,2021-03-03T12:28:42Z,NONE,"One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`. 

I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/268#issuecomment-789409126,https://api.github.com/repos/simonw/datasette/issues/268,789409126,MDEyOklzc3VlQ29tbWVudDc4OTQwOTEyNg==,649467,mhalle,2021-03-03T03:57:15Z,2021-03-03T03:58:40Z,NONE,"In FTS5, I think doing an FTS search is actually much easier than doing a join against the main table like datasette does now. In fact, FTS5 external content tables provide a transparent interface back to the original table or view.

Here's what I'm currently doing:
* build a view that joins whatever tables I want and rename the columns to non-joiny names (e.g, `chapter.name AS chapter_name` in the view where needed)
* Create an FTS5 table with `content=""viewname""`
* As described in the ""external content tables"" section (https://www.sqlite.org/fts5.html#external_content_tables), sql queries can be made directly to the FTS table, which behind the covers makes select calls to the content table when the content of the original columns are needed.
* In addition, you get ""rank"" and ""bm25()"" available to you when you select on the _fts table.

Unfortunately, datasette doesn't currently seem happy being coerced into doing a real query on an fts5 table.  This works:
```select col1, col2, col3 from table_fts where coll1=""value"" and table_fts match escape_fts(""search term"") order by rank```

But this doesn't work in the datasette SQL query interface:
```select col1, col2, col3 from table_fts where coll1=""value"" and table_fts match escape_fts(:search) order by rank```   (the ""search"" input text field doesn't show up)

For what datasette is doing right now, I think you could just use contentless fts5 tables (`content=""""`), since all you care about is the rowid since all you're doing a subselect to get the rowid anyway. In fts5, that's just a contentless table.

I guess if you want to follow this suggestion, you'd need a somewhat different code path for fts5.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,
https://github.com/simonw/datasette/issues/1238#issuecomment-789186458,https://api.github.com/repos/simonw/datasette/issues/1238,789186458,MDEyOklzc3VlQ29tbWVudDc4OTE4NjQ1OA==,198537,rgieseke,2021-03-02T20:19:30Z,2021-03-02T20:19:30Z,CONTRIBUTOR,A custom `templates/index.html` seems to work and custom `pages` as a workaround with moving them to `pages/base_url_dir`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1247#issuecomment-787616446,https://api.github.com/repos/simonw/datasette/issues/1247,787616446,MDEyOklzc3VlQ29tbWVudDc4NzYxNjQ0Ng==,9599,simonw,2021-03-01T03:50:37Z,2021-03-01T03:50:37Z,OWNER,"I like the `.add_memory_database()` option. I also like that it makes it more obvious that this is a capability of Datasette, since I'm excited to see more plugins, features and tests that take advantage of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",818430405,datasette.add_memory_database() method,
https://github.com/simonw/datasette/issues/1247#issuecomment-787616158,https://api.github.com/repos/simonw/datasette/issues/1247,787616158,MDEyOklzc3VlQ29tbWVudDc4NzYxNjE1OA==,9599,simonw,2021-03-01T03:49:27Z,2021-03-01T03:49:27Z,OWNER,"A couple of options:
```python
datasette.add_memory_database(""test_json_array"")
# or make that first argument to add_database() optional and support:
datasette.add_database(memory_name=""test_json_array"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",818430405,datasette.add_memory_database() method,
https://github.com/simonw/datasette/issues/1246#issuecomment-787611153,https://api.github.com/repos/simonw/datasette/issues/1246,787611153,MDEyOklzc3VlQ29tbWVudDc4NzYxMTE1Mw==,9599,simonw,2021-03-01T03:30:57Z,2021-03-01T03:30:57Z,OWNER,"I'm going to try a new pattern for testing this, enabled by #1151 - the test will create a new named in-memory database, write some records to it and then run some test facets against that. This will save me from having to add yet another fixtures table for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817597268,Suggest for ArrayFacet possibly confused by blank values,
https://github.com/simonw/datasette/issues/1005#issuecomment-787536267,https://api.github.com/repos/simonw/datasette/issues/1005,787536267,MDEyOklzc3VlQ29tbWVudDc4NzUzNjI2Nw==,9599,simonw,2021-02-28T22:30:37Z,2021-02-28T22:30:37Z,OWNER,It's out! https://github.com/encode/httpx/releases/tag/0.17.0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787532279,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787532279,MDEyOklzc3VlQ29tbWVudDc4NzUzMjI3OQ==,9599,simonw,2021-02-28T22:09:37Z,2021-02-28T22:09:37Z,OWNER,Microsoft's playwright Python library solves this problem by code generating both their sync AND their async libraries https://github.com/microsoft/playwright-python/tree/master/scripts,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787198202,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787198202,MDEyOklzc3VlQ29tbWVudDc4NzE5ODIwMg==,9599,simonw,2021-02-27T22:33:58Z,2021-02-27T22:33:58Z,OWNER,"Hah or use this trick, which genuinely rewrites the code at runtime using a class decorator! https://github.com/python-happybase/aiohappybase/blob/0990ef45cfdb720dc987afdb4957a0fac591cb99/aiohappybase/sync/_util.py#L19-L32","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787195536,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787195536,MDEyOklzc3VlQ29tbWVudDc4NzE5NTUzNg==,9599,simonw,2021-02-27T22:13:24Z,2021-02-27T22:13:24Z,OWNER,Some other interesting background reading: https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html - in particular see how SQLALchemy has a `await conn.run_sync(meta.drop_all)` mechanism for running methods that haven't themselves been provided in an async version,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787190562,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787190562,MDEyOklzc3VlQ29tbWVudDc4NzE5MDU2Mg==,9599,simonw,2021-02-27T22:04:00Z,2021-02-27T22:04:00Z,OWNER,"From the poster here: https://github.com/sethmlarson/pycon-async-sync-poster/blob/master/poster.pdf

<img width=""624"" alt=""pycon-async-sync-poster_poster_pdf_at_master_·_sethmlarson_pycon-async-sync-poster"" src=""https://user-images.githubusercontent.com/9599/109401634-9f0a1400-7904-11eb-8b3a-37df0678b8dc.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787186826,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787186826,MDEyOklzc3VlQ29tbWVudDc4NzE4NjgyNg==,9599,simonw,2021-02-27T22:01:54Z,2021-02-27T22:01:54Z,OWNER,`unasync` is an implementation of the exact pattern I was talking about above - it uses the `tokenize` module from the Python standard library to apply some clever rules to transform an async codebase into a sync one. https://unasync.readthedocs.io/en/latest/ - implementation here: https://github.com/python-trio/unasync/blob/v0.5.0/src/unasync/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787175126,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787175126,MDEyOklzc3VlQ29tbWVudDc4NzE3NTEyNg==,9599,simonw,2021-02-27T21:55:05Z,2021-02-27T21:55:05Z,OWNER,"""how to use some new tools to more easily maintain a codebase that supports both async and synchronous I/O and multiple async libraries"" - yeah that's exactly what I need, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787150276,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787150276,MDEyOklzc3VlQ29tbWVudDc4NzE1MDI3Ng==,37962604,polyrand,2021-02-27T21:27:26Z,2021-02-27T21:27:26Z,NONE,"I had this resource by Seth Michael Larson saved https://github.com/sethmlarson/pycon-async-sync-poster I haven't had a look at it, but it may contain useful info.

On twitter, I mentioned passing an aiosqlite connection during the `Database` creation. I'm not 100% familiar with the `sqlite-utils` codebase, so I may be wrong here, but maybe decorating internal functions could be an option? Then they are awaited or not inside the decorator depending on how they are called.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787144523,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787144523,MDEyOklzc3VlQ29tbWVudDc4NzE0NDUyMw==,9599,simonw,2021-02-27T21:18:46Z,2021-02-27T21:18:46Z,OWNER,"Here's a really wild idea: I wonder if it would be possible to run a source transformation against either the sync or the async versions of the code to produce the equivalent for the other paradigm?

Could that even be as simple as a set of regular expressions against the `await ...` version that strips out or replaces the `await` and `async def` and `async for` statements?

If so... I could maintain just the async version, generate the sync version with a script and rely on robust unit testing to guarantee that this actually works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787142066,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787142066,MDEyOklzc3VlQ29tbWVudDc4NzE0MjA2Ng==,9599,simonw,2021-02-27T21:17:10Z,2021-02-27T21:17:10Z,OWNER,"I have a hunch this is actually going to be quite difficult, due to the internal complexity of some of the `sqlite-utils` API methods.

Consider `db[table].extract(...)` for example. It does a whole bunch of extra queries inside the method - each of those would need to be turned into an `await` call for the async version. Here's the method body today:

https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1060-L1152

Writing this method twice - looking similar but with `await ...` tucked in before every internal method it calls that needs to execute SQL - is going to be pretty messy.

One thing that would help a LOT is figuring out how to share the majority of the test code. If the exact same tests could run against both the sync and async versions with a bit of test trickery, maintaining parallel implementations would at least be a bit more feasible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787121933,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787121933,MDEyOklzc3VlQ29tbWVudDc4NzEyMTkzMw==,25778,eyeseast,2021-02-27T19:18:57Z,2021-02-27T19:18:57Z,CONTRIBUTOR,"I think HTTPX gets it exactly right, with a clear separation between sync and async clients, each with a basically identical API. (I'm about to switch [feed-to-sqlite](https://github.com/eyeseast/feed-to-sqlite) over to it, from Requests, to eventually make way for async support.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787120136,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787120136,MDEyOklzc3VlQ29tbWVudDc4NzEyMDEzNg==,9599,simonw,2021-02-27T19:04:47Z,2021-02-27T19:04:47Z,OWNER,"Another option here would be to add https://github.com/omnilib/aiosqlite/blob/main/aiosqlite/core.py as a dependency - it's four years old now and actively marinated, and the code is pretty small so it looks like a solid, stable, reliable dependency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787118691,https://api.github.com/repos/simonw/sqlite-utils/issues/242,787118691,MDEyOklzc3VlQ29tbWVudDc4NzExODY5MQ==,9599,simonw,2021-02-27T18:53:23Z,2021-02-27T18:53:23Z,OWNER,"Datasette has its own implementation of a write queue for exactly this purpose - and there's no reason at all that should stay in Datasette rather than being extracted out and moved over here to `sqlite-utils`.

One small concern I have is around the API design. I'd want to keep supporting the existing synchronous API while also providing a similar API with await-based methods.

What are some good examples of libraries that do this? I like how https://www.python-httpx.org/ handles it, maybe that's a good example to imitate?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817989436,Async support,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-786925280,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,786925280,MDEyOklzc3VlQ29tbWVudDc4NjkyNTI4MA==,9599,simonw,2021-02-26T22:23:10Z,2021-02-26T22:23:10Z,MEMBER,"Thanks!

I requested my Gmail export from takeout - once that arrives I'll test it against this and then merge the PR.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/issues/1238#issuecomment-786849095,https://api.github.com/repos/simonw/datasette/issues/1238,786849095,MDEyOklzc3VlQ29tbWVudDc4Njg0OTA5NQ==,9599,simonw,2021-02-26T19:29:38Z,2021-02-26T19:29:38Z,OWNER,"Here's the test I wrote:
```diff
git diff tests/test_custom_pages.py
diff --git a/tests/test_custom_pages.py b/tests/test_custom_pages.py
index 6a23192..5a71f56 100644
--- a/tests/test_custom_pages.py
+++ b/tests/test_custom_pages.py
@@ -2,11 +2,19 @@ import pathlib
 import pytest
 from .fixtures import make_app_client
 
+TEST_TEMPLATE_DIRS = str(pathlib.Path(__file__).parent / ""test_templates"")
+
 
 @pytest.fixture(scope=""session"")
 def custom_pages_client():
+    with make_app_client(template_dir=TEST_TEMPLATE_DIRS) as client:
+        yield client
+
+
+@pytest.fixture(scope=""session"")
+def custom_pages_client_with_base_url():
     with make_app_client(
-        template_dir=str(pathlib.Path(__file__).parent / ""test_templates"")
+        template_dir=TEST_TEMPLATE_DIRS, config={""base_url"": ""/prefix/""}
     ) as client:
         yield client
 
@@ -23,6 +31,12 @@ def test_request_is_available(custom_pages_client):
     assert ""path:/request"" == response.text
 
 
+def test_custom_pages_with_base_url(custom_pages_client_with_base_url):
+    response = custom_pages_client_with_base_url.get(""/prefix/request"")
+    assert 200 == response.status
+    assert ""path:/prefix/request"" == response.text
+
+
 def test_custom_pages_nested(custom_pages_client):
     response = custom_pages_client.get(""/nested/nest"")
     assert 200 == response.status
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-786848654,https://api.github.com/repos/simonw/datasette/issues/1238,786848654,MDEyOklzc3VlQ29tbWVudDc4Njg0ODY1NA==,9599,simonw,2021-02-26T19:28:48Z,2021-02-26T19:28:48Z,OWNER,"I added a debug line just before `for regex, wildcard_template` here:

https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1148-L1155

And it showed that for some reason `request.path` is `/prefix/prefix/request` here - the prefix got doubled somehow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1238#issuecomment-786841261,https://api.github.com/repos/simonw/datasette/issues/1238,786841261,MDEyOklzc3VlQ29tbWVudDc4Njg0MTI2MQ==,9599,simonw,2021-02-26T19:13:44Z,2021-02-26T19:13:44Z,OWNER,Sounds like a bug - thanks for reporting this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813899472,Custom pages don't work with base_url setting,
https://github.com/simonw/datasette/issues/1246#issuecomment-786840734,https://api.github.com/repos/simonw/datasette/issues/1246,786840734,MDEyOklzc3VlQ29tbWVudDc4Njg0MDczNA==,9599,simonw,2021-02-26T19:12:39Z,2021-02-26T19:12:47Z,OWNER,"Could I take this part:
```python
             suggested_facet_sql = """""" 
                 select distinct json_type({column}) 
                 from ({sql}) 
             """""".format( 
                 column=escape_sqlite(column), sql=self.sql 
             ) 
```
And add `where {column} is not null and {column} != ''` perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817597268,Suggest for ArrayFacet possibly confused by blank values,
https://github.com/simonw/datasette/issues/1246#issuecomment-786840425,https://api.github.com/repos/simonw/datasette/issues/1246,786840425,MDEyOklzc3VlQ29tbWVudDc4Njg0MDQyNQ==,9599,simonw,2021-02-26T19:11:56Z,2021-02-26T19:11:56Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/facets.py#L271-L295,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817597268,Suggest for ArrayFacet possibly confused by blank values,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786830832,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786830832,MDEyOklzc3VlQ29tbWVudDc4NjgzMDgzMg==,9599,simonw,2021-02-26T18:52:40Z,2021-02-26T18:52:40Z,OWNER,"Could this handle lists of objects too? That would be pretty amazing - if the column has a `[{...}, {...}]` list in it could turn that into a many-to-many.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/datasette/issues/1240#issuecomment-786813506,https://api.github.com/repos/simonw/datasette/issues/1240,786813506,MDEyOklzc3VlQ29tbWVudDc4NjgxMzUwNg==,9599,simonw,2021-02-26T18:19:46Z,2021-02-26T18:19:46Z,OWNER,"Linking to rows from custom queries is a lot harder - because given an arbitrary string of SQL it's difficult to analyze it and figure out which (if any) of the returned columns represent a primary key.

It's possible to manually write a SQL query that returns a column that will be treated as a link to another page using this plugin, but it's not particularly straight-forward: https://datasette.io/plugins/datasette-json-html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814591962,Allow facetting on custom queries,
https://github.com/simonw/datasette/issues/1240#issuecomment-786812716,https://api.github.com/repos/simonw/datasette/issues/1240,786812716,MDEyOklzc3VlQ29tbWVudDc4NjgxMjcxNg==,9599,simonw,2021-02-26T18:18:18Z,2021-02-26T18:18:18Z,OWNER,"Agreed, this would be extremely useful. I'd love to be able to facet against custom queries. It's a fair bit of work to implement but it's not impossible. Closing this as a duplicate of #972.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",814591962,Allow facetting on custom queries,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786795132,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786795132,MDEyOklzc3VlQ29tbWVudDc4Njc5NTEzMg==,9599,simonw,2021-02-26T17:45:53Z,2021-02-26T17:45:53Z,OWNER,If there's no primary key in the JSON could use the `hash_id` mechanism.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786794435,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786794435,MDEyOklzc3VlQ29tbWVudDc4Njc5NDQzNQ==,9599,simonw,2021-02-26T17:44:38Z,2021-02-26T17:44:38Z,OWNER,This came up in office hours!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/datasette/issues/1244#issuecomment-786786645,https://api.github.com/repos/simonw/datasette/issues/1244,786786645,MDEyOklzc3VlQ29tbWVudDc4Njc4NjY0NQ==,9599,simonw,2021-02-26T17:30:38Z,2021-02-26T17:30:38Z,OWNER,"New paragraph at the top of https://docs.datasette.io/en/latest/writing_plugins.html

> Want to start by looking at an example? The [Datasette plugins directory](https://datasette.io/plugins) lists more than 50 open source plugins with code you can explore. The [plugin hooks](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hooks) page includes links to example plugins for each of the documented hooks.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",817528452,Plugin tip: look at the examples linked from the hooks page,
https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786050562,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786050562,MDEyOklzc3VlQ29tbWVudDc4NjA1MDU2Mg==,9599,simonw,2021-02-25T16:57:56Z,2021-02-25T16:57:56Z,OWNER,"`sqlite-utils create-view` currently has a `--ignore` option, so adding that to `sqlite-utils drop-view` and `sqlite-utils drop-table` makes sense as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore",
https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786049686,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786049686,MDEyOklzc3VlQ29tbWVudDc4NjA0OTY4Ng==,9599,simonw,2021-02-25T16:56:42Z,2021-02-25T16:56:42Z,OWNER,"So:
```python
    db[""my_table""].drop(ignore=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore",
https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786049394,https://api.github.com/repos/simonw/sqlite-utils/issues/237,786049394,MDEyOklzc3VlQ29tbWVudDc4NjA0OTM5NA==,9599,simonw,2021-02-25T16:56:14Z,2021-02-25T16:56:14Z,OWNER,"Other methods (`db.create_view()` for example) have `ignore=True` to mean ""don't throw an error if this causes a problem"", so I'm good with adding that to `.drop_view()`.

I don't like using it as the default partly because that would be a very minor breaking API change, but mainly because I don't want to hide mistakes people make - e.g. if you mistype the name of the table you are trying to drop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815554385,"db[""my_table""].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore",
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786037219,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786037219,MDEyOklzc3VlQ29tbWVudDc4NjAzNzIxOQ==,9599,simonw,2021-02-25T16:39:23Z,2021-02-25T16:39:23Z,OWNER,"Example from the docs:
```pycon
>>> db = sqlite_utils.Database(memory=True)
>>> db[""dogs""].insert({""name"": ""Cleo""})
>>> for pk, row in db[""dogs""].pks_and_rows_where():
...     print(pk, row)
1 {'rowid': 1, 'name': 'Cleo'}

>>> db[""dogs_with_pk""].insert({""id"": 5, ""name"": ""Cleo""}, pk=""id"")
>>> for pk, row in db[""dogs_with_pk""].pks_and_rows_where():
...     print(pk, row)
5 {'id': 5, 'name': 'Cleo'}

>>> db[""dogs_with_compound_pk""].insert(
...     {""species"": ""dog"", ""id"": 3, ""name"": ""Cleo""},
...     pk=(""species"", ""id"")
... )
>>> for pk, row in db[""dogs_with_compound_pk""].pks_and_rows_where():
...     print(pk, row)
('dog', 3) {'species': 'dog', 'id': 3, 'name': 'Cleo'}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786036355,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786036355,MDEyOklzc3VlQ29tbWVudDc4NjAzNjM1NQ==,9599,simonw,2021-02-25T16:38:07Z,2021-02-25T16:38:07Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#listing-rows-with-their-primary-keys,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786035142,https://api.github.com/repos/simonw/sqlite-utils/issues/239,786035142,MDEyOklzc3VlQ29tbWVudDc4NjAzNTE0Mg==,9599,simonw,2021-02-25T16:36:17Z,2021-02-25T16:36:17Z,OWNER,WIP in a pull request.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786016380,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786016380,MDEyOklzc3VlQ29tbWVudDc4NjAxNjM4MA==,9599,simonw,2021-02-25T16:10:01Z,2021-02-25T16:10:01Z,OWNER,"I prototyped this and I like it:
```
In [1]: import sqlite_utils
In [2]: db = sqlite_utils.Database(""/Users/simon/Dropbox/Development/datasette/fixtures.db"")
In [3]: list(db[""compound_primary_key""].pks_and_rows_where())
Out[3]: [(('a', 'b'), {'pk1': 'a', 'pk2': 'b', 'content': 'c'})]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786007209,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786007209,MDEyOklzc3VlQ29tbWVudDc4NjAwNzIwOQ==,9599,simonw,2021-02-25T15:57:50Z,2021-02-25T15:57:50Z,OWNER,`table.pks_and_rows_where(...)` is explicit and I think less ambiguous than the other options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786006794,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786006794,MDEyOklzc3VlQ29tbWVudDc4NjAwNjc5NA==,9599,simonw,2021-02-25T15:57:17Z,2021-02-25T15:57:28Z,OWNER,"I quite like `pks_with_rows_where(...)` - but grammatically it suggests it will return the primary keys that exist where their rows match the criteria - ""pks with rows"" can be interpreted as ""pks for the rows that..."" as opposed to ""pks accompanied by rows""","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786005078,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786005078,MDEyOklzc3VlQ29tbWVudDc4NjAwNTA3OA==,9599,simonw,2021-02-25T15:54:59Z,2021-02-25T15:56:16Z,OWNER,"Is `pk_rows_where()` a good name? It sounds like it returns ""primary key rows"" which isn't a thing. It actually returns rows along with their primary key.

Other options:

- `table.rows_with_pk_where(...)` - should this return `(row, pk)` rather than `(pk, row)`?
- `table.rows_where_pk(...)`
- `table.pk_and_rows_where(...)`
- `table.pk_with_rows_where(...)`
- `table.pks_with_rows_where(...)` - because rows is pluralized, so pks should be pluralized too?
- `table.pks_rows_where(...)`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786001768,https://api.github.com/repos/simonw/sqlite-utils/issues/240,786001768,MDEyOklzc3VlQ29tbWVudDc4NjAwMTc2OA==,9599,simonw,2021-02-25T15:50:28Z,2021-02-25T15:52:12Z,OWNER,"One option: `.rows_where()` could grow a `ensure_pk=True` option which checks to see if the table is a `rowid` table and, if it is, includes that in the `select`.

Or... how about you can call `.rows_where(..., pks=True)` and it will yield `(pk, rowdict)` tuple pairs instead of just returning the sequence of dictionaries?

I'm always a little bit nervous of methods that vary their return type based on their arguments. Maybe this would be a separate method instead?
```python
    for pk, row in table.pk_rows_where(...):
        # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816560819,table.pks_and_rows_where() method returning primary keys along with the rows,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785992158,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785992158,MDEyOklzc3VlQ29tbWVudDc4NTk5MjE1OA==,9599,simonw,2021-02-25T15:37:04Z,2021-02-25T15:37:04Z,OWNER,"Here's the current implementation of `.extract()`: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L1049-L1074

Tricky detail here: I create the lookup table first, based on the types of the columns that are being extracted.

I need to do this because extraction currently uses unique tuples of values, so the table has to be created in advance.

But if I'm using these new expand functions to figure out what's going to be extracted, I don't know the names of the columns and their types in advance. I'm only going to find those out during the transformation.

This may turn out to be incompatible with how `.extract()` works at the moment. I may need a new method, `.extract_expand()` perhaps? It could be simpler - work only against a single column for example.

I can still use the existing `sqlite-utils extract` CLI command though, with a `--json` flag and a rule that you can't run it against multiple columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983837,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785983837,MDEyOklzc3VlQ29tbWVudDc4NTk4MzgzNw==,9599,simonw,2021-02-25T15:25:21Z,2021-02-25T15:28:57Z,OWNER,"Problem with calling this argument `transform=` is that the term ""transform"" already means something else in this library.

I could use `convert=` instead.

... but that doesn't instantly make me think of turning a value into multiple columns.

How about `expand=`? I've not used that term anywhere yet.

    db[""Reports""].extract([""Reported by""], expand={""Reported by"": json.loads})

I think that works. You're expanding a single value into several columns of information.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983070,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785983070,MDEyOklzc3VlQ29tbWVudDc4NTk4MzA3MA==,9599,simonw,2021-02-25T15:24:17Z,2021-02-25T15:24:17Z,OWNER,I'm going to go with last-wins - so if multiple transform functions return the same key the last one will over-write the others.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980813,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785980813,MDEyOklzc3VlQ29tbWVudDc4NTk4MDgxMw==,9599,simonw,2021-02-25T15:21:02Z,2021-02-25T15:23:47Z,OWNER,"Maybe the Python version takes an optional dictionary mapping column names to transformation functions? It could then merge all of those results together - and maybe throw an error if the same key is produced by more than one column.

```python
    db[""Reports""].extract([""Reported by""], transform={""Reported by"": json.loads})
```
Or it could have an option for different strategies if keys collide: first wins, last wins, throw exception, add a prefix to the new column name. That feels a bit too complex for an edge-case though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980083,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785980083,MDEyOklzc3VlQ29tbWVudDc4NTk4MDA4Mw==,9599,simonw,2021-02-25T15:20:02Z,2021-02-25T15:20:02Z,OWNER,It would be OK if the CLI version only allows you to specify a single column if you are using the `--json` option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979769,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785979769,MDEyOklzc3VlQ29tbWVudDc4NTk3OTc2OQ==,9599,simonw,2021-02-25T15:19:37Z,2021-02-25T15:19:37Z,OWNER,For the Python version I'd like to be able to provide a transformation callback function - which can be `json.loads` but could also be anything else which accepts the value of the current column and returns a Python dictionary of columns and their values to use in the new table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979192,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785979192,MDEyOklzc3VlQ29tbWVudDc4NTk3OTE5Mg==,9599,simonw,2021-02-25T15:18:46Z,2021-02-25T15:18:46Z,OWNER,"Likewise the `sqlite-utils extract` command takes one or more columns:
```
Usage: sqlite-utils extract [OPTIONS] PATH TABLE COLUMNS...

  Extract one or more columns into a separate table

Options:
  --table TEXT             Name of the other table to extract columns to
  --fk-column TEXT         Name of the foreign key column to add to the table
  --rename <TEXT TEXT>...  Rename this column in extracted table
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785978689,https://api.github.com/repos/simonw/sqlite-utils/issues/239,785978689,MDEyOklzc3VlQ29tbWVudDc4NTk3ODY4OQ==,9599,simonw,2021-02-25T15:18:03Z,2021-02-25T15:18:03Z,OWNER,"The Python `.extract()` method currently starts like this:
```python
def extract(self, columns, table=None, fk_column=None, rename=None):
        rename = rename or {}
        if isinstance(columns, str):
            columns = [columns]
        if not set(columns).issubset(self.columns_dict.keys()):
            raise InvalidColumns(
                ""Invalid columns {} for table with columns {}"".format(
                    columns, list(self.columns_dict.keys())
                )
            )
        ...
```
Note that it takes a list of columns (and treats a string as a single item list). That's because it can be called with a list of columns and it will use them to populate another table of unique tuples of those column values.

So a new mechanism that can instead read JSON values from a single column needs to be compatible with that existing design.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816526538,sqlite-utils extract could handle nested objects,
https://github.com/simonw/sqlite-utils/issues/238#issuecomment-785972074,https://api.github.com/repos/simonw/sqlite-utils/issues/238,785972074,MDEyOklzc3VlQ29tbWVudDc4NTk3MjA3NA==,9599,simonw,2021-02-25T15:08:36Z,2021-02-25T15:08:36Z,OWNER,I bet the bug is in here: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L593-L602,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",816523763,.add_foreign_key() corrupts database if column contains a space,
https://github.com/simonw/datasette/pull/1243#issuecomment-785485597,https://api.github.com/repos/simonw/datasette/issues/1243,785485597,MDEyOklzc3VlQ29tbWVudDc4NTQ4NTU5Nw==,22429695,codecov[bot],2021-02-25T00:28:30Z,2021-02-25T00:28:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=h1) Report
> Merging [#1243](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=desc) (887bfd2) into [main](https://codecov.io/gh/simonw/datasette/commit/726f781c50e88f557437f6490b8479c3d6fabfc2?el=desc) (726f781) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1243/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1243   +/-   ##
=======================================
  Coverage   91.56%   91.56%           
=======================================
  Files          34       34           
  Lines        4242     4242           
=======================================
  Hits         3884     3884           
  Misses        358      358           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=footer). Last update [726f781...32652d9](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",815955014,fix small typo,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-784638394,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,784638394,MDEyOklzc3VlQ29tbWVudDc4NDYzODM5NA==,306240,UtahDave,2021-02-24T00:36:18Z,2021-02-24T00:36:18Z,NONE,I noticed that @simonw is using black for formatting. I ran black on my additions in this PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/issues/1241#issuecomment-784567547,https://api.github.com/repos/simonw/datasette/issues/1241,784567547,MDEyOklzc3VlQ29tbWVudDc4NDU2NzU0Nw==,9599,simonw,2021-02-23T22:45:56Z,2021-02-23T22:46:12Z,OWNER,"I really like the way the Share feature on Stack Overflow works: https://stackoverflow.com/questions/18934149/how-can-i-use-postgresqls-text-column-type-in-django
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1241#issuecomment-784347646,https://api.github.com/repos/simonw/datasette/issues/1241,784347646,MDEyOklzc3VlQ29tbWVudDc4NDM0NzY0Ng==,7107523,Kabouik,2021-02-23T16:55:26Z,2021-02-23T16:57:39Z,NONE,"> I think it's possible that many users these days no longer assume they can paste a URL from the browser address bar (if they ever understood that at all) because to many apps are SPAs with broken URLs.

Absolutely, that's why I thought my corner case with `iframe` preventing access to the datasette URL could actually be relevant in more general situations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1241#issuecomment-784334931,https://api.github.com/repos/simonw/datasette/issues/1241,784334931,MDEyOklzc3VlQ29tbWVudDc4NDMzNDkzMQ==,9599,simonw,2021-02-23T16:37:26Z,2021-02-23T16:37:26Z,OWNER,"A ""Share link"" button would only be needed on the table page and the arbitrary query page I think - and maybe on the row page, especially as that page starts to grow more features in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1241#issuecomment-784333768,https://api.github.com/repos/simonw/datasette/issues/1241,784333768,MDEyOklzc3VlQ29tbWVudDc4NDMzMzc2OA==,9599,simonw,2021-02-23T16:35:51Z,2021-02-23T16:35:51Z,OWNER,"This can definitely be done with a plugin.

Adding to Datasette itself is an interesting idea. I think it's possible that many users these days no longer assume they can paste a URL from the browser address bar (if they ever understood that at all) because to many apps are SPAs with broken URLs.

The shareable URLs are actually a key feature of Datasette - so maybe they should be highlighted in the default UI?

I built a ""copy to clipboard"" feature for `datasette-copyable` and wrote up how that works here: https://til.simonwillison.net/javascript/copy-button","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/1240#issuecomment-784312460,https://api.github.com/repos/simonw/datasette/issues/1240,784312460,MDEyOklzc3VlQ29tbWVudDc4NDMxMjQ2MA==,7107523,Kabouik,2021-02-23T16:07:10Z,2021-02-23T16:08:28Z,NONE,"Likewise, while answering to another issue regarding the Vega plugin, I realized that there is no such way of linking rows after a custom query, I only get this ""Link"" column with individual URLs for the default SQL view:

![ss-2021-02-23_170559](https://user-images.githubusercontent.com/7107523/108871491-1e3fd500-75f1-11eb-8f76-5d5a82cc14d7.png)

Or is it there and I am just missing the option in my custom queries?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814591962,Allow facetting on custom queries,
https://github.com/simonw/datasette/issues/1218#issuecomment-784157345,https://api.github.com/repos/simonw/datasette/issues/1218,784157345,MDEyOklzc3VlQ29tbWVudDc4NDE1NzM0NQ==,1244799,soobrosa,2021-02-23T12:12:17Z,2021-02-23T12:12:17Z,NONE,"Topline this fixed the same problem for me.
```
brew install python@3.7
ln -s /usr/local/opt/python@3.7/bin/python3.7 /usr/local/opt/python/bin/python3.7
pip3 uninstall -y numpy
pip3 uninstall -y setuptools
pip3 install setuptools
pip3 install numpy
pip3 install datasette-publish-fly
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803356942, /usr/local/opt/python3/bin/python3.6: bad interpreter: No such file or directory,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-783794520,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5,783794520,MDEyOklzc3VlQ29tbWVudDc4Mzc5NDUyMA==,306240,UtahDave,2021-02-23T01:13:54Z,2021-02-23T01:13:54Z,NONE,"Also, @simonw I created a test based off the existing tests. I think it's working correctly","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813880401,WIP: Add Gmail takeout mbox import,
https://github.com/simonw/datasette/issues/1239#issuecomment-783774084,https://api.github.com/repos/simonw/datasette/issues/1239,783774084,MDEyOklzc3VlQ29tbWVudDc4Mzc3NDA4NA==,9599,simonw,2021-02-23T00:18:56Z,2021-02-23T00:19:18Z,OWNER,"Bug is here: https://github.com/simonw/datasette/blob/42caabf7e9e6e4d69ef6dd7de16f2cd96bc79d5b/datasette/filters.py#L149-L165

Those `json_each` lines should be:

    select {t}.rowid from {t}, json_each([{t}].[{c}]) j","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",813978858,JSON filter fails if column contains spaces,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-783688547,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,783688547,MDEyOklzc3VlQ29tbWVudDc4MzY4ODU0Nw==,306240,UtahDave,2021-02-22T21:31:28Z,2021-02-22T21:31:28Z,NONE,"@Btibert3 I've opened a PR with my initial attempt at this.  Would you be willing to give this a try?

https://github.com/dogsheep/google-takeout-to-sqlite/pull/5","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/simonw/datasette/issues/1237#issuecomment-783676548,https://api.github.com/repos/simonw/datasette/issues/1237,783676548,MDEyOklzc3VlQ29tbWVudDc4MzY3NjU0OA==,9599,simonw,2021-02-22T21:10:19Z,2021-02-22T21:10:25Z,OWNER,This is another change which is a little bit hard to figure out because I haven't solved #878 yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812704869,?_pretty=1 option for pretty-printing JSON output,
https://github.com/simonw/datasette/issues/1234#issuecomment-783674659,https://api.github.com/repos/simonw/datasette/issues/1234,783674659,MDEyOklzc3VlQ29tbWVudDc4MzY3NDY1OQ==,9599,simonw,2021-02-22T21:06:28Z,2021-02-22T21:06:28Z,OWNER,"I'm not going to work on this for a while, but if anyone has needs or ideas around that they can add them to this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811505638,Runtime support for ATTACHing multiple databases,
https://github.com/simonw/datasette/issues/1236#issuecomment-783674038,https://api.github.com/repos/simonw/datasette/issues/1236,783674038,MDEyOklzc3VlQ29tbWVudDc4MzY3NDAzOA==,9599,simonw,2021-02-22T21:05:21Z,2021-02-22T21:05:21Z,OWNER,It's good on mobile - iOS at least. Going to close this open new issues if anyone reports bugs.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/sqlite-utils/issues/220#issuecomment-783662968,https://api.github.com/repos/simonw/sqlite-utils/issues/220,783662968,MDEyOklzc3VlQ29tbWVudDc4MzY2Mjk2OA==,649467,mhalle,2021-02-22T20:44:51Z,2021-02-22T20:44:51Z,NONE,"Actually, coming back to this, I have a clearer use case for enabling fts generation for views: making it easier to bring in text from lookup tables and other joins.  

The datasette documentation describes populating an fts table like so:
```
INSERT INTO ""items_fts"" (rowid, name, description, category_name)
    SELECT items. rowid,
    items.name,
    items.description,
    categories.name
    FROM items JOIN categories ON items.category_id=categories.id;
```
Alternatively if you have fts support in sqlite_utils for views (which sqlite and fts5 support), you can do the same thing just by creating a view that captures the above joins as columns, then creating an fts table from that view. Such an fts table can be created using sqlite_utils, where one created with your method can't. 

The resulting fts table can then be used by a whole family of related tables and views in the manner you described earlier in this issue. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",783778672,Better error message for *_fts methods against views,
https://github.com/simonw/datasette/issues/1166#issuecomment-783560017,https://api.github.com/repos/simonw/datasette/issues/1166,783560017,MDEyOklzc3VlQ29tbWVudDc4MzU2MDAxNw==,94334,thorn0,2021-02-22T18:00:57Z,2021-02-22T18:13:11Z,NONE,"Hi! I don't think Prettier supports this syntax for globs: `datasette/static/*[!.min].js` Are you sure that works?
Prettier uses https://github.com/mrmlnc/fast-glob, which in turn uses https://github.com/micromatch/micromatch, and the docs for these packages don't mention this syntax. As per the docs, square brackets should work as in regexes (`foo-[1-5].js`).

Tested it. Apparently, it works as a negated character class in regexes (like `[^.min]`). I wonder where this syntax comes from. Micromatch doesn't support that:

```js
micromatch(['static/table.js', 'static/n.js'], ['static/*[!.min].js']);
// result: [""static/n.js""] -- brackets are treated like [!.min] in regexes, without negation
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/782#issuecomment-783265830,https://api.github.com/repos/simonw/datasette/issues/782,783265830,MDEyOklzc3VlQ29tbWVudDc4MzI2NTgzMA==,30665,frankieroberto,2021-02-22T10:21:14Z,2021-02-22T10:21:14Z,NONE,"@simonw:

> The problem there is that ?_size=x isn't actually doing the same thing as the SQL limit keyword.

Interesting! Although I don't think it matters too much what the underlying implementation is - I more meant that `limit` is familiar to developers conceptually as ""up to and including this number, if they exist"", whereas ""size"" is potentially more ambiguous. However, it's probably no big deal either way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782789598,https://api.github.com/repos/simonw/datasette/issues/782,782789598,MDEyOklzc3VlQ29tbWVudDc4Mjc4OTU5OA==,9599,simonw,2021-02-21T03:30:02Z,2021-02-21T03:30:02Z,OWNER,Another benefit to default:object - I could include a key that shows a list of available extras. I could then use that to power an interactive API explorer.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782765665,https://api.github.com/repos/simonw/datasette/issues/782,782765665,MDEyOklzc3VlQ29tbWVudDc4Mjc2NTY2NQ==,9599,simonw,2021-02-20T23:34:41Z,2021-02-20T23:34:41Z,OWNER,"OK, I'm back to the ""top level object as the default"" side of things now - it's pretty much unanimous at this point, and it's certainly true that it's not a decision you'll even regret.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782756398,https://api.github.com/repos/simonw/datasette/issues/782,782756398,MDEyOklzc3VlQ29tbWVudDc4Mjc1NjM5OA==,601316,simonrjones,2021-02-20T22:05:48Z,2021-02-20T22:05:48Z,NONE,"> I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default.

I agree it is more predictable if the top level item is an object with a rows or data object that contains an array of data, which then allows for other top-level meta data. 

I can see the argument for removing this and just using an array for convenience - but I think that's OK as an option (as you have now).

Rather than have lots of top-level keys you could have a ""meta"" object to contain non-data stuff. You could use something like ""links"" for API endpoint URLs (or use a standard like HAL). Which would then leave the top level a bit cleaner - if that's what you what. 

Have you had much feedback from users who use the Datasette API a lot?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782748501,https://api.github.com/repos/simonw/datasette/issues/782,782748501,MDEyOklzc3VlQ29tbWVudDc4Mjc0ODUwMQ==,9599,simonw,2021-02-20T20:58:18Z,2021-02-20T20:58:18Z,OWNER,"Yet another option: support a `?_path=x` option which returns a nested path from the result. So you could do this:

`/github/commits.json?_path=rows` - to get back a top-level array pulled from the `""rows""` key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782748093,https://api.github.com/repos/simonw/datasette/issues/782,782748093,MDEyOklzc3VlQ29tbWVudDc4Mjc0ODA5Mw==,9599,simonw,2021-02-20T20:54:52Z,2021-02-20T20:54:52Z,OWNER,"> Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.

Adding a `?_pretty=1` option that does that is a great idea, I'm filing a ticket for it: #1237","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747878,https://api.github.com/repos/simonw/datasette/issues/782,782747878,MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzg3OA==,9599,simonw,2021-02-20T20:53:11Z,2021-02-20T20:53:11Z,OWNER,"... though thinking about this further, I could re-implement the `select * from commits` (but only return a max of 10 results) feature using a nested `select * from (select * from commits) limit 10` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747743,https://api.github.com/repos/simonw/datasette/issues/782,782747743,MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzc0Mw==,9599,simonw,2021-02-20T20:52:10Z,2021-02-20T20:52:10Z,OWNER,"> Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?

The problem there is that `?_size=x` isn't actually doing the same thing as the SQL `limit` keyword. Consider this query:

https://latest-with-plugins.datasette.io/github?sql=select+*+from+commits - `select * from commits`

Datasette returns 1,000 results, and shows a ""Custom SQL query returning more than 1,000 rows"" message at the top. That's the `size` kicking in - I only fetch the first 1,000 results from the cursor to avoid exhausting resources. In the JSON version of that at https://latest-with-plugins.datasette.io/github.json?sql=select+*+from+commits there's a `""truncated"": true` key to let you know what happened.

I find myself using `?_size=2` against Datasette occasionally if I know the rows being returned are really big and I don't want to load 10+MB of HTML.

This is only really a concern for arbitrary SQL queries though - for table pages such as https://latest-with-plugins.datasette.io/github/commits?_size=10 adding `?_size=10` actually puts a `limit 10` on the underlying SQL query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782747164,https://api.github.com/repos/simonw/datasette/issues/782,782747164,MDEyOklzc3VlQ29tbWVudDc4Mjc0NzE2NA==,9599,simonw,2021-02-20T20:47:16Z,2021-02-20T20:47:16Z,OWNER,(I started a thread on Twitter about this: https://twitter.com/simonw/status/1363220355318358016),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782746755,https://api.github.com/repos/simonw/datasette/issues/782,782746755,MDEyOklzc3VlQ29tbWVudDc4Mjc0Njc1NQ==,30665,frankieroberto,2021-02-20T20:44:05Z,2021-02-20T20:44:05Z,NONE,"Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?

I like the idea of specifying a limit of 0 if you don’t want any rows data - and returning an empty array under the `rows` key seems fine.

Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782746633,https://api.github.com/repos/simonw/datasette/issues/782,782746633,MDEyOklzc3VlQ29tbWVudDc4Mjc0NjYzMw==,9599,simonw,2021-02-20T20:43:07Z,2021-02-20T20:43:07Z,OWNER,"Another option: `.json` always returns an object with a list of keys that gets increased through adding `?_extra=` parameters.

`.jsona` always returns a JSON array of objects

I had something similar to this in Datasette a few years ago - a `.jsono` extension, which still redirects to the `shape=array` version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782745199,https://api.github.com/repos/simonw/datasette/issues/782,782745199,MDEyOklzc3VlQ29tbWVudDc4Mjc0NTE5OQ==,30665,frankieroberto,2021-02-20T20:32:03Z,2021-02-20T20:32:03Z,NONE,"I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default. Mainly because it allows you to add extra keys in a backwards-compatible way. Also just seems more expected somehow.

The API design guidance for the UK government also recommends this: https://www.gov.uk/guidance/gds-api-technical-and-data-standards#use-json

I also strongly dislike having versioned APIs (eg with a `/v1/` path prefix, as it invariably means that old versions stop working at some point, even though the bit of the API you’re using might not have changed at all.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782742233,https://api.github.com/repos/simonw/datasette/issues/782,782742233,MDEyOklzc3VlQ29tbWVudDc4Mjc0MjIzMw==,9599,simonw,2021-02-20T20:09:16Z,2021-02-20T20:09:16Z,OWNER,I just noticed that https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0&_trace=1 executes 35 SQL queries at the moment! A great reminder that a big improvement from this change will be a reduction in queries through not calculating things like suggested facets unless they are explicitly requested.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782741719,https://api.github.com/repos/simonw/datasette/issues/782,782741719,MDEyOklzc3VlQ29tbWVudDc4Mjc0MTcxOQ==,9599,simonw,2021-02-20T20:05:04Z,2021-02-20T20:05:04Z,OWNER,"> The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.

This is something I've not thought very hard about. If there's an error, I need to return a top-level object, not a top-level array, so I can provide details of the error.

But this means that client code will have to handle this difference - it will have to know that the returned data can be array-shaped if nothing went wrong, and object-shaped if there's an error.

The HTTP status code helps here - calling client code can know that a 200 status code means there will be an array, but an error status code means an object.

If developers really hate that the shape could be different, they can always use `?_extra=next` to ensure that the top level item is an object whether or not an error occurred. So I think this is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782741107,https://api.github.com/repos/simonw/datasette/issues/782,782741107,MDEyOklzc3VlQ29tbWVudDc4Mjc0MTEwNw==,9599,simonw,2021-02-20T20:00:22Z,2021-02-20T20:00:22Z,OWNER,"A really exciting opportunity this opens up is for parallel execution - the `facets()` and `suggested_facets()` and `total()` async functions could be called in parallel, which could speed things up if I'm confident the SQLite thread pool can execute on multiple CPU cores (it should be able to because the Python `sqlite3` module releases the GIL while it's executing C code).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740985,https://api.github.com/repos/simonw/datasette/issues/782,782740985,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDk4NQ==,9599,simonw,2021-02-20T19:59:21Z,2021-02-20T19:59:21Z,OWNER,"This design should be influenced by how it's implemented.

One implementation that could be nice is that each of the keys that can be requested - `next_url`, `total` etc - maps to an `async def` function which can do the work. So that expensive `count(*)` will only be executed by the `async def total` function if it is requested.

This raises more questions: Both `next` and `next_url` work off the same underlying data, so if they are both requested can we re-use the work that `next` does somehow? Maybe by letting these functions depend on each other (so `next_url()` knows to first call `next()`, but only if it hasn't been called already.

I think I need to flesh out the full default collection of `?_extra=` parameters in order to design how they will work under the hood.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740604,https://api.github.com/repos/simonw/datasette/issues/782,782740604,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDYwNA==,9599,simonw,2021-02-20T19:56:21Z,2021-02-20T19:56:33Z,OWNER,"I think I want to support `?_extra=next_url,total` in addition to `?_extra=next_url&_extra=total` - partly because it's less characters to type, and also because I know there exist URL handling library that don't know how to handle the same parameter multiple times (though they're going to break against Datasette already, so it's not a big deal).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782740488,https://api.github.com/repos/simonw/datasette/issues/782,782740488,MDEyOklzc3VlQ29tbWVudDc4Mjc0MDQ4OA==,9599,simonw,2021-02-20T19:55:23Z,2021-02-20T19:55:23Z,OWNER,"Am I saying you won't get back a key in the response unless you explicitly request it, either by name or by specifying a bundle of extras (e.g. `all` or `paginated`)?

The `""truncated"": true` key that tells you that your arbitrary query returned more than X results but was truncated is pretty important, do I really want people to have to opt-in to that one?

Also: having bundles like `all` or `paginated` live in the same namespace as single keys like `next_url` or `total` is a little odd - you can't tell by looking at them if they'll add a key called `all` or if they'll add a bunch of other stuff.

Maybe bundles could be prefixed with something, perhaps an underscore? `?_extra=_all` and `?_extra=_paginated` for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782739926,https://api.github.com/repos/simonw/datasette/issues/782,782739926,MDEyOklzc3VlQ29tbWVudDc4MjczOTkyNg==,9599,simonw,2021-02-20T19:51:30Z,2021-02-20T19:52:19Z,OWNER,"Demos:

- https://latest-with-plugins.datasette.io/github/commits.json-preview
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url&_extra=total
- https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782709425,https://api.github.com/repos/simonw/datasette/issues/782,782709425,MDEyOklzc3VlQ29tbWVudDc4MjcwOTQyNQ==,9599,simonw,2021-02-20T16:24:54Z,2021-02-20T16:24:54Z,OWNER,Having shortcuts means I could support `?_extra=all` for returning ALL possible keys.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782709270,https://api.github.com/repos/simonw/datasette/issues/782,782709270,MDEyOklzc3VlQ29tbWVudDc4MjcwOTI3MA==,9599,simonw,2021-02-20T16:23:51Z,2021-02-20T16:24:11Z,OWNER,"Also how would you opt out of returning the `""rows""` key? I sometimes want to do this - if I want to get back just the count or just the facets for example.

Some options:

* `/fixtures/roadside_attractions.json?_extra=total&_extra=-rows`
* `/fixtures/roadside_attractions.json?_extra=total&_skip=rows`
* `/fixtures/roadside_attractions.json?_extra=total&_size=0`

I quite like that last one with `?_size=0`. I think it would still return `""rows"": []` but that's OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/782#issuecomment-782708938,https://api.github.com/repos/simonw/datasette/issues/782,782708938,MDEyOklzc3VlQ29tbWVudDc4MjcwODkzOA==,9599,simonw,2021-02-20T16:22:14Z,2021-02-20T16:22:14Z,OWNER,"I'm leaning back in the direction of a flat JSON array of objects as the default - this:

`/fixtures/roadside_attractions.json`

Would return:

```json
[
    {
      ""pk"": 1,
      ""name"": ""The Mystery Spot"",
      ""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
      ""latitude"": 37.0167,
      ""longitude"": -122.0024
    },
    {
      ""pk"": 2,
      ""name"": ""Winchester Mystery House"",
      ""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
      ""latitude"": 37.3184,
      ""longitude"": -121.9511
    },
    {
      ""pk"": 3,
      ""name"": ""Burlingame Museum of PEZ Memorabilia"",
      ""address"": ""214 California Drive, Burlingame, CA 94010"",
      ""latitude"": 37.5793,
      ""longitude"": -122.3442
    },
    {
      ""pk"": 4,
      ""name"": ""Bigfoot Discovery Museum"",
      ""address"": ""5497 Highway 9, Felton, CA 95018"",
      ""latitude"": 37.0414,
      ""longitude"": -122.0725
    }
]
```
To get the version that includes pagination information you would use the `?_extra=` parameter. For example:

`/fixtures/roadside_attractions.json?_extra=total&_extra=next_url`

```json
{
  ""rows"": [
    {
      ""pk"": 1,
      ""name"": ""The Mystery Spot"",
      ""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"",
      ""latitude"": 37.0167,
      ""longitude"": -122.0024
    },
    {
      ""pk"": 2,
      ""name"": ""Winchester Mystery House"",
      ""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"",
      ""latitude"": 37.3184,
      ""longitude"": -121.9511
    },
    {
      ""pk"": 3,
      ""name"": ""Burlingame Museum of PEZ Memorabilia"",
      ""address"": ""214 California Drive, Burlingame, CA 94010"",
      ""latitude"": 37.5793,
      ""longitude"": -122.3442
    },
    {
      ""pk"": 4,
      ""name"": ""Bigfoot Discovery Museum"",
      ""address"": ""5497 Highway 9, Felton, CA 95018"",
      ""latitude"": 37.0414,
      ""longitude"": -122.0725
    }
  ],
  ""total"": 4,
  ""next_url"": null
}
```
ANY usage of the `?_extra=` parameter would turn the list into an object with a `""rows""` key.

Opting in to the `total` is nice because it's actually expensive to run a count, so only doing a count if the user requests it feels good.

But... having to add `?_extra=total&_extra=next_url` for the common case of wanting both the total count and the URL to get the next page of results is a bit verbose. So maybe support aliases, like `?_extra=paginated` which is a shortcut for `?_extra=total&_extra=next_url`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1236#issuecomment-782464306,https://api.github.com/repos/simonw/datasette/issues/1236,782464306,MDEyOklzc3VlQ29tbWVudDc4MjQ2NDMwNg==,9599,simonw,2021-02-19T23:57:32Z,2021-02-19T23:57:32Z,OWNER,Need to test this on mobile.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782464215,https://api.github.com/repos/simonw/datasette/issues/1236,782464215,MDEyOklzc3VlQ29tbWVudDc4MjQ2NDIxNQ==,9599,simonw,2021-02-19T23:57:13Z,2021-02-19T23:57:13Z,OWNER,Now live on https://latest.datasette.io/_memory,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782462049,https://api.github.com/repos/simonw/datasette/issues/1236,782462049,MDEyOklzc3VlQ29tbWVudDc4MjQ2MjA0OQ==,9599,simonw,2021-02-19T23:51:12Z,2021-02-19T23:51:12Z,OWNER,"![resize-demo](https://user-images.githubusercontent.com/9599/108573758-4914eb00-72ca-11eb-989c-e642eee68021.gif)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782459550,https://api.github.com/repos/simonw/datasette/issues/1236,782459550,MDEyOklzc3VlQ29tbWVudDc4MjQ1OTU1MA==,9599,simonw,2021-02-19T23:45:30Z,2021-02-19T23:45:30Z,OWNER,"Encoded using https://meyerweb.com/eric/tools/dencoder/

`%3Csvg%20aria-labelledby%3D%22cm-drag-to-resize%22%20role%3D%22img%22%20fill%3D%22%23ccc%22%20stroke%3D%22%23ccc%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2016%2016%22%20width%3D%2216%22%20height%3D%2216%22%3E%0A%20%20%3Ctitle%20id%3D%22cm-drag-to-resize%22%3EDrag%20to%20resize%3C%2Ftitle%3E%0A%20%20%3Cpath%20fill-rule%3D%22evenodd%22%20d%3D%22M1%202.75A.75.75%200%20011.75%202h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%202.75zm0%205A.75.75%200%20011.75%207h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%207.75zM1.75%2012a.75.75%200%20100%201.5h12.5a.75.75%200%20100-1.5H1.75z%22%3E%3C%2Fpath%3E%0A%3C%2Fsvg%3E`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782459405,https://api.github.com/repos/simonw/datasette/issues/1236,782459405,MDEyOklzc3VlQ29tbWVudDc4MjQ1OTQwNQ==,9599,simonw,2021-02-19T23:45:02Z,2021-02-19T23:45:02Z,OWNER,"I'm going to use a variant of the Datasette menu icon. Here it is in `#ccc` with an ARIA label:

```svg
<svg aria-labelledby=""cm-drag-to-resize"" role=""img"" fill=""#ccc"" stroke=""#ccc"" xmlns=""http://www.w3.org/2000/svg"" viewBox=""0 0 16 16"" width=""16"" height=""16"">
  <title id=""cm-drag-to-resize"">Drag to resize</title>
  <path fill-rule=""evenodd"" d=""M1 2.75A.75.75 0 011.75 2h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 2.75zm0 5A.75.75 0 011.75 7h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 7.75zM1.75 12a.75.75 0 100 1.5h12.5a.75.75 0 100-1.5H1.75z""></path>
</svg>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782458983,https://api.github.com/repos/simonw/datasette/issues/1236,782458983,MDEyOklzc3VlQ29tbWVudDc4MjQ1ODk4Mw==,9599,simonw,2021-02-19T23:43:34Z,2021-02-19T23:43:34Z,OWNER,"I only want it to resize up and down, not left to right - so I'm not keen on the default resize handle:

<img width=""722"" alt=""cm-resize_demo"" src=""https://user-images.githubusercontent.com/9599/108573363-364de680-72c9-11eb-8741-5112463ebfaa.png"">

https://rawgit.com/Sphinxxxx/cm-resize/master/demo/index.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1236#issuecomment-782458744,https://api.github.com/repos/simonw/datasette/issues/1236,782458744,MDEyOklzc3VlQ29tbWVudDc4MjQ1ODc0NA==,9599,simonw,2021-02-19T23:42:42Z,2021-02-19T23:42:42Z,OWNER,I can use https://github.com/Sphinxxxx/cm-resize for this,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",812228314,Ability to increase size of the SQL editor window,
https://github.com/simonw/datasette/issues/1212#issuecomment-782430028,https://api.github.com/repos/simonw/datasette/issues/1212,782430028,MDEyOklzc3VlQ29tbWVudDc4MjQzMDAyOA==,4488943,kbaikov,2021-02-19T22:54:13Z,2021-02-19T22:54:13Z,CONTRIBUTOR,I will close this issue since it appears only in my particular setup.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797651831,Tests are very slow. ,
https://github.com/simonw/datasette/issues/619#issuecomment-782246111,https://api.github.com/repos/simonw/datasette/issues/619,782246111,MDEyOklzc3VlQ29tbWVudDc4MjI0NjExMQ==,9599,simonw,2021-02-19T18:11:22Z,2021-02-19T18:11:22Z,OWNER,"Big usability improvement, see also #1236","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL",
https://github.com/simonw/datasette/pull/1229#issuecomment-782053455,https://api.github.com/repos/simonw/datasette/issues/1229,782053455,MDEyOklzc3VlQ29tbWVudDc4MjA1MzQ1NQ==,295329,camallen,2021-02-19T12:47:19Z,2021-02-19T12:47:19Z,CONTRIBUTOR,I believe this pr and #1031 are related and fix the same issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810507413,ensure immutable databses when starting in configuration directory mode with,
https://github.com/simonw/sqlite-utils/issues/236#issuecomment-781825726,https://api.github.com/repos/simonw/sqlite-utils/issues/236,781825726,MDEyOklzc3VlQ29tbWVudDc4MTgyNTcyNg==,9599,simonw,2021-02-19T05:10:41Z,2021-02-19T05:10:41Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#attaching-additional-databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811680502,--attach command line option for attaching extra databases,
https://github.com/simonw/sqlite-utils/issues/113#issuecomment-781825187,https://api.github.com/repos/simonw/sqlite-utils/issues/113,781825187,MDEyOklzc3VlQ29tbWVudDc4MTgyNTE4Nw==,9599,simonw,2021-02-19T05:09:12Z,2021-02-19T05:09:12Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#attaching-additional-databases,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621286870,Syntactic sugar for ATTACH DATABASE,
https://github.com/simonw/datasette/issues/283#issuecomment-781764561,https://api.github.com/repos/simonw/datasette/issues/283,781764561,MDEyOklzc3VlQ29tbWVudDc4MTc2NDU2MQ==,9599,simonw,2021-02-19T02:10:21Z,2021-02-19T02:10:21Z,OWNER,This feature is now released! https://docs.datasette.io/en/stable/changelog.html#v0-55,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1235#issuecomment-781736855,https://api.github.com/repos/simonw/datasette/issues/1235,781736855,MDEyOklzc3VlQ29tbWVudDc4MTczNjg1NQ==,9599,simonw,2021-02-19T00:52:47Z,2021-02-19T01:47:53Z,OWNER,"I bumped the two lines in the `Dockerfile` to `FROM python:3.7.10-slim-stretch as build` and ran this to build it:

    docker build -f Dockerfile -t datasetteproject/datasette:python-3-7-10 .

Then I ran it with:

    docker run -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette:python-3-7-10 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

http://0.0.0.0:8001/-/versions confirmed that it was now running Python 3.7.10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811589344,Upgrade Python version used by official Datasette Docker image,
https://github.com/simonw/datasette/issues/1235#issuecomment-781735887,https://api.github.com/repos/simonw/datasette/issues/1235,781735887,MDEyOklzc3VlQ29tbWVudDc4MTczNTg4Nw==,9599,simonw,2021-02-19T00:50:21Z,2021-02-19T00:50:55Z,OWNER,"I'll bump to `3.7.10` for the moment - the fix for 3.8 isn't out until March 1st according to https://news.ycombinator.com/item?id=26186434

https://www.python.org/downloads/release/python-3710/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811589344,Upgrade Python version used by official Datasette Docker image,
https://github.com/simonw/datasette/issues/283#issuecomment-781670827,https://api.github.com/repos/simonw/datasette/issues/283,781670827,MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw==,9599,simonw,2021-02-18T22:16:46Z,2021-02-18T22:16:46Z,OWNER,"Demo is now live here: https://latest.datasette.io/_memory

The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/pull/1232#issuecomment-781599929,https://api.github.com/repos/simonw/datasette/issues/1232,781599929,MDEyOklzc3VlQ29tbWVudDc4MTU5OTkyOQ==,22429695,codecov[bot],2021-02-18T19:59:54Z,2021-02-18T22:06:42Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=h1) Report
> Merging [#1232](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=desc) (8876499) into [main](https://codecov.io/gh/simonw/datasette/commit/4df548e7668b5b21d64a267964951e67894f4712?el=desc) (4df548e) will **increase** coverage by `0.03%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1232/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main    #1232      +/-   ##
==========================================
+ Coverage   91.42%   91.46%   +0.03%     
==========================================
  Files          32       32              
  Lines        3955     3970      +15     
==========================================
+ Hits         3616     3631      +15     
  Misses        339      339              
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.68% <100.00%> (+0.06%)` | :arrow_up: |
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `76.62% <100.00%> (+0.36%)` | :arrow_up: |
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.19% <100.00%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=footer). Last update [4df548e...8876499](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/issues/283#issuecomment-781665560,https://api.github.com/repos/simonw/datasette/issues/283,781665560,MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA==,9599,simonw,2021-02-18T22:06:14Z,2021-02-18T22:06:14Z,OWNER,"The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases.

It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you.

So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection.

Also worth noting: plugins that implement the [prepare_connection()](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette) hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/pull/1232#issuecomment-781651283,https://api.github.com/repos/simonw/datasette/issues/1232,781651283,MDEyOklzc3VlQ29tbWVudDc4MTY1MTI4Mw==,9599,simonw,2021-02-18T21:37:55Z,2021-02-18T21:37:55Z,OWNER,"UI listing the attached tables:

<img width=""888"" alt=""_memory"" src=""https://user-images.githubusercontent.com/9599/108424809-8015ce80-71ee-11eb-8752-03c459f89320.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781641728,https://api.github.com/repos/simonw/datasette/issues/1232,781641728,MDEyOklzc3VlQ29tbWVudDc4MTY0MTcyOA==,9599,simonw,2021-02-18T21:19:34Z,2021-02-18T21:19:34Z,OWNER,"I tested the demo deployment like this:
```
datasette publish cloudrun fixtures.db extra_database.db \                                      
            -m fixtures.json \
            --plugins-dir=plugins \
            --branch=crossdb \
            --extra-options=""--setting template_debug 1 --crossdb"" \
            --install=pysqlite3-binary \
            --service=datasette-latest-crossdb
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781637292,https://api.github.com/repos/simonw/datasette/issues/1232,781637292,MDEyOklzc3VlQ29tbWVudDc4MTYzNzI5Mg==,9599,simonw,2021-02-18T21:11:31Z,2021-02-18T21:11:31Z,OWNER,Due to bug #1233 I'm going to publish the additional database as `extra_database.db` rather than `extra database.db` as it is used in the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/issues/1233#issuecomment-781636590,https://api.github.com/repos/simonw/datasette/issues/1233,781636590,MDEyOklzc3VlQ29tbWVudDc4MTYzNjU5MA==,9599,simonw,2021-02-18T21:10:08Z,2021-02-18T21:10:08Z,OWNER,I think the bug is here: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/__init__.py#L349-L373,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811458446,"""datasette publish cloudrun"" cannot publish files with spaces in their name",
https://github.com/simonw/datasette/pull/1232#issuecomment-781634819,https://api.github.com/repos/simonw/datasette/issues/1232,781634819,MDEyOklzc3VlQ29tbWVudDc4MTYzNDgxOQ==,9599,simonw,2021-02-18T21:06:43Z,2021-02-18T21:06:43Z,OWNER,I'll document this option on https://docs.datasette.io/en/stable/sql_queries.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781629841,https://api.github.com/repos/simonw/datasette/issues/1232,781629841,MDEyOklzc3VlQ29tbWVudDc4MTYyOTg0MQ==,9599,simonw,2021-02-18T20:57:23Z,2021-02-18T20:57:23Z,OWNER,"The new warning looks like this:

<img width=""514"" alt=""datasette_—_pipenv_shell_▸_Python_—_182×66"" src=""https://user-images.githubusercontent.com/9599/108420562-d41db480-71e8-11eb-87e3-1cddd65627c2.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781598585,https://api.github.com/repos/simonw/datasette/issues/1232,781598585,MDEyOklzc3VlQ29tbWVudDc4MTU5ODU4NQ==,9599,simonw,2021-02-18T19:57:30Z,2021-02-18T19:57:30Z,OWNER,It would also be neat if https://latest.datasette.io/ had multiple databases attached in order to demonstrate this feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/pull/1232#issuecomment-781594632,https://api.github.com/repos/simonw/datasette/issues/1232,781594632,MDEyOklzc3VlQ29tbWVudDc4MTU5NDYzMg==,9599,simonw,2021-02-18T19:50:21Z,2021-02-18T19:50:21Z,OWNER,"It would be neat if the `/_memory` page showed a list of attached databases, to indicate that the `--crossdb` option is working and give people links to click to start running queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811407131,--crossdb option for joining across databases,
https://github.com/simonw/datasette/issues/283#issuecomment-781593169,https://api.github.com/repos/simonw/datasette/issues/283,781593169,MDEyOklzc3VlQ29tbWVudDc4MTU5MzE2OQ==,9599,simonw,2021-02-18T19:47:34Z,2021-02-18T19:47:34Z,OWNER,"I have a working version now, moving development to a pull request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-781591015,https://api.github.com/repos/simonw/datasette/issues/283,781591015,MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ==,9599,simonw,2021-02-18T19:44:02Z,2021-02-18T19:44:02Z,OWNER,For the moment I'm going to hard-code a `SQLITE_LIMIT_ATTACHED=10` constant and only attach the first 10 databases to the `_memory` connection.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-781574786,https://api.github.com/repos/simonw/datasette/issues/283,781574786,MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng==,9599,simonw,2021-02-18T19:15:37Z,2021-02-18T19:15:37Z,OWNER,`select * from pragma_database_list();` is useful - shows all attached databases for the current connection.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-781573676,https://api.github.com/repos/simonw/datasette/issues/283,781573676,MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng==,9599,simonw,2021-02-18T19:13:30Z,2021-02-18T19:13:30Z,OWNER,"It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/1231#issuecomment-781560989,https://api.github.com/repos/simonw/datasette/issues/1231,781560989,MDEyOklzc3VlQ29tbWVudDc4MTU2MDk4OQ==,9599,simonw,2021-02-18T18:50:53Z,2021-02-18T18:50:53Z,OWNER,Ideally I'd figure out a way to replicate this error in a concurrent unit test.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1231#issuecomment-781560865,https://api.github.com/repos/simonw/datasette/issues/1231,781560865,MDEyOklzc3VlQ29tbWVudDc4MTU2MDg2NQ==,9599,simonw,2021-02-18T18:50:38Z,2021-02-18T18:50:38Z,OWNER,"I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 9e15a16..1681c9d 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -217,6 +217,7 @@ class Datasette:
         self.inspect_data = inspect_data
         self.immutables = set(immutables or [])
         self.databases = collections.OrderedDict()
+        self._refresh_schemas_lock = threading.Lock()
         if memory or not self.files:
             self.add_database(Database(self, is_memory=True), name=""_memory"")
         # memory_name is a random string so that each Datasette instance gets its own
@@ -324,6 +325,13 @@ class Datasette:
         self.client = DatasetteClient(self)
 
     async def refresh_schemas(self):
+        return
+        if self._refresh_schemas_lock.locked():
+            return
+        with self._refresh_schemas_lock:
+            await self._refresh_schemas()
+
+    async def _refresh_schemas(self):
         internal_db = self.databases[""_internal""]
         if not self.internal_db_created:
             await init_internal_db(internal_db)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811367257,Race condition errors in new refresh_schemas() mechanism,
https://github.com/simonw/datasette/issues/1226#issuecomment-781546512,https://api.github.com/repos/simonw/datasette/issues/1226,781546512,MDEyOklzc3VlQ29tbWVudDc4MTU0NjUxMg==,9599,simonw,2021-02-18T18:26:19Z,2021-02-18T18:26:19Z,OWNER,This broke CI: https://github.com/simonw/datasette/runs/1929355965?check_suite_focus=true,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/simonw/datasette/issues/1226#issuecomment-781530157,https://api.github.com/repos/simonw/datasette/issues/1226,781530157,MDEyOklzc3VlQ29tbWVudDc4MTUzMDE1Nw==,9599,simonw,2021-02-18T18:00:15Z,2021-02-18T18:00:15Z,OWNER,"I can use `click.IntRange(min=None, max=None)` for this. https://click.palletsprojects.com/en/7.x/options/#ranges - inclusive on both edges.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-781451701,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,781451701,MDEyOklzc3VlQ29tbWVudDc4MTQ1MTcwMQ==,203343,Btibert3,2021-02-18T16:06:21Z,2021-02-18T16:06:21Z,NONE,Awesome!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/simonw/datasette/issues/1230#issuecomment-781330466,https://api.github.com/repos/simonw/datasette/issues/1230,781330466,MDEyOklzc3VlQ29tbWVudDc4MTMzMDQ2Ng==,7107523,Kabouik,2021-02-18T13:06:22Z,2021-02-18T15:22:15Z,NONE,"[Edit] Oh, I just saw the ""Load all"" button under the cluster map as well as the [setting to alter the max number or results](https://docs.datasette.io/en/stable/settings.html#max-returned-rows). So I guess this issue only is about the Vega charts.

<details>
Note that datasette-cluster-map also seems to be limited to 998 displayed points: 

![ss-2021-02-18_140548](https://user-images.githubusercontent.com/7107523/108361225-15fb2a80-71ea-11eb-9a19-d885e8513f55.png)
</details>","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",811054000,"Vega charts are plotted only for rows on the visible page, cluster maps only for rows in the remaining pages",
https://github.com/simonw/datasette/issues/283#issuecomment-781077127,https://api.github.com/repos/simonw/datasette/issues/283,781077127,MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw==,9599,simonw,2021-02-18T05:56:30Z,2021-02-18T05:57:34Z,OWNER,I'm going to to try prototyping the `--crossdb` option that causes `/_memory` to connect to all databases as a starting point and see how well that works.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/simonw/datasette/issues/283#issuecomment-780991910,https://api.github.com/repos/simonw/datasette/issues/283,780991910,MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA==,9308268,rayvoelker,2021-02-18T02:13:56Z,2021-02-18T02:13:56Z,NONE,"I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining?

I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots

https://ilsweb.cincinnatilibrary.org/collection-analysis/

![image](https://user-images.githubusercontent.com/9308268/108294883-ba3a8e00-7164-11eb-9206-fcd5a8cdd883.png)

and thanks again for such an amazing tool!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",325958506,Support cross-database joins,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-780817596,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4,780817596,MDEyOklzc3VlQ29tbWVudDc4MDgxNzU5Ng==,306240,UtahDave,2021-02-17T20:01:35Z,2021-02-17T20:01:35Z,NONE,I've got this almost working. Just needs some polish,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778380836,Feature Request: Gmail,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-779785638,https://api.github.com/repos/simonw/sqlite-utils/issues/227,779785638,MDEyOklzc3VlQ29tbWVudDc3OTc4NTYzOA==,295329,camallen,2021-02-16T11:48:03Z,2021-02-16T11:48:03Z,NONE,Thank you @simonw ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,Error reading csv files with large column data,
https://github.com/simonw/datasette/issues/1226#issuecomment-779467451,https://api.github.com/repos/simonw/datasette/issues/1226,779467451,MDEyOklzc3VlQ29tbWVudDc3OTQ2NzQ1MQ==,9599,simonw,2021-02-15T22:02:46Z,2021-02-15T22:02:46Z,OWNER,"I'm OK with the current error message shown if you try to use too low a port:
```
datasette fivethirtyeight.db -p 800 
INFO:     Started server process [45511]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
ERROR:    [Errno 13] error while attempting to bind on address ('127.0.0.1', 800): permission denied
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/simonw/datasette/issues/1226#issuecomment-779467160,https://api.github.com/repos/simonw/datasette/issues/1226,779467160,MDEyOklzc3VlQ29tbWVudDc3OTQ2NzE2MA==,9599,simonw,2021-02-15T22:01:53Z,2021-02-15T22:01:53Z,OWNER,"This check needs to happen in two places:

https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L222-L227

https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L328-L333","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808843401,--port option should validate port is between 0 and 65535,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779416619,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779416619,MDEyOklzc3VlQ29tbWVudDc3OTQxNjYxOQ==,9599,simonw,2021-02-15T19:40:57Z,2021-02-15T21:27:55Z,OWNER,"Tried this experiment (not proper binary search, it only searches downwards):
```python
import sqlite3

db = sqlite3.connect("":memory:"")

def tryit(n):
    sql = ""select 1 where 1 in ({})"".format("", "".join(""?"" for i in range(n)))
    db.execute(sql, [0 for i in range(n)])


def find_limit(min=0, max=5_000_000):
    value = max
    while True:
        print('Trying', value)
        try:
            tryit(value)
            return value
        except:
            value = value // 2
```
Running `find_limit()` with those default parameters takes about 1.47s on my laptop:
```
In [9]: %timeit find_limit()
Trying 5000000
Trying 2500000...
1.47 s ± 28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Interestingly the value it suggested was 156250 - suggesting that the macOS `sqlite3` binary with a 500,000 limit isn't the same as whatever my Python is using here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779448912,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779448912,MDEyOklzc3VlQ29tbWVudDc3OTQ0ODkxMg==,9599,simonw,2021-02-15T21:09:50Z,2021-02-15T21:09:50Z,OWNER,"I fiddled around and replaced that line with `batch_size = SQLITE_MAX_VARS // num_columns` - which evaluated to `10416` for this particular file. That got me this:

       40.71s user 1.81s system 98% cpu 43.081 total

43s is definitely better than 56s, but it's still not as big as the  ~26.5s to ~3.5s improvement described by @simonwiles at the top of this issue. I wonder what I'm missing here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779446652,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779446652,MDEyOklzc3VlQ29tbWVudDc3OTQ0NjY1Mg==,9599,simonw,2021-02-15T21:04:19Z,2021-02-15T21:04:19Z,OWNER,"... but it looks like `batch_size` is hard-coded to 100, rather than `None` - which means it's not being calculated using that value:

https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L704

And

https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L1877","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779445423,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779445423,MDEyOklzc3VlQ29tbWVudDc3OTQ0NTQyMw==,9599,simonw,2021-02-15T21:00:44Z,2021-02-15T21:01:09Z,OWNER,"I tried changing the hard-coded value from 999 to 156_250 and running `sqlite-utils insert` against a 500MB CSV file, with these results:
```
(sqlite-utils) sqlite-utils % time sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
  [###################################-]   99%  00:00:00sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv
44.74s user 7.61s system 92% cpu 56.601 total
# Increased the setting here
(sqlite-utils) sqlite-utils % time sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
  [###################################-]   99%  00:00:00sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv
39.40s user 5.15s system 96% cpu 46.320 total
```
Not as big a difference as I was expecting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779417723,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779417723,MDEyOklzc3VlQ29tbWVudDc3OTQxNzcyMw==,9599,simonw,2021-02-15T19:44:02Z,2021-02-15T19:47:00Z,OWNER,"`%timeit find_limit(max=1_000_000)` took 378ms on my laptop

`%timeit find_limit(max=500_000)` took 197ms

`%timeit find_limit(max=200_000)` reported 53ms per loop

`%timeit find_limit(max=100_000)` reported 26.8ms per loop.

All of these are still slow enough that I'm not comfortable running this search for every time the library is imported. Allowing users to opt-in to this as a performance enhancement might be better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779409770,https://api.github.com/repos/simonw/sqlite-utils/issues/147,779409770,MDEyOklzc3VlQ29tbWVudDc3OTQwOTc3MA==,9599,simonw,2021-02-15T19:23:11Z,2021-02-15T19:23:11Z,OWNER,"On my Mac right now I'm seeing a limit of 500,000:
```
% sqlite3 -cmd "".limits variable_number""
     variable_number 500000
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158,SQLITE_MAX_VARS maybe hard-coded too low,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778854808,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778854808,MDEyOklzc3VlQ29tbWVudDc3ODg1NDgwOA==,9599,simonw,2021-02-14T22:46:54Z,2021-02-14T22:46:54Z,OWNER,Fix is released in 3.5.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,Error reading csv files with large column data,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778851721,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778851721,MDEyOklzc3VlQ29tbWVudDc3ODg1MTcyMQ==,9599,simonw,2021-02-14T22:23:46Z,2021-02-14T22:23:46Z,OWNER,I called this `--no-headers` for consistency with the existing output option: https://github.com/simonw/sqlite-utils/blob/427dace184c7da57f4a04df07b1e84cdae3261e8/sqlite_utils/cli.py#L61-L64,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778849394,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778849394,MDEyOklzc3VlQ29tbWVudDc3ODg0OTM5NA==,9599,simonw,2021-02-14T22:06:53Z,2021-02-14T22:06:53Z,OWNER,"For the moment I think just adding `--no-header` - which causes column names ""unknown1,unknown2,..."" to be used - should be enough.

Users can import with that option, then use `sqlite-utils transform --rename` to rename them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778844016,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778844016,MDEyOklzc3VlQ29tbWVudDc3ODg0NDAxNg==,9599,simonw,2021-02-14T21:22:45Z,2021-02-14T21:22:45Z,OWNER,"I'm going to use this pattern from https://stackoverflow.com/a/15063941
```python
import sys
import csv
maxInt = sys.maxsize

while True:
    # decrease the maxInt value by factor 10 
    # as long as the OverflowError occurs.

    try:
        csv.field_size_limit(maxInt)
        break
    except OverflowError:
        maxInt = int(maxInt/10)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,Hitting `_csv.Error: field larger than field limit (131072)`,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843503,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778843503,MDEyOklzc3VlQ29tbWVudDc3ODg0MzUwMw==,9599,simonw,2021-02-14T21:18:51Z,2021-02-14T21:18:51Z,OWNER,"I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving `ctypes`. I'm a bit loathe to add a dependency on `ctypes` though - even though it's in the Python standard library I worry that it might not be available on some architectures.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,Hitting `_csv.Error: field larger than field limit (131072)`,
https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843362,https://api.github.com/repos/simonw/sqlite-utils/issues/229,778843362,MDEyOklzc3VlQ29tbWVudDc3ODg0MzM2Mg==,9599,simonw,2021-02-14T21:17:53Z,2021-02-14T21:17:53Z,OWNER,Same issue as #227.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807817197,Hitting `_csv.Error: field larger than field limit (131072)`,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811746,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778811746,MDEyOklzc3VlQ29tbWVudDc3ODgxMTc0Ng==,9599,simonw,2021-02-14T17:39:30Z,2021-02-14T21:16:54Z,OWNER,"I'm going to detach this from the #131 column types idea.

The three things I need to handle here are:

- The CSV file doesn't have a header row at all, so I need to specify what the column names should be
- The CSV file DOES have a header row but I want to ignore it and use alternative column names
- The CSV doesn't have a header row at all and I want to automatically use `unknown1,unknown2...` so I can start exploring it as quickly as possible.

Here's a potential design that covers the first two:

`--replace-header=""foo,bar,baz""` - ignore whatever is in the first row and pretend it was this instead
`--add-header=""foo,bar,baz""` - add a first row with these details, to use as the header

It doesn't cover the ""give me unknown column names"" case though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778843086,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778843086,MDEyOklzc3VlQ29tbWVudDc3ODg0MzA4Ng==,9599,simonw,2021-02-14T21:15:43Z,2021-02-14T21:15:43Z,OWNER,"I'm not convinced the `.has_header()` rules are useful for the kind of CSV files I work with: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L383

```python
    def has_header(self, sample):
        # Creates a dictionary of types of data in each column. If any
        # column is of a single type (say, integers), *except* for the first
        # row, then the first row is presumed to be labels. If the type
        # can't be determined, it is assumed to be a string in which case
        # the length of the string is the determining factor: if all of the
        # rows except for the first are the same length, it's a header.
        # Finally, a 'vote' is taken at the end for each column, adding or
        # subtracting from the likelihood of the first row being a header.
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778842982,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778842982,MDEyOklzc3VlQ29tbWVudDc3ODg0Mjk4Mg==,9599,simonw,2021-02-14T21:15:11Z,2021-02-14T21:15:11Z,OWNER,"Implementation tip: I have code that reads the first row and uses it as headers here: https://github.com/simonw/sqlite-utils/blob/8f042ae1fd323995d966a94e8e6df85cc843b938/sqlite_utils/cli.py#L689-L691

So If I want to use `unknown1,unknown2...` I can do that by reading the first row, counting the number of columns, generating headers based on that range and then continuing to build that generator (maybe with `itertools.chain()` to replay the record we already read).

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778841704,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778841704,MDEyOklzc3VlQ29tbWVudDc3ODg0MTcwNA==,9599,simonw,2021-02-14T21:05:20Z,2021-02-14T21:05:20Z,OWNER,This has also been reported in #229.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,Error reading csv files with large column data,
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778841547,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778841547,MDEyOklzc3VlQ29tbWVudDc3ODg0MTU0Nw==,9599,simonw,2021-02-14T21:04:13Z,2021-02-14T21:04:13Z,OWNER,I added a test and fixed this in #234 - thanks for the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,fix for problem in Table.insert_all on search for columns per chunk of rows,
https://github.com/simonw/sqlite-utils/issues/234#issuecomment-778841278,https://api.github.com/repos/simonw/sqlite-utils/issues/234,778841278,MDEyOklzc3VlQ29tbWVudDc3ODg0MTI3OA==,9599,simonw,2021-02-14T21:02:11Z,2021-02-14T21:02:11Z,OWNER,"I managed to replicate this in a test:
```python
def test_insert_all_with_extra_columns_in_later_chunks(fresh_db):
    chunk = [
        {""record"": ""Record 1""},
        {""record"": ""Record 2""},
        {""record"": ""Record 3""},
        {""record"": ""Record 4"", ""extra"": 1},
    ]
    fresh_db[""t""].insert_all(chunk, batch_size=2, alter=True)
    assert list(fresh_db[""t""].rows) == [
        {""record"": ""Record 1"", ""extra"": None},
        {""record"": ""Record 2"", ""extra"": None},
        {""record"": ""Record 3"", ""extra"": None},
        {""record"": ""Record 4"", ""extra"": 1},
    ]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808046597,.insert_all() fails if subsequent chunks contain additional columns,
https://github.com/simonw/sqlite-utils/pull/225#issuecomment-778834504,https://api.github.com/repos/simonw/sqlite-utils/issues/225,778834504,MDEyOklzc3VlQ29tbWVudDc3ODgzNDUwNA==,9599,simonw,2021-02-14T20:09:30Z,2021-02-14T20:09:30Z,OWNER,Thanks for this. I'm going to try and get the test suite to run in Windows on GitHub Actions.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797159961,fix for problem in Table.insert_all on search for columns per chunk of rows,
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778829456,https://api.github.com/repos/simonw/sqlite-utils/issues/231,778829456,MDEyOklzc3VlQ29tbWVudDc3ODgyOTQ1Ng==,9599,simonw,2021-02-14T19:37:52Z,2021-02-14T19:37:52Z,OWNER,"I'm going to add `limit` and `offset` to the following methods:

- `rows_where()`
- `search_sql()`
- `search()`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808028757,"limit=X, offset=Y parameters for more Python methods",
https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778828758,https://api.github.com/repos/simonw/sqlite-utils/issues/231,778828758,MDEyOklzc3VlQ29tbWVudDc3ODgyODc1OA==,9599,simonw,2021-02-14T19:33:14Z,2021-02-14T19:33:14Z,OWNER,The `limit=` parameter is currently only available on the `.search()` method - it would make sense to add this to other methods as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808028757,"limit=X, offset=Y parameters for more Python methods",
https://github.com/simonw/sqlite-utils/pull/224#issuecomment-778828495,https://api.github.com/repos/simonw/sqlite-utils/issues/224,778828495,MDEyOklzc3VlQ29tbWVudDc3ODgyODQ5NQ==,9599,simonw,2021-02-14T19:31:06Z,2021-02-14T19:31:06Z,OWNER,I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792297010,Add fts offset docs.,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778827570,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778827570,MDEyOklzc3VlQ29tbWVudDc3ODgyNzU3MA==,9599,simonw,2021-02-14T19:24:20Z,2021-02-14T19:24:20Z,OWNER,Here's the implementation in Python: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L204-L225,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778824361,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778824361,MDEyOklzc3VlQ29tbWVudDc3ODgyNDM2MQ==,9599,simonw,2021-02-14T18:59:22Z,2021-02-14T18:59:22Z,OWNER,"I think I've got it. I can use `io.BufferedReader()` to get an object I can run `.peek(2048)` on, then wrap THAT in `io.TextIOWrapper`:

```python
    encoding = encoding or ""utf-8""
    buffered = io.BufferedReader(json_file, buffer_size=4096)
    decoded = io.TextIOWrapper(buffered, encoding=encoding, line_buffering=True)
    if pk and len(pk) == 1:
        pk = pk[0]
    if csv or tsv:
        if sniff:
            # Read first 2048 bytes and use that to detect
            first_bytes = buffered.peek(2048)
            print('first_bytes', first_bytes)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778821403,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778821403,MDEyOklzc3VlQ29tbWVudDc3ODgyMTQwMw==,9599,simonw,2021-02-14T18:38:16Z,2021-02-14T18:38:16Z,OWNER,"There are two code paths here that matter:

- For a regular file, can read the first 2048 bytes, then `.seek(0)` before continuing. That's easy.
- `stdin` is harder. I need to read and buffer the first 2048 bytes, then pass an object to `csv.reader()` which will replay that chunk and then play the rest of stdin.

I'm a bit stuck on the second one. Ideally I could use something like `itertools.chain()` but I can't find an alternative for file-like objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778818639,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778818639,MDEyOklzc3VlQ29tbWVudDc3ODgxODYzOQ==,9599,simonw,2021-02-14T18:22:38Z,2021-02-14T18:22:38Z,OWNER,Maybe I shouldn't be using `StreamReader` at all - https://www.python.org/dev/peps/pep-0400/ suggests that it should be deprecated in favour of `io.TextIOWrapper`. I'm using `StreamReader` due to this line: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L667-L668,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778817494,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778817494,MDEyOklzc3VlQ29tbWVudDc3ODgxNzQ5NA==,9599,simonw,2021-02-14T18:16:06Z,2021-02-14T18:16:06Z,OWNER,"Types involved:
```
(Pdb) type(json_file.raw)
<class '_io.FileIO'>
(Pdb) type(json_file)
<class 'encodings.utf_8.StreamReader'>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778816333,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778816333,MDEyOklzc3VlQ29tbWVudDc3ODgxNjMzMw==,9599,simonw,2021-02-14T18:08:44Z,2021-02-14T18:08:44Z,OWNER,"No, you can't `.seek(0)` on stdin:
```
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py"", line 678, in insert_upsert_implementation
    json_file.raw.seek(0)
OSError: [Errno 29] Illegal seek
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778815740,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778815740,MDEyOklzc3VlQ29tbWVudDc3ODgxNTc0MA==,9599,simonw,2021-02-14T18:05:03Z,2021-02-14T18:05:03Z,OWNER,"The challenge here is how to read the first 2048 bytes and then reset the incoming file.

The Python docs example looks like this:

```python
with open('example.csv', newline='') as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(1024))
    csvfile.seek(0)
    reader = csv.reader(csvfile, dialect)
```
Here's the relevant code in `sqlite-utils`: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L671-L679

The challenge is going to be having the `--sniff` option work with the progress bar. Here's how `file_progress()` works: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/utils.py#L106-L113

If `file.raw` is `stdin` can I do the equivalent of `csvfile.seek(0)` on it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778812684,https://api.github.com/repos/simonw/sqlite-utils/issues/230,778812684,MDEyOklzc3VlQ29tbWVudDc3ODgxMjY4NA==,9599,simonw,2021-02-14T17:45:16Z,2021-02-14T17:45:16Z,OWNER,"Running this could take any CSV (or TSV) file and automatically detect the delimiter. If no header row is detected it could add `unknown1,unknown2` headers:

    sqlite-utils insert db.db data file.csv --sniff

(Using `--sniff` would imply `--csv`)

This could be called `--sniffer` instead but I like `--sniff` better.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",808008305,--sniff option for sniffing delimiters,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778812050,MDEyOklzc3VlQ29tbWVudDc3ODgxMjA1MA==,9599,simonw,2021-02-14T17:41:30Z,2021-02-14T17:41:30Z,OWNER,"I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811934,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778811934,MDEyOklzc3VlQ29tbWVudDc3ODgxMTkzNA==,9599,simonw,2021-02-14T17:40:48Z,2021-02-14T17:40:48Z,OWNER,"Another pattern that might be useful is to generate a header that is just ""unknown1,unknown2,unknown3"" for each of the columns in the rest of the file. This makes it easy to e.g. facet-explore within Datasette to figure out the correct names, then use `sqlite-utils transform --rename` to rename the columns.

I needed to do that for the https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778511347,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778511347,MDEyOklzc3VlQ29tbWVudDc3ODUxMTM0Nw==,9599,simonw,2021-02-12T23:27:50Z,2021-02-12T23:27:50Z,OWNER,"For the moment, a workaround can be to `cat` an additional row onto the start of the file.

    echo ""name,url,description"" | cat - missing_headings.csv | sqlite-utils insert blah.db table - --csv","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778510528,https://api.github.com/repos/simonw/sqlite-utils/issues/131,778510528,MDEyOklzc3VlQ29tbWVudDc3ODUxMDUyOA==,9599,simonw,2021-02-12T23:25:06Z,2021-02-12T23:25:06Z,OWNER,"If `-c` isn't available, maybe `-t` or `--type` would work for specifying column types:
```
sqlite-utils insert db.db images images.tsv \
  --tsv \
  --type id int \
  --type score float
```
or
```
sqlite-utils insert db.db images images.tsv \
  --tsv \
  -t id int \
  -t score float
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778508887,https://api.github.com/repos/simonw/sqlite-utils/issues/131,778508887,MDEyOklzc3VlQ29tbWVudDc3ODUwODg4Nw==,9599,simonw,2021-02-12T23:20:11Z,2021-02-12T23:20:11Z,OWNER,"Annoyingly `-c` is currently a shortcut for `--csv` - so I'd have to do a major version bump to use that.

https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L601-L603

Particularly annoying because I attempted to remove the `-c` shortcut in https://github.com/simonw/sqlite-utils/commit/2c00567aac6d9c79087cfff0d054f64922b1473d#diff-76294b3d4afeb27e74e738daa01c26dd4dc9ccb6f4477451483a2ece1095902eL48 but forgot to remove it from the input options (I removed it from the output options).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",675753042,sqlite-utils insert: options for column types,
https://github.com/simonw/datasette/issues/1220#issuecomment-778467759,https://api.github.com/repos/simonw/datasette/issues/1220,778467759,MDEyOklzc3VlQ29tbWVudDc3ODQ2Nzc1OQ==,30607,aborruso,2021-02-12T21:35:17Z,2021-02-12T21:35:17Z,NONE,Thank you,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806743116,Installing datasette via docker: Path 'fixtures.db' does not exist,
https://github.com/simonw/datasette/issues/1220#issuecomment-778439617,https://api.github.com/repos/simonw/datasette/issues/1220,778439617,MDEyOklzc3VlQ29tbWVudDc3ODQzOTYxNw==,7476523,bobwhitelock,2021-02-12T20:33:27Z,2021-02-12T20:33:27Z,CONTRIBUTOR,"That Docker command will mount your current directory inside the Docker container at `/mnt` - so you shouldn't need to change anything locally, just run

```
docker run -p 8001:8001 -v `pwd`:/mnt \
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
```

and it will use the `fixtures.db` file within your current directory","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806743116,Installing datasette via docker: Path 'fixtures.db' does not exist,
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770069864,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60,770069864,MDEyOklzc3VlQ29tbWVudDc3MDA2OTg2NA==,22578954,daniel-butler,2021-01-29T21:52:05Z,2021-02-12T18:29:43Z,CONTRIBUTOR,"For the purposes below I am assuming the organization I would get all the repositories and their related commits from is called `gh-organization`. The github's owner id of gh-orgnization is `123456789`.

```bash
github-to-sqlite  repos github.db gh-organization
```

I'm on a windows computer running git bash to be able to use the `|` command. This works for me
```bash
sqlite3 github.db ""SELECT full_name FROM repos WHERE owner = '123456789';"" | tr '\n\r' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }
```

On a pure linux system I think this would work because the new line character is normally `\n`
```bash
sqlite3 github.db ""SELECT full_name FROM repos WHERE owner = '123456789';"" | tr '\n' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }`
```

As expected I ran into rate limit issues #51 
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797097140,Use Data from SQLite in other commands,
https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778349672,https://api.github.com/repos/simonw/sqlite-utils/issues/228,778349672,MDEyOklzc3VlQ29tbWVudDc3ODM0OTY3Mg==,9599,simonw,2021-02-12T18:00:43Z,2021-02-12T18:00:43Z,OWNER,"I could combine this with #131 to allow types to be specified in addition to column names.

Probably need an option that means ""ignore the existing heading row and use this one instead"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807437089,--no-headers option for CSV and TSV,
https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778349142,https://api.github.com/repos/simonw/sqlite-utils/issues/227,778349142,MDEyOklzc3VlQ29tbWVudDc3ODM0OTE0Mg==,9599,simonw,2021-02-12T17:59:35Z,2021-02-12T17:59:35Z,OWNER,It looks like I can at least bump this size limit up to the maximum allowed by Python - I'll take a look at that. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",807174161,Error reading csv files with large column data,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778246347,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,778246347,MDEyOklzc3VlQ29tbWVudDc3ODI0NjM0Nw==,41546558,RhetTbull,2021-02-12T15:00:43Z,2021-02-12T15:00:43Z,CONTRIBUTOR,"Yes, Big Sur Photos database doesn't have `ZGENERICASSET` table.   PR #31 will fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778014990,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,778014990,MDEyOklzc3VlQ29tbWVudDc3ODAxNDk5MA==,675335,leafgarland,2021-02-12T06:54:14Z,2021-02-12T06:54:14Z,NONE,"Ahh, that might be because macOS Big Sur has changed the structure of the photos db. Might need to wait for a later release, there is a PR which adds support for Big Sur. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/simonw/datasette/issues/1220#issuecomment-778008752,https://api.github.com/repos/simonw/datasette/issues/1220,778008752,MDEyOklzc3VlQ29tbWVudDc3ODAwODc1Mg==,30607,aborruso,2021-02-12T06:37:34Z,2021-02-12T06:37:34Z,NONE,"I have used my path, I'm running it from the folder in wich I have the db.

Do I must an absolute path?

Do I must create exactly that folder?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806743116,Installing datasette via docker: Path 'fixtures.db' does not exist,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778002092,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,778002092,MDEyOklzc3VlQ29tbWVudDc3ODAwMjA5Mg==,11855322,robmarkcole,2021-02-12T06:19:32Z,2021-02-12T06:19:32Z,NONE,"hi @leafgarland that results in a new error:
```
(venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos apple-photos photos.db
Traceback (most recent call last):
  File ""/Users/robin/datasette/venv/bin/dogsheep-photos"", line 8, in <module>
    sys.exit(cli())
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py"", line 206, in apple_photos
    db.conn.execute(
sqlite3.OperationalError: no such table: attached.ZGENERICASSET
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-777951854,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33,777951854,MDEyOklzc3VlQ29tbWVudDc3Nzk1MTg1NA==,675335,leafgarland,2021-02-12T03:54:39Z,2021-02-12T03:54:39Z,NONE,"I think that is a typo in the docs, you can use

    > dogsheep-photos apple-photos photos.db","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803338729,photo-to-sqlite: command not found,
https://github.com/simonw/datasette/pull/1223#issuecomment-777949755,https://api.github.com/repos/simonw/datasette/issues/1223,777949755,MDEyOklzc3VlQ29tbWVudDc3Nzk0OTc1NQ==,22429695,codecov[bot],2021-02-12T03:45:31Z,2021-02-12T03:45:31Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=h1) Report
> Merging [#1223](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=desc) (d1cd1f2) into [main](https://codecov.io/gh/simonw/datasette/commit/9603d893b9b72653895318c9104d754229fdb146?el=desc) (9603d89) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1223/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1223   +/-   ##
=======================================
  Coverage   91.42%   91.42%           
=======================================
  Files          32       32           
  Lines        3955     3955           
=======================================
  Hits         3616     3616           
  Misses        339      339           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=footer). Last update [9603d89...d1cd1f2](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806918878,Add compile option to Dockerfile to fix failing test (fixes #696),
https://github.com/simonw/datasette/issues/1220#issuecomment-777927946,https://api.github.com/repos/simonw/datasette/issues/1220,777927946,MDEyOklzc3VlQ29tbWVudDc3NzkyNzk0Ng==,7476523,bobwhitelock,2021-02-12T02:29:54Z,2021-02-12T02:29:54Z,CONTRIBUTOR,"According to https://github.com/simonw/datasette/blob/master/docs/installation.rst#using-docker it should be

```
docker run -p 8001:8001 -v `pwd`:/mnt \
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
```

This uses `/mnt/fixtures.db` whereas you're using `fixtures.db` - did you try using this path instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806743116,Installing datasette via docker: Path 'fixtures.db' does not exist,
https://github.com/simonw/datasette/issues/1221#issuecomment-777901052,https://api.github.com/repos/simonw/datasette/issues/1221,777901052,MDEyOklzc3VlQ29tbWVudDc3NzkwMTA1Mg==,9599,simonw,2021-02-12T01:09:54Z,2021-02-12T01:09:54Z,OWNER,"I also tested this manually. I generated certificate files like so:

    cd /tmp
    python -m trustme

This created `/tmp/server.pem`, `/tmp/client.pem` and `/tmp/server.key`

Then I started Datasette like this:

    datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem

And exercise it using `curl` like so:

    /tmp % curl --cacert /tmp/client.pem 'https://localhost:8001/_memory.json'
    {""database"": ""_memory"", ""path"": ""/_memory"", ""size"": 0, ""tables"": [], ""hidden_count"": 0, ""views"": [], ""queries"": [],
    ""private"": false, ""allow_execute_sql"": true, ""query_ms"": 0.8843200000114848}

Note that without the `--cacert` option I get an error:

```
/tmp % curl 'https://localhost:8001/_memory.json' 
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806849424,Support SSL/TLS directly,
https://github.com/simonw/datasette/issues/1221#issuecomment-777887190,https://api.github.com/repos/simonw/datasette/issues/1221,777887190,MDEyOklzc3VlQ29tbWVudDc3Nzg4NzE5MA==,9599,simonw,2021-02-12T00:29:18Z,2021-02-12T00:29:18Z,OWNER,I can use this recipe to start a `datasette` server in a sub-process during the pytest run and exercise it with real HTTP requests: https://til.simonwillison.net/pytest/subprocess-server,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806849424,Support SSL/TLS directly,
https://github.com/simonw/datasette/issues/1221#issuecomment-777883452,https://api.github.com/repos/simonw/datasette/issues/1221,777883452,MDEyOklzc3VlQ29tbWVudDc3Nzg4MzQ1Mg==,9599,simonw,2021-02-12T00:19:30Z,2021-02-12T00:19:40Z,OWNER,"Uvicorn supports these options: https://www.uvicorn.org/#command-line-options
```
  --ssl-keyfile TEXT              SSL key file
  --ssl-certfile TEXT             SSL certificate file
  --ssl-keyfile-password TEXT     SSL keyfile password
  --ssl-version INTEGER           SSL version to use (see stdlib ssl module's)
                                  [default: 2]

  --ssl-cert-reqs INTEGER         Whether client certificate is required (see
                                  stdlib ssl module's)  [default: 0]

  --ssl-ca-certs TEXT             CA certificates file
  --ssl-ciphers TEXT              Ciphers to use (see stdlib ssl module's)
                                  [default: TLSv1]
```
For the moment I'm going to support just `--ssl-keyfile` and `--ssl-certfile` as arguments to `datasette serve`. I'll add other options if people ask for them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",806849424,Support SSL/TLS directly,
https://github.com/dogsheep/evernote-to-sqlite/pull/10#issuecomment-777839351,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10,777839351,MDEyOklzc3VlQ29tbWVudDc3NzgzOTM1MQ==,9599,simonw,2021-02-11T22:37:55Z,2021-02-11T22:37:55Z,MEMBER,"I've merged these changes by hand now, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770712149,BugFix for encoding and not update info.,
https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7,777827396,MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng==,9599,simonw,2021-02-11T22:13:14Z,2021-02-11T22:13:14Z,MEMBER,My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743297582,evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace',
https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9,777821383,MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw==,9599,simonw,2021-02-11T22:01:28Z,2021-02-11T22:01:28Z,MEMBER,"Aha! I think I've figured out what's going on here.

The CData blocks containing the notes look like this:

`<![CDATA[<!DOCTYPE en-note SYSTEM ""http://xml.evernote.com/pub/enml2.dtd""><en-note><div>This note includes two images.</div><div><br /></div>...`

The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:

```
<!--=========== External character mnemonic entities ===================-->

<!ENTITY % HTMLlat1 PUBLIC
   ""-//W3C//ENTITIES Latin 1 for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent"">
%HTMLlat1;

<!ENTITY % HTMLsymbol PUBLIC
   ""-//W3C//ENTITIES Symbols for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent"">
%HTMLsymbol;

<!ENTITY % HTMLspecial PUBLIC
   ""-//W3C//ENTITIES Special for XHTML//EN""
   ""http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent"">
%HTMLspecial;
```
So I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:

```python
{'Aacute': 'Á',
 'aacute': 'á',
 'Aacute;': 'Á',
 'aacute;': 'á',
 'Abreve;': 'Ă',
 'abreve;': 'ă',
 'ac;': '∾',
 'acd;': '∿',
# ...
}
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",748372469,ParseError: undefined entity &scaron;,
https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777798330,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11,777798330,MDEyOklzc3VlQ29tbWVudDc3Nzc5ODMzMA==,9599,simonw,2021-02-11T21:18:58Z,2021-02-11T21:18:58Z,MEMBER,Thanks for the fix!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792851444,XML parse error,
https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777690332,https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11,777690332,MDEyOklzc3VlQ29tbWVudDc3NzY5MDMzMg==,3613583,dskrad,2021-02-11T18:16:01Z,2021-02-11T18:16:01Z,NONE,"I solved this issue by modifying line 31 of utils.py

    content = ET.tostring(ET.fromstring(content_xml.strip())).decode(""utf-8"")","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792851444,XML parse error,
https://github.com/simonw/datasette/issues/1200#issuecomment-777178728,https://api.github.com/repos/simonw/datasette/issues/1200,777178728,MDEyOklzc3VlQ29tbWVudDc3NzE3ODcyOA==,9599,simonw,2021-02-11T03:13:59Z,2021-02-11T03:13:59Z,OWNER,"I came up with the need for this while playing with this tool: https://calands.datasettes.com/calands?sql=select%0D%0A++AsGeoJSON(geometry)%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+'%25mini%25'+and%0D%0A++Intersects(GeomFromGeoJSON(%3Afreedraw)%2C+geometry)+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+(%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+'CPAD_2020a_SuperUnits'%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON(%3Afreedraw)%0D%0A++)&freedraw={""type""%3A""MultiPolygon""%2C""coordinates""%3A[[[[-122.42202758789064%2C37.82280243352759]%2C[-122.39868164062501%2C37.823887203271454]%2C[-122.38220214843751%2C37.81846319511331]%2C[-122.35061645507814%2C37.77071473849611]%2C[-122.34924316406251%2C37.74465712069939]%2C[-122.37258911132814%2C37.703380457832374]%2C[-122.39044189453125%2C37.690340943717715]%2C[-122.41241455078126%2C37.680559803205135]%2C[-122.44262695312501%2C37.67295135774715]%2C[-122.47283935546876%2C37.67295135774715]%2C[-122.52502441406251%2C37.68382032669382]%2C[-122.53463745117189%2C37.6892542140253]%2C[-122.54699707031251%2C37.690340943717715]%2C[-122.55798339843751%2C37.72945260537781]%2C[-122.54287719726564%2C37.77831314799672]%2C[-122.49893188476564%2C37.81303878836991]%2C[-122.46185302734376%2C37.82822612280363]%2C[-122.42889404296876%2C37.82822612280363]%2C[-122.42202758789064%2C37.82280243352759]]]]} - before I fixed https://github.com/simonw/datasette-leaflet-geojson/issues/16 it was loading a LOT of maps, which felt bad. I wanted to be able to link people to that page with a hard limit on the number of rows displayed on that page.

It's mainly to guard against unexpected behaviour from limit-less queries though. It's not a very high priority feature!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792890765,?_size=10 option for the arbitrary query page would be useful,
https://github.com/simonw/datasette/issues/1200#issuecomment-777132761,https://api.github.com/repos/simonw/datasette/issues/1200,777132761,MDEyOklzc3VlQ29tbWVudDc3NzEzMjc2MQ==,7476523,bobwhitelock,2021-02-11T00:29:52Z,2021-02-11T00:29:52Z,CONTRIBUTOR,I'm probably missing something but what's the use case here - what would this offer over adding `limit 10` to the query?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792890765,?_size=10 option for the arbitrary query page would be useful,
https://github.com/simonw/datasette/issues/1219#issuecomment-775442039,https://api.github.com/repos/simonw/datasette/issues/1219,775442039,MDEyOklzc3VlQ29tbWVudDc3NTQ0MjAzOQ==,9599,simonw,2021-02-08T20:39:52Z,2021-02-08T22:13:00Z,OWNER,"This comment helped me find a pattern for running Scalene against the Datasette test suite: https://github.com/emeryberger/scalene/issues/70#issuecomment-755245858

```
pip install scalene
```
Then I created a file called `run_tests.py` with the following contents:
```python
if __name__ == ""__main__"":
    import sys, pytest
    pytest.main(sys.argv)
```
Then I ran this:
```
scalene --profile-all run_tests.py -sv -x .
```
But... it quit with a segmentation fault!
```
(datasette) datasette % scalene --profile-all run_tests.py -sv -x .
======================================================================== test session starts ========================================================================
platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- python
cachedir: .pytest_cache
rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
plugins: asyncio-0.14.0, timeout-1.4.2
collecting ... Fatal Python error: Segmentation fault

Current thread 0x0000000110c1edc0 (most recent call first):
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 553 in detect_json1
  File ""/Users/simon/Dropbox/Development/datasette/datasette/filters.py"", line 168 in Filters
  File ""/Users/simon/Dropbox/Development/datasette/datasette/filters.py"", line 94 in <module>
  File ""<frozen importlib._bootstrap>"", line 219 in _call_with_frames_removed
  File ""<frozen importlib._bootstrap_external>"", line 783 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 27 in <module>
  File ""<frozen importlib._bootstrap>"", line 219 in _call_with_frames_removed
  File ""<frozen importlib._bootstrap_external>"", line 783 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 42 in <module>
  File ""<frozen importlib._bootstrap>"", line 219 in _call_with_frames_removed
  File ""<frozen importlib._bootstrap_external>"", line 783 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""/Users/simon/Dropbox/Development/datasette/tests/test_api.py"", line 1 in <module>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/assertion/rewrite.py"", line 170 in exec_module
  File ""<frozen importlib._bootstrap>"", line 671 in _load_unlocked
  File ""<frozen importlib._bootstrap>"", line 975 in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 991 in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1014 in _gcd_import
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py"", line 127 in import_module
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/pathlib.py"", line 520 in import_path
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 552 in _importtestmodule
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 484 in _getobj
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 288 in obj
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 500 in _inject_setup_module_fixture
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py"", line 487 in collect
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 324 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 294 in from_call
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 324 in pytest_make_collect_report
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py"", line 187 in _multicall
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 84 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 93 in _hookexec
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py"", line 286 in __call__
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py"", line 441 in collect_one_node
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 768 in genitems
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 771 in genitems
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 568 in _perform_collect
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 516 in perform_collect
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 306 in pytest_collection
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py"", line 187 in _multicall
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 84 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 93 in _hookexec
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py"", line 286 in __call__
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 295 in _main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 240 in wrap_session
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py"", line 289 in pytest_cmdline_main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py"", line 187 in _multicall
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 84 in <lambda>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py"", line 93 in _hookexec
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py"", line 286 in __call__
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/config/__init__.py"", line 157 in main
  File ""run_tests.py"", line 3 in <module>
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/scalene_profiler.py"", line 1525 in main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py"", line 7 in main
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py"", line 14 in <module>
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py"", line 87 in _run_code
  File ""/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py"", line 194 in _run_module_as_main
Scalene error: received signal SIGSEGV
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803929694,Try profiling Datasette using scalene,
https://github.com/simonw/datasette/issues/1219#issuecomment-775497449,https://api.github.com/repos/simonw/datasette/issues/1219,775497449,MDEyOklzc3VlQ29tbWVudDc3NTQ5NzQ0OQ==,9599,simonw,2021-02-08T22:11:34Z,2021-02-08T22:11:34Z,OWNER,"https://github.com/emeryberger/scalene/issues/110 reports a ""received signal SIGSEGV"" error that was fixed by upgrading to the latest Scalene version, but I'm running that already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",803929694,Try profiling Datasette using scalene,
https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774730656,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9,774730656,MDEyOklzc3VlQ29tbWVudDc3NDczMDY1Ng==,635179,merwok,2021-02-07T18:45:04Z,2021-02-07T18:45:04Z,NONE,"That URL uses TLS 1.3, but maybe only if the client supports it.
It could be your Python version or your SSL library that’s not recent enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",801780625,SSL Error,
https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774726123,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9,774726123,MDEyOklzc3VlQ29tbWVudDc3NDcyNjEyMw==,12669260,jfeiwell,2021-02-07T18:21:08Z,2021-02-07T18:21:08Z,NONE,@simonw any ideas here?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",801780625,SSL Error,
https://github.com/simonw/datasette/issues/1217#issuecomment-774528913,https://api.github.com/repos/simonw/datasette/issues/1217,774528913,MDEyOklzc3VlQ29tbWVudDc3NDUyODkxMw==,639730,virtadpt,2021-02-06T19:23:41Z,2021-02-06T19:23:41Z,NONE,I've had a lot of success running it as an OpenFaaS lambda.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",802513359,Possible to deploy as a python app (for Rstudio connect server)?,
https://github.com/simonw/datasette/issues/1217#issuecomment-774385092,https://api.github.com/repos/simonw/datasette/issues/1217,774385092,MDEyOklzc3VlQ29tbWVudDc3NDM4NTA5Mg==,6165713,plpxsk,2021-02-06T02:49:11Z,2021-02-06T02:49:11Z,NONE,"A good reference seems to be the note to run `datasette` as a module in https://github.com/simonw/datasette/pull/556
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",802513359,Possible to deploy as a python app (for Rstudio connect server)?,
https://github.com/simonw/sqlite-utils/issues/223#issuecomment-774373829,https://api.github.com/repos/simonw/sqlite-utils/issues/223,774373829,MDEyOklzc3VlQ29tbWVudDc3NDM3MzgyOQ==,9599,simonw,2021-02-06T01:39:47Z,2021-02-06T01:39:47Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-insert-csv-tsv-delimiter,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788527932,--delimiter option for CSV import,
https://github.com/simonw/datasette/issues/1208#issuecomment-774286962,https://api.github.com/repos/simonw/datasette/issues/1208,774286962,MDEyOklzc3VlQ29tbWVudDc3NDI4Njk2Mg==,4488943,kbaikov,2021-02-05T21:02:39Z,2021-02-05T21:02:39Z,CONTRIBUTOR,@simonw could you please take a look at the PR 1211 that fixes this issue?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",794554881,A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-774217792,https://api.github.com/repos/simonw/sqlite-utils/issues/203,774217792,MDEyOklzc3VlQ29tbWVudDc3NDIxNzc5Mg==,1049910,drkane,2021-02-05T18:44:13Z,2021-02-05T18:44:13Z,NONE,"Thanks for looking at this - home schooling kids has prevented me from replying. 

I'd struggled with how to adapt the API for the foreign keys too - I definitely tried the String/Tuple approach. I hadn't considered the breaking changes that would introduce though. I can take a look at this and try and make the change - see which of your options works best.

I've got a workaround for the use-case I was looking at this for, so it wouldn't be a problem for me if it was put on the back burner until a hypothetical v4.0 anyway.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/datasette/issues/1210#issuecomment-773977128,https://api.github.com/repos/simonw/datasette/issues/1210,773977128,MDEyOklzc3VlQ29tbWVudDc3Mzk3NzEyOA==,525780,heyarne,2021-02-05T11:30:34Z,2021-02-05T11:30:34Z,NONE,"Thanks for your quick reply! Having changed my `metadata.yml`, queries AND database I can't really reproduce it anymore, sorry. But at least I'm happy to say that it works now! :) Thanks again for the super nifty tool, very appreciated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796234313,Immutable Database w/ Canned Queries,
https://github.com/simonw/datasette/issues/1216#issuecomment-772796111,https://api.github.com/repos/simonw/datasette/issues/1216,772796111,MDEyOklzc3VlQ29tbWVudDc3Mjc5NjExMQ==,9599,simonw,2021-02-03T20:20:48Z,2021-02-03T20:20:48Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/datasette/app.py#L620-L632,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",800669347,"/-/databases should reflect connection order, not alphabetical order",
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-772408273,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56,772408273,MDEyOklzc3VlQ29tbWVudDc3MjQwODI3Mw==,42315895,gsajko,2021-02-03T10:36:36Z,2021-02-03T10:36:36Z,NONE,"I figured it out.
Those tweets are in database, because somebody quote tweeted them, or retweeted them.
And if you grab quoted tweet or reweeted tweet from other tweet json, It doesn't grab all of the details.

So if someone quote tweeted a quote tweet, the second quote tweet won't have `quoted_status`. 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796736607,Not all quoted statuses get fetched?,
https://github.com/simonw/datasette/issues/1212#issuecomment-772007663,https://api.github.com/repos/simonw/datasette/issues/1212,772007663,MDEyOklzc3VlQ29tbWVudDc3MjAwNzY2Mw==,4488943,kbaikov,2021-02-02T21:36:56Z,2021-02-02T21:36:56Z,CONTRIBUTOR,"How do you get 4-5 minutes?
I run my tests in WSL 2, so may be i need to try a real linux VM.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797651831,Tests are very slow. ,
https://github.com/simonw/datasette/issues/1214#issuecomment-772001787,https://api.github.com/repos/simonw/datasette/issues/1214,772001787,MDEyOklzc3VlQ29tbWVudDc3MjAwMTc4Nw==,9599,simonw,2021-02-02T21:28:53Z,2021-02-02T21:28:53Z,OWNER,"Fix is now live on https://latest.datasette.io/fixtures/searchable?_search=terry - clearing ""terry"" and re-submitting the form now works as expected.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799693777,Re-submitting filter form duplicates _x querystring arguments,
https://github.com/simonw/datasette/issues/1214#issuecomment-771992628,https://api.github.com/repos/simonw/datasette/issues/1214,771992628,MDEyOklzc3VlQ29tbWVudDc3MTk5MjYyOA==,9599,simonw,2021-02-02T21:15:18Z,2021-02-02T21:15:18Z,OWNER,"The cause of this bug is form fields which begin with `_` but ARE displayed as form inputs on the page - hence should not be duplicated in an `<input type=""hidden"">` element.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799693777,Re-submitting filter form duplicates _x querystring arguments,
https://github.com/simonw/datasette/issues/1214#issuecomment-771992025,https://api.github.com/repos/simonw/datasette/issues/1214,771992025,MDEyOklzc3VlQ29tbWVudDc3MTk5MjAyNQ==,9599,simonw,2021-02-02T21:14:16Z,2021-02-02T21:14:16Z,OWNER,"As a result, navigating to https://github-to-sqlite.dogsheep.net/github/labels?_search=help and clearing out the `_search` field then submitting the form does NOT clear the search term.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799693777,Re-submitting filter form duplicates _x querystring arguments,
https://github.com/simonw/datasette/issues/1212#issuecomment-771976561,https://api.github.com/repos/simonw/datasette/issues/1212,771976561,MDEyOklzc3VlQ29tbWVudDc3MTk3NjU2MQ==,9599,simonw,2021-02-02T20:53:27Z,2021-02-02T20:53:27Z,OWNER,"It would be great if we could get `python-xdist` to run too - I tried it in the past and gave up when I ran into those race conditions, but I've not done any further digging to see if there's a way to fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797651831,Tests are very slow. ,
https://github.com/simonw/datasette/issues/1212#issuecomment-771975941,https://api.github.com/repos/simonw/datasette/issues/1212,771975941,MDEyOklzc3VlQ29tbWVudDc3MTk3NTk0MQ==,9599,simonw,2021-02-02T20:52:36Z,2021-02-02T20:52:36Z,OWNER,"37 minutes, wow! They're a little slow for me (4-5 minutes perhaps) but not nearly that bad.

Thanks for running that profile. I think you're right: figuring out how to use more session scopes would definitely help.

The `:memory:` idea is interesting too. The new `memory_name=` feature added in #1151 (released in Datasette 0.54) could help a lot here, since it allows Datasette instances to share the same in-memory database across multiple HTTP requests and connections.

Note that `memory_name=` also persists within test runs themselves, independently of any `scope=` options on the fixtures. That might actually help us here!

I'd be delighted if you explored this issue further, especially the option of using `memory_name=` for the fixtures databases used by the tests.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797651831,Tests are very slow. ,
https://github.com/simonw/datasette/issues/1213#issuecomment-771968675,https://api.github.com/repos/simonw/datasette/issues/1213,771968675,MDEyOklzc3VlQ29tbWVudDc3MTk2ODY3NQ==,9599,simonw,2021-02-02T20:41:55Z,2021-02-02T20:41:55Z,OWNER,"So maybe I could a special response header which ASGI middleware can pick up that means ""Don't attempt to gzip this, just stream it through"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799663959,gzip support for HTML (and JSON) responses,
https://github.com/simonw/datasette/issues/1213#issuecomment-771968177,https://api.github.com/repos/simonw/datasette/issues/1213,771968177,MDEyOklzc3VlQ29tbWVudDc3MTk2ODE3Nw==,9599,simonw,2021-02-02T20:41:13Z,2021-02-02T20:41:13Z,OWNER,"Starlette accumulates the full response body in a `body` variable and then does this:
```python
        elif message_type == ""http.response.body"":
            # Remaining body in streaming GZip response.
            body = message.get(""body"", b"""")
            more_body = message.get(""more_body"", False)

            self.gzip_file.write(body)
            if not more_body:
                self.gzip_file.close()

            message[""body""] = self.gzip_buffer.getvalue()
            self.gzip_buffer.seek(0)
            self.gzip_buffer.truncate()

            await self.send(message)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799663959,gzip support for HTML (and JSON) responses,
https://github.com/simonw/datasette/issues/1213#issuecomment-771965281,https://api.github.com/repos/simonw/datasette/issues/1213,771965281,MDEyOklzc3VlQ29tbWVudDc3MTk2NTI4MQ==,9599,simonw,2021-02-02T20:37:08Z,2021-02-02T20:39:24Z,OWNER,Starlette's gzip middleware implementation is here: https://github.com/encode/starlette/blob/0.14.2/starlette/middleware/gzip.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",799663959,gzip support for HTML (and JSON) responses,
https://github.com/simonw/datasette/pull/1211#issuecomment-771127458,https://api.github.com/repos/simonw/datasette/issues/1211,771127458,MDEyOklzc3VlQ29tbWVudDc3MTEyNzQ1OA==,4488943,kbaikov,2021-02-01T20:13:39Z,2021-02-01T20:13:39Z,CONTRIBUTOR,Ping @simonw ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797649915,Use context manager instead of plain open,
https://github.com/simonw/datasette/pull/1159#issuecomment-770865698,https://api.github.com/repos/simonw/datasette/issues/1159,770865698,MDEyOklzc3VlQ29tbWVudDc3MDg2NTY5OA==,552629,lovasoa,2021-02-01T13:42:29Z,2021-02-01T13:42:29Z,NONE,@simonw : Could you have a look at this ? I think this really improves readability.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/datasette/pull/1211#issuecomment-770343684,https://api.github.com/repos/simonw/datasette/issues/1211,770343684,MDEyOklzc3VlQ29tbWVudDc3MDM0MzY4NA==,22429695,codecov[bot],2021-01-31T08:03:40Z,2021-01-31T08:03:40Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=h1) Report
> Merging [#1211](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=desc) (e33ccaa) into [main](https://codecov.io/gh/simonw/datasette/commit/dde3c500c73ace33529672f7d862b76753d309cc?el=desc) (dde3c50) will **decrease** coverage by `0.00%`.
> The diff coverage is `92.85%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1211/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main    #1211      +/-   ##
==========================================
- Coverage   91.54%   91.53%   -0.01%     
==========================================
  Files          32       32              
  Lines        3948     3959      +11     
==========================================
+ Hits         3614     3624      +10     
- Misses        334      335       +1     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `77.29% <66.66%> (-0.31%)` | :arrow_down: |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.62% <100.00%> (+<0.01%)` | :arrow_up: |
| [datasette/publish/cloudrun.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY2xvdWRydW4ucHk=) | `96.96% <100.00%> (+0.09%)` | :arrow_up: |
| [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.73% <100.00%> (+0.60%)` | :arrow_up: |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.13% <100.00%> (+0.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=footer). Last update [dde3c50...e33ccaa](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797649915,Use context manager instead of plain open,
https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-770150526,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51,770150526,MDEyOklzc3VlQ29tbWVudDc3MDE1MDUyNg==,22578954,daniel-butler,2021-01-30T03:44:19Z,2021-01-30T03:47:24Z,CONTRIBUTOR,I don't have much experience with github's rate limiting. In my day job we use the [tenacity library](https://github.com/jd/tenacity) to handle http errors we get.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703246031,github-to-sqlite should handle rate limits better,
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770112248,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60,770112248,MDEyOklzc3VlQ29tbWVudDc3MDExMjI0OA==,22578954,daniel-butler,2021-01-30T00:01:03Z,2021-01-30T01:14:42Z,CONTRIBUTOR,"Yes that would be cool! I wouldn't mind helping. Is this the meat of it? https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/utils.py#L512

It looks like the cli option is added with this decorator : https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/cli.py#L14

I looked a bit at utils.py in the GitHub repository. I was surprised at the amount of manual mapping of the API response you had to do to get this to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797097140,Use Data from SQLite in other commands,
https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60,770071568,MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA==,9599,simonw,2021-01-29T21:56:15Z,2021-01-29T21:56:15Z,MEMBER,"I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow:

https://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82

`twitter-to-sqlite` actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach

It lets you do things like:

```
$ twitter-to-sqlite users-lookup my.db --sql=""select follower_id from following"" --ids
```
Maybe I should add something similar to `github-to-sqlite`? Feels like it could be really useful.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",797097140,Use Data from SQLite in other commands,
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769973212,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56,769973212,MDEyOklzc3VlQ29tbWVudDc2OTk3MzIxMg==,42315895,gsajko,2021-01-29T18:29:02Z,2021-01-29T18:31:55Z,NONE,"I think it was with `twitter-to-sqlite home-timeline home.db -a  auth.json --since`
and Im using only this command to grab tweets 

from cron tab
`2,7,12,17,22,27,32,37,42,47,52,57 * * * * run-one /home/gsajko/miniconda3/bin/twitter-to-sqlite home-timeline /home/gsajko/work/custom_twitter_feed/home.db -a /home/gsajko/work/custom_twitter_feed/auth/auth.json --since`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796736607,Not all quoted statuses get fetched?,
https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56,769957751,MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ==,9599,simonw,2021-01-29T17:59:40Z,2021-01-29T17:59:40Z,MEMBER,"This is interesting - how did you create that initial table? Was this using the `twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip` command, or something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796736607,Not all quoted statuses get fetched?,
https://github.com/simonw/datasette/issues/1207#issuecomment-769534187,https://api.github.com/repos/simonw/datasette/issues/1207,769534187,MDEyOklzc3VlQ29tbWVudDc2OTUzNDE4Nw==,9599,simonw,2021-01-29T02:37:19Z,2021-01-29T02:37:19Z,OWNER,https://docs.datasette.io/en/latest/testing_plugins.html#using-pdb-for-errors-thrown-inside-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793881756,"Document the Datasette(..., pdb=True) testing pattern",
https://github.com/simonw/datasette/issues/1209#issuecomment-769455370,https://api.github.com/repos/simonw/datasette/issues/1209,769455370,MDEyOklzc3VlQ29tbWVudDc2OTQ1NTM3MA==,9599,simonw,2021-01-28T23:00:21Z,2021-01-28T23:00:21Z,OWNER,"Good catch on the workaround here. The root problem is that `datasette-template-sql` looks for the first available databsae if you don't provide it with a `database=` argument, and in Datasette 0.54 the first available database changed to being the new `_internal` database.

Is this a bug? I think it is - because the documented behaviour on https://docs.datasette.io/en/stable/internals.html#get-database-name is this:

> `name` - string, optional
>
> The name to be used for this database - this will be used in the URL path, e.g. `/dbname`. If not specified Datasette will pick one based on the filename or memory name.

Since the new behaviour differs from what was in the documentation I'm going to treat this as a bug and fix it.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",795367402,v0.54 500 error from sql query in custom template; code worked in v0.53; found a workaround,
https://github.com/simonw/datasette/issues/1205#issuecomment-769453074,https://api.github.com/repos/simonw/datasette/issues/1205,769453074,MDEyOklzc3VlQ29tbWVudDc2OTQ1MzA3NA==,9599,simonw,2021-01-28T22:54:49Z,2021-01-28T22:55:02Z,OWNER,"
I also checked that the following works:

    echo '{""foo"": ""bar""}' | sqlite-utils insert _memory.db demo -
    datasette _memory.db --memory

Sure enough, it results in the following Datasette homepage - thanks to #509

<img width=""274"" alt=""Datasette___memory___memory_2"" src=""https://user-images.githubusercontent.com/9599/106208790-c8564980-6178-11eb-8b8b-053a9f1d0193.png"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793027837,Rename /:memory: to /_memory,
https://github.com/simonw/datasette/issues/1205#issuecomment-769452084,https://api.github.com/repos/simonw/datasette/issues/1205,769452084,MDEyOklzc3VlQ29tbWVudDc2OTQ1MjA4NA==,9599,simonw,2021-01-28T22:52:23Z,2021-01-28T22:52:23Z,OWNER,Here are the redirect tests: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/tests/test_api.py#L635-L648,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793027837,Rename /:memory: to /_memory,
https://github.com/simonw/datasette/issues/1205#issuecomment-769442165,https://api.github.com/repos/simonw/datasette/issues/1205,769442165,MDEyOklzc3VlQ29tbWVudDc2OTQ0MjE2NQ==,9599,simonw,2021-01-28T22:30:16Z,2021-01-28T22:30:27Z,OWNER,"I'm going to do this, with redirects from `/:memory:*`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793027837,Rename /:memory: to /_memory,
https://github.com/simonw/datasette/issues/1210#issuecomment-769274591,https://api.github.com/repos/simonw/datasette/issues/1210,769274591,MDEyOklzc3VlQ29tbWVudDc2OTI3NDU5MQ==,9599,simonw,2021-01-28T18:10:02Z,2021-01-28T18:10:02Z,OWNER,That definitely sounds like a bug! Can you provide a copy of your `metadata.JSON` and the command-line you are using to launch Datasette?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",796234313,Immutable Database w/ Canned Queries,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-767888743,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,767888743,MDEyOklzc3VlQ29tbWVudDc2Nzg4ODc0Mw==,19328961,henry501,2021-01-26T23:07:41Z,2021-01-26T23:07:41Z,NONE,"My import got much further with the applied fixes than 0.21.3, but not 100%. I do appear to have all of the tweets imported at least. 
Not sure when I'll have a chance to look further to try to fix or see what didn't make it into the import.

Here's my output:

```
direct-messages-group: not yet implemented
branch-links: not yet implemented
periscope-expired-broadcasts: not yet implemented
direct-messages: not yet implemented
mute: not yet implemented
periscope-comments-made-by-user: not yet implemented
periscope-ban-information: not yet implemented
periscope-profile-description: not yet implemented
screen-name-change: not yet implemented
manifest: not yet implemented
fleet: not yet implemented
user-link-clicks: not yet implemented
periscope-broadcast-metadata: not yet implemented
contact: not yet implemented
fleet-mute: not yet implemented
device-token: not yet implemented
protected-history: not yet implemented
direct-message-mute: not yet implemented
Traceback (most recent call last):
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/bin/twitter-to-sqlite"", line 33, in <module>
    sys.exit(load_entry_point('twitter-to-sqlite==0.21.3', 'console_scripts', 'twitter-to-sqlite')())
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/cli.py"", line 772, in import_
    archive.import_from_file(db, filepath.name, open(filepath, ""rb"").read())
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 233, in import_from_file
    to_insert = transformer(data)
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 21, in callback
    return {filename: [fn(item) for item in data]}
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 21, in <listcomp>
    return {filename: [fn(item) for item in data]}
  File ""/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py"", line 88, in ageinfo
    return item[""ageMeta""][""ageInfo""]
KeyError: 'ageInfo'

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/simonw/datasette/issues/1208#issuecomment-767823684,https://api.github.com/repos/simonw/datasette/issues/1208,767823684,MDEyOklzc3VlQ29tbWVudDc2NzgyMzY4NA==,9599,simonw,2021-01-26T20:58:51Z,2021-01-26T20:58:51Z,OWNER,"This is a good catch - I've been lazy about this, but you're right that it's an issue that needs cleaning up.  Would be very happy to apply a PR, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",794554881,A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper,
https://github.com/simonw/datasette/issues/1151#issuecomment-767762551,https://api.github.com/repos/simonw/datasette/issues/1151,767762551,MDEyOklzc3VlQ29tbWVudDc2Nzc2MjU1MQ==,9599,simonw,2021-01-26T19:07:44Z,2021-01-26T19:07:44Z,OWNER,Mentioned in https://simonwillison.net/2021/Jan/25/datasette/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/991#issuecomment-767761155,https://api.github.com/repos/simonw/datasette/issues/991,767761155,MDEyOklzc3VlQ29tbWVudDc2Nzc2MTE1NQ==,9599,simonw,2021-01-26T19:05:21Z,2021-01-26T19:06:36Z,OWNER,"Idea: implement this using the existing table view, with a custom template called `table-internal-bb0ec0-tables.html` - that's the custom template listed in the HTML comments at the bottom of https://latest.datasette.io/_internal/tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage,
https://github.com/simonw/datasette/issues/1201#issuecomment-766991680,https://api.github.com/repos/simonw/datasette/issues/1201,766991680,MDEyOklzc3VlQ29tbWVudDc2Njk5MTY4MA==,9599,simonw,2021-01-25T17:42:21Z,2021-01-25T17:42:21Z,OWNER,https://docs.datasette.io/en/stable/changelog.html#v0-54,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/pull/1206#issuecomment-766589070,https://api.github.com/repos/simonw/datasette/issues/1206,766589070,MDEyOklzc3VlQ29tbWVudDc2NjU4OTA3MA==,22429695,codecov[bot],2021-01-25T06:50:30Z,2021-01-25T17:31:11Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=h1) Report
> Merging [#1206](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=desc) (06480e1) into [main](https://codecov.io/gh/simonw/datasette/commit/a5ede3cdd455e2bb1a1fb2f4e1b5a9855caf5179?el=desc) (a5ede3c) will **not change** coverage.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1206/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1206   +/-   ##
=======================================
  Coverage   91.53%   91.53%           
=======================================
  Files          32       32           
  Lines        3947     3947           
=======================================
  Hits         3613     3613           
  Misses        334      334           
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/version.py](https://codecov.io/gh/simonw/datasette/pull/1206/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZlcnNpb24ucHk=) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=footer). Last update [a5ede3c...571476d](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793086333,Release 0.54,
https://github.com/simonw/datasette/pull/1206#issuecomment-766588371,https://api.github.com/repos/simonw/datasette/issues/1206,766588371,MDEyOklzc3VlQ29tbWVudDc2NjU4ODM3MQ==,9599,simonw,2021-01-25T06:49:06Z,2021-01-25T06:49:06Z,OWNER,"Last thing to do: write up the annotated version of these release notes, assign it a URL on my blog and link to it from the release notes here so I can publish them simultaneously.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793086333,Release 0.54,
https://github.com/simonw/datasette/pull/1206#issuecomment-766588020,https://api.github.com/repos/simonw/datasette/issues/1206,766588020,MDEyOklzc3VlQ29tbWVudDc2NjU4ODAyMA==,9599,simonw,2021-01-25T06:48:20Z,2021-01-25T06:48:20Z,OWNER,"Issues to reference in the commit message: #509, #1091, #1150, #1151, #1166, #1167, #1178, #1181, #1182, #1184, #1185, #1186, #1187, #1194, #1198","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793086333,Release 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766586151,https://api.github.com/repos/simonw/datasette/issues/1201,766586151,MDEyOklzc3VlQ29tbWVudDc2NjU4NjE1MQ==,9599,simonw,2021-01-25T06:44:43Z,2021-01-25T06:44:43Z,OWNER,"OK, release notes are ready to merge from that branch. I'll ship the release in the morning, to give me time to write the accompanying annotated release notes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766545604,https://api.github.com/repos/simonw/datasette/issues/1201,766545604,MDEyOklzc3VlQ29tbWVudDc2NjU0NTYwNA==,9599,simonw,2021-01-25T05:14:31Z,2021-01-25T05:14:31Z,OWNER,"The two big ticket items are `<script type=""module"">` support and the new `_internal` mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766545442,https://api.github.com/repos/simonw/datasette/issues/1201,766545442,MDEyOklzc3VlQ29tbWVudDc2NjU0NTQ0Mg==,9599,simonw,2021-01-25T05:13:59Z,2021-01-25T05:13:59Z,OWNER,"The big stuff:

- Database(memory_name=) for shared in-memory databases, closes #1151 
- The `_internal` database -  #1150
-  script type=module support, closes #1186 , #1187
- Improved design for the `.add_database()` method 8919f99c2f7f245aca7f94bd53d5ac9d04aa42b5 - which means databases with the same stem can now be opened, #509
- Adopted Prettier #1166 

Smaller:

- force_https_urls on for publish cloudrun, refs #1178 
- Fixed bug in example nginx config, refs #1091 
-  Shrunk ecosystem docs in favour of datasette.io, closes #1182 
-  request.full_path property, closes #1184 
-  Better PRAGMA error message, closes #1185 
-  publish heroku now uses python-3.8.7 
-  Plugin testing documentation on using pytest-httpx Closes #1198
-  Contributing docs for Black and Prettier, closes #1167 
-  All ?_ parameters now copied to hidden form fields, closes #1194 
- Fixed bug loading database called 'test-database (1).sqlite' - Closes #1181.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/1201#issuecomment-766543387,https://api.github.com/repos/simonw/datasette/issues/1201,766543387,MDEyOklzc3VlQ29tbWVudDc2NjU0MzM4Nw==,9599,simonw,2021-01-25T05:07:40Z,2021-01-25T05:13:29Z,OWNER,Changes: https://github.com/simonw/datasette/compare/0.53...a5ede3cdd455e2bb1a1fb2f4e1b5a9855caf5179,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792904595,Release notes for Datasette 0.54,
https://github.com/simonw/datasette/issues/983#issuecomment-766536076,https://api.github.com/repos/simonw/datasette/issues/983,766536076,MDEyOklzc3VlQ29tbWVudDc2NjUzNjA3Ng==,9599,simonw,2021-01-25T04:43:53Z,2021-01-25T04:43:53Z,OWNER,"... actually not going to include this in 0.54, I need to write a couple of plugins myself using it before I even make it available in preview.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1195#issuecomment-766535046,https://api.github.com/repos/simonw/datasette/issues/1195,766535046,MDEyOklzc3VlQ29tbWVudDc2NjUzNTA0Ng==,9599,simonw,2021-01-25T04:40:08Z,2021-01-25T04:40:08Z,OWNER,"Also: should the view name really be the same for both of these pages?

- https://latest.datasette.io/fixtures?sql=select+*+from+facetable
- https://latest.datasette.io/fixtures/neighborhood_search

Where one of them is a canned query and the other is an arbitrary query?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/datasette/issues/1195#issuecomment-766534748,https://api.github.com/repos/simonw/datasette/issues/1195,766534748,MDEyOklzc3VlQ29tbWVudDc2NjUzNDc0OA==,9599,simonw,2021-01-25T04:38:56Z,2021-01-25T04:38:56Z,OWNER,"Here's a diff showing how far I got before I abandoned this particular effort:

```diff
diff --git a/datasette/views/base.py b/datasette/views/base.py
index a21b929..04e4aa9 100644
--- a/datasette/views/base.py
+++ b/datasette/views/base.py
@@ -120,7 +120,7 @@ class BaseView:
         handler = getattr(self, request.method.lower(), None)
         return await handler(request, *args, **kwargs)
 
-    async def render(self, templates, request, context=None):
+    async def render(self, templates, request, context=None, view_name=None):
         context = context or {}
         template = self.ds.jinja_env.select_template(templates)
         template_context = {
@@ -135,7 +135,7 @@ class BaseView:
         }
         return Response.html(
             await self.ds.render_template(
-                template, template_context, request=request, view_name=self.name
+                template, template_context, request=request, view_name=view_name
             )
         )
 
@@ -155,7 +155,7 @@ class BaseView:
 
 
 class DataView(BaseView):
-    name = """"
+    view_name = ""no-view-name""
     re_named_parameter = re.compile("":([a-zA-Z0-9_]+)"")
 
     async def options(self, request, *args, **kwargs):
@@ -414,6 +414,10 @@ class DataView(BaseView):
             args[""table""] = urllib.parse.unquote_plus(args[""table""])
         return _format, args
 
+    async def get_view_name(self, request, database, hash, **kwargs):
+        # Defaults to self.view_name, but can be over-ridden by subclasses
+        return self.view_name
+
     async def view_get(self, request, database, hash, correct_hash_provided, **kwargs):
         _format, kwargs = await self.get_format(request, database, kwargs)
 
@@ -424,6 +428,8 @@ class DataView(BaseView):
             # HTML views default to expanding all foreign key labels
             kwargs[""default_labels""] = True
 
+        view_name = await self.get_view_name(request, database, hash, **kwargs)
+
         extra_template_data = {}
         start = time.perf_counter()
         status_code = 200
@@ -489,7 +495,7 @@ class DataView(BaseView):
                 database=database,
                 table=data.get(""table""),
                 request=request,
-                view_name=self.name,
+                view_name=view_name,
                 # These will be deprecated in Datasette 1.0:
                 args=request.args,
                 data=data,
@@ -533,7 +539,7 @@ class DataView(BaseView):
                     database=database,
                     table=data.get(""table""),
                     request=request,
-                    view_name=self.name,
+                    view_name=view_name,
                 )
                 it_can_render = await await_me_maybe(it_can_render)
                 if it_can_render:
@@ -565,7 +571,7 @@ class DataView(BaseView):
             }
             if ""metadata"" not in context:
                 context[""metadata""] = self.ds.metadata
-            r = await self.render(templates, request=request, context=context)
+            r = await self.render(templates, request=request, context=context, view_name=view_name)
             r.status = status_code
 
         ttl = request.args.get(""_ttl"", None)
diff --git a/datasette/views/database.py b/datasette/views/database.py
index f6fd579..e425213 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -23,7 +23,11 @@ from .base import DatasetteError, DataView
 
 
 class DatabaseView(DataView):
-    name = ""database""
+    async def get_view_name(self, request, db_name, table_and_format):
+        if request.args.get(""sql""):
+            return ""query""
+        else:
+            return ""database""
 
     async def data(self, request, database, hash, default_labels=False, _size=None):
         await self.check_permissions(
@@ -145,7 +149,7 @@ class DatabaseView(DataView):
 
 
 class DatabaseDownload(DataView):
-    name = ""database_download""
+    view_name = ""database_download""
 
     async def view_get(self, request, database, hash, correct_hash_present, **kwargs):
         await self.check_permissions(
diff --git a/datasette/views/index.py b/datasette/views/index.py
index b6b8cbe..d750e3d 100644
--- a/datasette/views/index.py
+++ b/datasette/views/index.py
@@ -16,7 +16,7 @@ COUNT_DB_SIZE_LIMIT = 100 * 1024 * 1024
 
 
 class IndexView(BaseView):
-    name = ""index""
+    view_name = ""index""
 
     async def get(self, request, as_format):
         await self.check_permission(request, ""view-instance"")
diff --git a/datasette/views/special.py b/datasette/views/special.py
index 9750dd0..dbd1e00 100644
--- a/datasette/views/special.py
+++ b/datasette/views/special.py
@@ -6,7 +6,7 @@ import secrets
 
 
 class JsonDataView(BaseView):
-    name = ""json_data""
+    view_name = ""json_data""
 
     def __init__(self, datasette, filename, data_callback, needs_request=False):
         self.ds = datasette
@@ -42,7 +42,7 @@ class JsonDataView(BaseView):
 
 
 class PatternPortfolioView(BaseView):
-    name = ""patterns""
+    view_name = ""patterns""
 
     async def get(self, request):
         await self.check_permission(request, ""view-instance"")
@@ -50,7 +50,7 @@ class PatternPortfolioView(BaseView):
 
 
 class AuthTokenView(BaseView):
-    name = ""auth_token""
+    view_name = ""auth_token""
 
     async def get(self, request):
         token = request.args.get(""token"") or """"
@@ -68,7 +68,7 @@ class AuthTokenView(BaseView):
 
 
 class LogoutView(BaseView):
-    name = ""logout""
+    view_name = ""logout""
 
     async def get(self, request):
         if not request.actor:
@@ -87,7 +87,7 @@ class LogoutView(BaseView):
 
 
 class PermissionsDebugView(BaseView):
-    name = ""permissions_debug""
+    view_name = ""permissions_debug""
 
     async def get(self, request):
         await self.check_permission(request, ""view-instance"")
@@ -102,7 +102,7 @@ class PermissionsDebugView(BaseView):
 
 
 class AllowDebugView(BaseView):
-    name = ""allow_debug""
+    view_name = ""allow_debug""
 
     async def get(self, request):
         errors = []
@@ -136,7 +136,7 @@ class AllowDebugView(BaseView):
 
 
 class MessagesDebugView(BaseView):
-    name = ""messages_debug""
+    view_name = ""messages_debug""
 
     async def get(self, request):
         await self.check_permission(request, ""view-instance"")
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 0a3504b..45d298a 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -257,7 +257,16 @@ class RowTableShared(DataView):
 
 
 class TableView(RowTableShared):
-    name = ""table""
+    view_name = ""table""
+
+    async def get_view_name(self, request, db_name, table_and_format):
+        canned_query = await self.ds.get_canned_query(
+            db_name, table_and_format, request.actor
+        )
+        if canned_query:
+            return ""query""
+        else:
+            return ""table""
 
     async def post(self, request, db_name, table_and_format):
         # Handle POST to a canned query
@@ -923,7 +932,7 @@ async def _sql_params_pks(db, table, pk_values):
 
 
 class RowView(RowTableShared):
-    name = ""row""
+    view_name = ""row""
 
     async def data(self, request, database, hash, table, pk_path, default_labels=False):
         await self.check_permissions(
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index 715c7c1..7ce2b1b 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -252,7 +252,7 @@ def test_plugin_config_file(app_client):
             },
         ),
         (
-            ""/fixtures/"",
+            ""/fixtures"",
             {
                 ""template"": ""database.html"",
                 ""database"": ""fixtures"",
@@ -285,6 +285,38 @@ def test_plugin_config_file(app_client):
                 ],
             },
         ),
+        (
+            ""/fixtures?sql=select+1+as+one"",
+            {
+                ""template"": ""query.html"",
+                ""database"": ""fixtures"",
+                ""table"": None,
+                ""config"": {""depth"": ""database""},
+                ""view_name"": ""query"",
+                ""request_path"": ""/fixtures"",
+                ""added"": 15,
+                ""columns"": [
+                    ""one"",
+                ],
+            },
+        ),
+        (
+            ""/fixtures/neighborhood_search"",
+            {
+                ""template"": ""query.html"",
+                ""database"": ""fixtures"",
+                ""table"": None,
+                ""config"": {""depth"": ""database""},
+                ""view_name"": ""query"",
+                ""request_path"": ""/fixtures/neighborhood_search"",
+                ""added"": 15,
+                ""columns"": [
+                    ""neighborhood"",
+                    ""name"",
+                    ""state"",
+                ],
+            },
+        ),
     ],
 )
 def test_hook_extra_body_script(app_client, path, expected_extra_body_script):
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/datasette/issues/1195#issuecomment-766534634,https://api.github.com/repos/simonw/datasette/issues/1195,766534634,MDEyOklzc3VlQ29tbWVudDc2NjUzNDYzNA==,9599,simonw,2021-01-25T04:38:30Z,2021-01-25T04:38:30Z,OWNER,"This has proved surprisingly difficult to implement, due to the weird way the QueryView is actually called. The class itself isn't used like other view classes - instead, the `.data()` methods of both `DatabaseView` and `TableView` dispatch out to `QueryView.data()` when they need to:

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L259-L270

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L290-L294

https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/database.py#L39-L44

It turns out this is a bad pattern because it makes changes like this one WAY harder than they should be.

I think I should clean this up as part of #878.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/datasette/pull/1204#issuecomment-766525337,https://api.github.com/repos/simonw/datasette/issues/1204,766525337,MDEyOklzc3VlQ29tbWVudDc2NjUyNTMzNw==,9599,simonw,2021-01-25T04:04:28Z,2021-01-25T04:04:28Z,OWNER,"Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",793002853,WIP: Plugin includes,
https://github.com/simonw/datasette/issues/987#issuecomment-766524141,https://api.github.com/repos/simonw/datasette/issues/987,766524141,MDEyOklzc3VlQ29tbWVudDc2NjUyNDE0MQ==,9599,simonw,2021-01-25T03:59:55Z,2021-01-25T03:59:55Z,OWNER,"This is joined with #1191 now, which I've bumped from 0.54.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/1191#issuecomment-766524016,https://api.github.com/repos/simonw/datasette/issues/1191,766524016,MDEyOklzc3VlQ29tbWVudDc2NjUyNDAxNg==,9599,simonw,2021-01-25T03:59:17Z,2021-01-25T03:59:17Z,OWNER,More work can happen in the PR: #1204,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1191#issuecomment-766523866,https://api.github.com/repos/simonw/datasette/issues/1191,766523866,MDEyOklzc3VlQ29tbWVudDc2NjUyMzg2Ng==,9599,simonw,2021-01-25T03:58:34Z,2021-01-25T03:58:34Z,OWNER,"I've got a good prototype working now, but I'm dropping this from the Datasette 0.54 milestone because it requires a bunch of additional work to make sure it is really well tested and documented.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1194#issuecomment-766491911,https://api.github.com/repos/simonw/datasette/issues/1194,766491911,MDEyOklzc3VlQ29tbWVudDc2NjQ5MTkxMQ==,9599,simonw,2021-01-25T02:02:15Z,2021-01-25T02:02:15Z,OWNER,I'm going to copy across anything starting with an underscore.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788447787,?_size= argument is not persisted by hidden form fields in the table filters,
https://github.com/simonw/datasette/issues/1167#issuecomment-766491566,https://api.github.com/repos/simonw/datasette/issues/1167,766491566,MDEyOklzc3VlQ29tbWVudDc2NjQ5MTU2Ng==,9599,simonw,2021-01-25T02:01:19Z,2021-01-25T02:01:19Z,OWNER,New documentation section here (I documented Black as well): https://docs.datasette.io/en/latest/contributing.html#code-formatting,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777145954,Add Prettier to contributing documentation,
https://github.com/simonw/datasette/issues/1167#issuecomment-766487520,https://api.github.com/repos/simonw/datasette/issues/1167,766487520,MDEyOklzc3VlQ29tbWVudDc2NjQ4NzUyMA==,9599,simonw,2021-01-25T01:44:43Z,2021-01-25T01:44:43Z,OWNER,"Thanks @benpickles, I just merged that in. I'll use it in the documentation.

    # To check code is conformant
    npm run prettier -- --check
    # To fix it if it isn't
    npm run fix
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777145954,Add Prettier to contributing documentation,
https://github.com/simonw/datasette/issues/983#issuecomment-766484915,https://api.github.com/repos/simonw/datasette/issues/983,766484915,MDEyOklzc3VlQ29tbWVudDc2NjQ4NDkxNQ==,9599,simonw,2021-01-25T01:33:29Z,2021-01-25T01:33:29Z,OWNER,I'm going to ship a version of this in Datasette 0.54 with a warning that the interface should be considered unstable (see #1202) so that we can start trying this out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1179#issuecomment-766484435,https://api.github.com/repos/simonw/datasette/issues/1179,766484435,MDEyOklzc3VlQ29tbWVudDc2NjQ4NDQzNQ==,9599,simonw,2021-01-25T01:31:36Z,2021-01-25T01:31:36Z,OWNER,Relevant existing tests: https://github.com/simonw/datasette/blob/461670a0b87efa953141b449a9a261919864ceb3/tests/test_utils.py#L365-L398,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-766484257,https://api.github.com/repos/simonw/datasette/issues/1179,766484257,MDEyOklzc3VlQ29tbWVudDc2NjQ4NDI1Nw==,9599,simonw,2021-01-25T01:30:57Z,2021-01-25T01:30:57Z,OWNER,"The challenge here is figuring out what the original path, without the `.format`, actually was - while taking into account that Datasette has a special case for tables that themselves end in a `.something`.

The `path_with_format()` function nearly does what we need here:

https://github.com/simonw/datasette/blob/b6a7b58fa01af0cd5a5e94bd17d686d283a46819/datasette/utils/__init__.py#L710-L729

It can be called with `replace_format=""csv""` to REMOVE the `.csv` format and replace it with something else.

Problem is, we want to use it to get rid of the format entirely.

We could update `path_with_format()` to accept `format=''` to mean ""remove the format entirely"", but it's a bit messy. It may be better to reconsider the design of `path_with_format()` and related utility functions entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1191#issuecomment-766466030,https://api.github.com/repos/simonw/datasette/issues/1191,766466030,MDEyOklzc3VlQ29tbWVudDc2NjQ2NjAzMA==,9599,simonw,2021-01-25T00:11:04Z,2021-01-25T00:11:04Z,OWNER,"I can combine this with #987 - each of these areas of the page can be wrapped in a `<div>` with a class that matches the name of the plugin hook, that way JavaScript plugins can append their content in the same place as Python plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1154#issuecomment-766465719,https://api.github.com/repos/simonw/datasette/issues/1154,766465719,MDEyOklzc3VlQ29tbWVudDc2NjQ2NTcxOQ==,9599,simonw,2021-01-25T00:09:22Z,2021-01-25T00:09:22Z,OWNER,"https://docs.datasette.io/en/latest/internals.html#the-internal-database

<img width=""738"" alt=""Internals_for_plugins_—_Datasette_documentation"" src=""https://user-images.githubusercontent.com/9599/105648087-76eb4900-5e5e-11eb-9c73-c00ba37b7b10.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771208009,Documentation for new _internal database and tables,
https://github.com/simonw/datasette/issues/1202#issuecomment-766464136,https://api.github.com/repos/simonw/datasette/issues/1202,766464136,MDEyOklzc3VlQ29tbWVudDc2NjQ2NDEzNg==,9599,simonw,2021-01-25T00:01:02Z,2021-01-25T00:01:02Z,OWNER,I'm going to use the existing `.. warning::` pattern for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792931244,Documentation convention for marking unstable APIs.,
https://github.com/simonw/datasette/issues/1090#issuecomment-766463496,https://api.github.com/repos/simonw/datasette/issues/1090,766463496,MDEyOklzc3VlQ29tbWVudDc2NjQ2MzQ5Ng==,9599,simonw,2021-01-24T23:57:00Z,2021-01-24T23:57:00Z,OWNER,Related: I built [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) which turns any canned query field called `freedraw` or `something_freedraw` into an interactive map that you can draw on to create a GeoJSON MultiPolygon.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",741862364,Custom widgets for canned query forms,
https://github.com/simonw/datasette/issues/1202#issuecomment-766462475,https://api.github.com/repos/simonw/datasette/issues/1202,766462475,MDEyOklzc3VlQ29tbWVudDc2NjQ2MjQ3NQ==,9599,simonw,2021-01-24T23:49:28Z,2021-01-24T23:50:33Z,OWNER,"Can use an ""admonition"" similar to this:
```sphinx
.. warning::
    Restricting access to tables and views in this way will NOT prevent users from querying them using arbitrary SQL queries, `like this <https://latest.datasette.io/fixtures?sql=select+*+from+facetable>`__ for example.
```
As seen on https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-tables-and-views

Documentation: https://docutils.sourceforge.io/docs/ref/rst/directives.html#specific-admonitions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792931244,Documentation convention for marking unstable APIs.,
https://github.com/simonw/datasette/issues/1154#issuecomment-766462197,https://api.github.com/repos/simonw/datasette/issues/1154,766462197,MDEyOklzc3VlQ29tbWVudDc2NjQ2MjE5Nw==,9599,simonw,2021-01-24T23:47:06Z,2021-01-24T23:47:06Z,OWNER,"I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771208009,Documentation for new _internal database and tables,
https://github.com/simonw/datasette/issues/1179#issuecomment-766434629,https://api.github.com/repos/simonw/datasette/issues/1179,766434629,MDEyOklzc3VlQ29tbWVudDc2NjQzNDYyOQ==,9599,simonw,2021-01-24T21:23:47Z,2021-01-24T21:23:47Z,OWNER,I'm just going to do `path` and `full_path` (which includes the querystring)`. The `datasette.absolute_url()` method can be used by plugins that need the full URL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1190#issuecomment-766433153,https://api.github.com/repos/simonw/datasette/issues/1190,766433153,MDEyOklzc3VlQ29tbWVudDc2NjQzMzE1Mw==,9599,simonw,2021-01-24T21:13:25Z,2021-01-24T21:13:25Z,OWNER,"This ties in to a bunch of other ideas that are in flight at the moment.

If you're publishing databases by uploading them, how do you attach metadata? Ideally by baking it into the database file itself, using the mechanism from #1169.

How could this interact with the `datasette insert` concept from #1163? Could you pass a CSV file to the `upload` command and have that converted and uploaded for you, or would you create the database file locally using `datasette insert` and then upload it as a separate `datasette upload` step?

Lots to think about here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098146,`datasette publish upload` mechanism for uploading databases to an existing Datasette instance,
https://github.com/simonw/datasette/issues/1190#issuecomment-766430644,https://api.github.com/repos/simonw/datasette/issues/1190,766430644,MDEyOklzc3VlQ29tbWVudDc2NjQzMDY0NA==,9599,simonw,2021-01-24T20:57:03Z,2021-01-24T20:57:03Z,OWNER,"I really like this idea. It feels like an opportunity for a plugin that adds two things: an API endpoint to Datasette for accepting uploaded databases, and a `datasette publish upload` subcommand which can upload files to that endpoint (with some kind of authentication mechanism).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098146,`datasette publish upload` mechanism for uploading databases to an existing Datasette instance,
https://github.com/simonw/datasette/issues/1197#issuecomment-766430111,https://api.github.com/repos/simonw/datasette/issues/1197,766430111,MDEyOklzc3VlQ29tbWVudDc2NjQzMDExMQ==,9599,simonw,2021-01-24T20:53:40Z,2021-01-24T20:53:40Z,OWNER,"https://devcenter.heroku.com/articles/slug-compiler#slug-size says that the maximum allowed size is 500MB - my hunch is that the Datasette application itself weighs in at only a dozen or so MB but I haven't measured it. So I would imagine anything up to around 450MB should work OK on Heroku.

Cloud Run works for up to about 2GB in my experience.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",791381623,DB size limit for publishing with Heroku,
https://github.com/simonw/datasette/issues/1198#issuecomment-766428183,https://api.github.com/repos/simonw/datasette/issues/1198,766428183,MDEyOklzc3VlQ29tbWVudDc2NjQyODE4Mw==,9599,simonw,2021-01-24T20:40:37Z,2021-01-24T20:40:37Z,OWNER,https://docs.datasette.io/en/latest/testing_plugins.html#testing-outbound-http-calls-with-pytest-httpx,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792625812,Plugin testing documentation on using pytest-httpx,
https://github.com/simonw/datasette/issues/1199#issuecomment-766181628,https://api.github.com/repos/simonw/datasette/issues/1199,766181628,MDEyOklzc3VlQ29tbWVudDc2NjE4MTYyOA==,9599,simonw,2021-01-23T21:25:18Z,2021-01-23T21:25:18Z,OWNER,"Comment thread here: https://news.ycombinator.com/item?id=25881911 - cperciva says:

>  There's an even better reason for databases to not write to memory mapped pages: Pages get synched out to disk at the kernel's leisure. This can be ok for a cache but it's definitely not what you want for a database!

But... Datasette is often used in read-only mode, so that disadvantage often doesn't apply.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792652391,Experiment with PRAGMA mmap_size=N,
https://github.com/simonw/sqlite-utils/pull/224#issuecomment-765678057,https://api.github.com/repos/simonw/sqlite-utils/issues/224,765678057,MDEyOklzc3VlQ29tbWVudDc2NTY3ODA1Nw==,37962604,polyrand,2021-01-22T20:53:06Z,2021-01-23T20:13:27Z,NONE,"I'm using the FTS methods in sqlite-utils for this website: [drwn.io](https://drwn.io/). I wanted to get pagination to have some kind of infinite scrolling in the landing page, and I ended up using that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",792297010,Add fts offset docs.,
https://github.com/simonw/datasette/issues/1191#issuecomment-765757433,https://api.github.com/repos/simonw/datasette/issues/1191,765757433,MDEyOklzc3VlQ29tbWVudDc2NTc1NzQzMw==,9599,simonw,2021-01-22T23:43:43Z,2021-01-22T23:43:43Z,OWNER,"Another potential use for this: plugins that provide authentication (like `datasette-auth-passwords` and `datasette-auth-github`) could use it to add a chunk of HTML to the ""permission denied"" page that links to their mechanism of authenticating.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1196#issuecomment-765639968,https://api.github.com/repos/simonw/datasette/issues/1196,765639968,MDEyOklzc3VlQ29tbWVudDc2NTYzOTk2OA==,2826376,QAInsights,2021-01-22T19:37:15Z,2021-01-22T19:37:15Z,NONE,I tried deployment in WSL. It is working fine https://jmeter.vercel.app/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",791237799,Access Denied Error in Windows,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765525338,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765525338,MDEyOklzc3VlQ29tbWVudDc2NTUyNTMzOA==,25372415,cobiadigital,2021-01-22T16:22:44Z,2021-01-22T16:22:44Z,NONE,"rs1333049 associated with coronary artery disease
https://www.snpedia.com/index.php/Rs1333049
```

select rsid, genotype, case genotype
  when 'CC' then '1.9x increased risk for coronary artery disease'
  when 'CG' then '1.5x increased risk for CAD'
  when 'GG' then 'normal'
end as interpretation from genome where rsid = 'rs1333049'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765523517,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765523517,MDEyOklzc3VlQ29tbWVudDc2NTUyMzUxNw==,25372415,cobiadigital,2021-01-22T16:20:25Z,2021-01-22T16:20:25Z,NONE,"rs53576: the oxytocin receptor (OXTR) gene

```
select rsid, genotype, case genotype
  when 'AA' then 'Lack of empathy?'
  when 'AG' then 'Lack of empathy?'
  when 'GG' then 'Optimistic and empathetic; handle stress well'
end as interpretation from genome where rsid = 'rs53576'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765506901,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765506901,MDEyOklzc3VlQ29tbWVudDc2NTUwNjkwMQ==,25372415,cobiadigital,2021-01-22T15:58:41Z,2021-01-22T15:58:58Z,NONE,"Both rs10757274 and rs2383206 can both indicate higher risks of heart disease
https://www.snpedia.com/index.php/Rs2383206

```
select rsid, genotype, case genotype
  when 'AA' then 'Normal'
  when 'AG' then '~1.2x increased risk for heart disease'
  when 'GG' then '~1.3x increased risk for heart disease'
end as interpretation from genome where rsid = 'rs10757274'
```

```
select rsid, genotype, case genotype
  when 'AA' then 'Normal'
  when 'AG' then '1.4x increased risk for heart disease'
  when 'GG' then '1.7x increased risk for heart disease'
end as interpretation from genome where rsid = 'rs2383206'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765502845,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765502845,MDEyOklzc3VlQ29tbWVudDc2NTUwMjg0NQ==,25372415,cobiadigital,2021-01-22T15:53:19Z,2021-01-22T15:53:19Z,NONE,"rs7903146 Influences risk of Type-2 diabetes
https://www.snpedia.com/index.php/Rs7903146
```
select rsid, genotype, case genotype
  when 'CC' then 'Normal (lower) risk of Type 2 Diabetes and Gestational Diabetes.'
  when 'CT' then '1.4x increased risk for diabetes (and perhaps colon cancer).'
  when 'TT' then '2x increased risk for Type-2 diabetes'
end as interpretation from genome where rsid = 'rs7903146'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765498984,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765498984,MDEyOklzc3VlQ29tbWVudDc2NTQ5ODk4NA==,25372415,cobiadigital,2021-01-22T15:48:25Z,2021-01-22T15:49:33Z,NONE,"The ""Warrior Gene"" https://www.snpedia.com/index.php/Rs4680

```
select rsid, genotype, case genotype
  when 'AA' then '(worrier) advantage in memory and attention tasks'
  when 'AG' then 'Intermediate dopamine levels, other effects'
  when 'GG' then '(warrior) multiple associations, see details'
end as interpretation from genome where rsid = 'rs4680'
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765495861,https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1,765495861,MDEyOklzc3VlQ29tbWVudDc2NTQ5NTg2MQ==,25372415,cobiadigital,2021-01-22T15:44:00Z,2021-01-22T15:44:00Z,NONE,"Risk of autoimmune disorders: https://www.snpedia.com/index.php/Genotype
```
select rsid, genotype, case genotype
  when 'AA' then '2x risk of rheumatoid arthritis and other autoimmune diseases'
  when 'GG' then 'Normal risk for autoimmune disorders'
end as interpretation from genome where rsid = 'rs2476601'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",496415321,Figure out some interesting example SQL queries,
https://github.com/simonw/datasette/issues/1195#issuecomment-763108730,https://api.github.com/repos/simonw/datasette/issues/1195,763108730,MDEyOklzc3VlQ29tbWVudDc2MzEwODczMA==,9599,simonw,2021-01-19T20:22:37Z,2021-01-19T20:22:37Z,OWNER,I can use this test: https://github.com/simonw/datasette/blob/c38c42948cbfddd587729413fd6082ba352eaece/tests/test_plugins.py#L238-L294,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",789336592,"view_name = ""query"" for the query page",
https://github.com/simonw/sqlite-utils/issues/223#issuecomment-762540514,https://api.github.com/repos/simonw/sqlite-utils/issues/223,762540514,MDEyOklzc3VlQ29tbWVudDc2MjU0MDUxNA==,9599,simonw,2021-01-19T01:14:58Z,2021-01-19T01:15:54Z,OWNER,"Example from https://docs.python.org/3/library/csv.html#csv.reader
```pycon
>>> import csv
>>> with open('eggs.csv', newline='') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
```
I'm going to add `--quotechar` as well as `--delimiter`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788527932,--delimiter option for CSV import,
https://github.com/simonw/datasette/issues/1175#issuecomment-762488336,https://api.github.com/repos/simonw/datasette/issues/1175,762488336,MDEyOklzc3VlQ29tbWVudDc2MjQ4ODMzNg==,758858,hannseman,2021-01-18T21:59:28Z,2021-01-18T22:00:31Z,NONE,"I encountered your issue when trying to find a solution and came up with the following, maybe it can help.

```python
import logging.config
from typing import Tuple

import structlog
import uvicorn

from example.config import settings

shared_processors: Tuple[structlog.types.Processor, ...] = (
    structlog.contextvars.merge_contextvars,
    structlog.stdlib.add_logger_name,
    structlog.stdlib.add_log_level,
    structlog.processors.TimeStamper(fmt=""iso""),
)

logging_config = {
    ""version"": 1,
    ""disable_existing_loggers"": False,
    ""formatters"": {
        ""json"": {
            ""()"": structlog.stdlib.ProcessorFormatter,
            ""processor"": structlog.processors.JSONRenderer(),
            ""foreign_pre_chain"": shared_processors,
        },
        ""console"": {
            ""()"": structlog.stdlib.ProcessorFormatter,
            ""processor"": structlog.dev.ConsoleRenderer(),
            ""foreign_pre_chain"": shared_processors,
        },
        **uvicorn.config.LOGGING_CONFIG[""formatters""],
    },
    ""handlers"": {
        ""default"": {
            ""level"": ""DEBUG"",
            ""class"": ""logging.StreamHandler"",
            ""formatter"": ""json"" if not settings.debug else ""console"",
        },
        ""uvicorn.access"": {
            ""level"": ""INFO"",
            ""class"": ""logging.StreamHandler"",
            ""formatter"": ""access"",
        },
        ""uvicorn.default"": {
            ""level"": ""INFO"",
            ""class"": ""logging.StreamHandler"",
            ""formatter"": ""default"",
        },
    },
    ""loggers"": {
        """": {""handlers"": [""default""], ""level"": ""INFO""},
        ""uvicorn.error"": {
            ""handlers"": [""default"" if not settings.debug else ""uvicorn.default""],
            ""level"": ""INFO"",
            ""propagate"": False,
        },
        ""uvicorn.access"": {
            ""handlers"": [""default"" if not settings.debug else ""uvicorn.access""],
            ""level"": ""INFO"",
            ""propagate"": False,
        },
    },
}


def setup_logging() -> None:
    structlog.configure(
        processors=[
            structlog.stdlib.filter_by_level,
            *shared_processors,
            structlog.stdlib.PositionalArgumentsFormatter(),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
        ],
        context_class=dict,
        logger_factory=structlog.stdlib.LoggerFactory(),
        wrapper_class=structlog.stdlib.AsyncBoundLogger,
        cache_logger_on_first_use=True,
    )
    logging.config.dictConfig(logging_config)
```

And then it'll be run on the startup event:
```python
@app.on_event(""startup"")
async def startup_event() -> None:
    setup_logging()
```

It depends on a setting called `debug` which controls if it should output the regular uvicorn logging or json. ","{""total_count"": 15, ""+1"": 7, ""-1"": 0, ""laugh"": 1, ""hooray"": 1, ""confused"": 0, ""heart"": 5, ""rocket"": 1, ""eyes"": 0}",779156520,Use structlog for logging,
https://github.com/simonw/datasette/issues/1036#issuecomment-762391426,https://api.github.com/repos/simonw/datasette/issues/1036,762391426,MDEyOklzc3VlQ29tbWVudDc2MjM5MTQyNg==,4997607,philshem,2021-01-18T17:45:00Z,2021-01-18T17:45:00Z,NONE,"It might be possible with this library: https://docs.python.org/3/library/imghdr.html

quick test of the downloaded blob:

```
>>> import imghdr
>>> imghdr.what('material_culture-1-image.blob')
'jpeg'
```

The output plugin would be cool. I'll look into making my first datasette plugin.  I'm also imagining displaying the image in the browser -- but that would be a step 2.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1194#issuecomment-762390568,https://api.github.com/repos/simonw/datasette/issues/1194,762390568,MDEyOklzc3VlQ29tbWVudDc2MjM5MDU2OA==,9599,simonw,2021-01-18T17:43:03Z,2021-01-18T17:43:03Z,OWNER,Should I just blanket copy over any query string argument that starts with an underscore? Any reason _not_ to do that?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788447787,?_size= argument is not persisted by hidden form fields in the table filters,
https://github.com/simonw/datasette/issues/1194#issuecomment-762390401,https://api.github.com/repos/simonw/datasette/issues/1194,762390401,MDEyOklzc3VlQ29tbWVudDc2MjM5MDQwMQ==,9599,simonw,2021-01-18T17:42:38Z,2021-01-18T17:42:38Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/views/table.py#L815-L827

It looks like there are other arguments that may not be persisted too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",788447787,?_size= argument is not persisted by hidden form fields in the table filters,
https://github.com/simonw/datasette/issues/1036#issuecomment-762387875,https://api.github.com/repos/simonw/datasette/issues/1036,762387875,MDEyOklzc3VlQ29tbWVudDc2MjM4Nzg3NQ==,9599,simonw,2021-01-18T17:36:36Z,2021-01-18T17:36:36Z,OWNER,"As you can see, I'm pretty paranoid about serving content with `Content-Type` HTTP headers because I'm so worried about execution vulnerabilities. I'm much more comfortable exploring that kind of thing in plugins, since that way people can opt-in to riskier features.

You found `datasette-media` which is my most comprehensive exploration of that idea so far - but there's definitely lots of room for more plugins along those lines.

Maybe even an output plugin? `.jpg` as an export format which returns the `BLOB` column for a row as a JPEG image with the correct `content-type` header (but first verifies that the binary content does indeed look like a real JPEG) could be interesting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/simonw/datasette/issues/1036#issuecomment-762385981,https://api.github.com/repos/simonw/datasette/issues/1036,762385981,MDEyOklzc3VlQ29tbWVudDc2MjM4NTk4MQ==,4997607,philshem,2021-01-18T17:32:13Z,2021-01-18T17:34:50Z,NONE,"Hi Simon

Just finding this old issue regarding downloading blobs. Nice work!

<img width=""212"" alt=""image"" src=""https://user-images.githubusercontent.com/4997607/104946741-df4bad80-59ba-11eb-96e3-727c85cc4dc6.png"">

As a feature request, maybe it would be possible to assign a blob column as a certain data type (e.g. `.jpg`) and then each blob could be downloaded as that type of file (perhaps if the file types were constrained to normal blobs that people store in sqlite databases, this could avoid the execution stuff mentioned above).

I guess the column blob-type definition could fit into this dropdown selection:

<img width=""201"" alt=""image"" src=""https://user-images.githubusercontent.com/4997607/104947000-479a8f00-59bb-11eb-87d9-1644e5940894.png"">

Let me know if I should open a new issue with a feature request. (This could slowly go in the direction of displaying image blob-types in the browser.)

Thanks for the great tool!


---

edit: just reading the rest of the twitter thread: https://twitter.com/simonw/status/1318685933256855552

perhaps this is already possible in some form with the plugin datasette-media: https://github.com/simonw/datasette-media","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI,
https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11,761967094,MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA==,9599,simonw,2021-01-18T04:11:13Z,2021-01-18T04:11:13Z,MEMBER,"I just got a similar error:

```
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py"", line 79, in save_checkin
    checkins_table.m2m(""users"", user, m2m_table=""with"", pk=""id"")
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 2048, in m2m
    id = other_table.insert(record, pk=pk, replace=True).last_pk
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1781, in insert
    return self.insert_all(
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1899, in insert_all
    self.insert_chunk(
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 1709, in insert_chunk
    result = self.db.execute(query, params)
  File ""/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py"", line 226, in execute
    return self.conn.execute(sql, parameters)
pysqlite3.dbapi2.OperationalError: table users has no column named countryCode

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743400216,Error thrown: sqlite3.OperationalError: table users has no column named lastName,
https://github.com/simonw/datasette/issues/1191#issuecomment-761705076,https://api.github.com/repos/simonw/datasette/issues/1191,761705076,MDEyOklzc3VlQ29tbWVudDc2MTcwNTA3Ng==,9599,simonw,2021-01-17T00:35:13Z,2021-01-17T00:37:51Z,OWNER,"I'm going to try using Jinja macros to implement this: https://jinja.palletsprojects.com/en/2.11.x/templates/#macros

Maybe using one of these tricks to auto-load the macro? http://codyaray.com/2015/05/auto-load-jinja2-macros","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1181#issuecomment-761703555,https://api.github.com/repos/simonw/datasette/issues/1181,761703555,MDEyOklzc3VlQ29tbWVudDc2MTcwMzU1NQ==,9599,simonw,2021-01-17T00:24:20Z,2021-01-17T00:24:40Z,OWNER,"Here's the incomplete sketch of a test - to go at the bottom of `test_cli.py`.

```python
@pytest.mark.parametrize(
    ""filename"", [""test-database (1).sqlite"", ""database (1).sqlite""]
)
def test_weird_database_names(ensure_eventloop, tmpdir, filename):
    # https://github.com/simonw/datasette/issues/1181
    runner = CliRunner()
    db_path = str(tmpdir / filename)
    sqlite3.connect(db_path).execute(""vacuum"")
    result1 = runner.invoke(cli, [db_path, ""--get"", ""/""])
    assert result1.exit_code == 0, result1.output
    homepage_html = result1.output
    assert False
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""",
https://github.com/simonw/datasette/issues/1191#issuecomment-761703368,https://api.github.com/repos/simonw/datasette/issues/1191,761703368,MDEyOklzc3VlQ29tbWVudDc2MTcwMzM2OA==,9599,simonw,2021-01-17T00:22:46Z,2021-01-17T00:22:46Z,OWNER,I'm going to prototype this in a branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1191#issuecomment-761703232,https://api.github.com/repos/simonw/datasette/issues/1191,761703232,MDEyOklzc3VlQ29tbWVudDc2MTcwMzIzMg==,9599,simonw,2021-01-17T00:21:31Z,2021-01-17T00:21:54Z,OWNER,"I think this ends up being a whole collection of new plugin hooks, something like:

- `include_table_top`
- `include_table_bottom`
- `include_row_top`
- `include_row_bottom`
- `include_database_top`
- `include_database_bottom`
- `include_query_bottom`
- `include_query_bottom`
- `include_index_bottom`
- `include_index_bottom`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/1191#issuecomment-761703022,https://api.github.com/repos/simonw/datasette/issues/1191,761703022,MDEyOklzc3VlQ29tbWVudDc2MTcwMzAyMg==,9599,simonw,2021-01-17T00:20:00Z,2021-01-17T00:20:00Z,OWNER,"Plugins that want to provide extra context to the template can already do so using the `extra_template_vars()` plugin hook.

So this hook could work by returning a list of template filenames to be included. Those templates can be bundled with the plugin. Since they are included they will have access to the template context and to any `extra_template_vars()` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/657#issuecomment-761179229,https://api.github.com/repos/simonw/datasette/issues/657,761179229,MDEyOklzc3VlQ29tbWVudDc2MTE3OTIyOQ==,9599,simonw,2021-01-15T20:24:35Z,2021-01-15T20:24:35Z,OWNER,"I'm not sure how I missed this issue but it's almost a year later and I'm finally taking a look at your Parquet work.

This is yet more evidence that allowing plugins to provide their own custom `Database` objects would be a good idea.

I started exploring what Datasette would like on PostgreSQL in #670 - my concern was that I would need to add a large amount of database abstraction code which would dramatically increase the complexity of the core project, but my thinking now is that it might be tractable - Datasette doesn't actually construct SQL in complex ways anywhere outside of the `TableView` class so abstracting away just that bit should be feasible.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",548591089,Allow creation of virtual tables at startup,
https://github.com/simonw/datasette/issues/1191#issuecomment-761103910,https://api.github.com/repos/simonw/datasette/issues/1191,761103910,MDEyOklzc3VlQ29tbWVudDc2MTEwMzkxMA==,9599,simonw,2021-01-15T18:19:29Z,2021-01-15T18:19:29Z,OWNER,This relates to #987 (documented HTML hooks for JavaScript plugins) but is not quite the same thing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/657#issuecomment-761101878,https://api.github.com/repos/simonw/datasette/issues/657,761101878,MDEyOklzc3VlQ29tbWVudDc2MTEwMTg3OA==,9599,simonw,2021-01-15T18:16:01Z,2021-01-15T18:16:01Z,OWNER,"I think the `startup()` plugin hook at https://docs.datasette.io/en/stable/plugin_hooks.html#startup-datasette should be able to fit this. You can write a plugin which uses that hook to execute `CREATE VIRTUAL TABLE` against  one or more databases when Datasette first starts running.

Would that work here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",548591089,Allow creation of virtual tables at startup,
https://github.com/simonw/sqlite-utils/issues/220#issuecomment-761015218,https://api.github.com/repos/simonw/sqlite-utils/issues/220,761015218,MDEyOklzc3VlQ29tbWVudDc2MTAxNTIxOA==,649467,mhalle,2021-01-15T15:40:08Z,2021-01-15T15:40:08Z,NONE,"Make sense. If you're coming from the sqlite3 side of things, rather than the datasette side, wanting the fts methods to work for views makes more sense. sqlite3 allows fts5 tables on views, so I was looking for CLI functionality to build the fts virtual tables. Ultimately, though, sharing fts virtual tables across tables and derivative views is likely more efficient. 

Maybe an explicit error message like, ""fts is not supported for views"" rather than just throwing an exception that the method doesn't exist"" might be helpful.  Not critical though.

Thanks.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",783778672,Better error message for *_fts methods against views,
https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-760950128,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55,760950128,MDEyOklzc3VlQ29tbWVudDc2MDk1MDEyOA==,21148,jacobian,2021-01-15T13:44:52Z,2021-01-15T13:44:52Z,CONTRIBUTOR,"I found and fixed another bug, this one around importing the tweets table. @simonw let me know if you'd prefer this broken out into multiple PRs, happy to do that if it makes review/merging easier.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779211940,Fix archive imports,
https://github.com/simonw/datasette/issues/1187#issuecomment-759875239,https://api.github.com/repos/simonw/datasette/issues/1187,759875239,MDEyOklzc3VlQ29tbWVudDc1OTg3NTIzOQ==,9599,simonw,2021-01-14T02:02:24Z,2021-01-14T02:02:31Z,OWNER,"This plugin hook currently returns a string of JavaScript. It could optionally return this instead of a string:
```json
{
  ""script"": ""string of JavaScript goes here"",
  ""module"": true
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",785588942,"extra_body_script() support for script type=""module""",
https://github.com/simonw/datasette/issues/1186#issuecomment-759874332,https://api.github.com/repos/simonw/datasette/issues/1186,759874332,MDEyOklzc3VlQ29tbWVudDc1OTg3NDMzMg==,9599,simonw,2021-01-14T01:59:35Z,2021-01-14T01:59:35Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript and https://docs.datasette.io/en/latest/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",785573793,"script type=""module"" support",
https://github.com/simonw/datasette/pull/1159#issuecomment-759306228,https://api.github.com/repos/simonw/datasette/issues/1159,759306228,MDEyOklzc3VlQ29tbWVudDc1OTMwNjIyOA==,552629,lovasoa,2021-01-13T08:59:31Z,2021-01-13T08:59:31Z,NONE,@simonw : Did you have the time to take a look at this ?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/sqlite-utils/issues/220#issuecomment-759098964,https://api.github.com/repos/simonw/sqlite-utils/issues/220,759098964,MDEyOklzc3VlQ29tbWVudDc1OTA5ODk2NA==,9599,simonw,2021-01-12T23:19:55Z,2021-01-12T23:19:55Z,OWNER,"I don't think it makes sense to call `.enable_fts()` on a view does it? When I'm working with views and FTS I tend to write my queries against a FTS table for one of the tables that is used by the view - https://docs.datasette.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view describes how I do that in Datasette for example.

Can you expand on your use-case for FTS and views? I'm ready to be convinced otherwise, but I don't see how it would work right now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",783778672,Better error message for *_fts methods against views,
https://github.com/simonw/datasette/issues/1185#issuecomment-759069342,https://api.github.com/repos/simonw/datasette/issues/1185,759069342,MDEyOklzc3VlQ29tbWVudDc1OTA2OTM0Mg==,9599,simonw,2021-01-12T22:13:18Z,2021-01-12T22:13:18Z,OWNER,I'm going to change the error message to list the allowed pragmas.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",784628163,"""Statement may not contain PRAGMA"" error is not strictly true",
https://github.com/simonw/datasette/issues/1185#issuecomment-759067427,https://api.github.com/repos/simonw/datasette/issues/1185,759067427,MDEyOklzc3VlQ29tbWVudDc1OTA2NzQyNw==,9599,simonw,2021-01-12T22:09:21Z,2021-01-12T22:09:21Z,OWNER,"That allow-list was added in #761 but is not currently documented. It's here in the code:

https://github.com/simonw/datasette/blob/8e8fc5cee5c78da8334495c6d6257d5612c40792/datasette/utils/__init__.py#L173-L186","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",784628163,"""Statement may not contain PRAGMA"" error is not strictly true",
https://github.com/simonw/datasette/issues/1185#issuecomment-759066777,https://api.github.com/repos/simonw/datasette/issues/1185,759066777,MDEyOklzc3VlQ29tbWVudDc1OTA2Njc3Nw==,9599,simonw,2021-01-12T22:07:58Z,2021-01-12T22:07:58Z,OWNER,"https://docs.datasette.io/en/stable/sql_queries.html?highlight=pragma#named-parameters documentation is out-of-date as well:

> Datasette disallows custom SQL containing the string PRAGMA, as SQLite pragma statements can be used to change database settings at runtime. If you need to include the string ""pragma"" in a query you can do so safely using a named parameter.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",784628163,"""Statement may not contain PRAGMA"" error is not strictly true",
https://github.com/simonw/datasette/issues/1091#issuecomment-758668359,https://api.github.com/repos/simonw/datasette/issues/1091,758668359,MDEyOklzc3VlQ29tbWVudDc1ODY2ODM1OQ==,6739646,tballison,2021-01-12T13:52:29Z,2021-01-12T13:52:29Z,NONE,"Y, thank you to both of you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-758448525,https://api.github.com/repos/simonw/datasette/issues/1091,758448525,MDEyOklzc3VlQ29tbWVudDc1ODQ0ODUyNQ==,19328961,henry501,2021-01-12T06:55:08Z,2021-01-12T06:55:08Z,NONE,"Great, really happy I could help! Reverse proxies get tricky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1183#issuecomment-758356640,https://api.github.com/repos/simonw/datasette/issues/1183,758356640,MDEyOklzc3VlQ29tbWVudDc1ODM1NjY0MA==,9599,simonw,2021-01-12T02:42:08Z,2021-01-12T02:42:08Z,OWNER,"Should Datasette have subcommands for this? `datasette enable-counts data.db` and `datasette disable-counts data.db` and `datasette reset-counts data.db` commands?

Maybe. The `sqlite-utils` CLI tool could be used here instead, but that won't be easily available if Datasette was installed as a standalone binary or using `brew install datasette` or `pipx install datasette`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782708469,"Take advantage of sqlite-utils cached table counts, if available",
https://github.com/simonw/datasette/issues/1183#issuecomment-758356097,https://api.github.com/repos/simonw/datasette/issues/1183,758356097,MDEyOklzc3VlQ29tbWVudDc1ODM1NjA5Nw==,9599,simonw,2021-01-12T02:40:30Z,2021-01-12T02:40:30Z,OWNER,"So how would this work?

I think I'm going to automatically use these values if the `_counts` table exists, unless a `ignore_counts_table` boolean setting has been set. I won't bother looking to see if the triggers have been created.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782708469,"Take advantage of sqlite-utils cached table counts, if available",
https://github.com/simonw/datasette/issues/1091#issuecomment-758283074,https://api.github.com/repos/simonw/datasette/issues/1091,758283074,MDEyOklzc3VlQ29tbWVudDc1ODI4MzA3NA==,9599,simonw,2021-01-11T23:12:46Z,2021-01-11T23:12:46Z,OWNER,Fantastic!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-758280611,https://api.github.com/repos/simonw/datasette/issues/1091,758280611,MDEyOklzc3VlQ29tbWVudDc1ODI4MDYxMQ==,6739646,tballison,2021-01-11T23:06:10Z,2021-01-11T23:06:10Z,NONE,"+1

Yep!  Fixes it.  If I navigate to https://corpora.tika.apache.org/datasette, I get a 404 (database not found: datasette), but if I navigate to https://corpora.tika.apache.org/datasette/file_profiles/, everything WORKS!

Thank you!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1182#issuecomment-757375858,https://api.github.com/repos/simonw/datasette/issues/1182,757375858,MDEyOklzc3VlQ29tbWVudDc1NzM3NTg1OA==,9599,simonw,2021-01-09T22:18:47Z,2021-01-09T22:18:47Z,OWNER,https://docs.datasette.io/en/latest/ecosystem.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782692159,"Retire ""Ecosystem"" page in favour of datasette.io/plugins and /tools",
https://github.com/simonw/datasette/issues/1182#issuecomment-757373741,https://api.github.com/repos/simonw/datasette/issues/1182,757373741,MDEyOklzc3VlQ29tbWVudDc1NzM3Mzc0MQ==,9599,simonw,2021-01-09T22:01:41Z,2021-01-09T22:01:41Z,OWNER,It can talk about Dogsheep too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782692159,"Retire ""Ecosystem"" page in favour of datasette.io/plugins and /tools",
https://github.com/simonw/datasette/issues/1182#issuecomment-757373082,https://api.github.com/repos/simonw/datasette/issues/1182,757373082,MDEyOklzc3VlQ29tbWVudDc1NzM3MzA4Mg==,9599,simonw,2021-01-09T21:55:33Z,2021-01-09T21:55:33Z,OWNER,I'll leave the page there but change it into more of a blurb about the existence of the plugins and tools directories.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",782692159,"Retire ""Ecosystem"" page in favour of datasette.io/plugins and /tools",
https://github.com/simonw/datasette/issues/1181#issuecomment-756487966,https://api.github.com/repos/simonw/datasette/issues/1181,756487966,MDEyOklzc3VlQ29tbWVudDc1NjQ4Nzk2Ng==,9599,simonw,2021-01-08T01:25:42Z,2021-01-08T01:25:42Z,OWNER,I'm going to add a unit test that tries a variety of weird database names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""",
https://github.com/simonw/datasette/issues/1181#issuecomment-756482163,https://api.github.com/repos/simonw/datasette/issues/1181,756482163,MDEyOklzc3VlQ29tbWVudDc1NjQ4MjE2Mw==,9599,simonw,2021-01-08T01:06:23Z,2021-01-08T01:06:54Z,OWNER,"Yes, that logic is definitely at fault. It looks like it applies `urllib.parse.unquote_plus()` AFTER it's tried to do the `-` hash splitting thing, which is why it's failing here:

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184-L198","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",781262510,"Certain database names results in 404: ""Database not found: None""",
https://github.com/simonw/datasette/issues/1091#issuecomment-756453945,https://api.github.com/repos/simonw/datasette/issues/1091,756453945,MDEyOklzc3VlQ29tbWVudDc1NjQ1Mzk0NQ==,9599,simonw,2021-01-07T23:42:50Z,2021-01-07T23:42:50Z,OWNER,"@henry501 it looks like you spotted a bug in the documentation - I just addressed that, the fix is now live here: https://docs.datasette.io/en/latest/deploying.html#running-datasette-behind-a-proxy","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-756453010,https://api.github.com/repos/simonw/datasette/issues/1091,756453010,MDEyOklzc3VlQ29tbWVudDc1NjQ1MzAxMA==,9599,simonw,2021-01-07T23:39:58Z,2021-01-07T23:40:25Z,OWNER,"@tballison I think that's the solution! It looks like you need to use this in your config:

`ProxyPass /datasette http://127.0.0.1:8001/datasette`

Instead of this:

`ProxyPass /datasette http://127.0.0.1:8001/`

Give that a go and let me know if it fixes it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-756425587,https://api.github.com/repos/simonw/datasette/issues/1091,756425587,MDEyOklzc3VlQ29tbWVudDc1NjQyNTU4Nw==,19328961,henry501,2021-01-07T22:27:19Z,2021-01-07T22:27:19Z,NONE,"I found this issue while troubleshooting the same behavior with an nginx reverse proxy. The solution was to make sure I set:

`proxy_pass http://server:8001/baseurl/ 
`
instead of just:

`proxy_pass http://server:8001
`
The custom SQL query and header links are now correct.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1171#issuecomment-756335394,https://api.github.com/repos/simonw/datasette/issues/1171,756335394,MDEyOklzc3VlQ29tbWVudDc1NjMzNTM5NA==,9599,simonw,2021-01-07T19:35:59Z,2021-01-07T19:35:59Z,OWNER,"I requested a D-U-N-S number as a first step in getting a developer certificate: https://developer.apple.com/support/D-U-N-S/
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1180#issuecomment-756312213,https://api.github.com/repos/simonw/datasette/issues/1180,756312213,MDEyOklzc3VlQ29tbWVudDc1NjMxMjIxMw==,9599,simonw,2021-01-07T18:56:24Z,2021-01-07T18:56:24Z,OWNER,The `async_call_with_supported_arguments` version should be able to await any of the lazy arguments that are awaitable - can use `await_me_maybe` for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780767542,Lazily evaluated arguments for call_with_supported_arguments,
https://github.com/simonw/datasette/issues/1180#issuecomment-755500475,https://api.github.com/repos/simonw/datasette/issues/1180,755500475,MDEyOklzc3VlQ29tbWVudDc1NTUwMDQ3NQ==,9599,simonw,2021-01-06T18:43:41Z,2021-01-06T18:43:41Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/__init__.py#L919-L940,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780767542,Lazily evaluated arguments for call_with_supported_arguments,
https://github.com/simonw/datasette/issues/1179#issuecomment-755495387,https://api.github.com/repos/simonw/datasette/issues/1179,755495387,MDEyOklzc3VlQ29tbWVudDc1NTQ5NTM4Nw==,9599,simonw,2021-01-06T18:39:23Z,2021-01-06T18:39:23Z,OWNER,"In that case maybe there are three new arguments: `path`, `full_path` and `url`.

I'll also add `request.full_path` for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-755492945,https://api.github.com/repos/simonw/datasette/issues/1179,755492945,MDEyOklzc3VlQ29tbWVudDc1NTQ5Mjk0NQ==,9599,simonw,2021-01-06T18:37:39Z,2021-01-06T18:37:39Z,OWNER,I think I'll call this `full_path` for consistency with Django.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-755489974,https://api.github.com/repos/simonw/datasette/issues/1179,755489974,MDEyOklzc3VlQ29tbWVudDc1NTQ4OTk3NA==,9599,simonw,2021-01-06T18:35:24Z,2021-01-06T18:35:24Z,OWNER,Django calls this ` HttpRequest.get_full_path()` - for the path plus the querystring.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1179#issuecomment-755486103,https://api.github.com/repos/simonw/datasette/issues/1179,755486103,MDEyOklzc3VlQ29tbWVudDc1NTQ4NjEwMw==,9599,simonw,2021-01-06T18:32:41Z,2021-01-06T18:34:11Z,OWNER,"This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. `/github/issue_comments` or `/github/issue_comments?_sort_desc=created_at`

Open questions:

- What should it be called? `path` could be misleading since it also includes the querystring.
- Should I provide a `url` or `full_url` version which includes `https://blah.com/...`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/782#issuecomment-755484384,https://api.github.com/repos/simonw/datasette/issues/782,755484384,MDEyOklzc3VlQ29tbWVudDc1NTQ4NDM4NA==,9599,simonw,2021-01-06T18:31:14Z,2021-01-06T18:31:57Z,OWNER,"In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3:
```python
import pandas
df = pandas.read_json(
  ""https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array""
)
```
And:
```javascript
d3 = require(""d3@5"")
rows = d3.json(
    ""https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array""
)
```
Once again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both `pandas.read_json()` and `d3.json()` - but it leaves nowhere to put the extra information like pagination and suchlike!

Even given this I still think the correct default is an object with `""rows""`, `""total""` and `""next_url""` keys. I should commit to that and implement it - this thought exercise has been running for far too long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/1178#issuecomment-755476820,https://api.github.com/repos/simonw/datasette/issues/1178,755476820,MDEyOklzc3VlQ29tbWVudDc1NTQ3NjgyMA==,9599,simonw,2021-01-06T18:24:47Z,2021-01-06T18:24:47Z,OWNER,"Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now.

I can't think of a reason anyone on Cloud Run would ever NOT want the `force_https_urls` option, but just in case I've made it so if you pass `--extra-options --setting force_https_urls off` to `publish cloudrun` your setting will be respected.

https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755468795,https://api.github.com/repos/simonw/datasette/issues/1178,755468795,MDEyOklzc3VlQ29tbWVudDc1NTQ2ODc5NQ==,9599,simonw,2021-01-06T18:14:35Z,2021-01-06T18:14:35Z,OWNER,Deploying that change now to test it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755163886,https://api.github.com/repos/simonw/datasette/issues/1178,755163886,MDEyOklzc3VlQ29tbWVudDc1NTE2Mzg4Ng==,9599,simonw,2021-01-06T08:37:51Z,2021-01-06T08:37:51Z,OWNER,"Easiest fix would be for `publish cloudrun` to set `force_https_urls`:

`datasette publish now` used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1179#issuecomment-755161574,https://api.github.com/repos/simonw/datasette/issues/1179,755161574,MDEyOklzc3VlQ29tbWVudDc1NTE2MTU3NA==,9599,simonw,2021-01-06T08:32:31Z,2021-01-06T08:32:31Z,OWNER,An optional `path` argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the `.Notebook` extension would be useful here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780278550,Make original path available to render hooks,
https://github.com/simonw/datasette/issues/1178#issuecomment-755160187,https://api.github.com/repos/simonw/datasette/issues/1178,755160187,MDEyOklzc3VlQ29tbWVudDc1NTE2MDE4Nw==,9599,simonw,2021-01-06T08:29:35Z,2021-01-06T08:29:35Z,OWNER,"https://latest-with-plugins.datasette.io/-/asgi-scope

```
{'asgi': {'spec_version': '2.1', 'version': '3.0'},
 'client': ('169.254.8.129', 54971),
 'headers': [(b'host', b'latest-with-plugins.datasette.io'),
             (b'user-agent',
              b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko'
              b'/20100101 Firefox/84.0'),
             (b'accept',
              b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
              b'webp,*/*;q=0.8'),
             (b'accept-language', b'en-US,en;q=0.5'),
             (b'dnt', b'1'),
             (b'cookie',
              b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1'
              b'.894633707.1607575712'),
             (b'upgrade-insecure-requests', b'1'),
             (b'x-client-data', b'CgSL6ZsV'),
             (b'x-cloud-trace-context',
              b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'),
             (b'x-forwarded-for', b'148.64.98.14'),
             (b'x-forwarded-proto', b'https'),
             (b'forwarded', b'for=""148.64.98.14"";proto=https'),
             (b'accept-encoding', b'gzip, deflate, br'),
             (b'content-length', b'0')],
 'http_version': '1.1',
 'method': 'GET',
 'path': '/-/asgi-scope',
 'query_string': b'',
 'raw_path': b'/-/asgi-scope',
 'root_path': '',
 'scheme': 'http',
 'server': ('169.254.8.130', 8080),
 'type': 'http'}
```
Note the `'scheme': 'http'` but also the `(b'x-forwarded-proto', b'https')`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1176#issuecomment-755159583,https://api.github.com/repos/simonw/datasette/issues/1176,755159583,MDEyOklzc3VlQ29tbWVudDc1NTE1OTU4Mw==,9599,simonw,2021-01-06T08:28:20Z,2021-01-06T08:28:20Z,OWNER,I used `from datasette.utils import path_with_format` in https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py just now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1178#issuecomment-755158310,https://api.github.com/repos/simonw/datasette/issues/1178,755158310,MDEyOklzc3VlQ29tbWVudDc1NTE1ODMxMA==,9599,simonw,2021-01-06T08:25:31Z,2021-01-06T08:25:31Z,OWNER,Moving this to the Datasette repo.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755157732,https://api.github.com/repos/simonw/datasette/issues/1178,755157732,MDEyOklzc3VlQ29tbWVudDc1NTE1NzczMg==,9599,simonw,2021-01-06T08:24:12Z,2021-01-06T08:24:12Z,OWNER,https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the `next_url` is `http://` when it should be `https://`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755157281,https://api.github.com/repos/simonw/datasette/issues/1178,755157281,MDEyOklzc3VlQ29tbWVudDc1NTE1NzI4MQ==,9599,simonw,2021-01-06T08:23:14Z,2021-01-06T08:23:14Z,OWNER,"https://latest-with-plugins.datasette.io/-/settings says `""force_https_urls"": false`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755157066,https://api.github.com/repos/simonw/datasette/issues/1178,755157066,MDEyOklzc3VlQ29tbWVudDc1NTE1NzA2Ng==,9599,simonw,2021-01-06T08:22:47Z,2021-01-06T08:22:47Z,OWNER,"Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613

```python
    def absolute_url(self, request, path):
        url = urllib.parse.urljoin(request.url, path)
        if url.startswith(""http://"") and self.setting(""force_https_urls""):
            url = ""https://"" + url[len(""http://"") :]
        return url
```
That looks like it should work. Needs more digging.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1178#issuecomment-755156606,https://api.github.com/repos/simonw/datasette/issues/1178,755156606,MDEyOklzc3VlQ29tbWVudDc1NTE1NjYwNg==,9599,simonw,2021-01-06T08:21:49Z,2021-01-06T08:21:49Z,OWNER,"https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/__init__.py#L18-L23

Maybe this is a bug in `datasette.absolute_url`? Perhaps it doesn't take the scheme into account.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",780267857,Use force_https_urls on when deploying with Cloud Run,
https://github.com/simonw/datasette/issues/1101#issuecomment-755134771,https://api.github.com/repos/simonw/datasette/issues/1101,755134771,MDEyOklzc3VlQ29tbWVudDc1NTEzNDc3MQ==,9599,simonw,2021-01-06T07:28:01Z,2021-01-06T07:28:01Z,OWNER,"With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the `stream_rows()` method could output `[` first, then each row followed by a comma, then `]` after the very last row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-755133937,https://api.github.com/repos/simonw/datasette/issues/1101,755133937,MDEyOklzc3VlQ29tbWVudDc1NTEzMzkzNw==,9599,simonw,2021-01-06T07:25:48Z,2021-01-06T07:26:43Z,OWNER,"Idea: instead of returning a dictionary, `register_output_renderer` could return an object. The object could have the following properties:

- `.extension` - the extension to use
- `.can_render(...)` - says if it can render this
- `.can_stream(...)` - says if streaming is supported
- `async .stream_rows(rows_iterator, send)` - method that loops through all rows and uses `send` to send them to the response in the correct format

I can then deprecate the existing `dict` return type for 1.0.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-755128038,https://api.github.com/repos/simonw/datasette/issues/1101,755128038,MDEyOklzc3VlQ29tbWVudDc1NTEyODAzOA==,9599,simonw,2021-01-06T07:10:22Z,2021-01-06T07:10:22Z,OWNER,"Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

    pandas.read_json(""https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on"", lines=True)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1171#issuecomment-754958998,https://api.github.com/repos/simonw/datasette/issues/1171,754958998,MDEyOklzc3VlQ29tbWVudDc1NDk1ODk5OA==,9599,simonw,2021-01-05T23:16:33Z,2021-01-05T23:16:33Z,OWNER,"That's really useful, thanks @rcoup ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/782#issuecomment-754958610,https://api.github.com/repos/simonw/datasette/issues/782,754958610,MDEyOklzc3VlQ29tbWVudDc1NDk1ODYxMA==,9599,simonw,2021-01-05T23:15:24Z,2021-01-05T23:15:24Z,OWNER,https://latest-with-plugins.datasette.io/fixtures/roadside_attraction_characteristics/1.json-preview returns a 500 error at the moment - a KeyError on 'filtered_table_rows_count'.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/576#issuecomment-754957658,https://api.github.com/repos/simonw/datasette/issues/576,754957658,MDEyOklzc3VlQ29tbWVudDc1NDk1NzY1OA==,9599,simonw,2021-01-05T23:12:50Z,2021-01-05T23:12:50Z,OWNER,See https://docs.datasette.io/en/stable/internals.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/576#issuecomment-754957563,https://api.github.com/repos/simonw/datasette/issues/576,754957563,MDEyOklzc3VlQ29tbWVudDc1NDk1NzU2Mw==,9599,simonw,2021-01-05T23:12:37Z,2021-01-05T23:12:37Z,OWNER,"I'm happy with how this has evolved, so I'm closing the issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497170355,Documented internals API for use in plugins,
https://github.com/simonw/datasette/issues/1176#issuecomment-754957378,https://api.github.com/repos/simonw/datasette/issues/1176,754957378,MDEyOklzc3VlQ29tbWVudDc1NDk1NzM3OA==,9599,simonw,2021-01-05T23:12:03Z,2021-01-05T23:12:03Z,OWNER,This needs to be done for Datasette 1.0. At the very least I need to ensure it's clear that `datasette.utils` is not part of the public API unless explicitly marked as such.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1176#issuecomment-754952146,https://api.github.com/repos/simonw/datasette/issues/1176,754952146,MDEyOklzc3VlQ29tbWVudDc1NDk1MjE0Ng==,9599,simonw,2021-01-05T22:57:26Z,2021-01-05T22:57:26Z,OWNER,Known public APIs might be worth adding type annotations to as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1176#issuecomment-754952040,https://api.github.com/repos/simonw/datasette/issues/1176,754952040,MDEyOklzc3VlQ29tbWVudDc1NDk1MjA0MA==,9599,simonw,2021-01-05T22:57:09Z,2021-01-05T22:57:09Z,OWNER,It might be neater to move all of the non-public functions into a separate module - `datasette.utils.internal` perhaps.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1176#issuecomment-754951786,https://api.github.com/repos/simonw/datasette/issues/1176,754951786,MDEyOklzc3VlQ29tbWVudDc1NDk1MTc4Ng==,9599,simonw,2021-01-05T22:56:27Z,2021-01-05T22:56:43Z,OWNER,"Idea: introduce a `@documented` decorator which marks specific functions as part of the public, documented API. The unit tests can then confirm that anything with that decorator is both documented and tested.
```python
@documented
def escape_css_string(s):
    return _css_re.sub(
        lambda m: ""\\"" + (f""{ord(m.group()):X}"".zfill(6)),
        s.replace(""\r\n"", ""\n""),
    )
```
Or maybe `@public`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779691739,"Policy on documenting ""public"" datasette.utils functions",
https://github.com/simonw/datasette/issues/1171#issuecomment-754911290,https://api.github.com/repos/simonw/datasette/issues/1171,754911290,MDEyOklzc3VlQ29tbWVudDc1NDkxMTI5MA==,59874,rcoup,2021-01-05T21:31:15Z,2021-01-05T21:31:15Z,NONE,"We did this for [Sno](https://sno.earth) under macOS — it's a PyInstaller binary/setup which uses [Packages](http://s.sudre.free.fr/Software/Packages/about.html) for packaging.

* [Building & Signing](https://github.com/koordinates/sno/blob/master/platforms/Makefile#L67-L95)
* [Packaging & Notarizing](https://github.com/koordinates/sno/blob/master/platforms/Makefile#L121-L215)
* [Github Workflow](https://github.com/koordinates/sno/blob/master/.github/workflows/build.yml#L228-L269) has the CI side of it

FYI (if you ever get to it) for Windows you need to get a code signing certificate. And if you want automated CI, you'll want to get an ""EV CodeSigning for HSM"" certificate from GlobalSign, which then lets you put the certificate into Azure Key Vault. Which you can use with [azuresigntool](https://github.com/vcsjones/AzureSignTool) to sign your code & installer. (Non-EV certificates are a waste of time, the user still gets big warnings at install time).
","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754729035,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,754729035,MDEyOklzc3VlQ29tbWVudDc1NDcyOTAzNQ==,21148,jacobian,2021-01-05T16:03:29Z,2021-01-05T16:03:29Z,CONTRIBUTOR,"I was able to fix this, at least enough to get _my_ archive to import. Not sure if there's more work to be done here or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-754728696,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55,754728696,MDEyOklzc3VlQ29tbWVudDc1NDcyODY5Ng==,21148,jacobian,2021-01-05T16:02:55Z,2021-01-05T16:02:55Z,CONTRIBUTOR,"This now works for me, though I'm entirely ensure if it's a just-my-export thing or a wider issue. Also, this doesn't contain any tests. So I'm not sure if there's more work to be done here, or if this is good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779211940,Fix archive imports,
https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754721153,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54,754721153,MDEyOklzc3VlQ29tbWVudDc1NDcyMTE1Mw==,21148,jacobian,2021-01-05T15:51:09Z,2021-01-05T15:51:09Z,CONTRIBUTOR,"Correction: the failure is on `lists-member.js` (I was thrown by the `block` variable name, but that's just a coincidence)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779088071,Archive import appears to be broken on recent exports,
https://github.com/simonw/datasette/issues/1175#issuecomment-754696725,https://api.github.com/repos/simonw/datasette/issues/1175,754696725,MDEyOklzc3VlQ29tbWVudDc1NDY5NjcyNQ==,9599,simonw,2021-01-05T15:12:30Z,2021-01-05T15:12:30Z,OWNER,Some tips here: https://github.com/tiangolo/fastapi/issues/78,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",779156520,Use structlog for logging,
https://github.com/simonw/datasette/issues/1167#issuecomment-754619930,https://api.github.com/repos/simonw/datasette/issues/1167,754619930,MDEyOklzc3VlQ29tbWVudDc1NDYxOTkzMA==,3637,benpickles,2021-01-05T12:57:57Z,2021-01-05T12:57:57Z,CONTRIBUTOR,"Not sure where exactly to put the actual docs (presumably somewhere in [docs/contributing.rst](https://github.com/simonw/datasette/blob/main/docs/contributing.rst)) but I've made a slight change to make it easier to run locally (copying [the approach in excalidraw](https://github.com/excalidraw/excalidraw/blob/ade2565f497243a5e428f4906d8ed80c872fd981/package.json#L90-L94)): https://github.com/simonw/datasette/compare/main...benpickles:prettier-docs

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777145954,Add Prettier to contributing documentation,
https://github.com/simonw/datasette/issues/859#issuecomment-647922203,https://api.github.com/repos/simonw/datasette/issues/859,647922203,MDEyOklzc3VlQ29tbWVudDY0NzkyMjIwMw==,3243482,abdusco,2020-06-23T05:44:58Z,2021-01-05T08:22:43Z,CONTRIBUTOR,"I'm seeing the problem on database page. Index page and table page runs quite fast.

- Tables have <10 columns (`id`, `url`, `title`, `body_html`, `date`, `author`, `meta` (for keeping unstructured json)). I've added index on `date` columns (using `sqlite-utils`) in addition to the index present on `id` columns. 
- All tables have FTS enabled on `text` and `varchar` columns (`title`, `body_html` etc) to speed up searching.
- There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with `thread_id`)

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642572841,Database page loads too slowly with many large tables (due to table counts),
https://github.com/simonw/datasette/issues/1173#issuecomment-754463845,https://api.github.com/repos/simonw/datasette/issues/1173,754463845,MDEyOklzc3VlQ29tbWVudDc1NDQ2Mzg0NQ==,9599,simonw,2021-01-05T07:41:43Z,2021-01-05T07:41:43Z,OWNER,"https://github.com/oleksis/pyinstaller-manylinux looks useful, via https://twitter.com/oleksis/status/1346341987876823040","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778682317,GitHub Actions workflow to build manylinux binary,
https://github.com/simonw/datasette/issues/1171#issuecomment-754296761,https://api.github.com/repos/simonw/datasette/issues/1171,754296761,MDEyOklzc3VlQ29tbWVudDc1NDI5Njc2MQ==,9599,simonw,2021-01-04T23:55:44Z,2021-01-04T23:55:44Z,OWNER,Bit uncomfortable that it looks like you need to include your Apple ID username and password in the CI configuration to do this. I'll use GitHub Secrets for this but I don't like it - I'll definitely setup a dedicated code signing account that's not my access-to-everything AppleID for this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754295380,https://api.github.com/repos/simonw/datasette/issues/1171,754295380,MDEyOklzc3VlQ29tbWVudDc1NDI5NTM4MA==,9599,simonw,2021-01-04T23:54:32Z,2021-01-04T23:54:32Z,OWNER,"https://github.com/search?l=YAML&q=gon+json&type=Code reveals some examples of people using `gon` in workflows.

These look useful:

* https://github.com/coherence/hub-server/blob/3b7e9c7c5bce9e244b14b854f1f89d66f53a5a39/.github/workflows/release_build.yml
* https://github.com/simoncozens/pilcrow/blob/5abc145e7fb9577086afe47b48fd730cb8195386/.github/workflows/buildapp.yaml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754287882,https://api.github.com/repos/simonw/datasette/issues/1171,754287882,MDEyOklzc3VlQ29tbWVudDc1NDI4Nzg4Mg==,9599,simonw,2021-01-04T23:40:10Z,2021-01-04T23:42:32Z,OWNER,"This looks VERY useful: https://github.com/mitchellh/gon - "" Sign, notarize, and package macOS CLI tools and applications written in any language. Available as both a CLI and a Go library.""

And it installs like this:

    brew install mitchellh/gon/gon","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754286783,https://api.github.com/repos/simonw/datasette/issues/1171,754286783,MDEyOklzc3VlQ29tbWVudDc1NDI4Njc4Mw==,9599,simonw,2021-01-04T23:38:18Z,2021-01-04T23:38:18Z,OWNER,Oh wow maybe I need to Notarize it too? https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/1171#issuecomment-754286618,https://api.github.com/repos/simonw/datasette/issues/1171,754286618,MDEyOklzc3VlQ29tbWVudDc1NDI4NjYxOA==,9599,simonw,2021-01-04T23:37:45Z,2021-01-04T23:37:45Z,OWNER,https://github.com/actions/virtual-environments/issues/1820#issuecomment-719549887 looks useful - not sure if those notes are for iOS or macOS though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778450486,GitHub Actions workflow to build and sign macOS binary executables,
https://github.com/simonw/datasette/issues/93#issuecomment-754285795,https://api.github.com/repos/simonw/datasette/issues/93,754285795,MDEyOklzc3VlQ29tbWVudDc1NDI4NTc5NQ==,9599,simonw,2021-01-04T23:35:13Z,2021-01-04T23:35:13Z,OWNER,Next step is to automate this all!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/1152#issuecomment-754285588,https://api.github.com/repos/simonw/datasette/issues/1152,754285588,MDEyOklzc3VlQ29tbWVudDc1NDI4NTU4OA==,9599,simonw,2021-01-04T23:34:30Z,2021-01-04T23:34:30Z,OWNER,"I think the way to do this is to have a new plugin hook that returns two SQL where clauses: one returning a list of resources that the user should be able to access (the allow-list) and one returning a list of resources they are explicitly forbidden from accessing (the deny-list). Either of these can be blank.

Datasette can then combine those into a full SQL query and use it to answer the question ""show me a list of resources that the user is allowed to perform action X on"". It can also answer the existing question, ""is user X allowed to perform action Y on resource Z""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/93#issuecomment-754233960,https://api.github.com/repos/simonw/datasette/issues/93,754233960,MDEyOklzc3VlQ29tbWVudDc1NDIzMzk2MA==,9599,simonw,2021-01-04T21:35:37Z,2021-01-04T21:35:37Z,OWNER,"I tested it by running a `tmate` session on the GitHub macOS machines, and it worked!
```
Mac-1609795972770:tmp runner$ wget 'https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip'
--2021-01-04 21:34:10--  https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream [following]
--2021-01-04 21:34:14--  https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream
Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.43.164
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.43.164|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8297283 (7.9M) [application/octet-stream]
Saving to: ‘datasette-0.53-macos-binary.zip’

datasette-0.53-maco 100%[===================>]   7.91M  --.-KB/s    in 0.1s    

2021-01-04 21:34:14 (73.4 MB/s) - ‘datasette-0.53-macos-binary.zip’ saved [8297283/8297283]

Mac-1609795972770:tmp runner$ unzip datasette-0.53-macos-binary.zip 
Archive:  datasette-0.53-macos-binary.zip
   creating: datasette-0.53-macos-binary/
  inflating: datasette-0.53-macos-binary/datasette  
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  serve*     Serve up specified SQLite database files with a web UI
  inspect
  install    Install Python packages - e.g.
  package    Package specified SQLite files into a new datasette Docker...
  plugins    List currently available plugins
  publish    Publish specified SQLite database files to the internet along...
  uninstall  Uninstall Python packages (e.g.
Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --get /-/versions.json
{""python"": {""version"": ""3.9.1"", ""full"": ""3.9.1 (default, Dec 10 2020, 10:36:35) \n[Clang 12.0.0 (clang-1200.0.32.27)]""}, ""datasette"": {""version"": ""0.53""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.13.3"", ""sqlite"": {""version"": ""3.34.0"", ""fts_versions"": [""FTS5"", ""FTS4"", ""FTS3""], ""extensions"": {""json1"": null}, ""compile_options"": [""COMPILER=clang-12.0.0"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_FTS3"", ""ENABLE_FTS3_PARENTHESIS"", ""ENABLE_FTS4"", ""ENABLE_FTS5"", ""ENABLE_GEOPOLY"", ""ENABLE_JSON1"", ""ENABLE_PREUPDATE_HOOK"", ""ENABLE_RTREE"", ""ENABLE_SESSION"", ""MAX_VARIABLE_NUMBER=250000"", ""THREADSAFE=1""]}}
Mac-1609795972770:tmp runner$ 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754229977,https://api.github.com/repos/simonw/datasette/issues/93,754229977,MDEyOklzc3VlQ29tbWVudDc1NDIyOTk3Nw==,9599,simonw,2021-01-04T21:28:01Z,2021-01-04T21:28:01Z,OWNER,"As an experiment, I put the macOS one in a zip file and attached it to the latest release:

```
mkdir datasette-0.53-macos-binary
cp dist/datasette datasette-0.53-macos-binary
zip -r datasette-0.53-macos-binary.zip datasette-0.53-macos-binary
```

It's available here: https://github.com/simonw/datasette/releases/tag/0.53 - download URL is https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754227543,https://api.github.com/repos/simonw/datasette/issues/93,754227543,MDEyOklzc3VlQ29tbWVudDc1NDIyNzU0Mw==,9599,simonw,2021-01-04T21:23:13Z,2021-01-04T21:23:13Z,OWNER,"```
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# dist/datasette --get /-/databases.json
[{""name"": "":memory:"", ""path"": null, ""size"": 0, ""is_mutable"": true, ""is_memory"": true, ""hash"": null}]
(pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# ls -lah dist/datasette 
-rwxr-xr-x 1 root root 8.9M Jan  4 21:05 dist/datasette
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754219002,https://api.github.com/repos/simonw/datasette/issues/93,754219002,MDEyOklzc3VlQ29tbWVudDc1NDIxOTAwMg==,9599,simonw,2021-01-04T21:06:49Z,2021-01-04T21:22:27Z,OWNER,"Works on Linux/Ubuntu too, except I had to do `export BASE=` on a separate line. I also did this:
```
apt-get install python3 python3-venv
python3 -m venv pyinstaller-venv
source pyinstaller-venv/bin/activate
pip install wheel
pip install datasette pyinstaller

export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__(""datasette"").__file__))')

pyinstaller -F \
    --add-data ""$DATASETTE_BASE/templates:datasette/templates"" \
    --add-data ""$DATASETTE_BASE/static:datasette/static"" \
    --hidden-import datasette.publish \
    --hidden-import datasette.publish.heroku \
    --hidden-import datasette.publish.cloudrun \
    --hidden-import datasette.facets \
    --hidden-import datasette.sql_functions \
    --hidden-import datasette.actor_auth_cookie \
    --hidden-import datasette.default_permissions \
    --hidden-import datasette.default_magic_parameters \
    --hidden-import datasette.blob_renderer \
    --hidden-import datasette.default_menu_links \
    --hidden-import uvicorn \
    --hidden-import uvicorn.logging \
    --hidden-import uvicorn.loops \
    --hidden-import uvicorn.loops.auto \
    --hidden-import uvicorn.protocols \
    --hidden-import uvicorn.protocols.http \
    --hidden-import uvicorn.protocols.http.auto \
    --hidden-import uvicorn.protocols.websockets \
    --hidden-import uvicorn.protocols.websockets.auto \
    --hidden-import uvicorn.lifespan \
    --hidden-import uvicorn.lifespan.on \
    $(which datasette)
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754218545,https://api.github.com/repos/simonw/datasette/issues/93,754218545,MDEyOklzc3VlQ29tbWVudDc1NDIxODU0NQ==,9599,simonw,2021-01-04T21:05:57Z,2021-01-04T21:05:57Z,OWNER,That BASE= trick seems to work with `zsh` but not with `bash`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754215392,https://api.github.com/repos/simonw/datasette/issues/93,754215392,MDEyOklzc3VlQ29tbWVudDc1NDIxNTM5Mg==,9599,simonw,2021-01-04T20:59:20Z,2021-01-04T21:03:14Z,OWNER,"Updated `pyinstaller` recipe - lots of hidden imports needed now:
```
pip install wheel
pip install datasette pyinstaller

BASE=$(python -c 'import os; print(os.path.dirname(__import__(""datasette"").__file__))') \
    pyinstaller -F \
        --add-data ""$BASE/templates:datasette/templates"" \
        --add-data ""$BASE/static:datasette/static"" \
        --hidden-import datasette.publish \
        --hidden-import datasette.publish.heroku \
        --hidden-import datasette.publish.cloudrun \
        --hidden-import datasette.facets \
        --hidden-import datasette.sql_functions \
        --hidden-import datasette.actor_auth_cookie \
        --hidden-import datasette.default_permissions \
        --hidden-import datasette.default_magic_parameters \
        --hidden-import datasette.blob_renderer \
        --hidden-import datasette.default_menu_links \
        --hidden-import uvicorn \
        --hidden-import uvicorn.logging \
        --hidden-import uvicorn.loops \
        --hidden-import uvicorn.loops.auto \
        --hidden-import uvicorn.protocols \
        --hidden-import uvicorn.protocols.http \
        --hidden-import uvicorn.protocols.http.auto \
        --hidden-import uvicorn.protocols.websockets \
        --hidden-import uvicorn.protocols.websockets.auto \
        --hidden-import uvicorn.lifespan \
        --hidden-import uvicorn.lifespan.on \
        $(which datasette)
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/93#issuecomment-754215793,https://api.github.com/repos/simonw/datasette/issues/93,754215793,MDEyOklzc3VlQ29tbWVudDc1NDIxNTc5Mw==,9599,simonw,2021-01-04T21:00:14Z,2021-01-04T21:00:14Z,OWNER,"```
(pyinstaller-datasette) pyinstaller-datasette % file dist/datasette
dist/datasette: Mach-O 64-bit executable x86_64
(pyinstaller-datasette) pyinstaller-datasette % ls -lah dist/datasette
-rwxr-xr-x  1 simon  wheel   8.0M Jan  4 12:58 dist/datasette
```
I'm surprised it's only 8MB!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",273944952,Package as standalone binary,
https://github.com/simonw/datasette/issues/983#issuecomment-754210356,https://api.github.com/repos/simonw/datasette/issues/983,754210356,MDEyOklzc3VlQ29tbWVudDc1NDIxMDM1Ng==,222245,carlmjohnson,2021-01-04T20:49:05Z,2021-01-04T20:49:05Z,NONE,"For reasons [I've written about elsewhere](https://blog.carlmjohnson.net/post/2020/time-to-kill-ie11/), I'm in favor of modules. It has several beneficial effects. One, old browsers just ignore it all together. Two, if you include the same plain script on the page more than once, it will be executed twice, but if you include the same module script on a page twice, it will only execute once. Three, you get a module local namespace, instead of having to use the global window namespace or a function private namespace.

OTOH, if you are going to use an old style script, the code from before isn't ideal, because you wipe out your registry if the script it included more than once. Also you may as well use object methods and splat arguments.

The event based architecture probably makes more sense though. Just make up some event names prefixed with `datasette:` and listen for them on the root. The only concern with that approach is it can sometimes be tricky to make sure your plugins are run after datasette has run. Maybe 

```js
function mycallback(){
  // whatever
}

if (window.datasette) {
  window.datasette.init(mycallback);
} else {
  document.addEventListener('datasette:init', mycallback);
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/668#issuecomment-754194996,https://api.github.com/repos/simonw/datasette/issues/668,754194996,MDEyOklzc3VlQ29tbWVudDc1NDE5NDk5Ng==,9599,simonw,2021-01-04T20:18:39Z,2021-01-04T20:18:39Z,OWNER,I fixed this in #1115 - you can run `--load-extension=spatialite` now and it will look for the extension in common places.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",563347679,Make it easier to load SpatiaLite,
https://github.com/simonw/datasette/issues/436#issuecomment-754193501,https://api.github.com/repos/simonw/datasette/issues/436,754193501,MDEyOklzc3VlQ29tbWVudDc1NDE5MzUwMQ==,9599,simonw,2021-01-04T20:15:41Z,2021-01-04T20:15:41Z,OWNER,"Sadly `publish.datasettes.com` was broken by changes to Zeit, and I don't think I'll be bringing it back.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",435819321,400 Error when trying to register new user via https://publish.datasettes.com/,
https://github.com/simonw/datasette/issues/371#issuecomment-754192873,https://api.github.com/repos/simonw/datasette/issues/371,754192873,MDEyOklzc3VlQ29tbWVudDc1NDE5Mjg3Mw==,9599,simonw,2021-01-04T20:14:28Z,2021-01-04T20:14:28Z,OWNER,"Now that Digital Ocean has App Platform this is less necessary, especially since the documentation covers how to use App Platform here: https://docs.datasette.io/en/stable/deploying.html#deploying-using-buildpacks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",377156339,datasette publish digitalocean plugin,
https://github.com/simonw/datasette/issues/102#issuecomment-754192267,https://api.github.com/repos/simonw/datasette/issues/102,754192267,MDEyOklzc3VlQ29tbWVudDc1NDE5MjI2Nw==,9599,simonw,2021-01-04T20:13:19Z,2021-01-04T20:13:19Z,OWNER,"I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274264175,datasette publish elasticbeanstalk,
https://github.com/simonw/datasette/issues/221#issuecomment-754191699,https://api.github.com/repos/simonw/datasette/issues/221,754191699,MDEyOklzc3VlQ29tbWVudDc1NDE5MTY5OQ==,9599,simonw,2021-01-04T20:12:14Z,2021-01-04T20:12:14Z,OWNER,I'm going to close this. Plugins can register their own CLI tools (see https://github.com/simonw/click-app) if they need to.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315142414,Allow plugins to add new cli sub commands ,
https://github.com/simonw/datasette/issues/221#issuecomment-754190952,https://api.github.com/repos/simonw/datasette/issues/221,754190952,MDEyOklzc3VlQ29tbWVudDc1NDE5MDk1Mg==,9599,simonw,2021-01-04T20:10:51Z,2021-01-04T20:10:51Z,OWNER,Is this still a good idea? I don't have any pressing need for it at the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315142414,Allow plugins to add new cli sub commands ,
https://github.com/simonw/datasette/issues/221#issuecomment-754190814,https://api.github.com/repos/simonw/datasette/issues/221,754190814,MDEyOklzc3VlQ29tbWVudDc1NDE5MDgxNA==,9599,simonw,2021-01-04T20:10:34Z,2021-01-04T20:10:34Z,OWNER,"For the `csvs-to-sqlite` case I'm going with `datasette insert` instead, see #1160.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315142414,Allow plugins to add new cli sub commands ,
https://github.com/simonw/datasette/issues/18#issuecomment-754188383,https://api.github.com/repos/simonw/datasette/issues/18,754188383,MDEyOklzc3VlQ29tbWVudDc1NDE4ODM4Mw==,9599,simonw,2021-01-04T20:05:48Z,2021-01-04T20:05:48Z,OWNER,"I'm not using Sanic any more, but this is still very feasible. If I ever do it I'll write a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",267739593,See if I can get a websockets interface working,
https://github.com/simonw/datasette/issues/103#issuecomment-754188099,https://api.github.com/repos/simonw/datasette/issues/103,754188099,MDEyOklzc3VlQ29tbWVudDc1NDE4ODA5OQ==,9599,simonw,2021-01-04T20:05:14Z,2021-01-04T20:05:14Z,OWNER,"Wontfix, Cloud Run is already implemented and is a better fit for Datasette.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274265878,datasette publish appengine,
https://github.com/simonw/datasette/issues/913#issuecomment-754187520,https://api.github.com/repos/simonw/datasette/issues/913,754187520,MDEyOklzc3VlQ29tbWVudDc1NDE4NzUyMA==,9599,simonw,2021-01-04T20:04:10Z,2021-01-04T20:04:10Z,OWNER,That's pretty elegant: each plugin gets its own namespace and can register new settings.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/913#issuecomment-754187326,https://api.github.com/repos/simonw/datasette/issues/913,754187326,MDEyOklzc3VlQ29tbWVudDc1NDE4NzMyNg==,9599,simonw,2021-01-04T20:03:50Z,2021-01-04T20:03:50Z,OWNER,"I renamed `--config` to `--setting` and changed it to work like this:

    datasette --setting sql_time_limit_ms 1000

Note the lack of colons.

This actually makes colons cleaner to use for plugins - I could support this:

    datasette --setting datasette-insert:unsafe 1","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",670209331,Mechanism for passing additional options to `datasette my.db` that affect plugins,
https://github.com/simonw/datasette/issues/1111#issuecomment-754184287,https://api.github.com/repos/simonw/datasette/issues/1111,754184287,MDEyOklzc3VlQ29tbWVudDc1NDE4NDI4Nw==,9599,simonw,2021-01-04T19:57:53Z,2021-01-04T19:57:53Z,OWNER,Relevant new feature in sqlite-utils: the ability to use triggers to maintain fast counts. This optimization could help a lot here. https://github.com/simonw/sqlite-utils/issues/212,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",751195017,Accessing a database's `.json` is slow for very large SQLite files,
https://github.com/simonw/datasette/issues/1164#issuecomment-754182058,https://api.github.com/repos/simonw/datasette/issues/1164,754182058,MDEyOklzc3VlQ29tbWVudDc1NDE4MjA1OA==,9599,simonw,2021-01-04T19:53:31Z,2021-01-04T19:53:31Z,OWNER,This will be helped by the new `package.json` added in #1170.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/pull/1170#issuecomment-754181646,https://api.github.com/repos/simonw/datasette/issues/1170,754181646,MDEyOklzc3VlQ29tbWVudDc1NDE4MTY0Ng==,9599,simonw,2021-01-04T19:52:40Z,2021-01-04T19:52:40Z,OWNER,Thank you very much!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778126516,Install Prettier via package.json,
https://github.com/simonw/datasette/issues/983#issuecomment-754181647,https://api.github.com/repos/simonw/datasette/issues/983,754181647,MDEyOklzc3VlQ29tbWVudDc1NDE4MTY0Nw==,11941245,jussiarpalahti,2021-01-04T19:52:40Z,2021-01-04T19:52:40Z,NONE,"I was thinking JavaScript plugins going with server side template extensions custom HTML. Attach my own widgets on there and listen for Datasette events to refresh when user interacts with main UI. Like a map view or table that updates according to selected column. There's certainly other ways to look at this. Perhaps you could list possible hooks or high level design doc on what would be possible with the plugin system?

Re: modules. I would like to see modules supported at least in development. The developer experience is so much better than what JavaScript coding has been in the past. With large parts of NPM at your disposal I’d imagine even less experienced coder can whisk a custom plugin in no time. Proper production build system (like one you get with Pika or Parcel) could package everything up into bundles that older browsers can understand. Though that does come with performance and size penalties alongside the added complexity. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1169#issuecomment-754007242,https://api.github.com/repos/simonw/datasette/issues/1169,754007242,MDEyOklzc3VlQ29tbWVudDc1NDAwNzI0Mg==,3637,benpickles,2021-01-04T14:29:57Z,2021-01-04T14:29:57Z,CONTRIBUTOR,I somewhat share your reluctance to add a package.json to seemingly every project out there but ultimately if they're project dependencies it's important they're managed within the codebase.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/pull/1170#issuecomment-754004715,https://api.github.com/repos/simonw/datasette/issues/1170,754004715,MDEyOklzc3VlQ29tbWVudDc1NDAwNDcxNQ==,3637,benpickles,2021-01-04T14:25:44Z,2021-01-04T14:25:44Z,CONTRIBUTOR,I was going to re-add the filter to only run Prettier when there have been changes in `datasette/static` but that would mean it wouldn't run when the package is updated. That plus the fact that [the last run of the job took only 8 seconds](https://github.com/benpickles/datasette/runs/1640121514) is why I decided not to re-add the filter.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778126516,Install Prettier via package.json,
https://github.com/simonw/datasette/pull/1170#issuecomment-754002859,https://api.github.com/repos/simonw/datasette/issues/1170,754002859,MDEyOklzc3VlQ29tbWVudDc1NDAwMjg1OQ==,22429695,codecov[bot],2021-01-04T14:22:52Z,2021-01-04T14:22:52Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=h1) Report
> Merging [#1170](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=desc) (a5761cc) into [main](https://codecov.io/gh/simonw/datasette/commit/1e8fa3ac7cb2d6e516c47c306c86ed2334fc3dc0?el=desc) (1e8fa3a) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1170/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1170   +/-   ##
=======================================
  Coverage   91.55%   91.55%           
=======================================
  Files          32       32           
  Lines        3932     3932           
=======================================
  Hits         3600     3600           
  Misses        332      332           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=footer). Last update [1e8fa3a...a5761cc](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",778126516,Install Prettier via package.json,
https://github.com/simonw/datasette/issues/983#issuecomment-753690280,https://api.github.com/repos/simonw/datasette/issues/983,753690280,MDEyOklzc3VlQ29tbWVudDc1MzY5MDI4MA==,9599,simonw,2021-01-03T23:13:30Z,2021-01-03T23:13:30Z,OWNER,"Oh that's interesting, I hadn't thought about plugins firing events - just responding to events fired by the rest of the application.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671902,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753671902,MDEyOklzc3VlQ29tbWVudDc1MzY3MTkwMg==,9599,simonw,2021-01-03T20:31:04Z,2021-01-03T20:32:13Z,OWNER,A `table.has_count_triggers` property.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671235,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753671235,MDEyOklzc3VlQ29tbWVudDc1MzY3MTIzNQ==,9599,simonw,2021-01-03T20:25:10Z,2021-01-03T20:25:10Z,OWNER,"To detect tables, look at the names of the triggers - `{table}{counts_table}_insert` and `{table}{counts_table}_delete`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671009,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753671009,MDEyOklzc3VlQ29tbWVudDc1MzY3MTAwOQ==,9599,simonw,2021-01-03T20:22:53Z,2021-01-03T20:22:53Z,OWNER,I think this should be accompanied by a `sqlite-utils reset-counts` command.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753670833,https://api.github.com/repos/simonw/sqlite-utils/issues/219,753670833,MDEyOklzc3VlQ29tbWVudDc1MzY3MDgzMw==,9599,simonw,2021-01-03T20:20:54Z,2021-01-03T20:20:54Z,OWNER,"This is a little tricky. We should assume that the existing values in the `_counts` table cannot be trusted at all when this method is called - so we should probably clear that table entirely and then re-populate it.

But that means we need to figure out which tables in the database have the counts triggers defined.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777707544,reset_counts() method and command,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753668099,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753668099,MDEyOklzc3VlQ29tbWVudDc1MzY2ODA5OQ==,9599,simonw,2021-01-03T19:55:53Z,2021-01-03T19:55:53Z,OWNER,So if you instantiate the `Database()` constructor with `use_counts_table=True` any access to the `.count` properties will go through this table - otherwise regular `count(*)` queries will be executed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753665521,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753665521,MDEyOklzc3VlQ29tbWVudDc1MzY2NTUyMQ==,9599,simonw,2021-01-03T19:31:33Z,2021-01-03T19:31:33Z,OWNER,"I'm having second thoughts about this being the default behaviour. It's pretty weird. I feel like HUGE databases that need this are rare, so having it on by default doesn't make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753662490,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753662490,MDEyOklzc3VlQ29tbWVudDc1MzY2MjQ5MA==,9599,simonw,2021-01-03T19:05:53Z,2021-01-03T19:05:53Z,OWNER,Idea: a `.execute_count()` method that never uses the cache.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661292,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753661292,MDEyOklzc3VlQ29tbWVudDc1MzY2MTI5Mg==,9599,simonw,2021-01-03T18:56:06Z,2021-01-03T18:56:23Z,OWNER,"Another option: on creation of the `Database()` object, check to see if the `_counts` table exists and use that as the default for a `use_counts_table` property. Also flip that property to `True` if the user calls `.enable_counts()` at any time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661158,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753661158,MDEyOklzc3VlQ29tbWVudDc1MzY2MTE1OA==,9599,simonw,2021-01-03T18:55:16Z,2021-01-03T18:55:16Z,OWNER,"Alternative implementation: provided `db.should_trust_counts` is `True`, try running the query:
```sql
select count from _counts where [table] = ?
```
If the query fails to return a result OR throws an error because the table doesn't exist, run the `count(*)` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660814,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753660814,MDEyOklzc3VlQ29tbWVudDc1MzY2MDgxNA==,9599,simonw,2021-01-03T18:53:05Z,2021-01-03T18:53:05Z,OWNER,"Here's the current `.count` property: https://github.com/simonw/sqlite-utils/blob/036ec6d32313487527c66dea613a3e7118b97459/sqlite_utils/db.py#L597-L609

It's implemented on `Queryable` which means it's available on both `Table` and `View` - the optimization doesn't make sense for views.

I'm a bit cautious about making that property so much more complex. In order to decide if it should try the `_counts` table first it needs to know:

- Should it be trusting the counts? I'm thinking a `.should_trust_counts` property on `Database` which defaults to `True` would be good - then advanced users can turn that off if they know the counts should not be trusted.
- Does the `_counts` table exist?
- Are the triggers defined?

Then it can do the query, and if the query fails it can fall back on the `count(*)`. That's quite a lot of extra activity though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660379,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753660379,MDEyOklzc3VlQ29tbWVudDc1MzY2MDM3OQ==,9599,simonw,2021-01-03T18:50:15Z,2021-01-03T18:50:15Z,OWNER,"```python
    def cached_counts(self, tables=None):
        sql = ""select [table], count from {}"".format(self._counts_table_name)
        if tables:
            sql += "" where [table] in ({})"".format("", "".join(""?"" for table in tables))
        return {r[0]: r[1] for r in self.execute(sql, tables).fetchall()}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/206#issuecomment-753659260,https://api.github.com/repos/simonw/sqlite-utils/issues/206,753659260,MDEyOklzc3VlQ29tbWVudDc1MzY1OTI2MA==,9599,simonw,2021-01-03T18:42:01Z,2021-01-03T18:42:01Z,OWNER,"```
% sqlite-utils insert blah.db blah global_power_plant_database.csv
Error: Invalid JSON - use --csv for CSV or --tsv for TSV files
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",761915790,sqlite-utils should suggest --csv if JSON parsing fails,
https://github.com/simonw/datasette/issues/1169#issuecomment-753657180,https://api.github.com/repos/simonw/datasette/issues/1169,753657180,MDEyOklzc3VlQ29tbWVudDc1MzY1NzE4MA==,9599,simonw,2021-01-03T18:23:30Z,2021-01-03T18:23:30Z,OWNER,"Also welcome in that PR would be a bit of documentation for contributors, see #1167 - but no problem if you leave that out, I'm happy to add it later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/issues/1169#issuecomment-753653260,https://api.github.com/repos/simonw/datasette/issues/1169,753653260,MDEyOklzc3VlQ29tbWVudDc1MzY1MzI2MA==,9599,simonw,2021-01-03T17:54:40Z,2021-01-03T17:54:40Z,OWNER,And @benpickles yes I would land that pull request straight away as-is. Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/issues/1169#issuecomment-753653033,https://api.github.com/repos/simonw/datasette/issues/1169,753653033,MDEyOklzc3VlQ29tbWVudDc1MzY1MzAzMw==,9599,simonw,2021-01-03T17:52:53Z,2021-01-03T17:52:53Z,OWNER,"Oh that's so frustrating! I was worried about that - I spotted a few runs that seemed faster and hoped that it meant that the package was coming out of the `~/.npm` cache, but evidently that's not the case.

You've convinced me that Datasette itself should have a `package.json` - the Dependabot argument is a really good one.

But... I'd really love to figure out a general pattern for using `npx` scripts in GitHub Actions workflows in a cache-friendly way. I have plenty of other projects that I'd love to run Prettier or Uglify or `puppeteer-cli` in without adding a `package.json` to them.

Any ideas? The best I can think of is for the workflow itself to write out a `package.json` file (using `echo '{ ... }' > package.json`) as part of the run - that way the cache should work (I think) but I don't get a misleading `package.json` file sitting in the repo.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777677671,Prettier package not actually being cached,
https://github.com/simonw/datasette/issues/983#issuecomment-753600999,https://api.github.com/repos/simonw/datasette/issues/983,753600999,MDEyOklzc3VlQ29tbWVudDc1MzYwMDk5OQ==,475613,MarkusH,2021-01-03T11:11:21Z,2021-01-03T11:11:21Z,NONE,"With regards to JS/Browser events, given your example of menu items that plugins could add, I could imagine this code to work:

```js
// as part of datasette
datasette.events.AddMenuItem = 'DatasetteAddMenuItemEvent';
document.addEventListener(datasette.events.AddMenuItem, (e) => {
  // do whatever is needed to add the menu item. Data comes from `e`
  alert(e.title + ' ' + e.link);
});

// as part of a plugin
const event = new Event(datasette.events.AddMenuItem, {link: '/foo/bar', title: 'Go somewhere'});
Document.dispatchEvent(event)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753587963,https://api.github.com/repos/simonw/datasette/issues/983,753587963,MDEyOklzc3VlQ29tbWVudDc1MzU4Nzk2Mw==,154364,dracos,2021-01-03T09:02:50Z,2021-01-03T10:00:05Z,NONE,"> but I'm already commited to requiring support for () => {} arrow functions

Don't think you are :) (e.g. gzipped, using arrow functions in my example saves 2 bytes over spelling out function). On FMS, past month, looking at popular browsers, looks like we'd have 95.41% arrow support, 94.19% module support, and 4.58% (mostly IE9/IE11/Safari 9) supporting neither.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753570710,https://api.github.com/repos/simonw/datasette/issues/983,753570710,MDEyOklzc3VlQ29tbWVudDc1MzU3MDcxMA==,9599,simonw,2021-01-03T05:29:56Z,2021-01-03T05:29:56Z,OWNER,"I thought about using browser events, but they don't quite match the API that I'm looking to provide. In particular, the great thing about Pluggy is that if you have multiple handlers registered for a specific plugin hook each of those handlers can return a value, and Pluggy will combine those values into a list of replies.

This is great for things like plugin hooks that add extra menu items - each plugin can return a menu item (maybe as a label/URL/click-callback object) and the calling code can then add all of those items to the menu. See https://docs.datasette.io/en/stable/plugin_hooks.html#table-actions-datasette-actor-database-table for a Python example.

I'm on the fence about relying on JavaScript modules. I need to think about browser compatibility for them - but I'm already commited to requiring support for `() => {}` arrow functions so maybe I'm committed to module support too already?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1160#issuecomment-753568428,https://api.github.com/repos/simonw/datasette/issues/1160,753568428,MDEyOklzc3VlQ29tbWVudDc1MzU2ODQyOA==,9599,simonw,2021-01-03T05:02:32Z,2021-01-03T05:02:32Z,OWNER,"Should this command include a `--fts` option for configuring full-text search on one-or-more columns?

I thought about doing that for `sqlite-utils insert` in https://github.com/simonw/sqlite-utils/issues/202 and decided not to because of the need to include extra options covering the FTS version, porter stemming options and whether or not to create triggers.

But maybe I can set sensible defaults for that with `datasette insert ... -f title -f body`? Worth thinking about a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/sqlite-utils/issues/202#issuecomment-753568264,https://api.github.com/repos/simonw/sqlite-utils/issues/202,753568264,MDEyOklzc3VlQ29tbWVudDc1MzU2ODI2NA==,9599,simonw,2021-01-03T05:00:24Z,2021-01-03T05:00:24Z,OWNER,"I'm not going to implement this, because it actually needs several additional options that already exist on `sqlite-utils enable-fts`:
```
  --fts4                 Use FTS4
  --fts5                 Use FTS5
  --tokenize TEXT        Tokenizer to use, e.g. porter
  --create-triggers      Create triggers to update the FTS tables when the
                         parent table changes.
```
I'd rather not add all four of those options to `sqlite-utils insert` just to support this shortcut.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",738514367,sqlite-utils insert -f colname - for configuring full-text search,
https://github.com/simonw/sqlite-utils/issues/202#issuecomment-753567969,https://api.github.com/repos/simonw/sqlite-utils/issues/202,753567969,MDEyOklzc3VlQ29tbWVudDc1MzU2Nzk2OQ==,9599,simonw,2021-01-03T04:55:17Z,2021-01-03T04:55:43Z,OWNER,"The long version of this can be `--fts`, same as in `csvs-to-sqlite`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",738514367,sqlite-utils insert -f colname - for configuring full-text search,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567932,MDEyOklzc3VlQ29tbWVudDc1MzU2NzkzMg==,9599,simonw,2021-01-03T04:54:43Z,2021-01-03T04:54:43Z,OWNER,"Another option: expand the `ForeignKey` object to have `.columns` and `.other_columns` properties in addition to the existing `.column` and `.other_column` properties. These new plural properties would always return a tuple, which would be a one-item tuple for a non-compound-foreign-key.

The question then is what should `.column` and `.other_column` return for compound foreign keys?

I'd be inclined to say they should return `None` - which would trigger errors in code that encounters a compound foreign key for the first time, but those errors would at least be a strong indicator as to what had gone wrong.

We can label `.column` and `.other_column` as deprecated and then remove them in `sqlite-utils 4.0`.

Since this would still be a breaking change in some minor edge-cases I'm thinking maybe 4.0 needs to happen in order to land this feature. I'm not opposed to doing that, I was just hoping it might be avoidable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567744,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567744,MDEyOklzc3VlQ29tbWVudDc1MzU2Nzc0NA==,9599,simonw,2021-01-03T04:51:44Z,2021-01-03T04:51:44Z,OWNER,"One way that this could avoid a breaking change would be to have `fk.column` and `fk.other_column` remain as strings for non-compound-foreign-keys, but turn into tuples for a compound foreign key.

This is a bit of an ugly API design, and it could still break existing code that encounters a compound foreign key for the first time - but it would leave code working for the more common case of a non-compound-foreign-key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567508,https://api.github.com/repos/simonw/sqlite-utils/issues/203,753567508,MDEyOklzc3VlQ29tbWVudDc1MzU2NzUwOA==,9599,simonw,2021-01-03T04:48:17Z,2021-01-03T04:48:17Z,OWNER,"Sorry for taking so long to review this!

This approach looks great to me - being able to optionally pass a tuple anywhere the API currently expects a column is smart, and it's consistent with how the `pk=` parameter works elsewhere.

There's just one problem I can see with this: the way it changes the `ForeignKey(...)` interface to always return a tuple for `.column` and `.other_column`, even if that tuple only contains a single item.

This represents a breaking change to the existing API - any code that expects `ForeignKey.column` to be a single string (which is any code that has been written against that) will break.

As such, I'd have to bump the major version of `sqlite-utils` to `4.0` in order to ship this.

Ideally I'd like to make this change in a way that doesn't represent an API compatibility break. I need to think a bit harder about how that might be achieved.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753566184,https://api.github.com/repos/simonw/sqlite-utils/issues/217,753566184,MDEyOklzc3VlQ29tbWVudDc1MzU2NjE4NA==,9599,simonw,2021-01-03T04:27:38Z,2021-01-03T04:27:38Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-strings-for-use-in-sql,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777543336,Rename .escape() to .quote(),
https://github.com/simonw/sqlite-utils/issues/216#issuecomment-753566156,https://api.github.com/repos/simonw/sqlite-utils/issues/216,753566156,MDEyOklzc3VlQ29tbWVudDc1MzU2NjE1Ng==,9599,simonw,2021-01-03T04:27:14Z,2021-01-03T04:27:14Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#introspection,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777540352,database.triggers_dict introspection property,
https://github.com/simonw/sqlite-utils/issues/218#issuecomment-753563757,https://api.github.com/repos/simonw/sqlite-utils/issues/218,753563757,MDEyOklzc3VlQ29tbWVudDc1MzU2Mzc1Nw==,9599,simonw,2021-01-03T03:49:51Z,2021-01-03T03:49:51Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-triggers,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777560474,"""sqlite-utils triggers"" command",
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753545757,MDEyOklzc3VlQ29tbWVudDc1MzU0NTc1Nw==,9599,simonw,2021-01-02T23:58:07Z,2021-01-02T23:58:07Z,OWNER,"Thought: maybe there should be a `.reset_counts()` method too, for if the table gets out of date with the triggers.

One way that could happen is if a table is dropped and recreated - the counts in the `_counts` table would likely no longer match the number of rows in that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545381,https://api.github.com/repos/simonw/sqlite-utils/issues/215,753545381,MDEyOklzc3VlQ29tbWVudDc1MzU0NTM4MQ==,9599,simonw,2021-01-02T23:52:52Z,2021-01-02T23:52:52Z,OWNER,Idea: a `db.cached_counts()` method that returns a dictionary of data from the `_counts` table. Call it with a list of tables to get back the counts for just those tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777535402,Use _counts to speed up counts,
https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753544914,https://api.github.com/repos/simonw/sqlite-utils/issues/217,753544914,MDEyOklzc3VlQ29tbWVudDc1MzU0NDkxNA==,9599,simonw,2021-01-02T23:47:42Z,2021-01-02T23:47:42Z,OWNER,https://github.com/simonw/sqlite-utils/blob/9a5c92b63e7917c93cc502478493c51c781b2ecc/sqlite_utils/db.py#L231-L239,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777543336,Rename .escape() to .quote(),
https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753535488,https://api.github.com/repos/simonw/sqlite-utils/issues/213,753535488,MDEyOklzc3VlQ29tbWVudDc1MzUzNTQ4OA==,9599,simonw,2021-01-02T22:03:48Z,2021-01-02T22:03:48Z,OWNER,"I got this error while prototyping this:

    too many levels of trigger recursion

It looks like that's because SQLite doesn't like triggers on a table that themselves then update that table - so I'm going to exclude the `_counts` table from this mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777529979,db.enable_counts() method,
https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753533775,https://api.github.com/repos/simonw/sqlite-utils/issues/213,753533775,MDEyOklzc3VlQ29tbWVudDc1MzUzMzc3NQ==,9599,simonw,2021-01-02T21:47:10Z,2021-01-02T21:47:10Z,OWNER,"I'm going to skip virtual tables, which I can identify using this property: https://github.com/simonw/sqlite-utils/blob/1cad7fad3e7a5b734088f5cc545b69a055e636da/sqlite_utils/db.py#L720-L726","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777529979,db.enable_counts() method,
https://github.com/simonw/datasette/issues/1012#issuecomment-753531657,https://api.github.com/repos/simonw/datasette/issues/1012,753531657,MDEyOklzc3VlQ29tbWVudDc1MzUzMTY1Nw==,45380,bollwyvl,2021-01-02T21:25:36Z,2021-01-02T21:25:36Z,CONTRIBUTOR,"Actually, on more research, I found out this is handled by the [trove-classifiers package](https://github.com/pypa/trove-classifiers/blob/master/src/trove_classifiers/__init__.py#L2) now, so it's just a one-liner pr instead of fire-up-a-docker-container-and-do-some-migrations","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py,
https://github.com/simonw/datasette/issues/1168#issuecomment-753524779,https://api.github.com/repos/simonw/datasette/issues/1168,753524779,MDEyOklzc3VlQ29tbWVudDc1MzUyNDc3OQ==,9599,simonw,2021-01-02T20:19:26Z,2021-01-02T20:19:26Z,OWNER,Idea: version the metadata scheme. If the table is called `_metadata_v1` it gives me a clear path to designing a new scheme in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/sqlite-utils/issues/212#issuecomment-753422324,https://api.github.com/repos/simonw/sqlite-utils/issues/212,753422324,MDEyOklzc3VlQ29tbWVudDc1MzQyMjMyNA==,9599,simonw,2021-01-02T03:00:34Z,2021-01-02T03:00:34Z,OWNER,"Here's a prototype:
```python
with db.conn:
    db.conn.executescript(""""""
CREATE TABLE IF NOT EXISTS [_counts] ([table] TEXT PRIMARY KEY, [count] INTEGER DEFAULT 0);
CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ai] AFTER INSERT ON [Street_Tree_List] BEGIN
    INSERT OR REPLACE INTO _counts
        VALUES ('Street_Tree_List', COALESCE(
            (SELECT count FROM _counts
                WHERE [table]='Street_Tree_List'),
            0) + 1);
END;
CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ad] AFTER DELETE ON [Street_Tree_List] BEGIN
    INSERT OR REPLACE INTO _counts
        VALUES ('Street_Tree_List', COALESCE(
            (SELECT count FROM _counts
                WHERE [table]='Street_Tree_List'),
            0) - 1);
END;
INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', (select count(*) from [Street_Tree_List]));
"""""")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777392020,Mechanism for maintaining cache of table counts using triggers,
https://github.com/simonw/sqlite-utils/issues/210#issuecomment-753406744,https://api.github.com/repos/simonw/sqlite-utils/issues/210,753406744,MDEyOklzc3VlQ29tbWVudDc1MzQwNjc0NA==,9599,simonw,2021-01-02T00:02:39Z,2021-01-02T00:02:39Z,OWNER,"It looks like https://github.com/ofajardo/pyreadr is a good library for this.

I won't add this to `sqlite-utils` because it's quite a bulky dependency for a relatively small feature.

Normally I'd write a `rdata-to-sqlite` tool similar to https://pypi.org/project/dbf-to-sqlite/ - but I'm actually working on a new plugin hook for Datasette that might be an even better fit for this. The idea is to allow Datasette plugins to define input formats - such as RData - which would then result in being able to import them on the command-line with `datasette insert my.db file.rdata` or by uploading a file through the Datasette web interface.

That work is happening over here: https://github.com/simonw/datasette/issues/1160 - I'll close this issue in favour of a sometime-in-the-future `datasette-import-rdata` plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",767685961,Support of RData files,
https://github.com/simonw/sqlite-utils/issues/209#issuecomment-753405835,https://api.github.com/repos/simonw/sqlite-utils/issues/209,753405835,MDEyOklzc3VlQ29tbWVudDc1MzQwNTgzNQ==,9599,simonw,2021-01-01T23:52:06Z,2021-01-01T23:52:06Z,OWNER,I just hit this one too. Such a weird bug!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",766156875,Test failure with sqlite 3.34 in test_cli.py::test_optimize,
https://github.com/simonw/datasette/issues/1168#issuecomment-753402423,https://api.github.com/repos/simonw/datasette/issues/1168,753402423,MDEyOklzc3VlQ29tbWVudDc1MzQwMjQyMw==,9599,simonw,2021-01-01T23:16:05Z,2021-01-01T23:16:05Z,OWNER,"One catch: solving the ""show me all metadata for everything in this Datasette instance"" problem.

Ideally there would be a SQLite table that can be queried for this. But the need to resolve the potentially complex set of precedence rules means that table would be difficult if not impossible to provide at run-time.

Ideally a denormalized table would be available that featured the results of running those precedence rule calculations. But how to handle keeping this up-to-date? It would need to be recalculated any time a `_metadata` table in any of the attached databases had an update.

This is a much larger problem - but one potential fix would be to use triggers to maintain a ""version number"" for the `_metadata` table - similar to SQLite's own built-in `schema_version` mechanism. Triggers could increment a counter any time a record in that table was added, deleted or updated.

Such a mechanism would have applications outside of just this `_metadata` system. The ability to attach a version number to any table and have it automatically incremented when that table changes (via triggers) could help with all kinds of other Datasette-at-scale problems, including things like cached table counts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753401001,https://api.github.com/repos/simonw/datasette/issues/1168,753401001,MDEyOklzc3VlQ29tbWVudDc1MzQwMTAwMQ==,9599,simonw,2021-01-01T23:01:45Z,2021-01-01T23:01:45Z,OWNER,I need to prototype this. Could I do that as a plugin? I think so - I could try out the algorithm for loading metadata and display it on pages using some custom templates.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753400420,https://api.github.com/repos/simonw/datasette/issues/1168,753400420,MDEyOklzc3VlQ29tbWVudDc1MzQwMDQyMA==,9599,simonw,2021-01-01T22:53:58Z,2021-01-01T22:53:58Z,OWNER,"Precedence idea:
- First priority is non-_internal metadata from other databases - if those conflict then pick then the alphabetically-ordered-first database name wins
- Next priority: `_internal` metadata, which should have been loaded from `metadata.json`
- Last priority: the `_metadata` table from that database itself, i.e. the default ""baked in"" metadata","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753400306,https://api.github.com/repos/simonw/datasette/issues/1168,753400306,MDEyOklzc3VlQ29tbWVudDc1MzQwMDMwNg==,9599,simonw,2021-01-01T22:52:44Z,2021-01-01T22:52:44Z,OWNER,"Also: probably load column metadata as part of the table metadata rather than loading column metadata individually, since it's going to be rare to want the metadata for a single column rather than for an entire table full of columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753400265,https://api.github.com/repos/simonw/datasette/issues/1168,753400265,MDEyOklzc3VlQ29tbWVudDc1MzQwMDI2NQ==,9599,simonw,2021-01-01T22:52:09Z,2021-01-01T22:52:09Z,OWNER,"From an implementation perspective, I think the way this works is SQL queries read the relevant metadata from ALL available metadata tables, then Python code solves the precedence rules to produce the final, combined metadata for a database/table/column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753399635,https://api.github.com/repos/simonw/datasette/issues/1168,753399635,MDEyOklzc3VlQ29tbWVudDc1MzM5OTYzNQ==,9599,simonw,2021-01-01T22:45:21Z,2021-01-01T22:50:21Z,OWNER,"Would also need to figure out the precedence rules:

- What happens if the database has a `_metadata` table with data that conflicts with a remote metadata record from another database? I think the other database should win, because that allows plugins to over-ride the default metadata for something.
- Do JSON values get merged together? So if one table provides a description and another provides a title do both values get returned?
- If a database has a `license`, does that ""cascade"" down to the tables? What about `source` and `about`?
- What if there are two databases (or more) that provide conflicting metadata for a table in some other database? Also, `_internal` may have loaded data from `metadata.json` that conflicts with some other remote table metadata definition.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753399428,https://api.github.com/repos/simonw/datasette/issues/1168,753399428,MDEyOklzc3VlQ29tbWVudDc1MzM5OTQyOA==,9599,simonw,2021-01-01T22:43:14Z,2021-01-01T22:43:22Z,OWNER,"Could this use a compound primary key on `database, table, column`? Does that work with null values?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753399366,https://api.github.com/repos/simonw/datasette/issues/1168,753399366,MDEyOklzc3VlQ29tbWVudDc1MzM5OTM2Ng==,9599,simonw,2021-01-01T22:42:37Z,2021-01-01T22:42:37Z,OWNER,"So what would the database schema for this look like?

I'm leaning towards a single table called `_metadata`, because that's a neater fit for baking the metadata into the database file along with the data that it is describing. Alternatively I could have multiple tables sharing that prefix - `_metadata_database` and `_metadata_tables` and `_metadata_columns` perhaps.

If it's just a single `_metadata` table, the schema could look like this:

| database | table | column | metadata |
| --- | --- | --- | --- |
|  | mytable |  | {""title"": ""My Table"" } |
|  | mytable | mycolumn | {""description"": ""Column description"" } |
| otherdb | othertable | | {""description"": ""Table in another DB"" } |

If the `database` column is `null` it means ""this is describing a table in the same database file as this `_metadata` table"".

The alternative to the `metadata` JSON column would be separate columns for each potential metadata value - `license`, `source`, `about`, `about_url` etc. But that makes it harder for people to create custom metadata fields.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753398542,https://api.github.com/repos/simonw/datasette/issues/1168,753398542,MDEyOklzc3VlQ29tbWVudDc1MzM5ODU0Mg==,9599,simonw,2021-01-01T22:37:24Z,2021-01-01T22:37:24Z,OWNER,"The direction I'm leaning in now is the following:

- Metadata always lives in SQLite tables
- These tables can be co-located with the database they describe (same DB file)
- ... or they can be in a different DB file and reference the other database that they are describing
- Metadata provided on startup in a `metadata.json` file is loaded into an in-memory metadata table using that same mechanism

Plugins that want to provide metadata can do so by populating a table. They could even maintain their own in-memory database for this, or they could write to the `_internal` in-memory database, or they could write to a table in a database on disk.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753392102,https://api.github.com/repos/simonw/datasette/issues/1168,753392102,MDEyOklzc3VlQ29tbWVudDc1MzM5MjEwMg==,9599,simonw,2021-01-01T22:06:33Z,2021-01-01T22:06:33Z,OWNER,"Some SQLite databases include SQL comments in the schema definition which tell you what each column means:

```sql
CREATE TABLE User
        -- A table comment
(
        uid INTEGER,    -- A field comment
        flags INTEGER   -- Another field comment
);
```
The problem with these is that they're not exposed to SQLite in any mechanism other than parsing the `CREATE TABLE` statement from the `sqlite_master` table to extract those columns.

I had an idea to build a plugin that could return these. That would be easy with a ""get metadata for this column"" plugin hook - in the absence of one a plugin could still run that reads the schemas on startup and uses them to populate a metadata database table somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753391869,https://api.github.com/repos/simonw/datasette/issues/1168,753391869,MDEyOklzc3VlQ29tbWVudDc1MzM5MTg2OQ==,9599,simonw,2021-01-01T22:04:30Z,2021-01-01T22:04:30Z,OWNER,"The sticking point here seems to be the plugin hook. Allowing plugins to over-ride the way the question ""give me the metadata for this database/table/column"" is answered makes the database-backed metadata mechanisms much more complicated to think about.

What if plugins didn't get to over-ride metadata in this way, but could instead update the metadata in a persistent Datasette-managed storage mechanism?

Then maybe Datasette could do the following:

- Maintain metadata in `_internal` that has been loaded from `metadata.json`
- Know how to check a database for baked-in metadata (maybe in a `_metadata` table)
- Know how to fall back on the `_internal` metadata if no baked-in metadata is available

If database files were optionally allowed to store metadata about tables that live in another database file this could perhaps solve the plugin needs - since an ""edit metadata"" plugin would be able to edit records in a separate, dedicated `metadata.db` database to store new information about tables in other files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753390791,https://api.github.com/repos/simonw/datasette/issues/1168,753390791,MDEyOklzc3VlQ29tbWVudDc1MzM5MDc5MQ==,9599,simonw,2021-01-01T22:00:42Z,2021-01-01T22:00:42Z,OWNER,"Here are the requirements I'm currently trying to satisfy:

- It should be possible to query the metadata for ALL attached tables in one place, potentially with pagination and filtering
- Metadata should be able to exist in the current `metadata.json` file
- It should also be possible to bundle metadata in a table in the SQLite database files themselves
- Plugins should be able to define their own special mechanisms for metadata. This is particularly interesting for providing a UI that allows users to edit the metadata for their existing tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753390262,https://api.github.com/repos/simonw/datasette/issues/1168,753390262,MDEyOklzc3VlQ29tbWVudDc1MzM5MDI2Mg==,9599,simonw,2021-01-01T21:58:11Z,2021-01-01T21:58:11Z,OWNER,"One possibility: plugins could write directly to that in-memory database table. But how would they know to write again should the server restart? Maybe they would write to it once when called by the `startup` plugin hook, and then update it (and their own backing store) when metadata changes for some reason. Feels a bit messy though.

Also: if I want to support metadata optionally living in a `_metadata` table colocated with the data in a SQLite database file itself, how would that affect the `metadata` columns in `_internal`? How often would Datasette denormalize and copy data across from the on-disk `_metadata` tables to the `_internal` in-memory columns?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753389938,https://api.github.com/repos/simonw/datasette/issues/1168,753389938,MDEyOklzc3VlQ29tbWVudDc1MzM4OTkzOA==,9599,simonw,2021-01-01T21:54:15Z,2021-01-01T21:54:15Z,OWNER,"So what if the `databases`, `tables` and `columns` tables in `_internal` each grew a new `metadata` text column?

These columns could be populated by Datasette on startup through reading the `metadata.json` file. But how would plugins interact with them?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753389477,https://api.github.com/repos/simonw/datasette/issues/1168,753389477,MDEyOklzc3VlQ29tbWVudDc1MzM4OTQ3Nw==,9599,simonw,2021-01-01T21:49:57Z,2021-01-01T21:49:57Z,OWNER,"What if metadata was stored in a JSON text column in the existing `_internal` tables? This would allow for users to invent additional metadata fields in the future beyond the current `license`, `license_url` etc fields - without needing a schema change.

The downside of JSON columns generally is that they're harder to run indexed queries against. For metadata I don't think that matters - even with 10,000 tables each with their own metadata a SQL query asking for e.g. ""everything that has Apache 2 as the license"" would return in just a few ms.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753388809,https://api.github.com/repos/simonw/datasette/issues/1168,753388809,MDEyOklzc3VlQ29tbWVudDc1MzM4ODgwOQ==,9599,simonw,2021-01-01T21:47:51Z,2021-01-01T21:47:51Z,OWNER,"A database that exposes metadata will have the same restriction as the new `_internal` database that exposes columns and tables, in that it needs to take permissions into account. A user should not be able to view metadata for tables that they are not able to see.

As such, I'd rather bundle any metadata tables into the existing `_internal` database so I don't have to solve that permissions problem in two places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/1168#issuecomment-753366024,https://api.github.com/repos/simonw/datasette/issues/1168,753366024,MDEyOklzc3VlQ29tbWVudDc1MzM2NjAyNA==,9599,simonw,2021-01-01T18:48:34Z,2021-01-01T18:48:34Z,OWNER,Also: in #188 I proposed bundling metadata in the SQLite database itself alongside the data. This is a great way of ensuring metadata travels with the data when it is downloaded as a SQLite `.db` file. But how would that play with the idea of an in-memory `_metadata` table? Could that table perhaps offer views that join data across multiple attached physical databases?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/issues/983#issuecomment-753224999,https://api.github.com/repos/simonw/datasette/issues/983,753224999,MDEyOklzc3VlQ29tbWVudDc1MzIyNDk5OQ==,11941245,jussiarpalahti,2020-12-31T23:29:36Z,2020-12-31T23:29:36Z,NONE,"I have yet to build Datasette plugin and am unfamiliar with Pluggy. Since browsers have event handling builtin Datasette could communicate with plugins through it. Handlers register as listeners for custom Datasette events and Datasette's JS can then trigger said events.

I was also wondering if you had looked at Javascript Modules for JS plugins? With services like Skypack (https://www.skypack.dev) NPM libraries can be loaded directly into browser, no build step needed. Same goes for local JS if you adhere to ES Module spec. 

If minification is required then tools such as Snowpack (https://www.snowpack.dev) could fit better. It uses https://github.com/evanw/esbuild for bundling and minification.

On plugins you'd simply:

```javascript
import {register} from '/assets/js/datasette'
register.on({'click' : my_func})
```

In Datasette HTML pages' head you'd merely import these files as modules one by one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1166#issuecomment-753224351,https://api.github.com/repos/simonw/datasette/issues/1166,753224351,MDEyOklzc3VlQ29tbWVudDc1MzIyNDM1MQ==,9599,simonw,2020-12-31T23:23:29Z,2020-12-31T23:23:29Z,OWNER,I should configure the action to only run if changes have been made within the `datasette/static` directory.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/983#issuecomment-753221646,https://api.github.com/repos/simonw/datasette/issues/983,753221646,MDEyOklzc3VlQ29tbWVudDc1MzIyMTY0Ng==,9599,simonw,2020-12-31T22:58:47Z,2020-12-31T22:58:47Z,OWNER,"https://github.com/mishoo/UglifyJS/issues/1905#issuecomment-300485490 says:

> `sourceMappingURL` aren't added by default in `3.x` due to one of the feature requests not to - some users are putting them within HTTP response headers instead.
> 
> So the command line for that would be:
> 
> ```js
> $ uglifyjs main.js -cmo main.min.js --source-map url=main.min.js.map
> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1164#issuecomment-753221362,https://api.github.com/repos/simonw/datasette/issues/1164,753221362,MDEyOklzc3VlQ29tbWVudDc1MzIyMTM2Mg==,9599,simonw,2020-12-31T22:55:57Z,2020-12-31T22:55:57Z,OWNER,"I had to add this as the first line in `table.min.js` for the source mapping to work:
```
//# sourceMappingURL=/-/static/table.min.js.map
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-753220665,https://api.github.com/repos/simonw/datasette/issues/1164,753220665,MDEyOklzc3VlQ29tbWVudDc1MzIyMDY2NQ==,9599,simonw,2020-12-31T22:49:36Z,2020-12-31T22:49:36Z,OWNER,"I started with a 7K `table.js` file.

`npx uglifyjs table.js --source-map -o table.min.js` gave me a 5.6K `table.min.js` file. 

`npx uglifyjs table.js --source-map -o table.min.js --compress --mangle` gave me 4.5K.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-753220412,https://api.github.com/repos/simonw/datasette/issues/1164,753220412,MDEyOklzc3VlQ29tbWVudDc1MzIyMDQxMg==,9599,simonw,2020-12-31T22:47:36Z,2020-12-31T22:47:36Z,OWNER,"I'm trying to minify `table.js` and I ran into a problem:

    Uglification failed. Unexpected character '`'

It turns out `uglify-js` doesn't support ES6 syntax!

But `uglify-es` does:

    npm install uglify-es

Annoyingly it looks like `uglify-es` uses the same CLI command, `uglifyjs`. So after installing it this seemed to work:

    npx uglifyjs table.js --source-map -o table.min.js

I really don't like how `npx uglifyjs` could mean different things depending on which package was installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/983#issuecomment-753219521,https://api.github.com/repos/simonw/datasette/issues/983,753219521,MDEyOklzc3VlQ29tbWVudDc1MzIxOTUyMQ==,9599,simonw,2020-12-31T22:39:52Z,2020-12-31T22:39:52Z,OWNER,For inlining the `plugins.min.js` file into the Jinja templates I could use the trick described here: https://stackoverflow.com/a/41404611 - which adds a `{{ include_file('file.txt') }}` function to Jinja.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753219407,https://api.github.com/repos/simonw/datasette/issues/983,753219407,MDEyOklzc3VlQ29tbWVudDc1MzIxOTQwNw==,9599,simonw,2020-12-31T22:38:45Z,2020-12-31T22:39:10Z,OWNER,"You'll be able to add JavaScript plugins using a bunch of different mechanisms:

- In a custom template, dropping the code in to a `<script>` block
- A bookmarklet that injects an extra script (I'm really excited to try this out)
- A separate `script.js` file that's loaded into Datasette using the `""extra_js_urls""` metadata option, documented here: https://docs.datasette.io/en/stable/custom_templates.html#custom-css-and-javascript
- A plugin you can install, like `datasette-vega` or `datasette-cluster-map` - since plugins can bundle their own script files that then get loaded on pages via this hook: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette
","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753218817,https://api.github.com/repos/simonw/datasette/issues/983,753218817,MDEyOklzc3VlQ29tbWVudDc1MzIxODgxNw==,173848,yozlet,2020-12-31T22:32:25Z,2020-12-31T22:32:25Z,NONE,"Amazing work! And you've put in far more work than I'd expect to reduce the payload (which is admirable).

So, to add a plugin with the current design, it goes in (a) the template or (b) a bookmarklet, right?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753217917,https://api.github.com/repos/simonw/datasette/issues/983,753217917,MDEyOklzc3VlQ29tbWVudDc1MzIxNzkxNw==,9599,simonw,2020-12-31T22:23:29Z,2020-12-31T22:23:36Z,OWNER,"If I'm going to do that, it would be good if subsequent plugins that register against the `load` event are executed straight away. That's a bit of a weird edge-case in plugin world - it would involve the bulkier code that gets loaded redefining how `datasette.plugins.register` works to special-case the `'load'` hook.

Maybe the tiny bootstrap code could define a `datasette.plugins.onload(callbackFunction)` method which gets upgraded later into something that fires straight away? Would add more bytes though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753217714,https://api.github.com/repos/simonw/datasette/issues/983,753217714,MDEyOklzc3VlQ29tbWVudDc1MzIxNzcxNA==,9599,simonw,2020-12-31T22:21:33Z,2020-12-31T22:21:33Z,OWNER,"Eventually I'd like to provide a whole bunch of other `datasette.X` utility functions that plugins can use - things like `datasette.addTabbedContentPane()` or similar.

But I don't want to inline those into the page.

So... I think the basic plugin system remains inline - maybe from an inlined file called `plugins-bootstrap.js`. Then a separate `plugins.js` contains the rest of the API functionality.

If a plugin wants to take advantage of those APIs, maybe it registers itself using `datasette.plugins.register('load', () => ...)` - that `load` hook can then be fired once the bulkier plugin code has been loaded.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/987#issuecomment-753217127,https://api.github.com/repos/simonw/datasette/issues/987,753217127,MDEyOklzc3VlQ29tbWVudDc1MzIxNzEyNw==,9599,simonw,2020-12-31T22:16:46Z,2020-12-31T22:16:46Z,OWNER,"I'm going to use `class=""plugin-content-pre-table""` rather than `id=` - just because I still want to be able to display all of this stuff on the single https://latest.datasette.io/-/patterns page so duplicate IDs are best avoided.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/983#issuecomment-753215761,https://api.github.com/repos/simonw/datasette/issues/983,753215761,MDEyOklzc3VlQ29tbWVudDc1MzIxNTc2MQ==,9599,simonw,2020-12-31T22:07:31Z,2020-12-31T22:07:31Z,OWNER,"I think I need to keep the mechanism whereby a plugin can return `undefined` in order to indicate that it has nothing to say for that specific item - that's borrowed from Pluggy and I've used it a bunch in my Python plugins. That makes the code a bit longer.

I'll write some example plugins to help me decide if the filtering-out-of-undefined mechanism is needed or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-753215545,https://api.github.com/repos/simonw/datasette/issues/983,753215545,MDEyOklzc3VlQ29tbWVudDc1MzIxNTU0NQ==,9599,simonw,2020-12-31T22:05:41Z,2020-12-31T22:05:41Z,OWNER,Using object destructuring like that is a great idea. I'm going to play with your version  - it's delightfully succinct.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1166#issuecomment-753214664,https://api.github.com/repos/simonw/datasette/issues/1166,753214664,MDEyOklzc3VlQ29tbWVudDc1MzIxNDY2NA==,9599,simonw,2020-12-31T21:58:04Z,2020-12-31T21:58:04Z,OWNER,Wrote a TIL about this: https://til.simonwillison.net/github-actions/prettier-github-actions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753211535,https://api.github.com/repos/simonw/datasette/issues/1166,753211535,MDEyOklzc3VlQ29tbWVudDc1MzIxMTUzNQ==,9599,simonw,2020-12-31T21:46:04Z,2020-12-31T21:46:04Z,OWNER,"https://github.com/simonw/datasette/runs/1631682372?check_suite_focus=true failed!

<img width=""693"" alt=""Trying_out_bad_formatting__refs__1166_·_simonw_datasette_8087091"" src=""https://user-images.githubusercontent.com/9599/103426449-841e5c00-4b6e-11eb-95e6-26432a7b27dd.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753210536,https://api.github.com/repos/simonw/datasette/issues/1166,753210536,MDEyOklzc3VlQ29tbWVudDc1MzIxMDUzNg==,9599,simonw,2020-12-31T21:45:19Z,2020-12-31T21:45:19Z,OWNER,"Oops, committed that bad formatting test to `main` instead of a branch!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753209192,https://api.github.com/repos/simonw/datasette/issues/1166,753209192,MDEyOklzc3VlQ29tbWVudDc1MzIwOTE5Mg==,9599,simonw,2020-12-31T21:44:22Z,2020-12-31T21:44:22Z,OWNER,"Tests passed in https://github.com/simonw/datasette/runs/1631677726?check_suite_focus=true

I'm going to try submitting a pull request with badly formatted JavaScript to see if it gets caught.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753200580,https://api.github.com/repos/simonw/datasette/issues/1166,753200580,MDEyOklzc3VlQ29tbWVudDc1MzIwMDU4MA==,9599,simonw,2020-12-31T21:38:06Z,2020-12-31T21:38:06Z,OWNER,"I think this should work:
```
- uses: actions/cache@v2
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/prettier.yml' }}
```
I'll use the `prettier.yml` workflow that I'm about to create as the cache key for the NPM cache.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753197957,https://api.github.com/repos/simonw/datasette/issues/1166,753197957,MDEyOklzc3VlQ29tbWVudDc1MzE5Nzk1Nw==,9599,simonw,2020-12-31T21:36:14Z,2020-12-31T21:36:14Z,OWNER,"Maybe not that action actually - I wanted to use a pre-built action to avoid installing Prettier every time, but that's what it seems to do: https://github.com/creyD/prettier_action/blob/bb361e2979cff283ca7684908deac8f95400e779/entrypoint.sh#L28-L37","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753195905,https://api.github.com/repos/simonw/datasette/issues/1166,753195905,MDEyOklzc3VlQ29tbWVudDc1MzE5NTkwNQ==,9599,simonw,2020-12-31T21:34:46Z,2020-12-31T21:34:46Z,OWNER,This action looks good - tag 3.2 is equivalent to this commit hash: https://github.com/creyD/prettier_action/tree/bb361e2979cff283ca7684908deac8f95400e779,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1166#issuecomment-753193475,https://api.github.com/repos/simonw/datasette/issues/1166,753193475,MDEyOklzc3VlQ29tbWVudDc1MzE5MzQ3NQ==,9599,simonw,2020-12-31T21:33:00Z,2020-12-31T21:33:00Z,OWNER,"I want a CI check that confirms that files conform to prettier - but only `datasette/static/*.js` files that are not already  minified.

This seems to do the job:

    npx prettier --check 'datasette/static/*[!.min].js'
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777140799,Adopt Prettier for JavaScript code formatting,
https://github.com/simonw/datasette/issues/1165#issuecomment-753033121,https://api.github.com/repos/simonw/datasette/issues/1165,753033121,MDEyOklzc3VlQ29tbWVudDc1MzAzMzEyMQ==,154364,dracos,2020-12-31T19:33:47Z,2020-12-31T19:33:47Z,NONE,"Sorry to go on about it, but it's my only example ;) And thought it might be of interest/use. Here is FixMyStreet's Cypress workflow https://github.com/mysociety/fixmystreet/blob/master/.github/workflows/cypress.yml with the master script that sets up server etc at https://github.com/mysociety/fixmystreet/blob/master/bin/browser-tests (that has features such as working inside/outside Vagrant, and can do JS code coverage) and then the tests are at https://github.com/mysociety/fixmystreet/tree/master/.cypress/cypress/integration","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/983#issuecomment-752882797,https://api.github.com/repos/simonw/datasette/issues/983,752882797,MDEyOklzc3VlQ29tbWVudDc1Mjg4Mjc5Nw==,154364,dracos,2020-12-31T08:07:59Z,2020-12-31T15:04:32Z,NONE,"If you're using arrow functions, you can presumably use default parameters, not much difference in support. That would save you 9 bytes. But OTOH you need `""use strict"";` to use arrow functions etc, and that's 13 bytes.

Your latest 250-byte one, with use strict, gzips to 199 bytes. The following might be 292 bytes, but compresses to 204, basically the same, and works in any browser (well, IE9+) at all:

`var datasette=datasette||{};datasette.plugins=function(){var d={};return{register:function(b,c,e){d[b]||(d[b]=[]);d[b].push([c,e])},call:function(b,c){c=c||{};var e=[];(d[b]||[]).forEach(function(a){a=a[0].apply(a[0],a[1].map(function(a){return c[a]}));void 0!==a&&e.push(a)});return e}}}();`

Source for that is below; I replaced the [fn,parameters] because closure-compiler includes a polyfill for that, and I ran `closure-compiler --language_out ECMASCRIPT3`:

```js
var datasette = datasette || {};
datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach((data) => {
                /* Call with the correct arguments */
                var result = data[0].apply(data[0], data[1].map(parameter => args[parameter]));
                if (result !== undefined) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752888552,https://api.github.com/repos/simonw/datasette/issues/983,752888552,MDEyOklzc3VlQ29tbWVudDc1Mjg4ODU1Mg==,154364,dracos,2020-12-31T08:33:11Z,2020-12-31T08:34:27Z,NONE,"If you could say that all hook functions had to accept one options parameter (and could use object destructuring if they wished to only see a subset), you could have this, which minifies (to all-browser-JS) to 200 bytes, gzips to 146, and works practically the same:

```js
var datasette = datasette || {};
datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn) => {
            registry[hook] = registry[hook] || [];
            registry[hook].push(fn);
        },
        call: (hook, args) => {
            var results = (registry[hook] || []).map(fn => fn(args||{}));
            return results;
        }
    };
})();
```

`var datasette=datasette||{};datasette.plugins=function(){var b={};return{register:function(a,c){b[a]=b[a]||[];b[a].push(c)},call:function(a,c){return(b[a]||[]).map(function(a){return a(c||{})})}}}();`

Called the same, definitions tiny bit different:

```js
datasette.plugins.register('numbers', ({a, b}) => a + b)
datasette.plugins.register('numbers', o => o.a * o.b)
datasette.plugins.call('numbers', {a: 4, b: 6})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1165#issuecomment-752846267,https://api.github.com/repos/simonw/datasette/issues/1165,752846267,MDEyOklzc3VlQ29tbWVudDc1Mjg0NjI2Nw==,9599,simonw,2020-12-31T05:10:41Z,2020-12-31T05:13:14Z,OWNER,"https://github.com/PostHog/posthog/tree/master/cypress/integration has some useful examples, linked from this article: https://posthog.com/blog/cypress-end-to-end-tests

Also useful: their workflow https://github.com/PostHog/posthog/blob/master/.github/workflows/e2e.yml","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752839433,https://api.github.com/repos/simonw/datasette/issues/1165,752839433,MDEyOklzc3VlQ29tbWVudDc1MjgzOTQzMw==,9599,simonw,2020-12-31T04:29:40Z,2020-12-31T04:29:40Z,OWNER,Important to absorb the slightly bizarre assertion syntax from Chai - docs here https://www.chaijs.com/api/bdd/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752828851,https://api.github.com/repos/simonw/datasette/issues/1165,752828851,MDEyOklzc3VlQ29tbWVudDc1MjgyODg1MQ==,9599,simonw,2020-12-31T03:19:38Z,2020-12-31T03:19:38Z,OWNER,"I got Cypress working! I added the `datasette.plugins` code to the table template and ran a test called `plugins.spec.js` using the following:
```javascript
context('datasette.plugins API', () => {
    beforeEach(() => {
      cy.visit('/fixtures/compound_three_primary_keys')
    });
    it('should exist', () => {
        let datasette;
        cy.window().then(win => {
            datasette = win.datasette;
        }).then(() => {
            expect(datasette).to.exist;
            expect(datasette.plugins).to.exist;
        });
    });
    it('should register and execute plugins', () => {
        let datasette;
        cy.window().then(win => {
            datasette = win.datasette;
        }).then(() => {
            expect(datasette.plugins.call('numbers')).to.deep.equal([]);
            // Register a plugin
            datasette.plugins.register(""numbers"", (a, b) => a + b, ['a', 'b']);
            var result = datasette.plugins.call(""numbers"", {a: 1, b: 2});
            expect(result).to.deep.equal([3]);
            // Second plugin
            datasette.plugins.register(""numbers"", (a, b) => a * b, ['a', 'b']);
            var result2 = datasette.plugins.call(""numbers"", {a: 1, b: 2});
            expect(result2).to.deep.equal([3, 2]);
        });
    });
});
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752780000,https://api.github.com/repos/simonw/datasette/issues/1165,752780000,MDEyOklzc3VlQ29tbWVudDc1Mjc4MDAwMA==,9599,simonw,2020-12-30T22:41:25Z,2020-12-30T22:41:25Z,OWNER,Jest works with Puppeteer: https://jestjs.io/docs/en/puppeteer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752779820,https://api.github.com/repos/simonw/datasette/issues/1165,752779820,MDEyOklzc3VlQ29tbWVudDc1Mjc3OTgyMA==,9599,simonw,2020-12-30T22:40:28Z,2020-12-30T22:40:28Z,OWNER,"I don't know if Jest on the command-line is the right tool for this. It works for the `plugins.js` script but I'm increasingly going to want to start adding tests for browser JavaScript features - like the https://github.com/simonw/datasette/blob/0.53/datasette/static/table.js script - which will need to run in a browser.

So maybe I should just find a browser testing solution and figure out how to run that under CI in GitHub Actions. Maybe https://www.cypress.io/ ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752779490,https://api.github.com/repos/simonw/datasette/issues/1165,752779490,MDEyOklzc3VlQ29tbWVudDc1Mjc3OTQ5MA==,9599,simonw,2020-12-30T22:38:43Z,2020-12-30T22:38:43Z,OWNER,Turned that into a TIL: https://til.simonwillison.net/javascript/jest-without-package-json,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/1165#issuecomment-752777744,https://api.github.com/repos/simonw/datasette/issues/1165,752777744,MDEyOklzc3VlQ29tbWVudDc1Mjc3Nzc0NA==,9599,simonw,2020-12-30T22:30:24Z,2020-12-30T22:30:24Z,OWNER,"https://www.valentinog.com/blog/jest/ was useful.

I created a `static/__tests__` folder and added this file as `plugins.spec.js`:

```javascript
const datasette = require(""../plugins.js"");

describe(""Datasette Plugins"", () => {
  test(""it should have datasette.plugins"", () => {
    expect(!!datasette.plugins).toEqual(true);
  });
  test(""registering a plugin should work"", () => {
    datasette.plugins.register(""numbers"", (a, b) => a + b, [""a"", ""b""]);
    var result = datasette.plugins.call(""numbers"", { a: 1, b: 2 });
    expect(result).toEqual([3]);
    datasette.plugins.register(""numbers"", (a, b) => a * b, [""a"", ""b""]);
    var result2 = datasette.plugins.call(""numbers"", { a: 1, b: 2 });
    expect(result2).toEqual([3, 2]);
  });
});
```

In `static/plugins.js` I put this:
```javascript
var datasette = datasette || {};
datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var result = fn.apply(fn, parameters.map(parameter => args[parameter]));
                if (result !== undefined) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();

module.exports = datasette;
```
Note the `module.exports` line at the end.

Then inside `static/` I ran the following command:

```
% npx jest -c '{}'
 PASS  __tests__/plugins.spec.js
  Datasette Plugins
    ✓ it should have datasette.plugins (3 ms)
    ✓ registering a plugin should work (1 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        1.163 s
Ran all test suites.
```
The `-c {}` was necessary because I didn't have a Jest configuration or a `package.json`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/983#issuecomment-752773508,https://api.github.com/repos/simonw/datasette/issues/983,752773508,MDEyOklzc3VlQ29tbWVudDc1Mjc3MzUwOA==,9599,simonw,2020-12-30T22:10:08Z,2020-12-30T22:11:34Z,OWNER,"https://twitter.com/dracos/status/1344402639476424706 points out that plugins returning 0 will be ignored.

This should probably check for `result !== undefined` instead - knocks the size up to 250.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752770488,https://api.github.com/repos/simonw/datasette/issues/983,752770488,MDEyOklzc3VlQ29tbWVudDc1Mjc3MDQ4OA==,9599,simonw,2020-12-30T21:55:35Z,2020-12-30T21:58:26Z,OWNER,"This one minifies to 241:
```javascript
var datasette = datasette || {};
datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var result = fn.apply(fn, parameters.map(parameter => args[parameter]));
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();
```
`var datasette=datasette||{};datasette.plugins=(()=>{var a={};return{register:(t,r,e)=>{a[t]||(a[t]=[]),a[t].push([r,e])},call:(t,r)=>{r=r||{};var e=[];return(a[t]||[]).forEach(([a,t])=>{var s=a.apply(a,t.map(a=>r[a]));s&&e.push(s)}),e}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752770133,https://api.github.com/repos/simonw/datasette/issues/983,752770133,MDEyOklzc3VlQ29tbWVudDc1Mjc3MDEzMw==,9599,simonw,2020-12-30T21:53:45Z,2020-12-30T21:54:22Z,OWNER,"FixMyStreet inlines some JavaScript, and it's always a good idea to copy what they're doing when it comes to web performance: https://github.com/mysociety/fixmystreet/blob/23e9564b58a86b783ce47f3c0bf837cbd4fe7282/templates/web/base/common_header_tags.html#L19-L25

Note `var fixmystreet=fixmystreet||{};` which is shorter - https://twitter.com/dracos/status/1344399909794045954","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1164#issuecomment-752769452,https://api.github.com/repos/simonw/datasette/issues/1164,752769452,MDEyOklzc3VlQ29tbWVudDc1Mjc2OTQ1Mg==,9599,simonw,2020-12-30T21:50:16Z,2020-12-30T21:50:16Z,OWNER,If I implement this I can automate the CodeMirror minification and remove the bit about running `uglify-js` against it from the documentation here: https://docs.datasette.io/en/0.53/contributing.html#upgrading-codemirror,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-752768785,https://api.github.com/repos/simonw/datasette/issues/1164,752768785,MDEyOklzc3VlQ29tbWVudDc1Mjc2ODc4NQ==,9599,simonw,2020-12-30T21:47:06Z,2020-12-30T21:47:06Z,OWNER,If I'm going to minify `table.js` I'd like to offer a source map for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-752768652,https://api.github.com/repos/simonw/datasette/issues/1164,752768652,MDEyOklzc3VlQ29tbWVudDc1Mjc2ODY1Mg==,9599,simonw,2020-12-30T21:46:29Z,2020-12-30T21:46:29Z,OWNER,Running https://skalman.github.io/UglifyJS-online/ against https://github.com/simonw/datasette/blob/0.53/datasette/static/table.js knocks it down from 7810 characters to 4643.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/983#issuecomment-752767500,https://api.github.com/repos/simonw/datasette/issues/983,752767500,MDEyOklzc3VlQ29tbWVudDc1Mjc2NzUwMA==,9599,simonw,2020-12-30T21:42:07Z,2020-12-30T21:42:07Z,OWNER,"Another option: have both ""dev"" and ""production"" versions of the plugin mechanism script. Make it easy to switch between the two. Build JavaScript unit tests that exercise the ""production"" APIs against the development version, and have extra tests that just work against the features in the development version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752767174,https://api.github.com/repos/simonw/datasette/issues/983,752767174,MDEyOklzc3VlQ29tbWVudDc1Mjc2NzE3NA==,9599,simonw,2020-12-30T21:40:44Z,2020-12-30T21:40:44Z,OWNER,Started a Twitter thread about this here: https://twitter.com/simonw/status/1344392603794477056,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752751490,https://api.github.com/repos/simonw/datasette/issues/983,752751490,MDEyOklzc3VlQ29tbWVudDc1Mjc1MTQ5MA==,9599,simonw,2020-12-30T20:40:04Z,2020-12-30T21:34:22Z,OWNER,"This one is 683 bytes with Uglify - I like how https://skalman.github.io/UglifyJS-online/ shows you the minified character count as you edit the script:
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};
    var definitions = {};
    var stringify = JSON.stringify;

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function isSubSet(a, b) {
        return a.every(parameter => b.includes(parameter))
    }

    return {
        _r: registry,
        define: (hook, parameters) => {
            definitions[hook] = parameters || [];
        },
        register: (hook, fn, parameters) => {
            parameters = parameters || extractParameters(fn);
            if (!definitions[hook]) {
                throw 'Hook ""' + hook + '"" not defined';
            }
            /* Check parameters is a subset of definitions[hook] */
            var validParameters = definitions[hook];
            if (!isSubSet(parameters, validParameters)) {
                throw '""' + hook + '"" valid args: ' + stringify(validParameters);
            }
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        
        call: (hook, args) => {
            args = args || {};
            if (!definitions[hook]) {
                throw '""' + hook + '"" hook not defined';
            }
            if (!isSubSet(Object.keys(args), definitions[hook])) {
                throw '""' + hook + '"" valid args: ' + stringify(definitions[hook]);
            }
            
            var implementations = registry[hook] || [];
            var results = [];
            implementations.forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }       
    };
})();
```
`window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={},r={},e=JSON.stringify;function i(t,r){return t.every(t=>r.includes(t))}return{_r:t,define:(t,e)=>{r[t]=e||[]},register:(a,n,o)=>{if(o=o||function(t){var r=/\((.*)\)/.exec(t.toString());return r&&r[1].trim()?r[1].split("","").map(t=>t.trim()):[]}(n),!r[a])throw'Hook ""'+a+'"" not defined';var d=r[a];if(!i(o,d))throw'""'+a+'"" valid args: '+e(d);t[a]||(t[a]=[]),t[a].push([n,o])},call:(a,n)=>{if(n=n||{},!r[a])throw'""'+a+'"" hook not defined';if(!i(Object.keys(n),r[a]))throw'""'+a+'"" valid args: '+e(r[a]);var o=t[a]||[],d=[];return o.forEach(([t,r])=>{var e=r.map(t=>n[t]),i=t.apply(t,e);i&&d.push(i)}),d}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752760815,https://api.github.com/repos/simonw/datasette/issues/983,752760815,MDEyOklzc3VlQ29tbWVudDc1Mjc2MDgxNQ==,9599,simonw,2020-12-30T21:15:41Z,2020-12-30T21:15:41Z,OWNER,"I'm going to write a few example plugins and try them out against the longer and shorter versions of the script, to get a better feel for how useful the longer versions with the error handling and explicit definition actually are.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752760054,https://api.github.com/repos/simonw/datasette/issues/983,752760054,MDEyOklzc3VlQ29tbWVudDc1Mjc2MDA1NA==,9599,simonw,2020-12-30T21:12:36Z,2020-12-30T21:14:05Z,OWNER,"I gotta admit that 262 byte version is pretty tempting, if it's going to end up in the `<head>` of every single page.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752759885,https://api.github.com/repos/simonw/datasette/issues/983,752759885,MDEyOklzc3VlQ29tbWVudDc1Mjc1OTg4NQ==,9599,simonw,2020-12-30T21:11:52Z,2020-12-30T21:14:00Z,OWNER,"262 bytes if I remove the parameter introspection code, instead requiring plugin authors to specify the arguments they take:
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();
```
`window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var a={};return{register:(t,e,r)=>{a[t]||(a[t]=[]),a[t].push([e,r])},call:(t,e)=>{e=e||{};var r=[];return(a[t]||[]).forEach(([a,t])=>{var s=t.map(a=>e[a]),d=a.apply(a,s);d&&r.push(d)}),r}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752758802,https://api.github.com/repos/simonw/datasette/issues/983,752758802,MDEyOklzc3VlQ29tbWVudDc1Mjc1ODgwMg==,9599,simonw,2020-12-30T21:07:33Z,2020-12-30T21:10:10Z,OWNER,"Removing the `datasette.plugin.define()` method and associated error handling reduces the uglified version from 683 bytes to 380 bytes. I think the error checking is worth the extra 303 bytes per page load, even if it's only really needed for a better developer experience.
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }
    return {
        register: (hook, fn, parameters) => {
            parameters = parameters || extractParameters(fn);
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        
        call: (hook, args) => {
            args = args || {};
            var implementations = registry[hook] || [];
            var results = [];
            implementations.forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();
```
`window.datasette=window.datasette||{},window.datasette.plugins=(()=>{var t={};return{register:(r,a,e)=>{e=e||function(t){var r=/\((.*)\)/.exec(t.toString());return r&&r[1].trim()?r[1].split("","").map(t=>t.trim()):[]}(a),t[r]||(t[r]=[]),t[r].push([a,e])},call:(r,a)=>{a=a||{};var e=t[r]||[],i=[];return e.forEach(([t,r])=>{var e=r.map(t=>a[t]),n=t.apply(t,e);n&&i.push(n)}),i}}})();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1165#issuecomment-752757910,https://api.github.com/repos/simonw/datasette/issues/1165,752757910,MDEyOklzc3VlQ29tbWVudDc1Mjc1NzkxMA==,9599,simonw,2020-12-30T21:04:18Z,2020-12-30T21:04:18Z,OWNER,https://jestjs.io/ looks worth trying here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776635426,Mechanism for executing JavaScript unit tests,
https://github.com/simonw/datasette/issues/983#issuecomment-752757289,https://api.github.com/repos/simonw/datasette/issues/983,752757289,MDEyOklzc3VlQ29tbWVudDc1Mjc1NzI4OQ==,9599,simonw,2020-12-30T21:02:20Z,2020-12-30T21:02:20Z,OWNER,I'm going to need to add JavaScript unit tests for this new plugin system.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/1164#issuecomment-752757075,https://api.github.com/repos/simonw/datasette/issues/1164,752757075,MDEyOklzc3VlQ29tbWVudDc1Mjc1NzA3NQ==,9599,simonw,2020-12-30T21:01:27Z,2020-12-30T21:01:27Z,OWNER,"I don't want Datasette contributors to need a working Node.js install to run the tests or work on Datasette unless they are explicitly working on the JavaScript.

I think I'm going to do this with a unit test that runs only if `upglify-js` is available on the path and confirms that the `*.min.js` version of each script in the repository correctly matches the results from running `uglify-js` against it.

That way if anyone checks in a change to JavaScript but forgets to run the minifier the tests will fail in CI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/1164#issuecomment-752756612,https://api.github.com/repos/simonw/datasette/issues/1164,752756612,MDEyOklzc3VlQ29tbWVudDc1Mjc1NjYxMg==,9599,simonw,2020-12-30T20:59:54Z,2020-12-30T20:59:54Z,OWNER,"I tried a few different pure-Python JavaScript minifying libraries and none of them produced results as good as https://www.npmjs.com/package/uglify-js for the plugin code I'm considering in #983.

So I think I'll need to rely on a Node.js tool for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776634318,Mechanism for minifying JavaScript that ships with Datasette,
https://github.com/simonw/datasette/issues/983#issuecomment-752750551,https://api.github.com/repos/simonw/datasette/issues/983,752750551,MDEyOklzc3VlQ29tbWVudDc1Mjc1MDU1MQ==,9599,simonw,2020-12-30T20:36:38Z,2020-12-30T20:37:48Z,OWNER,"This version minifies to 702 characters:
```javascript
window.datasette = window.datasette || {};
window.datasette.plugins = (() => {
    var registry = {};
    var definitions = {};
    var stringify = JSON.stringify;

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function isSubSet(a, b) {
        return a.every(parameter => b.includes(parameter))
    }

    return {
        _registry: registry,
        define: (hook, parameters) => {
            definitions[hook] = parameters || [];
        },
        register: (hook, fn, parameters) => {
            parameters = parameters || extractParameters(fn);
            if (!definitions[hook]) {
                throw '""' + hook + '"" is not a defined hook';
            }
            /* Check parameters is a subset of definitions[hook] */
            var validParameters = definitions[hook];
            if (!isSubSet(parameters, validParameters)) {
                throw '""' + hook + '"" valid args are ' + stringify(validParameters);
            }
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        
        call: (hook, args) => {
            args = args || {};
            if (!definitions[hook]) {
                throw '""' + hook + '"" hook is not defined';
            }
            if (!isSubSet(Object.keys(args), definitions[hook])) {
                throw '""' + hook + '"" valid args: ' + stringify(definitions[hook]);
            }
            
            var implementations = registry[hook] || [];
            var results = [];
            implementations.forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var callWith = parameters.map(parameter => args[parameter]);
                var result = fn.apply(fn, callWith);
                if (result) {
                    results.push(result);
                }
            });
            return results;
        }       
    };
})();
```
Or 701 characters using https://skalman.github.io/UglifyJS-online/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752749189,https://api.github.com/repos/simonw/datasette/issues/983,752749189,MDEyOklzc3VlQ29tbWVudDc1Mjc0OTE4OQ==,9599,simonw,2020-12-30T20:31:28Z,2020-12-30T20:31:28Z,OWNER,"Using raw string exceptions, `throw '""' + hook + '"" hook has not been defined';`, knocks it down to 795 characters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752748496,https://api.github.com/repos/simonw/datasette/issues/983,752748496,MDEyOklzc3VlQ29tbWVudDc1Mjc0ODQ5Ng==,9599,simonw,2020-12-30T20:28:48Z,2020-12-30T20:28:48Z,OWNER,If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752747999,https://api.github.com/repos/simonw/datasette/issues/983,752747999,MDEyOklzc3VlQ29tbWVudDc1Mjc0Nzk5OQ==,9599,simonw,2020-12-30T20:27:00Z,2020-12-30T20:27:00Z,OWNER,"I need to decide how this code is going to be loaded. Putting it in a blocking `<script>` element in the head would work, but I'd rather not block loading of the rest of the page. Using a `<script async>` method would be nicer, but then I have to worry about plugins attempting to register themselves before the page has fully loaded.

Running it through https://javascript-minifier.com/ produces this, which is 855 characters - so maybe I could inline that into the header of the page?

`window.datasette={},window.datasette.plugins=function(){var r={},n={};function e(r,n){return r.every(r=>n.includes(r))}return{define:function(r,e){n[r]=e||[]},register:function(t,i,o){if(o=o||function(r){var n=/\((.*)\)/.exec(r.toString());return n&&n[1].trim()?n[1].split("","").map(r=>r.trim()):[]}(i),!n[t])throw new Error('""'+t+'"" is not a defined plugin hook');if(!n[t])throw new Error('""'+t+'"" is not a defined plugin hook');var a=n[t];if(!e(o,a))throw new Error('""'+t+'"" valid parameters are '+JSON.stringify(a));r[t]||(r[t]=[]),r[t].push([i,o])},_registry:r,call:function(t,i){if(i=i||{},!n[t])throw new Error('""'+t+'"" hook has not been defined');if(!e(Object.keys(i),n[t]))throw new Error('""'+t+'"" valid arguments are '+JSON.stringify(n[t]));var o=r[t]||[],a=[];return o.forEach(([r,n])=>{var e=n.map(r=>i[r]),t=r.apply(r,e);t&&a.push(t)}),a}}}();`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752747169,https://api.github.com/repos/simonw/datasette/issues/983,752747169,MDEyOklzc3VlQ29tbWVudDc1Mjc0NzE2OQ==,9599,simonw,2020-12-30T20:24:07Z,2020-12-30T20:24:07Z,OWNER,"This version adds `datasette.plugins.define()` plus extra validation of both `.register()` and `.call()`:
```javascript
window.datasette = {};
window.datasette.plugins = (function() {
    var registry = {};
    var definitions = {};

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function define(hook, parameters) {
        definitions[hook] = parameters || [];
    }

    function isSubSet(a, b) {
        return a.every(parameter => b.includes(parameter))
    }

    function register(hook, fn, parameters) {
        parameters = parameters || extractParameters(fn);
        if (!definitions[hook]) {
            throw new Error('""' + hook + '"" is not a defined plugin hook');
        }
        if (!definitions[hook]) {
            throw new Error('""' + hook + '"" is not a defined plugin hook');
        }
        /* Check parameters is a subset of definitions[hook] */
        var validParameters = definitions[hook];
        if (!isSubSet(parameters, validParameters)) {
            throw new Error('""' + hook + '"" valid parameters are ' + JSON.stringify(validParameters));
        }
        if (!registry[hook]) {
            registry[hook] = [];
        }
        registry[hook].push([fn, parameters]);
    }

    function call(hook, args) {
        args = args || {};
        if (!definitions[hook]) {
            throw new Error('""' + hook + '"" hook has not been defined');
        }
        if (!isSubSet(Object.keys(args), definitions[hook])) {
            throw new Error('""' + hook + '"" valid arguments are ' + JSON.stringify(definitions[hook]));
        }
        
        var implementations = registry[hook] || [];
        var results = [];
        implementations.forEach(([fn, parameters]) => {
            /* Call with the correct arguments */
            var callWith = parameters.map(parameter => args[parameter]);
            var result = fn.apply(fn, callWith);
            if (result) {
                results.push(result);
            }
        });
        return results;
    }
    return {
        define: define,
        register: register,
        _registry: registry,
        call: call
    };
})();
```
Usage:
```javascript
datasette.plugins.define('numbers', ['a', 'b'])
datasette.plugins.register('numbers', (a, b) => a + b)
datasette.plugins.register('numbers', (a, b) => a * b)
datasette.plugins.call('numbers', {a: 4, b: 6})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752744311,https://api.github.com/repos/simonw/datasette/issues/983,752744311,MDEyOklzc3VlQ29tbWVudDc1Mjc0NDMxMQ==,9599,simonw,2020-12-30T20:12:50Z,2020-12-30T20:13:02Z,OWNER,"This could work to define a plugin hook:
```javascript
datasette.plugins.define('numbers', ['a' ,'b'])
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752744195,https://api.github.com/repos/simonw/datasette/issues/983,752744195,MDEyOklzc3VlQ29tbWVudDc1Mjc0NDE5NQ==,9599,simonw,2020-12-30T20:12:26Z,2020-12-30T20:12:26Z,OWNER,"This implementation doesn't have an equivalent of ""hookspecs"" which can identify if a registered plugin implementation matches a known signature. I should add that, it will provide a better developer experience if someone has a typo.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752742669,https://api.github.com/repos/simonw/datasette/issues/983,752742669,MDEyOklzc3VlQ29tbWVudDc1Mjc0MjY2OQ==,9599,simonw,2020-12-30T20:07:05Z,2020-12-30T20:07:18Z,OWNER,"Initial prototype:
```javascript
window.datasette = {};
window.datasette.plugins = (function() {
    var registry = {};

    function extractParameters(fn) {
        var match = /\((.*)\)/.exec(fn.toString());
        if (match && match[1].trim()) {
            return match[1].split(',').map(s => s.trim());
        } else {
            return [];
        }
    }

    function register(hook, fn, parameters) {
        parameters = parameters || extractParameters(fn);
        if (!registry[hook]) {
            registry[hook] = [];
        }
        registry[hook].push([fn, parameters]);
    }

    function call(hook, args) {
        args = args || {};
        var implementations = registry[hook] || [];
        var results = [];
        implementations.forEach(([fn, parameters]) => {
            /* Call with the correct arguments */
            var callWith = parameters.map(parameter => args[parameter]);
            var result = fn.apply(fn, callWith);
            if (result) {
                results.push(result);
            }
        });
        return results;
    }
    return {
        register: register,
        _registry: registry,
        call: call
    };
})();
```
Usage example:
```javascript
datasette.plugins.register('numbers', (a, b) => a + b)
datasette.plugins.register('numbers', (a, b) => a * b)
datasette.plugins.call('numbers', {a: 4, b: 6})
/* Returns [10, 24] */
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752729035,https://api.github.com/repos/simonw/datasette/issues/983,752729035,MDEyOklzc3VlQ29tbWVudDc1MjcyOTAzNQ==,9599,simonw,2020-12-30T19:15:56Z,2020-12-30T19:16:44Z,OWNER,"The `column_actions` hook is the obvious first place to try this out. What are some demo plugins I could build for it?

- Word cloud for this column
- Count values (essentially linking to the SQL query for that column, as an extended version of the facet counts) - would be great if this could include pagination somehow, via #856.
- Extract this column into a separate table
- Add an index to this column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752722863,https://api.github.com/repos/simonw/datasette/issues/983,752722863,MDEyOklzc3VlQ29tbWVudDc1MjcyMjg2Mw==,9599,simonw,2020-12-30T18:52:39Z,2020-12-30T18:52:39Z,OWNER,"Then to call the plugins:
```javascript
datasette.plugins.call('column_actions', {database: 'database', table: 'table'})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752721840,https://api.github.com/repos/simonw/datasette/issues/983,752721840,MDEyOklzc3VlQ29tbWVudDc1MjcyMTg0MA==,9599,simonw,2020-12-30T18:48:53Z,2020-12-30T18:51:51Z,OWNER,"Potential design:
```javascript
datasette.plugins.register('column_actions', function(database, table, column, actor) {
    /* ... *l
})
```
Or if you want to be explicit to survive minification:
```javascript
datasette.plugins.register('column_actions', function(database, table, column, actor) {
    /* ... *l
}, ['database', 'table', 'column', 'actor'])
```
I'm making that list of parameter names an optional third argument to the `register()` function. If that argument isn't passed, introspection will be used to figure out the parameter names.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752721069,https://api.github.com/repos/simonw/datasette/issues/983,752721069,MDEyOklzc3VlQ29tbWVudDc1MjcyMTA2OQ==,9599,simonw,2020-12-30T18:46:10Z,2020-12-30T18:46:10Z,OWNER,"Pluggy does dependency injection by introspecting the named arguments to the Python function, which I really like.

That's tricker in JavaScript. It looks like the only way to introspect a function is to look at the `.toString()` representation of it and parse the `(parameter, list)` using a regular expression.

Even more challenging: JavaScript developers love minifying their code, and minification can shorten the function parameter names.

From https://code-maven.com/dependency-injection-in-angularjs it looks like Angular.js does dependency injection and solves this by letting you optionally provide a separate list of the arguments your function uses:

```javascript
   angular.module('DemoApp', [])
   .controller('DemoController', ['$scope', '$log', function($scope, $log) {
       $scope.message = ""Hello World"";
       $log.debug('logging hello');
   }]);
```

I can copy that approach: I'll introspect by default, but provide a documented mechanism for explicitly listing your parameter names so that if you know your plugin code will be minified you can use that instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752715412,https://api.github.com/repos/simonw/datasette/issues/983,752715412,MDEyOklzc3VlQ29tbWVudDc1MjcxNTQxMg==,9599,simonw,2020-12-30T18:25:31Z,2020-12-30T18:25:31Z,OWNER,I'm going to introduce a global `datasette` object which holds all the documented JavaScript API for plugin authors.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/983#issuecomment-752715236,https://api.github.com/repos/simonw/datasette/issues/983,752715236,MDEyOklzc3VlQ29tbWVudDc1MjcxNTIzNg==,9599,simonw,2020-12-30T18:24:54Z,2020-12-30T18:24:54Z,OWNER,"I think I'm going to try building a very lightweight clone of the core API design of Pluggy - not the advanced features, just the idea that plugins can register and a call to `plugin.nameOfHook()` will return the concatenated results of all of the registered hooks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/issues/987#issuecomment-752714747,https://api.github.com/repos/simonw/datasette/issues/987,752714747,MDEyOklzc3VlQ29tbWVudDc1MjcxNDc0Nw==,9599,simonw,2020-12-30T18:23:08Z,2020-12-30T18:23:20Z,OWNER,"In terms of ""places to put your plugin content"", the simplest solution I can think of is something like this:
```html
<div id=""plugin-content-pre-table""></div>
```
Alternative designs:

- A documented JavaScript function that returns the CSS selector where plugins should put their content
- A documented JavaScript function that returns a DOM node where plugins should put their content. This would allow the JavaScript to create the element if it does not already exist (though it wouldn't be obvious WHERE that element should be created)
- Documented JavaScript functions for things like ""append this node/HTML to the place-where-plugins-go""

I think the original option - an empty `<div>` with a known `id` attribute - is the right one to go with here. It's the simplest, it's very easy for custom template authors to understand and it acknowledges that plugins may have all kinds of extra crazy stuff they want to do - like checking in that div to see if another plugin has written to it already, for example.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/987#issuecomment-752697279,https://api.github.com/repos/simonw/datasette/issues/987,752697279,MDEyOklzc3VlQ29tbWVudDc1MjY5NzI3OQ==,9599,simonw,2020-12-30T17:23:27Z,2020-12-30T17:23:32Z,OWNER,Related problem: right now if you're writing custom template it's not at all obvious how to write them such that visualization plugins like `datasette-cluster-map` will continue to work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/987#issuecomment-752696499,https://api.github.com/repos/simonw/datasette/issues/987,752696499,MDEyOklzc3VlQ29tbWVudDc1MjY5NjQ5OQ==,9599,simonw,2020-12-30T17:21:08Z,2020-12-30T17:21:08Z,OWNER,"More generally, I need to document certain areas of the page that JavaScript plugins are invited to append their content to - such that plugin authors can use them and feel confident that future changes to the Datasette templates won't break their plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712984738,Documented HTML hooks for JavaScript plugin authors,
https://github.com/simonw/datasette/issues/1160#issuecomment-752275611,https://api.github.com/repos/simonw/datasette/issues/1160,752275611,MDEyOklzc3VlQ29tbWVudDc1MjI3NTYxMQ==,9599,simonw,2020-12-29T23:32:04Z,2020-12-29T23:32:04Z,OWNER,"If I can get this working for CSV, TSV, JSON and JSON-NL that should be enough to exercise the API design pretty well across both streaming and non-streaming formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752274509,https://api.github.com/repos/simonw/datasette/issues/1160,752274509,MDEyOklzc3VlQ29tbWVudDc1MjI3NDUwOQ==,9599,simonw,2020-12-29T23:26:02Z,2020-12-29T23:26:02Z,OWNER,"The documentation for this plugin hook is going to be pretty detailed, since it involves writing custom classes.

I'll stick it all on the existing hooks page for the moment, but I should think about breaking up the plugin hook documentation into a page-per-hook in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1163#issuecomment-752274340,https://api.github.com/repos/simonw/datasette/issues/1163,752274340,MDEyOklzc3VlQ29tbWVudDc1MjI3NDM0MA==,9599,simonw,2020-12-29T23:25:02Z,2020-12-29T23:25:02Z,OWNER,This will be built on top of `httpx` since that's already a dependency.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776128565,"""datasette insert data.db url-to-csv""",
https://github.com/simonw/datasette/issues/1160#issuecomment-752274078,https://api.github.com/repos/simonw/datasette/issues/1160,752274078,MDEyOklzc3VlQ29tbWVudDc1MjI3NDA3OA==,9599,simonw,2020-12-29T23:23:39Z,2020-12-29T23:23:39Z,OWNER,"If I design this right I can ship a full version of the command-line `datasette insert` command in a release without doing any work at all on the Web UI version of it - that UI can then come later, without needing any changes to be made to the plugin hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752273873,https://api.github.com/repos/simonw/datasette/issues/1160,752273873,MDEyOklzc3VlQ29tbWVudDc1MjI3Mzg3Mw==,9599,simonw,2020-12-29T23:22:30Z,2020-12-29T23:22:30Z,OWNER,"How much of this should I get done in a branch before merging into `main`?

The challenge here is the plugin hook design: ideally I don't want an incomplete plugin hook design in `main` since that could be a blocker for a release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752273400,https://api.github.com/repos/simonw/datasette/issues/1160,752273400,MDEyOklzc3VlQ29tbWVudDc1MjI3MzQwMA==,9599,simonw,2020-12-29T23:19:46Z,2020-12-29T23:19:46Z,OWNER,I'm going to break out some separate tickets.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752273306,https://api.github.com/repos/simonw/datasette/issues/1160,752273306,MDEyOklzc3VlQ29tbWVudDc1MjI3MzMwNg==,9599,simonw,2020-12-29T23:19:15Z,2020-12-29T23:19:15Z,OWNER,It would be nice if this abstraction could support progress bars as well. These won't necessarily work for every format - or they might work for things loaded from files but not things loaded over URLs (if the `content-length` HTTP header is missing) - but if they ARE possible it would be good to provide them - both for the CLI interface and the web insert UI.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752267905,https://api.github.com/repos/simonw/datasette/issues/1160,752267905,MDEyOklzc3VlQ29tbWVudDc1MjI2NzkwNQ==,9599,simonw,2020-12-29T22:52:09Z,2020-12-29T22:52:09Z,OWNER,"What's the simplest thing that could possible work? I think it's `datasette insert blah.db data.csv` - no URL handling, no other formats.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752266076,https://api.github.com/repos/simonw/datasette/issues/1160,752266076,MDEyOklzc3VlQ29tbWVudDc1MjI2NjA3Ng==,9599,simonw,2020-12-29T22:42:23Z,2020-12-29T22:42:59Z,OWNER,"Aside: maybe `datasette insert` works against simple files, but a later mechanism called `datasette import` allows plugins to register sub-commands, like `datasette import github ...` or `datasette import jira ...` or whatever.

This would be useful for import mechanisms that are likely to need their own custom set of command-line options unique to that source.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752265600,https://api.github.com/repos/simonw/datasette/issues/1160,752265600,MDEyOklzc3VlQ29tbWVudDc1MjI2NTYwMA==,9599,simonw,2020-12-29T22:39:56Z,2020-12-29T22:39:56Z,OWNER,"Does it definitely make sense to break this operation up into the code that turns the incoming format into a iterator of dictionaries, then the code that inserts those into the database using `sqlite-utils`?

That seems right for simple imports, where the incoming file represents a sequence of records in a single table. But what about more complex formats? What if a format needs to be represented as multiple tables?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752259345,https://api.github.com/repos/simonw/datasette/issues/1160,752259345,MDEyOklzc3VlQ29tbWVudDc1MjI1OTM0NQ==,9599,simonw,2020-12-29T22:11:54Z,2020-12-29T22:11:54Z,OWNER,"Important detail from https://docs.python.org/3/library/csv.html#csv.reader

> If *csvfile* is a file object, it should be opened with `newline=''`. [1]
>
> [...]
>
> If `newline=''` is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use `\r\n` linendings on write an extra `\r` will be added. It should always be safe to specify `newline=''`, since the csv module does its own ([universal](https://docs.python.org/3/glossary.html#term-universal-newlines)) newline handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752257666,https://api.github.com/repos/simonw/datasette/issues/1160,752257666,MDEyOklzc3VlQ29tbWVudDc1MjI1NzY2Ng==,9599,simonw,2020-12-29T22:09:18Z,2020-12-29T22:09:18Z,OWNER,"### Figuring out the API design

I want to be able to support different formats, and be able to parse them into tables either streaming or in one go depending on if the format supports that.

Ideally I want to be able to pull the first 1,024 bytes for the purpose of detecting the format, then replay those bytes again later. I'm considering this a stretch goal though.

CSV is easy to parse as a stream - here’s [how sqlite-utils does it](https://github.com/simonw/sqlite-utils/blob/f1277f638f3a54a821db6e03cb980adad2f2fa35/sqlite_utils/cli.py#L630):

        dialect = ""excel-tab"" if tsv else ""excel""
        with file_progress(json_file, silent=silent) as json_file:
            reader = csv_std.reader(json_file, dialect=dialect)
            headers = next(reader)
            docs = (dict(zip(headers, row)) for row in reader)

Problem: using `db.insert_all()` could block for a long time on a big set of rows. Probably easiest to batch the records before calling `insert_all()` and then run a batch at a time using a `db.execute_write_fn()` call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1161#issuecomment-752253095,https://api.github.com/repos/simonw/datasette/issues/1161,752253095,MDEyOklzc3VlQ29tbWVudDc1MjI1MzA5NQ==,9599,simonw,2020-12-29T21:49:57Z,2020-12-29T21:49:57Z,OWNER,https://github.com/Homebrew/homebrew-core/pull/67983,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",776101101,Update a whole bunch of links to datasette.io instead of datasette.readthedocs.io,
https://github.com/simonw/datasette/issues/1160#issuecomment-752236520,https://api.github.com/repos/simonw/datasette/issues/1160,752236520,MDEyOklzc3VlQ29tbWVudDc1MjIzNjUyMA==,9599,simonw,2020-12-29T20:48:51Z,2020-12-29T20:48:51Z,OWNER,It would be neat if `datasette insert` could accept a `--plugins-dir` option which allowed one-off format plugins to be registered. Bit tricky to implement since the `--format` Click option will already be populated by that plugin hook call.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751925934,https://api.github.com/repos/simonw/datasette/issues/1160,751925934,MDEyOklzc3VlQ29tbWVudDc1MTkyNTkzNA==,9599,simonw,2020-12-29T02:40:13Z,2020-12-29T20:25:57Z,OWNER,"Basic command design:

    datasette insert data.db blah.csv

The options can include:

- `--format` to specify the exact format - without this it will be guessed based on the filename
- `--table` to specify the table (otherwise the filename is used)
- `--pk` to specify one or more primary key columns
- `--replace` to specify that existing rows with a matching primary key should be replaced
- `--upsert` to specify that existing matching rows should be upserted
- `--ignore` to ignore matching rows
- `--alter` to alter the table to add missing columns
- `--type column type` to specify the type of a column - useful when working with CSV or TSV files","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752208036,https://api.github.com/repos/simonw/datasette/issues/1160,752208036,MDEyOklzc3VlQ29tbWVudDc1MjIwODAzNg==,9599,simonw,2020-12-29T19:06:35Z,2020-12-29T19:06:35Z,OWNER,"If I'm going to execute 1000s of writes in an `async def` operation it may make sense to break that up into smaller chunks, so as not to block the event loop for too long.

https://stackoverflow.com/a/36648102 and https://github.com/python/asyncio/issues/284 confirm that `await asyncio.sleep(0)` is the recommended way of doing this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-752203909,https://api.github.com/repos/simonw/datasette/issues/1160,752203909,MDEyOklzc3VlQ29tbWVudDc1MjIwMzkwOQ==,9599,simonw,2020-12-29T18:54:19Z,2020-12-29T18:54:19Z,OWNER,More thoughts on this: the key mechanism that populates the tables needs to be an `aysnc def` method of some sort so that it can run as part of the async loop in core Datasette - for importing from web uploads.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/417#issuecomment-752098906,https://api.github.com/repos/simonw/datasette/issues/417,752098906,MDEyOklzc3VlQ29tbWVudDc1MjA5ODkwNg==,82988,psychemedia,2020-12-29T14:34:30Z,2020-12-29T14:34:50Z,CONTRIBUTOR,"FWIW, I had a look at `watchdog` for a `datasette` powered Jupyter notebook search tool: https://github.com/ouseful-testing/nbsearch/blob/main/nbsearch/nbwatchdog.py

Not a production thing, just an experiment trying to explore what might be possible...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",421546944,Datasette Library,
https://github.com/simonw/datasette/issues/1160#issuecomment-751947991,https://api.github.com/repos/simonw/datasette/issues/1160,751947991,MDEyOklzc3VlQ29tbWVudDc1MTk0Nzk5MQ==,9599,simonw,2020-12-29T05:06:50Z,2020-12-29T05:07:03Z,OWNER,"Given the URL option could it be possible for plugins to ""subscribe"" to URLs that keep on streaming?

    datasette insert db.db https://example.con/streaming-api \
      --format api-stream","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751946262,https://api.github.com/repos/simonw/datasette/issues/1160,751946262,MDEyOklzc3VlQ29tbWVudDc1MTk0NjI2Mg==,9599,simonw,2020-12-29T04:56:12Z,2020-12-29T04:56:32Z,OWNER,"Potential design for this: a `datasette memory` command which takes most of the same arguments as `datasette serve` but starts an in-memory database and treats the command arguments as things that should be inserted into that in-memory database.

    tail -f access.log | datasette memory - \
      --format clf -p 8002 -o","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751945094,https://api.github.com/repos/simonw/datasette/issues/1160,751945094,MDEyOklzc3VlQ29tbWVudDc1MTk0NTA5NA==,9599,simonw,2020-12-29T04:48:11Z,2020-12-29T04:48:11Z,OWNER,"It would be pretty cool if you could launch Datasette directly against an insert-compatible file or URL without first having to load it into a SQLite database file.

Or imagine being able to tail a log file and like that directly into a new Datasette process, which then runs a web server with the UI while simultaneously continuing to load new entries from that log into the in-memory SQLite database that it is serving...

Not quite sure what that CLI interface would look like. Maybe treat that as a future stretch goal for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751943837,https://api.github.com/repos/simonw/datasette/issues/1160,751943837,MDEyOklzc3VlQ29tbWVudDc1MTk0MzgzNw==,9599,simonw,2020-12-29T04:40:30Z,2020-12-29T04:40:30Z,OWNER,"The `insert` command should also accept URLs - anything starting with `http://` or `https://`.

It should accept more than one file name at a time for bulk inserts.

if using a URL that URL will be passed to the method that decides if a plugin implementation can handle the import or not. This will allow plugins to register themselves for specific websites.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751926437,https://api.github.com/repos/simonw/datasette/issues/1160,751926437,MDEyOklzc3VlQ29tbWVudDc1MTkyNjQzNw==,9599,simonw,2020-12-29T02:43:21Z,2020-12-29T02:43:37Z,OWNER,"Default formats to support:

- CSV
- TSV
- JSON and newline-delimited JSON
- YAML

Each of these will be implemented as a default plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751926218,https://api.github.com/repos/simonw/datasette/issues/1160,751926218,MDEyOklzc3VlQ29tbWVudDc1MTkyNjIxOA==,9599,simonw,2020-12-29T02:41:57Z,2020-12-29T02:41:57Z,OWNER,"Other names I considered:

- `datasette load`
- `datasette import` - I decided to keep this name available for any future work that might involve plugins that help import data from APIs as opposed to inserting it from files","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/1160#issuecomment-751926095,https://api.github.com/repos/simonw/datasette/issues/1160,751926095,MDEyOklzc3VlQ29tbWVudDc1MTkyNjA5NQ==,9599,simonw,2020-12-29T02:41:15Z,2020-12-29T02:41:15Z,OWNER,"The UI can live at `/-/insert` and be available by default to the `root` user only. It can offer the following:

- Upload a file and have the import type detected (equivalent to `datasette insert data.db thatfile.csv`)
- Copy and paste the data to be inserted into a textarea
- API equivalents of these","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",775666296,"""datasette insert"" command and plugin hook",
https://github.com/simonw/datasette/issues/675#issuecomment-751826749,https://api.github.com/repos/simonw/datasette/issues/675,751826749,MDEyOklzc3VlQ29tbWVudDc1MTgyNjc0OQ==,9599,simonw,2020-12-28T18:49:21Z,2020-12-28T18:49:21Z,OWNER,"That `--exec` could help solve all sorts of other problems too, like needing to `apt-get install` extra packages or download files from somewhere using `wget`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-751826621,https://api.github.com/repos/simonw/datasette/issues/675,751826621,MDEyOklzc3VlQ29tbWVudDc1MTgyNjYyMQ==,9599,simonw,2020-12-28T18:48:51Z,2020-12-28T18:48:51Z,OWNER,"I could make `--include` work if I also had a mechanism for running some shell commands inside the container at the end of the build - which users could then use to move files into the correct place.

    datasette publish cloudrun my.db --include src/ --exec 'mv /app/src/config.yml /etc/conf/config.yml'","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/417#issuecomment-751504136,https://api.github.com/repos/simonw/datasette/issues/417,751504136,MDEyOklzc3VlQ29tbWVudDc1MTUwNDEzNg==,212369,drewda,2020-12-27T19:02:06Z,2020-12-27T19:02:06Z,NONE,"Very much looking forward to seeing this functionality come together. This is probably out-of-scope for an initial release, but in the future it could be useful to also think of how to run this is a container'ized context. For example, an immutable datasette container that points to an S3 bucket of SQLite DBs or CSVs. Or an immutable datasette container pointing to a NFS volume elsewhere on a Kubernetes cluster.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",421546944,Datasette Library,
https://github.com/simonw/datasette/issues/1150#issuecomment-751476406,https://api.github.com/repos/simonw/datasette/issues/1150,751476406,MDEyOklzc3VlQ29tbWVudDc1MTQ3NjQwNg==,18221871,noklam,2020-12-27T14:51:39Z,2020-12-27T14:51:39Z,NONE,"I like the idea of _internal, it's a nice way to get a data catalog quickly. I wonder if this trick applies to db other than SQLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-751375487,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59,751375487,MDEyOklzc3VlQ29tbWVudDc1MTM3NTQ4Nw==,631242,frosencrantz,2020-12-26T17:08:44Z,2020-12-26T17:08:44Z,CONTRIBUTOR,"Hi @simonw,  do I need to do anything else for this PR to be considered to be included? I've tried using this project and it is quite nice to be able to explore a repository, but noticed that a couple commands don't allow you to use authorization from the environment variable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771872303,Remove unneeded exists=True for -a/--auth flag.,
https://github.com/simonw/datasette/issues/417#issuecomment-751127485,https://api.github.com/repos/simonw/datasette/issues/417,751127485,MDEyOklzc3VlQ29tbWVudDc1MTEyNzQ4NQ==,9599,simonw,2020-12-24T22:58:05Z,2020-12-24T22:58:05Z,OWNER,"That's a great idea. I'd ruled that out because working with the different operating system versions of those is tricky, but if `watchdog` can handle those differences for me this could be a really good option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",421546944,Datasette Library,
https://github.com/simonw/datasette/issues/417#issuecomment-751127384,https://api.github.com/repos/simonw/datasette/issues/417,751127384,MDEyOklzc3VlQ29tbWVudDc1MTEyNzM4NA==,1279360,dyllan-to-you,2020-12-24T22:56:48Z,2020-12-24T22:56:48Z,NONE,"Instead of scanning the directory every 10s, have you considered listening for the native system events to notify you of updates?

I think python has a nice module to do this for you called [watchdog](https://pypi.org/project/watchdog/)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",421546944,Datasette Library,
https://github.com/dogsheep/dogsheep-photos/issues/28#issuecomment-751125270,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/28,751125270,MDEyOklzc3VlQ29tbWVudDc1MTEyNTI3MA==,129786,jmelloy,2020-12-24T22:26:22Z,2020-12-24T22:26:22Z,NONE,This comes around if you’ve run the photo export without running an s3 upload. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",624490929,Invalid SQL no such table: main.uploads,
https://github.com/simonw/datasette/pull/1159#issuecomment-750849460,https://api.github.com/repos/simonw/datasette/issues/1159,750849460,MDEyOklzc3VlQ29tbWVudDc1MDg0OTQ2MA==,22429695,codecov[bot],2020-12-24T11:07:35Z,2020-12-24T11:29:21Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=h1) Report
> Merging [#1159](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=desc) (c820abd) into [main](https://codecov.io/gh/simonw/datasette/commit/a882d679626438ba0d809944f06f239bcba8ee96?el=desc) (a882d67) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1159/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1159   +/-   ##
=======================================
  Coverage   91.55%   91.55%           
=======================================
  Files          32       32           
  Lines        3930     3930           
=======================================
  Hits         3598     3598           
  Misses        332      332           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=footer). Last update [a882d67...c820abd](https://codecov.io/gh/simonw/datasette/pull/1159?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/simonw/datasette/pull/1158#issuecomment-750390741,https://api.github.com/repos/simonw/datasette/issues/1158,750390741,MDEyOklzc3VlQ29tbWVudDc1MDM5MDc0MQ==,9599,simonw,2020-12-23T17:05:32Z,2020-12-23T17:05:32Z,OWNER,"Thanks for this!

I'm fine keeping the `os.path` stuff as is.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",773913793,Modernize code to Python 3.6+,
https://github.com/simonw/datasette/pull/1158#issuecomment-750389683,https://api.github.com/repos/simonw/datasette/issues/1158,750389683,MDEyOklzc3VlQ29tbWVudDc1MDM4OTY4Mw==,6774676,eumiro,2020-12-23T17:02:50Z,2020-12-23T17:02:50Z,CONTRIBUTOR,"The dict/set suggestion comes from `pyupgrade --py36-plus`, but then had to `black` the change.

The rest comes from PyCharm's Inspect code function. I reviewed all the suggestions and fixed a thing or two, such as leading/trailing spaces in the docstrings or turned around the chained conditions.

Then I tried to convert all `os.path/glob/open` to `Path`, but there were some local test issues, so I'll have to start over in smaller chunks if you want to have that too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",773913793,Modernize code to Python 3.6+,
https://github.com/simonw/datasette/pull/1158#issuecomment-750373603,https://api.github.com/repos/simonw/datasette/issues/1158,750373603,MDEyOklzc3VlQ29tbWVudDc1MDM3MzYwMw==,9599,simonw,2020-12-23T16:26:21Z,2020-12-23T16:26:21Z,OWNER,"Did you use a tool to find these or are they all from manual code inspection?

They look good - I particularly like the set/dict  literals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",773913793,Modernize code to Python 3.6+,
https://github.com/simonw/datasette/pull/1158#issuecomment-750373496,https://api.github.com/repos/simonw/datasette/issues/1158,750373496,MDEyOklzc3VlQ29tbWVudDc1MDM3MzQ5Ng==,22429695,codecov[bot],2020-12-23T16:26:06Z,2020-12-23T16:26:06Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=h1) Report
> Merging [#1158](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=desc) (37ce72f) into [main](https://codecov.io/gh/simonw/datasette/commit/90eba4c3ca569c57e96bce314e7ac8caf67d884e?el=desc) (90eba4c) will **not change** coverage.
> The diff coverage is `87.50%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1158/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1158   +/-   ##
=======================================
  Coverage   91.55%   91.55%           
=======================================
  Files          32       32           
  Lines        3930     3930           
=======================================
  Hits         3598     3598           
  Misses        332      332           
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `77.41% <ø> (ø)` | |
| [datasette/facets.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZhY2V0cy5weQ==) | `89.04% <ø> (ø)` | |
| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <ø> (ø)` | |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <ø> (ø)` | |
| [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <ø> (ø)` | |
| [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.02% <ø> (ø)` | |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <50.00%> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.11% <100.00%> (ø)` | |
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `92.13% <100.00%> (ø)` | |
| ... and [1 more](https://codecov.io/gh/simonw/datasette/pull/1158/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=footer). Last update [90eba4c...37ce72f](https://codecov.io/gh/simonw/datasette/pull/1158?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",773913793,Modernize code to Python 3.6+,
https://github.com/simonw/datasette/issues/1099#issuecomment-749845797,https://api.github.com/repos/simonw/datasette/issues/1099,749845797,MDEyOklzc3VlQ29tbWVudDc0OTg0NTc5Nw==,9599,simonw,2020-12-23T00:13:29Z,2020-12-23T00:14:25Z,OWNER,"Also need to solve displaying these links in the opposite direction:

https://latest.datasette.io/_internal/tables/fixtures,facet_cities

<img width=""617"" alt=""_internal__tables"" src=""https://user-images.githubusercontent.com/9599/102944742-84cd3900-4470-11eb-95a0-bfaf00b17e82.png"">

That page should link to lists of records in columns, foreign_keys and indexes - like this example: https://latest.datasette.io/fixtures/roadside_attractions/1

<img width=""712"" alt=""fixtures__roadside_attractions"" src=""https://user-images.githubusercontent.com/9599/102944816-c1009980-4470-11eb-9dbd-50d0ba4fd137.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1152#issuecomment-748206874,https://api.github.com/repos/simonw/datasette/issues/1152,748206874,MDEyOklzc3VlQ29tbWVudDc0ODIwNjg3NA==,9599,simonw,2020-12-18T17:03:00Z,2020-12-22T23:58:04Z,OWNER,"Another permissions thought: what if ALL Datasette permissions were default-deny, and plugins could only grant permission to things, not block permission?

Right now a plugin can reply `False` to block, `True` to allow or `None` for ""I have no opinion on this, ask someone else"" - but even I'm confused by the interactions between block and allow and I implemented the system!

If everything in Datasette was default-deny then the user could use `--public-view` as an option when starting the server to default-allow view actions.

More importantly: plugins could return SQL statements that select a list of databases/tables the user is allowed access to. These could then be combined with `UNION` to create a full list of available resources.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747920515,https://api.github.com/repos/simonw/datasette/issues/1152,747920515,MDEyOklzc3VlQ29tbWVudDc0NzkyMDUxNQ==,9599,simonw,2020-12-18T07:29:21Z,2020-12-22T23:57:29Z,OWNER,Could I solve this using a configured canned query against the `_internal` tables with the actor's properties as inputs?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1099#issuecomment-749771231,https://api.github.com/repos/simonw/datasette/issues/1099,749771231,MDEyOklzc3VlQ29tbWVudDc0OTc3MTIzMQ==,9599,simonw,2020-12-22T20:54:25Z,2020-12-22T20:54:25Z,OWNER,"https://latest.datasette.io/_internal/foreign_keys (use https://latest.datasette.io/login-as-root first) is now a compound foreign key table:
```sql
CREATE TABLE foreign_keys (
        ""database_name"" TEXT,
        ""table_name"" TEXT,
        ""id"" INTEGER,
        ""seq"" INTEGER,
        ""table"" TEXT,
        ""from"" TEXT,
        ""to"" TEXT,
        ""on_update"" TEXT,
        ""on_delete"" TEXT,
        ""match"" TEXT,
        PRIMARY KEY (database_name, table_name, id, seq),
        FOREIGN KEY (database_name) REFERENCES databases(database_name),
        FOREIGN KEY (database_name, table_name) REFERENCES tables(database_name, table_name)
    );
```
Currently the `database_name` column becomes a link (because it's a single foreign key) but the `table_name` one remains a non-link:

<img width=""1079"" alt=""_internal__foreign_keys__24_rows"" src=""https://user-images.githubusercontent.com/9599/102932110-87ba3080-4454-11eb-9ad5-b70a65129588.png"">

My original idea for compound foreign keys was to turn both of those columns into links, but that doesn't fit here because `database_name` is already part of a different foreign key.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1152#issuecomment-749750995,https://api.github.com/repos/simonw/datasette/issues/1152,749750995,MDEyOklzc3VlQ29tbWVudDc0OTc1MDk5NQ==,9599,simonw,2020-12-22T20:05:30Z,2020-12-22T20:05:30Z,OWNER,"#1150 is landed now, which means there's a new, hidden `_internal` SQLite in-memory database containing all of the tables and databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/509#issuecomment-749749948,https://api.github.com/repos/simonw/datasette/issues/509,749749948,MDEyOklzc3VlQ29tbWVudDc0OTc0OTk0OA==,9599,simonw,2020-12-22T20:03:10Z,2020-12-22T20:03:10Z,OWNER,"If you open multiple files with the same filename, e.g. like this:

    datasette fixtures.db templates/fixtures.db plugins/fixtures.db

You'll now get this:

<img width=""702"" alt=""Datasette__fixtures__fixtures_2__fixtures_3"" src=""https://user-images.githubusercontent.com/9599/102928494-a4069f00-444d-11eb-9c1e-382a4e27bb51.png"">
","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",456568880,Support opening multiple databases with the same stem,
https://github.com/simonw/datasette/issues/509#issuecomment-749738241,https://api.github.com/repos/simonw/datasette/issues/509,749738241,MDEyOklzc3VlQ29tbWVudDc0OTczODI0MQ==,9599,simonw,2020-12-22T19:38:14Z,2020-12-22T19:38:14Z,OWNER,I'm fixing this in #1155.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456568880,Support opening multiple databases with the same stem,
https://github.com/simonw/datasette/issues/1155#issuecomment-749723557,https://api.github.com/repos/simonw/datasette/issues/1155,749723557,MDEyOklzc3VlQ29tbWVudDc0OTcyMzU1Nw==,9599,simonw,2020-12-22T19:08:27Z,2020-12-22T19:08:27Z,OWNER,"I'm going to have the `.add_database()` method select the name used in the path, de-duping against any existing names. It will then set database.name to that so that the database has access to its own name.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748356492,https://api.github.com/repos/simonw/datasette/issues/1155,748356492,MDEyOklzc3VlQ29tbWVudDc0ODM1NjQ5Mg==,9599,simonw,2020-12-18T22:49:32Z,2020-12-22T01:13:05Z,OWNER,"There's some messy code that needs fixing here. The `datasette.databases` dictionary right now has a key that corresponds to the `/_internal` URL in the path, and a value that's a `Database()` object. BUT... the `Database()` object doesn't know what its key is.

While fixing this I should fix the issue where Datasette gets confused by multiple databases with the same stem: https://github.com/simonw/datasette/issues/509","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1157#issuecomment-749179460,https://api.github.com/repos/simonw/datasette/issues/1157,749179460,MDEyOklzc3VlQ29tbWVudDc0OTE3OTQ2MA==,9599,simonw,2020-12-21T20:24:19Z,2020-12-21T20:24:19Z,OWNER,"Three places to fix:

https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/tracer.py#L40-L42

https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/utils/__init__.py#L139-L152

https://github.com/simonw/datasette/blob/dcdfb2c301341d45b66683e3e3be72f9c7585b2f/datasette/views/base.py#L460-L461","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",772438273,Use time.perf_counter() instead of time.time() to measure performance,
https://github.com/simonw/datasette/issues/1155#issuecomment-749176936,https://api.github.com/repos/simonw/datasette/issues/1155,749176936,MDEyOklzc3VlQ29tbWVudDc0OTE3NjkzNg==,9599,simonw,2020-12-21T20:18:15Z,2020-12-21T20:18:15Z,OWNER,"Fun query:
```sql
select table_name, group_concat(name, ', ') from columns group by database_name, table_name
```
https://latest.datasette.io/_internal?sql=select+table_name%2C+group_concat%28name%2C+%27%2C+%27%29+from+columns+group+by+database_name%2C+table_name","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-749170608,https://api.github.com/repos/simonw/datasette/issues/1155,749170608,MDEyOklzc3VlQ29tbWVudDc0OTE3MDYwOA==,9599,simonw,2020-12-21T20:01:47Z,2020-12-21T20:01:47Z,OWNER,I removed that `MEMORY` object() in dcdfb2c301341d45b66683e3e3be72f9c7585b2f,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1156#issuecomment-749158111,https://api.github.com/repos/simonw/datasette/issues/1156,749158111,MDEyOklzc3VlQ29tbWVudDc0OTE1ODExMQ==,9599,simonw,2020-12-21T19:33:45Z,2020-12-21T19:33:45Z,OWNER,"One reason for this change: it means I can use that database for more stuff. I've been thinking about moving metadata storage there for example, which fits a database called `_internal` but not one called `_schemas`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",772408750,Rename _schemas to _internal,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-748562330,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,748562330,MDEyOklzc3VlQ29tbWVudDc0ODU2MjMzMA==,41546558,RhetTbull,2020-12-20T04:45:08Z,2020-12-20T04:45:08Z,CONTRIBUTOR,Fixes the issue mentioned here: https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748562288,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,748562288,MDEyOklzc3VlQ29tbWVudDc0ODU2MjI4OA==,41546558,RhetTbull,2020-12-20T04:44:22Z,2020-12-20T04:44:22Z,CONTRIBUTOR,"@nickvazz @simonw I opened a [PR](https://github.com/dogsheep/dogsheep-photos/pull/31) that replaces the SQL for `ZCOMPUTEDASSETATTRIBUTES` to use osxphotos which now exposes all this data and has been updated for Big Sur.  I did regression tests to confirm the extracted data is identical, with one exception which should not affect operation: the old code pulled data from `ZCOMPUTEDASSETATTRIBUTES` for missing photos while the main loop ignores missing photos and does not add them to `apple_photos`.  The new code does not add rows to the `apple_photos_scores` table for missing photos.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436779,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,748436779,MDEyOklzc3VlQ29tbWVudDc0ODQzNjc3OQ==,41546558,RhetTbull,2020-12-19T07:49:00Z,2020-12-19T07:49:00Z,CONTRIBUTOR,@nickvazz ZGENERICASSET changed to ZASSET in Big Sur. Here's a list of other changes to the schema in Big Sur: https://github.com/RhetTbull/osxphotos/wiki/Changes-in-Photos-6---Big-Sur,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436115,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15,748436115,MDEyOklzc3VlQ29tbWVudDc0ODQzNjExNQ==,8573886,nickvazz,2020-12-19T07:43:38Z,2020-12-19T07:47:36Z,NONE,"Hey Simon! I really enjoy datasette so far, just started trying it out today following your iPhone photos [example](https://simonwillison.net/2020/May/21/dogsheep-photos/). 

I am not sure if you had run into this or not, but it seems like they might have changed one of the column names from
`ZGENERICASSET` to `ZASSET`. Should I open a PR? 

Would change:
- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L209-L213)
- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L238)
- [here](https://github.com/dogsheep/dogsheep-photos/blob/master/dogsheep_photos/cli.py#L240)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",612151767,Expose scores from ZCOMPUTEDASSETATTRIBUTES,
https://github.com/dogsheep/twitter-to-sqlite/issues/53#issuecomment-748436453,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53,748436453,MDEyOklzc3VlQ29tbWVudDc0ODQzNjQ1Mw==,27,anotherjesse,2020-12-19T07:47:01Z,2020-12-19T07:47:01Z,NONE,"I think this should probably be closed as won't fix.

Attempting to make a patch for this I realized that the since_id would limit to tweets posted since that since_id, not when it was favorited.  So favoriting something in the older would be missed if you used `--since` with a cron script

Better to just use `--stop_after` set to a couple hundred","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771324837,--since support for favorites,
https://github.com/dogsheep/dogsheep-photos/issues/21#issuecomment-748436195,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/21,748436195,MDEyOklzc3VlQ29tbWVudDc0ODQzNjE5NQ==,8573886,nickvazz,2020-12-19T07:44:32Z,2020-12-19T07:44:49Z,NONE,"I have also run into this a bit, would it be possible to post your `requirements.txt` so I can try and reproduce your [blog post](https://simonwillison.net/2020/May/21/dogsheep-photos/)?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",615474990,bpylist.archiver.CircularReference: archive has a cycle with uid(13),
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426877,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426877,MDEyOklzc3VlQ29tbWVudDc0ODQyNjg3Nw==,9599,simonw,2020-12-19T06:16:11Z,2020-12-19T06:16:11Z,MEMBER,"Here's why:

    if ""fts5"" in str(e):

But the error being raised here is:

    sqlite3.OperationalError: no such column: to

I'm going to attempt the escaped on on every error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426663,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426663,MDEyOklzc3VlQ29tbWVudDc0ODQyNjY2Mw==,9599,simonw,2020-12-19T06:14:06Z,2020-12-19T06:14:06Z,MEMBER,Looks like I already do that here: https://github.com/dogsheep/dogsheep-beta/blob/9ba4401017ac24ffa3bc1db38e0910ea49de7616/dogsheep_beta/__init__.py#L141-L146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426581,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426581,MDEyOklzc3VlQ29tbWVudDc0ODQyNjU4MQ==,9599,simonw,2020-12-19T06:13:17Z,2020-12-19T06:13:17Z,MEMBER,"One fix for this could be to try running the raw query, but if it throws an error run it again with the query escaped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/dogsheep/dogsheep-beta/issues/31#issuecomment-748426501,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31,748426501,MDEyOklzc3VlQ29tbWVudDc0ODQyNjUwMQ==,9599,simonw,2020-12-19T06:12:22Z,2020-12-19T06:12:22Z,MEMBER,I deliberately added support for advanced FTS in https://github.com/dogsheep/dogsheep-beta/commit/cbb2491b85d7ff416d6d429b60109e6c2d6d50b9 for #13 but that's the cause of this bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771316301,"Searching for ""github-to-sqlite"" throws an error",
https://github.com/simonw/datasette/issues/1155#issuecomment-748397998,https://api.github.com/repos/simonw/datasette/issues/1155,748397998,MDEyOklzc3VlQ29tbWVudDc0ODM5Nzk5OA==,9599,simonw,2020-12-19T01:28:18Z,2020-12-19T01:28:18Z,OWNER,"`datasette-graphql` returns an error due to this issue:

<img width=""878"" alt=""GraphiQL_and_fixtures__compound_three_primary_keys__1_001_rows"" src=""https://user-images.githubusercontent.com/9599/102677257-61e01380-4156-11eb-8f88-fe26b9c4a40b.png"">

On the console:
```
INFO:     127.0.0.1:63116 - ""POST /graphql/_internal HTTP/1.1"" 500 Internal Server Error
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/promise/promise.py"", line 844, in handle_future_result
    resolve(future.result())
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/datasette_graphql/utils.py"", line 603, in resolve_table
    data, _, _ = await view.data(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 304, in data
    db = self.ds.databases[database]
graphql.error.located_error.GraphQLLocatedError: ':memory:c6dd5abe1a757a7de00d99b699175bd33d9a575f05b5751bf856b8656fb07edd'
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748368660,https://api.github.com/repos/simonw/datasette/issues/1155,748368660,MDEyOklzc3VlQ29tbWVudDc0ODM2ODY2MA==,9599,simonw,2020-12-18T23:18:04Z,2020-12-19T01:12:00Z,OWNER,"A `Database` should have a `.name` which is unique across the Datasette instance and is used in the URL. The `path` should be optional, only set for file databases. A new `.memory_name` property can be used for shared memory databases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748368938,https://api.github.com/repos/simonw/datasette/issues/1155,748368938,MDEyOklzc3VlQ29tbWVudDc0ODM2ODkzOA==,9599,simonw,2020-12-18T23:19:04Z,2020-12-18T23:19:04Z,OWNER,`Database` internal class is documented here: https://docs.datasette.io/en/latest/internals.html#database-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748368384,https://api.github.com/repos/simonw/datasette/issues/1155,748368384,MDEyOklzc3VlQ29tbWVudDc0ODM2ODM4NA==,9599,simonw,2020-12-18T23:17:00Z,2020-12-18T23:17:00Z,OWNER,Here's the commit where I added it. https://github.com/simonw/datasette/commit/9743e1d91b5f0a2b3c1c0bd6ffce8739341f43c4 - I didn't yet have the `.add_database()` mechanism. Today the `MEMORY` object bit is no longer needed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/1155#issuecomment-748367922,https://api.github.com/repos/simonw/datasette/issues/1155,748367922,MDEyOklzc3VlQ29tbWVudDc0ODM2NzkyMg==,9599,simonw,2020-12-18T23:15:24Z,2020-12-18T23:15:24Z,OWNER,"The code for building up that `.databases` dictionary is a bit convoluted. Here's the code that adds a `:memory:` database if the user specified `--memory` OR if there are no files to be attached:

https://github.com/simonw/datasette/blob/ebc7aa287c99fe6114b79aeab8efb8d4489a6182/datasette/app.py#L221-L241

I'm not sure why I wrote it this way, instead of just calling `.add_database("":memory:"", Database(..., is_memory=True)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771216293,Better internal database_name for _internal database,
https://github.com/simonw/datasette/issues/509#issuecomment-748356637,https://api.github.com/repos/simonw/datasette/issues/509,748356637,MDEyOklzc3VlQ29tbWVudDc0ODM1NjYzNw==,9599,simonw,2020-12-18T22:50:03Z,2020-12-18T22:50:03Z,OWNER,Related problem caused by the new `_schemas` database - if a user attempts to open their own `_schemas.db` file it will fail. I'd like to open and mount that as `/_schemas_` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456568880,Support opening multiple databases with the same stem,
https://github.com/simonw/datasette/issues/1150#issuecomment-748354841,https://api.github.com/repos/simonw/datasette/issues/1150,748354841,MDEyOklzc3VlQ29tbWVudDc0ODM1NDg0MQ==,9599,simonw,2020-12-18T22:43:49Z,2020-12-18T22:43:49Z,OWNER,"For a demo, visit https://latest.datasette.io/login-as-root and then hit https://latest.datasette.io/_schemas","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-748352106,https://api.github.com/repos/simonw/datasette/issues/1150,748352106,MDEyOklzc3VlQ29tbWVudDc0ODM1MjEwNg==,9599,simonw,2020-12-18T22:34:40Z,2020-12-18T22:34:40Z,OWNER,"Needs documentation, but I can wait to write that until I've tested out the feature a bit more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-748351350,https://api.github.com/repos/simonw/datasette/issues/1150,748351350,MDEyOklzc3VlQ29tbWVudDc0ODM1MTM1MA==,9599,simonw,2020-12-18T22:32:13Z,2020-12-18T22:32:13Z,OWNER,Getting all the tests to pass is tricky because this adds a whole extra database to Datasette - and there's various code that loops through `ds.databases` as part of the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/493#issuecomment-748305976,https://api.github.com/repos/simonw/datasette/issues/493,748305976,MDEyOklzc3VlQ29tbWVudDc0ODMwNTk3Ng==,50527,jefftriplett,2020-12-18T20:34:39Z,2020-12-18T20:34:39Z,CONTRIBUTOR,"I can't keep up with the renaming contexts, but I like having the ability to run datasette+ datasette-ripgrep against different configs: 

```shell
datasette serve --metadata=./metadata.json
```

I have one for all of my code and one per client who has lots of code. So as long as I can point to datasette to something, it's easy to work with. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/1150#issuecomment-748260875,https://api.github.com/repos/simonw/datasette/issues/1150,748260875,MDEyOklzc3VlQ29tbWVudDc0ODI2MDg3NQ==,9599,simonw,2020-12-18T18:55:12Z,2020-12-18T18:55:12Z,OWNER,"I'm going to move the code into a `utils/schemas.py` module, to avoid further extending the `Datasette` class definition and to make it more easily testable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-748260118,https://api.github.com/repos/simonw/datasette/issues/1150,748260118,MDEyOklzc3VlQ29tbWVudDc0ODI2MDExOA==,9599,simonw,2020-12-18T18:54:12Z,2020-12-18T18:54:12Z,OWNER,"I'm going to tidy this up and land it. A couple of additional decisions:

- The database will be called `/_schemas`
- By default it will only be visible to `root` - thus avoiding having to solve the permissions problem with regards to users seeing schemas for tables that are otherwise invisible to them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/493#issuecomment-747966232,https://api.github.com/repos/simonw/datasette/issues/493,747966232,MDEyOklzc3VlQ29tbWVudDc0Nzk2NjIzMg==,9599,simonw,2020-12-18T09:19:41Z,2020-12-18T09:19:41Z,OWNER,Is there any reason to keep `--setting` rather than moving those items into a `configure.json` file with all the configuration options that currently live in `metadata.json`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/1152#issuecomment-747921195,https://api.github.com/repos/simonw/datasette/issues/1152,747921195,MDEyOklzc3VlQ29tbWVudDc0NzkyMTE5NQ==,9599,simonw,2020-12-18T07:31:25Z,2020-12-18T07:31:25Z,OWNER,It's also a really good fit for the new mechanism that's coming together in #1150.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747920852,https://api.github.com/repos/simonw/datasette/issues/1152,747920852,MDEyOklzc3VlQ29tbWVudDc0NzkyMDg1Mg==,9599,simonw,2020-12-18T07:30:22Z,2020-12-18T07:30:22Z,OWNER,Redefining all Datasette permissions in terms of SQL queries that return the set of databases and tables that the user is allowed to interact with does feel VERY Datasette-y.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747920087,https://api.github.com/repos/simonw/datasette/issues/1152,747920087,MDEyOklzc3VlQ29tbWVudDc0NzkyMDA4Nw==,9599,simonw,2020-12-18T07:27:58Z,2020-12-18T07:28:30Z,OWNER,"I want to keep the existing `metadata.json` ""allow"" blocks mechanism working. Note that if you have 1,000 tables and a permissions policy you won't be using ""allow"" blocks, you'll be using a more sophisticated permissions plugin instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747919782,https://api.github.com/repos/simonw/datasette/issues/1152,747919782,MDEyOklzc3VlQ29tbWVudDc0NzkxOTc4Mg==,9599,simonw,2020-12-18T07:27:01Z,2020-12-18T07:27:01Z,OWNER,"Perhaps this can be solved by keeping the existing plugin hooks and adding new, optional ones for bulk lookups.

If your plugin doesn't implement the bulk lookup hooks Datasette will do an inefficient loop through everything checking permissions on each one.

If you DO implement it you can speed things up dramatically.

Not sure if this would solve the homepage problem though, where you might need to run 1,000 table permission checks. That's more a case where you want to think in terms of a SQL where clause.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1150#issuecomment-747893704,https://api.github.com/repos/simonw/datasette/issues/1150,747893704,MDEyOklzc3VlQ29tbWVudDc0Nzg5MzcwNA==,9599,simonw,2020-12-18T06:19:13Z,2020-12-18T06:19:13Z,OWNER,I'm not going to block this issue on permissions - I will tackle the efficient bulk permissions problem in #1152.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1152#issuecomment-747893423,https://api.github.com/repos/simonw/datasette/issues/1152,747893423,MDEyOklzc3VlQ29tbWVudDc0Nzg5MzQyMw==,9599,simonw,2020-12-18T06:18:24Z,2020-12-18T06:18:24Z,OWNER,"What would Datasette's permission hooks look like if they all dealt with sets of items rather than individual items? So plugins could return a set of items that the user has permission to access, or even a WHERE clause?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747892731,https://api.github.com/repos/simonw/datasette/issues/1152,747892731,MDEyOklzc3VlQ29tbWVudDc0Nzg5MjczMQ==,9599,simonw,2020-12-18T06:16:29Z,2020-12-18T06:16:29Z,OWNER,"One enormous advantage I have is that after #1150 I will have a database table full of databases and tables that I can execute queries against.

This means I could calculate visible tables using SQL where clauses, which should be easily fast enough even against ten thousand plus tables.

The catch is the permissions hooks. Since I haven't hit Datasette 1.0 yet maybe I should redesign those hooks to work against the new in-memory database schema stuff?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1152#issuecomment-747891854,https://api.github.com/repos/simonw/datasette/issues/1152,747891854,MDEyOklzc3VlQ29tbWVudDc0Nzg5MTg1NA==,9599,simonw,2020-12-18T06:14:09Z,2020-12-18T06:14:15Z,OWNER,"This is a classic challenge in permissions systems. If I want Datasette to be able to handle thousands of tables I need a reasonable solution for it.

Twitter conversation: https://twitter.com/simonw/status/1339791768842248192","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770598024,Efficiently calculate list of databases/tables a user can view,
https://github.com/simonw/datasette/issues/1150#issuecomment-747864831,https://api.github.com/repos/simonw/datasette/issues/1150,747864831,MDEyOklzc3VlQ29tbWVudDc0Nzg2NDgzMQ==,9599,simonw,2020-12-18T04:46:18Z,2020-12-18T04:46:18Z,OWNER,"The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75

A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747864080,https://api.github.com/repos/simonw/datasette/issues/1150,747864080,MDEyOklzc3VlQ29tbWVudDc0Nzg2NDA4MA==,9599,simonw,2020-12-18T04:43:29Z,2020-12-18T04:43:29Z,OWNER,"I may be overthinking that problem. Many queries are fast in SQLite. If a Datasette instance has 1,000 connected tables will even that be a performance problem for permission checks? I should benchmark to find out.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747862001,https://api.github.com/repos/simonw/datasette/issues/1150,747862001,MDEyOklzc3VlQ29tbWVudDc0Nzg2MjAwMQ==,9599,simonw,2020-12-18T04:35:34Z,2020-12-18T04:35:34Z,OWNER,"I do need to solve the permissions problem properly though, because one of the goals of this system is to provide a paginated, searchable list of databases and tables for the homepage of the instance - #991.

As such, the homepage will need to be able to display only the tables and databases that the user has permission to view.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747861556,https://api.github.com/repos/simonw/datasette/issues/1150,747861556,MDEyOklzc3VlQ29tbWVudDc0Nzg2MTU1Ng==,9599,simonw,2020-12-18T04:33:45Z,2020-12-18T04:33:45Z,OWNER,"One solution on permissions: if Datasette had an efficient way of saying ""list the tables that this user has access to"" I could use that as a filter any time the user views the schema information. The implementation could be tricky though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747861357,https://api.github.com/repos/simonw/datasette/issues/1150,747861357,MDEyOklzc3VlQ29tbWVudDc0Nzg2MTM1Nw==,9599,simonw,2020-12-18T04:32:52Z,2020-12-18T04:32:52Z,OWNER,"I need to figure out how this will interact with Datasette permissions.

If some tables are private, but others are public, should users be able to see the private tables listed in the schema metadata?

If not, how can that mechanism work?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747833639,https://api.github.com/repos/simonw/datasette/issues/1150,747833639,MDEyOklzc3VlQ29tbWVudDc0NzgzMzYzOQ==,9599,simonw,2020-12-18T02:49:40Z,2020-12-18T03:52:12Z,OWNER,"I'm going to use five tables to start off with:

- `databases` - a list of databases. Each one has a `name`, `path` (if it's on disk), `is_memory`, `schema_version`
- `tables` - a list of tables. Each row is `database_name`, `table_name`, `sql` (the create table statement) - may add more tables in the future, in particular maybe a `last_row_count` to cache results of counting the rows.
- `columns` - a list of columns. It's the output of `pragma_table_xinfo` with the `database_name` and `table_name` columns added at the beginning.
- `foreign_keys` - a list of foreign keys - `pragma_foreign_key_list` output plus `database_name` and `table_name`.
- `indexes` - a list of indexes - `pragma_table_xinfo` output plus `database_name` and `table_name`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747847405,https://api.github.com/repos/simonw/datasette/issues/1150,747847405,MDEyOklzc3VlQ29tbWVudDc0Nzg0NzQwNQ==,9599,simonw,2020-12-18T03:36:04Z,2020-12-18T03:36:04Z,OWNER,I could have another table that stores the combined rows from `sqlite_máster` on every connected database so I have a copy of the schema SQL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747847180,https://api.github.com/repos/simonw/datasette/issues/1150,747847180,MDEyOklzc3VlQ29tbWVudDc0Nzg0NzE4MA==,9599,simonw,2020-12-18T03:35:15Z,2020-12-18T03:35:15Z,OWNER,"Simpler implementation idea: a Datasette method `.refresh_schemas()` which loops through all known databases, checks their schema version and updates the in-memory schemas database if they have changed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747834762,https://api.github.com/repos/simonw/datasette/issues/1150,747834762,MDEyOklzc3VlQ29tbWVudDc0NzgzNDc2Mg==,9599,simonw,2020-12-18T02:53:22Z,2020-12-18T02:53:22Z,OWNER,I think I'm going to have to build this without using the `pragma_x()` SQL functions as they were only added in 3.16 in 2017-01-02 and I've seen plenty of Datasette instances running on older versions of SQLite.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747834462,https://api.github.com/repos/simonw/datasette/issues/1150,747834462,MDEyOklzc3VlQ29tbWVudDc0NzgzNDQ2Mg==,9599,simonw,2020-12-18T02:52:19Z,2020-12-18T02:52:26Z,OWNER,Maintaining this database will be the responsibility of a subclass of `Database` called `_SchemaDatabase` which will be managed by the `Datasette` instance.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747834113,https://api.github.com/repos/simonw/datasette/issues/1150,747834113,MDEyOklzc3VlQ29tbWVudDc0NzgzNDExMw==,9599,simonw,2020-12-18T02:51:13Z,2020-12-18T02:51:20Z,OWNER,"SQLite uses `indexes` rather than `indices` as the plural, so I'll go with that: https://sqlite.org/lang_createindex.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747809670,https://api.github.com/repos/simonw/datasette/issues/1150,747809670,MDEyOklzc3VlQ29tbWVudDc0NzgwOTY3MA==,9599,simonw,2020-12-18T01:29:30Z,2020-12-18T01:29:30Z,OWNER,I've been rediscovering the pattern I already documented in this TIL: https://github.com/simonw/til/blob/main/sqlite/list-all-columns-in-a-database.md#better-alternative-using-a-join,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747807891,https://api.github.com/repos/simonw/datasette/issues/1150,747807891,MDEyOklzc3VlQ29tbWVudDc0NzgwNzg5MQ==,9599,simonw,2020-12-18T01:23:59Z,2020-12-18T01:23:59Z,OWNER,"https://www.sqlite.org/pragma.html#pragfunc says:
>  * This feature is experimental and is subject to change. Further documentation will become available if and when the table-valued functions for PRAGMAs feature becomes officially supported. 
>  * The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747807289,https://api.github.com/repos/simonw/datasette/issues/1150,747807289,MDEyOklzc3VlQ29tbWVudDc0NzgwNzI4OQ==,9599,simonw,2020-12-18T01:22:05Z,2020-12-18T01:22:05Z,OWNER,"Here's a simpler query pattern (not using CTEs so should work on older versions of SQLite) - this one lists all indexes for all tables:
```sql
select
  sqlite_master.name as 'table',
  indexes.*
from
  sqlite_master
  join pragma_index_list(sqlite_master.name) indexes
where
  sqlite_master.type = 'table'
```
https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+%27table%27%2C%0D%0A++indexes.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_index_list%28sqlite_master.name%29+indexes%0D%0Awhere%0D%0A++sqlite_master.type+%3D+%27table%27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747805275,https://api.github.com/repos/simonw/datasette/issues/1150,747805275,MDEyOklzc3VlQ29tbWVudDc0NzgwNTI3NQ==,9599,simonw,2020-12-18T01:15:27Z,2020-12-18T01:16:17Z,OWNER,"This query uses a join to pull foreign key information for every table: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_foreign_key_list%28tables.name%29+foo

```sql
with tables as (
  select
    name
  from
    sqlite_master
  where
    type = 'table'
)
select
  tables.name as 'table',
  foo.*
from
  tables
  join pragma_foreign_key_list(tables.name) foo
```

Same query for `pragma_table_xinfo`: https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%0D%0Aselect%0D%0A++tables.name+as+%27table%27%2C%0D%0A++foo.*%0D%0Afrom%0D%0A++tables%0D%0A++join+pragma_table_xinfo%28tables.name%29+foo","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747804254,https://api.github.com/repos/simonw/datasette/issues/1150,747804254,MDEyOklzc3VlQ29tbWVudDc0NzgwNDI1NA==,9599,simonw,2020-12-18T01:12:13Z,2020-12-18T01:12:13Z,OWNER,"Prototype: https://latest.datasette.io/fixtures?sql=select+%27facetable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27facetable%27%29%0D%0Aunion%0D%0Aselect+%27searchable%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27searchable%27%29%0D%0Aunion%0D%0Aselect+%27compound_three_primary_keys%27+as+%27table%27%2C+*+from+pragma_table_xinfo%28%27compound_three_primary_keys%27%29

```sql
select 'facetable' as 'table', * from pragma_table_xinfo('facetable')
union
select 'searchable' as 'table', * from pragma_table_xinfo('searchable')
union
select 'compound_three_primary_keys' as 'table', * from pragma_table_xinfo('compound_three_primary_keys')
```


table | cid | name | type | notnull | dflt_value | pk | hidden
-- | -- | -- | -- | -- | -- | -- | --
compound_three_primary_keys | 0 | pk1 | varchar(30) | 0 |   | 1 | 0
compound_three_primary_keys | 1 | pk2 | varchar(30) | 0 |   | 2 | 0
compound_three_primary_keys | 2 | pk3 | varchar(30) | 0 |   | 3 | 0
compound_three_primary_keys | 3 | content | text | 0 |   | 0 | 0
facetable | 0 | pk | integer | 0 |   | 1 | 0
facetable | 1 | created | text | 0 |   | 0 | 0
facetable | 2 | planet_int | integer | 0 |   | 0 | 0
facetable | 3 | on_earth | integer | 0 |   | 0 | 0
facetable | 4 | state | text | 0 |   | 0 | 0
facetable | 5 | city_id | integer | 0 |   | 0 | 0
facetable | 6 | neighborhood | text | 0 |   | 0 | 0
facetable | 7 | tags | text | 0 |   | 0 | 0
facetable | 8 | complex_array | text | 0 |   | 0 | 0
facetable | 9 | distinct_some_null |   | 0 |   | 0 | 0
searchable | 0 | pk | integer | 0 |   | 1 | 0
searchable | 1 | text1 | text | 0 |   | 0 | 0
searchable | 2 | text2 | text | 0 |   | 0 | 0
searchable | 3 | name with . and spaces | text | 0 |   | 0 | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747803268,https://api.github.com/repos/simonw/datasette/issues/1150,747803268,MDEyOklzc3VlQ29tbWVudDc0NzgwMzI2OA==,9599,simonw,2020-12-18T01:08:40Z,2020-12-18T01:08:40Z,OWNER,"Next step: design a schema for the in-memory database table that exposes all of the tables.

I want to support things like:

- Show me all of the tables
- Show me the columns in a table
- Show me all tables that contain a `tags` column
- Show me the indexes
- Show me every table configured for full-text search

Maybe a starting point would be to build concrete tables using the results of things like `PRAGMA foreign_key_list(table)` and `PRAGMA table_xinfo(table)` - note though that `table_xinfo` is SQLite 3.26.0 or higher, as shown here: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/datasette/utils/__init__.py#L563-L579","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1151#issuecomment-747801751,https://api.github.com/repos/simonw/datasette/issues/1151,747801751,MDEyOklzc3VlQ29tbWVudDc0NzgwMTc1MQ==,9599,simonw,2020-12-18T01:03:39Z,2020-12-18T01:03:39Z,OWNER,"This feature is illustrated by the tests: https://github.com/simonw/datasette/blob/5e9895c67f08e9f42acedd3d6d29512ac446e15f/tests/test_internals_database.py#L469-L496

I added new documentation for the `Datasette()` constructor here as well: https://docs.datasette.io/en/latest/internals.html#database-ds-path-none-is-mutable-false-is-memory-false-memory-name-none","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747801084,https://api.github.com/repos/simonw/datasette/issues/1151,747801084,MDEyOklzc3VlQ29tbWVudDc0NzgwMTA4NA==,9599,simonw,2020-12-18T01:01:26Z,2020-12-18T01:01:26Z,OWNER,"I tested this with a one-off plugin and it worked!

```python
from datasette import hookimpl
from datasette.database import Database


@hookimpl
def startup(datasette):
    datasette.add_database(""statistics"", Database(
        datasette,
        memory_name=""statistics""
    ))
```
This created a `/statistics` database when I ran `datasette` - and if I installed https://github.com/simonw/datasette-write I could then create tables in it which persisted until I restarted the server.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747784199,https://api.github.com/repos/simonw/datasette/issues/1151,747784199,MDEyOklzc3VlQ29tbWVudDc0Nzc4NDE5OQ==,9599,simonw,2020-12-18T00:09:36Z,2020-12-18T00:09:36Z,OWNER,"Is it possible to connect to a memory database in read-only mode?

`file:foo?mode=memory&cache=shared&mode=ro` isn't valid because it features `mode=` more than once.

https://stackoverflow.com/a/40548682 suggests using `PRAGMA query_only` on the connection instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747775245,https://api.github.com/repos/simonw/datasette/issues/1151,747775245,MDEyOklzc3VlQ29tbWVudDc0Nzc3NTI0NQ==,9599,simonw,2020-12-17T23:43:41Z,2020-12-17T23:56:27Z,OWNER,"I'm going to add an argument to the `Database()` constructor which means ""connect to named in-memory database called X"".

```python
db = Database(ds, memory_name=""datasette"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747779056,https://api.github.com/repos/simonw/datasette/issues/1151,747779056,MDEyOklzc3VlQ29tbWVudDc0Nzc3OTA1Ng==,9599,simonw,2020-12-17T23:55:57Z,2020-12-17T23:55:57Z,OWNER,Wait I do use it - if you run `datasette --memory` - which is useful for trying things out in SQL that doesn't need to run against a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747775792,https://api.github.com/repos/simonw/datasette/issues/1151,747775792,MDEyOklzc3VlQ29tbWVudDc0Nzc3NTc5Mg==,9599,simonw,2020-12-17T23:45:20Z,2020-12-17T23:45:20Z,OWNER,"Do I use the current `is_memory=` boolean anywhere at the moment?

https://ripgrep.datasette.io/-/ripgrep?pattern=is_memory - doesn't look like it.

I may remove that feature, since it's not actually useful, and replace it with a mechanism for creating shared named memory databases instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747774855,https://api.github.com/repos/simonw/datasette/issues/1151,747774855,MDEyOklzc3VlQ29tbWVudDc0Nzc3NDg1NQ==,9599,simonw,2020-12-17T23:42:34Z,2020-12-17T23:42:34Z,OWNER,"This worked as a prototype:
```diff
diff --git a/datasette/database.py b/datasette/database.py
index 412e0c5..a90e617 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -24,11 +24,12 @@ connections = threading.local()
 
 
 class Database:
-    def __init__(self, ds, path=None, is_mutable=False, is_memory=False):
+    def __init__(self, ds, path=None, is_mutable=False, is_memory=False, uri=None):
         self.ds = ds
         self.path = path
         self.is_mutable = is_mutable
         self.is_memory = is_memory
+        self.uri = uri
         self.hash = None
         self.cached_size = None
         self.cached_table_counts = None
@@ -46,6 +47,8 @@ class Database:
                 }
 
     def connect(self, write=False):
+        if self.uri:
+            return sqlite3.connect(self.uri, uri=True, check_same_thread=False)
         if self.is_memory:
             return sqlite3.connect("":memory:"")
         # mode=ro or immutable=1?
```
Then in `ipython`:
```
from datasette.app import Datasette
from datasette.database import Database
ds = Datasette([])
db = Database(ds, uri=""file:datasette?mode=memory&cache=shared"", is_memory=True)
await db.execute_write(""create table foo (bar text)"")
await db.table_names()
# Outputs [""foo""]
db2 = Database(ds, uri=""file:datasette?mode=memory&cache=shared"", is_memory=True)
await db2.table_names()
# Also outputs [""foo""]
```

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747770581,https://api.github.com/repos/simonw/datasette/issues/1151,747770581,MDEyOklzc3VlQ29tbWVudDc0Nzc3MDU4MQ==,9599,simonw,2020-12-17T23:31:18Z,2020-12-17T23:32:07Z,OWNER,"This works in `ipython`:
```
In [1]: import sqlite3

In [2]: c1 = sqlite3.connect(""file:datasette?mode=memory&cache=shared"", uri=True)

In [3]: c2 = sqlite3.connect(""file:datasette?mode=memory&cache=shared"", uri=True)

In [4]: c1.executescript(""CREATE TABLE hello (world TEXT)"")
Out[4]: <sqlite3.Cursor at 0x1104addc0>

In [5]: c1.execute(""select * from sqlite_master"").fetchall()
Out[5]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

In [6]: c2.execute(""select * from sqlite_master"").fetchall()
Out[6]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

In [7]: c3 = sqlite3.connect(""file:datasette?mode=memory&cache=shared"", uri=True)

In [9]: c3.execute(""select * from sqlite_master"").fetchall()
Out[9]: [('table', 'hello', 'hello', 2, 'CREATE TABLE hello (world TEXT)')]

In [10]: c4 = sqlite3.connect(""file:datasette?mode=memory"", uri=True)

In [11]: c4.execute(""select * from sqlite_master"").fetchall()
Out[11]: []
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747770082,https://api.github.com/repos/simonw/datasette/issues/1151,747770082,MDEyOklzc3VlQ29tbWVudDc0Nzc3MDA4Mg==,9599,simonw,2020-12-17T23:29:53Z,2020-12-17T23:29:53Z,OWNER,I'm going to try with `file:datasette?mode=memory&cache=shared`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1151#issuecomment-747769830,https://api.github.com/repos/simonw/datasette/issues/1151,747769830,MDEyOklzc3VlQ29tbWVudDc0Nzc2OTgzMA==,9599,simonw,2020-12-17T23:29:08Z,2020-12-17T23:29:08Z,OWNER,"https://sqlite.org/inmemorydb.html

>  The database ceases to exist as soon as the database connection is closed. Every :memory: database is distinct from every other. So, opening two database connections each with the filename "":memory:"" will create two independent in-memory databases.
>
> [...]
>
> The special `"":memory:""` filename also works when using URI filenames. For example:
>
>      rc = sqlite3_open(""file::memory:"", &db);
>
> [...]
>
>  However, the same in-memory database can be opened by two or more database connections as follows:
>
>      rc = sqlite3_open(""file::memory:?cache=shared"", &db);
>
> [...]
> If two or more distinct but shareable in-memory databases are needed in a single process, then the mode=memory query parameter can be used with a URI filename to create a named in-memory database:
>
>     rc = sqlite3_open(""file:memdb1?mode=memory&cache=shared"", &db);
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770448622,Database class mechanism for cross-connection in-memory databases,
https://github.com/simonw/datasette/issues/1150#issuecomment-747768112,https://api.github.com/repos/simonw/datasette/issues/1150,747768112,MDEyOklzc3VlQ29tbWVudDc0Nzc2ODExMg==,9599,simonw,2020-12-17T23:25:21Z,2020-12-17T23:25:21Z,OWNER,"Next challenge: figure out how to use the `Database` class from https://github.com/simonw/datasette/blob/0.53/datasette/database.py for an in-memory database which persists data for the duration of the lifetime of the server, and allows access to that in-memory database from multiple threads in a way that lets them see each other's changes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747767598,https://api.github.com/repos/simonw/datasette/issues/1150,747767598,MDEyOklzc3VlQ29tbWVudDc0Nzc2NzU5OA==,9599,simonw,2020-12-17T23:24:03Z,2020-12-17T23:24:03Z,OWNER,"I'm going to assume that even the heaviest user will have trouble going beyond a few hundred database files, so this is fine.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747767499,https://api.github.com/repos/simonw/datasette/issues/1150,747767499,MDEyOklzc3VlQ29tbWVudDc0Nzc2NzQ5OQ==,9599,simonw,2020-12-17T23:23:44Z,2020-12-17T23:23:44Z,OWNER,Grabbing the schema version of 380 files in the root directory takes 70ms.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747767055,https://api.github.com/repos/simonw/datasette/issues/1150,747767055,MDEyOklzc3VlQ29tbWVudDc0Nzc2NzA1NQ==,9599,simonw,2020-12-17T23:22:41Z,2020-12-17T23:22:41Z,OWNER,"It's just recursion that's expensive. I created 380 empty SQLite databases in a folder and timed `list(pathlib.Path(""/tmp"").glob(""*.db""));` and it took 0.002s.

So maybe I tell users that all SQLite databases have to be in the root folder.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747766310,https://api.github.com/repos/simonw/datasette/issues/1150,747766310,MDEyOklzc3VlQ29tbWVudDc0Nzc2NjMxMA==,9599,simonw,2020-12-17T23:20:49Z,2020-12-17T23:20:49Z,OWNER,"I tried against my entire `~/Development/Dropbox` folder - deeply nested with 381 SQLite database files in sub-folders - and it took 25s! But it turned out 23.9s of that was the call to `pathlib.Path(""/Users/simon/Dropbox/Development"").glob('**/*.db')`.

So it looks like connecting to a SQLite database file and getting the schema version is extremely fast. Scanning directories is slower.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747764712,https://api.github.com/repos/simonw/datasette/issues/1150,747764712,MDEyOklzc3VlQ29tbWVudDc0Nzc2NDcxMg==,9599,simonw,2020-12-17T23:16:31Z,2020-12-17T23:16:31Z,OWNER,"Quick micro-benchmark, run against a folder with 46 database files adding up to 1.4GB total:
```python
import pathlib, sqlite3, time

paths = list(pathlib.Path(""."").glob('*.db'))

def schema_version(path):
    db = sqlite3.connect(path)
    version = db.execute(""PRAGMA schema_version"").fetchall()[0]
    db.close()
    return version

def all():
    versions = {}
    for path in paths:
        versions[path.name] = schema_version(path)
    return versions

start = time.time(); all(); print(time.time() - start)
# 0.012346982955932617
```
So that's 12ms.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747754229,https://api.github.com/repos/simonw/datasette/issues/1150,747754229,MDEyOklzc3VlQ29tbWVudDc0Nzc1NDIyOQ==,9599,simonw,2020-12-17T23:04:38Z,2020-12-17T23:04:38Z,OWNER,"Open question: will this work for hundreds of database files, or is the overhead of connecting to each of 100 databases in turn to run `PRAGMA schema_version` too high?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/1150#issuecomment-747754082,https://api.github.com/repos/simonw/datasette/issues/1150,747754082,MDEyOklzc3VlQ29tbWVudDc0Nzc1NDA4Mg==,9599,simonw,2020-12-17T23:04:13Z,2020-12-17T23:04:13Z,OWNER,"Pages that need a list of all databases - the index page and /-/databases for example - could trigger a ""check for new directories in the configured directories"" scan.

That scan would run at most once every 5 (n) seconds - the check is triggered if it’s run more recently than that it doesn’t run.

Hopefully this means it could be done as a blocking operation, rather than trying to run it in a thread.

When it runs it scans for *.db or *.sqlite files (maybe one or two other extensions) that it hasn’t seen before. It also checks that the existing list of known database files still exists.

If it finds any new ones it connects to them once to run `.schema`. It also runs `PRAGMA schema_version` on each known database so that it can compare the schema version number to the last one it saw. That's how it detects if there are new tables or if the cached schema needs to be updated.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",770436876,Maintain an in-memory SQLite table of connected databases and their tables,
https://github.com/simonw/datasette/issues/461#issuecomment-747734273,https://api.github.com/repos/simonw/datasette/issues/461,747734273,MDEyOklzc3VlQ29tbWVudDc0NzczNDI3Mw==,9599,simonw,2020-12-17T22:14:46Z,2020-12-17T22:14:46Z,OWNER,"I've been thinking about this a bunch. For Datasette to be useful as a private repository of data (Datasette Library, #417) it's crucial that it can handle a much, much larger number of databases.

This makes me worry about how many connections (and open file handles) it makes sense to have open at one time.

I realize now that this is much less of a problem for private instances. Public instances on the internet could get traffic to any database at any time, so connections could easily get out of control. A private instance with only a few users could instead get away with only opening connections to databases in ""active use"".

This does however make it even more important for Datasette to maintain a cached set of metadata about the tables - which is also needed to power this feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",443021509,Paginate + search for databases/tables on the homepage,
https://github.com/simonw/datasette/issues/1005#issuecomment-747209115,https://api.github.com/repos/simonw/datasette/issues/1005,747209115,MDEyOklzc3VlQ29tbWVudDc0NzIwOTExNQ==,9599,simonw,2020-12-17T05:11:04Z,2020-12-17T05:11:04Z,OWNER,Tracking ticket for the next HTTPX release is https://github.com/encode/httpx/pull/1403,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718259202,Remove xfail tests when new httpx is released,
https://github.com/simonw/datasette/issues/741#issuecomment-747208543,https://api.github.com/repos/simonw/datasette/issues/741,747208543,MDEyOklzc3VlQ29tbWVudDc0NzIwODU0Mw==,9599,simonw,2020-12-17T05:09:03Z,2020-12-17T05:09:03Z,OWNER,I really like this in `datasette-publish-vercel` - I'm definitely going to bring this to the other publish implementations as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/1149#issuecomment-747207787,https://api.github.com/repos/simonw/datasette/issues/1149,747207787,MDEyOklzc3VlQ29tbWVudDc0NzIwNzc4Nw==,9599,simonw,2020-12-17T05:06:16Z,2020-12-17T05:06:16Z,OWNER,"So, an idea: what if Datasette's default CSS applied only to elements with classes - or maybe to childen of a `body class=""datasette""` element? In such a way that you could write your own custom HTML that reused elements of Datasette's CSS - the cog menu styling for example - but only on an opt-in basis?","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769520939,Make it easier to theme Datasette with CSS,
https://github.com/simonw/datasette/issues/1149#issuecomment-747207487,https://api.github.com/repos/simonw/datasette/issues/1149,747207487,MDEyOklzc3VlQ29tbWVudDc0NzIwNzQ4Nw==,9599,simonw,2020-12-17T05:05:08Z,2020-12-17T05:05:08Z,OWNER,"I think what I want is for it to be easy to reuse portions of Datasette's CSS - the bit that styles the cog menu for example - without pulling in the whole thing.

I tried linking in the `<link rel=""stylesheet"" href=""/-/static/app.css"">` stylesheet and the page broke, wildly:

<img width=""1030"" alt=""content__categories__3_rows_and_Datasette_search__pytest"" src=""https://user-images.githubusercontent.com/9599/102446222-2290b600-3fe2-11eb-9d3a-e32ca72d1dac.png"">

That's because Datasette's [built-in CSS](https://github.com/simonw/datasette/blob/0.53/datasette/static/app.css) applies styles directly to a whole bunch of different tags - `body`, `header`, `footer` etc - which means that if you import that stylesheet it can play havoc with the site you have already built.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769520939,Make it easier to theme Datasette with CSS,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747130908,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2,747130908,MDEyOklzc3VlQ29tbWVudDc0NzEzMDkwOA==,231498,khimaros,2020-12-17T00:47:04Z,2020-12-17T00:47:43Z,NONE,"it looks like almost all of the memory consumption is coming from `json.load()`.

another direction here may be to use the new ""Semantic Location History"" data which is already broken down by year and month.

it also provides much more interesting data, such as estimated address, form of travel, etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769376447,killed by oomkiller on large location-history,
https://github.com/dogsheep/google-takeout-to-sqlite/issues/2#issuecomment-747126777,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2,747126777,MDEyOklzc3VlQ29tbWVudDc0NzEyNjc3Nw==,9599,simonw,2020-12-17T00:36:52Z,2020-12-17T00:36:52Z,MEMBER,The memory profiler tricks I used in https://github.com/dogsheep/healthkit-to-sqlite/issues/7 could help figure out what's going on here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769376447,killed by oomkiller on large location-history,
https://github.com/simonw/datasette/issues/675#issuecomment-747070709,https://api.github.com/repos/simonw/datasette/issues/675,747070709,MDEyOklzc3VlQ29tbWVudDc0NzA3MDcwOQ==,9599,simonw,2020-12-16T22:09:15Z,2020-12-16T22:09:15Z,OWNER,"The other way this could work is passing a single argument - the file (or directory) to be copied in - and assuming it should always go in the `/app` root. Something like:

    datasette publish cloudrun my.db --include src/ --include dogsheep-beta.yml

Which would add `/app/src/...` and `/app/dogsheep-beta.yml`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-747068624,https://api.github.com/repos/simonw/datasette/issues/675,747068624,MDEyOklzc3VlQ29tbWVudDc0NzA2ODYyNA==,9599,simonw,2020-12-16T22:04:42Z,2020-12-16T22:04:42Z,OWNER,"I can't just use `COPY /path/to/blah.yml /app` in the `Dockerfile` because it runs on the Google Cloud Build servers, not on the user's laptop - so I need to first copy the files they specify to that temporary directory that gets uploaded to the cloud, then rewrite the `COPY` lines in the `Dockerfile` to copy from there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-747067864,https://api.github.com/repos/simonw/datasette/issues/675,747067864,MDEyOklzc3VlQ29tbWVudDc0NzA2Nzg2NA==,9599,simonw,2020-12-16T22:02:55Z,2020-12-16T22:02:55Z,OWNER,"But since we're already running `COPY . /app` anything that's made it into the temporary directory will get copied into `/app`.

But... I feel the usability of the command will be better if users can use absolute paths on the `target` side:

    datasette publish cloudrun my.db --cp dogsheep-beta.yml /app","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/675#issuecomment-747066629,https://api.github.com/repos/simonw/datasette/issues/675,747066629,MDEyOklzc3VlQ29tbWVudDc0NzA2NjYyOQ==,9599,simonw,2020-12-16T21:59:58Z,2020-12-16T22:00:48Z,OWNER,"Note that `datasette publish cloudrun` uses a working directory of `/app` - so users will need to copy their files into `/app` if that's where they need to live.

https://github.com/simonw/datasette/blob/17cbbb1f7f230b39650afac62dd16476626001b5/datasette/utils/__init__.py#L348-L357","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/simonw/datasette/issues/1148#issuecomment-747065487,https://api.github.com/repos/simonw/datasette/issues/1148,747065487,MDEyOklzc3VlQ29tbWVudDc0NzA2NTQ4Nw==,9599,simonw,2020-12-16T21:57:29Z,2020-12-16T21:57:29Z,OWNER,I filed a new public bug in their issue tracker here: https://github.com/vercel/vercel/issues/5575,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",767561886,Syntax error with + symbol when deployed to Vercel,
https://github.com/simonw/datasette/issues/1148#issuecomment-747062909,https://api.github.com/repos/simonw/datasette/issues/1148,747062909,MDEyOklzc3VlQ29tbWVudDc0NzA2MjkwOQ==,9599,simonw,2020-12-16T21:51:54Z,2020-12-16T21:51:54Z,OWNER,"This is a really frustrating bug with Vercel: https://github.com/simonw/datasette-publish-vercel/issues/28

`+` characters in URLs get translated into spaces before they get to Datasette. They know about the bug and said they were working on a fix a few months ago, but looks like it's still a problem.

A workaround is to avoid `+` and use `-` instead - I think this SQL query does the same thing as yours:

https://aws-partners-singapore.vercel.app/partners?sql=select%0D%0A++A.launch_rank%2C%0D%0A++A.partner_info%0D%0Afrom%0D%0A++summary+A%0D%0A++INNER+JOIN+summary+B+ON+A.launch_rank+%3E%3D+B.launch_rank+-+3%0D%0A++AND+A.launch_rank+-4+%3C%3D+B.launch_rank%0D%0AWHERE%0D%0A++B.%22partner_info%22+LIKE+%27%25Palo+Alto%25%27

```sql
select
  A.launch_rank,
  A.partner_info
from
  summary A
  INNER JOIN summary B ON A.launch_rank >= B.launch_rank - 3
  AND A.launch_rank -4 <= B.launch_rank
WHERE
  B.""partner_info"" LIKE '%Palo Alto%'
```
I've been moving projects from Vercel to Cloud Run when they run into this, but that's not a great situation to be in.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",767561886,Syntax error with + symbol when deployed to Vercel,
https://github.com/simonw/datasette/issues/675#issuecomment-747059277,https://api.github.com/repos/simonw/datasette/issues/675,747059277,MDEyOklzc3VlQ29tbWVudDc0NzA1OTI3Nw==,9599,simonw,2020-12-16T21:43:52Z,2020-12-16T21:43:52Z,OWNER,"It turns out I need this for a couple of projects:

- [datasette-ripgrep](https://github.com/simonw/datasette-ripgrep) needs to ship a whole bunch of source code files up in a known location. I worked around this with a nasty hack involving `--static` but it would be better if I wasn't doing that.
- [dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) uses an additional `dogsheep-beta.yml` configuration file in the project root (a sibling to `metadata.yml`) which needs to be included when publishing - see https://github.com/simonw/datasette.io/issues/21#issuecomment-747058067

I want this for `datasette publish cloudrun`, not just for `datasette package`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",567902704,--cp option for datasette publish and datasette package for shipping additional files and directories,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747034481,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747034481,MDEyOklzc3VlQ29tbWVudDc0NzAzNDQ4MQ==,9599,simonw,2020-12-16T21:17:05Z,2020-12-16T21:17:05Z,MEMBER,I'm just going to add `q` for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747031608,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747031608,MDEyOklzc3VlQ29tbWVudDc0NzAzMTYwOA==,9599,simonw,2020-12-16T21:15:18Z,2020-12-16T21:15:18Z,MEMBER,Should I pass any other details to the `display_sql` here as well?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747030964,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747030964,MDEyOklzc3VlQ29tbWVudDc0NzAzMDk2NA==,9599,simonw,2020-12-16T21:14:54Z,2020-12-16T21:14:54Z,MEMBER,"To do this I'll need the search term to be passed to the `display_sql` SQL query:
https://github.com/dogsheep/dogsheep-beta/blob/4890ec87b5e2ec48940f32c9ad1f5aae25c75a4d/dogsheep_beta/__init__.py#L164-L171","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-747029636,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,747029636,MDEyOklzc3VlQ29tbWVudDc0NzAyOTYzNg==,9599,simonw,2020-12-16T21:14:03Z,2020-12-16T21:14:03Z,MEMBER,"I think I can do this as a cunning trick in `display_sql`. Consider this example query: https://til.simonwillison.net/tils?sql=select%0D%0A++path%2C%0D%0A++snippet%28til_fts%2C+-1%2C+%27b4de2a49c8%27%2C+%278c94a2ed4b%27%2C+%27...%27%2C+60%29+as+snippet%0D%0Afrom%0D%0A++til%0D%0A++join+til_fts+on+til.rowid+%3D+til_fts.rowid%0D%0Awhere%0D%0A++til_fts+match+escape_fts%28%3Aq%29%0D%0A++and+path+%3D+%27asgi_lifespan-test-httpx.md%27%0D%0A&q=pytest

```sql
select
  path,
  snippet(til_fts, -1, 'b4de2a49c8', '8c94a2ed4b', '...', 60) as snippet
from
  til
  join til_fts on til.rowid = til_fts.rowid
where
  til_fts match escape_fts(:q)
  and path = 'asgi_lifespan-test-httpx.md'
```
The `and path = 'asgi_lifespan-test-httpx.md'` bit means we only get back a specific document - but the snippet highlighting is applied to it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets,
https://github.com/simonw/datasette/issues/1143#issuecomment-746827083,https://api.github.com/repos/simonw/datasette/issues/1143,746827083,MDEyOklzc3VlQ29tbWVudDc0NjgyNzA4Mw==,9599,simonw,2020-12-16T18:56:07Z,2020-12-16T18:56:07Z,OWNER,"I think the right way to do this is to support multiple optional `--cors-origin=` pattern values, like you suggested.","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746735889,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58,746735889,MDEyOklzc3VlQ29tbWVudDc0NjczNTg4OQ==,9599,simonw,2020-12-16T17:59:50Z,2020-12-16T17:59:50Z,MEMBER,"I don't want to add a full HTML parser (like BeautifulSoup) as a dependency for this feature. Since the HTML comes from a single, trusted source (GitHub) I could probably handle this using [regular expressions](https://stackoverflow.com/a/1732454).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769150394,Readme HTML has broken internal links,
https://github.com/dogsheep/github-to-sqlite/issues/58#issuecomment-746734412,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58,746734412,MDEyOklzc3VlQ29tbWVudDc0NjczNDQxMg==,9599,simonw,2020-12-16T17:58:56Z,2020-12-16T17:58:56Z,MEMBER,"I'm going to rewrite those `<a href=""#filtering-tables"">` links to `<a href=""#user-content-filtering-tables"">` - but only if a corresponding `id=""user-content-filtering-tables""` element exists.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",769150394,Readme HTML has broken internal links,
https://github.com/simonw/datasette/issues/1142#issuecomment-745162571,https://api.github.com/repos/simonw/datasette/issues/1142,745162571,MDEyOklzc3VlQ29tbWVudDc0NTE2MjU3MQ==,6622733,nitinpaultifr,2020-12-15T09:22:58Z,2020-12-15T09:22:58Z,NONE,"You're right, probably more straightforward to have the links for JSON. I was imagining to toggle the `href` for the 'Export JSON' link (button) to the selected shape, but it'll probably be needlessly complex in the end.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1143#issuecomment-744618787,https://api.github.com/repos/simonw/datasette/issues/1143,744618787,MDEyOklzc3VlQ29tbWVudDc0NDYxODc4Nw==,114388,yurivish,2020-12-14T18:15:00Z,2020-12-15T02:21:53Z,NONE,"From a quick look at the README, it does seem to do everything I need, thanks!

I think the argument for inclusion in core is to lower the chances of unwanted data access. A local server can be accessed by anybody who can make an HTTP request to your computer regardless of CORS rules, but the default `*` rule additionally opens up access to the local instance to any website you visit while it is running. 

That's probably not what people typically intend, particularly when the data is of a sensitive nature. A default of requiring the user to specify the origin (allowing `*` but encouraging a narrower scope) would solve this problem entirely, I think.


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/simonw/datasette/issues/1143#issuecomment-744757558,https://api.github.com/repos/simonw/datasette/issues/1143,744757558,MDEyOklzc3VlQ29tbWVudDc0NDc1NzU1OA==,9599,simonw,2020-12-14T22:42:10Z,2020-12-14T22:42:10Z,OWNER,"This may involve a breaking change to the CLI settings interface, so I'm adding this to the 1.0 milestone.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/simonw/datasette/issues/1143#issuecomment-744756861,https://api.github.com/repos/simonw/datasette/issues/1143,744756861,MDEyOklzc3VlQ29tbWVudDc0NDc1Njg2MQ==,9599,simonw,2020-12-14T22:40:28Z,2020-12-14T22:40:28Z,OWNER,"That's a very convincing argument. I'm keen on making sure Datasette is ""secure by default"" so you're right, encouraging finely grains CORS rules in core rather than leaving that to a plugin sounds like the right call.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/simonw/datasette/issues/1142#issuecomment-744576894,https://api.github.com/repos/simonw/datasette/issues/1142,744576894,MDEyOklzc3VlQ29tbWVudDc0NDU3Njg5NA==,9599,simonw,2020-12-14T17:03:13Z,2020-12-14T17:03:13Z,OWNER,"I'm not sure about the radio boxes for JSON, just because you can't right-click on a radio box and copy it to your clipboard like you can with links. Worth trying it out though.

The radio boxes for that CSV option are definitely the right way to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1142#issuecomment-744563209,https://api.github.com/repos/simonw/datasette/issues/1142,744563209,MDEyOklzc3VlQ29tbWVudDc0NDU2MzIwOQ==,9599,simonw,2020-12-14T16:41:11Z,2020-12-14T16:41:11Z,OWNER,"To check out and start the server:

    /tmp % git clone git@github.com:nitinpaul/datasette
    Cloning into 'datasette'...
    remote: Enumerating objects: 124, done.
    # ...
    datasette % python3 -m venv venv
    datasette % source venv/bin/activate
    (venv) datasette % pip install -e '.[test]'
    Obtaining file:///private/tmp/datasette
    Collecting asgiref<3.4.0,>=3.2.10
    Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
    # ...
    (venv) datasette % datasette
    INFO:     Started server process [24002]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

And to run the tests:

    (venv) datasette % pytest
    ======================================================================== test session starts ========================================================================
    platform darwin -- Python 3.9.1, pytest-6.1.2, py-1.10.0, pluggy-0.13.1
    SQLite: 3.34.0
    rootdir: /private/tmp/datasette, configfile: pytest.ini
    plugins: asyncio-0.14.0, timeout-1.4.2
    collected 841 items                                                                                                                                                 

    tests/test_package.py ..                                                                                                                                      [  0%]
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1142#issuecomment-744522099,https://api.github.com/repos/simonw/datasette/issues/1142,744522099,MDEyOklzc3VlQ29tbWVudDc0NDUyMjA5OQ==,6622733,nitinpaultifr,2020-12-14T15:37:47Z,2020-12-14T15:37:47Z,NONE,"Alright I could give it a try! This might be a stupid question, can you tell me how to run the server from my fork? So that I can test the changes?","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1144#issuecomment-744489028,https://api.github.com/repos/simonw/datasette/issues/1144,744489028,MDEyOklzc3VlQ29tbWVudDc0NDQ4OTAyOA==,475613,MarkusH,2020-12-14T14:47:11Z,2020-12-14T14:47:11Z,NONE,"Thanks for opening the issue, @simonw. Let me elaborate on my Tweets.

[datasette-chartjs](https://github.com/MarkusH/datasette-chartjs) provides drop down lists to pick the chart visualization (e.g. bar, line, doughnut, pie, ...) as well as the column used for the ""x axis"" (e.g. time).

A user can change the values on-demand. The chart will be redrawn w/o querying the database again.

However, if a user wants to change the underlying query, they will use the SQL field provided by datasette or any of the other datasette built-in features to amend a query. In order to maintain a user's selections for the plugin, datasette-chartjs copies some parts of [datasette-vega](https://github.com/simonw/datasette-vega) which persist the chosen visualization and column in the hash part of a URL (the stuff behind the `#`). The plugin load the config from the hash upon initialization on the next page and use it accordingly.

Additionally, datasette-vega and datasette-chartjs need to make sure to include the hash in all links and forms that cause a reload of the page. This is, such that the config persists between clicks.

This ticket is about moving thes parts into datasette that provide the functionality to do so. This includes:

1. a way to load config options with a given prefix from the current URL hash
1. a way to update the current URL hash with a new config value or a bunch of config options
1. updating all necessary links and forms on the current page to include the URL hash whenever its updated
1. to prevent leaking config options to external pages, only ""internal"" links should be updated

There's another, optional, feature that we might want to think about during the design phase: the scope of the config. Links within a datasette instance have 1 of 3 scopes:

1. global, for the whole datasette project
1. database, for all tables in a database
1. table, only for a table within a database

When updating the links and forms as pointed out in 3. above, it might be worth considering which links need to be updated. I could imagine a plugin that wants to persist some setting across all tables within a database but another setting only within a table.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",765637324,JavaScript to help plugins interact with the fragment part of the URL,
https://github.com/simonw/datasette/pull/1145#issuecomment-744475543,https://api.github.com/repos/simonw/datasette/issues/1145,744475543,MDEyOklzc3VlQ29tbWVudDc0NDQ3NTU0Mw==,22429695,codecov[bot],2020-12-14T14:26:25Z,2020-12-14T14:26:25Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=h1) Report
> Merging [#1145](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=desc) (a8588f9) into [main](https://codecov.io/gh/simonw/datasette/commit/0c616f732cee79db80cad830917666f41b344262?el=desc) (0c616f7) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1145/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1145   +/-   ##
=======================================
  Coverage   91.41%   91.41%           
=======================================
  Files          31       31           
  Lines        3881     3881           
=======================================
  Hits         3548     3548           
  Misses        333      333           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=footer). Last update [0c616f7...a8588f9](https://codecov.io/gh/simonw/datasette/pull/1145?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",766494367,"Update pytest requirement from <6.2.0,>=5.2.2 to >=5.2.2,<6.3.0",
https://github.com/simonw/datasette/issues/276#issuecomment-744461856,https://api.github.com/repos/simonw/datasette/issues/276,744461856,MDEyOklzc3VlQ29tbWVudDc0NDQ2MTg1Ng==,296686,robintw,2020-12-14T14:04:57Z,2020-12-14T14:04:57Z,NONE,"I'm looking into using datasette with a database with spatialite geometry columns, and came across this issue. Has there been any progress on this since 2018?

In one of my tables I'm just storing lat/lon points in a spatialite point geometry, and I've managed to make datasette-cluster-map display the points by extracting the lat and lon in SQL - using something like `select ... ST_X(location) as longitude, ST_Y(location) as latitude from Blah`. Something more 'built-in' would be great though - particularly for the tables I have that store more complex geometries.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324835838,Handle spatialite geometry columns better,
https://github.com/simonw/datasette/issues/1142#issuecomment-744251252,https://api.github.com/repos/simonw/datasette/issues/1142,744251252,MDEyOklzc3VlQ29tbWVudDc0NDI1MTI1Mg==,9599,simonw,2020-12-14T07:56:38Z,2020-12-14T07:56:38Z,OWNER,That's a really solid design for this! I'd be very happy to review a pull request - you should be able to implement this with just template edits and some CSS changes I think.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1143#issuecomment-744249157,https://api.github.com/repos/simonw/datasette/issues/1143,744249157,MDEyOklzc3VlQ29tbWVudDc0NDI0OTE1Nw==,9599,simonw,2020-12-14T07:53:15Z,2020-12-14T07:53:15Z,OWNER,"Does this plugin do everything you need? https://github.com/simonw/datasette-cors

I'm open to arguments as to why this should be in core rather than in a plugin - I'm on the fence about that at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",764059235,"More flexible CORS support in core, to encourage good security practices",
https://github.com/simonw/datasette/issues/741#issuecomment-744142692,https://api.github.com/repos/simonw/datasette/issues/741,744142692,MDEyOklzc3VlQ29tbWVudDc0NDE0MjY5Mg==,9599,simonw,2020-12-14T03:28:56Z,2020-12-14T03:28:56Z,OWNER,I'm going to try this out on `datasette-publish-vercel` first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""",
https://github.com/simonw/datasette/issues/983#issuecomment-744066249,https://api.github.com/repos/simonw/datasette/issues/983,744066249,MDEyOklzc3VlQ29tbWVudDc0NDA2NjI0OQ==,9599,simonw,2020-12-13T20:47:52Z,2020-12-13T20:47:52Z,OWNER,"@yozlet just spotted this comment. Wow that is interesting!

With the right plugin hooks on the page (see also #987) one relatively simple way to do that could be with bookmarklets - users could install bookmarklets which, when executed against a Datasette page in their browser, use the existing JavaScript plugin integration points to add all kinds of functionality.

Doing full sandboxing is certainly daunting, but it looks like Figma figured it out so TIL it's technically feasible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",712260429,JavaScript plugin hooks mechanism similar to pluggy,
https://github.com/simonw/datasette/pull/1031#issuecomment-744003454,https://api.github.com/repos/simonw/datasette/issues/1031,744003454,MDEyOklzc3VlQ29tbWVudDc0NDAwMzQ1NA==,299380,frankier,2020-12-13T12:52:56Z,2020-12-13T12:52:56Z,NONE,"Please let me know if there's anything I can do to help get this merged.

This is causing problems for me because it means when I build my Docker image my databases aren't considered immutable, which I would like them to be so that a download link is produced.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed,
https://github.com/simonw/datasette/issues/1142#issuecomment-743998792,https://api.github.com/repos/simonw/datasette/issues/1142,743998792,MDEyOklzc3VlQ29tbWVudDc0Mzk5ODc5Mg==,6622733,nitinpaultifr,2020-12-13T12:14:06Z,2020-12-13T12:14:06Z,NONE,"Agreed, it would definitely provide better controls. However, I do feel it makes for a bit of inconsistent UX for the 'Advanced export' section, with links to download for JSON, checkboxes and radio buttons + button to download for CSV. Do you think this example makes the UX a bit nicer/consistent?

![Screenshot 2020-12-13 at 5 38 43 PM](https://user-images.githubusercontent.com/6622733/102011444-1dc1cd00-3d6a-11eb-9e38-5af198161e80.png)

I could give it a try if you'd like but I've never contributed to an actual project!
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743966801,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743966801,MDEyOklzc3VlQ29tbWVudDc0Mzk2NjgwMQ==,9599,simonw,2020-12-13T07:25:23Z,2020-12-13T07:25:23Z,OWNER,"CLI documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#analyzing-tables

Python library documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#analyzing-a-column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/sqlite-utils/pull/203#issuecomment-743966289,https://api.github.com/repos/simonw/sqlite-utils/issues/203,743966289,MDEyOklzc3VlQ29tbWVudDc0Mzk2NjI4OQ==,9599,simonw,2020-12-13T07:20:51Z,2020-12-13T07:20:51Z,OWNER,Sorry for not reviewing this yet! I'll try to carve out time to look at it in the next few days.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743384829,changes to allow for compound foreign keys,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743956666,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743956666,MDEyOklzc3VlQ29tbWVudDc0Mzk1NjY2Ng==,9599,simonw,2020-12-13T05:44:49Z,2020-12-13T05:44:49Z,OWNER,"Example output:
```
% sqlite-utils analyze-tables github.db tags             
tags.repo: (1/3)

  Total rows: 261
  Null rows: 0
  Blank rows: 0

  Distinct values: 14

  Most common:
    88: 107914493
    75: 140912432
    27: 206156866
    21: 207052882
    17: 197431109
    8: 197882382
    5: 256834907
    5: 205429375
    4: 248903544
    3: 206202864

  Least common:
    1: 209590345
    2: 206649770
    2: 303218369
    3: 206202864
    3: 213286752
    4: 248903544
    5: 205429375
    5: 256834907
    8: 197882382
    17: 197431109

tags.name: (2/3)

  Total rows: 261
  Null rows: 0
  Blank rows: 0

  Distinct values: 175

  Most common:
    10: 0.2
    9: 0.1
    7: 0.3
    6: 0.4
    5: 0.7
    5: 0.5
    5: 0.1a
    4: 0.9
    4: 0.8
    4: 0.6

  Least common:
    1: 0.1.1
    1: 0.11.1
    1: 0.1a2
    1: 0.20.1
    1: 0.21.1
    1: 0.21.2
    1: 0.21.3
    1: 0.22
    1: 0.22.1
    1: 0.23

tags.sha: (3/3)

  Total rows: 261
  Null rows: 0
  Blank rows: 0

  Distinct values: 261
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/datasette/issues/1142#issuecomment-743913004,https://api.github.com/repos/simonw/datasette/issues/1142,743913004,MDEyOklzc3VlQ29tbWVudDc0MzkxMzAwNA==,9599,simonw,2020-12-12T22:17:46Z,2020-12-12T22:17:46Z,OWNER,"You're actually choosing between two options here: the 100 rows you can see on the screen, or the x,000 rows that match the current query.

Maybe a radio box would be more obvious?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1142#issuecomment-743912875,https://api.github.com/repos/simonw/datasette/issues/1142,743912875,MDEyOklzc3VlQ29tbWVudDc0MzkxMjg3NQ==,9599,simonw,2020-12-12T22:16:38Z,2020-12-12T22:16:38Z,OWNER,"Yeah, maybe with the number of rows to make it completely clear. `Include all 2,455 rows` perhaps.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/datasette/issues/1142#issuecomment-743732440,https://api.github.com/repos/simonw/datasette/issues/1142,743732440,MDEyOklzc3VlQ29tbWVudDc0MzczMjQ0MA==,6622733,nitinpaultifr,2020-12-12T09:56:40Z,2020-12-12T09:56:40Z,NONE,'Include all rows' seem like a fairly obvious alternative,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763361458,"""Stream all rows"" is not at all obvious",
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708524,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708524,MDEyOklzc3VlQ29tbWVudDc0MzcwODUyNA==,9599,simonw,2020-12-12T05:48:20Z,2020-12-12T05:48:32Z,OWNER,"```
% sqlite-utils analyze-tables ../datasette/fixtures.db facetable --column pk
1/1: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708325,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708325,MDEyOklzc3VlQ29tbWVudDc0MzcwODMyNQ==,9599,simonw,2020-12-12T05:46:27Z,2020-12-12T05:46:27Z,OWNER,"It would be neat if you could optionally specify a subset of columns to analyze, using `-c` or `--column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708169,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708169,MDEyOklzc3VlQ29tbWVudDc0MzcwODE2OQ==,9599,simonw,2020-12-12T05:44:46Z,2020-12-12T05:44:46Z,OWNER,"If there are less than ten values is it worth outputting them twice, once in `most_common` and then in reverse in `least_common`? Feels redundant - I think I should leave `least_common` empty in that case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743708080,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743708080,MDEyOklzc3VlQ29tbWVudDc0MzcwODA4MA==,9599,simonw,2020-12-12T05:43:45Z,2020-12-12T05:43:45Z,OWNER,"CLI output looks like this at the moment, which is bad:
```
 % sqlite-utils analyze-tables ../datasette/fixtures.db facetable
1/10: ColumnDetails(table='facetable', column='pk', total_rows=15, num_null=0, num_blank=0, num_distinct=15, most_common=None, least_common=None)
2/10: ColumnDetails(table='facetable', column='created', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[('2019-01-17 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-14 08:00:00', 4), ('2019-01-16 08:00:00', 3)], least_common=[('2019-01-16 08:00:00', 3), ('2019-01-14 08:00:00', 4), ('2019-01-15 08:00:00', 4), ('2019-01-17 08:00:00', 4)])
3/10: ColumnDetails(table='facetable', column='planet_int', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (2, 1)], least_common=[(2, 1), (1, 14)])
4/10: ColumnDetails(table='facetable', column='on_earth', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[(1, 14), (0, 1)], least_common=[(0, 1), (1, 14)])
5/10: ColumnDetails(table='facetable', column='state', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('CA', 10), ('MI', 4), ('MC', 1)], least_common=[('MC', 1), ('MI', 4), ('CA', 10)])
6/10: ColumnDetails(table='facetable', column='city_id', total_rows=15, num_null=0, num_blank=0, num_distinct=4, most_common=[(1, 6), (3, 4), (2, 4), (4, 1)], least_common=[(4, 1), (2, 4), (3, 4), (1, 6)])
7/10: ColumnDetails(table='facetable', column='neighborhood', total_rows=15, num_null=0, num_blank=0, num_distinct=14, most_common=[('Downtown', 2), ('Tenderloin', 1), ('SOMA', 1), ('Mission', 1), ('Mexicantown', 1), ('Los Feliz', 1), ('Koreatown', 1), ('Hollywood', 1), ('Hayes Valley', 1), ('Greektown', 1)], least_common=[('Arcadia Planitia', 1), ('Bernal Heights', 1), ('Corktown', 1), ('Dogpatch', 1), ('Greektown', 1), ('Hayes Valley', 1), ('Hollywood', 1), ('Koreatown', 1), ('Los Feliz', 1), ('Mexicantown', 1)])
8/10: ColumnDetails(table='facetable', column='tags', total_rows=15, num_null=0, num_blank=0, num_distinct=3, most_common=[('[]', 13), ('[""tag1"", ""tag3""]', 1), ('[""tag1"", ""tag2""]', 1)], least_common=[('[""tag1"", ""tag2""]', 1), ('[""tag1"", ""tag3""]', 1), ('[]', 13)])
9/10: ColumnDetails(table='facetable', column='complex_array', total_rows=15, num_null=0, num_blank=0, num_distinct=2, most_common=[('[]', 14), ('[{""foo"": ""bar""}]', 1)], least_common=[('[{""foo"": ""bar""}]', 1), ('[]', 14)])
10/10: ColumnDetails(table='facetable', column='distinct_some_null', total_rows=15, num_null=13, num_blank=0, num_distinct=2, most_common=[(None, 13), ('two', 1), ('one', 1)], least_common=[('one', 1), ('two', 1), (None, 13)])
(sqlite-utils) sqlite-utils % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/pull/208#issuecomment-743707969,https://api.github.com/repos/simonw/sqlite-utils/issues/208,743707969,MDEyOklzc3VlQ29tbWVudDc0MzcwNzk2OQ==,9599,simonw,2020-12-12T05:42:26Z,2020-12-12T05:43:06Z,OWNER,"Should truncate values in the least/most common JSON array to a sensible length, otherwise you end up with stuff like this:
```json
[
    [
        ""b'\\x00\\x05barry\\x03\\x01\\x02\\x00\\x00\\x03cat\\x03\\x01\\x03\\x00\\x00\\x03dog\\x08\\x01\\x01\\x01\\x03\\x00\\x01\\x03\\x00\\x00\\x07panther\\x05\\x01\\x01\\x02\\x02\\x00\\x01\\x03uma\\x05\\x02\\x01\\x02\\x02\\x00\\x00\\x04sara\\x05\\x02\\x01\\x01\\x02\\x00\\x00\\x05terry\\x08\\x01\\x01\\x01\\x02\\x00\\x01\\x02\\x00\\x00\\x06weasel\\x05\\x02\\x01\\x01\\x03\\x00'"",
        1
    ]
]
```
This example also shows that binary values (like those in `_fts` tables) look a bit weird, but I think I'm OK with that since binary data can't be represented neatly in JSON anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763320133,sqlite-utils analyze-tables command and table.analyze_column() method,
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701697,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743701697,MDEyOklzc3VlQ29tbWVudDc0MzcwMTY5Nw==,9599,simonw,2020-12-12T04:39:51Z,2020-12-12T04:39:51Z,OWNER,"CLI could be:

    sqlite-utils analyze-tables

To analyze all tables or:

    sqlite-utils analyze-tables table1 table2

To analyze specific tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701599,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743701599,MDEyOklzc3VlQ29tbWVudDc0MzcwMTU5OQ==,9599,simonw,2020-12-12T04:38:52Z,2020-12-12T04:39:07Z,OWNER,I'll add a `table.analyze_column(column)` method which is used by the CLI tool - with a note that this is an unstable interface which may change in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/sqlite-utils/issues/207#issuecomment-743701422,https://api.github.com/repos/simonw/sqlite-utils/issues/207,743701422,MDEyOklzc3VlQ29tbWVudDc0MzcwMTQyMg==,9599,simonw,2020-12-12T04:37:14Z,2020-12-12T04:38:25Z,OWNER,"Prototype:
```python
from collections import namedtuple

ColumnDetails = namedtuple(""ColumnDetails"", (""column"", ""num_null"", ""num_blank"", ""num_distinct"", ""most_common"", ""least_common""))

def analyze_column(db, table, column, values=10):
    num_null = db.execute(""select count(*) from [{}] where [{}] is null"".format(table, column)).fetchone()[0]
    num_blank = db.execute(""select count(*) from [{}] where [{}] = ''"".format(table, column)).fetchone()[0]
    num_distinct = db.execute(""select count(distinct [{}]) from [{}]"".format(column, table)).fetchone()[0]
    most_common = None
    least_common = None
    if num_distinct != 1:
        most_common = [(r[0], r[1]) for r in db.execute(
            ""select [{}], count(*) from [{}] group by [{}] order by count(*) desc limit "".format(column, table, column, values)
        ).fetchall()]
        if num_distinct <= values:
            # No need to run the query if it will just return the results in revers order
            least_common = most_common[::-1]
        else:
            least_common = [(r[0], r[1]) for r in db.execute(
                ""select [{}], count(*) from [{}] group by [{}] order by count(*) limit {}"".format(column, table, column, values)
            ).fetchall()]
    return ColumnDetails(column, num_null, num_blank, num_distinct, most_common, least_common)


def analyze_table(db, table):
    for column in db[table].columns:
        details = analyze_column(db, table, column.name)
        print(details)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",763283616,sqlite-utils analyze-tables command,
https://github.com/simonw/datasette/issues/998#issuecomment-743080047,https://api.github.com/repos/simonw/datasette/issues/998,743080047,MDEyOklzc3VlQ29tbWVudDc0MzA4MDA0Nw==,6371750,JBPressac,2020-12-11T09:25:09Z,2020-12-11T09:25:09Z,CONTRIBUTOR,"Hello Simon,
I have a similar problem with horizontal scrollbar display with Datasette version 0.51 and superior for a table with more than 30 rows. With Datasette 0.50, the horizontal scrollbar is displayed, if I upgrade Datasette to 0.51 and superior, the horizontal scrollbar disappears.

Datasette 0.50: horizontal scrollbar

![2020-12-11 10_23_28-CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US](https://user-images.githubusercontent.com/6371750/101885620-a5f17800-3b9a-11eb-8870-654e7d4372ca.png)

Datasette 0.51 and superior: no horizontal scrollbar

![2020-12-11 10_24_55-CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US](https://user-images.githubusercontent.com/6371750/101885782-dfc27e80-3b9a-11eb-9d55-6c9a56227bf2.png)

Thanks,","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page,
https://github.com/simonw/sqlite-utils/issues/205#issuecomment-742737794,https://api.github.com/repos/simonw/sqlite-utils/issues/205,742737794,MDEyOklzc3VlQ29tbWVudDc0MjczNzc5NA==,9599,simonw,2020-12-10T19:18:22Z,2020-12-10T19:18:22Z,OWNER,"Yup, it looks like you're using a window function that was added in SQLite 3.25.0: https://www.sqlite.org/changes.html#version_3_25_0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760960559,"sqlite3.OperationalError: near ""("": syntax error",
https://github.com/simonw/sqlite-utils/issues/205#issuecomment-742299584,https://api.github.com/repos/simonw/sqlite-utils/issues/205,742299584,MDEyOklzc3VlQ29tbWVudDc0MjI5OTU4NA==,765871,kaihendry,2020-12-10T07:24:22Z,2020-12-10T07:24:22Z,NONE,Bumping to ubuntu-20.04 appears to have solved my syntax error. 🤷,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760960559,"sqlite3.OperationalError: near ""("": syntax error",
https://github.com/simonw/datasette/issues/1134#issuecomment-742260116,https://api.github.com/repos/simonw/datasette/issues/1134,742260116,MDEyOklzc3VlQ29tbWVudDc0MjI2MDExNg==,2181410,clausjuhl,2020-12-10T05:57:17Z,2020-12-10T05:57:17Z,NONE,"Hi Simon

Thank you for the quick fix! And glad you like our use of Datasette (launches 1. january 2021). It's a site that currently (more to come) makes all minutes and their annexes from Aarhus City Council and the major committees (1997-2019) available to the public. So we're putting Datasette to good use :)","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 2, ""rocket"": 0, ""eyes"": 0}",760312579,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",
https://github.com/simonw/datasette/issues/1091#issuecomment-741992106,https://api.github.com/repos/simonw/datasette/issues/1091,741992106,MDEyOklzc3VlQ29tbWVudDc0MTk5MjEwNg==,9599,simonw,2020-12-09T19:19:54Z,2020-12-09T20:27:45Z,OWNER,"Could you try removing the `ProxyPassReverse /datasette http://0.0.0.0:8001` line?

My hunch is that `ProxyPassReverse` is rewriting some of the links in the HTML (or maybe in the HTTP headers) in a way that breaks things.

Normally you would need `ProxyPassReverse` to compensate for the underlying application being unable to rewrite its links - but Datasette's `base_url` setting causes Datasette to rewrite all of the links for you, so `ProxyPassReverse` should be unneccessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1134#issuecomment-742024588,https://api.github.com/repos/simonw/datasette/issues/1134,742024588,MDEyOklzc3VlQ29tbWVudDc0MjAyNDU4OA==,9599,simonw,2020-12-09T20:19:59Z,2020-12-09T20:20:33Z,OWNER,https://byraadsarkivet.aarhus.dk/db/cases?_searchmode=raw&_search=sundhedsfrem%2A is an absolutely beautiful example of a themed Datasette! Very excited to show this to people.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760312579,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",
https://github.com/simonw/datasette/issues/1134#issuecomment-742023775,https://api.github.com/repos/simonw/datasette/issues/1134,742023775,MDEyOklzc3VlQ29tbWVudDc0MjAyMzc3NQ==,9599,simonw,2020-12-09T20:18:23Z,2020-12-09T20:18:23Z,OWNER,A fix for this should be available if you upgrade to 0.52.5,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760312579,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",
https://github.com/simonw/datasette/issues/1091#issuecomment-742023541,https://api.github.com/repos/simonw/datasette/issues/1091,742023541,MDEyOklzc3VlQ29tbWVudDc0MjAyMzU0MQ==,9599,simonw,2020-12-09T20:17:54Z,2020-12-09T20:17:54Z,OWNER,OK that is really weird. I'll have another go at replicating this locally.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1136#issuecomment-742023111,https://api.github.com/repos/simonw/datasette/issues/1136,742023111,MDEyOklzc3VlQ29tbWVudDc0MjAyMzExMQ==,9599,simonw,2020-12-09T20:17:02Z,2020-12-09T20:17:02Z,OWNER,Documentation for this procedure is now here: https://docs.datasette.io/en/latest/contributing.html#releasing-bug-fixes-from-a-branch,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742022222,https://api.github.com/repos/simonw/datasette/issues/1136,742022222,MDEyOklzc3VlQ29tbWVudDc0MjAyMjIyMg==,9599,simonw,2020-12-09T20:15:24Z,2020-12-09T20:15:51Z,OWNER,Used this procedure for the first time for 0.52.5 - deploy run here: https://github.com/simonw/datasette/actions/runs/411465648 - PyPI release here: https://pypi.org/project/datasette/0.52.5/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742017622,https://api.github.com/repos/simonw/datasette/issues/1136,742017622,MDEyOklzc3VlQ29tbWVudDc0MjAxNzYyMg==,9599,simonw,2020-12-09T20:06:47Z,2020-12-09T20:06:47Z,OWNER,"Then I can ship the release directly from that branch, creating the tag as part of the release process:

<img width=""1057"" alt=""New_release_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/101681442-ffc93500-3a16-11eb-8801-3a0e357a2bc1.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742014881,https://api.github.com/repos/simonw/datasette/issues/1136,742014881,MDEyOklzc3VlQ29tbWVudDc0MjAxNDg4MQ==,9599,simonw,2020-12-09T20:01:27Z,2020-12-09T20:01:27Z,OWNER,"I'll write the release notes in the branch, then cherry-pick them over to `main`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742014366,https://api.github.com/repos/simonw/datasette/issues/1136,742014366,MDEyOklzc3VlQ29tbWVudDc0MjAxNDM2Ng==,9599,simonw,2020-12-09T20:00:35Z,2020-12-09T20:00:35Z,OWNER,"Actually I'll start from 0.52.4 and then cherry-pick the fixes.

    git branch 0.52.x 0.52.4
    git checkout 0.52.x
    git cherry-pick COMMIT","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1091#issuecomment-742010306,https://api.github.com/repos/simonw/datasette/issues/1091,742010306,MDEyOklzc3VlQ29tbWVudDc0MjAxMDMwNg==,6739646,tballison,2020-12-09T19:53:18Z,2020-12-09T19:59:52Z,NONE,"I can't imagine this helps (esp. given your point about potential rewrites), but you can see that /datasette/ was correctly added to the sql form, but not to the ""export-links""

 <img width=""484"" alt=""Screen Shot 2020-12-09 at 2 51 09 PM"" src=""https://user-images.githubusercontent.com/6739646/101680055-234baa00-3a2e-11eb-8650-2b369bc6f031.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1134#issuecomment-742012324,https://api.github.com/repos/simonw/datasette/issues/1134,742012324,MDEyOklzc3VlQ29tbWVudDc0MjAxMjMyNA==,9599,simonw,2020-12-09T19:57:05Z,2020-12-09T19:57:05Z,OWNER,Thanks for the bug report!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760312579,"""_searchmode=raw"" throws an index out of range error when combined with ""_search_COLUMN""",
https://github.com/simonw/datasette/issues/1136#issuecomment-742009294,https://api.github.com/repos/simonw/datasette/issues/1136,742009294,MDEyOklzc3VlQ29tbWVudDc0MjAwOTI5NA==,9599,simonw,2020-12-09T19:51:18Z,2020-12-09T19:51:18Z,OWNER,"Likewise, Read The Docs publishes as stable the docs from the latest tagged release, so I would expect that to work fine as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742009101,https://api.github.com/repos/simonw/datasette/issues/1136,742009101,MDEyOklzc3VlQ29tbWVudDc0MjAwOTEwMQ==,9599,simonw,2020-12-09T19:50:53Z,2020-12-09T19:50:53Z,OWNER,"My concern is if this will  break anything about CI. I don't think it will - the code that deploys the latest `main` to https://latest.datasette.io/ should be unaffected, and the checkout code in `publish.yml` should check out the correct code based on the tag used for that release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1136#issuecomment-742008087,https://api.github.com/repos/simonw/datasette/issues/1136,742008087,MDEyOklzc3VlQ29tbWVudDc0MjAwODA4Nw==,9599,simonw,2020-12-09T19:48:56Z,2020-12-09T19:48:56Z,OWNER,I think I'm going to create a branch called `0.52.x` that starts with `8ae0f9f7f0d644b0161165a1084f53acd2786f7c` and then tag the release from there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",760621356,Establish pattern for release branches to support bug fixes,
https://github.com/simonw/datasette/issues/1091#issuecomment-742001510,https://api.github.com/repos/simonw/datasette/issues/1091,742001510,MDEyOklzc3VlQ29tbWVudDc0MjAwMTUxMA==,6739646,tballison,2020-12-09T19:36:42Z,2020-12-09T19:38:04Z,NONE,"I don't think this fixes it:

```
grep -R datasette .
./sites-available/000-default.conf:        ProxyPass /datasette http://127.0.0.1:8001/
./sites-available/000-default.conf:        #ProxyPassReverse /datasette http://127.0.0.1:8001/
./sites-available/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
./sites-available/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001
./sites-enabled/corpora-le-ssl.conf:  ProxyPass /datasette http://0.0.0.0:8001
./sites-enabled/corpora-le-ssl.conf:  #ProxyPassReverse /datasette http://0.0.0.0:8001
```

And I confirmed that I actually restarted the server. :rofl: 

https://corpora.tika.apache.org/datasette/file_profiles","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-741804334,https://api.github.com/repos/simonw/datasette/issues/1091,741804334,MDEyOklzc3VlQ29tbWVudDc0MTgwNDMzNA==,6739646,tballison,2020-12-09T14:26:05Z,2020-12-09T14:26:05Z,NONE,"Anything we can do to help debug this?  Thank you, again!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/766#issuecomment-741665253,https://api.github.com/repos/simonw/datasette/issues/766,741665253,MDEyOklzc3VlQ29tbWVudDc0MTY2NTI1Mw==,2181410,clausjuhl,2020-12-09T09:59:05Z,2020-12-09T09:59:05Z,NONE,Hi Simon. Any news on using wildcard-searches with datasette? Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",617323873,Enable wildcard-searches by default,
https://github.com/simonw/datasette/issues/1133#issuecomment-740850920,https://api.github.com/repos/simonw/datasette/issues/1133,740850920,MDEyOklzc3VlQ29tbWVudDc0MDg1MDkyMA==,9599,simonw,2020-12-08T18:55:59Z,2020-12-08T18:55:59Z,OWNER,Inspiration was this script: https://gist.github.com/simonw/f6e3cd29fde5d15ea9cd746c942046ba - which pipes output through `tail -n +2` to strip off the headers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",759695780,Option to omit header row in CSV export,
https://github.com/simonw/datasette/issues/1133#issuecomment-740850057,https://api.github.com/repos/simonw/datasette/issues/1133,740850057,MDEyOklzc3VlQ29tbWVudDc0MDg1MDA1Nw==,9599,simonw,2020-12-08T18:55:29Z,2020-12-08T18:55:29Z,OWNER,Can work on this as part of #1062.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",759695780,Option to omit header row in CSV export,
https://github.com/simonw/sqlite-utils/pull/204#issuecomment-740796067,https://api.github.com/repos/simonw/sqlite-utils/issues/204,740796067,MDEyOklzc3VlQ29tbWVudDc0MDc5NjA2Nw==,9599,simonw,2020-12-08T17:49:22Z,2020-12-08T17:49:22Z,OWNER,"Great catch, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752888228,use jsonify_if_need for sql updates,
https://github.com/simonw/datasette/issues/815#issuecomment-740385032,https://api.github.com/repos/simonw/datasette/issues/815,740385032,MDEyOklzc3VlQ29tbWVudDc0MDM4NTAzMg==,9599,simonw,2020-12-08T05:26:09Z,2020-12-08T05:26:16Z,OWNER,"Sure! It's a bit of a fiddle one - I've not found an approach that I like, but I also haven't thought about it since June. I'd love to see what you come up with!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/815#issuecomment-740383884,https://api.github.com/repos/simonw/datasette/issues/815,740383884,MDEyOklzc3VlQ29tbWVudDc0MDM4Mzg4NA==,11761973,sturzl,2020-12-08T05:23:18Z,2020-12-08T05:23:18Z,NONE,hey! I'd like to take a look at this if you're open to a PR for it,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",634663505,Group permission checks by request on /-/permissions debug page,
https://github.com/simonw/datasette/issues/1132#issuecomment-740228858,https://api.github.com/repos/simonw/datasette/issues/1132,740228858,MDEyOklzc3VlQ29tbWVudDc0MDIyODg1OA==,9599,simonw,2020-12-07T22:50:36Z,2020-12-07T22:50:36Z,OWNER,"Documented here: https://docs.datasette.io/en/latest/json_api.html#column-filter-arguments

Demo: https://latest.datasette.io/fixtures/facetable?tags__arraynotcontains=tag2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",758899581,New filter: array does not contain,
https://github.com/simonw/datasette/issues/1131#issuecomment-739414118,https://api.github.com/repos/simonw/datasette/issues/1131,739414118,MDEyOklzc3VlQ29tbWVudDczOTQxNDExOA==,9599,simonw,2020-12-05T20:48:33Z,2020-12-05T20:48:33Z,OWNER,"Oddly enough, I tried fixing this with `sys.stderr.write(""{}\n"".format(e))` - but my Click `CLIRunner` tests failed because `result.stderr` was an empty string. Adding `sys.stderr.flush()` to the code that output errors fixed that issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/simonw/datasette/issues/398#issuecomment-739357330,https://api.github.com/repos/simonw/datasette/issues/398,739357330,MDEyOklzc3VlQ29tbWVudDczOTM1NzMzMA==,9599,simonw,2020-12-05T19:36:27Z,2020-12-05T19:36:27Z,OWNER,This was fixed in #749 by 88ac538b41a4753c3de9b509c3a0e13077f66182,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",398011658,Ensure downloading a 100+MB SQLite database file works,
https://github.com/simonw/datasette/pull/1128#issuecomment-739355855,https://api.github.com/repos/simonw/datasette/issues/1128,739355855,MDEyOklzc3VlQ29tbWVudDczOTM1NTg1NQ==,9599,simonw,2020-12-05T19:34:57Z,2020-12-05T19:34:57Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756867924,Fix startup error on windows,
https://github.com/simonw/datasette/issues/1131#issuecomment-739083673,https://api.github.com/repos/simonw/datasette/issues/1131,739083673,MDEyOklzc3VlQ29tbWVudDczOTA4MzY3Mw==,9599,simonw,2020-12-05T00:02:10Z,2020-12-05T00:02:10Z,OWNER,"https://clig.dev/#the-basics

> **Send messaging to stderr**. Log messages, errors, and so on should all be sent to stderr. This means that when commands are piped together, these messages are displayed to the user and not fed into the next command.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/simonw/datasette/issues/1131#issuecomment-739083472,https://api.github.com/repos/simonw/datasette/issues/1131,739083472,MDEyOklzc3VlQ29tbWVudDczOTA4MzQ3Mg==,9599,simonw,2020-12-05T00:01:12Z,2020-12-05T00:01:12Z,OWNER,Here's why: https://github.com/simonw/datasette/blob/37f87b5e52e7f8ddd1c4ffcf368bd7a62a406a6d/datasette/database.py#L158-L163,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",757481949,"""datasette inspect"" outputs invalid JSON if an error is logged",
https://github.com/dogsheep/dogsheep-photos/pull/29#issuecomment-739058820,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/29,739058820,MDEyOklzc3VlQ29tbWVudDczOTA1ODgyMA==,9599,simonw,2020-12-04T22:32:35Z,2020-12-04T22:32:35Z,MEMBER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",638375985,Fixed bug in SQL query for photo scores,
https://github.com/simonw/datasette/pull/1130#issuecomment-738907852,https://api.github.com/repos/simonw/datasette/issues/1130,738907852,MDEyOklzc3VlQ29tbWVudDczODkwNzg1Mg==,3243482,abdusco,2020-12-04T17:22:29Z,2020-12-04T17:31:25Z,CONTRIBUTOR,"EDIT: I misunderstood the problem. This seems like a fix better suited for Safari. But I don't have any Apple device to test it.

```css
body {
  min-height: 100vh;
  min-height: -webkit-fill-available;
}
html {
  height: -webkit-fill-available;
}
```
https://css-tricks.com/css-fix-for-100vh-in-mobile-webkit/

---

It's actually not that difficult to fix.
Well, this is actually a workaround to keep viewport in place.

I usually put a transition (forgot to do it here) that keeps page from resizing.

```css
.container {
  min-height: 100vh;
  transition: height 10000s steps(0);
}
```

`steps()` function prevents excessive layout calculations, and lets the page snap back into place (10000s ~= 3h later) in a single step.
This fix also prevents page from jumping around when the keyboard pops up and down.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756876238,Fix footer not sticking to bottom in short pages,
https://github.com/simonw/datasette/issues/188#issuecomment-738905376,https://api.github.com/repos/simonw/datasette/issues/188,738905376,MDEyOklzc3VlQ29tbWVudDczODkwNTM3Ng==,9599,simonw,2020-12-04T17:18:34Z,2020-12-04T17:18:34Z,OWNER,This is likely to be covered by plugin hooks: #860 for the metadata and after investigating in #1042 it looks like the existing `prepare_jinja2_environment` hook may already be enough to load templates from the database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",309047460,Ability to bundle metadata and templates inside the SQLite file,
https://github.com/simonw/datasette/issues/111#issuecomment-738904347,https://api.github.com/repos/simonw/datasette/issues/111,738904347,MDEyOklzc3VlQ29tbWVudDczODkwNDM0Nw==,9599,simonw,2020-12-04T17:16:56Z,2020-12-04T17:16:56Z,OWNER,This is STILL a good idea.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata,
https://github.com/simonw/datasette/pull/1130#issuecomment-738897582,https://api.github.com/repos/simonw/datasette/issues/1130,738897582,MDEyOklzc3VlQ29tbWVudDczODg5NzU4Mg==,9599,simonw,2020-12-04T17:03:30Z,2020-12-04T17:03:30Z,OWNER,"I deployed this to https://datasette-issue-1129.vercel.app/ (using `datasette publish vercel fixtures.db --branch 8d4c69c6fb0ef741a19070f5172017ea3522e83c --about_url https://github.com/simonw/datasette/issues/1129 --about datasette/issues/1129 --project datasette-issue-1129`) - weirdly, on Mobile Safari the footer appears just below the visible window:

![RPReplay_Final1607100726](https://user-images.githubusercontent.com/9599/101191950-336b2000-360f-11eb-8f14-ed83bd86515c.gif)

I've seen other problems with fixed footers on Mobile Safari too: at Eventbrite this was a really nasty problem for us to figure out: https://www.eventbrite.com/engineering/mobile-safari-why/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756876238,Fix footer not sticking to bottom in short pages,
https://github.com/simonw/datasette/pull/1130#issuecomment-738620153,https://api.github.com/repos/simonw/datasette/issues/1130,738620153,MDEyOklzc3VlQ29tbWVudDczODYyMDE1Mw==,22429695,codecov[bot],2020-12-04T07:34:48Z,2020-12-04T07:34:48Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=h1) Report
> Merging [#1130](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=desc) (8d4c69c) into [main](https://codecov.io/gh/simonw/datasette/commit/49d8fc056844d5a537d6cfd96dab0dd5686fe718?el=desc) (49d8fc0) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1130/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1130   +/-   ##
=======================================
  Coverage   91.42%   91.42%           
=======================================
  Files          31       31           
  Lines        3873     3873           
=======================================
  Hits         3541     3541           
  Misses        332      332           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=footer). Last update [49d8fc0...8d4c69c](https://codecov.io/gh/simonw/datasette/pull/1130?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756876238,Fix footer not sticking to bottom in short pages,
https://github.com/simonw/datasette/pull/1128#issuecomment-738613497,https://api.github.com/repos/simonw/datasette/issues/1128,738613497,MDEyOklzc3VlQ29tbWVudDczODYxMzQ5Nw==,22429695,codecov[bot],2020-12-04T07:17:12Z,2020-12-04T07:17:12Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=h1) Report
> Merging [#1128](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=desc) (7004c3b) into [main](https://codecov.io/gh/simonw/datasette/commit/49d8fc056844d5a537d6cfd96dab0dd5686fe718?el=desc) (49d8fc0) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1128/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main    #1128      +/-   ##
==========================================
- Coverage   91.42%   91.42%   -0.01%     
==========================================
  Files          31       31              
  Lines        3873     3872       -1     
==========================================
- Hits         3541     3540       -1     
  Misses        332      332              
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1128/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `92.13% <ø> (-0.04%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=footer). Last update [49d8fc0...7004c3b](https://codecov.io/gh/simonw/datasette/pull/1128?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756867924,Fix startup error on windows,
https://github.com/simonw/datasette/issues/1125#issuecomment-738554392,https://api.github.com/repos/simonw/datasette/issues/1125,738554392,MDEyOklzc3VlQ29tbWVudDczODU1NDM5Mg==,9599,simonw,2020-12-04T04:16:57Z,2020-12-04T04:16:57Z,OWNER,"https://latest.datasette.io/-/versions now shows this:
```json
{
    ""python"": {
        ""version"": ""3.8.6"",
        ""full"": ""3.8.6 (default, Nov 18 2020, 13:49:49) \n[GCC 8.3.0]""
    },
    ""datasette"": {
        ""version"": ""0.52.3"",
        ""note"": ""49d8fc056844d5a537d6cfd96dab0dd5686fe718""
    },
    ""asgi"": ""3.0"",
    ""uvicorn"": ""0.12.3"",
    ""sqlite"": {
        ""version"": ""3.33.0"",
        ""fts_versions"": [
            ""FTS5"",
            ""FTS4"",
            ""FTS3""
        ],
        ""extensions"": {
            ""json1"": null
        },
        ""compile_options"": []
    },
    ""pysqlite3"": ""0.4.4""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738551280,https://api.github.com/repos/simonw/datasette/issues/1125,738551280,MDEyOklzc3VlQ29tbWVudDczODU1MTI4MA==,9599,simonw,2020-12-04T04:03:54Z,2020-12-04T04:03:54Z,OWNER,"I'm going to check `pkg_resources.get_distribution(""pysqlite3-binary"").version` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738550588,https://api.github.com/repos/simonw/datasette/issues/1125,738550588,MDEyOklzc3VlQ29tbWVudDczODU1MDU4OA==,9599,simonw,2020-12-04T04:01:10Z,2020-12-04T04:01:10Z,OWNER,"Urgh, figuring out the version of `pysqlite3` is WAY harder than I expected. The `getversion` module looks like the smartest attempt at solving this problem generally, but I'd like to avoid adding another dependency just for this: https://github.com/smarie/python-getversion","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1125#issuecomment-738548693,https://api.github.com/repos/simonw/datasette/issues/1125,738548693,MDEyOklzc3VlQ29tbWVudDczODU0ODY5Mw==,9599,simonw,2020-12-04T03:52:51Z,2020-12-04T03:52:51Z,OWNER,"That didn't work - https://latest.datasette.io/-/versions isn't showing the package.

I bet that's because I'm actually installing `pysqlite3-binary` here: https://github.com/simonw/datasette/blob/e2fea36540e952d8d72c1bd0af7144b85b7a4671/.github/workflows/deploy-latest.yml#L57","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1126#issuecomment-738548393,https://api.github.com/repos/simonw/datasette/issues/1126,738548393,MDEyOklzc3VlQ29tbWVudDczODU0ODM5Mw==,9599,simonw,2020-12-04T03:51:38Z,2020-12-04T03:51:38Z,OWNER,That worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756761963,Switch to google-github-actions/setup-gcloud for demo deploy,
https://github.com/simonw/datasette/issues/1125#issuecomment-738347171,https://api.github.com/repos/simonw/datasette/issues/1125,738347171,MDEyOklzc3VlQ29tbWVudDczODM0NzE3MQ==,9599,simonw,2020-12-03T22:04:52Z,2020-12-03T22:04:52Z,OWNER,"```
pkg_resources.get_distribution(""pysqlite3"").version
Out[14]: '0.4.4'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756622648,Show pysqlite3 version on /-/versions,
https://github.com/simonw/datasette/issues/1124#issuecomment-738215686,https://api.github.com/repos/simonw/datasette/issues/1124,738215686,MDEyOklzc3VlQ29tbWVudDczODIxNTY4Ng==,9599,simonw,2020-12-03T18:50:48Z,2020-12-03T21:42:02Z,OWNER,I'm going to punt on writing a unit test for this (not sure how I'd simulate those symlinks) - I'll manually test it and push out a dot release instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738215487,https://api.github.com/repos/simonw/datasette/issues/1124,738215487,MDEyOklzc3VlQ29tbWVudDczODIxNTQ4Nw==,9599,simonw,2020-12-03T18:50:26Z,2020-12-03T21:41:25Z,OWNER,"This fix works - calling `.resolve()` on the `root_path` before the comparison to ensure symlinks are resolved:
```python
        # Ensure full_path is within root_path to avoid weird ""../"" tricks
        try:
            print(""full_path={}, root_path={}"".format(full_path, root_path))
            full_path.relative_to(root_path.resolve())
        except ValueError as e:
            print(""  ValueError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738213342,https://api.github.com/repos/simonw/datasette/issues/1124,738213342,MDEyOklzc3VlQ29tbWVudDczODIxMzM0Mg==,9599,simonw,2020-12-03T18:46:22Z,2020-12-03T21:40:51Z,OWNER,"I replaced that function with this code:
```python
def asgi_static(root_path, chunk_size=4096, headers=None, content_type=None):
    async def inner_static(request, send):
        path = request.scope[""url_route""][""kwargs""][""path""]
        print(""path ="", path)
        try:
            full_path = (Path(root_path) / path).resolve().absolute()
        except FileNotFoundError as e:
            print(""FileNotFoundError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
        if full_path.is_dir():
            await asgi_send_html(send, ""403: Directory listing is not allowed"", 403)
            return
        # Ensure full_path is within root_path to avoid weird ""../"" tricks
        try:
            print(""full_path={}, root_path={}"".format(full_path, root_path))
            full_path.relative_to(root_path)
        except ValueError as e:
            print(""  ValueError:"", e)
            await asgi_send_html(send, ""404"", 404)
            return
        try:
            await asgi_send_file(send, full_path, chunk_size=chunk_size)
        except FileNotFoundError:
            await asgi_send_html(send, ""404"", 404)
            return

    return inner_static
```
Edited using `vi /home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/utils/asgi.py`

The output shows me what the bug is:
```
$ datasette --get /-/static/app.css --pdb
app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
path = app.css
full_path=/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css, root_path=/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static
  ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'
404
```
`  ValueError: '/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css' does not start with '/home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages/datasette/static'`

One is `../lib/python3.7/..` and the other is `../lib64/python3.7/..` - there's clearly some kind of symlink in play here which I'm not taking into account.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738313399,https://api.github.com/repos/simonw/datasette/issues/1124,738313399,MDEyOklzc3VlQ29tbWVudDczODMxMzM5OQ==,9599,simonw,2020-12-03T21:10:54Z,2020-12-03T21:10:54Z,OWNER,Confirmed that installing a fresh copy of Datasette 0.52.3 on that server works correctly as expected.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738224865,https://api.github.com/repos/simonw/datasette/issues/1124,738224865,MDEyOklzc3VlQ29tbWVudDczODIyNDg2NQ==,9599,simonw,2020-12-03T19:01:52Z,2020-12-03T19:01:52Z,OWNER,"https://github.com/simonw/datasette/runs/1494631261
```
/home/runner/work/datasette/datasette/tests/test_html.py:81: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File ""/home/runner/work/datasette/datasette/datasette/app.py"", line 1039, in route_path
    response = await view(request, send)
  File ""/home/runner/work/datasette/datasette/datasette/utils/asgi.py"", line 297, in inner_static
    full_path.relative_to(root_path.resolve())
AttributeError: 'str' object has no attribute 'resolve'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738220067,https://api.github.com/repos/simonw/datasette/issues/1124,738220067,MDEyOklzc3VlQ29tbWVudDczODIyMDA2Nw==,9599,simonw,2020-12-03T18:58:17Z,2020-12-03T18:58:17Z,OWNER,"I tested this by running:

    pipx uninstall datasette
    pipx install 'https://github.com/simonw/datasette/archive/6b4c55efea3e9d34d92cbe5f0066553ad9b14071.zip'

To replace that version of Datasette (in the correct virtual environment) with this patch. It worked!

```
[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/static/app.css
/* Reset and Page Setup ==================================================== */
...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738211776,https://api.github.com/repos/simonw/datasette/issues/1124,738211776,MDEyOklzc3VlQ29tbWVudDczODIxMTc3Ng==,9599,simonw,2020-12-03T18:43:21Z,2020-12-03T18:43:21Z,OWNER,I'm suspicious of this code here:https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/utils/asgi.py#L284-L307,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738211152,https://api.github.com/repos/simonw/datasette/issues/1124,738211152,MDEyOklzc3VlQ29tbWVudDczODIxMTE1Mg==,9599,simonw,2020-12-03T18:42:12Z,2020-12-03T18:42:12Z,OWNER,"Added a line to print out `app_root` from https://github.com/simonw/datasette/blob/e048791a9a2686f47d81a2c8aa88aa1966d82521/datasette/app.py#L848-L853
```
app_root =  /home/ec2-user/.local/pipx/venvs/datasette/lib64/python3.7/site-packages
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1124#issuecomment-738209642,https://api.github.com/repos/simonw/datasette/issues/1124,738209642,MDEyOklzc3VlQ29tbWVudDczODIwOTY0Mg==,9599,simonw,2020-12-03T18:39:19Z,2020-12-03T18:39:19Z,OWNER,"The CSS files are in the expected location:
```
[ec2-user@ip-172-31-30-7 ~]$ find /home/ec2-user/.local/pipx/venvs/datasette | grep css
/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/app.css
/home/ec2-user/.local/pipx/venvs/datasette/lib/python3.7/site-packages/datasette/static/codemirror-5.57.0.min.css
```
Wow it's running an ANCIENT version of SQLite:
```
[ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/versions.json
{""python"": {""version"": ""3.7.9"", ""full"": ""3.7.9 (default, Aug 27 2020, 21:58:41) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)]""}, ""datasette"": {""version"": ""0.52.2""}, ""asgi"": ""3.0"", ""uvicorn"": ""0.12.3"", ""sqlite"": {""version"": ""3.7.17"", ""fts_versions"": [""FTS4"", ""FTS3""], ""extensions"": {}, ""compile_options"": [""DISABLE_DIRSYNC"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_FTS3"", ""ENABLE_RTREE"", ""ENABLE_UNLOCK_NOTIFY"", ""SECURE_DELETE"", ""TEMP_STORE=1"", ""THREADSAFE=1""]}}
```
http://www.sqlite.org/releaselog/3_7_17.html - SQLite Release 3.7.17 On 2013-05-20","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",756439516,Datasette on Amazon Linux on ARM returns 404 for static assets,
https://github.com/simonw/datasette/issues/1121#issuecomment-737591281,https://api.github.com/repos/simonw/datasette/issues/1121,737591281,MDEyOklzc3VlQ29tbWVudDczNzU5MTI4MQ==,9599,simonw,2020-12-03T01:03:18Z,2020-12-03T01:03:18Z,OWNER,"Demo: https://latest.datasette.io/fixtures?_bot=1

<img width=""729"" alt=""fixtures"" src=""https://user-images.githubusercontent.com/9599/100950004-43c0b500-34c0-11eb-918c-aa959376461f.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",754178780,Table actions cog is misaligned,
https://github.com/simonw/datasette/issues/1100#issuecomment-737589314,https://api.github.com/repos/simonw/datasette/issues/1100,737589314,MDEyOklzc3VlQ29tbWVudDczNzU4OTMxNA==,9599,simonw,2020-12-03T00:57:35Z,2020-12-03T00:57:35Z,OWNER,"Fixed in the demo:
```
% curl -XOPTIONS https://latest.datasette.io/fixtures.json
ok%
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",747702144,Error on OPTIONS request to database,
https://github.com/simonw/datasette/issues/1123#issuecomment-737586248,https://api.github.com/repos/simonw/datasette/issues/1123,737586248,MDEyOklzc3VlQ29tbWVudDczNzU4NjI0OA==,9599,simonw,2020-12-03T00:47:37Z,2020-12-03T00:47:37Z,OWNER,"Affected tests:
```
FAILED tests/test_plugins.py::test_hook_table_actions[facetable] - AssertionE...
FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - Assertio...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",755721275,"Table actions hook are order dependent, should not be",
https://github.com/simonw/datasette/issues/1100#issuecomment-737581719,https://api.github.com/repos/simonw/datasette/issues/1100,737581719,MDEyOklzc3VlQ29tbWVudDczNzU4MTcxOQ==,9599,simonw,2020-12-03T00:35:23Z,2020-12-03T00:35:23Z,OWNER,"Replicated this against the live demo as well:
```
/tmp % curl -XOPTIONS https://latest.datasette.io/fixtures.json
{""ok"": false, ""error"": ""object Response can't be used in 'await' expression"", ""status"": 500, ""title"": null}%                                                                                                                                  /tmp % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",747702144,Error on OPTIONS request to database,
https://github.com/simonw/datasette/pull/1122#issuecomment-737580813,https://api.github.com/repos/simonw/datasette/issues/1122,737580813,MDEyOklzc3VlQ29tbWVudDczNzU4MDgxMw==,9599,simonw,2020-12-03T00:33:09Z,2020-12-03T00:33:09Z,OWNER,"This is a very neat fix, thank you.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",754179035,Fix misaligned table actions cog,
https://github.com/simonw/datasette/issues/749#issuecomment-737580084,https://api.github.com/repos/simonw/datasette/issues/749,737580084,MDEyOklzc3VlQ29tbWVudDczNzU4MDA4NA==,9599,simonw,2020-12-03T00:31:14Z,2020-12-03T00:31:14Z,OWNER,"This works!

```
/tmp % wget 'https://covid-19.datasettes.com/covid.db'
--2020-12-02 16:28:02--  https://covid-19.datasettes.com/covid.db
Resolving covid-19.datasettes.com (covid-19.datasettes.com)... 172.217.5.83
Connecting to covid-19.datasettes.com (covid-19.datasettes.com)|172.217.5.83|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/octet-stream]
Saving to: ‘covid.db’

covid.db                [              <=>   ] 306.42M  3.27MB/s    in 98s     

2020-12-02 16:29:40 (3.13 MB/s) - ‘covid.db’ saved [321306624]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/simonw/datasette/issues/749#issuecomment-737563699,https://api.github.com/repos/simonw/datasette/issues/749,737563699,MDEyOklzc3VlQ29tbWVudDczNzU2MzY5OQ==,9599,simonw,2020-12-02T23:45:42Z,2020-12-02T23:45:42Z,OWNER,"I asked about this on Twitter - https://twitter.com/steren/status/1334281184965140483

> You simply need to send the `Transfer-Encoding: chunked` header.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",610829227,Cloud Run fails to serve database files larger than 32MB,
https://github.com/simonw/datasette/issues/942#issuecomment-737463116,https://api.github.com/repos/simonw/datasette/issues/942,737463116,MDEyOklzc3VlQ29tbWVudDczNzQ2MzExNg==,9599,simonw,2020-12-02T20:02:10Z,2020-12-02T20:03:01Z,OWNER,"My idea is that if you installed my proposed plugin you wouldn't need `metadata.json` at all - your metadata would instead live in a table in the connected SQLite database files - either one table per database (so the metadata can live in the same place as the data) or maybe also in a dedicated separate database file, for if you want to add metadata to an otherwise read-only database.

The plugin would then provide a UI for editing that metadata - maybe by configuring some writable canned queries or maybe something more custom than that. Or you could edit the metadata by manually editing the SQLite database file (or loading data into it using a tool like [yaml-to-sqlite](https://github.com/simonw/yaml-to-sqlite)).","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-737428262,https://api.github.com/repos/simonw/datasette/issues/942,737428262,MDEyOklzc3VlQ29tbWVudDczNzQyODI2Mg==,596279,zaneselvans,2020-12-02T18:55:21Z,2020-12-02T18:55:21Z,NONE,"Are you thinking that those metadata tables would be added to the SQLite DB by Datasette, when you tell it to wrap up the database, with the metadata coming from the `metadata.json`? Would it be easy to allow the prepopulation of those tables in the database itself? We've been struggling with the best way to make sure that the data is always accompanied by metadata, and baking it all into the database itself would be nice, since then we wouldn't need to worry about separately distributing different files in different contexts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/942#issuecomment-737402392,https://api.github.com/repos/simonw/datasette/issues/942,737402392,MDEyOklzc3VlQ29tbWVudDczNzQwMjM5Mg==,9599,simonw,2020-12-02T18:08:55Z,2020-12-02T18:08:55Z,OWNER,"SQLite does let you add comments in your CREATE TABLE statements:
```sql
CREATE TABLE something (
    id integer primary key, -- integer primary key
    created text -- created date as ISO datetime
);
```
But the only mechanism for reading those back is to retrieve that `CREATE TABLE` block of SQL from the `sqlite_master` table and run a parser against it.

I've so far resisted adding a SQL syntax parser to Datasette for complexity reasons - though I'm increasingly thinking I'll need to do it at some point.

I think I'll leave this to plugins. I'm definitely going to build a plugin that lets you store metadata for tables and columns in a SQLite database table, which will then support interactively editing metadata through a UI.

A plugin which extracts column comments from the SQLite CREATE TABLE comments would be feasible too, if I design the plugin hooks well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1111#issuecomment-736322290,https://api.github.com/repos/simonw/datasette/issues/1111,736322290,MDEyOklzc3VlQ29tbWVudDczNjMyMjI5MA==,3243482,abdusco,2020-12-01T08:54:47Z,2020-12-01T08:54:47Z,CONTRIBUTOR,"Somewhat related: https://github.com/simonw/datasette/issues/859
I fixed the issue with forking and disabling the counts for hidden tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",751195017,Accessing a database's `.json` is slow for very large SQLite files,
https://github.com/simonw/datasette/pull/1122#issuecomment-736318377,https://api.github.com/repos/simonw/datasette/issues/1122,736318377,MDEyOklzc3VlQ29tbWVudDczNjMxODM3Nw==,22429695,codecov[bot],2020-12-01T08:47:33Z,2020-12-01T08:47:33Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=h1) Report
> Merging [#1122](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=desc) (94ea22f) into [main](https://codecov.io/gh/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267?el=desc) (a970276) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1122/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1122   +/-   ##
=======================================
  Coverage   91.49%   91.49%           
=======================================
  Files          31       31           
  Lines        3856     3856           
=======================================
  Hits         3528     3528           
  Misses        328      328           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=footer). Last update [a970276...94ea22f](https://codecov.io/gh/simonw/datasette/pull/1122?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",754179035,Fix misaligned table actions cog,
https://github.com/simonw/datasette/issues/942#issuecomment-736173084,https://api.github.com/repos/simonw/datasette/issues/942,736173084,MDEyOklzc3VlQ29tbWVudDczNjE3MzA4NA==,596279,zaneselvans,2020-12-01T02:20:58Z,2020-12-01T02:20:58Z,NONE,"Are there common patterns for storing column-based metadata inside SQLite itself? I know Postgres allows ""comment"" fields, which this is kind of trying to replicate. Should the `units` and `description` and possibly other per-column metadata fields be combined into a single (tabular?) structure, that would be displayed above the data on the table / query results page?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1119#issuecomment-736142201,https://api.github.com/repos/simonw/datasette/issues/1119,736142201,MDEyOklzc3VlQ29tbWVudDczNjE0MjIwMQ==,9599,simonw,2020-12-01T00:41:14Z,2020-12-01T00:41:14Z,OWNER,"On my laptop:

<img width=""1123"" alt=""fixtures__generated_columns__1_row"" src=""https://user-images.githubusercontent.com/9599/100681839-8734eb00-3329-11eb-8aea-813bdc18efa1.png"">

https://latest.datasette.io/-/versions is running SQLite 3.27.2 at the moment so it won't show that table until it gets to 3.31.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753876808,"Include generated columns in fixtures.db, if SQLite version supports it",
https://github.com/simonw/datasette/pull/1120#issuecomment-736135125,https://api.github.com/repos/simonw/datasette/issues/1120,736135125,MDEyOklzc3VlQ29tbWVudDczNjEzNTEyNQ==,22429695,codecov[bot],2020-12-01T00:22:36Z,2020-12-01T00:22:36Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=h1) Report
> Merging [#1120](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=desc) (ddad8db) into [main](https://codecov.io/gh/simonw/datasette/commit/461670a0b87efa953141b449a9a261919864ceb3?el=desc) (461670a) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1120/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1120   +/-   ##
=======================================
  Coverage   91.48%   91.49%           
=======================================
  Files          31       31           
  Lines        3852     3856    +4     
=======================================
+ Hits         3524     3528    +4     
  Misses        328      328           
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1120/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.10% <100.00%> (ø)` | |
| [datasette/utils/sqlite.py](https://codecov.io/gh/simonw/datasette/pull/1120/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3NxbGl0ZS5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=footer). Last update [461670a...ddad8db](https://codecov.io/gh/simonw/datasette/pull/1120?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753898359,generated_columns table in fixtures.py,
https://github.com/simonw/datasette/pull/1117#issuecomment-736088949,https://api.github.com/repos/simonw/datasette/issues/1117,736088949,MDEyOklzc3VlQ29tbWVudDczNjA4ODk0OQ==,2789593,nattaylor,2020-11-30T22:15:58Z,2020-11-30T22:23:19Z,NONE,"I just deployed this and its working great.

~In a very unscientific benchmark my response times went from around 22-25ms to 33-36ms, but I didn't even dig enough to confirm the latency is related to the change.  It's on a VPS, so maybe the load changed.~ I don't see any difference in performance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736067475,https://api.github.com/repos/simonw/datasette/issues/1117,736067475,MDEyOklzc3VlQ29tbWVudDczNjA2NzQ3NQ==,22429695,codecov[bot],2020-11-30T21:28:22Z,2020-11-30T21:28:22Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=h1) Report
> Merging [#1117](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=desc) (ccdf2c6) into [main](https://codecov.io/gh/simonw/datasette/commit/dea3c508b39528e566d711c38a467b3d372d220b?el=desc) (dea3c50) will **decrease** coverage by `0.00%`.
> The diff coverage is `95.23%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1117/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##             main    #1117      +/-   ##
==========================================
- Coverage   91.48%   91.48%   -0.01%     
==========================================
  Files          30       31       +1     
  Lines        3841     3852      +11     
==========================================
+ Hits         3514     3524      +10     
- Misses        327      328       +1     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1117/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.10% <87.50%> (-0.20%)` | :arrow_down: |
| [datasette/utils/sqlite.py](https://codecov.io/gh/simonw/datasette/pull/1117/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3NxbGl0ZS5weQ==) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=footer). Last update [dea3c50...ccdf2c6](https://codecov.io/gh/simonw/datasette/pull/1117?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/issues/1116#issuecomment-736030599,https://api.github.com/repos/simonw/datasette/issues/1116,736030599,MDEyOklzc3VlQ29tbWVudDczNjAzMDU5OQ==,9599,simonw,2020-11-30T20:41:41Z,2020-11-30T20:41:41Z,OWNER,"Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

> ### 2018-12-01 (3.26.0)
> 
> -  Added [PRAGMA table_xinfo](https://www.sqlite.org/pragma.html#pragma_table_xinfo) that works just like [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that it also shows [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables.

CI is running 3.22.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/pull/1117#issuecomment-736029337,https://api.github.com/repos/simonw/datasette/issues/1117,736029337,MDEyOklzc3VlQ29tbWVudDczNjAyOTMzNw==,9599,simonw,2020-11-30T20:39:06Z,2020-11-30T20:39:06Z,OWNER,"Here's the problem: https://www.sqlite.org/changes.html#version_3_26_0

> ### 2018-12-01 (3.26.0)
> 
> -  Added [PRAGMA table_xinfo](https://www.sqlite.org/pragma.html#pragma_table_xinfo) that works just like [PRAGMA table_info](https://www.sqlite.org/pragma.html#pragma_table_info) except that it also shows [hidden columns](https://www.sqlite.org/vtab.html#hiddencol) in virtual tables.

CI is running 3.22.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736028726,https://api.github.com/repos/simonw/datasette/issues/1117,736028726,MDEyOklzc3VlQ29tbWVudDczNjAyODcyNg==,9599,simonw,2020-11-30T20:37:50Z,2020-11-30T20:37:50Z,OWNER,"This kind of problem is why I have a `tmate` workflow:

<img width=""1086"" alt=""Actions_·_simonw_datasette"" src=""https://user-images.githubusercontent.com/9599/100660845-72dff680-3307-11eb-89dc-211d226f68dd.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736023089,https://api.github.com/repos/simonw/datasette/issues/1117,736023089,MDEyOklzc3VlQ29tbWVudDczNjAyMzA4OQ==,9599,simonw,2020-11-30T20:26:27Z,2020-11-30T20:26:27Z,OWNER,"On my laptop:
```
platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
SQLite: 3.33.0
```
In CI they are all SQLite: 3.22.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/pull/1117#issuecomment-736018609,https://api.github.com/repos/simonw/datasette/issues/1117,736018609,MDEyOklzc3VlQ29tbWVudDczNjAxODYwOQ==,9599,simonw,2020-11-30T20:17:31Z,2020-11-30T20:17:31Z,OWNER,I need to replicate these failures on my laptop. My hunch is that this is down to the version of SQLite available to Python.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753767911,Support for generated columns,
https://github.com/simonw/datasette/issues/1116#issuecomment-736015487,https://api.github.com/repos/simonw/datasette/issues/1116,736015487,MDEyOklzc3VlQ29tbWVudDczNjAxNTQ4Nw==,9599,simonw,2020-11-30T20:11:07Z,2020-11-30T20:11:07Z,OWNER,Working on this in a pull request: https://github.com/simonw/datasette/pull/1117,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736014372,https://api.github.com/repos/simonw/datasette/issues/1116,736014372,MDEyOklzc3VlQ29tbWVudDczNjAxNDM3Mg==,9599,simonw,2020-11-30T20:08:48Z,2020-11-30T20:08:48Z,OWNER,"Ouch, the tests pass on my laptop but failed in CI: https://github.com/simonw/datasette/actions/runs/392367997

Lots of failures look like this:
```
ERROR: conn=<sqlite3.Connection object at 0x7f44f0494030>, sql = 'select rowid,  from facetable order by rowid limit 51', params = {}: near ""from"": syntax error
```
Note the `select rowid, from...` - so it looks like invalid SQL queries are being constructed maybe due to mis-detecting columns somehow.

I wonder why it didn't fail on my laptop?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736010720,https://api.github.com/repos/simonw/datasette/issues/1116,736010720,MDEyOklzc3VlQ29tbWVudDczNjAxMDcyMA==,9599,simonw,2020-11-30T20:01:53Z,2020-11-30T20:01:53Z,OWNER,"I'm OK exposing hidden columns, unless someone comes up with a pressing reason not to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736005833,https://api.github.com/repos/simonw/datasette/issues/1116,736005833,MDEyOklzc3VlQ29tbWVudDczNjAwNTgzMw==,2789593,nattaylor,2020-11-30T19:54:39Z,2020-11-30T19:54:39Z,NONE,"@simonw thanks for investigating so quickly.  If it is undesirable to change that hidden behavior, maybe something like this is a suitable workaround:
```
SELECT * FROM pragma_table_xinfo('deeds') where hidden in (0,2);
0|body|TEXT|0||0|0
1|id|INT GENERATED ALWAYS|0||0|2
2|consideration|INT GENERATED ALWAYS|0||0|2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-736004383,https://api.github.com/repos/simonw/datasette/issues/1116,736004383,MDEyOklzc3VlQ29tbWVudDczNjAwNDM4Mw==,9599,simonw,2020-11-30T19:51:51Z,2020-11-30T19:51:51Z,OWNER,"This change will also have an impact on how hidden virtual FTS tables are displayed, since apparently those have some hidden columns: https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable_fts%27%29

| cid | name | type | notnull | dflt_value | pk | hidden |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | text1 |   | 0 |   | 0 | 0 |
| 1 | text2 |   | 0 |   | 0 | 0 |
| 2 | name with . and spaces |   | 0 |   | 0 | 0 |
| 3 | searchable_fts |   | 0 |   | 0 | 1 |
| 4 | docid |   | 0 |   | 0 | 1 |
| 5 | __langid |   | 0 |   | 0 | 1 |","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-735995695,https://api.github.com/repos/simonw/datasette/issues/1116,735995695,MDEyOklzc3VlQ29tbWVudDczNTk5NTY5NQ==,9599,simonw,2020-11-30T19:34:15Z,2020-11-30T19:34:15Z,OWNER,"Generated column support was added in SQLite 3.31.0, so any unit tests I write for this should use skipIf to only run on that version or later.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-735993935,https://api.github.com/repos/simonw/datasette/issues/1116,735993935,MDEyOklzc3VlQ29tbWVudDczNTk5MzkzNQ==,9599,simonw,2020-11-30T19:30:44Z,2020-11-30T19:32:15Z,OWNER,"It looks like `PRAGMA table_info` skips ""hidden"" columns: https://www.sqlite.org/pragma.html#pragma_table_info

But `PRAGMA table_xinfo` does not: https://www.sqlite.org/pragma.html#pragma_table_xinfo

Compare https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_info%28%27searchable%27%29 to https://latest.datasette.io/fixtures?sql=select+*+from+pragma_table_xinfo%28%27searchable%27%29 - the `xinfo` one has an additional `hidden` column.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/1116#issuecomment-735992106,https://api.github.com/repos/simonw/datasette/issues/1116,735992106,MDEyOklzc3VlQ29tbWVudDczNTk5MjEwNg==,9599,simonw,2020-11-30T19:27:10Z,2020-11-30T19:27:10Z,OWNER,"I'm treating this as a bug - these columns should definitely be visible in Datasette.

I created my own test database using SQLite from Homebrew like this:
```
/usr/local/Cellar/sqlite/3.33.0/bin/sqlite3 deeds.db << EOF
CREATE TABLE deeds (
    body TEXT,
    id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
    consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
);
INSERT INTO deeds (body) VALUES ('{
    ""id"": 1,
    ""consideration"": ""This is the consideration""
}');
EOF
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753668177,GENERATED column support,
https://github.com/simonw/datasette/issues/263#issuecomment-735960132,https://api.github.com/repos/simonw/datasette/issues/263,735960132,MDEyOklzc3VlQ29tbWVudDczNTk2MDEzMg==,9599,simonw,2020-11-30T18:25:17Z,2020-11-30T18:25:17Z,OWNER,Fixing this would unblock this issue for switching `datasette-graphql` to using `datasette.client` internally: https://github.com/simonw/datasette-graphql/issues/61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323671577,Facets should not execute for ?shape=array|object,
https://github.com/dogsheep/github-to-sqlite/issues/53#issuecomment-735485677,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53,735485677,MDEyOklzc3VlQ29tbWVudDczNTQ4NTY3Nw==,9599,simonw,2020-11-30T00:36:09Z,2020-11-30T00:36:09Z,MEMBER,Given rate limits (see #51) this command might be better implemented by running a `git clone` into a temporary directory - doing so would retrieve all of the files in one go.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753000405,Command for fetching file contents,
https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-735484186,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51,735484186,MDEyOklzc3VlQ29tbWVudDczNTQ4NDE4Ng==,9599,simonw,2020-11-30T00:29:31Z,2020-11-30T00:29:31Z,MEMBER,"This just caused a failure in deploying the demo: https://github.com/dogsheep/github-to-sqlite/runs/1471304407?check_suite_focus=true

```
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/bin/github-to-sqlite"", line 33, in <module>
    sys.exit(load_entry_point('github-to-sqlite', 'console_scripts', 'github-to-sqlite')())
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 829, in __call__
    return self.main(*args, **kwargs)
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 782, in main
    rv = self.invoke(ctx)
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/opt/hostedtoolcache/Python/3.8.6/x64/lib/python3.8/site-packages/click/core.py"", line 610, in invoke
    return callback(*args, **kwargs)
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 142, in issue_comments
    for comment in utils.fetch_issue_comments(repo, token, issue):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 380, in fetch_issue_comments
    for comments in paginate(url, headers):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 472, in paginate
    raise GitHubError.from_response(response)
github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)
Error: Process completed with exit code 1.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703246031,github-to-sqlite should handle rate limits better,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483820,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735483820,MDEyOklzc3VlQ29tbWVudDczNTQ4MzgyMA==,9599,simonw,2020-11-30T00:27:47Z,2020-11-30T00:27:47Z,MEMBER,"So it looks like anything that pulls reviews needs to pull each review, then for each one pull the comments.

I'm going to consider this blocked on smarter rate limit handling in #51.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735483604,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735483604,MDEyOklzc3VlQ29tbWVudDczNTQ4MzYwNA==,9599,simonw,2020-11-30T00:26:50Z,2020-11-30T00:26:50Z,MEMBER,"It seems like there's a lot missing from that - those aren't particularly interesting given the data that is returned.

From the docs at https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#reviews it looks like each review consists of multiple comments, and the comments are where the useful material is - https://docs.github.com/en/free-pro-team@latest/rest/reference/pulls#list-comments-for-a-pull-request-review

`github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews/503368921/comments --accept 'application/vnd.github.v3+json'`

```json
[
    {
        ""id"": 500603838,
        ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwMzgzOA=="",
        ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838"",
        ""pull_request_review_id"": 503368921,
        ""diff_hunk"": ""@@ -0,0 +1,370 @@\n+[\n+    {\n+        \""url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571\"",\n+        \""id\"": 313384926,\n+        \""node_id\"": \""MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2\"",\n+        \""html_url\"": \""https://github.com/simonw/datasette/pull/571\"",\n+        \""diff_url\"": \""https://github.com/simonw/datasette/pull/571.diff\"",\n+        \""patch_url\"": \""https://github.com/simonw/datasette/pull/571.patch\"",\n+        \""issue_url\"": \""https://api.github.com/repos/simonw/datasette/issues/571\"",\n+        \""number\"": 571,\n+        \""state\"": \""closed\"",\n+        \""locked\"": false,\n+        \""title\"": \""detect_fts now works with alternative table escaping\"",\n+        \""user\"": {\n+          \""login\"": \""simonw\"",\n+          \""id\"": 9599,\n+          \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+          \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+          \""gravatar_id\"": \""\"",\n+          \""url\"": \""https://api.github.com/users/simonw\"",\n+          \""html_url\"": \""https://github.com/simonw\"",\n+          \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+          \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+          \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+          \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+          \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+          \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+          \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+          \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+          \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+          \""type\"": \""User\"",\n+          \""site_admin\"": false\n+        },\n+        \""body\"": \""Fixes #570\"",\n+        \""created_at\"": \""2019-09-03T00:23:39Z\"",\n+        \""updated_at\"": \""2019-09-03T00:32:28Z\"",\n+        \""closed_at\"": \""2019-09-03T00:32:28Z\"",\n+        \""merged_at\"": \""2019-09-03T00:32:28Z\"",\n+        \""merge_commit_sha\"": \""2dc5c8dc259a0606162673d394ba8cc1c6f54428\"",\n+        \""assignee\"": null,\n+        \""assignees\"": [\n+\n+        ],\n+        \""requested_reviewers\"": [\n+\n+        ],\n+        \""requested_teams\"": [\n+\n+        ],\n+        \""labels\"": [\n+\n+        ],\n+        \""milestone\"": null,\n+        \""draft\"": false,\n+        \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/commits\"",\n+        \""review_comments_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/comments\"",\n+        \""review_comment_url\"": \""https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\"",\n+        \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/issues/571/comments\"",\n+        \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\"",\n+        \""head\"": {\n+          \""label\"": \""simonw:detect-fts\"",\n+          \""ref\"": \""detect-fts\"",\n+          \""sha\"": \""a85239f69261c10f1a9f90514c8b5d113cb94585\"",\n+          \""user\"": {\n+            \""login\"": \""simonw\"",\n+            \""id\"": 9599,\n+            \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+            \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+            \""gravatar_id\"": \""\"",\n+            \""url\"": \""https://api.github.com/users/simonw\"",\n+            \""html_url\"": \""https://github.com/simonw\"",\n+            \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+            \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+            \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+            \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+            \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+            \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+            \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+            \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+            \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+            \""type\"": \""User\"",\n+            \""site_admin\"": false\n+          },\n+          \""repo\"": {\n+            \""id\"": 107914493,\n+            \""node_id\"": \""MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\"",\n+            \""name\"": \""datasette\"",\n+            \""full_name\"": \""simonw/datasette\"",\n+            \""private\"": false,\n+            \""owner\"": {\n+              \""login\"": \""simonw\"",\n+              \""id\"": 9599,\n+              \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+              \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+              \""gravatar_id\"": \""\"",\n+              \""url\"": \""https://api.github.com/users/simonw\"",\n+              \""html_url\"": \""https://github.com/simonw\"",\n+              \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+              \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+              \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+              \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+              \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+              \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+              \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+              \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+              \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+              \""type\"": \""User\"",\n+              \""site_admin\"": false\n+            },\n+            \""html_url\"": \""https://github.com/simonw/datasette\"",\n+            \""description\"": \""An open source multi-tool for exploring and publishing data\"",\n+            \""fork\"": false,\n+            \""url\"": \""https://api.github.com/repos/simonw/datasette\"",\n+            \""forks_url\"": \""https://api.github.com/repos/simonw/datasette/forks\"",\n+            \""keys_url\"": \""https://api.github.com/repos/simonw/datasette/keys{/key_id}\"",\n+            \""collaborators_url\"": \""https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\"",\n+            \""teams_url\"": \""https://api.github.com/repos/simonw/datasette/teams\"",\n+            \""hooks_url\"": \""https://api.github.com/repos/simonw/datasette/hooks\"",\n+            \""issue_events_url\"": \""https://api.github.com/repos/simonw/datasette/issues/events{/number}\"",\n+            \""events_url\"": \""https://api.github.com/repos/simonw/datasette/events\"",\n+            \""assignees_url\"": \""https://api.github.com/repos/simonw/datasette/assignees{/user}\"",\n+            \""branches_url\"": \""https://api.github.com/repos/simonw/datasette/branches{/branch}\"",\n+            \""tags_url\"": \""https://api.github.com/repos/simonw/datasette/tags\"",\n+            \""blobs_url\"": \""https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\"",\n+            \""git_tags_url\"": \""https://api.github.com/repos/simonw/datasette/git/tags{/sha}\"",\n+            \""git_refs_url\"": \""https://api.github.com/repos/simonw/datasette/git/refs{/sha}\"",\n+            \""trees_url\"": \""https://api.github.com/repos/simonw/datasette/git/trees{/sha}\"",\n+            \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/{sha}\"",\n+            \""languages_url\"": \""https://api.github.com/repos/simonw/datasette/languages\"",\n+            \""stargazers_url\"": \""https://api.github.com/repos/simonw/datasette/stargazers\"",\n+            \""contributors_url\"": \""https://api.github.com/repos/simonw/datasette/contributors\"",\n+            \""subscribers_url\"": \""https://api.github.com/repos/simonw/datasette/subscribers\"",\n+            \""subscription_url\"": \""https://api.github.com/repos/simonw/datasette/subscription\"",\n+            \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/commits{/sha}\"",\n+            \""git_commits_url\"": \""https://api.github.com/repos/simonw/datasette/git/commits{/sha}\"",\n+            \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/comments{/number}\"",\n+            \""issue_comment_url\"": \""https://api.github.com/repos/simonw/datasette/issues/comments{/number}\"",\n+            \""contents_url\"": \""https://api.github.com/repos/simonw/datasette/contents/{+path}\"",\n+            \""compare_url\"": \""https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\"",\n+            \""merges_url\"": \""https://api.github.com/repos/simonw/datasette/merges\"",\n+            \""archive_url\"": \""https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\"",\n+            \""downloads_url\"": \""https://api.github.com/repos/simonw/datasette/downloads\"",\n+            \""issues_url\"": \""https://api.github.com/repos/simonw/datasette/issues{/number}\"",\n+            \""pulls_url\"": \""https://api.github.com/repos/simonw/datasette/pulls{/number}\"",\n+            \""milestones_url\"": \""https://api.github.com/repos/simonw/datasette/milestones{/number}\"",\n+            \""notifications_url\"": \""https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\"",\n+            \""labels_url\"": \""https://api.github.com/repos/simonw/datasette/labels{/name}\"",\n+            \""releases_url\"": \""https://api.github.com/repos/simonw/datasette/releases{/id}\"",\n+            \""deployments_url\"": \""https://api.github.com/repos/simonw/datasette/deployments\"",\n+            \""created_at\"": \""2017-10-23T00:39:03Z\"",\n+            \""updated_at\"": \""2020-07-27T20:42:15Z\"",\n+            \""pushed_at\"": \""2020-07-26T01:21:05Z\"",\n+            \""git_url\"": \""git://github.com/simonw/datasette.git\"",\n+            \""ssh_url\"": \""git@github.com:simonw/datasette.git\"",\n+            \""clone_url\"": \""https://github.com/simonw/datasette.git\"",\n+            \""svn_url\"": \""https://github.com/simonw/datasette\"",\n+            \""homepage\"": \""http://datasette.readthedocs.io/\"",\n+            \""size\"": 3487,\n+            \""stargazers_count\"": 3642,\n+            \""watchers_count\"": 3642,\n+            \""language\"": \""Python\"",\n+            \""has_issues\"": true,\n+            \""has_projects\"": false,\n+            \""has_downloads\"": true,\n+            \""has_wiki\"": true,\n+            \""has_pages\"": false,\n+            \""forks_count\"": 206,\n+            \""mirror_url\"": null,\n+            \""archived\"": false,\n+            \""disabled\"": false,\n+            \""open_issues_count\"": 190,\n+            \""license\"": {\n+              \""key\"": \""apache-2.0\"",\n+              \""name\"": \""Apache License 2.0\"",\n+              \""spdx_id\"": \""Apache-2.0\"",\n+              \""url\"": \""https://api.github.com/licenses/apache-2.0\"",\n+              \""node_id\"": \""MDc6TGljZW5zZTI=\""\n+            },\n+            \""forks\"": 206,\n+            \""open_issues\"": 190,\n+            \""watchers\"": 3642,\n+            \""default_branch\"": \""master\""\n+          }\n+        },\n+        \""base\"": {\n+          \""label\"": \""simonw:master\"",\n+          \""ref\"": \""master\"",\n+          \""sha\"": \""f04deebec4f3842f7bd610cd5859de529f77d50e\"",\n+          \""user\"": {\n+            \""login\"": \""simonw\"",\n+            \""id\"": 9599,\n+            \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+            \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+            \""gravatar_id\"": \""\"",\n+            \""url\"": \""https://api.github.com/users/simonw\"",\n+            \""html_url\"": \""https://github.com/simonw\"",\n+            \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+            \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+            \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+            \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+            \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+            \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+            \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+            \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+            \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+            \""type\"": \""User\"",\n+            \""site_admin\"": false\n+          },\n+          \""repo\"": {\n+            \""id\"": 107914493,\n+            \""node_id\"": \""MDEwOlJlcG9zaXRvcnkxMDc5MTQ0OTM=\"",\n+            \""name\"": \""datasette\"",\n+            \""full_name\"": \""simonw/datasette\"",\n+            \""private\"": false,\n+            \""owner\"": {\n+              \""login\"": \""simonw\"",\n+              \""id\"": 9599,\n+              \""node_id\"": \""MDQ6VXNlcjk1OTk=\"",\n+              \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"",\n+              \""gravatar_id\"": \""\"",\n+              \""url\"": \""https://api.github.com/users/simonw\"",\n+              \""html_url\"": \""https://github.com/simonw\"",\n+              \""followers_url\"": \""https://api.github.com/users/simonw/followers\"",\n+              \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"",\n+              \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"",\n+              \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"",\n+              \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"",\n+              \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"",\n+              \""repos_url\"": \""https://api.github.com/users/simonw/repos\"",\n+              \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"",\n+              \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"",\n+              \""type\"": \""User\"",\n+              \""site_admin\"": false\n+            },\n+            \""html_url\"": \""https://github.com/simonw/datasette\"",\n+            \""description\"": \""An open source multi-tool for exploring and publishing data\"",\n+            \""fork\"": false,\n+            \""url\"": \""https://api.github.com/repos/simonw/datasette\"",\n+            \""forks_url\"": \""https://api.github.com/repos/simonw/datasette/forks\"",\n+            \""keys_url\"": \""https://api.github.com/repos/simonw/datasette/keys{/key_id}\"",\n+            \""collaborators_url\"": \""https://api.github.com/repos/simonw/datasette/collaborators{/collaborator}\"",\n+            \""teams_url\"": \""https://api.github.com/repos/simonw/datasette/teams\"",\n+            \""hooks_url\"": \""https://api.github.com/repos/simonw/datasette/hooks\"",\n+            \""issue_events_url\"": \""https://api.github.com/repos/simonw/datasette/issues/events{/number}\"",\n+            \""events_url\"": \""https://api.github.com/repos/simonw/datasette/events\"",\n+            \""assignees_url\"": \""https://api.github.com/repos/simonw/datasette/assignees{/user}\"",\n+            \""branches_url\"": \""https://api.github.com/repos/simonw/datasette/branches{/branch}\"",\n+            \""tags_url\"": \""https://api.github.com/repos/simonw/datasette/tags\"",\n+            \""blobs_url\"": \""https://api.github.com/repos/simonw/datasette/git/blobs{/sha}\"",\n+            \""git_tags_url\"": \""https://api.github.com/repos/simonw/datasette/git/tags{/sha}\"",\n+            \""git_refs_url\"": \""https://api.github.com/repos/simonw/datasette/git/refs{/sha}\"",\n+            \""trees_url\"": \""https://api.github.com/repos/simonw/datasette/git/trees{/sha}\"",\n+            \""statuses_url\"": \""https://api.github.com/repos/simonw/datasette/statuses/{sha}\"",\n+            \""languages_url\"": \""https://api.github.com/repos/simonw/datasette/languages\"",\n+            \""stargazers_url\"": \""https://api.github.com/repos/simonw/datasette/stargazers\"",\n+            \""contributors_url\"": \""https://api.github.com/repos/simonw/datasette/contributors\"",\n+            \""subscribers_url\"": \""https://api.github.com/repos/simonw/datasette/subscribers\"",\n+            \""subscription_url\"": \""https://api.github.com/repos/simonw/datasette/subscription\"",\n+            \""commits_url\"": \""https://api.github.com/repos/simonw/datasette/commits{/sha}\"",\n+            \""git_commits_url\"": \""https://api.github.com/repos/simonw/datasette/git/commits{/sha}\"",\n+            \""comments_url\"": \""https://api.github.com/repos/simonw/datasette/comments{/number}\"",\n+            \""issue_comment_url\"": \""https://api.github.com/repos/simonw/datasette/issues/comments{/number}\"",\n+            \""contents_url\"": \""https://api.github.com/repos/simonw/datasette/contents/{+path}\"",\n+            \""compare_url\"": \""https://api.github.com/repos/simonw/datasette/compare/{base}...{head}\"",\n+            \""merges_url\"": \""https://api.github.com/repos/simonw/datasette/merges\"",\n+            \""archive_url\"": \""https://api.github.com/repos/simonw/datasette/{archive_format}{/ref}\"",\n+            \""downloads_url\"": \""https://api.github.com/repos/simonw/datasette/downloads\"",\n+            \""issues_url\"": \""https://api.github.com/repos/simonw/datasette/issues{/number}\"",\n+            \""pulls_url\"": \""https://api.github.com/repos/simonw/datasette/pulls{/number}\"",\n+            \""milestones_url\"": \""https://api.github.com/repos/simonw/datasette/milestones{/number}\"",\n+            \""notifications_url\"": \""https://api.github.com/repos/simonw/datasette/notifications{?since,all,participating}\"",\n+            \""labels_url\"": \""https://api.github.com/repos/simonw/datasette/labels{/name}\"",\n+            \""releases_url\"": \""https://api.github.com/repos/simonw/datasette/releases{/id}\"",\n+            \""deployments_url\"": \""https://api.github.com/repos/simonw/datasette/deployments\"",\n+            \""created_at\"": \""2017-10-23T00:39:03Z\"",\n+            \""updated_at\"": \""2020-07-27T20:42:15Z\"",\n+            \""pushed_at\"": \""2020-07-26T01:21:05Z\"",\n+            \""git_url\"": \""git://github.com/simonw/datasette.git\"",\n+            \""ssh_url\"": \""git@github.com:simonw/datasette.git\"",\n+            \""clone_url\"": \""https://github.com/simonw/datasette.git\"",\n+            \""svn_url\"": \""https://github.com/simonw/datasette\"",\n+            \""homepage\"": \""http://datasette.readthedocs.io/\"",\n+            \""size\"": 3487,\n+            \""stargazers_count\"": 3642,\n+            \""watchers_count\"": 3642,\n+            \""language\"": \""Python\"",\n+            \""has_issues\"": true,\n+            \""has_projects\"": false,\n+            \""has_downloads\"": true,\n+            \""has_wiki\"": true,\n+            \""has_pages\"": false,\n+            \""forks_count\"": 206,\n+            \""mirror_url\"": null,\n+            \""archived\"": false,\n+            \""disabled\"": false,\n+            \""open_issues_count\"": 190,\n+            \""license\"": {\n+              \""key\"": \""apache-2.0\"",\n+              \""name\"": \""Apache License 2.0\"",\n+              \""spdx_id\"": \""Apache-2.0\"",\n+              \""url\"": \""https://api.github.com/licenses/apache-2.0\"",\n+              \""node_id\"": \""MDc6TGljZW5zZTI=\""\n+            },\n+            \""forks\"": 206,\n+            \""open_issues\"": 190,\n+            \""watchers\"": 3642,\n+            \""default_branch\"": \""master\""\n+          }\n+        },\n+        \""_links\"": {\n+          \""self\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571\""\n+          },\n+          \""html\"": {\n+            \""href\"": \""https://github.com/simonw/datasette/pull/571\""\n+          },\n+          \""issue\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/issues/571\""\n+          },\n+          \""comments\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/issues/571/comments\""\n+          },\n+          \""review_comments\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/comments\""\n+          },\n+          \""review_comment\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/comments{/number}\""\n+          },\n+          \""commits\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/pulls/571/commits\""\n+          },\n+          \""statuses\"": {\n+            \""href\"": \""https://api.github.com/repos/simonw/datasette/statuses/a85239f69261c10f1a9f90514c8b5d113cb94585\""\n+          }\n+        },\n+        \""author_association\"": \""OWNER\"",\n+        \""active_lock_reason\"": null,\n+        \""merged\"": true,\n+        \""mergeable\"": null,\n+        \""rebaseable\"": null,\n+        \""mergeable_state\"": \""unknown\"",\n+        \""merged_by\"": {"",
        ""path"": ""tests/pull_requests.json"",
        ""position"": 342,
        ""original_position"": 342,
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": ""Running this should create a `merged_by` column on the `pull_requests` table which is a foreign key to the `users` table."",
        ""created_at"": ""2020-10-06T21:22:47Z"",
        ""updated_at"": ""2020-10-20T20:56:33Z"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""self"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500603838""
            },
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500603838""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 500606198,
        ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjE5OA=="",
        ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198"",
        ""pull_request_review_id"": 503368921,
        ""diff_hunk"": ""@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+    return json.load(open(pathlib.Path(__file__).parent / \""pull_requests.json\""))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+    db = sqlite_utils.Database(memory=True)\n+    db[\""repos\""].insert(\n+        {\""id\"": 1},\n+        pk=\""id\"",\n+        columns={\""organization\"": int, \""topics\"": str, \""name\"": str, \""description\"": str},\n+    )\n+    utils.save_pull_requests(db, pull_requests, {\""id\"": 1})\n+    return db\n+\n+\n+def test_tables(db):\n+    assert {\""pull_requests\"", \""users\"", \""repos\"", \""milestones\""} == set(\n+        db.table_names()\n+    )\n+    assert {\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""repo\"", other_table=\""repos\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"",\n+            column=\""milestone\"",\n+            other_table=\""milestones\"",\n+            other_column=\""id\"",\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""assignee\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""user\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+    } == set(db[\""pull_requests\""].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+    pull_request_rows = list(db[\""pull_requests\""].rows)\n+    assert [\n+        {\n+            'id': 313384926,\n+            'node_id': 'MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2',\n+            'number': 571,\n+            'state': 'closed',\n+            'locked': 0,\n+            'title': 'detect_fts now works with alternative table escaping',\n+            'user': 9599,\n+            'body': 'Fixes #570',\n+            'created_at': '2019-09-03T00:23:39Z',\n+            'updated_at': '2019-09-03T00:32:28Z',\n+            'closed_at': '2019-09-03T00:32:28Z',\n+            'merged_at': '2019-09-03T00:32:28Z',\n+            'merge_commit_sha': '2dc5c8dc259a0606162673d394ba8cc1c6f54428',\n+            'assignee': None,\n+            'milestone': None,\n+            'draft': 0,\n+            'head': 'a85239f69261c10f1a9f90514c8b5d113cb94585',\n+            'base': 'f04deebec4f3842f7bd610cd5859de529f77d50e',\n+            'author_association': 'OWNER',\n+            'merged': 1,\n+            'mergeable': None,\n+            'rebaseable': None,\n+            'mergeable_state': 'unknown',\n+            'merged_by': '{\""login\"": \""simonw\"", \""id\"": 9599, \""node_id\"": \""MDQ6VXNlcjk1OTk=\"", \""avatar_url\"": \""https://avatars0.githubusercontent.com/u/9599?v=4\"", \""gravatar_id\"": \""\"", \""url\"": \""https://api.github.com/users/simonw\"", \""html_url\"": \""https://github.com/simonw\"", \""followers_url\"": \""https://api.github.com/users/simonw/followers\"", \""following_url\"": \""https://api.github.com/users/simonw/following{/other_user}\"", \""gists_url\"": \""https://api.github.com/users/simonw/gists{/gist_id}\"", \""starred_url\"": \""https://api.github.com/users/simonw/starred{/owner}{/repo}\"", \""subscriptions_url\"": \""https://api.github.com/users/simonw/subscriptions\"", \""organizations_url\"": \""https://api.github.com/users/simonw/orgs\"", \""repos_url\"": \""https://api.github.com/users/simonw/repos\"", \""events_url\"": \""https://api.github.com/users/simonw/events{/privacy}\"", \""received_events_url\"": \""https://api.github.com/users/simonw/received_events\"", \""type\"": \""User\"", \""site_admin\"": false}',"",
        ""path"": ""tests/test_pull_requests.py"",
        ""position"": null,
        ""original_position"": 76,
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": ""See above - this should be 9599, an integer reference to the row in the users table."",
        ""created_at"": ""2020-10-06T21:27:43Z"",
        ""updated_at"": ""2020-10-20T20:56:33Z"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""self"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606198""
            },
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606198""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 500606665,
        ""node_id"": ""MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDUwMDYwNjY2NQ=="",
        ""url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665"",
        ""pull_request_review_id"": 503368921,
        ""diff_hunk"": ""@@ -0,0 +1,124 @@\n+from github_to_sqlite import utils\n+import pytest\n+import pathlib\n+import sqlite_utils\n+from sqlite_utils.db import ForeignKey\n+import json\n+\n+\n+@pytest.fixture\n+def pull_requests():\n+    return json.load(open(pathlib.Path(__file__).parent / \""pull_requests.json\""))\n+\n+\n+@pytest.fixture\n+def db(pull_requests):\n+    db = sqlite_utils.Database(memory=True)\n+    db[\""repos\""].insert(\n+        {\""id\"": 1},\n+        pk=\""id\"",\n+        columns={\""organization\"": int, \""topics\"": str, \""name\"": str, \""description\"": str},\n+    )\n+    utils.save_pull_requests(db, pull_requests, {\""id\"": 1})\n+    return db\n+\n+\n+def test_tables(db):\n+    assert {\""pull_requests\"", \""users\"", \""repos\"", \""milestones\""} == set(\n+        db.table_names()\n+    )\n+    assert {\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""repo\"", other_table=\""repos\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"",\n+            column=\""milestone\"",\n+            other_table=\""milestones\"",\n+            other_column=\""id\"",\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""assignee\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+        ForeignKey(\n+            table=\""pull_requests\"", column=\""user\"", other_table=\""users\"", other_column=\""id\""\n+        ),\n+    } == set(db[\""pull_requests\""].foreign_keys)\n+\n+\n+def test_pull_requests(db):\n+    pull_request_rows = list(db[\""pull_requests\""].rows)\n+    assert [\n+        {\n+            'id': 313384926,"",
        ""path"": ""tests/test_pull_requests.py"",
        ""position"": null,
        ""original_position"": 53,
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": ""Minor detail: I use Black for this repo, which requires double quotes - running \""black .\"" in the root directory (with the latest version of Black) should handle this for you."",
        ""created_at"": ""2020-10-06T21:28:31Z"",
        ""updated_at"": ""2020-10-20T20:56:33Z"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""self"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/comments/500606665""
            },
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#discussion_r500606665""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""original_commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    }
]
```
That's a lot more interesting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482546,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735482546,MDEyOklzc3VlQ29tbWVudDczNTQ4MjU0Ng==,9599,simonw,2020-11-30T00:22:02Z,2020-11-30T00:22:02Z,MEMBER,"As for reviews... here's the output of `github-to-sqlite get https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48/reviews --accept 'application/vnd.github.v3+json'`

```json
[
    {
        ""id"": 503368921,
        ""node_id"": ""MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTAzMzY4OTIx"",
        ""user"": {
            ""login"": ""simonw"",
            ""id"": 9599,
            ""node_id"": ""MDQ6VXNlcjk1OTk="",
            ""avatar_url"": ""https://avatars0.githubusercontent.com/u/9599?u=5968723deb1a55b82620e106f5ca58e9b11a0942&v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/simonw"",
            ""html_url"": ""https://github.com/simonw"",
            ""followers_url"": ""https://api.github.com/users/simonw/followers"",
            ""following_url"": ""https://api.github.com/users/simonw/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/simonw/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/simonw/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/simonw/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/simonw/orgs"",
            ""repos_url"": ""https://api.github.com/users/simonw/repos"",
            ""events_url"": ""https://api.github.com/users/simonw/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/simonw/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": """",
        ""state"": ""CHANGES_REQUESTED"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""MEMBER"",
        ""_links"": {
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-503368921""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""submitted_at"": ""2020-10-06T21:28:40Z"",
        ""commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 513118561,
        ""node_id"": ""MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTE4NTYx"",
        ""user"": {
            ""login"": ""adamjonas"",
            ""id"": 755825,
            ""node_id"": ""MDQ6VXNlcjc1NTgyNQ=="",
            ""avatar_url"": ""https://avatars1.githubusercontent.com/u/755825?v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/adamjonas"",
            ""html_url"": ""https://github.com/adamjonas"",
            ""followers_url"": ""https://api.github.com/users/adamjonas/followers"",
            ""following_url"": ""https://api.github.com/users/adamjonas/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/adamjonas/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/adamjonas/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/adamjonas/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/adamjonas/orgs"",
            ""repos_url"": ""https://api.github.com/users/adamjonas/repos"",
            ""events_url"": ""https://api.github.com/users/adamjonas/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/adamjonas/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": """",
        ""state"": ""COMMENTED"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""CONTRIBUTOR"",
        ""_links"": {
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513118561""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""submitted_at"": ""2020-10-20T20:45:05Z"",
        ""commit_id"": ""4f33b850bd37829262dd29e1c520afffebedc19c""
    },
    {
        ""id"": 513127529,
        ""node_id"": ""MDE3OlB1bGxSZXF1ZXN0UmV2aWV3NTEzMTI3NTI5"",
        ""user"": {
            ""login"": ""adamjonas"",
            ""id"": 755825,
            ""node_id"": ""MDQ6VXNlcjc1NTgyNQ=="",
            ""avatar_url"": ""https://avatars1.githubusercontent.com/u/755825?v=4"",
            ""gravatar_id"": """",
            ""url"": ""https://api.github.com/users/adamjonas"",
            ""html_url"": ""https://github.com/adamjonas"",
            ""followers_url"": ""https://api.github.com/users/adamjonas/followers"",
            ""following_url"": ""https://api.github.com/users/adamjonas/following{/other_user}"",
            ""gists_url"": ""https://api.github.com/users/adamjonas/gists{/gist_id}"",
            ""starred_url"": ""https://api.github.com/users/adamjonas/starred{/owner}{/repo}"",
            ""subscriptions_url"": ""https://api.github.com/users/adamjonas/subscriptions"",
            ""organizations_url"": ""https://api.github.com/users/adamjonas/orgs"",
            ""repos_url"": ""https://api.github.com/users/adamjonas/repos"",
            ""events_url"": ""https://api.github.com/users/adamjonas/events{/privacy}"",
            ""received_events_url"": ""https://api.github.com/users/adamjonas/received_events"",
            ""type"": ""User"",
            ""site_admin"": false
        },
        ""body"": """",
        ""state"": ""COMMENTED"",
        ""html_url"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529"",
        ""pull_request_url"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48"",
        ""author_association"": ""CONTRIBUTOR"",
        ""_links"": {
            ""html"": {
                ""href"": ""https://github.com/dogsheep/github-to-sqlite/pull/48#pullrequestreview-513127529""
            },
            ""pull_request"": {
                ""href"": ""https://api.github.com/repos/dogsheep/github-to-sqlite/pulls/48""
            }
        },
        ""submitted_at"": ""2020-10-20T20:57:33Z"",
        ""commit_id"": ""3a0d5c498f9faae4e40aab204cd01b965a4f61f3""
    }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/46#issuecomment-735482187,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/46,735482187,MDEyOklzc3VlQ29tbWVudDczNTQ4MjE4Nw==,9599,simonw,2020-11-30T00:20:11Z,2020-11-30T00:20:11Z,MEMBER,"Pull request are now added, thanks to  @adamjonas.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",664485022,Feature: pull request reviews and comments,
https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735465708,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54,735465708,MDEyOklzc3VlQ29tbWVudDczNTQ2NTcwOA==,9599,simonw,2020-11-29T22:08:46Z,2020-11-29T22:08:46Z,MEMBER,"Demo:
- https://github-to-sqlite.dogsheep.net/github/steps?_facet=repo
- https://github-to-sqlite.dogsheep.net/github/workflows
- https://github-to-sqlite.dogsheep.net/github/jobs","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753026003,github-to-sqlite workflows command,
https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464493,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54,735464493,MDEyOklzc3VlQ29tbWVudDczNTQ2NDQ5Mw==,9599,simonw,2020-11-29T21:57:32Z,2020-11-29T21:57:32Z,MEMBER,`$ github-to-sqlite workflows github.db simonw/datasette dogsheep/github-to-sqlite`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753026003,github-to-sqlite workflows command,
https://github.com/dogsheep/github-to-sqlite/issues/54#issuecomment-735464438,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54,735464438,MDEyOklzc3VlQ29tbWVudDczNTQ2NDQzOA==,9599,simonw,2020-11-29T21:57:08Z,2020-11-29T21:57:08Z,MEMBER,Inspired by this tweet from Michael Heap https://twitter.com/mheap/status/1333108608817631238,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",753026003,github-to-sqlite workflows command,
https://github.com/simonw/datasette/issues/1114#issuecomment-735447635,https://api.github.com/repos/simonw/datasette/issues/1114,735447635,MDEyOklzc3VlQ29tbWVudDczNTQ0NzYzNQ==,9599,simonw,2020-11-29T20:16:32Z,2020-11-29T20:17:11Z,OWNER,"The new Docker container is pushed to Docker Hub too. Here's it in action:

```
% docker pull datasetteproject/datasette
% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette --version
datasette, version 0.52.1
% docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/nps-spatialite.db --load-extension=spatialite
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
https://github.com/simonw/datasette/issues/1115#issuecomment-735447198,https://api.github.com/repos/simonw/datasette/issues/1115,735447198,MDEyOklzc3VlQ29tbWVudDczNTQ0NzE5OA==,9599,simonw,2020-11-29T20:12:49Z,2020-11-29T20:12:49Z,OWNER,"```
$ datasette ../shapefile-to-sqlite/nps-spatialite.db
Usage: datasette serve [OPTIONS] [FILES]...

Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

Try adding the --load-extension=spatialite option.

Read more: https://docs.datasette.io/en/stable/spatialite.html
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752995227,SpatiaLite error could suggest --load-extension=spatialite,
https://github.com/simonw/datasette/issues/1099#issuecomment-735444858,https://api.github.com/repos/simonw/datasette/issues/1099,735444858,MDEyOklzc3VlQ29tbWVudDczNTQ0NDg1OA==,9599,simonw,2020-11-29T19:51:58Z,2020-11-29T19:51:58Z,OWNER,"My fix in deb0be4ae56f191f121239b29e83dd53b62d6305 for #1098 was to have Datasette deliberately pretend that compound foreign keys don't exist: https://github.com/simonw/datasette/blob/deb0be4ae56f191f121239b29e83dd53b62d6305/datasette/utils/__init__.py#L470-L495

This workaround will need to be rethought to implement real support for them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743371103,Support linking to compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-735444698,https://api.github.com/repos/simonw/datasette/issues/1098,735444698,MDEyOklzc3VlQ29tbWVudDczNTQ0NDY5OA==,9599,simonw,2020-11-29T19:50:14Z,2020-11-29T19:50:14Z,OWNER,Demo of the fix: https://latest.datasette.io/fixtures/foreign_key_references (the compound foreign key columns do not link to anything),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-735443654,https://api.github.com/repos/simonw/datasette/issues/1098,735443654,MDEyOklzc3VlQ29tbWVudDczNTQ0MzY1NA==,9599,simonw,2020-11-29T19:41:01Z,2020-11-29T19:41:01Z,OWNER,Fix is out in 0.52.1: https://docs.datasette.io/en/latest/changelog.html#v0-52-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1114#issuecomment-735443626,https://api.github.com/repos/simonw/datasette/issues/1114,735443626,MDEyOklzc3VlQ29tbWVudDczNTQ0MzYyNg==,9599,simonw,2020-11-29T19:40:49Z,2020-11-29T19:40:49Z,OWNER,Fix is out in 0.52.1: https://docs.datasette.io/en/latest/changelog.html#v0-52-1,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
https://github.com/simonw/datasette/issues/1098#issuecomment-735442226,https://api.github.com/repos/simonw/datasette/issues/1098,735442226,MDEyOklzc3VlQ29tbWVudDczNTQ0MjIyNg==,9599,simonw,2020-11-29T19:28:04Z,2020-11-29T19:28:04Z,OWNER,"For the moment I'm going to solve this by teaching Datasette's internal introspection methods - in particular these ones - to ignore compound foreign keys entirely:

https://github.com/simonw/datasette/blob/e800ffcf7cc6a915eb554b369c654f87162575e5/datasette/utils/__init__.py#L470-L505","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/123#issuecomment-735440555,https://api.github.com/repos/simonw/datasette/issues/123,735440555,MDEyOklzc3VlQ29tbWVudDczNTQ0MDU1NQ==,11912854,jsancho-gpl,2020-11-29T19:12:30Z,2020-11-29T19:12:30Z,NONE,"[datasette-connectors](https://github.com/pytables/datasette-connectors) provides an API for making connectors for any file based database. For example, [datasette-pytables](https://github.com/pytables/datasette-pytables) is a connector for HDF5 files, so now is possible to use this type of files with Datasette.

It'd be nice if Datasette coud provide that API directly, for other file formats and for urls too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats,
https://github.com/simonw/datasette/issues/1114#issuecomment-735436014,https://api.github.com/repos/simonw/datasette/issues/1114,735436014,MDEyOklzc3VlQ29tbWVudDczNTQzNjAxNA==,2182,danp,2020-11-29T18:33:30Z,2020-11-29T18:33:30Z,CONTRIBUTOR,Thank you!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
https://github.com/simonw/datasette/issues/1114#issuecomment-735429041,https://api.github.com/repos/simonw/datasette/issues/1114,735429041,MDEyOklzc3VlQ29tbWVudDczNTQyOTA0MQ==,9599,simonw,2020-11-29T17:36:18Z,2020-11-29T17:36:18Z,OWNER,"This is a great catch, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752966476,--load-extension=spatialite not working with datasetteproject/datasette docker image,
https://github.com/simonw/datasette/issues/1102#issuecomment-735356882,https://api.github.com/repos/simonw/datasette/issues/1102,735356882,MDEyOklzc3VlQ29tbWVudDczNTM1Njg4Mg==,9599,simonw,2020-11-29T07:47:22Z,2020-11-29T07:47:22Z,OWNER,Live: https://docs.datasette.io/en/latest/testing_plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749289611,Plugin testing docs should show datasette.client,
https://github.com/simonw/datasette/issues/1111#issuecomment-735320736,https://api.github.com/repos/simonw/datasette/issues/1111,735320736,MDEyOklzc3VlQ29tbWVudDczNTMyMDczNg==,9599,simonw,2020-11-29T02:46:23Z,2020-11-29T02:46:23Z,OWNER,"This is a really useful bug report, thanks! I agree: more aggressive timeouts on table counts sounds like the right solution here.

I've learned that avoiding `count(*)` is crucial for handling these larger databases. Datasette has been getting better about that over time but there are still some edge-cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",751195017,Accessing a database's `.json` is slow for very large SQLite files,
https://github.com/simonw/datasette/issues/1108#issuecomment-735320499,https://api.github.com/repos/simonw/datasette/issues/1108,735320499,MDEyOklzc3VlQ29tbWVudDczNTMyMDQ5OQ==,9599,simonw,2020-11-29T02:42:42Z,2020-11-29T02:42:42Z,OWNER,https://docs.datasette.io/en/stable/config.html now redirects correctly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750087350,Configure /en/stable/config.html redirect when I ship 0.52,
https://github.com/simonw/datasette/issues/1113#issuecomment-735304071,https://api.github.com/repos/simonw/datasette/issues/1113,735304071,MDEyOklzc3VlQ29tbWVudDczNTMwNDA3MQ==,9599,simonw,2020-11-28T23:23:31Z,2020-11-28T23:23:31Z,OWNER,https://github.com/simonw/datasette/blob/bbde835a1fec01458e8d00929e7bab6d6a5ba948/datasette/views/table.py#L1013-L1026 - looks like I fixed this in #1088.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752789159,500 error on row page if query against foreign keys hits time limit,
https://github.com/simonw/datasette/pull/1112#issuecomment-735283033,https://api.github.com/repos/simonw/datasette/issues/1112,735283033,MDEyOklzc3VlQ29tbWVudDczNTI4MzAzMw==,9599,simonw,2020-11-28T19:53:36Z,2020-11-28T19:53:36Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752749485,Fix --metadata doc usage,
https://github.com/simonw/datasette/issues/493#issuecomment-735281577,https://api.github.com/repos/simonw/datasette/issues/493,735281577,MDEyOklzc3VlQ29tbWVudDczNTI4MTU3Nw==,50527,jefftriplett,2020-11-28T19:39:53Z,2020-11-28T19:39:53Z,CONTRIBUTOR,"I was confused by `--config` and I tried passing the json from datasette-ripgrep into `config.json` just as a wild guess. 

A short term solution might be pointing out in plugins that their snippet json can go in `metadata.json` at least makes it easier to search for config options or to know where to start if someone is new. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/pull/1112#issuecomment-735279733,https://api.github.com/repos/simonw/datasette/issues/1112,735279733,MDEyOklzc3VlQ29tbWVudDczNTI3OTczMw==,22429695,codecov[bot],2020-11-28T19:24:28Z,2020-11-28T19:24:28Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=h1) Report
> Merging [#1112](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=desc) (1a30fc2) into [main](https://codecov.io/gh/simonw/datasette/commit/37d18a5bce08c9ee53c080f613bae84fc2ccc853?el=desc) (37d18a5) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1112/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1112   +/-   ##
=======================================
  Coverage   91.44%   91.44%           
=======================================
  Files          30       30           
  Lines        3833     3833           
=======================================
  Hits         3505     3505           
  Misses        328      328           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=footer). Last update [37d18a5...1a30fc2](https://codecov.io/gh/simonw/datasette/pull/1112?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752749485,Fix --metadata doc usage,
https://github.com/simonw/datasette/pull/1112#issuecomment-735279355,https://api.github.com/repos/simonw/datasette/issues/1112,735279355,MDEyOklzc3VlQ29tbWVudDczNTI3OTM1NQ==,50527,jefftriplett,2020-11-28T19:21:09Z,2020-11-28T19:21:09Z,CONTRIBUTOR,"(Even more annoying is that I see my editor leaked an extra delete space at the end of the line. I'm happy to rebuild this to be less annoying, but you probably don't want the changelog update either way)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",752749485,Fix --metadata doc usage,
https://github.com/simonw/datasette/issues/1110#issuecomment-733432916,https://api.github.com/repos/simonw/datasette/issues/1110,733432916,MDEyOklzc3VlQ29tbWVudDczMzQzMjkxNg==,9599,simonw,2020-11-25T03:04:29Z,2020-11-25T03:04:29Z,OWNER,Already have a pattern for extra packages in the form of the `--spatialite` feature: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/utils/__init__.py#L50-L55,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750330029,datasette publish option for installing extra apt-get packages,
https://github.com/simonw/datasette/issues/1110#issuecomment-733432768,https://api.github.com/repos/simonw/datasette/issues/1110,733432768,MDEyOklzc3VlQ29tbWVudDczMzQzMjc2OA==,9599,simonw,2020-11-25T03:04:00Z,2020-11-25T03:04:00Z,OWNER,"Design:

    datasette publish cloudrun blah.db --apt-get-install ripgrep --install datasette-ripgrep","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750330029,datasette publish option for installing extra apt-get packages,
https://github.com/simonw/datasette/issues/860#issuecomment-733288841,https://api.github.com/repos/simonw/datasette/issues/860,733288841,MDEyOklzc3VlQ29tbWVudDczMzI4ODg0MQ==,9599,simonw,2020-11-24T23:19:47Z,2020-11-24T23:20:24Z,OWNER,Here's what I have today - it's an undocumented `datasette.metadata()` method that returns a full JSON dictionary of values OR a single value if the optional `key=` argument is provided: https://github.com/simonw/datasette/blob/f2e2bfcdd9ad4891f3f66c9104c09943d943ffe4/datasette/app.py#L357-L388,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-733288522,https://api.github.com/repos/simonw/datasette/issues/860,733288522,MDEyOklzc3VlQ29tbWVudDczMzI4ODUyMg==,9599,simonw,2020-11-24T23:18:47Z,2020-11-24T23:18:47Z,OWNER,"In #942 I want to add support for per-column metadata - which means this new lookup mechanism will need to be able to answer the question ""what description is available for this column"".

So what should the `.metadata()` method look like? A couple of options:

- `datasette.metadata(""description"", table=x, database=y)` - can take optional `column=` too.
- `datasette.table_metadata(""description"", table=x, database=y)` and `datasette.database_metadata(""description"", database=y)` and so on - multiple methods for the different types of metadata.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-733287619,https://api.github.com/repos/simonw/datasette/issues/860,733287619,MDEyOklzc3VlQ29tbWVudDczMzI4NzYxOQ==,9599,simonw,2020-11-24T23:16:21Z,2020-11-24T23:16:21Z,OWNER,I'll also allow any key to be looked up - so if users want to invent their own metadata keys other than the default `license_url` etc they can do so.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-733287416,https://api.github.com/repos/simonw/datasette/issues/860,733287416,MDEyOklzc3VlQ29tbWVudDczMzI4NzQxNg==,9599,simonw,2020-11-24T23:15:44Z,2020-11-24T23:15:44Z,OWNER,"I'm going to go with a plugin hook (and Datasette method) that returns individual values - so you ask it for e.g. the `license_url` for a specific table and it returns a string or `None`.

The default plugin hook implementation that ships with Datasette will then implement cascading lookups against `metadata.json` - but other plugins will be able to provide their own implementations, which should make it easy to build a plugin that lets you keep metadata in a database file and edit it interactively.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1107#issuecomment-733261501,https://api.github.com/repos/simonw/datasette/issues/1107,733261501,MDEyOklzc3VlQ29tbWVudDczMzI2MTUwMQ==,9599,simonw,2020-11-24T22:09:11Z,2020-11-24T22:09:11Z,OWNER,Documentation: https://docs.datasette.io/en/latest/internals.html#setting-key,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1107#issuecomment-733257071,https://api.github.com/repos/simonw/datasette/issues/1107,733257071,MDEyOklzc3VlQ29tbWVudDczMzI1NzA3MQ==,9599,simonw,2020-11-24T21:59:32Z,2020-11-24T21:59:32Z,OWNER,I'm going to make this a documented method in https://docs.datasette.io/en/latest/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1105#issuecomment-733249176,https://api.github.com/repos/simonw/datasette/issues/1105,733249176,MDEyOklzc3VlQ29tbWVudDczMzI0OTE3Ng==,9599,simonw,2020-11-24T21:40:28Z,2020-11-24T21:40:28Z,OWNER,This rebranding is complete - #1107 is a follow-up internal refactor.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749982022,Rebrand config as settings,
https://github.com/simonw/datasette/issues/1106#issuecomment-733248437,https://api.github.com/repos/simonw/datasette/issues/1106,733248437,MDEyOklzc3VlQ29tbWVudDczMzI0ODQzNw==,9599,simonw,2020-11-24T21:38:50Z,2020-11-24T21:38:50Z,OWNER,"I used an ""exact redirect"" instead and it worked:

<img width=""654"" alt=""Edit_Redirects___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/100154418-60bf0d80-2e5a-11eb-92bf-527a14edb5c5.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1106#issuecomment-733247101,https://api.github.com/repos/simonw/datasette/issues/1106,733247101,MDEyOklzc3VlQ29tbWVudDczMzI0NzEwMQ==,9599,simonw,2020-11-24T21:35:29Z,2020-11-24T21:36:04Z,OWNER,"<img width=""861"" alt=""Edit_Redirects___Read_the_Docs"" src=""https://user-images.githubusercontent.com/9599/100153884-a3341a80-2e59-11eb-8d8c-7027c714b9cf.png"">

https://docs.datasette.io/en/latest/config.html isn't redirecting though, even after I tried running a rebuild of the `latest` version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1106#issuecomment-733245596,https://api.github.com/repos/simonw/datasette/issues/1106,733245596,MDEyOklzc3VlQ29tbWVudDczMzI0NTU5Ng==,9599,simonw,2020-11-24T21:32:11Z,2020-11-24T21:32:11Z,OWNER,https://docs.datasette.io/en/latest/settings.html is now live - need to redirect https://docs.datasette.io/en/latest/config.html to it using the ReadTheDocs redirects interface.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1107#issuecomment-733245097,https://api.github.com/repos/simonw/datasette/issues/1107,733245097,MDEyOklzc3VlQ29tbWVudDczMzI0NTA5Nw==,9599,simonw,2020-11-24T21:31:10Z,2020-11-24T21:31:10Z,OWNER,"Most of these use `plugin_config` which is unaffected. It looks like the only code I need to worry about is this trick in `datasette-graphl`: https://github.com/simonw/datasette-graphql/blob/483c9a9e203bb90365def3df8b8f01dda1e75865/datasette_graphql/utils.py#L456-L460

```python
class DatasetteSpecialConfig(wrapt.ObjectProxy):
    def config(self, key):
        if key == ""suggest_facets"":
            return False
        return self.__wrapped__.config(key)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1107#issuecomment-733244471,https://api.github.com/repos/simonw/datasette/issues/1107,733244471,MDEyOklzc3VlQ29tbWVudDczMzI0NDQ3MQ==,9599,simonw,2020-11-24T21:29:59Z,2020-11-24T21:29:59Z,OWNER,"I ran `rg '.config\(' datasette-*/` in my top-level directory:
```
datasette-sentry/test_datasette_sentry.py:    def plugin_config(self, name):
datasette-sentry/datasette_sentry.py:    config = datasette.plugin_config(""datasette-sentry"") or {}
datasette-render-markdown/datasette_render_markdown/__init__.py:        datasette.plugin_config(
datasette-render-images/datasette_render_images.py:        plugin_config = datasette.plugin_config(""datasette-render-images"") or {}
datasette-render-html/datasette_render_html.py:    config = datasette.plugin_config(
datasette-render-timestamps/datasette_render_timestamps/__init__.py:        datasette.plugin_config(
datasette-permissions-sql/datasette_permissions_sql/__init__.py:        for rule in datasette.plugin_config(""datasette-permissions-sql"") or []:
datasette-mask-columns/datasette_mask_columns/__init__.py:            datasette.plugin_config(""datasette-mask-columns"", database=database) or {}
datasette-mask-columns/datasette_mask_columns/__init__.py:    masks = datasette.plugin_config(""datasette-mask-columns"", database=database) or {}
datasette-media/datasette_media/__init__.py:    plugin_config = datasette.plugin_config(""datasette-media"") or {}
datasette-insert/datasette_insert/__init__.py:    plugin_config = datasette.plugin_config(""datasette-insert"") or {}
datasette-indieauth/datasette_indieauth/__init__.py:    plugin_config = datasette.plugin_config(""datasette-indieauth"") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config(""datasette-graphql"") or {}
datasette-graphql/datasette_graphql/__init__.py:    config = datasette.plugin_config(""datasette-graphql"") or {}
datasette-graphql/datasette_graphql/utils.py:            auto_camelcase=(datasette.plugin_config(""datasette-graphql"") or {}).get(
datasette-graphql/datasette_graphql/utils.py:    table_plugin_config = datasette.plugin_config(
datasette-graphql/datasette_graphql/utils.py:    def config(self, key):
datasette-graphql/datasette_graphql/utils.py:        return self.__wrapped__.config(key)
datasette-init/datasette_init/__init__.py:        config = datasette.plugin_config(""datasette-init"")
datasette-edit-templates/datasette_edit_templates/__init__.py:    plugin_config = datasette.plugin_config(""datasette-edit-templates"") or {}
datasette-cors/datasette_cors.py:    config = datasette.plugin_config(""datasette-cors"") or {}
datasette-cluster-map-old/build/lib/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-cluster-map/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-cluster-map/tests/test_cluster_map.py:async def test_plugin_config(db_path, config, table, expected_fragments):
datasette-configure-asgi/datasette_configure_asgi.py:        configs = datasette.plugin_config(""datasette-configure-asgi"") or []
datasette-configure-asgi/test_datasette_configure_asgi.py:    def plugin_config(self, name):
datasette-cluster-map-old/datasette_cluster_map/__init__.py:        datasette.plugin_config(""datasette-cluster-map"", database=database, table=table)
datasette-auth-passwords/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config(""datasette-auth-passwords"") or {}
datasette-auth-github/datasette_auth_github/views.py:def verify_config(config):
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config(""datasette-auth-github"")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-auth-github/datasette_auth_github/views.py:    config = datasette.plugin_config(""datasette-auth-github"")
datasette-auth-github/datasette_auth_github/views.py:    verify_config(config)
datasette-atom/datasette_atom/__init__.py:        plugin_config = datasette.plugin_config(""datasette-atom"")
datasette-auth-google/datasette_auth_google/__init__.py:    config = datasette.plugin_config(""datasette-auth-github"") or {}
datasette-auth-existing-cookies/test_datasette_auth_existing_cookies.py:    def plugin_config(self, name):
datasette-auth-passwords/build/lib/datasette_auth_passwords/__init__.py:    config = datasette.plugin_config(""datasette-auth-passwords"") or {}
datasette-annotate/datasette_annotate/utils.py:    plugin_config = datasette.plugin_config(""datasette-annotate"") or {}
datasette-auth-existing-cookies/datasette_auth_existing_cookies/__init__.py:    config = datasette.plugin_config(""datasette-auth-existing-cookies"") or {}
datasette-auth-simple/datasette_auth_simple/__init__.py:datasette.plugin_config(""datasette-auth-simple"")
datasette-auth-tokens/datasette_auth_tokens/__init__.py:        config = datasette.plugin_config(""datasette-auth-tokens"") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config(""datasette-block-robots"") or {}
datasette-block-robots/datasette_block_robots/__init__.py:    config = datasette.plugin_config(""datasette-block-robots"") or {}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1107#issuecomment-733241949,https://api.github.com/repos/simonw/datasette/issues/1107,733241949,MDEyOklzc3VlQ29tbWVudDczMzI0MTk0OQ==,9599,simonw,2020-11-24T21:24:26Z,2020-11-24T21:24:26Z,OWNER,Are there any plugins that use this API even though it isn't documented?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",750079085,Rename datasette.config() method to datasette.setting(),
https://github.com/simonw/datasette/issues/1106#issuecomment-733221359,https://api.github.com/repos/simonw/datasette/issues/1106,733221359,MDEyOklzc3VlQ29tbWVudDczMzIyMTM1OQ==,9599,simonw,2020-11-24T20:40:21Z,2020-11-24T20:40:21Z,OWNER,https://readthedocs.org/dashboard/datasette/redirects/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749983857,Rebrand and redirect config.rst as settings.rst,
https://github.com/simonw/datasette/issues/1104#issuecomment-733212084,https://api.github.com/repos/simonw/datasette/issues/1104,733212084,MDEyOklzc3VlQ29tbWVudDczMzIxMjA4NA==,9599,simonw,2020-11-24T20:20:33Z,2020-11-24T20:20:33Z,OWNER,I'll throw an error if a `config.json` file is detected.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749981663,config.json in directory config mode should be settings.json,
https://github.com/simonw/datasette/issues/226#issuecomment-733198051,https://api.github.com/repos/simonw/datasette/issues/226,733198051,MDEyOklzc3VlQ29tbWVudDczMzE5ODA1MQ==,9599,simonw,2020-11-24T19:52:46Z,2020-11-24T19:52:46Z,OWNER,This is well handled now: https://github.com/simonw/datasette/tree/0.51.1/tests/plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",315738696,Unit tests for installable plugins,
https://github.com/simonw/datasette/issues/1105#issuecomment-733190827,https://api.github.com/repos/simonw/datasette/issues/1105,733190827,MDEyOklzc3VlQ29tbWVudDczMzE5MDgyNw==,9599,simonw,2020-11-24T19:38:02Z,2020-11-24T19:38:02Z,OWNER,I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749982022,Rebrand config as settings,
https://github.com/simonw/datasette/issues/1103#issuecomment-733189737,https://api.github.com/repos/simonw/datasette/issues/1103,733189737,MDEyOklzc3VlQ29tbWVudDczMzE4OTczNw==,9599,simonw,2020-11-24T19:35:45Z,2020-11-24T19:35:45Z,OWNER,Part of #1105,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749979454,Rename /-/config to /-/settings,
https://github.com/simonw/datasette/issues/992#issuecomment-733189693,https://api.github.com/repos/simonw/datasette/issues/992,733189693,MDEyOklzc3VlQ29tbWVudDczMzE4OTY5Mw==,9599,simonw,2020-11-24T19:35:38Z,2020-11-24T19:35:38Z,OWNER,Part of #1105,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/1104#issuecomment-733189620,https://api.github.com/repos/simonw/datasette/issues/1104,733189620,MDEyOklzc3VlQ29tbWVudDczMzE4OTYyMA==,9599,simonw,2020-11-24T19:35:30Z,2020-11-24T19:35:30Z,OWNER,Part of #1105,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749981663,config.json in directory config mode should be settings.json,
https://github.com/simonw/datasette/issues/1103#issuecomment-733187586,https://api.github.com/repos/simonw/datasette/issues/1103,733187586,MDEyOklzc3VlQ29tbWVudDczMzE4NzU4Ng==,9599,simonw,2020-11-24T19:31:23Z,2020-11-24T19:31:23Z,OWNER,I'll set up a redirect from `/-/config` to `/-/settings`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749979454,Rename /-/config to /-/settings,
https://github.com/simonw/datasette/issues/992#issuecomment-733180289,https://api.github.com/repos/simonw/datasette/issues/992,733180289,MDEyOklzc3VlQ29tbWVudDczMzE4MDI4OQ==,9599,simonw,2020-11-24T19:16:30Z,2020-11-24T19:16:30Z,OWNER,Need to figure out the `--setting foo bar` alternative for this `--config foo:bar` logic: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/cli.py#L31-L63,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/992#issuecomment-733176252,https://api.github.com/repos/simonw/datasette/issues/992,733176252,MDEyOklzc3VlQ29tbWVudDczMzE3NjI1Mg==,9599,simonw,2020-11-24T19:07:49Z,2020-11-24T19:07:49Z,OWNER,I'm going to keep `--config` for the moment but show a deprecation warning that it will be gone in Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/992#issuecomment-733175965,https://api.github.com/repos/simonw/datasette/issues/992,733175965,MDEyOklzc3VlQ29tbWVudDczMzE3NTk2NQ==,9599,simonw,2020-11-24T19:07:13Z,2020-11-24T19:07:13Z,OWNER,"This is blocking progress on other metadata tickets like #860 because I want to split the concept of concrete metadata (source, license, etc) from configuration that currently lives in metadata (default sort order, default facets).

I'm going to solve this next to unblock that stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714449879,"Change ""--config foo:bar"" to ""--setting foo bar""",
https://github.com/simonw/datasette/issues/860#issuecomment-733175454,https://api.github.com/repos/simonw/datasette/issues/860,733175454,MDEyOklzc3VlQ29tbWVudDczMzE3NTQ1NA==,9599,simonw,2020-11-24T19:06:07Z,2020-11-24T19:06:07Z,OWNER,"I see two ways this plugin hook could work. It could be asked about a specific instance, database or table and return the full metadata for that object. OR it could ask for a specific metadata field - e.g. `source_url` for table X, and return that.

The more finely grained one would allow plugins to implement their own cascading rules pretty easily. Is there a reason it would be better for the hook to return an entire block of JSON for a specific table or database?

I also need to decide if this hook is just going to be about source/license/about displayed metadata, or if it will include the functionality that has been sneaking into `metadata.json` over time - stuff like page size, default sort order or default facets.

Perhaps I should split those out into a ""configuration"" concept first, after renaming `--config` to `--setting` in #992.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1101#issuecomment-732544590,https://api.github.com/repos/simonw/datasette/issues/1101,732544590,MDEyOklzc3VlQ29tbWVudDczMjU0NDU5MA==,9599,simonw,2020-11-24T02:22:55Z,2020-11-24T02:22:55Z,OWNER,"The trick I'm using here is to follow the `next_url` in order to paginate through all of the matching results. The loop calls the `data()` method multiple times, once for each page of results: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L304-L307","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1101#issuecomment-732543700,https://api.github.com/repos/simonw/datasette/issues/1101,732543700,MDEyOklzc3VlQ29tbWVudDczMjU0MzcwMA==,9599,simonw,2020-11-24T02:20:30Z,2020-11-24T02:20:30Z,OWNER,"Current design: https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette

```python
@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""test"",
        ""render"": render_demo,
        ""can_render"": can_render_demo,  # Optional
    }
```
Where `render_demo` looks something like this:
```python
async def render_demo(datasette, columns, rows):
    db = datasette.get_database()
    result = await db.execute(""select sqlite_version()"")
    first_row = "" | "".join(columns)
    lines = [first_row]
    lines.append(""="" * len(first_row))
    for row in rows:
        lines.append("" | "".join(row))
    return Response(
        ""\n"".join(lines),
        content_type=""text/plain; charset=utf-8"",
        headers={""x-sqlite-version"": result.first()[0]}
    )
```
Meanwhile here's where the CSV streaming mode is implemented: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/views/base.py#L297-L380","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",749283032,register_output_renderer() should support streaming data,
https://github.com/simonw/datasette/issues/1096#issuecomment-732542285,https://api.github.com/repos/simonw/datasette/issues/1096,732542285,MDEyOklzc3VlQ29tbWVudDczMjU0MjI4NQ==,9599,simonw,2020-11-24T02:16:22Z,2020-11-24T02:16:22Z,OWNER,"I'd like to implement this by first extending the `register_output_renderer()` hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743359646,TSV should be a default export option,
https://github.com/simonw/datasette/issues/860#issuecomment-731658059,https://api.github.com/repos/simonw/datasette/issues/860,731658059,MDEyOklzc3VlQ29tbWVudDczMTY1ODA1OQ==,9599,simonw,2020-11-22T00:31:47Z,2020-11-22T00:33:48Z,OWNER,"Documented behaviour right now, for metadata set at the instance level, is: https://docs.datasette.io/en/stable/metadata.html

> The above metadata will be displayed on the index page of your Datasette-powered site. The source and license information will also be included in the footer of every page served by Datasette.
>
> ...
> 
> Metadata at the top level of the JSON will be shown on the index page and in the footer on every page of the site. The license and source is expected to apply to all of your data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-731657660,https://api.github.com/repos/simonw/datasette/issues/860,731657660,MDEyOklzc3VlQ29tbWVudDczMTY1NzY2MA==,9599,simonw,2020-11-22T00:27:32Z,2020-11-22T00:31:54Z,OWNER,"Open question: how should cascading work? If a table is missing a field but the database or instance has it, should that value cascade down to the table?

It feels like `license` should definitely cascade: if an instance lists a certain `license` that should absolutely filter through to all databases and tables.

But... should the other fields cascade? Cascading `description` doesn't feel right at all, and neither does `title`.

What about `about` and `about_url` and `source` and `source_url`? I'm a bit torn on whether they should cascade or not. I'm leaning towards cascading them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/860#issuecomment-731657486,https://api.github.com/repos/simonw/datasette/issues/860,731657486,MDEyOklzc3VlQ29tbWVudDczMTY1NzQ4Ng==,9599,simonw,2020-11-22T00:25:34Z,2020-11-22T00:25:34Z,OWNER,"There are three layers of metadata: table, database and instance.

Currently the metadata fields are (ignoring not-quite-metadata like `sort` and `sort_desc`):

- `title`
- `description` (or `description_html`)
- `about` / `about_url`
- `source` / `source_url`
- `license` / `license_url`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642651572,Plugin hook for instance/database/table metadata,
https://github.com/simonw/datasette/issues/1084#issuecomment-731654132,https://api.github.com/repos/simonw/datasette/issues/1084,731654132,MDEyOklzc3VlQ29tbWVudDczMTY1NDEzMg==,9599,simonw,2020-11-21T23:45:59Z,2020-11-21T23:45:59Z,OWNER,https://datasette-graphql-demo.datasette.io/github/users now demonstrates the fix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731653083,https://api.github.com/repos/simonw/datasette/issues/1084,731653083,MDEyOklzc3VlQ29tbWVudDczMTY1MzA4Mw==,9599,simonw,2020-11-21T23:35:07Z,2020-11-21T23:35:07Z,OWNER,I got to use CSS `calc()` for this: https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/datasette/static/app.css#L367-L371,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731652991,https://api.github.com/repos/simonw/datasette/issues/1084,731652991,MDEyOklzc3VlQ29tbWVudDczMTY1Mjk5MQ==,9599,simonw,2020-11-21T23:34:22Z,2020-11-21T23:34:22Z,OWNER,"Fixed by positioning the menu relative to the `<div class=""page-header"">` element rather than the cog icon:

<img width=""966"" alt=""data"" src=""https://user-images.githubusercontent.com/9599/99889950-02eab580-2c0f-11eb-85dc-dd4bc840a92e.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1084#issuecomment-731644064,https://api.github.com/repos/simonw/datasette/issues/1084,731644064,MDEyOklzc3VlQ29tbWVudDczMTY0NDA2NA==,9599,simonw,2020-11-21T22:10:15Z,2020-11-21T22:10:15Z,OWNER,"Another example of this bug: https://datasette-graphql-demo.datasette.io/github/users

<img width=""350"" alt=""github__users__4_823_rows_and_Comparing_1_1___main_·_simonw_datasette-graphql"" src=""https://user-images.githubusercontent.com/9599/99888639-3889a180-2c03-11eb-83b9-41f2eb12287e.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",737394470,Table/database action menu cut off if too short ,
https://github.com/simonw/datasette/issues/1094#issuecomment-731260091,https://api.github.com/repos/simonw/datasette/issues/1094,731260091,MDEyOklzc3VlQ29tbWVudDczMTI2MDA5MQ==,4808085,bapowell,2020-11-20T16:11:29Z,2020-11-20T16:11:29Z,NONE,"I can confirm this issue, running version 0.51.1 under Windows.

Fixed by commenting out the following line near the top of datasette\utils\asgi.py :

`#from os import EX_CANTCREAT`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743011397,import EX_CANTCREAT means datasette fails to work on Windows,
https://github.com/simonw/datasette/issues/511#issuecomment-730893729,https://api.github.com/repos/simonw/datasette/issues/511,730893729,MDEyOklzc3VlQ29tbWVudDczMDg5MzcyOQ==,4060506,Carib0u,2020-11-20T06:35:13Z,2020-11-20T06:35:13Z,NONE,"Trying to run on Windows today, I get an error from the utils/asgi.py module.  

It's trying `from os import EX_CANTCREAT` which is Unix-only.  I commented this line out, and (so far) it's working. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",456578474,Get Datasette tests passing on Windows in GitHub Actions,
https://github.com/dogsheep/twitter-to-sqlite/issues/52#issuecomment-729484478,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52,729484478,MDEyOklzc3VlQ29tbWVudDcyOTQ4NDQ3OA==,4169772,fatihky,2020-11-18T07:12:45Z,2020-11-18T07:12:45Z,NONE,I'm so sorry that you already have `--since_id` option and that's enough for the case I've mentioned. Thank you for this excellent tool!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",745393298,Discussion: Adding support for fetching only fresh tweets,
https://github.com/simonw/datasette/issues/1091#issuecomment-729045320,https://api.github.com/repos/simonw/datasette/issues/1091,729045320,MDEyOklzc3VlQ29tbWVudDcyOTA0NTMyMA==,6739646,tballison,2020-11-17T16:31:00Z,2020-11-17T16:31:00Z,NONE,We're using mod_proxy.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-729018386,https://api.github.com/repos/simonw/datasette/issues/1091,729018386,MDEyOklzc3VlQ29tbWVudDcyOTAxODM4Ng==,6739646,tballison,2020-11-17T15:48:58Z,2020-11-17T15:48:58Z,NONE,"I don't think we are, but I'll check with Maruan.

I think this is the relevant part of our config?

```
  Alias ""/base/"" ""/usr/share/corpora/""
  <Directory ""/usr/share/corpora/"">
    Options +Indexes -Multiviews
    AllowOverride None
  </Directory>

  ProxyPreserveHost On

  ProxyPass /datasette http://0.0.0.0:8001
  ProxyPassReverse /datasette http://0.0.0.0:8001

</VirtualHost>
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-728262974,https://api.github.com/repos/simonw/datasette/issues/1091,728262974,MDEyOklzc3VlQ29tbWVudDcyODI2Mjk3NA==,9599,simonw,2020-11-16T19:05:08Z,2020-11-16T19:05:08Z,OWNER,I have a hunch that there may be some extra configuration in play here - could Apache itself be rewriting some of the links using [mod_proxy_html](https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html)?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11,727692413,MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw==,9599,simonw,2020-11-16T02:15:22Z,2020-11-16T02:15:22Z,MEMBER,"Thanks, I'll look into this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743400216,Error thrown: sqlite3.OperationalError: table users has no column named lastName,
https://github.com/simonw/datasette/issues/1098#issuecomment-727656208,https://api.github.com/repos/simonw/datasette/issues/1098,727656208,MDEyOklzc3VlQ29tbWVudDcyNzY1NjIwOA==,9599,simonw,2020-11-15T23:26:14Z,2020-11-15T23:26:14Z,OWNER,"Schema for that broken example:
```sql
CREATE TABLE generators_eia860 (
	id INTEGER NOT NULL, 
	plant_id_eia INTEGER, 
	generator_id TEXT, 
	report_date DATE, 
	operational_status_code TEXT, 
	operational_status TEXT, 
	ownership_code TEXT, 
	utility_id_eia INTEGER, 
	capacity_mw FLOAT, 
	summer_capacity_mw FLOAT, 
	winter_capacity_mw FLOAT, 
	energy_source_code_1 TEXT, 
	energy_source_code_2 TEXT, 
	energy_source_code_3 TEXT, 
	energy_source_code_4 TEXT, 
	energy_source_code_5 TEXT, 
	energy_source_code_6 TEXT, 
	fuel_type_code_pudl TEXT, 
	multiple_fuels BOOLEAN, 
	deliver_power_transgrid BOOLEAN, 
	syncronized_transmission_grid BOOLEAN, 
	turbines_num INTEGER, 
	planned_modifications BOOLEAN, 
	planned_net_summer_capacity_uprate_mw FLOAT, 
	planned_net_winter_capacity_uprate_mw FLOAT, 
	planned_uprate_date DATE, 
	planned_net_summer_capacity_derate_mw FLOAT, 
	planned_net_winter_capacity_derate_mw FLOAT, 
	planned_derate_date DATE, 
	planned_new_prime_mover_code TEXT, 
	planned_energy_source_code_1 TEXT, 
	planned_repower_date DATE, 
	other_planned_modifications BOOLEAN, 
	other_modifications_date DATE, 
	planned_retirement_date DATE, 
	carbon_capture BOOLEAN, 
	startup_source_code_1 TEXT, 
	startup_source_code_2 TEXT, 
	startup_source_code_3 TEXT, 
	startup_source_code_4 TEXT, 
	technology_description TEXT, 
	turbines_inverters_hydrokinetics TEXT, 
	time_cold_shutdown_full_load_code TEXT, 
	planned_new_capacity_mw FLOAT, 
	cofire_fuels BOOLEAN, 
	switch_oil_gas BOOLEAN, 
	nameplate_power_factor FLOAT, 
	minimum_load_mw FLOAT, 
	uprate_derate_during_year BOOLEAN, 
	uprate_derate_completed_date DATE, 
	current_planned_operating_date DATE, 
	summer_estimated_capability_mw FLOAT, 
	winter_estimated_capability_mw FLOAT, 
	retirement_date DATE, 
	PRIMARY KEY (id), 
	FOREIGN KEY(plant_id_eia, generator_id) REFERENCES generators_entity_eia (plant_id_eia, generator_id), 
	FOREIGN KEY(utility_id_eia) REFERENCES utilities_entity_eia (utility_id_eia), 
	CHECK (multiple_fuels IN (0, 1)), 
	CHECK (deliver_power_transgrid IN (0, 1)), 
	CHECK (syncronized_transmission_grid IN (0, 1)), 
	CHECK (planned_modifications IN (0, 1)), 
	CHECK (other_planned_modifications IN (0, 1)), 
	CHECK (carbon_capture IN (0, 1)), 
	CHECK (cofire_fuels IN (0, 1)), 
	CHECK (switch_oil_gas IN (0, 1)), 
	CHECK (uprate_derate_during_year IN (0, 1))
);
```
https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_entity_eia is:
```sql
CREATE TABLE generators_entity_eia (
	plant_id_eia INTEGER NOT NULL, 
	generator_id TEXT NOT NULL, 
	prime_mover_code TEXT, 
	duct_burners BOOLEAN, 
	operating_date DATE, 
	topping_bottoming_code TEXT, 
	solid_fuel_gasification BOOLEAN, 
	pulverized_coal_tech BOOLEAN, 
	fluidized_bed_tech BOOLEAN, 
	subcritical_tech BOOLEAN, 
	supercritical_tech BOOLEAN, 
	ultrasupercritical_tech BOOLEAN, 
	stoker_tech BOOLEAN, 
	other_combustion_tech BOOLEAN, 
	bypass_heat_recovery BOOLEAN, 
	rto_iso_lmp_node_id TEXT, 
	rto_iso_location_wholesale_reporting_id TEXT, 
	associated_combined_heat_power BOOLEAN, 
	original_planned_operating_date DATE, 
	operating_switch TEXT, 
	previously_canceled BOOLEAN, 
	PRIMARY KEY (plant_id_eia, generator_id), 
	FOREIGN KEY(plant_id_eia) REFERENCES plants_entity_eia (plant_id_eia), 
	CHECK (duct_burners IN (0, 1)), 
	CHECK (solid_fuel_gasification IN (0, 1)), 
	CHECK (pulverized_coal_tech IN (0, 1)), 
	CHECK (fluidized_bed_tech IN (0, 1)), 
	CHECK (subcritical_tech IN (0, 1)), 
	CHECK (supercritical_tech IN (0, 1)), 
	CHECK (ultrasupercritical_tech IN (0, 1)), 
	CHECK (stoker_tech IN (0, 1)), 
	CHECK (other_combustion_tech IN (0, 1)), 
	CHECK (bypass_heat_recovery IN (0, 1)), 
	CHECK (associated_combined_heat_power IN (0, 1)), 
	CHECK (previously_canceled IN (0, 1))
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/issues/1098#issuecomment-727655636,https://api.github.com/repos/simonw/datasette/issues/1098,727655636,MDEyOklzc3VlQ29tbWVudDcyNzY1NTYzNg==,9599,simonw,2020-11-15T23:22:27Z,2020-11-15T23:22:27Z,OWNER,"Need to replicate this in the fixtures, then fix it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743370900,Foreign key links break for compound foreign keys,
https://github.com/simonw/datasette/pull/1097#issuecomment-727655018,https://api.github.com/repos/simonw/datasette/issues/1097,727655018,MDEyOklzc3VlQ29tbWVudDcyNzY1NTAxOA==,22429695,codecov[bot],2020-11-15T23:18:18Z,2020-11-15T23:18:18Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=h1) Report
> Merging [#1097](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=desc) (e89211d) into [main](https://codecov.io/gh/simonw/datasette/commit/5eb8e9bf250b26e30b017d39a392c33973997656?el=desc) (5eb8e9b) will **not change** coverage.
> The diff coverage is `84.61%`.

[![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1097/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##             main    #1097   +/-   ##
=======================================
  Coverage   91.38%   91.38%           
=======================================
  Files          30       30           
  Lines        3785     3785           
=======================================
  Hits         3459     3459           
  Misses        326      326           
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `73.63% <0.00%> (ø)` | |
| [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <ø> (ø)` | |
| [datasette/publish/common.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY29tbW9uLnB5) | `94.73% <ø> (ø)` | |
| [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `81.60% <0.00%> (ø)` | |
| [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.16% <ø> (ø)` | |
| [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.12% <50.00%> (ø)` | |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.46% <66.66%> (ø)` | |
| [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <77.77%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.01% <86.20%> (ø)` | |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.92% <92.30%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/simonw/datasette/pull/1097/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=footer). Last update [5eb8e9b...e89211d](https://codecov.io/gh/simonw/datasette/pull/1097?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",743369188,Use f-strings,
https://github.com/simonw/datasette/issues/942#issuecomment-727626657,https://api.github.com/repos/simonw/datasette/issues/942,727626657,MDEyOklzc3VlQ29tbWVudDcyNzYyNjY1Nw==,9599,simonw,2020-11-15T19:54:44Z,2020-11-15T19:54:44Z,OWNER,This will also benefit from the metadata plugin hook: #860 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681334912,Support column descriptions in metadata.json,
https://github.com/simonw/datasette/issues/1091#issuecomment-727233553,https://api.github.com/repos/simonw/datasette/issues/1091,727233553,MDEyOklzc3VlQ29tbWVudDcyNzIzMzU1Mw==,9599,simonw,2020-11-14T16:46:52Z,2020-11-14T16:46:52Z,OWNER,@tballison could I see the section of your Apache config that configures the proxying to `/datasette/`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726801731,https://api.github.com/repos/simonw/datasette/issues/1091,726801731,MDEyOklzc3VlQ29tbWVudDcyNjgwMTczMQ==,6739646,tballison,2020-11-13T14:40:56Z,2020-11-13T14:40:56Z,NONE,"My headers aren't clickable/sortable with custom sql, but I think that's by design.

In the default view, https://corpora.tika.apache.org/datasette/file_profiles/file_profiles, ah, y, now I see that the headers should be sortable, but you're right the base_url is not applied.

base_url works with ""View and Edit SQL"" and with ""(advanced)""

As you point out, does not work with the export csv, json, other or with the ""Next page"" navigational button at the bottom.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/1091#issuecomment-726798745,https://api.github.com/repos/simonw/datasette/issues/1091,726798745,MDEyOklzc3VlQ29tbWVudDcyNjc5ODc0NQ==,6739646,tballison,2020-11-13T14:35:22Z,2020-11-13T14:35:22Z,NONE,"I'm starting this with docker like so:

`docker run --name datasette -d -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/file_profiles.db --config sql_time_limit_ms:120000 --config max_returned_rows:100000 --config base_url:/datasette/ --config cache_size_kb:50000`

I'm not doing any templating or anything else custom.

Apropos of nothing, I swapped out a simpler db, so this query should now work:

https://corpora.tika.apache.org/datasette/file_profiles?sql=select%0D%0A++*%0D%0Afrom%0D%0A++file_profiles+fp%0D%0Alimit%0D%0A++10","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742011049,.json and .csv exports fail to apply base_url,
https://github.com/simonw/datasette/issues/268#issuecomment-726419027,https://api.github.com/repos/simonw/datasette/issues/268,726419027,MDEyOklzc3VlQ29tbWVudDcyNjQxOTAyNw==,9599,simonw,2020-11-13T00:09:04Z,2020-11-13T00:09:04Z,OWNER,Part of the challenge here is that this is the first time the `TableView` will have had a complete rewrite of the SQL it is going to execute. That SQL is currently constructed here: https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/table.py#L628-L636,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search,