# issue_comments

html_url: https://github.com/simonw/datasette/issues/1269#issuecomment-804261610
issue_url: https://api.github.com/repos/simonw/datasette/issues/1269
node_id: MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA==
created_at: 2021-03-22T17:40:41Z
updated_at: 2021-03-22T17:40:41Z
author_association: OWNER
body:
  #1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty `count(*)` query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804263434
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA==
created_at: 2021-03-22T17:43:25Z
updated_at: 2021-03-22T17:43:25Z
author_association: OWNER
body:
  I figured out the cause of the hang in #1268 - it was caused by `select count(*) from SpatialIndex` interacting badly with the `set_progress_handler()` mechanism I was using to implement query time limits. #1271 has a replacement for that using `asyncio.wait_for()` and `conn.interrupt()` which should resolve the SpatiaLite issue too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1271#issuecomment-804265042
issue_url: https://api.github.com/repos/simonw/datasette/issues/1271
node_id: MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg==
created_at: 2021-03-22T17:45:45Z
updated_at: 2021-03-22T17:45:45Z
author_association: OWNER
body:
  I can remove this code too: 
  https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L190-L192

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1271#issuecomment-804299406
issue_url: https://api.github.com/repos/simonw/datasette/issues/1271
node_id: MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg==
created_at: 2021-03-22T18:36:14Z
updated_at: 2021-03-22T21:49:27Z
author_association: OWNER
body:
  This isn't actually working - the outer code attempts to send an `.interrupt()` call to the connection object via the `connections` thread-local, which doesn't work because it's a thread-local so the connection isn't visible to that code.

  Need to figure out how to communicate with that thread properly.

  Also a test that fails in this particular case would be a good idea!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804309510
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA==
created_at: 2021-03-22T18:50:50Z
updated_at: 2021-03-22T18:50:50Z
author_association: OWNER
body:
  Ideally I'd like to use the Debian stable `python:3.9.2-slim-buster` base image but install SpatiaLite from Debian unstable here: https://packages.debian.org/sid/libspatialite7

  This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804310353
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw==
created_at: 2021-03-22T18:52:12Z
updated_at: 2021-03-22T18:52:12Z
author_association: OWNER
body:
  This Dockerfile:
  ```dockerfile
  FROM python:3.9.2-slim-buster as build

  # Setup build dependencies
  RUN apt update \
      && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
      libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
      && apt clean

  RUN wget "https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz" && tar xzf sqlite-autoconf-3340100.tar.gz \
      && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
      CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \
      && make && make install

  RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz" && tar zxf freexl-1.0.6.tar.gz \
      && cd freexl-1.0.6 && ./configure && make && make install

  RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz" && tar zxf libspatialite-5.0.1.tar.gz \
      && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install

  RUN wget "http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install

  RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz" && tar zxf spatialite-tools-5.0.0.tar.gz \
      && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install

  # Add local code to the image instead of fetching from pypi.
  #COPY . /datasette
  #RUN pip install /datasette
  RUN pip install datasette

  FROM python:3.9.2-slim-buster

  # Copy python dependencies and spatialite libraries
  COPY --from=build /usr/local/lib/ /usr/local/lib/
  # Copy executables
  COPY --from=build /usr/local/bin /usr/local/bin
  # Copy spatial extensions
  COPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

  ENV LD_LIBRARY_PATH=/usr/local/lib

  EXPOSE 8001
  CMD ["datasette"]
  ```
  Produced a 448MB image.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804317545
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ==
created_at: 2021-03-22T19:03:22Z
updated_at: 2021-03-22T19:03:22Z
author_association: OWNER
body:
  This Dockerfile:
  ```dockerfile
  FROM python:3.9.2-slim-buster as build

  # software-properties-common provides add-apt-repository
  RUN apt-get update && \
      apt-get -y install software-properties-common && \
      add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
      apt-get update && \
      apt-get -t sid install -y libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/

  RUN pip install datasette

  EXPOSE 8001
  CMD ["datasette"]
  ```
  Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804338678
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA==
created_at: 2021-03-22T19:33:43Z
updated_at: 2021-03-22T19:33:43Z
author_association: OWNER
body:
  Replacing `rm -rf /var/lib/{apt,dpkg,cache,log}/` with 
  ```
      rm -rf /var/lib/apt && \
      rm -rf /var/lib/dpkg
  ```
  Got the size down to 305MB.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804347152
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg==
created_at: 2021-03-22T19:47:56Z
updated_at: 2021-03-22T19:48:03Z
author_association: OWNER
body:
  I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804363687
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw==
created_at: 2021-03-22T20:15:00Z
updated_at: 2021-03-22T20:15:00Z
author_association: OWNER
body:
  ```
  RUN pip install datasette && \
      find /usr/local/lib -name '__pycache__' | xargs rm -r
  ```
  That dropped it to 265MB.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804368372
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg==
created_at: 2021-03-22T20:22:43Z
updated_at: 2021-03-22T20:22:43Z
author_association: OWNER
body:
  ```dockerfile
  FROM python:3.9.2-slim-buster as build

  # software-properties-common provides add-apt-repository
  RUN apt-get update && \
      apt-get -y --no-install-recommends install software-properties-common && \
      add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
      apt-get update && \
      apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/apt && \
      rm -rf /var/lib/dpkg

  RUN pip install datasette && \
      find /usr/local/lib -name '__pycache__' | xargs rm -r && \
      rm -rf /root/.cache/pip

  EXPOSE 8001
  CMD ["datasette"]
  ```
  262 MB

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804372977
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw==
created_at: 2021-03-22T20:30:37Z
updated_at: 2021-03-22T20:30:37Z
author_association: OWNER
body:
  I tried copying just the `mod_spatialite.so` file into a second stage build but it failed. So I ran `bash` in a working image and used `ldd` to figure out what it was linked to:
  ```
  root@39683f91e588:/usr/lib/x86_64-linux-gnu# ldd mod_spatialite.so
  	linux-vdso.so.1 (0x00007ffd021f4000)
  	libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f5c75412000)
  	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5c753f0000)
  	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5c752ac000)
  	libminizip.so.1 => /usr/lib/x86_64-linux-gnu/libminizip.so.1 (0x00007f5c750a0000)
  	librttopo.so.1 => /usr/lib/x86_64-linux-gnu/librttopo.so.1 (0x00007f5c75028000)
  	libfreexl.so.1 => /usr/lib/x86_64-linux-gnu/libfreexl.so.1 (0x00007f5c7501c000)
  	libproj.so.19 => /usr/lib/x86_64-linux-gnu/libproj.so.19 (0x00007f5c74ca7000)
  	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c74a89000)
  	libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f5c74967000)
  	libgeos_c.so.1 => /usr/lib/x86_64-linux-gnu/libgeos_c.so.1 (0x00007f5c7492b000)
  	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c74766000)
  	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5c74760000)
  	libicuuc.so.67 => /usr/lib/x86_64-linux-gnu/libicuuc.so.67 (0x00007f5c74575000)
  	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5c7454d000)
  	/lib64/ld-linux-x86-64.so.2 (0x00007f5c75d49000)
  	libtiff.so.5 => /usr/lib/x86_64-linux-gnu/libtiff.so.5 (0x00007f5c744c7000)
  	libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f5c74439000)
  	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5c7426c000)
  	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5c74250000)
  	libgeos-3.9.0.so => /usr/lib/x86_64-linux-gnu/libgeos-3.9.0.so (0x00007f5c74040000)
  	libicudata.so.67 => /usr/lib/x86_64-linux-gnu/libicudata.so.67 (0x00007f5c72527000)
  	libwebp.so.6 => /usr/lib/x86_64-linux-gnu/libwebp.so.6 (0x00007f5c724bc000)
  	libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f5c7241c000)
  	libjbig.so.0 => /usr/lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f5c7220e000)
  	libjpeg.so.62 => /usr/lib/x86_64-linux-gnu/libjpeg.so.62 (0x00007f5c72188000)
  	libdeflate.so.0 => /usr/lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f5c7216c000)
  	libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f5c72144000)
  	libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f5c72125000)
  	librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f5c71f08000)
  	libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f5c71eda000)
  	libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f5c71ec5000)
  	libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f5c71e8d000)
  	libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f5c71ce0000)
  	libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f5c71c93000)
  	libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f5c71bb3000)
  	libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f5c71b7f000)
  	libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f5c71b77000)
  	libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f5c71b23000)
  	liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f5c71b12000)
  	libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f5c7198e000)
  	libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f5c71955000)
  	libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f5c718d0000)
  	libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f5c717b2000)
  	libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f5c71683000)
  	libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f5c71470000)
  	libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f5c71461000)
  	libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f5c71458000)
  	libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5c7143e000)
  	libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f5c71421000)
  	libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f5c713fe000)
  	libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f5c713f4000)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804379644
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA==
created_at: 2021-03-22T20:41:23Z
updated_at: 2021-03-22T20:41:23Z
author_association: OWNER
body:
  I tried adding `apt-get remove -y software-properties-common &&` to remove `software-properties-common` but it made no difference to the image size.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804380181
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ==
created_at: 2021-03-22T20:42:16Z
updated_at: 2021-03-22T20:42:16Z
author_association: OWNER
body:
  Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804384196
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng==
created_at: 2021-03-22T20:48:46Z
updated_at: 2021-03-22T20:48:46Z
author_association: OWNER
body:
  I think part of the reason it's smaller is that I ran `pip install datasette` instead of using `COPY . /datasette` followed by `pip install /datasette`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-804404544
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA==
created_at: 2021-03-22T21:22:56Z
updated_at: 2021-03-22T21:24:24Z
author_association: OWNER
body:
  Final version of Dockerfile which installs the specified version from GitHub:

      docker build . -t datasette-spatialite --build-arg VERSION=0.55

  ```dockerfile
  FROM python:3.9.2-slim-buster as build

  # Version of Datasette to install, e.g. 0.55
  #   docker build . -t datasette --build-arg VERSION=0.55
  ARG VERSION

  # software-properties-common provides add-apt-repository
  # which we need in order to install a more recent release
  # of libsqlite3-mod-spatialite from the sid distribution
  RUN apt-get update && \
      apt-get -y --no-install-recommends install software-properties-common && \
      add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
      apt-get update && \
      apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
      apt-get remove -y software-properties-common && \
      apt clean && \
      rm -rf /var/lib/apt && \
      rm -rf /var/lib/dpkg

  RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \
      find /usr/local/lib -name '__pycache__' | xargs rm -r && \
      rm -rf /root/.cache/pip

  EXPOSE 8001
  CMD ["datasette"]
  ```
  Run against 0.55 this produces an image of 262MB

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/88#issuecomment-804471733
issue_url: https://api.github.com/repos/simonw/datasette/issues/88
node_id: MDEyOklzc3VlQ29tbWVudDgwNDQ3MTczMw==
created_at: 2021-03-22T23:46:36Z
updated_at: 2021-03-22T23:46:36Z
author_association: CONTRIBUTOR
body:
  Google Map API limits seem to prevent https://nhs-england-map.netlify.com from being a working demo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/163#issuecomment-804539729
issue_url: https://api.github.com/repos/simonw/datasette/issues/163
node_id: MDEyOklzc3VlQ29tbWVudDgwNDUzOTcyOQ==
created_at: 2021-03-23T02:41:14Z
updated_at: 2021-03-23T02:41:14Z
author_association: CONTRIBUTOR
body:
  I'm visiting old issues for context while learning datasette. Let me know if okay to make the occasional comment like this one.
  querystring argument now located at:
  https://docs.datasette.io/en/latest/settings.html#sql-time-limit-ms

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1159#issuecomment-804639427
issue_url: https://api.github.com/repos/simonw/datasette/issues/1159
node_id: MDEyOklzc3VlQ29tbWVudDgwNDYzOTQyNw==
created_at: 2021-03-23T05:56:02Z
updated_at: 2021-03-23T05:56:02Z
author_association: CONTRIBUTOR
body:
  With just three facets, I like it, but it does take more horizontal space. Would be nice to have a switch somewhere, enabling either original compact option or this proposed more-readable option. Also some control over word wrap (width setting) and facet spacing. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1159#issuecomment-804698315
issue_url: https://api.github.com/repos/simonw/datasette/issues/1159
node_id: MDEyOklzc3VlQ29tbWVudDgwNDY5ODMxNQ==
created_at: 2021-03-23T07:58:28Z
updated_at: 2021-03-23T07:58:38Z
author_association: NONE
body:
  @mroswell Did you try it with more columns ? The display is flexible and  columns get closer  as new ones are added.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-805033155
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwNTAzMzE1NQ==
created_at: 2021-03-23T16:12:13Z
updated_at: 2021-03-23T16:12:13Z
author_association: OWNER
body:
  Don't forget to update this bit of the docs: https://docs.datasette.io/en/0.55/spatialite.html#building-spatialite-from-source

  > The packaged versions of SpatiaLite usually provide SpatiaLite 4.3.0a. For an example of how to build the most recent unstable version, 4.4.0-RC0 (which includes the powerful [VirtualKNN module](https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN)), take a look at the [Datasette Dockerfile](https://github.com/simonw/datasette/blob/master/Dockerfile).

  See also #1273

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-805041522
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: MDEyOklzc3VlQ29tbWVudDgwNTA0MTUyMg==
created_at: 2021-03-23T16:22:46Z
updated_at: 2021-03-23T16:22:46Z
author_association: OWNER
body:
  That's a good idea. I could do that with JavaScript - loading YAML and converting it to JSON in JavaScript shouldn't be hard, and it's better than JSON-to-YAML because there's only one correct JSON representation of a YAML file whereas you can represent a JSON document in YAML in a bunch of different ways.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-805042880
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: MDEyOklzc3VlQ29tbWVudDgwNTA0Mjg4MA==
created_at: 2021-03-23T16:24:32Z
updated_at: 2021-03-23T16:24:32Z
author_association: OWNER
body:
  ... actually I think I would do that conversion in Python. The client-side YAML parsers all look a little bit heavy to me in terms of additional page weight.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-805047117
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: MDEyOklzc3VlQ29tbWVudDgwNTA0NzExNw==
created_at: 2021-03-23T16:30:15Z
updated_at: 2021-03-23T16:46:06Z
author_association: OWNER
body:
  https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js is only 12.5KB zipped, 38KB total - so that's not a bad option.

  https://github.com/nodeca/js-yaml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-805050163
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: MDEyOklzc3VlQ29tbWVudDgwNTA1MDE2Mw==
created_at: 2021-03-23T16:34:35Z
updated_at: 2021-03-23T16:35:32Z
author_association: OWNER
body:
  https://docs.datasette.io/en/stable/metadata.html has this example:
  ```yaml
  title: Demonstrating Metadata from YAML
  description_html: |-
    <p>This description includes a long HTML string</p>
    <ul>
      <li>YAML is better for embedding HTML strings than JSON!</li>
    </ul>
  license: ODbL
  license_url: https://opendatacommons.org/licenses/odbl/
  databases:
    fixtures:
      tables:
        no_primary_key:
          hidden: true
      queries:
        neighborhood_search:
          sql: |-
            select neighborhood, facet_cities.name, state
            from facetable join facet_cities on facetable.city_id = facet_cities.id
            where neighborhood like '%' || :text || '%' order by neighborhood;
          title: Search neighborhoods
          description_html: |-
            <p>This demonstrates <em>basic</em> LIKE search
  ```

  I ran this in the browser dev tools:
  ```javascript
  var s = document.createElement('script')
  s.src = 'https://cdnjs.cloudflare.com/ajax/libs/js-yaml/4.0.0/js-yaml.min.js'
  document.head.appendChild(s)
  var yamlExample = document.querySelector('.highlight-yaml').textContent);
  console.log(JSON.stringify(window.jsyaml.load(yamlExample), null, 4))
  ```
  And got:
  ```json
  {
      "title": "Demonstrating Metadata from YAML",
      "description_html": "<p>This description includes a long HTML string</p>\n<ul>\n  <li>YAML is better for embedding HTML strings than JSON!</li>\n</ul>",
      "license": "ODbL",
      "license_url": "https://opendatacommons.org/licenses/odbl/",
      "databases": {
          "fixtures": {
              "tables": {
                  "no_primary_key": {
                      "hidden": true
                  }
              },
              "queries": {
                  "neighborhood_search": {
                      "sql": "select neighborhood, facet_cities.name, state\nfrom facetable join facet_cities on facetable.city_id = facet_cities.id\nwhere neighborhood like '%' || :text || '%' order by neighborhood;",
                      "title": "Search neighborhoods",
                      "description_html": "<p>This demonstrates <em>basic</em> LIKE search"
                  }
              }
          }
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-805055291
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: MDEyOklzc3VlQ29tbWVudDgwNTA1NTI5MQ==
created_at: 2021-03-23T16:41:31Z
updated_at: 2021-03-23T16:41:31Z
author_association: OWNER
body:
  One downside of doing this conversion in JavaScript: it's much harder to get the same JSON syntax highlighting as that provided by Sphinx:

  <img width="655" alt="Metadata_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/112183909-ef9b2700-8bbb-11eb-882b-aeb387c29e61.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-805056806
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: MDEyOklzc3VlQ29tbWVudDgwNTA1NjgwNg==
created_at: 2021-03-23T16:43:38Z
updated_at: 2021-03-23T16:43:38Z
author_association: OWNER
body:
  I used this code to get that:
  ```javascript
  var jsonVersion = JSON.stringify(window.jsyaml.load(document.querySelector('.highlight-yaml').textContent), null, 4);
  div.querySelector('.highlight pre').innerText = jsonVersion;
  div.querySelector('.highlight pre').style.whiteSpace = 'pre-wrap'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1270#issuecomment-805058241
issue_url: https://api.github.com/repos/simonw/datasette/issues/1270
node_id: MDEyOklzc3VlQ29tbWVudDgwNTA1ODI0MQ==
created_at: 2021-03-23T16:45:39Z
updated_at: 2021-03-23T16:45:39Z
author_association: OWNER
body:
  I managed to build SpatiaLite such that this isn't necessary any more. I'm still interested in pursuing this further though - it feels like it could be a more robust way of implementing timeouts, but I need to prove to myself that it's better (maybe better performance, or handles more edge-cases?). Not sure how to prove that yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1153#issuecomment-805109341
issue_url: https://api.github.com/repos/simonw/datasette/issues/1153
node_id: MDEyOklzc3VlQ29tbWVudDgwNTEwOTM0MQ==
created_at: 2021-03-23T17:55:48Z
updated_at: 2021-03-23T18:41:57Z
author_association: OWNER
body:
  Beginnings of a UI element for switching between them:
  ```html
  <div style="border: 1px solid rgb(225, 228, 229);
    background-color: rgb(238, 255, 204);
    padding: 0.3em;
    position: relative;
    top: 3px;
    font-family: courier;">
  <a href="#" style="display: inline-block; padding-left: 0px; padding-right: 2em;">JSON</a>
  <a href="#" style="display: inline-block;">YAML</a>
  </div>
  ```
  <img width="646" alt="Metadata_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/112194637-51f92500-8bc6-11eb-9662-3faa7ef37538.png">

  That `<pre>` has a padding of 12px, so using 12px padding on the tab links should get them to line up better.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1274#issuecomment-805214307
issue_url: https://api.github.com/repos/simonw/datasette/issues/1274
node_id: MDEyOklzc3VlQ29tbWVudDgwNTIxNDMwNw==
created_at: 2021-03-23T20:12:29Z
updated_at: 2021-03-23T20:12:29Z
author_association: CONTRIBUTOR
body:
  One issue I could see with adding first class support for metadata in hjson format is that this would require adding an additional dependency to handle this, for a feature that would be unused by many users. I wonder if this could fit in as a plugin instead; if a hook existed for loading metadata (maybe as part of https://github.com/simonw/datasette/issues/860) the metadata could then come from any source, as specified by plugins, e.g. hjson, toml, XML, a database table etc.

  Until/unless this exists, a few ideas for how you could add comments:
  - Using YAML as you suggest.
  - A common pattern is adding a `"comment"` key for comments to any object in JSON - I don't think including an unnecessary key like this would break anything in Datasette, but not certain.
  - You could use another tool as a preprocessor for your JSON metadata - e.g. hjson or Jsonnet. You'd write the metadata in that format, and then convert that into JSON to actually use as your final metadata.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1274#issuecomment-805216038
issue_url: https://api.github.com/repos/simonw/datasette/issues/1274
node_id: MDEyOklzc3VlQ29tbWVudDgwNTIxNjAzOA==
created_at: 2021-03-23T20:14:53Z
updated_at: 2021-03-23T20:14:53Z
author_association: OWNER
body:
  Yes this is one of the main reasons I'm planning to switch to encouraging YAML be default instead of JSON (while still supporting JSON) - YAML supports comments and multi-line strings.

  See #1153 for YAML by default in the documentation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/741#issuecomment-806010960
issue_url: https://api.github.com/repos/simonw/datasette/issues/741
node_id: MDEyOklzc3VlQ29tbWVudDgwNjAxMDk2MA==
created_at: 2021-03-24T17:19:42Z
updated_at: 2021-03-24T17:19:42Z
author_association: NONE
body:
  Ah, okay so `--extra-options` applies to both `datasette publish` and `datasette package`?  There wren't any examples of it being used with `publish` in the docs, so this tripped me up for a bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/741#issuecomment-806166575
issue_url: https://api.github.com/repos/simonw/datasette/issues/741
node_id: MDEyOklzc3VlQ29tbWVudDgwNjE2NjU3NQ==
created_at: 2021-03-24T20:30:33Z
updated_at: 2021-03-24T20:30:33Z
author_association: OWNER
body:
  `datasette package` is a mostly unmaintained feature at this point - it has a bit of test coverage but I've not made any improvements to it in a few years, and I don't use it for my own projects.

  I'll make this change to `package` at the same time as I land it for `publish` though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1258#issuecomment-807459633
issue_url: https://api.github.com/repos/simonw/datasette/issues/1258
node_id: MDEyOklzc3VlQ29tbWVudDgwNzQ1OTYzMw==
created_at: 2021-03-25T20:48:33Z
updated_at: 2021-03-25T20:49:34Z
author_association: NONE
body:
  What about allowing default parameters when defining the query in metadata.yml?  Something like:
  ```
  databases:
    fec:
      queries:
        search_by_name:
          params:
              - q
          default-param-values:
               q: "text to search"
          sql: |-
            SELECT...
  ```

  For now, I'm using a custom database-<file>.html file that hardcodes a default param in the link, but I'd rather not customize the template just for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807642041
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: MDEyOklzc3VlQ29tbWVudDgwNzY0MjA0MQ==
created_at: 2021-03-25T22:39:22Z
updated_at: 2021-03-25T22:39:22Z
author_association: OWNER
body:
  Here's the full current implementation of that tool: https://github.com/simonw/sqlite-transform/blob/0.5/sqlite_transform/cli.py

  My current plan is to make this functionality available as the following:

      sqlite-utils convert jsonsplit mydb.db mytable mycolumn
      sqlite-utils convert parsedatetime mydb.db mytable mycolumn
      sqlite-utils convert parsedate mydb.db mytable mycolumn
      sqlite-utils convert lambda mydb.db mytable mycolumn --code='str(value).upper()'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/251#issuecomment-807647791
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/251
node_id: MDEyOklzc3VlQ29tbWVudDgwNzY0Nzc5MQ==
created_at: 2021-03-25T22:42:48Z
updated_at: 2021-03-25T22:44:31Z
author_association: OWNER
body:
  Idea: enhance `lambda` to allow it to return a dictionary of values, which will then be used to populate new columns. Use a `--multicolumn` option to indicate this:

      sqlite-utils convert lambda mydb.db mytable mycolumn \
        --code '{"first_name": value.split()[0], "last_name": value.split()[1]}' \
        --multicolumn --drop

  The `--drop` means "drop the `mycolumn` column after making this change".

  Maybe `--multi` is a better name than `--multicolumn` here, since either way it's going to need additional explanation somewhere.

  Would this overlap with #239 at all?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/252#issuecomment-808302971
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/252
node_id: MDEyOklzc3VlQ29tbWVudDgwODMwMjk3MQ==
created_at: 2021-03-26T15:21:38Z
updated_at: 2021-03-26T15:21:38Z
author_association: OWNER
body:
  Already got that! It's the `--nl` option - works for both importing and exporting data: https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-json-data

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1276#issuecomment-808642405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1276
node_id: MDEyOklzc3VlQ29tbWVudDgwODY0MjQwNQ==
created_at: 2021-03-27T03:53:18Z
updated_at: 2021-03-27T03:53:18Z
author_association: OWNER
body:
  That's really odd. What version of SQLite are you using on the server? You can tell by visiting `https://your-site/-/versions`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1272#issuecomment-808647937
issue_url: https://api.github.com/repos/simonw/datasette/issues/1272
node_id: MDEyOklzc3VlQ29tbWVudDgwODY0NzkzNw==
created_at: 2021-03-27T04:23:19Z
updated_at: 2021-03-27T04:23:36Z
author_association: OWNER
body:
  Part of the challenge here is only running if a Docker daemon is available. I think this pattern works, in `tests/test_dockerfile.py`:
  ```python
  import httpx
  import pathlib
  import pytest
  import subprocess

  root = pathlib.Path(__file__).parent.parent

  def docker_is_available():
      try:
          client = httpx.Client(
              transport=httpx.HTTPTransport(uds="/var/run/docker.sock")
          )
          client.get("http://docker/info")
          return True
      except httpx.ConnectError:
          return False


  @pytest.fixture
  def build_container():
      assert (root / "Dockerfile").exists()
      subprocess.check_call([
          "docker", "build", str(root), "-t", "datasette-dockerfile-test"
      ])


  @pytest.mark.skipif(not docker_is_available(),
      reason="Docker is not available"
  )
  def test_dockerfile(build_container):
      output = subprocess.check_output([
          "docker", "run", "datasette-dockerfile-test", "datasette", "--get", "/_memory?sql=select+1&shape=_array"
      ])
      assert False, "Implement better assertion here"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1272#issuecomment-808648974
issue_url: https://api.github.com/repos/simonw/datasette/issues/1272
node_id: MDEyOklzc3VlQ29tbWVudDgwODY0ODk3NA==
created_at: 2021-03-27T04:29:42Z
updated_at: 2021-03-27T04:29:42Z
author_association: OWNER
body:
  I'm skipping this for the moment because the new Dockerfile shape introduced in https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

  Will need to change that if I want to do this unit tests thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-808649322
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwODY0OTMyMg==
created_at: 2021-03-27T04:31:28Z
updated_at: 2021-03-27T04:31:28Z
author_association: OWNER
body:
  One last test of that Dockerfile:
  ```
  (datasette) datasette % docker build -f Dockerfile -t datasetteproject/datasette:0.55a --build-arg VERSION=0.55 .
  (datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' | jq
  {
    "python": {
      "version": "3.9.2",
      "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
    },
    "datasette": {
      "version": "0.55"
    },
    "asgi": "3.0",
    "uvicorn": "0.13.4",
    "sqlite": {
      "version": "3.27.2",
      "fts_versions": [
        "FTS5",
        "FTS4",
        "FTS3"
      ],
      "extensions": {
        "json1": null
      },
      "compile_options": [
        "COMPILER=gcc-8.3.0",
        "ENABLE_COLUMN_METADATA",
        "ENABLE_DBSTAT_VTAB",
        "ENABLE_FTS3",
        "ENABLE_FTS3_PARENTHESIS",
        "ENABLE_FTS3_TOKENIZER",
        "ENABLE_FTS4",
        "ENABLE_FTS5",
        "ENABLE_JSON1",
        "ENABLE_LOAD_EXTENSION",
        "ENABLE_PREUPDATE_HOOK",
        "ENABLE_RTREE",
        "ENABLE_SESSION",
        "ENABLE_STMTVTAB",
        "ENABLE_UNLOCK_NOTIFY",
        "ENABLE_UPDATE_DELETE_LIMIT",
        "HAVE_ISNAN",
        "LIKE_DOESNT_MATCH_BLOBS",
        "MAX_SCHEMA_RETRY=25",
        "MAX_VARIABLE_NUMBER=250000",
        "OMIT_LOOKASIDE",
        "SECURE_DELETE",
        "SOUNDEX",
        "TEMP_STORE=1",
        "THREADSAFE=1",
        "USE_URI"
      ]
    }
  }
  (datasette) datasette % docker run datasetteproject/datasette:0.55a datasette --get '/-/versions.json' --load-extension=spatialite | jq
  {
    "python": {
      "version": "3.9.2",
      "full": "3.9.2 (default, Feb 19 2021, 17:23:45) \n[GCC 8.3.0]"
    },
    "datasette": {
      "version": "0.55"
    },
    "asgi": "3.0",
    "uvicorn": "0.13.4",
    "sqlite": {
      "version": "3.27.2",
      "fts_versions": [
        "FTS5",
        "FTS4",
        "FTS3"
      ],
      "extensions": {
        "json1": null,
        "spatialite": "5.0.1"
      },
      "compile_options": [
        "COMPILER=gcc-8.3.0",
        "ENABLE_COLUMN_METADATA",
        "ENABLE_DBSTAT_VTAB",
        "ENABLE_FTS3",
        "ENABLE_FTS3_PARENTHESIS",
        "ENABLE_FTS3_TOKENIZER",
        "ENABLE_FTS4",
        "ENABLE_FTS5",
        "ENABLE_JSON1",
        "ENABLE_LOAD_EXTENSION",
        "ENABLE_PREUPDATE_HOOK",
        "ENABLE_RTREE",
        "ENABLE_SESSION",
        "ENABLE_STMTVTAB",
        "ENABLE_UNLOCK_NOTIFY",
        "ENABLE_UPDATE_DELETE_LIMIT",
        "HAVE_ISNAN",
        "LIKE_DOESNT_MATCH_BLOBS",
        "MAX_SCHEMA_RETRY=25",
        "MAX_VARIABLE_NUMBER=250000",
        "OMIT_LOOKASIDE",
        "SECURE_DELETE",
        "SOUNDEX",
        "TEMP_STORE=1",
        "THREADSAFE=1",
        "USE_URI"
      ]
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-808649480
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwODY0OTQ4MA==
created_at: 2021-03-27T04:32:10Z
updated_at: 2021-03-27T04:32:10Z
author_association: OWNER
body:
  I'll close this issue after I ship Datasette 0.56 and confirm that the Dockerfile was correctly built and published to Docker Hub.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1258#issuecomment-808651088
issue_url: https://api.github.com/repos/simonw/datasette/issues/1258
node_id: MDEyOklzc3VlQ29tbWVudDgwODY1MTA4OA==
created_at: 2021-03-27T04:41:52Z
updated_at: 2021-03-27T04:42:14Z
author_association: OWNER
body:
  Right now they look like this:
  ```yaml
  databases:
    fixtures:
      queries:
        neighborhood_search:
          params:
          - text
  ```
  In addition to being able to specify defaults, I'd also like to add other things in the future - most significantly the ability to specify a different input widget (e.g. textarea v.s. single-line input)

  So maybe this looks like:
  ```yaml
  params:
  - name: text
    default: ""
  - name: age
    widget: number
  ```

reactions:
  {"total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1278#issuecomment-808652008
issue_url: https://api.github.com/repos/simonw/datasette/issues/1278
node_id: MDEyOklzc3VlQ29tbWVudDgwODY1MjAwOA==
created_at: 2021-03-27T04:47:17Z
updated_at: 2021-03-27T04:47:17Z
author_association: OWNER
body:
  https://github.com/simonw/timezones-api is that project, it's pretty old now. I'll try to get it running on Cloud Run.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1273#issuecomment-808756921
issue_url: https://api.github.com/repos/simonw/datasette/issues/1273
node_id: MDEyOklzc3VlQ29tbWVudDgwODc1NjkyMQ==
created_at: 2021-03-27T16:19:45Z
updated_at: 2021-03-27T16:26:28Z
author_association: OWNER
body:
  I have a better recipe for using spatial indexes now on https://simonwillison.net/2021/Jan/24/drawing-shapes-spatialite/
  ```sql
  select
    AsGeoJSON(geometry), *
  from
    CPAD_2020a_SuperUnits
  where
    PARK_NAME like '%mini%' and
    Intersects(GeomFromGeoJSON(:freedraw), geometry) = 1
    and CPAD_2020a_SuperUnits.rowid in (
      select
        rowid
      from
        SpatialIndex
      where
        f_table_name = 'CPAD_2020a_SuperUnits'
        and search_frame = GeomFromGeoJSON(:freedraw)
    )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1090#issuecomment-808757155
issue_url: https://api.github.com/repos/simonw/datasette/issues/1090
node_id: MDEyOklzc3VlQ29tbWVudDgwODc1NzE1NQ==
created_at: 2021-03-27T16:21:43Z
updated_at: 2021-03-27T16:21:43Z
author_association: OWNER
body:
  Idea for these: imitate https://django-sql-dashboard.readthedocs.io/en/latest/widgets.html#custom-widgets and drive them with templates.

  So a custom widget type of `textarea` would look for a template called `widgets/textarea.html` - which means users could define brand new custom widgets just by creating their own template files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1273#issuecomment-808757721
issue_url: https://api.github.com/repos/simonw/datasette/issues/1273
node_id: MDEyOklzc3VlQ29tbWVudDgwODc1NzcyMQ==
created_at: 2021-03-27T16:25:48Z
updated_at: 2021-03-27T16:25:48Z
author_association: OWNER
body:
  > This will give you back an additional column of GeoJSON. You can copy and paste GeoJSON from this column into the debugging tool at geojson.io to visualize it on a map.

  That should promote `datasette-leaflet-geojson` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1279#issuecomment-808762613
issue_url: https://api.github.com/repos/simonw/datasette/issues/1279
node_id: MDEyOklzc3VlQ29tbWVudDgwODc2MjYxMw==
created_at: 2021-03-27T17:03:37Z
updated_at: 2021-03-27T17:03:37Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1279?src=pr&el=h1) Report
  > Merging [#1279](https://codecov.io/gh/simonw/datasette/pull/1279?src=pr&el=desc) (14d8977) into [main](https://codecov.io/gh/simonw/datasette/commit/3fcfc8513465339ac5f055296cbb67f5262af02b?el=desc) (3fcfc85) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1279/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1279?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1279   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1279?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1279?src=pr&el=footer). Last update [3fcfc85...14d8977](https://codecov.io/gh/simonw/datasette/pull/1279?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1276#issuecomment-808978808
issue_url: https://api.github.com/repos/simonw/datasette/issues/1276
node_id: MDEyOklzc3VlQ29tbWVudDgwODk3ODgwOA==
created_at: 2021-03-28T23:32:58Z
updated_at: 2021-03-28T23:33:58Z
author_association: OWNER
body:
  I just managed to replicate this bug on Glitch: https://nicar-2020.glitch.me/data

  > Invalid SQL
  > no such table: pragma_database_list

  https://nicar-2020.glitch.me/-/versions says:

  ```json
  {
      "python": {
          "version": "3.7.10",
          "full": "3.7.10 (default, Feb 20 2021, 21:21:24) \n[GCC 5.4.0 20160609]"
      },
      "datasette": {
          "version": "0.55"
      },
      "asgi": "3.0",
      "uvicorn": "0.13.4",
      "sqlite": {
          "version": "3.11.0",
          "fts_versions": [
              "FTS4",
              "FTS3"
          ],
          "extensions": {
              "json1": null
          },
          "compile_options": [
              "ENABLE_COLUMN_METADATA",
              "ENABLE_DBSTAT_VTAB",
              "ENABLE_FTS3",
              "ENABLE_FTS3_PARENTHESIS",
              "ENABLE_JSON1",
              "ENABLE_LOAD_EXTENSION",
              "ENABLE_RTREE",
              "ENABLE_UNLOCK_NOTIFY",
              "ENABLE_UPDATE_DELETE_LIMIT",
              "HAVE_ISNAN",
              "LIKE_DOESNT_MATCH_BLOBS",
              "MAX_SCHEMA_RETRY=25",
              "OMIT_LOOKASIDE",
              "SECURE_DELETE",
              "SOUNDEX",
              "SYSTEM_MALLOC",
              "TEMP_STORE=1",
              "THREADSAFE=1"
          ]
      }
  }
  ```
  That's [SQLite 3.11.0 from 2016-02-15](https://www.sqlite.org/releaselog/3_11_0.html) with no FTS5.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1276#issuecomment-808979049
issue_url: https://api.github.com/repos/simonw/datasette/issues/1276
node_id: MDEyOklzc3VlQ29tbWVudDgwODk3OTA0OQ==
created_at: 2021-03-28T23:34:38Z
updated_at: 2021-03-28T23:34:38Z
author_association: OWNER
body:
  The Glitch server logs showed:

  > `ERROR: conn=<sqlite3.Connection object at 0x7f88ebadd8f0>, sql = 'select seq, name, file from pragma_database_list() where seq > 0', params = None: no such table: pragma_database_list`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1276#issuecomment-808979218
issue_url: https://api.github.com/repos/simonw/datasette/issues/1276
node_id: MDEyOklzc3VlQ29tbWVudDgwODk3OTIxOA==
created_at: 2021-03-28T23:35:53Z
updated_at: 2021-03-28T23:36:26Z
author_association: OWNER
body:
  Here's where I run that: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L249-L253

  That's from when I added the `--crossdb` option in #1232: https://github.com/simonw/datasette/commit/6f41c8a2bef309a66588b2875c3e24d26adb4850#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9R249

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1276#issuecomment-808979608
issue_url: https://api.github.com/repos/simonw/datasette/issues/1276
node_id: MDEyOklzc3VlQ29tbWVudDgwODk3OTYwOA==
created_at: 2021-03-28T23:38:34Z
updated_at: 2021-03-28T23:38:34Z
author_association: OWNER
body:
  Aha! https://www.sqlite.org/pragma.html says:

  >  The table-valued functions for PRAGMA feature was added in SQLite version 3.16.0 (2017-01-02). Prior versions of SQLite cannot use this feature. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1276#issuecomment-808981968
issue_url: https://api.github.com/repos/simonw/datasette/issues/1276
node_id: MDEyOklzc3VlQ29tbWVudDgwODk4MTk2OA==
created_at: 2021-03-28T23:52:31Z
updated_at: 2021-03-28T23:52:31Z
author_association: OWNER
body:
  Testing this on Glitch by adding `https://github.com/simonw/datasette/archive/48d5e0e6ac8975cfd869d4e8c69c64ca0c65e29e.zip` as a dependency. That fixed it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1280#issuecomment-808983160
issue_url: https://api.github.com/repos/simonw/datasette/issues/1280
node_id: MDEyOklzc3VlQ29tbWVudDgwODk4MzE2MA==
created_at: 2021-03-28T23:59:28Z
updated_at: 2021-03-29T00:10:05Z
author_association: OWNER
body:
  Might be easier to do this using https://github.com/coleifer/pysqlite3 rather than try to replace the system `sqlite3` on the Ubuntu GitHub Actions instances.

  These instructions should help: https://github.com/coleifer/pysqlite3#building-a-statically-linked-library

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1260#issuecomment-808988697
issue_url: https://api.github.com/repos/simonw/datasette/issues/1260
node_id: MDEyOklzc3VlQ29tbWVudDgwODk4ODY5Nw==
created_at: 2021-03-29T00:22:21Z
updated_at: 2021-03-29T00:22:21Z
author_association: OWNER
body:
  This is interesting!

  I've decided to apply a subset of these - the `if` and `elif` blocks are a deliberate style choice from me, because I find code clearer when it has if/else as opposed to relying on early termination. Likewise the iteration against `.keys()` on dictionaries.

  I like the other fixes though, I'm about to land them in a separate commit that credits you.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-808998719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwODk5ODcxOQ==
created_at: 2021-03-29T00:57:13Z
updated_at: 2021-03-29T00:57:13Z
author_association: OWNER
body:
  I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true

  It pushed a new `latest` tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated

      docker pull datasetteproject/datasette:latest

  And then:

      docker run datasetteproject/datasette:latest datasette \
        --load-extension=spatialite \
        --get /-/versions.json | jq .sqlite

  Outputs:
  ```json
  {
    "version": "3.27.2",
    "fts_versions": [
      "FTS5",
      "FTS4",
      "FTS3"
    ],
    "extensions": {
      "json1": null,
      "spatialite": "5.0.1"
    },
    "compile_options": [
      "COMPILER=gcc-8.3.0",
      "ENABLE_COLUMN_METADATA",
      "ENABLE_DBSTAT_VTAB",
      "ENABLE_FTS3",
      "ENABLE_FTS3_PARENTHESIS",
      "ENABLE_FTS3_TOKENIZER",
      "ENABLE_FTS4",
      "ENABLE_FTS5",
      "ENABLE_JSON1",
      "ENABLE_LOAD_EXTENSION",
      "ENABLE_PREUPDATE_HOOK",
      "ENABLE_RTREE",
      "ENABLE_SESSION",
      "ENABLE_STMTVTAB",
      "ENABLE_UNLOCK_NOTIFY",
      "ENABLE_UPDATE_DELETE_LIMIT",
      "HAVE_ISNAN",
      "LIKE_DOESNT_MATCH_BLOBS",
      "MAX_SCHEMA_RETRY=25",
      "MAX_VARIABLE_NUMBER=250000",
      "OMIT_LOOKASIDE",
      "SECURE_DELETE",
      "SOUNDEX",
      "TEMP_STORE=1",
      "THREADSAFE=1",
      "USE_URI"
    ]
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1281#issuecomment-808999525
issue_url: https://api.github.com/repos/simonw/datasette/issues/1281
node_id: MDEyOklzc3VlQ29tbWVudDgwODk5OTUyNQ==
created_at: 2021-03-29T01:00:38Z
updated_at: 2021-03-29T01:00:38Z
author_association: OWNER
body:
  Here's the diff between `Dockerfile` in 0.54.1 and 0.56: https://github.com/simonw/datasette/compare/0.54.1...0.56#diff-551d1fcf87f78cc3bc18a7b332a4dc5d8773a512062df881c5aba28a6f5c48d7

  <img width="1357" alt="Banners_and_Alerts_and_Comparing_0_54_1___0_56_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/112774760-79e5ef80-8fef-11eb-886a-5d546d6d2f3d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1281#issuecomment-809000903
issue_url: https://api.github.com/repos/simonw/datasette/issues/1281
node_id: MDEyOklzc3VlQ29tbWVudDgwOTAwMDkwMw==
created_at: 2021-03-29T01:05:10Z
updated_at: 2021-03-29T01:05:10Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/runs/1763835467?check_suite_focus=true for Datasette 0.54 worked, and the output included this:

  ```
  Successfully tagged ***/datasette:0.54
  The push refers to repository [docker.io/***/datasette]
  aedd33c6b161: Preparing
  ...
  aedd33c6b161: Pushed
  0.54: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
  aedd33c6b161: Preparing
  ...
  5dacd731af1b: Layer already exists
  latest: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
  ```
  Here's that same section of output from the 0.56 release:
  ```
  Successfully tagged ***/datasette:0.56
  Using default tag: latest
  The push refers to repository [docker.io/***/datasette]
  4d4a9976adcc: Preparing
  ...
  9b2132a0d5cf: Pushed
  latest: digest: sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a size: 1793
  ```
  The difference here is the "Using default tag: latest" bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1281#issuecomment-809001273
issue_url: https://api.github.com/repos/simonw/datasette/issues/1281
node_id: MDEyOklzc3VlQ29tbWVudDgwOTAwMTI3Mw==
created_at: 2021-03-29T01:06:45Z
updated_at: 2021-03-29T01:06:45Z
author_association: OWNER
body:
  https://docs.docker.com/engine/reference/commandline/push/#push-all-tags-of-an-image

  > Use the `-a` (or `--all-tags`) option to push all tags of a local image.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1281#issuecomment-809001653
issue_url: https://api.github.com/repos/simonw/datasette/issues/1281
node_id: MDEyOklzc3VlQ29tbWVudDgwOTAwMTY1Mw==
created_at: 2021-03-29T01:08:31Z
updated_at: 2021-03-29T01:08:31Z
author_association: OWNER
body:
  I'm going to attempt to fix this manually for the 0.56 release, by building and tagging it by hand and then pushing the 0.56 tag to Docker Hub.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1281#issuecomment-809007255
issue_url: https://api.github.com/repos/simonw/datasette/issues/1281
node_id: MDEyOklzc3VlQ29tbWVudDgwOTAwNzI1NQ==
created_at: 2021-03-29T01:32:18Z
updated_at: 2021-03-29T01:32:18Z
author_association: OWNER
body:
  I'm going to build a new GitHub Actions workflow for this that lets me manually specify a tag to build and push as a Docker image.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1281#issuecomment-809008760
issue_url: https://api.github.com/repos/simonw/datasette/issues/1281
node_id: MDEyOklzc3VlQ29tbWVudDgwOTAwODc2MA==
created_at: 2021-03-29T01:38:21Z
updated_at: 2021-03-29T01:38:21Z
author_association: OWNER
body:
  Got this error:
  ```
  "docker tag" requires exactly 2 arguments.
  See 'docker tag --help'.

  Usage:  docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

  Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1281#issuecomment-809009580
issue_url: https://api.github.com/repos/simonw/datasette/issues/1281
node_id: MDEyOklzc3VlQ29tbWVudDgwOTAwOTU4MA==
created_at: 2021-03-29T01:41:48Z
updated_at: 2021-03-29T01:41:48Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/runs/2214871602?check_suite_focus=true worked:

  <img width="1001" alt="Banners_and_Alerts_and_Actions_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/112776676-270f3680-8ff5-11eb-90d1-533d4e82f677.png">

  Here's the 0.56 image on Docker Hub: https://hub.docker.com/layers/datasetteproject/datasette/0.56/images/sha256-701fc0f299a0ea79434a4852c46dab351254b9ac25dbe3c5f36fd5360caf52f9?context=explore

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1031#issuecomment-809010713
issue_url: https://api.github.com/repos/simonw/datasette/issues/1031
node_id: MDEyOklzc3VlQ29tbWVudDgwOTAxMDcxMw==
created_at: 2021-03-29T01:46:45Z
updated_at: 2021-03-29T01:46:45Z
author_association: OWNER
body:
  Sorry I didn't get to this PR sooner. I've joint-credited you in the release notes for this fix: https://docs.datasette.io/en/stable/changelog.html#v0-56

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/696#issuecomment-809548363
issue_url: https://api.github.com/repos/simonw/datasette/issues/696
node_id: MDEyOklzc3VlQ29tbWVudDgwOTU0ODM2Mw==
created_at: 2021-03-29T17:04:19Z
updated_at: 2021-03-29T17:04:19Z
author_association: OWNER
body:
  I tried this just now against Datasette 0.56 with the new Dockerfile from #1249 (that uses SQLite and SpatiaLite installed with `apt-get install`) and the tests all passed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1282#issuecomment-809667320
issue_url: https://api.github.com/repos/simonw/datasette/issues/1282
node_id: MDEyOklzc3VlQ29tbWVudDgwOTY2NzMyMA==
created_at: 2021-03-29T19:52:35Z
updated_at: 2021-03-29T19:52:35Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=h1) Report
  > Merging [#1282](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=desc) (08f7427) into [main](https://codecov.io/gh/simonw/datasette/commit/0486303b60ce2784fd2e2ecdbecf304b7d6e6659?el=desc) (0486303) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1282/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1282   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=footer). Last update [0486303...08f7427](https://codecov.io/gh/simonw/datasette/pull/1282?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1284#issuecomment-810740486
issue_url: https://api.github.com/repos/simonw/datasette/issues/1284
node_id: MDEyOklzc3VlQ29tbWVudDgxMDc0MDQ4Ng==
created_at: 2021-03-31T03:57:55Z
updated_at: 2021-03-31T03:57:55Z
author_association: OWNER
body:
  You're right, doing this is really hard at the moment - I'm not sure I know how I would tackle this either, and it's something I've wanted in the past!

  I'll have a think about this one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1284#issuecomment-810779928
issue_url: https://api.github.com/repos/simonw/datasette/issues/1284
node_id: MDEyOklzc3VlQ29tbWVudDgxMDc3OTkyOA==
created_at: 2021-03-31T05:40:12Z
updated_at: 2021-03-31T05:40:12Z
author_association: CONTRIBUTOR
body:
  Maybe the addition of two template files: 'one_database_index.html' and 'one_table_index.html' would be a better idea than the documentation diff idea. (They could include commented instructions to rename the preferred template 'index.html', along with any other necessary guidance.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-810943882
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: MDEyOklzc3VlQ29tbWVudDgxMDk0Mzg4Mg==
created_at: 2021-03-31T10:03:55Z
updated_at: 2021-03-31T10:03:55Z
author_association: NONE
body:
  +1 on using nested queries to achieve this! Would be great as streaming CSV is an amazing feature.

  Some UX/DX details:

  I was expecting it to work to simply add `&_stream=on` to custom SQL queries because the docs say 

  > Any Datasette table, view or **custom SQL query** can be exported as CSV.

  After a bit of testing back and forth I realized streaming only works for full tables. 

  Would love this feature because I'm using `pandas.read_csv` to paint graphs from custom queries and the graphs are cut off because of the 1000 row limit. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-811362316
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31
node_id: MDEyOklzc3VlQ29tbWVudDgxMTM2MjMxNg==
created_at: 2021-03-31T19:14:39Z
updated_at: 2021-03-31T19:14:39Z
author_association: NONE
body:
  👋  could I help somehow for this to be merged? As Big Sur is going to be more used as the time goes I think it would be nice to merge and publish a new version. Nice work!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1287#issuecomment-812661269
issue_url: https://api.github.com/repos/simonw/datasette/issues/1287
node_id: MDEyOklzc3VlQ29tbWVudDgxMjY2MTI2OQ==
created_at: 2021-04-02T18:45:08Z
updated_at: 2021-04-02T18:45:19Z
author_association: OWNER
body:
  A few places:

  https://github.com/simonw/datasette/blob/7b1a9a1999eb9326ce8ec830d75ac200e5279c46/Dockerfile#L1

  https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L350

  https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/tests/test_package.py#L14

  https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/publish/heroku.py#L177-L178

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1287#issuecomment-812662026
issue_url: https://api.github.com/repos/simonw/datasette/issues/1287
node_id: MDEyOklzc3VlQ29tbWVudDgxMjY2MjAyNg==
created_at: 2021-04-02T18:46:20Z
updated_at: 2021-04-02T18:46:20Z
author_association: OWNER
body:
  https://devcenter.heroku.com/articles/python-support#supported-runtimes looks like Heroku still only have 3.9.2 for the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1286#issuecomment-812663107
issue_url: https://api.github.com/repos/simonw/datasette/issues/1286
node_id: MDEyOklzc3VlQ29tbWVudDgxMjY2MzEwNw==
created_at: 2021-04-02T18:49:22Z
updated_at: 2021-04-02T18:49:22Z
author_association: OWNER
body:
  This makes senses - showing an array as `["blah", "blah2", "blah3"]` isn't particularly human-friendly!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1286#issuecomment-812664443
issue_url: https://api.github.com/repos/simonw/datasette/issues/1286
node_id: MDEyOklzc3VlQ29tbWVudDgxMjY2NDQ0Mw==
created_at: 2021-04-02T18:52:45Z
updated_at: 2021-04-02T18:52:51Z
author_association: OWNER
body:
  Idea: default to displaying single-dimension JSON arrays of strings as a comma-separated list but show the comma in a different colour - something like this:

  <img width="561" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/113445005-dce9d480-93a9-11eb-901c-0c2fb079c150.png">

  I used this HTML for the prototype (re-using `.type-int` just to get the colour):
  ```html
  <td class="col-tags type-str">tag1<span class="type-int">, </span>tag2</td>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1286#issuecomment-812679221
issue_url: https://api.github.com/repos/simonw/datasette/issues/1286
node_id: MDEyOklzc3VlQ29tbWVudDgxMjY3OTIyMQ==
created_at: 2021-04-02T19:34:01Z
updated_at: 2021-04-02T19:34:01Z
author_association: CONTRIBUTOR
body:
  This shows the city in a different color (and not the comma), but I get the idea, and I like it. (Ooh, could be nice to have the gear have an option in array fields to show as bullets or commas or semicolons...)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1255#issuecomment-812680519
issue_url: https://api.github.com/repos/simonw/datasette/issues/1255
node_id: MDEyOklzc3VlQ29tbWVudDgxMjY4MDUxOQ==
created_at: 2021-04-02T19:37:57Z
updated_at: 2021-04-02T19:37:57Z
author_association: NONE
body:
  Hello,

  I'm also experiencing a timeout in my environment. I don't know if it's because I need more indexes or a more powerful system.

  My data has 1,271,111 and when I try to create a facet, there's a time out. I've tried this on two different rows that should significantly filter down data: `CITY` and `PARTY_REG`.

  Simon's johns_hopkins_csse_daily_reports has more rows and it setup with two facets on load. He does have four indexes created, though. Do I need more indexes?

  I have one simple one so far:
  ```
  CREATE INDEX [idx_party_reg]
      ON [county_active] ([PARTY_REG]);
  ```

  I'm running Datasette 0.56 installed via pip with Python 3.7.3.

  `4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux`

  ```
  $ cat /etc/os-release
  PRETTY_NAME="Debian GNU/Linux 10 (buster)"
  NAME="Debian GNU/Linux"
  VERSION_ID="10"
  VERSION="10 (buster)"
  VERSION_CODENAME=buster
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1255#issuecomment-812710120
issue_url: https://api.github.com/repos/simonw/datasette/issues/1255
node_id: MDEyOklzc3VlQ29tbWVudDgxMjcxMDEyMA==
created_at: 2021-04-02T20:50:08Z
updated_at: 2021-04-02T20:50:08Z
author_association: NONE
body:
  Hello again,

  I was able to get my facets running with this `settings.json`, which was lifted from one of Simon's datasette's and slightly modified.

  ```
  {
      "default_page_size": 100,
      "max_returned_rows": 1000,
      "num_sql_threads": 3,
      "sql_time_limit_ms": 9000,
      "default_facet_size": 10,
      "facet_time_limit_ms": 9000,
      "facet_suggest_time_limit_ms": 500,
      "hash_urls": false,
      "allow_facet": true,
      "suggest_facets": false,
      "default_cache_ttl": 5,
      "default_cache_ttl_hashed": 31536000,
      "cache_size_kb": 0,
      "allow_csv_stream": true,
      "max_csv_mb": 100,
      "truncate_cells_html": 2048,
      "template_debug": false,
      "base_url": "/"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1245#issuecomment-812711365
issue_url: https://api.github.com/repos/simonw/datasette/issues/1245
node_id: MDEyOklzc3VlQ29tbWVudDgxMjcxMTM2NQ==
created_at: 2021-04-02T20:53:35Z
updated_at: 2021-04-02T20:53:35Z
author_association: NONE
body:
  Yes, I agree. 

  Alternatively, maybe the header could be at the top and bottom, above the next page button.

  Maybe even have the header 50 records down?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1289#issuecomment-812767460
issue_url: https://api.github.com/repos/simonw/datasette/issues/1289
node_id: MDEyOklzc3VlQ29tbWVudDgxMjc2NzQ2MA==
created_at: 2021-04-03T00:48:26Z
updated_at: 2021-04-03T00:48:26Z
author_association: OWNER
body:
  On my Mac `pytest-xdist` ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1289#issuecomment-812768915
issue_url: https://api.github.com/repos/simonw/datasette/issues/1289
node_id: MDEyOklzc3VlQ29tbWVudDgxMjc2ODkxNQ==
created_at: 2021-04-03T00:59:15Z
updated_at: 2021-04-03T00:59:26Z
author_association: OWNER
body:
  Looks like `-n auto` only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true
  ```
  ============================= test session starts ==============================
  platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
  SQLite: 3.31.1
  rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini
  plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0
  gw0 I / gw1 I
  gw0 [878] / gw1 [878]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1289#issuecomment-812803256
issue_url: https://api.github.com/repos/simonw/datasette/issues/1289
node_id: MDEyOklzc3VlQ29tbWVudDgxMjgwMzI1Ng==
created_at: 2021-04-03T03:29:25Z
updated_at: 2021-04-03T03:29:25Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/actions/runs/713207828 ran with `pytest-xdist` in 4m22s:

  <img width="1202" alt="Run_tests_in_CI_using_pytest-xdist__refs__1289_·_simonw_datasette_d63fc61" src="https://user-images.githubusercontent.com/9599/113466592-f151bf80-93f1-11eb-9d74-b830e96c19c8.png">

  Here's the test suite running on regular `pytest` in 5m13s:

  <img width="1162" alt="©_2017-2021_·_simonw_datasette_59ef4a2" src="https://user-images.githubusercontent.com/9599/113466606-00387200-93f2-11eb-95a3-24521c628cbe.png">

  Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1290#issuecomment-812804178
issue_url: https://api.github.com/repos/simonw/datasette/issues/1290
node_id: MDEyOklzc3VlQ29tbWVudDgxMjgwNDE3OA==
created_at: 2021-04-03T03:39:16Z
updated_at: 2021-04-03T03:41:29Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=h1) Report
  > Merging [#1290](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=desc) (2fb1e42) into [main](https://codecov.io/gh/simonw/datasette/commit/87b583a128986982552421d2510e467e74ac5046?el=desc) (87b583a) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1290/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1290   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=footer). Last update [87b583a...2fb1e42](https://codecov.io/gh/simonw/datasette/pull/1290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/916#issuecomment-812804998
issue_url: https://api.github.com/repos/simonw/datasette/issues/916
node_id: MDEyOklzc3VlQ29tbWVudDgxMjgwNDk5OA==
created_at: 2021-04-03T03:47:45Z
updated_at: 2021-04-03T03:47:45Z
author_association: OWNER
body:
  I found one example of an implementation of reversed keyset pagination here: https://github.com/tvainika/objection-keyset-pagination/blob/cb21a493c96daa6e63c302efae6718d09aa11661/index.js#L74-L79

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/502#issuecomment-812813732
issue_url: https://api.github.com/repos/simonw/datasette/issues/502
node_id: MDEyOklzc3VlQ29tbWVudDgxMjgxMzczMg==
created_at: 2021-04-03T05:16:54Z
updated_at: 2021-04-03T05:16:54Z
author_association: CONTRIBUTOR
body:
  For what it's worth, if anyone finds this in the future, I was having the same issue. 

  After digging through the code, it turned out that the database download is only available if it the db served in immutable mode, so `datasette serve -i xyz.db` rather than the doc's quickstart recommendation of `datasette serve xyz.db`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1291#issuecomment-812815358
issue_url: https://api.github.com/repos/simonw/datasette/issues/1291
node_id: MDEyOklzc3VlQ29tbWVudDgxMjgxNTM1OA==
created_at: 2021-04-03T05:32:50Z
updated_at: 2021-06-05T19:48:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1291](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8789157) into [main](https://codecov.io/gh/simonw/datasette/commit/0a7621f96f8ad14da17e7172e8a7bce24ef78966?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0a7621f) will **not change** coverage.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 8789157 differs from pull request most recent head 9bf089f. Consider uploading reports for the commit 9bf089f to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1291/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1291   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0a7621f...9bf089f](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1287#issuecomment-812935384
issue_url: https://api.github.com/repos/simonw/datasette/issues/1287
node_id: MDEyOklzc3VlQ29tbWVudDgxMjkzNTM4NA==
created_at: 2021-04-03T22:38:33Z
updated_at: 2021-04-03T22:38:33Z
author_association: OWNER
body:
  https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/916#issuecomment-812940457
issue_url: https://api.github.com/repos/simonw/datasette/issues/916
node_id: MDEyOklzc3VlQ29tbWVudDgxMjk0MDQ1Nw==
created_at: 2021-04-03T23:28:40Z
updated_at: 2021-04-03T23:28:40Z
author_association: OWNER
body:
  I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a `?_next=x` token that would provide the previous page sorted back in the expected default order.

  The alternative would be to have a `?_previous=x` token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in "hit next page, then hit previous page" returning you to a new state which features rows in the reverse order.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/916#issuecomment-812940907
issue_url: https://api.github.com/repos/simonw/datasette/issues/916
node_id: MDEyOklzc3VlQ29tbWVudDgxMjk0MDkwNw==
created_at: 2021-04-03T23:33:41Z
updated_at: 2021-04-03T23:33:41Z
author_association: OWNER
body:
  Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr

  Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r

  ```sql
  select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0)
    or
  (pk1 = :p0 and pk2 > :p1)
    or
  (pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101
  ```
  Where `p0` is `a`, `p1` is `h` and `p2` is `r`.

  Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - `a`, `d`, `v`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/916#issuecomment-812941112
issue_url: https://api.github.com/repos/simonw/datasette/issues/916
node_id: MDEyOklzc3VlQ29tbWVudDgxMjk0MTExMg==
created_at: 2021-04-03T23:35:55Z
updated_at: 2021-04-03T23:35:55Z
author_association: OWNER
body:
  I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r

  ```sql
  select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0)
    or
  (pk1 = :p0 and pk2 < :p1)
    or
  (pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99
  ```
  Which returned `a-d-v` as desired. I messed around with it to find the `limit 1 offset 99` values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/916#issuecomment-812941340
issue_url: https://api.github.com/repos/simonw/datasette/issues/916
node_id: MDEyOklzc3VlQ29tbWVudDgxMjk0MTM0MA==
created_at: 2021-04-03T23:38:37Z
updated_at: 2021-04-03T23:38:37Z
author_association: OWNER
body:
  Same query again with `a, d, v` returns 0 results, which is also as we would want: it signifies that we are back to the very first page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=d&p2=v

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/916#issuecomment-812941818
issue_url: https://api.github.com/repos/simonw/datasette/issues/916
node_id: MDEyOklzc3VlQ29tbWVudDgxMjk0MTgxOA==
created_at: 2021-04-03T23:43:11Z
updated_at: 2021-04-03T23:43:11Z
author_association: OWNER
body:
  Relevant code is some of the most complex in all of Datasette.

  https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594

  And

  https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771

  I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1292#issuecomment-813109789
issue_url: https://api.github.com/repos/simonw/datasette/issues/1292
node_id: MDEyOklzc3VlQ29tbWVudDgxMzEwOTc4OQ==
created_at: 2021-04-04T22:37:47Z
updated_at: 2021-04-04T22:37:47Z
author_association: OWNER
body:
  Could maybe replace this code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/utils/__init__.py#L1021-L1026

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813112546
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExMjU0Ng==
created_at: 2021-04-04T23:02:45Z
updated_at: 2021-04-04T23:02:45Z
author_association: OWNER
body:
  I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.

  The SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.

  Maybe it would be possible to use them via `ctypes`? My hunch is that I would have to re-implement the full `sqlite3` module with `ctypes`, which sounds daunting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813113175
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExMzE3NQ==
created_at: 2021-04-04T23:07:01Z
updated_at: 2021-04-04T23:07:01Z
author_association: OWNER
body:
  A more promising route I found involved the `db.set_authorizer` method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!

  ```pycon
  >>> def print_args(*args, **kwargs):
  ...    print("args", args, "kwargs", kwargs)
  ...    return sqlite3.SQLITE_OK

  >>> db = sqlite3.connect("fixtures.db")
  >>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()
  args (21, None, None, None, None) kwargs {}
  args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
  args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
  args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
  args (20, 'facetable', 'pk', 'main', None) kwargs {}
  args (20, 'facetable', 'created', 'main', None) kwargs {}
  args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
  args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
  args (20, 'facetable', 'state', 'main', None) kwargs {}
  args (20, 'facetable', 'city_id', 'main', None) kwargs {}
  args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
  args (20, 'facetable', 'tags', 'main', None) kwargs {}
  args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
  args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
  ```
  Those `20` values (where 20 is `SQLITE_READ`) looked like they were checking permissions for the columns in the order they would be returned!

  Then I found a snag:

  ```pycon
  In [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')
  args (21, None, None, None, None) kwargs {}
  args (31, None, 'max', None, None) kwargs {}
  args (20, 'facetable', 'pk', 'main', None) kwargs {}
  args (21, None, None, None, None) kwargs {}
  args (20, 'facetable', '', None, None) kwargs {}
  ```
  Once a subselect is involved the order of the `20` checks no longer matches the order in which the columns are returned from the query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813113218
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExMzIxOA==
created_at: 2021-04-04T23:07:25Z
updated_at: 2021-04-04T23:07:25Z
author_association: OWNER
body:
  Here are all of the available constants:
  ```pycon
  In [3]: for k in dir(sqlite3):
     ...:     if k.startswith("SQLITE_"):
     ...:         print(k, getattr(sqlite3, k))
     ...: 
  SQLITE_ALTER_TABLE 26
  SQLITE_ANALYZE 28
  SQLITE_ATTACH 24
  SQLITE_CREATE_INDEX 1
  SQLITE_CREATE_TABLE 2
  SQLITE_CREATE_TEMP_INDEX 3
  SQLITE_CREATE_TEMP_TABLE 4
  SQLITE_CREATE_TEMP_TRIGGER 5
  SQLITE_CREATE_TEMP_VIEW 6
  SQLITE_CREATE_TRIGGER 7
  SQLITE_CREATE_VIEW 8
  SQLITE_CREATE_VTABLE 29
  SQLITE_DELETE 9
  SQLITE_DENY 1
  SQLITE_DETACH 25
  SQLITE_DONE 101
  SQLITE_DROP_INDEX 10
  SQLITE_DROP_TABLE 11
  SQLITE_DROP_TEMP_INDEX 12
  SQLITE_DROP_TEMP_TABLE 13
  SQLITE_DROP_TEMP_TRIGGER 14
  SQLITE_DROP_TEMP_VIEW 15
  SQLITE_DROP_TRIGGER 16
  SQLITE_DROP_VIEW 17
  SQLITE_DROP_VTABLE 30
  SQLITE_FUNCTION 31
  SQLITE_IGNORE 2
  SQLITE_INSERT 18
  SQLITE_OK 0
  SQLITE_PRAGMA 19
  SQLITE_READ 20
  SQLITE_RECURSIVE 33
  SQLITE_REINDEX 27
  SQLITE_SAVEPOINT 32
  SQLITE_SELECT 21
  SQLITE_TRANSACTION 22
  SQLITE_UPDATE 23
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813113403
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExMzQwMw==
created_at: 2021-04-04T23:08:48Z
updated_at: 2021-04-04T23:08:48Z
author_association: OWNER
body:
  Worth noting that adding `limit 0` to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:
  ```pycon
  In [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()
      ...: 
  args (21, None, None, None, None) kwargs {}
  args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
  args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
  args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
  args (20, 'facetable', 'pk', 'main', None) kwargs {}
  args (20, 'facetable', 'created', 'main', None) kwargs {}
  args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
  args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
  args (20, 'facetable', 'state', 'main', None) kwargs {}
  args (20, 'facetable', 'city_id', 'main', None) kwargs {}
  args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
  args (20, 'facetable', 'tags', 'main', None) kwargs {}
  args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
  args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
  args (20, 'facetable', 'pk', 'main', None) kwargs {}
  args (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}
  Out[20]: []
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813113653
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExMzY1Mw==
created_at: 2021-04-04T23:10:49Z
updated_at: 2021-04-04T23:10:49Z
author_association: OWNER
body:
  One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?

  Then I could load that extension and run a SQL query something like this:

  ```
  select database, table, column from analyze_query(:sql_query)
  ```
  Where `analyze_query(...)` would be a fancy virtual table function of some sort that uses the underlying `sqlite3_column_database_name()` C functions to analyze the SQL query and return details of what it would return.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813115414
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExNTQxNA==
created_at: 2021-04-04T23:23:34Z
updated_at: 2021-04-04T23:23:34Z
author_association: OWNER
body:
  The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment `select *` came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the `*`.

  A Python parser approach might be good enough to handle a subset of queries - those that don't use `select *` for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813115607
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExNTYwNw==
created_at: 2021-04-04T23:25:15Z
updated_at: 2021-04-04T23:25:15Z
author_association: OWNER
body:
  Oh wow, I just spotted https://github.com/macbre/sql-metadata

  > Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.

  It's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813116177
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzExNjE3Nw==
created_at: 2021-04-04T23:31:00Z
updated_at: 2021-04-04T23:31:00Z
author_association: OWNER
body:
  Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:

  <img width="597" alt="sql-metadata_-_Jupyter_Notebook" src="https://user-images.githubusercontent.com/9599/113524385-216ca000-9563-11eb-8a7e-cbbe5dfc02f6.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813134072
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg==
created_at: 2021-04-05T01:18:37Z
updated_at: 2021-04-05T01:18:37Z
author_association: OWNER
body:
  Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by `explain select ...`.

  Copying the reply I posted to this thread:

  That's really useful, thanks! It looks like it _might_ be possible for me to reconstruct where each column came from using the `explain select` output.

  Here's a complex example: <https://calands.datasettes.com/calands?sql=explain+select%0D%0A++AsGeoJSON%28geometry%29%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+%27%25mini%25%27+and%0D%0A++Intersects%28GeomFromGeoJSON%28%3Afreedraw%29%2C+geometry%29+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+%27CPAD_2020a_SuperUnits%27%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON%28%3Afreedraw%29%0D%0A++%29&freedraw=%7B%22type%22%3A%22MultiPolygon%22%2C%22coordinates%22%3A%5B%5B%5B%5B-122.42202758789064%2C37.82280243352759%5D%2C%5B-122.39868164062501%2C37.823887203271454%5D%2C%5B-122.38220214843751%2C37.81846319511331%5D%2C%5B-122.35061645507814%2C37.77071473849611%5D%2C%5B-122.34924316406251%2C37.74465712069939%5D%2C%5B-122.37258911132814%2C37.703380457832374%5D%2C%5B-122.39044189453125%2C37.690340943717715%5D%2C%5B-122.41241455078126%2C37.680559803205135%5D%2C%5B-122.44262695312501%2C37.67295135774715%5D%2C%5B-122.47283935546876%2C37.67295135774715%5D%2C%5B-122.52502441406251%2C37.68382032669382%5D%2C%5B-122.53463745117189%2C37.6892542140253%5D%2C%5B-122.54699707031251%2C37.690340943717715%5D%2C%5B-122.55798339843751%2C37.72945260537781%5D%2C%5B-122.54287719726564%2C37.77831314799672%5D%2C%5B-122.49893188476564%2C37.81303878836991%5D%2C%5B-122.46185302734376%2C37.82822612280363%5D%2C%5B-122.42889404296876%2C37.82822612280363%5D%2C%5B-122.42202758789064%2C37.82280243352759%5D%5D%5D%5D%7D>

  It looks like the opcodes I need to inspect are `OpenRead`, `Column` and `ResultRow`.

  `OpenRead` tells me which tables are being opened - the `p2` value (in this case 51) corresponds to the `rootpage` column in `sqlite_master` here: <https://calands.datasettes.com/calands?sql=select+*+from+sqlite_master> - it gets assigned to the register in `p1`.

  The `Column` opcodes tell me which columns are being read - `p1` is that table reference, and `p2` is the `cid` of the column within that table.

  The `ResultRow` opcode then tells me which columns are used in the results. `15 16` means start at the 15th and then read the next 16 columns.

  I think this might work!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813134227
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzEzNDIyNw==
created_at: 2021-04-05T01:19:31Z
updated_at: 2021-04-05T01:19:31Z
author_association: OWNER
body:
  |   addr | opcode        |   p1 |   p2 |   p3 | p4                    |   p5 | comment   |
  |--------|---------------|------|------|------|-----------------------|------|-----------|
  |      0 | Init          |    0 |   47 |    0 |                       |   00 |           |
  |      1 | OpenRead      |    0 |   51 |    0 | 15                    |   00 |           |
  |      2 | Integer       |   15 |    2 |    0 |                       |   00 |           |
  |      3 | Once          |    0 |   15 |    0 |                       |   00 |           |
  |      4 | OpenEphemeral |    2 |    1 |    0 | k(1,)                 |   00 |           |
  |      5 | VOpen         |    1 |    0 |    0 | vtab:3E692C362158     |   00 |           |
  |      6 | String8       |    0 |    5 |    0 | CPAD_2020a_SuperUnits |   00 |           |
  |      7 | SCopy         |    7 |    6 |    0 |                       |   00 |           |
  |      8 | Integer       |    2 |    3 |    0 |                       |   00 |           |
  |      9 | Integer       |    2 |    4 |    0 |                       |   00 |           |
  |     10 | VFilter       |    1 |   15 |    3 |                       |   00 |           |
  |     11 | Rowid         |    1 |    8 |    0 |                       |   00 |           |
  |     12 | MakeRecord    |    8 |    1 |    9 | C                     |   00 |           |
  |     13 | IdxInsert     |    2 |    9 |    8 | 1                     |   00 |           |
  |     14 | VNext         |    1 |   11 |    0 |                       |   00 |           |
  |     15 | Return        |    2 |    0 |    0 |                       |   00 |           |
  |     16 | Rewind        |    2 |   46 |    0 |                       |   00 |           |
  |     17 | Column        |    2 |    0 |    1 |                       |   00 |           |
  |     18 | IsNull        |    1 |   45 |    0 |                       |   00 |           |
  |     19 | SeekRowid     |    0 |   45 |    1 |                       |   00 |           |
  |     20 | Column        |    0 |    2 |   11 |                       |   00 |           |
  |     21 | Function0     |    1 |   10 |    9 | like(2)               |   02 |           |
  |     22 | IfNot         |    9 |   45 |    1 |                       |   00 |           |
  |     23 | Column        |    0 |   14 |   13 |                       |   00 |           |
  |     24 | Function0     |    1 |   12 |    9 | intersects(2)         |   02 |           |
  |     25 | Ne            |   14 |   45 |    9 |                       |   51 |           |
  |     26 | Column        |    0 |   14 |    9 |                       |   00 |           |
  |     27 | Function0     |    0 |    9 |   15 | asgeojson(1)          |   01 |           |
  |     28 | Rowid         |    0 |   16 |    0 |                       |   00 |           |
  |     29 | Column        |    0 |    1 |   17 |                       |   00 |           |
  |     30 | Column        |    0 |    2 |   18 |                       |   00 |           |
  |     31 | Column        |    0 |    3 |   19 |                       |   00 |           |
  |     32 | Column        |    0 |    4 |   20 |                       |   00 |           |
  |     33 | Column        |    0 |    5 |   21 |                       |   00 |           |
  |     34 | Column        |    0 |    6 |   22 |                       |   00 |           |
  |     35 | Column        |    0 |    7 |   23 |                       |   00 |           |
  |     36 | Column        |    0 |    8 |   24 |                       |   00 |           |
  |     37 | Column        |    0 |    9 |   25 |                       |   00 |           |
  |     38 | Column        |    0 |   10 |   26 |                       |   00 |           |
  |     39 | Column        |    0 |   11 |   27 |                       |   00 |           |
  |     40 | RealAffinity  |   27 |    0 |    0 |                       |   00 |           |
  |     41 | Column        |    0 |   12 |   28 |                       |   00 |           |
  |     42 | Column        |    0 |   13 |   29 |                       |   00 |           |
  |     43 | Column        |    0 |   14 |   30 |                       |   00 |           |
  |     44 | ResultRow     |   15 |   16 |    0 |                       |   00 |           |
  |     45 | Next          |    2 |   17 |    0 |                       |   00 |           |
  |     46 | Halt          |    0 |    0 |    0 |                       |   00 |           |
  |     47 | Transaction   |    0 |    0 |  265 | 0                     |   01 |           |
  |     48 | Variable      |    1 |   31 |    0 | :freedraw             |   00 |           |
  |     49 | Function0     |    1 |   31 |    7 | geomfromgeojson(1)    |   01 |           |
  |     50 | String8       |    0 |   10 |    0 | %mini%                |   00 |           |
  |     51 | Variable      |    1 |   32 |    0 | :freedraw             |   00 |           |
  |     52 | Function0     |    1 |   32 |   12 | geomfromgeojson(1)    |   01 |           |
  |     53 | Integer       |    1 |   14 |    0 |                       |   00 |           |
  |     54 | Goto          |    0 |    1 |    0 |                       |   00 |           |

  Essential documentation for understanding that output: https://www.sqlite.org/opcode.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813134386
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzEzNDM4Ng==
created_at: 2021-04-05T01:20:28Z
updated_at: 2021-08-13T00:42:30Z
author_association: OWNER
body:
  ... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the `Variable` opcodes.

  [UPDATE: it did indeed do that, see #1421]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813134637
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzEzNDYzNw==
created_at: 2021-04-05T01:21:59Z
updated_at: 2021-04-05T01:21:59Z
author_association: OWNER
body:
  http://www.sqlite.org/draft/lang_explain.html says:

  > Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

  I'm going to keep exploring this though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813162622
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzE2MjYyMg==
created_at: 2021-04-05T03:34:24Z
updated_at: 2021-04-05T03:40:35Z
author_association: OWNER
body:
  This almost works, but throws errors with some queries (anything with a `rowid` column for example) - it needs a bunch of test coverage.
  ```python
  def columns_for_query(conn, sql):
      rows = conn.execute('explain ' + sql).fetchall()
      table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
      names_by_rootpage = dict(
          conn.execute(
              'select rootpage, name from sqlite_master where rootpage in ({})'.format(
                  ', '.join(map(str, table_rootpage_by_register.values()))
              )
          )
      )
      columns_by_column_register = {}
      for row in rows:
          if row['opcode'] == 'Column':
              addr, opcode, table_id, cid, column_register, p4, p5, comment = row
              table = names_by_rootpage[table_rootpage_by_register[table_id]]
              columns_by_column_register[column_register] = (table, cid)
      result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
      registers = list(range(result_row["p1"], result_row["p1"] + result_row["p2"] - 1))
      all_column_names = {}
      for table in names_by_rootpage.values():
          table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
          for row in table_xinfo:
              all_column_names[(table, row["cid"])] = row["name"]
      final_output = []
      for r in registers:
          try:
              table, cid = columns_by_column_register[r]
              final_output.append((table, all_column_names[table, cid]))
          except KeyError:
              final_output.append((None, None))
      return final_output
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813164282
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg==
created_at: 2021-04-05T03:42:26Z
updated_at: 2021-04-05T03:42:36Z
author_association: OWNER
body:
  Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the `explain select...` query. Using `defaultdict` seems to work there:

  ```pycon
  >>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))
  >>> [dict(r) for r in rows if r['opcode'] == 'Variable']
  [{'addr': 2,
    'opcode': 'Variable',
    'p1': 1,
    'p2': 1,
    'p3': 0,
    'p4': ':id',
    'p5': 0,
    'comment': None}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/620#issuecomment-813167335
issue_url: https://api.github.com/repos/simonw/datasette/issues/620
node_id: MDEyOklzc3VlQ29tbWVudDgxMzE2NzMzNQ==
created_at: 2021-04-05T03:57:22Z
updated_at: 2021-04-05T03:57:22Z
author_association: OWNER
body:
  This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/35#issuecomment-813249000
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35
node_id: MDEyOklzc3VlQ29tbWVudDgxMzI0OTAwMA==
created_at: 2021-04-05T07:37:57Z
updated_at: 2021-04-05T07:37:57Z
author_association: NONE
body:
  There are trained ML models used in Photoprism:
  - https://dl.photoprism.org/tensorflow/nasnet.zip
  - https://dl.photoprism.org/tensorflow/nsfw.zip

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813438771
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzQzODc3MQ==
created_at: 2021-04-05T14:58:48Z
updated_at: 2021-04-05T14:58:48Z
author_association: OWNER
body:
  I may need to do something special for rowid columns - there is a `RowId` opcode that might come into play here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813445512
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzQ0NTUxMg==
created_at: 2021-04-05T15:11:40Z
updated_at: 2021-04-05T15:11:40Z
author_association: OWNER
body:
  Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1293#issuecomment-813480043
issue_url: https://api.github.com/repos/simonw/datasette/issues/1293
node_id: MDEyOklzc3VlQ29tbWVudDgxMzQ4MDA0Mw==
created_at: 2021-04-05T16:16:17Z
updated_at: 2021-04-05T16:16:17Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:

  ```sql
  CREATE VIEW paginated_view AS
      SELECT
          content,
          '- ' || content || ' -' AS content_extra
      FROM no_primary_key;
  ```
  So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1286#issuecomment-815978405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1286
node_id: MDEyOklzc3VlQ29tbWVudDgxNTk3ODQwNQ==
created_at: 2021-04-08T16:47:29Z
updated_at: 2021-04-10T03:59:00Z
author_association: CONTRIBUTOR
body:
  This worked for me:                      
  `<td class="col-{{ cell.column|to_css_class }} type-{{ cell.value_type }}">{{ cell.value | replace('", "','; ') | replace('[\"','') | replace('\"]','')}}</td>`

  I'm sure there is a prettier (and more flexible) way, but for now, this is ever-so-much more pleasant to look at. 

  ------ AFTER:
  <img width="778" alt="Screen Shot 2021-04-08 at 12 27 36 PM" src="https://user-images.githubusercontent.com/192568/114062829-f2f20c00-9865-11eb-891d-b0c348a9b433.png">

  ------ BEFORE:
  <img width="795" alt="Screen Shot 2021-04-08 at 12 25 54 PM" src="https://user-images.githubusercontent.com/192568/114062871-01402800-9866-11eb-91ce-91efe4ee45cd.png">



  (Note: I didn't figure out how to have one item have no semicolon, while multi-items close with a semicolon, but this is good enough for now. I also didn't figure out how to set up a new jinja filter. I don't want to add to /datasette/utils/__init__.py as I assume that would get overwritten when upgrading datasette. Having a starter guide on creating jinja filters in datasette would be helpful. (The jinja documentation isn't datasette-specific enough for me to quite nail it.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1295#issuecomment-817301355
issue_url: https://api.github.com/repos/simonw/datasette/issues/1295
node_id: MDEyOklzc3VlQ29tbWVudDgxNzMwMTM1NQ==
created_at: 2021-04-11T12:40:25Z
updated_at: 2021-04-11T12:41:06Z
author_association: OWNER
body:
  I could have a page about error codes in the docs, then have `https://datasette.io/E123` style URLs for each error core which are shown when that error occurs and redirect to the corresponding documentation section.

  Can enforce these with a documentation unit test.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1296#issuecomment-817403642
issue_url: https://api.github.com/repos/simonw/datasette/issues/1296
node_id: MDEyOklzc3VlQ29tbWVudDgxNzQwMzY0Mg==
created_at: 2021-04-12T00:29:05Z
updated_at: 2021-07-20T08:52:12Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1296](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (527a056) into [main](https://codecov.io/gh/simonw/datasette/commit/c73af5dd72305f6a01ea94a2c76d52e5e26de38b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c73af5d) will **decrease** coverage by `0.11%`.
  > The diff coverage is `n/a`.

  > :exclamation: Current head 527a056 differs from pull request most recent head 8f00c31. Consider uploading reports for the commit 8f00c31 to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1296/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1296      +/-   ##
  ==========================================
  - Coverage   91.62%   91.51%   -0.12%     
  ==========================================
    Files          34       34              
    Lines        4371     4255     -116     
  ==========================================
  - Hits         4005     3894     -111     
  + Misses        366      361       -5     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `81.60% <0.00%> (-1.35%)` | :arrow_down: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <0.00%> (-0.42%)` | :arrow_down: |
  | [datasette/facets.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZhY2V0cy5weQ==) | `89.04% <0.00%> (-0.41%)` | :arrow_down: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.13% <0.00%> (-0.21%)` | :arrow_down: |
  | [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.02% <0.00%> (-0.18%)` | :arrow_down: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.19% <0.00%> (-0.10%)` | :arrow_down: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.88% <0.00%> (-0.07%)` | :arrow_down: |
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.36% <0.00%> (-0.07%)` | :arrow_down: |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <0.00%> (ø)` | |
  | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.38% <0.00%> (ø)` | |
  | ... and [5 more](https://codecov.io/gh/simonw/datasette/pull/1296/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [c73af5d...8f00c31](https://codecov.io/gh/simonw/datasette/pull/1296?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/830#issuecomment-817414881
issue_url: https://api.github.com/repos/simonw/datasette/issues/830
node_id: MDEyOklzc3VlQ29tbWVudDgxNzQxNDg4MQ==
created_at: 2021-04-12T01:06:34Z
updated_at: 2021-04-12T01:07:27Z
author_association: CONTRIBUTOR
body:
  Related: #1285, including arguments for natural breaks, equal interval, etc. modeled after choropleth map legends.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1296#issuecomment-819467759
issue_url: https://api.github.com/repos/simonw/datasette/issues/1296
node_id: MDEyOklzc3VlQ29tbWVudDgxOTQ2Nzc1OQ==
created_at: 2021-04-14T12:07:37Z
updated_at: 2021-04-14T12:11:36Z
author_association: CONTRIBUTOR
body:
  > Removing /var/lib/apt and /var/lib/dpkg makes apt and dpkg unusable in
  images based on this one. Running `apt-get clean` and removing
  /var/lib/apt/lists achieves similar size savings.

  this PR helps me as removing the /var/lib/apt and /var/lib/dpkg directories breaks my ability to add packages when using `datasetteproject/datasette:0.56` as a base image.


  ---- 
  Shorterm workaround for me was to use this in my Dockerfile
  ```
  FROM datasetteproject/datasette:0.56

  RUN mkdir -p /var/lib/apt
  RUN mkdir -p /var/lib/dpkg
  RUN mkdir -p /var/lib/dpkg/updates
  RUN mkdir -p /var/lib/dpkg/info
  RUN touch /var/lib/dpkg/status

  RUN apt-get update # and install your packages etc
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1196#issuecomment-819775388
issue_url: https://api.github.com/repos/simonw/datasette/issues/1196
node_id: MDEyOklzc3VlQ29tbWVudDgxOTc3NTM4OA==
created_at: 2021-04-14T19:28:38Z
updated_at: 2021-04-14T19:28:38Z
author_association: NONE
body:
  @QAInsights I'm having a similar problem when publishing to Cloud Run on Windows. It's not able to access certain packages in my conda environment where Datasette is installed. Can you explain how you got it to work in WSL? Were you able to access the .db file in the Windows file system? Thank you.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1300#issuecomment-821970965
issue_url: https://api.github.com/repos/simonw/datasette/issues/1300
node_id: MDEyOklzc3VlQ29tbWVudDgyMTk3MDk2NQ==
created_at: 2021-04-18T10:41:15Z
updated_at: 2021-04-18T10:41:15Z
author_association: CONTRIBUTOR
body:
  If I change the hookspec and add a row parameter, it works

  https://github.com/simonw/datasette/blob/7a2ed9f8a119e220b66d67c7b9e07cbab47b1196/datasette/hookspecs.py#L58

  ```
  def render_cell(value, column, row, table, database, datasette):
  ```

  But to generate a URL, I need the primary keys, but I can't call `pks = await db.primary_keys(table)`  inside a sync function. I can't call `datasette.utils.detect_primary_keys` either, because the db connection is not publicly exposed (AFAICT).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1303#issuecomment-822486113
issue_url: https://api.github.com/repos/simonw/datasette/issues/1303
node_id: MDEyOklzc3VlQ29tbWVudDgyMjQ4NjExMw==
created_at: 2021-04-19T13:55:24Z
updated_at: 2021-04-19T13:55:24Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1303?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1303](https://codecov.io/gh/simonw/datasette/pull/1303?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (c348ff1) into [main](https://codecov.io/gh/simonw/datasette/commit/0a7621f96f8ad14da17e7172e8a7bce24ef78966?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0a7621f) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1303/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1303?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1303   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1303?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1303?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0a7621f...c348ff1](https://codecov.io/gh/simonw/datasette/pull/1303?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1298#issuecomment-823064725
issue_url: https://api.github.com/repos/simonw/datasette/issues/1298
node_id: MDEyOklzc3VlQ29tbWVudDgyMzA2NDcyNQ==
created_at: 2021-04-20T07:57:14Z
updated_at: 2021-04-20T07:57:14Z
author_association: NONE
body:
  My suggestions, originally made on twitter, but might be better here now:

  1. Could have a CSS shadow (one of the comments on https://stackoverflow.com/questions/44793453/how-do-i-add-a-top-and-bottom-shadow-while-scrolling-but-only-when-needed is a codepen for horizontal instead of vertical);

  2. Could give the table a max-height (either the window or work out the available space) so that it is both vertically/horizontally scrollable and you don't have to scroll to the bottom in order to see this;

  3. On a desktop browser, what I think I'd want is an absolute grid to work with - left query/filters, TR chart (or map), BR table. No problem with scrolling then. Here is a mockup I made when this was about the map plugin:
  ![image](https://user-images.githubusercontent.com/154364/115358389-82c47e00-a1b5-11eb-8a63-0ca14fd23d32.png)
  ![image](https://user-images.githubusercontent.com/154364/115358454-97087b00-a1b5-11eb-9501-cf884ae72d7c.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1298#issuecomment-823093669
issue_url: https://api.github.com/repos/simonw/datasette/issues/1298
node_id: MDEyOklzc3VlQ29tbWVudDgyMzA5MzY2OQ==
created_at: 2021-04-20T08:38:10Z
updated_at: 2021-04-20T08:40:22Z
author_association: CONTRIBUTOR
body:
  @dracos I appreciate your ideas!

  1. Ooh, I like this: https://codepen.io/astro87/pen/LYRQNbd?editors=1100 (That's the codepen from your linked stackoverflow.)
  2. I worry that a max height will be a problem when my facets are open. (I've got 35 active ingredients, and so I've set the default_facet_size to 35.)
  3. I don't understand this one. I'm observing the screenshot... very helpful!  (Ah, okay, TR = Top Right and BR = Bottom Right. Absolute grid refers to position style.) All the scroll bars look a little wonky to me. I've also got a lot of facets, and prefer the extra horizontal space so that not as many facets disappear below the fold. My site also has end users... some will be on mobile... not sure what the absolute grid would do there... 
  4. (I still think a hover-arrow that scrolls upon click would help, too...)

  But meanwhile, I'm going to go ahead and see if I can apply that shadow. (Never would've thought of that.) Hmmm... I'm not an SCSS person. This looks helpful! https://jsonformatter.org/scss-to-css

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1298#issuecomment-823102978
issue_url: https://api.github.com/repos/simonw/datasette/issues/1298
node_id: MDEyOklzc3VlQ29tbWVudDgyMzEwMjk3OA==
created_at: 2021-04-20T08:51:23Z
updated_at: 2021-04-20T08:51:23Z
author_association: NONE
body:
  2. Max height would still let you scroll the page to underneath the facets to the table, but would mean the table would never take up more than your window size, so the horizontal scrollbar would be visible as soon as the table took up the size of the window.
  3. Yes, this wouldn't be for mobile :) It'd be desktop-only styling. On mobile you can scroll much more easily with touch, anyway. In your case, perhaps better would be the whole top half would be facets, bottom left quadrant chart, bottom right table. Depends upon the particular use case, as you say.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/173#issuecomment-823961091
issue_url: https://api.github.com/repos/simonw/datasette/issues/173
node_id: MDEyOklzc3VlQ29tbWVudDgyMzk2MTA5MQ==
created_at: 2021-04-21T10:37:05Z
updated_at: 2021-04-21T10:37:36Z
author_association: NONE
body:
  I have the feeling that the text visible to users is 95% present in template files ([datasette/templates](https://github.com/simonw/datasette/tree/main/datasette/templates)). The python code mainly contains error messages.

  In the current situation, the best way to provide a localized frontend is to translate the templates and [configure datasette to use them](https://docs.datasette.io/en/stable/custom_templates.html). I think I'm going to do it for French.

  If we want localization to be better integrated, for the python code, I think [gettext](https://docs.python.org/3/library/gettext.html#localizing-your-application) is the way to go. The .po can be translated in user-friendly tools such as Transifex and Crowdin.

  For the templates, I'm not sure how we could do it cleanly and easy to maintain. Maybe the tools above could parse HTML and detect the strings to be translated.

  In any case, localization implementing l10n is just the first step: a continuous process must be setup to maintain the translations and produce new ones while datasette keeps getting new features.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1306#issuecomment-824866566
issue_url: https://api.github.com/repos/simonw/datasette/issues/1306
node_id: MDEyOklzc3VlQ29tbWVudDgyNDg2NjU2Ng==
created_at: 2021-04-22T13:59:04Z
updated_at: 2021-04-22T13:59:04Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1306?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1306](https://codecov.io/gh/simonw/datasette/pull/1306?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (115332c) into [main](https://codecov.io/gh/simonw/datasette/commit/6ed9238178a56da5fb019f37fb1e1e15886be1d1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (6ed9238) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1306/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1306   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4256    +1     
  =======================================
  + Hits         3894     3895    +1     
    Misses        361      361           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1306/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.42% <100.00%> (+0.06%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1306?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1306?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [6ed9238...115332c](https://codecov.io/gh/simonw/datasette/pull/1306?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1308#issuecomment-826040676
issue_url: https://api.github.com/repos/simonw/datasette/issues/1308
node_id: MDEyOklzc3VlQ29tbWVudDgyNjA0MDY3Ng==
created_at: 2021-04-24T05:58:35Z
updated_at: 2021-04-24T05:58:35Z
author_association: OWNER
body:
  Here's why: https://github.com/simonw/datasette/blob/6ed9238178a56da5fb019f37fb1e1e15886be1d1/datasette/static/app.css#L435-L437

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1308#issuecomment-826040909
issue_url: https://api.github.com/repos/simonw/datasette/issues/1308
node_id: MDEyOklzc3VlQ29tbWVudDgyNjA0MDkwOQ==
created_at: 2021-04-24T06:01:21Z
updated_at: 2021-04-24T06:01:21Z
author_association: OWNER
body:
  Demo:
  ```
  echo '{"link": "https://example.com/"}' | sqlite-utils insert link.db link -
  ```
  <img width="467" alt="link__link__1_row" src="https://user-images.githubusercontent.com/9599/115949159-cfdf7300-a487-11eb-84d1-437d954d1841.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1308#issuecomment-826041458
issue_url: https://api.github.com/repos/simonw/datasette/issues/1308
node_id: MDEyOklzc3VlQ29tbWVudDgyNjA0MTQ1OA==
created_at: 2021-04-24T06:07:01Z
updated_at: 2021-04-24T06:07:01Z
author_association: OWNER
body:
  I can use `td.type-pk` instead - here's the existing HTML:
  ```html
  <td class="col-Link type-pk"><a href="/link/link/1">1</a></td>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/173#issuecomment-826784306
issue_url: https://api.github.com/repos/simonw/datasette/issues/173
node_id: MDEyOklzc3VlQ29tbWVudDgyNjc4NDMwNg==
created_at: 2021-04-26T12:10:01Z
updated_at: 2021-04-26T12:10:01Z
author_association: NONE
body:
  I found a neat tutorial to set up gettext with jinja2: http://siongui.github.io/2016/01/17/i18n-python-web-application-by-gettext-jinja2/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1309#issuecomment-827911909
issue_url: https://api.github.com/repos/simonw/datasette/issues/1309
node_id: MDEyOklzc3VlQ29tbWVudDgyNzkxMTkwOQ==
created_at: 2021-04-27T20:35:15Z
updated_at: 2021-04-27T20:35:15Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1309?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1309](https://codecov.io/gh/simonw/datasette/pull/1309?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (20fc3fe) into [main](https://codecov.io/gh/simonw/datasette/commit/a4bb2abce0764d49d255e5379f9e9c70981834ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a4bb2ab) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1309/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1309?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1309   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1309?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1309?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a4bb2ab...20fc3fe](https://codecov.io/gh/simonw/datasette/pull/1309?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1310#issuecomment-828670621
issue_url: https://api.github.com/repos/simonw/datasette/issues/1310
node_id: MDEyOklzc3VlQ29tbWVudDgyODY3MDYyMQ==
created_at: 2021-04-28T18:12:08Z
updated_at: 2021-04-28T18:12:08Z
author_association: NONE
body:
  Apparently, beside a string, Reponse could also [work with bytes](https://github.com/simonw/datasette/blob/master/datasette/utils/asgi.py#L338).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1311#issuecomment-828683322
issue_url: https://api.github.com/repos/simonw/datasette/issues/1311
node_id: MDEyOklzc3VlQ29tbWVudDgyODY4MzMyMg==
created_at: 2021-04-28T18:30:49Z
updated_at: 2021-04-28T18:30:49Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1311?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1311](https://codecov.io/gh/simonw/datasette/pull/1311?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (baf3030) into [main](https://codecov.io/gh/simonw/datasette/commit/a4bb2abce0764d49d255e5379f9e9c70981834ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a4bb2ab) will **increase** coverage by `0.07%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1311/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1311?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1311      +/-   ##
  ==========================================
  + Coverage   91.51%   91.58%   +0.07%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3897       +3     
  + Misses        361      358       -3     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1311?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1311/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `93.68% <0.00%> (+0.74%)` | :arrow_up: |
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1311/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `98.18% <0.00%> (+1.81%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1311?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1311?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a4bb2ab...baf3030](https://codecov.io/gh/simonw/datasette/pull/1311?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1313#issuecomment-829265979
issue_url: https://api.github.com/repos/simonw/datasette/issues/1313
node_id: MDEyOklzc3VlQ29tbWVudDgyOTI2NTk3OQ==
created_at: 2021-04-29T14:04:13Z
updated_at: 2021-04-29T14:04:13Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1313?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1313](https://codecov.io/gh/simonw/datasette/pull/1313?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (3cd7ad4) into [main](https://codecov.io/gh/simonw/datasette/commit/a4bb2abce0764d49d255e5379f9e9c70981834ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a4bb2ab) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1313/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1313?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1313   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1313?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1313?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a4bb2ab...3cd7ad4](https://codecov.io/gh/simonw/datasette/pull/1313?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1314#issuecomment-829349118
issue_url: https://api.github.com/repos/simonw/datasette/issues/1314
node_id: MDEyOklzc3VlQ29tbWVudDgyOTM0OTExOA==
created_at: 2021-04-29T15:43:32Z
updated_at: 2021-04-29T15:43:32Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1314?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1314](https://codecov.io/gh/simonw/datasette/pull/1314?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (98eea0b) into [main](https://codecov.io/gh/simonw/datasette/commit/a4bb2abce0764d49d255e5379f9e9c70981834ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a4bb2ab) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1314/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1314?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1314   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1314?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1314?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a4bb2ab...98eea0b](https://codecov.io/gh/simonw/datasette/pull/1314?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1313#issuecomment-829352402
issue_url: https://api.github.com/repos/simonw/datasette/issues/1313
node_id: MDEyOklzc3VlQ29tbWVudDgyOTM1MjQwMg==
created_at: 2021-04-29T15:47:23Z
updated_at: 2021-04-29T15:47:23Z
author_association: CONTRIBUTOR
body:
  This pull request will no longer be automatically closed when a new version is found as this pull request was created by Dependabot Preview and this repo is using a `version: 2` config file. You can close this pull request and let Dependabot re-create it the next time it checks for updates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1310#issuecomment-829885904
issue_url: https://api.github.com/repos/simonw/datasette/issues/1310
node_id: MDEyOklzc3VlQ29tbWVudDgyOTg4NTkwNA==
created_at: 2021-04-30T06:58:46Z
updated_at: 2021-04-30T07:26:11Z
author_association: NONE
body:
  I made it work with openpyxl. I'm not sure all the code under `@hookimpl` is necessary... but it works :)

  ```python
  from datasette import hookimpl
  from datasette.utils.asgi import Response
  from openpyxl import Workbook
  from openpyxl.writer.excel import save_virtual_workbook
  from openpyxl.cell import WriteOnlyCell
  from openpyxl.styles import Alignment, Font, PatternFill
  from tempfile import NamedTemporaryFile

  def render_spreadsheet(rows):
      wb = Workbook(write_only=True)
      ws = wb.create_sheet()
      ws = wb.active
      ws.title = "decp"

      columns = rows[0].keys()
      headers = []
      for col in columns :
          c = WriteOnlyCell(ws, col)
          c.fill = PatternFill("solid", fgColor="DDEFFF")
          headers.append(c)
      ws.append(headers)

      for row in rows:
          wsRow = []
          for col in columns:
              c = WriteOnlyCell(ws, row[col])
              if col == "objet" :
                  c.alignment = Alignment(wrapText = True)
              wsRow.append(c)
          ws.append(wsRow)

      with NamedTemporaryFile() as tmp:
          wb.save(tmp.name)
          tmp.seek(0)
          return Response(
              tmp.read(),
              headers={
                  'Content-Disposition': 'attachment; filename=decp.xlsx',
                  'Content-type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
              }
          )

  @hookimpl
  def register_output_renderer():
      return {"extension": "xlsx",
      "render": render_spreadsheet,
      "can_render": lambda: False}

  ```

  The key part was to find the right function to wrap the spreadsheet object `wb`. `NamedTemporaryFile()` did it!

  I'll update this issue when the plugin is packaged and ready for broader use.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-831004775
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDgzMTAwNDc3NQ==
created_at: 2021-05-03T03:46:23Z
updated_at: 2021-05-03T03:46:23Z
author_association: NONE
body:
  RS1800955 is related to novelty seeking and ADHD
  https://www.snpedia.com/index.php/Rs1800955

  `select rsid, genotype, case genotype
    when 'CC' then 'increased susceptibility to novelty seeking'
    when 'CT' then 'increased susceptibility to novelty seeking'
    when 'TT' then 'normal'
  end as interpretation from genome where rsid = 'rs1800955'`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1318#issuecomment-832676649
issue_url: https://api.github.com/repos/simonw/datasette/issues/1318
node_id: MDEyOklzc3VlQ29tbWVudDgzMjY3NjY0OQ==
created_at: 2021-05-05T13:13:45Z
updated_at: 2021-05-05T13:13:45Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1318?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1318](https://codecov.io/gh/simonw/datasette/pull/1318?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (e06c099) into [main](https://codecov.io/gh/simonw/datasette/commit/1b697539f5b53cec3fe13c0f4ada13ba655c88c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b69753) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1318/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1318?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1318      +/-   ##
  ==========================================
  + Coverage   91.51%   91.53%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3895       +1     
  + Misses        361      360       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1318?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1318/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.31% <0.00%> (+0.17%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1318?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1318?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1b69753...e06c099](https://codecov.io/gh/simonw/datasette/pull/1318?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1300#issuecomment-833132571
issue_url: https://api.github.com/repos/simonw/datasette/issues/1300
node_id: MDEyOklzc3VlQ29tbWVudDgzMzEzMjU3MQ==
created_at: 2021-05-06T00:16:50Z
updated_at: 2021-05-06T00:18:05Z
author_association: CONTRIBUTOR
body:
  I ended up using some JS as a workaround. 

  First, add a JS file in `metadata.yaml`:

  ```yaml
  extra_js_urls:
    - '/static/app.js'
  ```
  then inside the script, find the blob download links and replace `.blob` extension in the url with `.jpg` and replace the links with `<img/>` elements. 
  You need to add an output formatter to serve `BLOB` columns as JPG. You can find the code in the first post.
  ~~Replacing `.blob` -> `.jpg` might not even be necessary, because browsers only care about the mime type, so you only need to serve the binary content with the right `content-type` header.~~. You need to replace the extension, otherwise the output renderer will not run.

  ```js
  window.addEventListener('DOMContentLoaded', () => {
      function renderBlobImages() {
          document.querySelectorAll('a[href*=".blob"]').forEach(el => {
              const img = document.createElement('img');
              img.className = 'blob-image';
              img.loading = 'lazy';
              img.src = el.href.replace('.blob', '.jpg');
              el.parentElement.replaceChild(img, el);
          });
      }

      renderBlobImages();
  });
  ```

  while this does the job, I'd prefer handling this in Python where it belongs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-834636796
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDgzNDYzNjc5Ng==
created_at: 2021-05-07T17:22:52Z
updated_at: 2021-05-07T17:22:52Z
author_association: OWNER
body:
  Related: Here's an implementation of a `get_metadata()` plugin hook by @brandonrobertz https://github.com/next-LI/datasette/commit/3fd8ce91f3108c82227bf65ff033929426c60437

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1296#issuecomment-835491318
issue_url: https://api.github.com/repos/simonw/datasette/issues/1296
node_id: MDEyOklzc3VlQ29tbWVudDgzNTQ5MTMxOA==
created_at: 2021-05-08T19:59:01Z
updated_at: 2021-05-08T19:59:01Z
author_association: CONTRIBUTOR
body:
  I have also found that ubuntu has fewer vulnerabilities than the buster based images.

  ```
  ➜  ~ docker pull python:3-buster
  ➜  ~ trivy image python:3-buster | head                             
  2021-04-28T17:14:29.313-0400    INFO    Detecting Debian vulnerabilities...
  2021-04-28T17:14:29.393-0400    INFO    Trivy skips scanning programming language libraries because no supported file was detected
  python:3-buster (debian 10.9)
  =============================
  Total: 1621 (UNKNOWN: 13, LOW: 1106, MEDIUM: 343, HIGH: 145, CRITICAL: 14)
  +------------------------------+---------------------+----------+------------------------------+---------------+--------------------------------------------------------------+
  |           LIBRARY            |  VULNERABILITY ID   | SEVERITY |      INSTALLED VERSION       | FIXED VERSION |                            TITLE                             |
  +------------------------------+---------------------+----------+------------------------------+---------------+--------------------------------------------------------------+
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1280#issuecomment-837166862
issue_url: https://api.github.com/repos/simonw/datasette/issues/1280
node_id: MDEyOklzc3VlQ29tbWVudDgzNzE2Njg2Mg==
created_at: 2021-05-10T19:07:46Z
updated_at: 2021-05-10T19:07:46Z
author_association: CONTRIBUTOR
body:
  Do you have a list of sqlite versions you want to test against?

  One cool thing I saw recently (that we started using) was using `import docker` within python, and then writing pytest functions which executed against the container

  [setup](https://github.com/StatCan/kubeflow-containers/blob/3c7dcfb5e7188982fb8ebcded82e84292720f720/conftest.py#L85)

  [example](https://github.com/StatCan/kubeflow-containers/blob/master/tests/jupyterlab-cpu/test_julia.py#L8-L18)

  The inspiration for this came from the [jupyter docker-stacks](https://github.com/jupyter/docker-stacks/blob/09fb66007615ea68d9bce8f8e1a2cf9402f1e432/test/test_packages.py#L107)

  So off the top of my head, could look at building the container with different sqlite versions as a build-arg, then run tests against the containers. Just brainstorming though

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1321#issuecomment-838458014
issue_url: https://api.github.com/repos/simonw/datasette/issues/1321
node_id: MDEyOklzc3VlQ29tbWVudDgzODQ1ODAxNA==
created_at: 2021-05-11T13:18:12Z
updated_at: 2021-05-11T13:18:12Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1321?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1321](https://codecov.io/gh/simonw/datasette/pull/1321?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (966ced8) into [main](https://codecov.io/gh/simonw/datasette/commit/1b697539f5b53cec3fe13c0f4ada13ba655c88c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b69753) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1321/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1321      +/-   ##
  ==========================================
  + Coverage   91.51%   91.53%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3895       +1     
  + Misses        361      360       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1321?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1321/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.31% <0.00%> (+0.17%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1321?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1321?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1b69753...966ced8](https://codecov.io/gh/simonw/datasette/pull/1321?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1325#issuecomment-839763094
issue_url: https://api.github.com/repos/simonw/datasette/issues/1325
node_id: MDEyOklzc3VlQ29tbWVudDgzOTc2MzA5NA==
created_at: 2021-05-12T13:14:29Z
updated_at: 2021-05-12T13:14:29Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1325?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1325](https://codecov.io/gh/simonw/datasette/pull/1325?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (4dea180) into [main](https://codecov.io/gh/simonw/datasette/commit/1b697539f5b53cec3fe13c0f4ada13ba655c88c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b69753) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1325/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1325      +/-   ##
  ==========================================
  + Coverage   91.51%   91.53%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3895       +1     
  + Misses        361      360       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1325?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1325/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.31% <0.00%> (+0.17%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1325?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1325?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1b69753...4dea180](https://codecov.io/gh/simonw/datasette/pull/1325?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1324#issuecomment-839763280
issue_url: https://api.github.com/repos/simonw/datasette/issues/1324
node_id: MDEyOklzc3VlQ29tbWVudDgzOTc2MzI4MA==
created_at: 2021-05-12T13:14:44Z
updated_at: 2021-05-12T13:14:44Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1324?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1324](https://codecov.io/gh/simonw/datasette/pull/1324?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (177456e) into [main](https://codecov.io/gh/simonw/datasette/commit/1b697539f5b53cec3fe13c0f4ada13ba655c88c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b69753) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1324/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1324?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1324      +/-   ##
  ==========================================
  + Coverage   91.51%   91.53%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3895       +1     
  + Misses        361      360       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1324?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1324/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.31% <0.00%> (+0.17%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1324?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1324?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1b69753...177456e](https://codecov.io/gh/simonw/datasette/pull/1324?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1323#issuecomment-839763331
issue_url: https://api.github.com/repos/simonw/datasette/issues/1323
node_id: MDEyOklzc3VlQ29tbWVudDgzOTc2MzMzMQ==
created_at: 2021-05-12T13:14:50Z
updated_at: 2021-05-17T17:25:17Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1323?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1323](https://codecov.io/gh/simonw/datasette/pull/1323?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cd69502) into [main](https://codecov.io/gh/simonw/datasette/commit/5e60bad40460f68122006ce704cfc163d6076f34?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (5e60bad) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1323/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1323      +/-   ##
  ==========================================
  + Coverage   91.51%   91.53%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3895       +1     
  + Misses        361      360       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1323?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1323/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.31% <0.00%> (+0.17%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1323?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1323?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9b3b7e2...cd69502](https://codecov.io/gh/simonw/datasette/pull/1323?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1330#issuecomment-842312696
issue_url: https://api.github.com/repos/simonw/datasette/issues/1330
node_id: MDEyOklzc3VlQ29tbWVudDg0MjMxMjY5Ng==
created_at: 2021-05-17T13:11:54Z
updated_at: 2021-05-17T13:11:54Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1330?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1330](https://codecov.io/gh/simonw/datasette/pull/1330?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cc1b504) into [main](https://codecov.io/gh/simonw/datasette/commit/1b697539f5b53cec3fe13c0f4ada13ba655c88c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b69753) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1330/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1330?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1330      +/-   ##
  ==========================================
  + Coverage   91.51%   91.53%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3895       +1     
  + Misses        361      360       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1330?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1330/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.31% <0.00%> (+0.17%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1330?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1330?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1b69753...cc1b504](https://codecov.io/gh/simonw/datasette/pull/1330?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1331#issuecomment-842497125
issue_url: https://api.github.com/repos/simonw/datasette/issues/1331
node_id: MDEyOklzc3VlQ29tbWVudDg0MjQ5NzEyNQ==
created_at: 2021-05-17T17:20:15Z
updated_at: 2021-05-17T17:20:15Z
author_association: OWNER
body:
  I wonder if there are any new 3.0 features we should be taking advantage of here that would justify pinning to 3.0 minimum?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1331#issuecomment-842499728
issue_url: https://api.github.com/repos/simonw/datasette/issues/1331
node_id: MDEyOklzc3VlQ29tbWVudDg0MjQ5OTcyOA==
created_at: 2021-05-17T17:24:30Z
updated_at: 2021-05-17T17:24:30Z
author_association: NONE
body:
  > I wonder if there are any new 3.0 features we should be taking advantage of here that would justify pinning to 3.0 minimum?

  The changelog reads like bug fixes and removal of deprecated parts to me

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1236#issuecomment-842798043
issue_url: https://api.github.com/repos/simonw/datasette/issues/1236
node_id: MDEyOklzc3VlQ29tbWVudDg0Mjc5ODA0Mw==
created_at: 2021-05-18T03:28:25Z
updated_at: 2021-05-18T03:28:25Z
author_association: CONTRIBUTOR
body:
  That corner handle looks like a hamburger menu to me. Note that the default resize handle is not limited to two-way resize: http://jsfiddle.net/LLrh7Lte/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/856#issuecomment-843065142
issue_url: https://api.github.com/repos/simonw/datasette/issues/856
node_id: MDEyOklzc3VlQ29tbWVudDg0MzA2NTE0Mg==
created_at: 2021-05-18T10:49:11Z
updated_at: 2021-05-18T10:49:29Z
author_association: NONE
body:
  Hi Simon, I'm using a canned query to do some geospatial stuff, but it maxes out at 1000 rows returned. I can't see any `Link` headers to follow to get the next page of data. Is there any way currently to work around this 1000 row limit for canned queries?

  Thanks,
  Jon

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/856#issuecomment-843291675
issue_url: https://api.github.com/repos/simonw/datasette/issues/856
node_id: MDEyOklzc3VlQ29tbWVudDg0MzI5MTY3NQ==
created_at: 2021-05-18T15:56:45Z
updated_at: 2021-05-18T15:56:45Z
author_association: OWNER
body:
  Tables and views get "stream all rows" at the moment, so one workaround is to define a SQL view for your query - this only works for queries that don't take any parameters though (although you may be able to define a view and then pass it extra fields using the Datasette table interface, like on https://latest.datasette.io/fixtures/paginated_view?content_extra__contains=9)

  I've explored this problem in a bit more detail in https://githu.com/simonw/django-sql-dashboard and I think I have a pattern that could work.

  For your canned query, you could implement the pattern yourself by setting up two canned queries that look something like this:

  https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+order+by+rowid+limit+1000

  ```sql
  select rowid, sha, author_date from commits order by rowid limit 1000
  ```
  That gets you the first set of 1,000 results. The important thing here is to order by a unique column, in this case `rowid` - because then subsequent pages can be loaded by a separate canned query that looks like this:
  ```sql
  select rowid, sha, author_date from commits where rowid > :after order by rowid limit 1000
  ```
  https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+sha%2C+author_date+from+commits+where+rowid+%3E+%3Aafter+order+by+rowid+limit+1000&after=1000

  You then need to write code which knows how to generate these queries - start with the first query with no `where` clause (or if you are using `rowid` you can just use the second query and pass it `?after=0` for the first call) - then keep calling the query passing in the last rowid you recieved as the `after` parameter.

  Basically this is an implementation of keyset pagination with a smart client. When Datasette grows the ability to do this itself it will work by executing this mechanism inside the Python code, which is how the "stream all rows" option for tables works at the moment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/256#issuecomment-843708246
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/256
node_id: MDEyOklzc3VlQ29tbWVudDg0MzcwODI0Ng==
created_at: 2021-05-19T03:06:02Z
updated_at: 2021-05-19T03:06:02Z
author_association: OWNER
body:
  Did you try running this with the `--alter` command-line option? That seems to work when I try it on a sample file here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/256#issuecomment-843715308
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/256
node_id: MDEyOklzc3VlQ29tbWVudDg0MzcxNTMwOA==
created_at: 2021-05-19T03:26:54Z
updated_at: 2021-05-19T03:26:54Z
author_association: OWNER
body:
  I decided to treat this as a usability bug: there's a workaround, which is adding `--alter` - but it's not at all obvious that you should do that. I addressed this in #259.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/259#issuecomment-843715502
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/259
node_id: MDEyOklzc3VlQ29tbWVudDg0MzcxNTUwMg==
created_at: 2021-05-19T03:27:33Z
updated_at: 2021-05-19T03:27:33Z
author_association: OWNER
body:
  ```
   % sqlite-utils insert test-big.db text big.json 
  Error: table text has no column named colun

  Try using --alter to add additional columns
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/253#issuecomment-843716042
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/253
node_id: MDEyOklzc3VlQ29tbWVudDg0MzcxNjA0Mg==
created_at: 2021-05-19T03:29:04Z
updated_at: 2021-05-19T03:29:04Z
author_association: OWNER
body:
  Thanks for pointing this out - the problem here is that these days the `https://latest.datasette.io/fixtures.db` example database includes a test for generated columns, which are a brand new feature in SQLite. Your local SQLite is an older version.

  I'm going to update the blog post to instead use a copy of `fixtures.db` that doesn't use that feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1335#issuecomment-844007155
issue_url: https://api.github.com/repos/simonw/datasette/issues/1335
node_id: MDEyOklzc3VlQ29tbWVudDg0NDAwNzE1NQ==
created_at: 2021-05-19T11:21:45Z
updated_at: 2021-05-19T11:21:45Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1335?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1335](https://codecov.io/gh/simonw/datasette/pull/1335?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (d91c18d) into [main](https://codecov.io/gh/simonw/datasette/commit/9b3b7e280ca718254b4ca15d40864297146a85b3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (9b3b7e2) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1335/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1335?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1335   +/-   ##
  =======================================
    Coverage   91.53%   91.53%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3895     3895           
    Misses        360      360           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1335?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1335?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [9b3b7e2...d91c18d](https://codecov.io/gh/simonw/datasette/pull/1335?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1331#issuecomment-844970776
issue_url: https://api.github.com/repos/simonw/datasette/issues/1331
node_id: MDEyOklzc3VlQ29tbWVudDg0NDk3MDc3Ng==
created_at: 2021-05-20T10:40:25Z
updated_at: 2021-05-20T10:40:25Z
author_association: NONE
body:
  Any chance you could push a new datasette release with the updated dependencies in the setup.py, @simonw?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1332#issuecomment-846479062
issue_url: https://api.github.com/repos/simonw/datasette/issues/1332
node_id: MDEyOklzc3VlQ29tbWVudDg0NjQ3OTA2Mg==
created_at: 2021-05-23T00:06:34Z
updated_at: 2021-05-23T00:06:34Z
author_association: OWNER
body:
  A URL parameter to modify that facet size is a really good idea.

  I thought I had an issue open for "..." linking to more results but I can't find it now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1332#issuecomment-846479212
issue_url: https://api.github.com/repos/simonw/datasette/issues/1332
node_id: MDEyOklzc3VlQ29tbWVudDg0NjQ3OTIxMg==
created_at: 2021-05-23T00:07:46Z
updated_at: 2021-05-23T00:07:46Z
author_association: OWNER
body:
  `?_facet_size=50` seems like a very reasonable thing to support - I'll cap it at whatever the `max_returned_rows` setting is (which defaults to 1000).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1332#issuecomment-846479730
issue_url: https://api.github.com/repos/simonw/datasette/issues/1332
node_id: MDEyOklzc3VlQ29tbWVudDg0NjQ3OTczMA==
created_at: 2021-05-23T00:11:31Z
updated_at: 2021-05-23T00:11:31Z
author_association: OWNER
body:
  Here's the code in question: https://github.com/simonw/datasette/blob/5e9672c9bb33e41686472db4aa427168f9e67dbe/datasette/facets.py#L187-L210

  It has access to the current request object as `self.request`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1332#issuecomment-846481801
issue_url: https://api.github.com/repos/simonw/datasette/issues/1332
node_id: MDEyOklzc3VlQ29tbWVudDg0NjQ4MTgwMQ==
created_at: 2021-05-23T00:36:20Z
updated_at: 2021-05-23T00:36:20Z
author_association: OWNER
body:
  Documentation:

  - https://docs.datasette.io/en/latest/facets.html#facets-in-query-strings at the bottom of the section
  - https://docs.datasette.io/en/latest/json_api.html#special-table-arguments at the bottom of the list of arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1337#issuecomment-846481896
issue_url: https://api.github.com/repos/simonw/datasette/issues/1337
node_id: MDEyOklzc3VlQ29tbWVudDg0NjQ4MTg5Ng==
created_at: 2021-05-23T00:37:52Z
updated_at: 2021-05-23T00:37:52Z
author_association: OWNER
body:
  The tricky thing about this is that ideally I'd like this to link to a paginated set of results - but arbitrary SQL queries do not yet support pagination, see #856.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1331#issuecomment-846482057
issue_url: https://api.github.com/repos/simonw/datasette/issues/1331
node_id: MDEyOklzc3VlQ29tbWVudDg0NjQ4MjA1Nw==
created_at: 2021-05-23T00:39:55Z
updated_at: 2021-05-23T00:39:55Z
author_association: OWNER
body:
  > I'm stuck also because datasette wants itsdangerous~=1.1 instead of allowing itsdangerous-2.0.0

  Bumped that dependency in b64d87204612a84663616e075f542499a5d82a03

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1326#issuecomment-846493370
issue_url: https://api.github.com/repos/simonw/datasette/issues/1326
node_id: MDEyOklzc3VlQ29tbWVudDg0NjQ5MzM3MA==
created_at: 2021-05-23T02:55:00Z
updated_at: 2021-05-23T02:55:00Z
author_association: OWNER
body:
  Currently you can do this with a custom SQL query, but as you've noticed this means you don't get automatic pagination: https://latest.datasette.io/fixtures.json?sql=select+pk%2C+created%2C+state+from+facetable+order+by+pk+limit+10&_shape=array

  I have an open issue to allow selecting specific columns on the table page: #615

  A workaround you can use today is to define a SQL view that selects just the tables you want. View pages behave exactly like tables in that they provide pagination by default.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846592392
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjU5MjM5Mg==
created_at: 2021-05-23T16:49:30Z
updated_at: 2021-05-23T16:49:44Z
author_association: OWNER
body:
  I started looking at this again, inspired by #1326. I have a new diff that works against the latest `main` branch.

  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 4879228..f4b2ee2 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -64,6 +64,36 @@ class Row:
   
   
   class RowTableShared(DataView):
  +    async def columns_to_select(self, db, table, request):
  +        table_columns = await db.table_columns(table)
  +        if "_col" in request.args and "_nocol" in request.args:
  +            raise DatasetteError("Cannot use _col and _nocol at the same time")
  +        if "_col" in request.args:
  +            new_columns = []
  +            for column in request.args.getlist("_col"):
  +                if column not in table_columns:
  +                    raise DatasetteError("_col={} is an invalid column".format(column))
  +                new_columns.append(column)
  +            return new_columns
  +        elif "_nocol" in request.args:
  +            # Return all columns EXCEPT these
  +            bad_columns = [
  +                column
  +                for column in request.args.getlist("_nocol")
  +                if column not in table_columns
  +            ]
  +            if bad_columns:
  +                raise DatasetteError(
  +                    "_nocol={} - invalid columns".format(", ".join(bad_columns))
  +                )
  +            return [
  +                column
  +                for column in table_columns
  +                if column not in request.args.getlist("_nocol")
  +            ]
  +        else:
  +            return table_columns
  +
       async def sortable_columns_for_table(self, database, table, use_rowid):
           db = self.ds.databases[database]
           table_metadata = self.ds.table_metadata(database, table)
  @@ -321,18 +351,16 @@ class TableView(RowTableShared):
           )
   
           pks = await db.primary_keys(table)
  -        table_column_details = await db.table_column_details(table)
  -        table_columns = [column.name for column in table_column_details]
  -
  -        select_columns = ", ".join(escape_sqlite(t) for t in table_columns)
  +        table_columns = await self.columns_to_select(db, table, request)
  +        select_clause = ", ".join(escape_sqlite(t) for t in table_columns)
   
           use_rowid = not pks and not is_view
           if use_rowid:
  -            select = f"rowid, {select_columns}"
  +            select = f"rowid, {select_clause}"
               order_by = "rowid"
               order_by_pks = "rowid"
           else:
  -            select = select_columns
  +            select = select_clause
               order_by_pks = ", ".join([escape_sqlite(pk) for pk in pks])
               order_by = order_by_pks
   
  @@ -715,6 +743,8 @@ class TableView(RowTableShared):
                   column = fk["column"]
                   if column not in columns_to_expand:
                       continue
  +                if column not in columns:
  +                    continue
                   expanded_columns.append(column)
                   # Gather the values
                   column_index = columns.index(column)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846592583
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjU5MjU4Mw==
created_at: 2021-05-23T16:50:45Z
updated_at: 2021-05-24T17:15:48Z
author_association: OWNER
body:
  Still needed:

  - Unit tests
  - Documentation

  And maybe a UI mechanism for this? I could at least add a "remove this column" item to the cog menu on columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846595091
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjU5NTA5MQ==
created_at: 2021-05-23T17:09:59Z
updated_at: 2021-05-23T17:09:59Z
author_association: OWNER
body:
  Here's that prototype of a "Hide this column" cog menu button:

  ![hide](https://user-images.githubusercontent.com/9599/119269916-effd7180-bbae-11eb-92fa-0ef374bda4b8.gif)

  Need a way to undo that once you've hidden a column - maybe a list of currently hidden columns that lets  you un-hide them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846595473
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjU5NTQ3Mw==
created_at: 2021-05-23T17:13:09Z
updated_at: 2021-05-23T17:13:09Z
author_association: OWNER
body:
  Natalie suggests a quick way to implement "undo" would be to add a "Show all columns" item to that menu which only appears when at least one column is hidden.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846599732
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjU5OTczMg==
created_at: 2021-05-23T17:46:45Z
updated_at: 2021-05-23T17:46:45Z
author_association: OWNER
body:
  I've changed my mind about forbidding `?_col=` and `?_nocol=` from being applied at the same time - I'm going to come up with a simple resolution rule instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846626567
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjYyNjU2Nw==
created_at: 2021-05-23T21:25:19Z
updated_at: 2021-05-23T21:25:19Z
author_association: OWNER
body:
  Here's a bug: removing the `rowid` column returns an error.

  ![rowid-bug](https://user-images.githubusercontent.com/9599/119277132-b179ae00-bbd2-11eb-8f2f-e228c7d7c264.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846626871
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjYyNjg3MQ==
created_at: 2021-05-23T21:27:36Z
updated_at: 2021-05-23T21:27:36Z
author_association: OWNER
body:
  A better interface for this would be a full list of columns each with a checkbox for making it visible on invisible - this could then be used to apply a bulk change (rather than refreshing the interface after every removed column) and it could also be easily designed to work on narrow mobile screens where the cog icon is not visible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846627163
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjYyNzE2Mw==
created_at: 2021-05-23T21:30:01Z
updated_at: 2021-05-23T21:30:01Z
author_association: OWNER
body:
  Interesting side-effect of this implementation is that you can both control column order and request the same column multiple times:

  `/fivethirtyeight/airline-safety%2Fairline-safety?_col=fatal_accidents_00_14&_col=fatalities_00_14&_col=airline&_col=airline`

  <img width="930" alt="fivethirtyeight__airline-safety_airline-safety__56_rows" src="https://user-images.githubusercontent.com/9599/119277243-58f6e080-bbd3-11eb-98be-69787cd9192d.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846627322
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjYyNzMyMg==
created_at: 2021-05-23T21:31:22Z
updated_at: 2021-05-23T21:31:22Z
author_association: OWNER
body:
  Would it be useful to allow this mechanism to alias columns, for example supporting one of the following:

  - `?_col=airline as name_of_airline`
  - `?_col=airline:name_of_airline`

  This could be handy for renaming columns to match a specific expected JSON output.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846659272
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjY1OTI3Mg==
created_at: 2021-05-24T00:44:47Z
updated_at: 2021-05-24T00:44:47Z
author_association: OWNER
body:
  I can't think of any good reasons to support passing the same column twice `?col=airline&_col=airline` so I'm going to de-duplicate and silently ignore the second one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846660103
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjY2MDEwMw==
created_at: 2021-05-24T00:47:00Z
updated_at: 2021-05-24T00:47:00Z
author_association: OWNER
body:
  > Here's a bug: removing the `rowid` column returns an error.

  Removing the `rowid` column should work. We can continue to show the `Link` column, ensuring users can still navigate to the row page for each row.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-846668419
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0NjY2ODQxOQ==
created_at: 2021-05-24T01:14:27Z
updated_at: 2021-05-24T01:14:27Z
author_association: OWNER
body:
  I tried allowing the removal of `rowid` and got this exception:
  ```
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 831, in extra_template
      display_columns, display_rows = await self.display_columns_and_rows(
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 163, in display_columns_and_rows
      pk_path = path_from_row_pks(row, pks, not pks, False)
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 82, in path_from_row_pks
      bits = [row["rowid"]]
  IndexError: No item with that key
  ```
  I'm going to disable the removal of `rowid` - or indeed any of the primary keys, since they are needed to construct the row permalink.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1339#issuecomment-847207804
issue_url: https://api.github.com/repos/simonw/datasette/issues/1339
node_id: MDEyOklzc3VlQ29tbWVudDg0NzIwNzgwNA==
created_at: 2021-05-24T17:22:13Z
updated_at: 2021-05-27T04:14:21Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1339?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1339](https://codecov.io/gh/simonw/datasette/pull/1339?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (3d8598b) into [main](https://codecov.io/gh/simonw/datasette/commit/a443dba82f43c22b03402a4f86c85558ccb526b8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a443dba) will **increase** coverage by `0.01%`.
  > The diff coverage is `91.66%`.

  > :exclamation: Current head 3d8598b differs from pull request most recent head 387c837. Consider uploading reports for the commit 387c837 to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1339/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1339?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1339      +/-   ##
  ==========================================
  + Coverage   91.55%   91.56%   +0.01%     
  ==========================================
    Files          34       34              
    Lines        4262     4280      +18     
  ==========================================
  + Hits         3902     3919      +17     
  - Misses        360      361       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1339?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1339/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.83% <90.00%> (-0.05%)` | :arrow_down: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1339/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.84% <100.00%> (-0.01%)` | :arrow_down: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1339/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.19% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1339?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1339?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a443dba...387c837](https://codecov.io/gh/simonw/datasette/pull/1339?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1320#issuecomment-847230866
issue_url: https://api.github.com/repos/simonw/datasette/issues/1320
node_id: MDEyOklzc3VlQ29tbWVudDg0NzIzMDg2Ng==
created_at: 2021-05-24T18:03:29Z
updated_at: 2021-05-24T18:03:29Z
author_association: OWNER
body:
  ```
  root@33c269900821:/# du -h /var/lib/dpkg
  6.3M	/var/lib/dpkg/info
  24K	/var/lib/dpkg/triggers
  4.0K	/var/lib/dpkg/updates
  20K	/var/lib/dpkg/alternatives
  4.0K	/var/lib/dpkg/parts
  6.7M	/var/lib/dpkg
  ```
  So `/var/lib/dpkg/info` is most of the space, I'll try just deleting that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1320#issuecomment-847232547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1320
node_id: MDEyOklzc3VlQ29tbWVudDg0NzIzMjU0Nw==
created_at: 2021-05-24T18:06:32Z
updated_at: 2021-05-24T18:06:32Z
author_association: OWNER
body:
  I'm testing tweaks to the Dockerfile by editing it and running:
  ```
  docker build . -t datasette-057a0 --build-arg VERSION=0.57a0
  ```
  Then running this to inspect the new image:
  ```
  docker run -it -p 8001:8001 datasette-057a0 bash
  ```
  After changing that line to `rm -rf /var/lib/dpkg/info/*` I can run `apt-get` successfully from inside that bash prompt:
  ```
  # apt-get update
  # apt-get install ripgrep
  # rg --version
  ripgrep 12.1.1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1320#issuecomment-847237524
issue_url: https://api.github.com/repos/simonw/datasette/issues/1320
node_id: MDEyOklzc3VlQ29tbWVudDg0NzIzNzUyNA==
created_at: 2021-05-24T18:15:56Z
updated_at: 2021-05-24T18:15:56Z
author_association: OWNER
body:
  Added some new documentation about that here: https://github.com/simonw/datasette/blob/c0a748e5c3f498fa8c139b420d07dd3dea612379/docs/installation.rst#installing-plugins

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1327#issuecomment-847240470
issue_url: https://api.github.com/repos/simonw/datasette/issues/1327
node_id: MDEyOklzc3VlQ29tbWVudDg0NzI0MDQ3MA==
created_at: 2021-05-24T18:21:44Z
updated_at: 2021-05-24T18:21:44Z
author_association: OWNER
body:
  I'm having trouble replicating this - I tried the following `metadata.json` in my local environment and it displayed correctly:
  ```json
  {
      "title": "မြန်မာစာ ("
  }
  ```
  What operating system are you using here? Is there a chance that your `metadata.json` file is saved in a text encoding other than utf-8?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1327#issuecomment-847271122
issue_url: https://api.github.com/repos/simonw/datasette/issues/1327
node_id: MDEyOklzc3VlQ29tbWVudDg0NzI3MTEyMg==
created_at: 2021-05-24T19:10:21Z
updated_at: 2021-05-24T19:10:21Z
author_association: NONE
body:
  wow, thanks a lot @simonw , problem is solved. I converted my current json file into utf-8 format with Python script. It's working now. I'm using with Window 10. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1340#issuecomment-847987346
issue_url: https://api.github.com/repos/simonw/datasette/issues/1340
node_id: MDEyOklzc3VlQ29tbWVudDg0Nzk4NzM0Ng==
created_at: 2021-05-25T15:49:55Z
updated_at: 2021-05-25T15:49:55Z
author_association: OWNER
body:
  Usual problem with this kind of UI: what would it do on mobile where hover isn't a thing? Plenty of other questions: if it opens a menu, what options might I put in there other than "Show more like this"?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-848425056
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDg0ODQyNTA1Ng==
created_at: 2021-05-26T03:22:32Z
updated_at: 2021-05-26T03:22:32Z
author_association: NONE
body:
  I've also been investigating serving postgresql databases over postgrest. I like the idea of hosting some static html + js on github, but having it backed by datasets I can update and control on the database server. I started from SQLite + datasette but would like to host larger datasets (with smaller materialized views exposed publicly). I think the postgrest model where all the authorization and ownership is defined in database role grants is really powerful. But I really miss being able to define an ad-hoc query in sql, then instantly link to a json representation of it like datasette does.

  P.S.: I've been sort of following along as you pop up in hacker news here and there. It's been great! Thanks for doing this all out in the open!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-849022714
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDg0OTAyMjcxNA==
created_at: 2021-05-26T18:33:47Z
updated_at: 2021-05-26T18:33:58Z
author_association: OWNER
body:
  Worth mentioning here: I've been doing a tun of research around running Datasette-like functionality against PostgreSQL in my https://github.com/simonw/django-sql-dashboard project - which will definitely inform the Datasette implementation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/292#issuecomment-849308907
issue_url: https://api.github.com/repos/simonw/datasette/issues/292
node_id: MDEyOklzc3VlQ29tbWVudDg0OTMwODkwNw==
created_at: 2021-05-27T04:25:01Z
updated_at: 2021-05-27T04:25:01Z
author_association: OWNER
body:
  Now that `?_col=` and `?_nocol=` are implemented I'm closing this ticket - other customizations can already be handled by defining SQL views or creating canned SQL queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/615#issuecomment-849309330
issue_url: https://api.github.com/repos/simonw/datasette/issues/615
node_id: MDEyOklzc3VlQ29tbWVudDg0OTMwOTMzMA==
created_at: 2021-05-27T04:26:10Z
updated_at: 2021-05-27T04:26:10Z
author_association: OWNER
body:
  Demos:

  - https://latest.datasette.io/fixtures/sortable?_nocol=sortable
  - https://latest.datasette.io/fixtures/sortable?_col=sortable&_col=text

  Also try the column cog menu on that page.

  Documentation: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1341#issuecomment-849310294
issue_url: https://api.github.com/repos/simonw/datasette/issues/1341
node_id: MDEyOklzc3VlQ29tbWVudDg0OTMxMDI5NA==
created_at: 2021-05-27T04:29:04Z
updated_at: 2021-05-27T04:29:04Z
author_association: OWNER
body:
  For implementation simplicity I'm going to show the item even if all of the columns are visible because `?_col=` has selected all of them - selecting it will still reset the view to having no `?_col=` or `?_nocol=` parameters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1319#issuecomment-849311884
issue_url: https://api.github.com/repos/simonw/datasette/issues/1319
node_id: MDEyOklzc3VlQ29tbWVudDg0OTMxMTg4NA==
created_at: 2021-05-27T04:33:51Z
updated_at: 2021-05-27T04:33:51Z
author_association: OWNER
body:
  This is so cool! I wonder how I can best test it? I have a Raspberry Pi I should dig out to try this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1319#issuecomment-849313380
issue_url: https://api.github.com/repos/simonw/datasette/issues/1319
node_id: MDEyOklzc3VlQ29tbWVudDg0OTMxMzM4MA==
created_at: 2021-05-27T04:38:44Z
updated_at: 2021-05-27T04:38:44Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1319](https://codecov.io/gh/simonw/datasette/pull/1319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cfca570) into [main](https://codecov.io/gh/simonw/datasette/commit/1b697539f5b53cec3fe13c0f4ada13ba655c88c7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b69753) will **increase** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1319/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1319      +/-   ##
  ==========================================
  + Coverage   91.51%   91.53%   +0.02%     
  ==========================================
    Files          34       34              
    Lines        4255     4255              
  ==========================================
  + Hits         3894     3895       +1     
  + Misses        361      360       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.31% <0.00%> (+0.17%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [1b69753...cfca570](https://codecov.io/gh/simonw/datasette/pull/1319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1337#issuecomment-849695117
issue_url: https://api.github.com/repos/simonw/datasette/issues/1337
node_id: MDEyOklzc3VlQ29tbWVudDg0OTY5NTExNw==
created_at: 2021-05-27T14:44:13Z
updated_at: 2021-05-27T14:44:13Z
author_association: OWNER
body:
  Here's a much easier way to implement this: implement `?_facet_size=max` (sets it to `max_returned_rows`, default 1000) and then link to that from the ellipsis.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1337#issuecomment-849721280
issue_url: https://api.github.com/repos/simonw/datasette/issues/1337
node_id: MDEyOklzc3VlQ29tbWVudDg0OTcyMTI4MA==
created_at: 2021-05-27T15:18:08Z
updated_at: 2021-05-27T15:18:08Z
author_association: OWNER
body:
  In order to cleanly implement this I need to expose the `path_with_replaced_args` utility function to Datasette's template engine. This is the first time this will become an exposed (and hence should-by-documented) API and I don't like its shape much.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1342#issuecomment-849722218
issue_url: https://api.github.com/repos/simonw/datasette/issues/1342
node_id: MDEyOklzc3VlQ29tbWVudDg0OTcyMjIxOA==
created_at: 2021-05-27T15:19:23Z
updated_at: 2021-05-27T15:19:23Z
author_association: OWNER
body:
  Current implementations: https://github.com/simonw/datasette/blob/51d788114035458d8f860d9ea6d74078e6c0ea0d/datasette/utils/__init__.py#L214-L272

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1342#issuecomment-849724108
issue_url: https://api.github.com/repos/simonw/datasette/issues/1342
node_id: MDEyOklzc3VlQ29tbWVudDg0OTcyNDEwOA==
created_at: 2021-05-27T15:21:54Z
updated_at: 2021-05-27T15:21:54Z
author_association: OWNER
body:
  Turns out some of these are already being exposed to the template context on an ad-hoc basis.

  In `QueryView`: https://github.com/simonw/datasette/blob/2bd9d54b2762c991e11950c22c88c0336158d49b/datasette/views/database.py#L295-L304

  In `TableView`: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1342#issuecomment-849724915
issue_url: https://api.github.com/repos/simonw/datasette/issues/1342
node_id: MDEyOklzc3VlQ29tbWVudDg0OTcyNDkxNQ==
created_at: 2021-05-27T15:23:01Z
updated_at: 2021-05-27T15:23:01Z
author_association: OWNER
body:
  Since they are already exposed (but undocumented) this isn't actually a blocker on #1337 like I thought it was. Should still clean this up in the future though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1337#issuecomment-849725291
issue_url: https://api.github.com/repos/simonw/datasette/issues/1337
node_id: MDEyOklzc3VlQ29tbWVudDg0OTcyNTI5MQ==
created_at: 2021-05-27T15:23:33Z
updated_at: 2021-05-27T15:23:33Z
author_association: OWNER
body:
  Turns out `path_with_replaced_args()` is already exposed to the `table.html` template context: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1337#issuecomment-849759440
issue_url: https://api.github.com/repos/simonw/datasette/issues/1337
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc1OTQ0MA==
created_at: 2021-05-27T16:09:34Z
updated_at: 2021-05-27T16:09:34Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/facetable?_facet_size=2&_facet=state&_facet=city_id

  ![demo-max](https://user-images.githubusercontent.com/9599/119860507-3f071780-becb-11eb-8698-64c8c6112983.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1337#issuecomment-849761508
issue_url: https://api.github.com/repos/simonw/datasette/issues/1337
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc2MTUwOA==
created_at: 2021-05-27T16:12:31Z
updated_at: 2021-05-27T16:12:31Z
author_association: OWNER
body:
  More exciting demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1332#issuecomment-849761894
issue_url: https://api.github.com/repos/simonw/datasette/issues/1332
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc2MTg5NA==
created_at: 2021-05-27T16:13:07Z
updated_at: 2021-05-27T16:13:07Z
author_association: OWNER
body:
  Added `?_facet_size=max` in #1337 and made the `...` truncated note link to that - here's a demo: https://fivethirtyeight.datasettes.com/fivethirtyeight/antiquities-act%2Factions_under_antiquities_act?_facet=states&_facet=current_agency&_facet=pres_or_congress

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1337#issuecomment-849762735
issue_url: https://api.github.com/repos/simonw/datasette/issues/1337
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc2MjczNQ==
created_at: 2021-05-27T16:14:14Z
updated_at: 2021-05-27T16:14:14Z
author_association: OWNER
body:
  Here's a demo showing that once you get above 1,000 results you still get the "..." but it no longer links to anything: https://fivethirtyeight.datasettes.com/fivethirtyeight/births%2FUS_births_2000-2014_SSA?_facet=births&_facet_size=max

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1319#issuecomment-849778373
issue_url: https://api.github.com/repos/simonw/datasette/issues/1319
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc3ODM3Mw==
created_at: 2021-05-27T16:37:50Z
updated_at: 2021-05-27T16:37:50Z
author_association: OWNER
body:
  Started [a conversation](https://twitter.com/simonw/status/1397953338407223303) about this on Twitter. Ideally I'd like to push an alpha release to Docker Hub so people can start testing these out before they make it into an official non-alpha release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1319#issuecomment-849780481
issue_url: https://api.github.com/repos/simonw/datasette/issues/1319
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc4MDQ4MQ==
created_at: 2021-05-27T16:41:03Z
updated_at: 2021-05-27T16:41:03Z
author_association: OWNER
body:
  It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the `latest` tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1343#issuecomment-849781747
issue_url: https://api.github.com/repos/simonw/datasette/issues/1343
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc4MTc0Nw==
created_at: 2021-05-27T16:43:04Z
updated_at: 2021-05-27T16:43:04Z
author_association: OWNER
body:
  I'm going to try pushing an alpha of https://github.com/simonw/datasette/releases/tag/0.57a0 using this workflow I built a few months ago: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/push_docker_tag.yml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1343#issuecomment-849782098
issue_url: https://api.github.com/repos/simonw/datasette/issues/1343
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc4MjA5OA==
created_at: 2021-05-27T16:43:40Z
updated_at: 2021-05-27T16:43:55Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/actions/workflows/push_docker_tag.yml

  <img width="944" alt="Actions_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/119864935-01f15400-bed0-11eb-8979-0a5cc1b6bcb4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1343#issuecomment-849783398
issue_url: https://api.github.com/repos/simonw/datasette/issues/1343
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc4MzM5OA==
created_at: 2021-05-27T16:45:41Z
updated_at: 2021-05-27T16:45:41Z
author_association: OWNER
body:
  https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated now shows this at the top: https://hub.docker.com/layers/datasetteproject/datasette/0.57a0/images/sha256-4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db?context=explore

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1343#issuecomment-849784005
issue_url: https://api.github.com/repos/simonw/datasette/issues/1343
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc4NDAwNQ==
created_at: 2021-05-27T16:46:37Z
updated_at: 2021-05-27T16:46:37Z
author_association: OWNER
body:
  That worked:
  ```
   % docker run -it datasetteproject/datasette:0.57a0
  Unable to find image 'datasetteproject/datasette:0.57a0' locally
  0.57a0: Pulling from datasetteproject/datasette
  75646c2fb410: Pull complete 
  edf8b70d3cc4: Pull complete 
  229daa3ebd94: Pull complete 
  65d21ac50a56: Pull complete 
  e083ad7cc3ca: Pull complete 
  ed4de1431432: Pull complete 
  1da5e8b27cca: Pull complete 
  Digest: sha256:4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db
  Status: Downloaded newer image for datasetteproject/datasette:0.57a0
  INFO:     Started server process [1]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1319#issuecomment-849785098
issue_url: https://api.github.com/repos/simonw/datasette/issues/1319
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc4NTA5OA==
created_at: 2021-05-27T16:48:21Z
updated_at: 2021-05-27T16:48:21Z
author_association: OWNER
body:
  OK, since I have a mechanism for deploying alpha versions now I'm going to merge this and use it to publish 0.57a1.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849788412
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc4ODQxMg==
created_at: 2021-05-27T16:53:28Z
updated_at: 2021-05-27T16:53:28Z
author_association: OWNER
body:
  (Should also update https://docs.datasette.io/en/stable/contributing.html#release-process with notes on how this works)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849793312
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc5MzMxMg==
created_at: 2021-05-27T17:01:16Z
updated_at: 2021-05-27T17:01:16Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/runs/2687196350 is a run of https://github.com/simonw/datasette/blob/main/.github/workflows/push_docker_tag.yml to publish the new `0.57a1` alpha to Docker Hub.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849797767
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTc5Nzc2Nw==
created_at: 2021-05-27T17:08:13Z
updated_at: 2021-05-27T17:08:13Z
author_association: OWNER
body:
  Related issue: #1272 - being able to run the test suite inside Docker inside GitHub Actions would be incredibly helpful here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849801969
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTgwMTk2OQ==
created_at: 2021-05-27T17:14:49Z
updated_at: 2021-05-27T17:14:49Z
author_association: OWNER
body:
  Job completed in 13m38s - results are here: https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated&name=0.57a1

  <img width="1286" alt="datasetteproject_datasette_Tags" src="https://user-images.githubusercontent.com/9599/119868791-5c8caf00-bed4-11eb-9e9f-ebca557621c0.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849804684
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTgwNDY4NA==
created_at: 2021-05-27T17:19:16Z
updated_at: 2021-05-27T17:19:16Z
author_association: OWNER
body:
  Having pushed that to Docker Hub the following _seems_ to work on my laptop:

      docker run --platform linux/s390x \
        -v `pwd`:/mnt -p 8001:8001 \
        datasetteproject/datasette:0.57a1 datasette -p 8001 -h 0.0.0.0

  I'd like to confirm that this is correctly emulating the architecture and running the correct published image though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849805778
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTgwNTc3OA==
created_at: 2021-05-27T17:21:11Z
updated_at: 2021-05-27T17:21:22Z
author_association: OWNER
body:
  This helps:
  ```
  ~ % docker run -it --platform linux/s390x -v `pwd`:/mnt \
    -p 8001:8001 datasetteproject/datasette:0.57a1 \
    python -c 'import platform; print(platform.uname())'
  ```
  `uname_result(system='Linux', node='d14916ca91df', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='s390x')`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849806581
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTgwNjU4MQ==
created_at: 2021-05-27T17:22:37Z
updated_at: 2021-05-27T17:34:27Z
author_association: OWNER
body:
  ```
  ~ % docker run -it --platform linux/arm/v7 -v `pwd`:/mnt -p 8001:8001 datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
  Unable to find image 'datasetteproject/datasette:0.57a1' locally
  0.57a1: Pulling from datasetteproject/datasette
  ee0a2cc24f29: Pull complete 
  ca231153300d: Pull complete 
  6073fc53d406: Pull complete 
  bb2a6a6421dd: Pull complete 
  14c15f441034: Pull complete 
  28d113f72ba5: Pull complete 
  40516c2c3785: Pull complete 
  Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
  Status: Downloaded newer image for datasetteproject/datasette:0.57a1
  uname_result(system='Linux', node='c2e1a05732ed', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='armv7l')
  ```
  Shorter version:
  ```
  ~ % docker run -it --platform linux/ppc64le datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
  Unable to find image 'datasetteproject/datasette:0.57a1' locally
  0.57a1: Pulling from datasetteproject/datasette
  c840eb5e9aed: Pull complete 
  fc17d8b577ec: Pull complete 
  712a0ad995f8: Pull complete 
  56101149914c: Pull complete 
  e6241dfe9d38: Pull complete 
  41ed85039888: Pull complete 
  26beeabe9213: Pull complete 
  Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
  Status: Downloaded newer image for datasetteproject/datasette:0.57a1
  uname_result(system='Linux', node='0fe85887fa30', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='ppc64le')
  ~ % 
  ```

  OK! Now I need to figure out how to run the full test suite on each one, then I can call this done.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849815361
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTgxNTM2MQ==
created_at: 2021-05-27T17:36:51Z
updated_at: 2021-05-27T17:36:51Z
author_association: OWNER
body:
  To run the test suite... need to install a few more dependencies: https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/setup.py#L71-L79

  And then mount the `tests/` folder into the container and run `pytest`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1344#issuecomment-849820019
issue_url: https://api.github.com/repos/simonw/datasette/issues/1344
node_id: MDEyOklzc3VlQ29tbWVudDg0OTgyMDAxOQ==
created_at: 2021-05-27T17:44:39Z
updated_at: 2021-05-27T17:52:58Z
author_association: OWNER
body:
  This pattern appears to work, executed at the root of a checkout of `datasette`:
  ```
  docker run -it -v `pwd`:/mnt --platform linux/amd64 \
    datasetteproject/datasette:0.57a1 bash -c '
      pip install "pytest>=5.2.2,<6.3.0" \
        "pytest-xdist>=2.2.1,<2.3" \
        "pytest-asyncio>=0.10,<0.16" \
        "beautifulsoup4>=4.8.1,<4.10.0" \
        "black==21.5b1" \
        "pytest-timeout>=1.4.2,<1.5" \
        "trustme>=0.7,<0.8" \
      && cd /mnt && pytest'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-850025106
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDg1MDAyNTEwNg==
created_at: 2021-05-28T00:18:30Z
updated_at: 2021-05-28T00:18:30Z
author_association: OWNER
body:
  I'm going to return a 400 HTTP status code for "bad request", under the assumption that the client sent bad SQL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-850057694
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDg1MDA1NzY5NA==
created_at: 2021-05-28T02:03:05Z
updated_at: 2021-05-28T02:03:05Z
author_association: OWNER
body:
  I nearly got this working, but I ran into one last problem: the code path for when an error is raised but the user specified `?_shape=array`. I'll open a draft PR with where I've got to so far.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1346#issuecomment-850058851
issue_url: https://api.github.com/repos/simonw/datasette/issues/1346
node_id: MDEyOklzc3VlQ29tbWVudDg1MDA1ODg1MQ==
created_at: 2021-05-28T02:06:46Z
updated_at: 2021-05-28T02:06:46Z
author_association: OWNER
body:
  The one test failure here actually illustrates a larger problem: if the user specifies `.json?_shape=array` but an error occurs, what should we do?

  Prior to this change we return the following JSON:
  ```json
  {
      "ok": false,
      "error": "You did not supply a value for binding 1.",
      "status": 500,
      "title": null
  }
  ```
  But this comes from the `handle_500` higher level code here: https://github.com/simonw/datasette/blob/eae3084b46e2c3931db12cdef79093ad0e644bce/datasette/app.py#L1251-L1263

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1346#issuecomment-850059328
issue_url: https://api.github.com/repos/simonw/datasette/issues/1346
node_id: MDEyOklzc3VlQ29tbWVudDg1MDA1OTMyOA==
created_at: 2021-05-28T02:08:10Z
updated_at: 2021-05-28T02:09:25Z
author_association: OWNER
body:
  This is the current test failure, but it actually another problem that we don't have tests in place for errors with different formats and shapes:

  ```
      def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_client):
          response = magic_parameters_client.get(
              "/data.json?sql=select+:_header_host&_shape=array"
          )
          assert 400 == response.status
  >       assert "You did not supply a value for binding 1." == response.json["error"]
  E       TypeError: list indices must be integers or slices, not str
  ```

  The test fails because `response.json` here is the empty list `[]`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1348#issuecomment-850077261
issue_url: https://api.github.com/repos/simonw/datasette/issues/1348
node_id: MDEyOklzc3VlQ29tbWVudDg1MDA3NzI2MQ==
created_at: 2021-05-28T03:05:38Z
updated_at: 2021-05-28T03:05:38Z
author_association: CONTRIBUTOR
body:
  Note, the CVEs are probably resolvable with this https://github.com/simonw/datasette/pull/1296 . My experience is that Ubuntu seems to manage these better? Though that is surprising :/ 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1348#issuecomment-850077484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1348
node_id: MDEyOklzc3VlQ29tbWVudDg1MDA3NzQ4NA==
created_at: 2021-05-28T03:06:16Z
updated_at: 2021-05-28T03:06:16Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1348](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (56cba8f) into [main](https://codecov.io/gh/simonw/datasette/commit/7b106e106000713bbee31b34d694b3dadbd4818c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7b106e1) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1348/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1348   +/-   ##
  =======================================
    Coverage   91.56%   91.56%           
  =======================================
    Files          34       34           
    Lines        4282     4282           
  =======================================
    Hits         3921     3921           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7b106e1...56cba8f](https://codecov.io/gh/simonw/datasette/pull/1348?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764253
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NDI1Mw==
created_at: 2021-05-29T03:57:54Z
updated_at: 2021-05-29T03:57:54Z
author_association: OWNER
body:
  The problem here is differentiating between a column with the name `date desc` and wanting to create a descending index on a column called `date`.

  This won't work:
  ```python
  db["ny_times_us_counties"].create_index(["date desc"], desc=True)
  ```
  Because we need to be able to create compound indexes with columns with different directions - for example:
  ```sql
  create index idx_age_desc_name on dogs (age desc, name)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764594
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NDU5NA==
created_at: 2021-05-29T04:00:54Z
updated_at: 2021-05-29T04:00:54Z
author_association: OWNER
body:
  A few options:
  ```python
  db["dogs"].create_index([("age", "desc"), "name"])

  db["dogs"].create_index([desc("age"), "name"])

  db["dogs"].create_index([db.desc("age"), "name"])
  ```
  The first option uses an optional tuple. The second two use a `desc()` function - the question is where should that live? 

  `sqlite_utils.desc(column)` or `db.desc(column)`  are both options.

  I don't like using the term `desc()` for "descending index" though - it feels like it should mean something more broad.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764655
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NDY1NQ==
created_at: 2021-05-29T04:01:41Z
updated_at: 2021-05-29T04:01:41Z
author_association: OWNER
body:
  Maybe:
  ```python
  db["dogs"].create_index([db.descending_index("age"), "name"])
  ```
  It's a little verbose but it's for a relatively rare activity and it does make it very clear what is going on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764700
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NDcwMA==
created_at: 2021-05-29T04:02:10Z
updated_at: 2021-05-29T04:02:10Z
author_association: OWNER
body:
  I could use `db.desc_index("age")` to match SQLite SQL syntax, which uses `desc` and not `descending`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765050
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NTA1MA==
created_at: 2021-05-29T04:05:24Z
updated_at: 2021-05-29T04:05:40Z
author_association: OWNER
body:
  Need to solve this for the CLI tool too. Currently that works like this: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes
  ```sh
  sqlite-utils create-index mydb.db mytable col1 [col2...]
  ```
  Even harder to decide how to add a descending option to this. Maybe like this?
  ```sh
  sqlite-utils create-index mydb.db mytable --direction col1 asc --direction col2 desc
  ```
  It's a bit gross though! We're saying here that if a single one of the columns you are creating an index for is in reverse direction you have to use `--direction` to specify each end every other index.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765291
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NTI5MQ==
created_at: 2021-05-29T04:07:48Z
updated_at: 2021-05-29T04:08:21Z
author_association: OWNER
body:
  For the CLI version I could say that you can use a `-` prefix to specify reverse direction:
  ```sh
  sqlite-utils create-index mydb.db dogs -age name
  ```
  No, that doesn't work - it could get confused with a command-line flag. I guess you could do this:
  ```
  sqlite-utils create-index mydb.db dogs "-age" name
  ```
  This does mean that if any of your column names begin with a hyphen you can't use the CLI to add indexes to them. Is that an acceptable limitation? Users can always use `sqlite-utils mydb.db "create index ..."` in that case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765450
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NTQ1MA==
created_at: 2021-05-29T04:09:13Z
updated_at: 2021-05-29T04:09:13Z
author_association: OWNER
body:
  Decisions: for the Python API I'm going with `db.DescIndex("column")` as the way to do this.

  For the CLI I'm going to do the "-age" thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766335
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NjMzNQ==
created_at: 2021-05-29T04:18:19Z
updated_at: 2021-05-29T04:18:19Z
author_association: OWNER
body:
  Annoyingly the `table.indexes` property won't indicate if an index is in regular or reverse order - because the SQLite `PRAGMA index_info(table)` statement doesn't indicate that either. You have to look at the `sqlite_master` index definition to tell if any of the columns are in reverse order:
  ```
  (Pdb) fresh_db.execute("select * from sqlite_master where type = 'index'").fetchall()
  [('index', 'idx_dogs_age_name', 'dogs', 3, 'CREATE INDEX [idx_dogs_age_name]\n    ON [dogs] ([age] desc, [name])')]
  (Pdb) fresh_db.execute("PRAGMA index_info('idx_dogs_age_name')").fetchall()
  [(0, 2, 'age'), (1, 0, 'name')]
  (Pdb) fresh_db.execute("PRAGMA index_info('idx_dogs_age_name')").description
  (('seqno', None, None, None, None, None, None), ('cid', None, None, None, None, None, None), ('name', None, None, None, None, None, None))
  (Pdb) dogs.indexes
  [Index(seq=0, name='idx_dogs_age_name', unique=0, origin='c', partial=0, columns=['age', 'name'])]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NjU1Mg==
created_at: 2021-05-29T04:20:40Z
updated_at: 2021-05-29T04:24:01Z
author_association: OWNER
body:
  `PRAGMA index_xinfo(table)` DOES return that data:
  ```
  (Pdb) [c[0] for c in fresh_db.execute("PRAGMA index_xinfo('idx_dogs_age_name')").description]
  ['seqno', 'cid', 'name', 'desc', 'coll', 'key']
  (Pdb) fresh_db.execute("PRAGMA index_xinfo('idx_dogs_age_name')").fetchall()
  [(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)]
  ```
  See https://sqlite.org/pragma.html#pragma_index_xinfo

  Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850767210
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2NzIxMA==
created_at: 2021-05-29T04:26:26Z
updated_at: 2021-05-29T04:28:31Z
author_association: OWNER
body:
  It's weird having to use `Database.DescIndex` - I'm going to put `DescIndex` in `sqlite_utils.db` directly and let people import it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850768315
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2ODMxNQ==
created_at: 2021-05-29T04:39:33Z
updated_at: 2021-05-29T04:39:33Z
author_association: OWNER
body:
  This doesn't work:
  ```
  sqlite-utils create-index mydb.db dogs "-age" name
  ```
  But this does:
  ```
  sqlite-utils create-index mydb.db dogs -- -age name
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850769067
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/260
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc2OTA2Nw==
created_at: 2021-05-29T04:48:10Z
updated_at: 2021-05-29T04:48:10Z
author_association: OWNER
body:
  I confirmed and it's possible to have a SQLite column with a hyphen at the start, confirmed using:
  ```
  % sqlite-utils create-table demo.db demo -- id integer name text -blah integer
  % sqlite-utils tables --schema demo.db -t
  table    schema
  -------  ---------------------
  demo     CREATE TABLE [demo] (
              [id] INTEGER,
              [name] TEXT,
              [-blah] INTEGER
           )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771017
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/250
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc3MTAxNw==
created_at: 2021-05-29T05:04:28Z
updated_at: 2021-05-29T05:04:28Z
author_association: OWNER
body:
  https://stackoverflow.com/a/44573867/6083 says:

  > There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771264
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/250
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc3MTI2NA==
created_at: 2021-05-29T05:06:13Z
updated_at: 2021-05-29T05:06:13Z
author_association: OWNER
body:
  The other option is to check if the file starts with `codecs.BOM_UTF8` - which is `b'\xef\xbb\xbf'`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850772428
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/250
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc3MjQyOA==
created_at: 2021-05-29T05:16:02Z
updated_at: 2021-05-29T05:28:57Z
author_association: OWNER
body:
  I needed to find some CSV files on my computer with a BOM at the beginning - I figured out this recipe:
  ```
  % rg -U -E none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .
  ```
  TIL here: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1349#issuecomment-850778230
issue_url: https://api.github.com/repos/simonw/datasette/issues/1349
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc3ODIzMA==
created_at: 2021-05-29T06:11:28Z
updated_at: 2021-05-29T06:11:28Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L315

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1349#issuecomment-850778273
issue_url: https://api.github.com/repos/simonw/datasette/issues/1349
node_id: MDEyOklzc3VlQ29tbWVudDg1MDc3ODI3Mw==
created_at: 2021-05-29T06:11:59Z
updated_at: 2021-05-29T06:11:59Z
author_association: OWNER
body:
  Related issue: https://github.com/simonw/datasette/issues/263 - "Facets should not execute for ?shape=array|object"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1345#issuecomment-851113886
issue_url: https://api.github.com/repos/simonw/datasette/issues/1345
node_id: MDEyOklzc3VlQ29tbWVudDg1MTExMzg4Ng==
created_at: 2021-05-31T01:52:27Z
updated_at: 2021-05-31T01:52:27Z
author_association: OWNER
body:
  Related issue: visit https://latest.datasette.io/fixtures/facetable?_facet=state and click "Hide this column" on the "state" cog menu and you get https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state which shows an error:

  > ### Invalid SQL
  > no such column: state

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1345#issuecomment-851114806
issue_url: https://api.github.com/repos/simonw/datasette/issues/1345
node_id: MDEyOklzc3VlQ29tbWVudDg1MTExNDgwNg==
created_at: 2021-05-31T01:55:56Z
updated_at: 2021-05-31T01:55:56Z
author_association: OWNER
body:
  Two options here:

  - Don't provide users with options that will lead to this situation - so no "Hide this column" option on pages that are already faceted by that column
  - Ignore facet selections for columns which are no longer visible

  I think I like the first option more.

  I could partially implement that in the `table.js` JavaScript by looking at the `?_facet=` parameters... but that won't cover the case where the facet is happening because of default facets configured in `metadata.yml`.

  Instead the JavaScript should look for evidence in the DOM that specific facets are enabled. This could also help me cover other types of faceting, such as `?_facet_array=` or even custom facets provided by plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1345#issuecomment-851115805
issue_url: https://api.github.com/repos/simonw/datasette/issues/1345
node_id: MDEyOklzc3VlQ29tbWVudDg1MTExNTgwNQ==
created_at: 2021-05-31T01:59:39Z
updated_at: 2021-05-31T01:59:39Z
author_association: OWNER
body:
  Maybe there's a concept here of the columns that are required by a selected facet? Those can then be included as `data-` attributes on the page, which will then impact which "Hide this column" options are available.

  I can also use them to provide a better error message than "no such column: state" - I can verify that `?_nocol` and `?_col` have not been used to disable the required columns.

  There is one other option here: I could still include the columns that are known to be needed for faceting in the faceting SQL queries, but leave them out of the query that is used to return the results! That's actually a pretty tempting (albeit more complex) option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1345#issuecomment-851116105
issue_url: https://api.github.com/repos/simonw/datasette/issues/1345
node_id: MDEyOklzc3VlQ29tbWVudDg1MTExNjEwNQ==
created_at: 2021-05-31T02:00:44Z
updated_at: 2021-05-31T02:00:44Z
author_association: OWNER
body:
  Maybe there's a short-term and longer-term solution for this - where the long-term solution is to use different columns in the faceting selects, while the short-term solution is to disable "Hide this column" for certain things.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1345#issuecomment-851117233
issue_url: https://api.github.com/repos/simonw/datasette/issues/1345
node_id: MDEyOklzc3VlQ29tbWVudDg1MTExNzIzMw==
created_at: 2021-05-31T02:04:35Z
updated_at: 2021-05-31T02:04:35Z
author_association: OWNER
body:
  That long-term solution may not be too difficult. The facets are calculated against `sql_no_limit` which is constructed here:

  https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L659-L665

  And used here:

  https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/datasette/views/table.py#L706-L718

  Crucially, `sql_no_limit` is ONLY used for faceting - nothing else uses it anywhere. So constructing it before constructing `sql` and taking `?_col=` and `?_nocol=` into account may not be a complex change.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1345#issuecomment-851120306
issue_url: https://api.github.com/repos/simonw/datasette/issues/1345
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEyMDMwNg==
created_at: 2021-05-31T02:14:36Z
updated_at: 2021-05-31T02:14:36Z
author_association: OWNER
body:
  Yes! This was easier than I thought. I'm going with that solution - where facets are calculated against all columns, ignoring `?_col=` and `?_nocol=` entirely.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1350#issuecomment-851122934
issue_url: https://api.github.com/repos/simonw/datasette/issues/1350
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEyMjkzNA==
created_at: 2021-05-31T02:23:47Z
updated_at: 2021-05-31T02:23:47Z
author_association: OWNER
body:
  I think `?_facets_off=1` is a good design for this.

  I considered `?_facet=` with an empty string but that's a bit weird, and it's not clear what that would do if used like `?_facet=state&_facet=`.

  So it definitely needs to be a separate named option.

  Actually, since I have `?_nocol=` I'm going to call this `?_nofacets=1` instead as that's a little more consistent.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1345#issuecomment-851127314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1345
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEyNzMxNA==
created_at: 2021-05-31T02:40:26Z
updated_at: 2021-05-31T02:40:44Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state - the state column is not selected but facet by state still works: 

  <img width="1134" alt="fixtures__facetable__15_rows_and_Why_Russians_do_not_smile" src="https://user-images.githubusercontent.com/9599/120132273-110a2780-c198-11eb-91a5-cafb77a78c97.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1349#issuecomment-851128524
issue_url: https://api.github.com/repos/simonw/datasette/issues/1349
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEyODUyNA==
created_at: 2021-05-31T02:44:44Z
updated_at: 2021-05-31T02:44:44Z
author_association: OWNER
body:
  Now that I have `?_nofacets=1` I can use that to fix this. The challenge is that in this block of code I need to modify the incoming request's query string arguments, which isn't something I've done before: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/base.py#L263-L270

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1349#issuecomment-851129019
issue_url: https://api.github.com/repos/simonw/datasette/issues/1349
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEyOTAxOQ==
created_at: 2021-05-31T02:46:38Z
updated_at: 2021-05-31T02:46:38Z
author_association: OWNER
body:
  I think the right way to do this is to construct a new `Request` with a modified ASGI scope: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/utils/asgi.py#L88-L99

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1349#issuecomment-851129464
issue_url: https://api.github.com/repos/simonw/datasette/issues/1349
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEyOTQ2NA==
created_at: 2021-05-31T02:48:06Z
updated_at: 2021-05-31T02:48:06Z
author_association: OWNER
body:
  Actually there is precedent for swapping out `request.scope` for a new scope, as seen here in the routing code: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/app.py#L1117-L1122

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1349#issuecomment-851131999
issue_url: https://api.github.com/repos/simonw/datasette/issues/1349
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEzMTk5OQ==
created_at: 2021-05-31T02:57:25Z
updated_at: 2021-05-31T02:57:25Z
author_association: OWNER
body:
  I'm having a really hard time figuring out how to unit test this - ideally I'd monitor which SQL queries are executed using the tracing mechanism, but that's not set up to work with anything other than HTML or JSON outputs: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1349#issuecomment-851133125
issue_url: https://api.github.com/repos/simonw/datasette/issues/1349
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEzMzEyNQ==
created_at: 2021-05-31T03:01:48Z
updated_at: 2021-05-31T03:01:48Z
author_association: OWNER
body:
  I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1351#issuecomment-851133471
issue_url: https://api.github.com/repos/simonw/datasette/issues/1351
node_id: MDEyOklzc3VlQ29tbWVudDg1MTEzMzQ3MQ==
created_at: 2021-05-31T03:02:59Z
updated_at: 2021-05-31T03:02:59Z
author_association: OWNER
body:
  Since traces only work with `text/html` and JSON at the moment, the easiest way to do this will be to wrap generated CSV in a HTML page in a textarea if the user specified `?_trace=1`:

  https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1284#issuecomment-851567204
issue_url: https://api.github.com/repos/simonw/datasette/issues/1284
node_id: MDEyOklzc3VlQ29tbWVudDg1MTU2NzIwNA==
created_at: 2021-05-31T15:42:10Z
updated_at: 2021-11-04T03:15:01Z
author_association: CONTRIBUTOR
body:
  I very much want to make:
    https://list.SaferDisinfectants.org/disinfectants/listN 
  have this URL:
   https://list.SaferDisinfectants.org/
   
  I'm using only one table page on the site, with no pagination. I'm not using the home page, though when I tried to move my table to the home page as mentioned above, I failed to figure out how. 

  I am using cloudflare, but I haven't figured out a forwarding or HTML re-write method of doing this, either.

  Is there any way I can get a prettier list URL? I'm on Vercel.

  (I have a wordpress site on the main domain on a separate host.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1353#issuecomment-852238201
issue_url: https://api.github.com/repos/simonw/datasette/issues/1353
node_id: MDEyOklzc3VlQ29tbWVudDg1MjIzODIwMQ==
created_at: 2021-06-01T15:53:57Z
updated_at: 2021-06-01T15:53:57Z
author_association: OWNER
body:
  I'm going to rename `?_nofacets=1` to `?_nofacet=1` to keep it consistent with the new `?_nocount=1` option (and because I don't like `?_nocounts=1`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1257#issuecomment-852681622
issue_url: https://api.github.com/repos/simonw/datasette/issues/1257
node_id: MDEyOklzc3VlQ29tbWVudDg1MjY4MTYyMg==
created_at: 2021-06-02T03:12:18Z
updated_at: 2021-06-02T03:12:18Z
author_association: OWNER
body:
  Created a test database like this:
  ```
  % sqlite-utils create-table quote-in-name.db "this'hasquoteinname" id integer name text --pk id
  % datasette quote-in-name.db -p 8025 --pdb
  INFO:     Started server process [86046]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8025 (Press CTRL+C to quit)
  > /Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py(530)detect_fts()
  -> rows = conn.execute(detect_fts_sql(table)).fetchall()
  (Pdb) c
  Traceback (most recent call last):
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1124, in route_path
      response = await view(request, send)
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 147, in view
      return await self.dispatch_request(
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request
      return await handler(request, *args, **kwargs)
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/index.py", line 72, in get
      "fts_table": await db.fts_table(table),
    File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 279, in fts_table
      return await self.execute_fn(lambda conn: detect_fts(conn, table))
    File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 155, in execute_fn
      return await asyncio.get_event_loop().run_in_executor(
    File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/concurrent/futures/thread.py", line 57, in run
      result = self.fn(*self.args, **self.kwargs)
    File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 153, in in_thread
      return fn(conn)
    File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 279, in <lambda>
      return await self.execute_fn(lambda conn: detect_fts(conn, table))
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 530, in detect_fts
      rows = conn.execute(detect_fts_sql(table)).fetchall()
  sqlite3.OperationalError: near "hasquoteinname": syntax error
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1257#issuecomment-852686827
issue_url: https://api.github.com/repos/simonw/datasette/issues/1257
node_id: MDEyOklzc3VlQ29tbWVudDg1MjY4NjgyNw==
created_at: 2021-06-02T03:26:27Z
updated_at: 2021-06-02T03:26:27Z
author_association: OWNER
body:
  I tried and failed to get this fix working for tables with double quotes in their name - I couldn't figure out what the double-quote-in-a-table-name version of this code would look like: https://github.com/simonw/datasette/blob/807de378d08752a0f05bb1b980a0a62620a70520/datasette/utils/__init__.py#L538-L548

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1346#issuecomment-852693854
issue_url: https://api.github.com/repos/simonw/datasette/issues/1346
node_id: MDEyOklzc3VlQ29tbWVudDg1MjY5Mzg1NA==
created_at: 2021-06-02T03:44:35Z
updated_at: 2021-06-02T03:44:35Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1346](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (3bffc35) into [main](https://codecov.io/gh/simonw/datasette/commit/7b106e106000713bbee31b34d694b3dadbd4818c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7b106e1) will **increase** coverage by `0.14%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1346/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1346      +/-   ##
  ==========================================
  + Coverage   91.56%   91.71%   +0.14%     
  ==========================================
    Files          34       34              
    Lines        4282     4332      +50     
  ==========================================
  + Hits         3921     3973      +52     
  + Misses        361      359       -2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.69% <ø> (-0.15%)` | :arrow_down: |
  | [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.20% <100.00%> (+0.17%)` | :arrow_up: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.36% <100.00%> (+0.05%)` | :arrow_up: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.41% <100.00%> (+0.39%)` | :arrow_up: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.28% <100.00%> (+0.09%)` | :arrow_up: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.90% <100.00%> (+0.07%)` | :arrow_up: |
  | [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `85.05% <0.00%> (+3.44%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7b106e1...3bffc35](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-852703357
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDg1MjcwMzM1Nw==
created_at: 2021-06-02T04:08:03Z
updated_at: 2021-06-02T04:08:03Z
author_association: OWNER
body:
  A SQL error now looks like this: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

  <img width="1023" alt="fixtures__select___from__foo_" src="https://user-images.githubusercontent.com/9599/120422579-691e6680-c31d-11eb-99ea-655bdc2a401d.png">

  I'm going to get rid of that "0 results" message if an error is shown.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-852706970
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDg1MjcwNjk3MA==
created_at: 2021-06-02T04:15:54Z
updated_at: 2021-06-02T04:15:54Z
author_association: OWNER
body:
  <img width="1009" alt="fixtures__select___from__foo__and_Facets_should_not_execute_for__shape_array_object_·_Issue__263_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/120423073-8a338700-c31e-11eb-8c8f-affc24708b32.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1306#issuecomment-852707676
issue_url: https://api.github.com/repos/simonw/datasette/issues/1306
node_id: MDEyOklzc3VlQ29tbWVudDg1MjcwNzY3Ng==
created_at: 2021-06-02T04:17:41Z
updated_at: 2021-06-02T04:17:41Z
author_association: OWNER
body:
  We need to add a test that illustrates the bug in #1305 so we can prove that this definitely fixes it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1306#issuecomment-852708574
issue_url: https://api.github.com/repos/simonw/datasette/issues/1306
node_id: MDEyOklzc3VlQ29tbWVudDg1MjcwODU3NA==
created_at: 2021-06-02T04:20:18Z
updated_at: 2021-06-02T04:20:18Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/115332ce76c0e867d9936406aaf4bcee6b1ef3cb/datasette/views/index.py#L77 helps here - looks like we can pass `?_sort=relationships` to trigger the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/893#issuecomment-852712469
issue_url: https://api.github.com/repos/simonw/datasette/issues/893
node_id: MDEyOklzc3VlQ29tbWVudDg1MjcxMjQ2OQ==
created_at: 2021-06-02T04:29:56Z
updated_at: 2021-06-02T04:29:56Z
author_association: OWNER
body:
  I think I fixed this a while ago, though I can't find the commit now. Please re-open if the bug still happens!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853525036
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/261
node_id: MDEyOklzc3VlQ29tbWVudDg1MzUyNTAzNg==
created_at: 2021-06-03T03:02:22Z
updated_at: 2021-06-03T03:02:22Z
author_association: OWNER
body:
  This would be a breaking change - and the fact that it returns auxiliary columns isn't particularly useful for most cases - "Auxiliary columns are additional columns needed to locate the table entry that corresponds to each index entry".

  Instead, I'm going to add a new property `table.xindexes` which exposes this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853530348
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/261
node_id: MDEyOklzc3VlQ29tbWVudDg1MzUzMDM0OA==
created_at: 2021-06-03T03:16:33Z
updated_at: 2021-06-03T03:16:33Z
author_association: OWNER
body:
  In prototyping this out I realize that I actually want to get back the name of each index, then for each of them the detailed list of index columns. Here's the test from my initial prototype:
  ```python
  def test_xindexes(fresh_db):
      fresh_db.executescript(
          """
          create table Gosh (c1 text, c2 text, c3 text);
          create index Gosh_c1 on Gosh(c1);
          create index Gosh_c2c3 on Gosh(c2, c3 desc);
      """
      )
      assert fresh_db["Gosh"].xindexes == [
          (
              "Gosh_c2c3",
              [
                  XIndex(seqno=0, cid=1, name="c2", desc=0, coll="BINARY", key=1),
                  XIndex(seqno=1, cid=2, name="c3", desc=1, coll="BINARY", key=1),
                  XIndex(seqno=2, cid=-1, name=None, desc=0, coll="BINARY", key=0),
              ],
          ),
          (
              "Gosh_c1",
              [
                  XIndex(seqno=0, cid=0, name="c1", desc=0, coll="BINARY", key=1),
                  XIndex(seqno=1, cid=-1, name=None, desc=0, coll="BINARY", key=0),
              ],
          ),
      ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853535559
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/261
node_id: MDEyOklzc3VlQ29tbWVudDg1MzUzNTU1OQ==
created_at: 2021-06-03T03:32:47Z
updated_at: 2021-06-03T03:32:47Z
author_association: OWNER
body:
  New design:
  ```python
  def test_xindexes(fresh_db):
      fresh_db.executescript(
          """
          create table Gosh (c1 text, c2 text, c3 text);
          create index Gosh_c1 on Gosh(c1);
          create index Gosh_c2c3 on Gosh(c2, c3 desc);
      """
      )
      assert fresh_db["Gosh"].xindexes == [
          XIndex(
              name="Gosh_c2c3",
              columns=[
                  XIndexColumn(seqno=0, cid=1, name="c2", desc=0, coll="BINARY", key=1),
                  XIndexColumn(seqno=1, cid=2, name="c3", desc=1, coll="BINARY", key=1),
                  XIndexColumn(seqno=2, cid=-1, name=None, desc=0, coll="BINARY", key=0),
              ],
          ),
          XIndex(
              name="Gosh_c1",
              columns=[
                  XIndexColumn(seqno=0, cid=0, name="c1", desc=0, coll="BINARY", key=1),
                  XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll="BINARY", key=0),
              ],
          ),
      ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853544493
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/263
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU0NDQ5Mw==
created_at: 2021-06-03T04:03:59Z
updated_at: 2021-06-03T04:03:59Z
author_association: OWNER
body:
  Here's how `sqlite-utils triggers` works: https://github.com/simonw/sqlite-utils/blob/9c67cb925253cd5ef54a1fe0496e0ff9caeacfd6/sqlite_utils/cli.py#L1266-L1277

  Running it from a SQL query makes it easy to support modifiers like `--csv` and `-t`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853545743
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/263
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU0NTc0Mw==
created_at: 2021-06-03T04:08:04Z
updated_at: 2021-06-03T04:08:04Z
author_association: OWNER
body:
  Figuring out the right queries:

  https://covid-19.datasettes.com/covid?sql=select+sqlite_master.name%2C+i.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Awhere+type+%3D+%27table%27

  This query shows all columns across all indexes across all tables:

  ```sql
  select i.name as index_name, xinfo.* from sqlite_master
  join pragma_index_list(sqlite_master.name) i
  join pragma_index_xinfo(index_name) xinfo
  where sqlite_master.type = 'table'
  ```
  https://covid-19.datasettes.com/covid?sql=select+i.name+as+index_name%2C+xinfo.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Ajoin+pragma_index_xinfo%28index_name%29+xinfo%0D%0Awhere+sqlite_master.type+%3D+%27table%27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853547681
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/263
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU0NzY4MQ==
created_at: 2021-06-03T04:14:28Z
updated_at: 2021-06-03T04:14:28Z
author_association: OWNER
body:
  This looks good:
  ```
  % sqlite-utils indexes /tmp/covid.db -t      
  index_name                                                seqno    cid  name                 desc  coll      key
  ------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
  idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
  idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
  idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
  idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
  idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
  idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
  idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
  idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1

  % sqlite-utils indexes /tmp/covid.db -t --aux
  index_name                                                seqno    cid  name                 desc  coll      key
  ------------------------------------------------------  -------  -----  -----------------  ------  ------  -----
  idx_johns_hopkins_csse_daily_reports_combined_key             0     12  combined_key            0  BINARY      1
  idx_johns_hopkins_csse_daily_reports_combined_key             1     -1                          0  BINARY      0
  idx_johns_hopkins_csse_daily_reports_country_or_region        0      1  country_or_region       0  BINARY      1
  idx_johns_hopkins_csse_daily_reports_country_or_region        1     -1                          0  BINARY      0
  idx_johns_hopkins_csse_daily_reports_province_or_state        0      2  province_or_state       0  BINARY      1
  idx_johns_hopkins_csse_daily_reports_province_or_state        1     -1                          0  BINARY      0
  idx_johns_hopkins_csse_daily_reports_day                      0      0  day                     0  BINARY      1
  idx_johns_hopkins_csse_daily_reports_day                      1     -1                          0  BINARY      0
  idx_ny_times_us_counties_date                                 0      0  date                    1  BINARY      1
  idx_ny_times_us_counties_date                                 1     -1                          0  BINARY      0
  idx_ny_times_us_counties_fips                                 0      3  fips                    0  BINARY      1
  idx_ny_times_us_counties_fips                                 1     -1                          0  BINARY      0
  idx_ny_times_us_counties_county                               0      1  county                  0  BINARY      1
  idx_ny_times_us_counties_county                               1     -1                          0  BINARY      0
  idx_ny_times_us_counties_state                                0      2  state                   0  BINARY      1
  idx_ny_times_us_counties_state                                1     -1                          0  BINARY      0
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853553754
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/264
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU1Mzc1NA==
created_at: 2021-06-03T04:32:42Z
updated_at: 2021-06-03T04:36:36Z
author_association: OWNER
body:
  This is a really interesting thought. I've so far resisted the temptation to add plugins to `sqlite-utils`, partly to avoid overlap with Datasette - but I'm open to discussing it.

  There's actually a way for you to do what you're describing using `datasette` on the command-line, though it's a little obscure - also Datasette doesn't yet have a GeoJSON output extension, though it really should have one.

  Here's an example using [datasette-yaml](https://datasette.io/plugins/datasette-yaml):

  ```
  datasette /tmp/covid.db --get='/covid/ny_times_us_counties.yaml'
  - rowid: 1
    date: '2020-01-21'
    county: Snohomish
    state: Washington
    fips: 53061
    cases: 1
    deaths: 0
  - rowid: 2
    date: '2020-01-22'
    county: Snohomish
    state: Washington
    fips: 53061
    cases: 1
    deaths: 0
  ```
  It even works with arbitrary SQL queries, though you might have to apply URL encoding to the `--get` string (this seems to work though):
  ```
  datasette /tmp/covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 2' 
  - date: '2020-01-21'
    county: Snohomish
    state: Washington
    fips: 53061
    cases: 1
    deaths: 0
  - date: '2020-01-22'
    county: Snohomish
    state: Washington
    fips: 53061
    cases: 1
    deaths: 0
  ```
  Here's the documentation for `--get`: https://docs.datasette.io/en/latest/getting_started.html#datasette-get

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1355#issuecomment-853557439
issue_url: https://api.github.com/repos/simonw/datasette/issues/1355
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU1NzQzOQ==
created_at: 2021-06-03T04:43:14Z
updated_at: 2021-06-03T04:43:14Z
author_association: OWNER
body:
  It's using `TestClient` at the moment which is a wrapper around `httpx` (as of ) that uses the `@async_to_sync` decorator to hide the async nature.

  https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/utils/testing.py#L102-L156

  Maybe the fix here is to switch the `--get` implementation to using `httpx` directly with https://www.python-httpx.org/async/#streaming-responses

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853558741
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/264
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU1ODc0MQ==
created_at: 2021-06-03T04:47:19Z
updated_at: 2021-06-03T04:47:19Z
author_association: OWNER
body:
  This inspired me to re-examine how `--get` works, hence this issue: https://github.com/simonw/datasette/issues/1355

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-853559915
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU1OTkxNQ==
created_at: 2021-06-03T04:50:52Z
updated_at: 2021-06-03T04:50:52Z
author_association: OWNER
body:
  What happens if you pass multiple databases? The `--query` would be executed against the first one. And if you pass `--crossdb` it would be executed against the `/_memory` database and would support cross-database joins.

  Key thing here is that output plugins are supported (also plugins that add new SQL functions), making many Datasette plugins usable from the command-line.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-853560237
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU2MDIzNw==
created_at: 2021-06-03T04:51:49Z
updated_at: 2021-06-03T04:51:49Z
author_association: OWNER
body:
  This feels like a relatively simple feature to implement that unlocks a whole new set of possible uses for Datasette - as described by @eyeseast in https://github.com/simonw/sqlite-utils/issues/264#issue-907642546.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-853560870
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU2MDg3MA==
created_at: 2021-06-03T04:53:47Z
updated_at: 2021-06-03T04:53:56Z
author_association: OWNER
body:
  This is also interesting when used in conjunction with the proposed `datasette insert` command from #1163 - Datasette would become a plugin-driven CLI tool for both ingesting and outputting data, as a side-effect of its web features.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-853562891
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU2Mjg5MQ==
created_at: 2021-06-03T04:59:40Z
updated_at: 2021-06-03T04:59:40Z
author_association: OWNER
body:
  It's weird that `--get` is documented on this page right now: https://docs.datasette.io/en/stable/getting_started.html#datasette-get

  If I implement this I should build a new "Datasette on the command-line" documentation page to cover both `--get` and `--query`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-853565850
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU2NTg1MA==
created_at: 2021-06-03T05:07:21Z
updated_at: 2021-06-03T05:07:21Z
author_association: OWNER
body:
  A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177.

  So there's not much point implementing this unless we first make plugins able to add custom streaming formats.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-853566337
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU2NjMzNw==
created_at: 2021-06-03T05:08:32Z
updated_at: 2021-06-03T05:08:32Z
author_association: OWNER
body:
  Also relevant: CSV streaming for canned queries in #526 - even better if we could stream ANY SQL query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/526#issuecomment-853567413
issue_url: https://api.github.com/repos/simonw/datasette/issues/526
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU2NzQxMw==
created_at: 2021-06-03T05:11:27Z
updated_at: 2021-06-03T05:11:27Z
author_association: OWNER
body:
  Another potential way to implement this would be to hold the SQLite connection open and execute the full query there.

  I've avoided this in the past due to concerns of resource exhaustion - if multiple requests attempt this at the same time all of the connections in the pool will become tied up and the site will be unable to respond to further requests.

  But... now that Datasette has authentication there's the possibility of making this feature only available to specific authenticated users - the `--root` user for example. Which avoids the danger while unlocking a super-useful feature.

  Not to mention people who are running Datasette privately on their own laptop, or the proposed `--query` CLI feature in #1356.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853567861
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/264
node_id: MDEyOklzc3VlQ29tbWVudDg1MzU2Nzg2MQ==
created_at: 2021-06-03T05:12:21Z
updated_at: 2021-06-03T05:12:21Z
author_association: OWNER
body:
  I think this is more likely to happen in Datasette than in sqlite-utils - see https://github.com/simonw/datasette/issues/1356 for thoughts on this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-853895159
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1Mzg5NTE1OQ==
created_at: 2021-06-03T14:03:59Z
updated_at: 2021-06-03T14:03:59Z
author_association: CONTRIBUTOR
body:
  (Putting thoughts here to keep the conversation in one place.)

  I think using datasette for this use-case is the right approach. I usually have both datasette and sqlite-utils installed in the same project, and that's where I'm trying out queries, so it probably makes the most sense to have datasette also manage the output (and maybe the input, too).

  It seems like both `--get` and `--query` could work better as subcommands, rather than options, if you're looking at building out a full CLI experience in datasette. It would give a cleaner separation in what you're trying to do and let each have its own dedicated options. So something like this:

  ```sh
  # run an arbitrary query
  datasette query covid.db "select * from ny_times_us_counties limit 1" --format yaml

  # run a canned query
  datasette get covid.db some-canned-query --format yaml
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1238#issuecomment-855270917
issue_url: https://api.github.com/repos/simonw/datasette/issues/1238
node_id: MDEyOklzc3VlQ29tbWVudDg1NTI3MDkxNw==
created_at: 2021-06-05T17:32:29Z
updated_at: 2021-06-05T17:32:29Z
author_association: OWNER
body:
  This looks like the cause: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/app.py#L1087-L1092

  Note how `path` is modified... but then we create a new `Request()` that uses the old scope, which has unmodified `scope["path"]` - and then the code later on looks at `request.scope["path"]` when deciding if the request matches:

  https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1154-L1155

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1238#issuecomment-855272693
issue_url: https://api.github.com/repos/simonw/datasette/issues/1238
node_id: MDEyOklzc3VlQ29tbWVudDg1NTI3MjY5Mw==
created_at: 2021-06-05T17:45:42Z
updated_at: 2021-06-05T17:45:42Z
author_association: OWNER
body:
  Applying this fix worked when I manually tested it:
  ```diff
           base_url = self.ds.setting("base_url")
           if base_url != "/" and path.startswith(base_url):
               path = "/" + path[len(base_url) :]
  +            scope = dict(scope, path=path, raw_path=path.encode("utf-8"))
           request = Request(scope, receive)
  ```
  But... the test I wrote still failed. My hunch is that this is because deep within the test framework requests go through `ds.client` which may be applying its own changes relevant to `base_url`:

  https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/utils/testing.py#L139

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1238#issuecomment-855278540
issue_url: https://api.github.com/repos/simonw/datasette/issues/1238
node_id: MDEyOklzc3VlQ29tbWVudDg1NTI3ODU0MA==
created_at: 2021-06-05T18:33:25Z
updated_at: 2021-06-05T18:33:25Z
author_association: OWNER
body:
  Got the test to pass by ensuring the tests don't accidentally double-rewrite the path.

  Ran into a new problem:
  ```
      @pytest.mark.asyncio
      @pytest.mark.parametrize(
          "prefix,expected_path", [(None, "/asgi-scope"), ("/prefix/", "/prefix/asgi-scope")]
      )
      async def test_client_path(datasette, prefix, expected_path):
          original_base_url = datasette._settings["base_url"]
          try:
              if prefix is not None:
                  datasette._settings["base_url"] = prefix
              response = await datasette.client.get("/asgi-scope")
              path = response.json()["path"]
  >           assert path == expected_path
  E           AssertionError: assert '/asgi-scope' == '/prefix/asgi-scope'
  E             - /prefix/asgi-scope
  E             ? -------
  E             + /asgi-scope
  ```
  That test confirms that messing around with the `base_url` doesn't modify the ASGI scope... but the fix I'm using for this issue DOES modify the ASGI scope.

  The question raised here is: should the ASGI scope stay unmodified when `base_url` is used?

  I think it should. It doesn't make sense to obscure the "real" path just to get custom pages to work properly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1238#issuecomment-855278998
issue_url: https://api.github.com/repos/simonw/datasette/issues/1238
node_id: MDEyOklzc3VlQ29tbWVudDg1NTI3ODk5OA==
created_at: 2021-06-05T18:37:16Z
updated_at: 2021-06-05T18:37:16Z
author_association: OWNER
body:
  Alternative idea: populate `request.scope` with a new `route_path` which is the base-url-stripped version, which we then use for other routing operations.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1357#issuecomment-855281774
issue_url: https://api.github.com/repos/simonw/datasette/issues/1357
node_id: MDEyOklzc3VlQ29tbWVudDg1NTI4MTc3NA==
created_at: 2021-06-05T18:59:07Z
updated_at: 2021-06-05T18:59:07Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1357](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b27643) into [main](https://codecov.io/gh/simonw/datasette/commit/6e9b07be92905011211d8df7a872fb7c1f2737b2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (6e9b07b) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1357/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1357   +/-   ##
  =======================================
    Coverage   91.71%   91.72%           
  =======================================
    Files          34       34           
    Lines        4333     4336    +3     
  =======================================
  + Hits         3974     3977    +3     
    Misses        359      359           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1357/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.38% <ø> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1357/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.71% <100.00%> (+0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [6e9b07b...1b27643](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1356#issuecomment-855282466
issue_url: https://api.github.com/repos/simonw/datasette/issues/1356
node_id: MDEyOklzc3VlQ29tbWVudDg1NTI4MjQ2Ng==
created_at: 2021-06-05T19:05:06Z
updated_at: 2021-06-05T19:05:06Z
author_association: OWNER
body:
  Yeah that's a good point. I avoided making them sub-commands because `datasette serve` already supports the multitude of other arguments they also need... but actually that was just me being lazy - I can easily share arguments between multiple functions like I do in `sqlite-utils` itself: https://github.com/simonw/sqlite-utils/blob/d1a372b3006e6cf7d2017b3ddc484bf5c033e45d/sqlite_utils/cli.py#L46

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1358#issuecomment-855288228
issue_url: https://api.github.com/repos/simonw/datasette/issues/1358
node_id: MDEyOklzc3VlQ29tbWVudDg1NTI4ODIyOA==
created_at: 2021-06-05T19:57:18Z
updated_at: 2021-06-05T19:57:18Z
author_association: OWNER
body:
  There's also a security fix I need to make, so I'm not going to block this on wrapping up the work on the new Docker image testing from #1344 before putting out this release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1360#issuecomment-855303776
issue_url: https://api.github.com/repos/simonw/datasette/issues/1360
node_id: MDEyOklzc3VlQ29tbWVudDg1NTMwMzc3Ng==
created_at: 2021-06-05T22:23:23Z
updated_at: 2021-06-05T22:23:23Z
author_association: OWNER
body:
  Worth noting that I found this issue myself, and to my knowledge it has not been uncovered by anyone else prior to the patch being released.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1361#issuecomment-855306347
issue_url: https://api.github.com/repos/simonw/datasette/issues/1361
node_id: MDEyOklzc3VlQ29tbWVudDg1NTMwNjM0Nw==
created_at: 2021-06-05T22:49:30Z
updated_at: 2021-06-05T22:49:30Z
author_association: OWNER
body:
  Stack Overflow: https://stackoverflow.com/a/49367679/6083

  > The answer was that `os.chdir()` had been set to the deleted directory by accident. The directory was missing, but the error happened (it seems) at the attempt to get it with `os.getcwd()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1361#issuecomment-855307292
issue_url: https://api.github.com/repos/simonw/datasette/issues/1361
node_id: MDEyOklzc3VlQ29tbWVudDg1NTMwNzI5Mg==
created_at: 2021-06-05T22:59:35Z
updated_at: 2021-06-05T22:59:35Z
author_association: OWNER
body:
  That broke things. Here's how `pytest-cov` fixed a similar issue: https://github.com/pytest-dev/pytest-cov/commit/7ccb1783bf8290447df58deeb41eae74296a6d9b

  See also https://github.com/nedbat/coveragepy/issues/881 and https://github.com/pytest-dev/pytest-cov/issues/306 and https://github.com/nedbat/coveragepy/issues/824

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-855369819
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDg1NTM2OTgxOQ==
created_at: 2021-06-06T09:40:18Z
updated_at: 2021-06-06T09:40:18Z
author_association: OWNER
body:
  > One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`.
  > 
  > I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue.

  That issue is fixed in #1276.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855418065
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQxODA2NQ==
created_at: 2021-06-06T15:37:11Z
updated_at: 2021-06-06T15:37:11Z
author_association: OWNER
body:
  The easiest way to apply CSP is to remove all inline `<script>` blocks (Datasette has a few) and instead serve JavaScript as separate linked files.

  It's possible to keep inline script blocks by calculating a hash of their content and adding a `Content-Security-Policy: script-src 'sha256-B2yPHKaXnvFWtRChIbabYmUBFZdVfKKXHbWtWidDVF8='` to the policy. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src

  This could be achieved with some Django template tricks, but it feels very risky - and done carelessly could end up calculating a hash of a reflected XSS attack!

  The biggest challenge I see around here involves plugins and custom templates. Adopting CSP would require plugins to avoid using any inline scripts, instead keeping their entire implementations in `.js` files.

  That's maybe not a bad thing, but it represents a big commitment. It would need to be adopted before Datasette 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855418401
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQxODQwMQ==
created_at: 2021-06-06T15:39:38Z
updated_at: 2021-06-06T15:39:38Z
author_association: OWNER
body:
  The security benefit of forcing all JavaScript plugins to be written as CSP-friendly external scripts is very compelling though.

  Other plugin-heavy ecosystems such as WordPress have suffered greatly from insecurely written plugins - could this be a huge security win for the Datasette ecosystem generally?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855418698
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQxODY5OA==
created_at: 2021-06-06T15:41:24Z
updated_at: 2021-06-06T15:41:24Z
author_association: OWNER
body:
  I think the best way to answer these questions is with some prototyping - of both Datasette and some of the existing JavaScript plugins.

  I can start with a `datasette-experimental-csp` plugin that sets the header (and could even run an optional report URI mechanism).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855418899
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQxODg5OQ==
created_at: 2021-06-06T15:42:55Z
updated_at: 2021-06-06T15:42:55Z
author_association: OWNER
body:
  Another consideration: testing that this works correctly could require adoption of a real browser test environment (probably Cypress or maybe Playwright) to execute tests that will fail if CSP is violated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855426314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQyNjMxNA==
created_at: 2021-06-06T16:38:04Z
updated_at: 2021-06-06T16:38:04Z
author_association: OWNER
body:
  The other option for inline scripts is the CSP nonce:

      Content-Security-Policy: script-src 'nonce-2726c7f26c'

  Then:

      <script nonce="2726c7f26c">
        var inline = 1;
      </script>

  Since an attacker can't guess what the nonce will be it prevents them from injecting their own script block - this seems easier to make available to plugins than a full hashing mechanism, just make `{{ csp_nonce() }}` available to the template.

  That template function can then be smart enough to set a flag which Datasette uses to decide if the `script-src 'nonce-2726c7f26c'` policy should be sent or not.

  Presumably this would also require adding `Content-Security-Policy` to the `Vary` header though, which will have a nasty effect on Cloudflare and Fastly and such like.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855426516
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQyNjUxNg==
created_at: 2021-06-06T16:39:34Z
updated_at: 2021-06-06T16:39:34Z
author_association: OWNER
body:
  The reason Datasette uses small inline scripts right now is to avoid the overhead of an extra HTTP request for a JavaScript file - but these are both inherently cachable and perform much better under HTTP/2 so that's likely a false optimization.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855426750
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQyNjc1MA==
created_at: 2021-06-06T16:41:30Z
updated_at: 2021-06-06T16:44:49Z
author_association: OWNER
body:
  This is from the current `base.html` template: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/base.html#L62-L66

  Which includes this: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/_close_open_menus.html#L1-L16

  The `body_scripts` bit is for this `extra_body_script` plugin hook, which is the thing that will be the most affected by implementing CSP: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855427396
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQyNzM5Ng==
created_at: 2021-06-06T16:46:17Z
updated_at: 2021-06-06T16:46:17Z
author_association: OWNER
body:
  Mind you, since that plugin hook looks like this:

  ```python
  @hookimpl
  def extra_body_script():
      return {
          "module": True,
          "script": "console.log('Your JavaScript goes here...')"
      }
  ```
  Having it calculate a sha256 hash wouldn't be difficult.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855428296
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQyODI5Ng==
created_at: 2021-06-06T16:53:20Z
updated_at: 2021-06-06T16:53:20Z
author_association: NONE
body:
  > Presumably this would also require adding Content-Security-Policy to the Vary header though, which will have a nasty effect on Cloudflare and Fastly and such like.

  No, because Vary header is about *request* headers that cause the response to vary, not response headers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855428601
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQyODYwMQ==
created_at: 2021-06-06T16:55:33Z
updated_at: 2021-06-06T16:55:33Z
author_association: OWNER
body:
  > No, because Vary header is about _request_ headers that cause the response to vary, not response headers.

  Hah, of course! Thanks for the correction. So the nonce mechanism would actually be pretty great here, especially for the `extra_body_script()` hook.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855429111
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQyOTExMQ==
created_at: 2021-06-06T16:59:05Z
updated_at: 2021-06-06T17:00:15Z
author_association: OWNER
body:
  Twitter conversation: https://twitter.com/simonw/status/1401565566045806594

  @dracos provided some really useful code examples there:

  > We generate it here: https://github.com/mysociety/fixmystreet/blob/e9fec4e567e7148ed128816e5770c2963be51af6/perllib/FixMyStreet/Cobrand/Default.pm#L89-L90
  And use it e.g. https://github.com/mysociety/fixmystreet/blob/ba6788cd25d8f471a4e3308403607627b4d2f4f6/templates/web/base/common_header_tags.html or https://github.com/mysociety/fixmystreet/blob/cb4f2b96364d151988b5c664888468b25cc62240/templates/web/fixmystreet.com/header/css.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1362#issuecomment-855430317
issue_url: https://api.github.com/repos/simonw/datasette/issues/1362
node_id: MDEyOklzc3VlQ29tbWVudDg1NTQzMDMxNw==
created_at: 2021-06-06T17:07:48Z
updated_at: 2021-06-06T17:07:48Z
author_association: OWNER
body:
  I guess I can offer a `disable_csp` setting so that people with complex custom templates aren't completely blocked from using them with Datasette, but maybe it would be better not to offer that? Or to offer it as a `datasette-insecure-csp` plugin instead?

  I like the idea of very actively encouraging CSP across all Datasette projects, but I'm nervous about making the software unusable for certain edge cases.

  Maybe require CSP and wait for someone to complain?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/266#issuecomment-855611939
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDg1NTYxMTkzOQ==
created_at: 2021-06-07T06:07:41Z
updated_at: 2021-06-07T06:07:41Z
author_association: OWNER
body:
  Looks like this is the way to do this:
  ```python
  Point = typing.NamedTuple(
      "Point", [('x', int), ('y', int)]
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1366#issuecomment-856147450
issue_url: https://api.github.com/repos/simonw/datasette/issues/1366
node_id: MDEyOklzc3VlQ29tbWVudDg1NjE0NzQ1MA==
created_at: 2021-06-07T18:02:13Z
updated_at: 2021-06-07T18:02:13Z
author_association: OWNER
body:
  The hack in question is this fixture, which I've been using in an ad-hoc manner to work around errors while running the tests: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/tests/conftest.py#L62-L75

  I don't understand the underlying issue well enough to know how to get rid of it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1366#issuecomment-856147969
issue_url: https://api.github.com/repos/simonw/datasette/issues/1366
node_id: MDEyOklzc3VlQ29tbWVudDg1NjE0Nzk2OQ==
created_at: 2021-06-07T18:03:03Z
updated_at: 2021-06-07T18:03:03Z
author_association: OWNER
body:
  Here's an example of a test that uses it. It's necessary because sometimes fixtures that create temporary directories break in unexpected ways:

  https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_plugins.py#L658-L666

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1367#issuecomment-856160770
issue_url: https://api.github.com/repos/simonw/datasette/issues/1367
node_id: MDEyOklzc3VlQ29tbWVudDg1NjE2MDc3MA==
created_at: 2021-06-07T18:22:33Z
updated_at: 2021-06-07T18:22:33Z
author_association: OWNER
body:
  Here's why: https://github.com/simonw/datasette/blob/03ec71193b9545536898a4bc7493274fec48bdd7/datasette/static/app.css#L455-L458

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1368#issuecomment-856182547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1368
node_id: MDEyOklzc3VlQ29tbWVudDg1NjE4MjU0Nw==
created_at: 2021-06-07T18:59:47Z
updated_at: 2021-06-07T23:04:25Z
author_association: CONTRIBUTOR
body:
  Note that if we went with a "update_metadata" hook, the hook signature would look something like this (it would return nothing):

  ```
  update_metadata(
    datasette=self, metadata=metadata, key=key, database=database, table=table,
    fallback=fallback
  )
  ```

  The Datasette function `_metadata_recursive_update(self, orig, updated)` would disappear into the plugins. Doing this, though, we'd lose the easy ability to make the local metadata.yaml immutable (since we'd no longer have the recursive update).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1365#issuecomment-856212136
issue_url: https://api.github.com/repos/simonw/datasette/issues/1365
node_id: MDEyOklzc3VlQ29tbWVudDg1NjIxMjEzNg==
created_at: 2021-06-07T19:54:04Z
updated_at: 2021-06-07T19:54:04Z
author_association: OWNER
body:
  I've hit this one too. I agree, fixing this in Datasette itself is better than fixing it in the tests across multiple other projects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/266#issuecomment-856231119
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/266
node_id: MDEyOklzc3VlQ29tbWVudDg1NjIzMTExOQ==
created_at: 2021-06-07T20:26:05Z
updated_at: 2021-06-07T20:26:05Z
author_association: OWNER
body:
  https://github.com/python/cpython/blob/2ab27c4af4ddf7528e1375e77c787c7fbb09b5e6/Lib/typing.py#L2173-L2195

  In Python 3.6 or higher can do this:

  ```python
  class Employee(NamedTuple):
      name: str
      id: int
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1370#issuecomment-857139881
issue_url: https://api.github.com/repos/simonw/datasette/issues/1370
node_id: MDEyOklzc3VlQ29tbWVudDg1NzEzOTg4MQ==
created_at: 2021-06-08T20:58:41Z
updated_at: 2021-06-08T20:58:41Z
author_association: OWNER
body:
  We can remove a bunch of unnecessary `str(path)` calls too - this search finds a bunch of possible candidates: https://ripgrep.datasette.io/-/ripgrep?pattern=str%5C%28.*%28db%7Cpath%29&glob=datasette%2F**%2F*.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1373#issuecomment-857684605
issue_url: https://api.github.com/repos/simonw/datasette/issues/1373
node_id: MDEyOklzc3VlQ29tbWVudDg1NzY4NDYwNQ==
created_at: 2021-06-09T13:15:31Z
updated_at: 2021-06-13T15:34:07Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1373](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (d117ba7) into [main](https://codecov.io/gh/simonw/datasette/commit/03418ee037057aa85204f5a3feb2066cbb6a9b3e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (03418ee) will **increase** coverage by `7.65%`.
  > The diff coverage is `93.29%`.

  > :exclamation: Current head d117ba7 differs from pull request most recent head 51ff366. Consider uploading reports for the commit 51ff366 to get more accurate results
  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1373/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1373      +/-   ##
  ==========================================
  + Coverage   84.02%   91.68%   +7.65%     
  ==========================================
    Files          28       34       +6     
    Lines        3774     4340     +566     
  ==========================================
  + Hits         3171     3979     +808     
  + Misses        603      361     -242     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <ø> (ø)` | |
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
  | [datasette/publish/common.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY29tbW9uLnB5) | `94.73% <ø> (ø)` | |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `76.33% <70.64%> (+1.19%)` | :arrow_up: |
  | [datasette/filters.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2ZpbHRlcnMucHk=) | `94.35% <77.77%> (ø)` | |
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <89.28%> (ø)` | |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `90.98% <92.00%> (-0.95%)` | :arrow_down: |
  | [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.73% <94.44%> (+1.13%)` | :arrow_up: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.73% <94.52%> (-0.27%)` | :arrow_down: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.36% <94.53%> (+0.55%)` | :arrow_up: |
  | ... and [29 more](https://codecov.io/gh/simonw/datasette/pull/1373/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [e797565...51ff366](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1371#issuecomment-858099514
issue_url: https://api.github.com/repos/simonw/datasette/issues/1371
node_id: MDEyOklzc3VlQ29tbWVudDg1ODA5OTUxNA==
created_at: 2021-06-09T21:03:49Z
updated_at: 2021-06-09T21:03:49Z
author_association: OWNER
body:
  I'll release these as an alpha straight away - it makes sense to have plugin hook changes available for people to test as alpha dependencies ASAP.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/858#issuecomment-858813675
issue_url: https://api.github.com/repos/simonw/datasette/issues/858
node_id: MDEyOklzc3VlQ29tbWVudDg1ODgxMzY3NQ==
created_at: 2021-06-10T17:27:46Z
updated_at: 2021-06-10T17:27:46Z
author_association: NONE
body:
  shell=True is added to line 56 (I guess it used to be 54) of heroku.py as detailed in the original issue. (for posterity)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/858#issuecomment-858831895
issue_url: https://api.github.com/repos/simonw/datasette/issues/858
node_id: MDEyOklzc3VlQ29tbWVudDg1ODgzMTg5NQ==
created_at: 2021-06-10T17:44:09Z
updated_at: 2021-06-10T17:44:09Z
author_association: NONE
body:
  any fixes for that recursive issue with temp file? I get it using both heroku and cloudrun, although it seems to still publish and deploy fine

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1374#issuecomment-859572791
issue_url: https://api.github.com/repos/simonw/datasette/issues/1374
node_id: MDEyOklzc3VlQ29tbWVudDg1OTU3Mjc5MQ==
created_at: 2021-06-11T13:12:58Z
updated_at: 2021-06-11T13:12:58Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1374](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0ef0dd5) into [main](https://codecov.io/gh/simonw/datasette/commit/cd7678fde65319d7b6955ce9f4678ba4b9e64b66?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cd7678f) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1374/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1374   +/-   ##
  =======================================
    Coverage   91.68%   91.68%           
  =======================================
    Files          34       34           
    Lines        4340     4340           
  =======================================
    Hits         3979     3979           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [cd7678f...0ef0dd5](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859894105
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg1OTg5NDEwNQ==
created_at: 2021-06-11T20:28:52Z
updated_at: 2021-06-11T20:28:52Z
author_association: OWNER
body:
  Out of interest, here are the rows from that table where `sql` is `null`: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++sqlite_master%0D%0Awhere%0D%0A++sql+is+null

  ```csv
  type,name,tbl_name,rootpage,sql
  index,sqlite_autoindex_simple_primary_key_1,simple_primary_key,3,
  index,sqlite_autoindex_primary_key_multiple_columns_1,primary_key_multiple_columns,5,
  index,sqlite_autoindex_primary_key_multiple_columns_explicit_label_1,primary_key_multiple_columns_explicit_label,7,
  index,sqlite_autoindex_compound_primary_key_1,compound_primary_key,9,
  index,sqlite_autoindex_compound_three_primary_keys_1,compound_three_primary_keys,11,
  index,sqlite_autoindex_foreign_key_references_1,foreign_key_references,14,
  index,sqlite_autoindex_sortable_1,sortable,16,
  index,sqlite_autoindex_Table With Space In Name_1,Table With Space In Name,20,
  index,sqlite_autoindex_table/with/slashes.csv_1,table/with/slashes.csv,22,
  index,sqlite_autoindex_complex_foreign_keys_1,complex_foreign_keys,24,
  index,sqlite_autoindex_custom_foreign_key_label_1,custom_foreign_key_label,26,
  index,sqlite_autoindex_tags_1,tags,31,
  index,sqlite_autoindex_searchable_tags_1,searchable_tags,34,
  index,sqlite_autoindex_searchable_fts_segdir_1,searchable_fts_segdir,37,
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859895540
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg1OTg5NTU0MA==
created_at: 2021-06-11T20:30:34Z
updated_at: 2021-06-11T20:30:34Z
author_association: OWNER
body:
  You can currently see the `sql` on the CLI using:

      % sqlite-utils rows fixtures.db sqlite_master -c name -c sql
      name                                                            sql
      --------------------------------------------------------------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------
      simple_primary_key                                              CREATE TABLE simple_primary_key (
                                                                        id varchar(30) primary key,
                                                                        content text
                                                                      )
      sqlite_autoindex_simple_primary_key_1
      primary_key_multiple_columns                                    CREATE TABLE primary_key_multiple_columns (
                                                                        id varchar(30) primary key,
                                                                        content text,
                                                                        content2 text
                                                                      )
      sqlite_autoindex_primary_key_multiple_columns_1
      primary_key_multiple_columns_explicit_label                     CREATE TABLE primary_key_multiple_columns_explicit_label (
                                                                        id varchar(30) primary key,
                                                                        content text,
                                                                        content2 text
                                                                      )
      sqlite_autoindex_primary_key_multiple_columns_explicit_label_1
      compound_primary_key                                            CREATE TABLE compound_primary_key (
                                                                        pk1 varchar(30),
                                                                        pk2 varchar(30),
                                                                        content text,
                                                                        PRIMARY KEY (pk1, pk2)
                                                                      )
      sqlite_autoindex_compound_primary_key_1
      compound_three_primary_keys                                     CREATE TABLE compound_three_primary_keys (
                                                                        pk1 varchar(30),
                                                                        pk2 varchar(30),
                                                                        pk3 varchar(30),
                                                                        content text,
                                                                        PRIMARY KEY (pk1, pk2, pk3)
                                                                      )
      sqlite_autoindex_compound_three_primary_keys_1
      foreign_key_references                                          CREATE TABLE foreign_key_references (
                                                                        pk varchar(30) primary key,
                                                                        foreign_key_with_label varchar(30),
                                                                        foreign_key_with_no_label varchar(30),
                                                                        FOREIGN KEY (foreign_key_with_label) REFERENCES simple_primary_key(id),
                                                                        FOREIGN KEY (foreign_key_with_no_label) REFERENCES primary_key_multiple_columns(id)
                                                                      )
      sqlite_autoindex_foreign_key_references_1
      sortable                                                        CREATE TABLE sortable (
                                                                        pk1 varchar(30),
                                                                        pk2 varchar(30),
                                                                        content text,
                                                                        sortable integer,
                                                                        sortable_with_nulls real,
                                                                        sortable_with_nulls_2 real,
                                                                        text text,
                                                                        PRIMARY KEY (pk1, pk2)
                                                                      )
      sqlite_autoindex_sortable_1
      no_primary_key                                                  CREATE TABLE no_primary_key (
                                                                        content text,
                                                                        a text,
                                                                        b text,
                                                                        c text
                                                                      )
      123_starts_with_digits                                          CREATE TABLE [123_starts_with_digits] (
                                                                        content text
                                                                      )
      paginated_view                                                  CREATE VIEW paginated_view AS
                                                                          SELECT
                                                                              content,
                                                                              '- ' || content || ' -' AS content_extra
                                                                          FROM no_primary_key
      Table With Space In Name                                        CREATE TABLE "Table With Space In Name" (
                                                                        pk varchar(30) primary key,
                                                                        content text
                                                                      )
      sqlite_autoindex_Table With Space In Name_1
      table/with/slashes.csv                                          CREATE TABLE "table/with/slashes.csv" (
                                                                        pk varchar(30) primary key,
                                                                        content text
                                                                      )
      sqlite_autoindex_table/with/slashes.csv_1
      complex_foreign_keys                                            CREATE TABLE "complex_foreign_keys" (
                                                                        pk varchar(30) primary key,
                                                                        f1 text,
                                                                        f2 text,
                                                                        f3 text,
                                                                        FOREIGN KEY ("f1") REFERENCES [simple_primary_key](id),
                                                                        FOREIGN KEY ("f2") REFERENCES [simple_primary_key](id),
                                                                        FOREIGN KEY ("f3") REFERENCES [simple_primary_key](id)
                                                                      )
      sqlite_autoindex_complex_foreign_keys_1
      custom_foreign_key_label                                        CREATE TABLE "custom_foreign_key_label" (
                                                                        pk varchar(30) primary key,
                                                                        foreign_key_with_custom_label text,
                                                                        FOREIGN KEY ("foreign_key_with_custom_label") REFERENCES [primary_key_multiple_columns_explicit_label](id)
                                                                      )
      sqlite_autoindex_custom_foreign_key_label_1
      units                                                           CREATE TABLE units (
                                                                        pk integer primary key,
                                                                        distance int,
                                                                        frequency int
                                                                      )
      searchable                                                      CREATE TABLE searchable (
                                                                        pk integer primary key,
                                                                        text1 text,
                                                                        text2 text,
                                                                        [name with . and spaces] text
                                                                      )
      searchable_fts                                                  CREATE VIRTUAL TABLE "searchable_fts"
                                                                          USING FTS3 (text1, text2, [name with . and spaces], content="searchable")
      searchable_fts_content                                          CREATE TABLE 'searchable_fts_content'(docid INTEGER PRIMARY KEY, 'c0text1', 'c1text2', 'c2name with . and spaces', 'c3content')
      searchable_fts_segments                                         CREATE TABLE 'searchable_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB)
      searchable_fts_segdir                                           CREATE TABLE 'searchable_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx))
      sqlite_autoindex_searchable_fts_segdir_1
      select                                                          CREATE TABLE [select] (
                                                                        [group] text,
                                                                        [having] text,
                                                                        [and] text
                                                                      )
      facet_cities                                                    CREATE TABLE facet_cities (
                                                                          id integer primary key,
                                                                          name text
                                                                      )
      simple_view                                                     CREATE VIEW simple_view AS
                                                                          SELECT content, upper(content) AS upper_content FROM simple_primary_key

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859898736
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDg1OTg5ODczNg==
created_at: 2021-06-11T20:37:44Z
updated_at: 2021-06-11T20:37:44Z
author_association: OWNER
body:
  From the prototype:
  ```
  % sqlite-utils schema 24ways.db
  CREATE TABLE [articles] (
                 [title] TEXT  ,
     [contents] TEXT  ,
     [year] TEXT  ,
     [author] TEXT  ,
     [author_slug] TEXT  ,
     [published] TEXT  ,
     [url] TEXT  ,
     [topic] TEXT  
          );
  CREATE VIRTUAL TABLE "articles_fts" USING FTS5 (
                  title, author, contents,
                  content="articles"
              );
  CREATE TABLE 'articles_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
  CREATE TABLE 'articles_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
  CREATE TABLE 'articles_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
  CREATE TABLE 'articles_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
  % sqlite-utils schema 24ways.db | sqlite3 /tmp/boo.db
  Error: near line 15: table 'articles_fts_data' already exists
  Error: near line 16: table 'articles_fts_idx' already exists
  Error: near line 17: table 'articles_fts_docsize' already exists
  Error: near line 18: table 'articles_fts_config' already exists
  ```
  The problem here is that the `CREATE VIRTUAL TABLE "articles_fts"...` line causes those next four tables to be created - but that means that piping the output of this command into `sqlite3` in order to re-create those tables throws errors.

  I don't think this matters. I see this tool as more for introspection than for recreating table structures.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/270#issuecomment-859986489
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/270
node_id: MDEyOklzc3VlQ29tbWVudDg1OTk4NjQ4OQ==
created_at: 2021-06-12T02:47:12Z
updated_at: 2021-06-12T02:47:12Z
author_association: OWNER
body:
  Can you expand on what you'd like to change here? The library and CLI tool already allow JSON data to be stored in columns:

  - https://sqlite-utils.datasette.io/en/stable/cli.html#nested-json-values
  - https://sqlite-utils.datasette.io/en/stable/python-api.html#storing-json

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/270#issuecomment-860031071
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/270
node_id: MDEyOklzc3VlQ29tbWVudDg2MDAzMTA3MQ==
created_at: 2021-06-12T10:00:24Z
updated_at: 2021-06-12T10:00:24Z
author_association: NONE
body:
  Sure, I am sorry if my message hasn't been clear enough. I am also a new user :)

  At the beginning, I just call `sqlite-utils insert "$db" "$table" "$jsonfile"` to create the database. sqlite-utils convert JSON values into `TEXT`, when it tries to determine the schema automatically. I then try to transform the table to set `JSON` as type:

  ```
  sqlite-utils transform species.sqlite species --type criteria json
  Usage: sqlite-utils transform [OPTIONS] PATH TABLE
  Try 'sqlite-utils transform --help' for help.

  Error: Invalid value for '--type': 'json' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/269#issuecomment-860031217
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/269
node_id: MDEyOklzc3VlQ29tbWVudDg2MDAzMTIxNw==
created_at: 2021-06-12T10:01:53Z
updated_at: 2021-06-12T10:01:53Z
author_association: NONE
body:
  `sqlite-utils transform` does not allow setting the column type to boolean:
  ```
  Error: Invalid value for '--type': 'bool' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1286#issuecomment-860047794
issue_url: https://api.github.com/repos/simonw/datasette/issues/1286
node_id: MDEyOklzc3VlQ29tbWVudDg2MDA0Nzc5NA==
created_at: 2021-06-12T12:36:15Z
updated_at: 2021-06-12T12:36:15Z
author_association: NONE
body:
  @mroswell That is a very nice solution. I wonder if custom classes, like `col-columnName-value` could be automatically added to cells when facets on such column are enabled, to allow custom styling without having to modify templates or add custom JavaScript code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-860063190
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57
node_id: MDEyOklzc3VlQ29tbWVudDg2MDA2MzE5MA==
created_at: 2021-06-12T14:46:44Z
updated_at: 2021-06-12T14:46:44Z
author_association: NONE
body:
  I'm having the same issue (same versions of python and twitter-to-sqlite). It's the `user-timeline` command. Other commands are working. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/271#issuecomment-860142489
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/271
node_id: MDEyOklzc3VlQ29tbWVudDg2MDE0MjQ4OQ==
created_at: 2021-06-13T02:53:06Z
updated_at: 2021-06-13T02:53:06Z
author_association: OWNER
body:
  Looks like this is the problem: https://github.com/simonw/sqlite-utils/blob/b0f9d1e494c9891ce407e27b0f5c6deeea361d30/sqlite_utils/db.py#L1724-L1742

  Note how `set_cols = [col for col in all_columns if col not in pks] ` can potentially return an empty list if ALL of the columns are primary keys - but the next line of code that assigns `sql2` continues regardless, when it should instead be skipped if there are no columns in `set_cols`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1376#issuecomment-860229226
issue_url: https://api.github.com/repos/simonw/datasette/issues/1376
node_id: MDEyOklzc3VlQ29tbWVudDg2MDIyOTIyNg==
created_at: 2021-06-13T15:29:45Z
updated_at: 2021-06-13T15:29:45Z
author_association: OWNER
body:
  Oh good catch - this is a SQLite version issue.

  The `fixtures.db` file used on https://latest.datasette.io/ includes a generated column (for testing purposes) which is a feature added in SQLite 3.31.0 on 2020-01-22.

  https://latest.datasette.io/-/versions

  But... it looks like the packaged Datasette Docker container doesn't have that SQLite version!

  I should fix that. I'm renaming this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1376#issuecomment-860229397
issue_url: https://api.github.com/repos/simonw/datasette/issues/1376
node_id: MDEyOklzc3VlQ29tbWVudDg2MDIyOTM5Nw==
created_at: 2021-06-13T15:31:02Z
updated_at: 2021-06-13T15:31:02Z
author_association: OWNER
body:
  Alternative fix would be to update that section of the documentation - if the container upgrade proves tricky I can fall back on that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1375#issuecomment-860230385
issue_url: https://api.github.com/repos/simonw/datasette/issues/1375
node_id: MDEyOklzc3VlQ29tbWVudDg2MDIzMDM4NQ==
created_at: 2021-06-13T15:37:49Z
updated_at: 2021-06-13T15:37:49Z
author_association: OWNER
body:
  There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
  Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

  I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning  string instead of the expected dictionary and breaking your code.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1375#issuecomment-860548546
issue_url: https://api.github.com/repos/simonw/datasette/issues/1375
node_id: MDEyOklzc3VlQ29tbWVudDg2MDU0ODU0Ng==
created_at: 2021-06-14T09:41:59Z
updated_at: 2021-06-14T09:41:59Z
author_association: NONE
body:
  > There is a feature for this at the moment, but it's a little bit hidden: you can use `?_json=col` to tell
  > Datasette that you would like a specific column to be exported as nested JSON: https://docs.datasette.io/en/stable/json_api.html#special-json-arguments

  Thanks :)
   
  > I considered trying to make this automatic - so it detects columns that appear to contain valid JSON and outputs them as nested objects - but the problem with that is that it can lead to inconsistent results - you might hit the API and find that not every column contains valid JSON (compared to the previous day) resulting in the API retuning string instead of the expected dictionary and breaking your code.

  If a developer is not sure if the JSON fields are valid, but then retrieves and parse them, it should handle errors too. Handling inconsistent data is necessary due to the nature of SQLite. A global or dataset option to render the data as they have been defined (JSON, boolean, etc.) when requesting JSON could allow the user to download a regular JSON from the browser without having to rely on APIs. I would guess someone could just make a custom template with an extra JSON-parsed download button otherwise :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-860895838
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64
node_id: MDEyOklzc3VlQ29tbWVudDg2MDg5NTgzOA==
created_at: 2021-06-14T18:23:21Z
updated_at: 2021-06-14T21:37:35Z
author_association: NONE
body:
  i have a basic working version at https://github.com/khimaros/github-to-sqlite

  this can be tested with `github-to-sqlite events.db khimaros/events`

  caveat: the GitHub API doesn't seem to provide a complete history of events.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861035862
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64
node_id: MDEyOklzc3VlQ29tbWVudDg2MTAzNTg2Mg==
created_at: 2021-06-14T22:29:20Z
updated_at: 2021-06-14T22:29:20Z
author_association: NONE
body:
  it looks like the v4 GraphQL API is the only way to get data beyond 90 days from GitHub.

  this is significant change, but may be worth considering in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861041597
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64
node_id: MDEyOklzc3VlQ29tbWVudDg2MTA0MTU5Nw==
created_at: 2021-06-14T22:44:54Z
updated_at: 2021-06-14T22:44:54Z
author_association: MEMBER
body:
  Have you found a way to access events in GraphQL? I can only see way to access a timeline of events for a single issue or a single pull request. See also https://github.community/t/get-event-equivalent-for-v4/13600/2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/64#issuecomment-861087651
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64
node_id: MDEyOklzc3VlQ29tbWVudDg2MTA4NzY1MQ==
created_at: 2021-06-15T00:48:37Z
updated_at: 2021-06-15T00:48:37Z
author_association: NONE
body:
  @simonw -- i've created an omega-query that fetched most of what was interesting to me for a single user.

  found by poking around in the "Explorer" tab in https://docs.github.com/en/graphql/overview/explorer

  note: pagination is still required via `first` and `last` but it seems to allow unlimited history.

  ```
  query MyQuery {
    __typename
    user(login: "<user>") {
      id
      pinnedItems(first: 100) {
        edges {
          node
        }
      }
      pullRequests(first: 100) {
        nodes {
          body
          title
          state
          createdAt
        }
      }
      createdAt
      issues(first: 100) {
        pageInfo {
          endCursor
          startCursor
        }
        nodes {
          title
          url
          createdAt
          body
        }
      }
      issueComments(first: 100) {
        edges {
          node {
            id
            updatedAt
            url
            body
          }
        }
      }
      repositories(first: 100) {
        nodes {
          createdAt
          description
          parent {
            name
          }
          pinnedIssues(first: 100) {
            edges {
              node {
                id
              }
            }
          }
          pinnedDiscussions(first: 100) {
            edges {
              node {
                id
              }
            }
          }
        }
      }
      starredRepositories(first: 100) {
        edges {
          node {
            id
          }
        }
      }
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1377#issuecomment-861087949
issue_url: https://api.github.com/repos/simonw/datasette/issues/1377
node_id: MDEyOklzc3VlQ29tbWVudDg2MTA4Nzk0OQ==
created_at: 2021-06-15T00:49:19Z
updated_at: 2021-06-15T00:49:19Z
author_association: OWNER
body:
  The new `skip_if_scope` mechanism in `asgi-csrf` https://github.com/simonw/asgi-csrf/issues/20 is designed to help here.

  Now I need to design a plugin hook that allows plugins to have an opinion on whether a specific `scope` should have CSRF skipped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1377#issuecomment-861089794
issue_url: https://api.github.com/repos/simonw/datasette/issues/1377
node_id: MDEyOklzc3VlQ29tbWVudDg2MTA4OTc5NA==
created_at: 2021-06-15T00:53:29Z
updated_at: 2021-06-15T00:53:29Z
author_association: OWNER
body:
  Potential hook names:

  - `skip_csrf(scope, datasette)`
  - ... I can't think of any other ones I would tolerate to be honest

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/269#issuecomment-861103967
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/269
node_id: MDEyOklzc3VlQ29tbWVudDg2MTEwMzk2Nw==
created_at: 2021-06-15T01:34:10Z
updated_at: 2021-06-15T01:34:10Z
author_association: OWNER
body:
  SQLite doesn't have the concept of a boolean column, so there's not much I can do here: https://www.sqlite.org/datatype3.html#boolean_datatype

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1130#issuecomment-861497548
issue_url: https://api.github.com/repos/simonw/datasette/issues/1130
node_id: MDEyOklzc3VlQ29tbWVudDg2MTQ5NzU0OA==
created_at: 2021-06-15T13:27:48Z
updated_at: 2021-06-15T13:27:48Z
author_association: CONTRIBUTOR
body:
  There's a workaround: https://css-tricks.com/css-fix-for-100vh-in-mobile-webkit/

  and a future fix: https://css-tricks.com/safari-15-new-ui-theme-colors-and-a-css-tricks-cameo/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861889437
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTg4OTQzNw==
created_at: 2021-06-15T23:03:26Z
updated_at: 2021-06-15T23:03:26Z
author_association: OWNER
body:
  Maybe also support `--csvt` as an alternative option which takes two arguments: the CSV path and the name of the table that should be created from it (rather than auto-detecting from the filename).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861890689
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTg5MDY4OQ==
created_at: 2021-06-15T23:06:37Z
updated_at: 2021-06-15T23:06:37Z
author_association: OWNER
body:
  How about `--json` and `--nl` and `--tsv` too? Imitating the format options for `sqlite-utils insert`.

  And what happens if you provide a filename too? I'm tempted to say that the `--csv` stuff still gets loaded into an in-memory database but it's given a name and can then be joined against using SQLite `memory.blah` syntax.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891110
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTg5MTExMA==
created_at: 2021-06-15T23:07:38Z
updated_at: 2021-06-15T23:07:38Z
author_association: OWNER
body:
  `--csvt` seems unnecessary to me: if people want to load different CSV files with the same filename (but in different directories) they will get an error unless they rename the files first.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861891693
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTg5MTY5Mw==
created_at: 2021-06-15T23:09:08Z
updated_at: 2021-06-15T23:09:08Z
author_association: OWNER
body:
  Problem: `--csv` and `--json` and `--nl` are already options for `sqlite-utils query` - need new non-conflicting names.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861944202
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTk0NDIwMg==
created_at: 2021-06-16T01:41:03Z
updated_at: 2021-06-16T01:41:03Z
author_association: CONTRIBUTOR
body:
  So, I do things like this a lot, too. I like the idea of piping in from stdin. Something like this would be nice to do in a makefile:

  ```sh
  cat file.csv | sqlite-utils --csv --table data - 'SELECT * FROM data WHERE col="whatever"' > filtered.csv
  ```

  If you assumed that you're always piping out the same format you're piping in, the option names don't have to change. Depends how much you want to change formats.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861984707
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTk4NDcwNw==
created_at: 2021-06-16T02:19:48Z
updated_at: 2021-06-16T02:19:48Z
author_association: OWNER
body:
  This is going to need to be a separate command, for relatively non-obvious reasons.

      sqlite-utils blah.db "select * from x"

  Is equivalent to this, because `query` is the default sub-command:

      sqlite-utils query blah.db "select * from x"

  But... this means that making the filename optional doesn't actually work - because then this is ambiguous:

      sqlite-utils --load-csv blah.csv "select * from blah"

  So instead, I'm going to add a new sub-command. I'm currently thinking `memory` to reflect that this command operates on an in-memory database:

      sqlite-utils memory --load-csv blah.csv "select * from blah"

  I still think I need to use `--load-csv` rather than `--csv` because one interesting use-case for this is loading in CSV and converting it to JSON, or vice-versa.

  Another option: allow multiple arguments which are filenames, and use the extension (or sniff the content) to decide what to do with them:

      sqlite-utils memory blah.csv foo.csv "select * from foo join blah on ..."

  This would require the last positional argument to always be a SQL query, and would treat all other positional arguments as files that should be imported into memory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTk4NTk0NA==
created_at: 2021-06-16T02:22:52Z
updated_at: 2021-06-16T02:22:52Z
author_association: OWNER
body:
  Another option: allow an optional `:suffix` specifying the type of the file. If this is missing we detect based on the filename.

      sqlite-utils memory somefile:csv "select * from somefile"

  One catch: how to treat `-` for standard input?

      cat blah.csv | sqlite-utils memory - "select * from stdin"

  That's fine for CSV, but what about TSV or JSON or nl-JSON? Maybe this:

      cat blah.csv | sqlite-utils memory -:json "select * from stdin"

  Bit weird though. The alternative would be to support this:

      cat blah.csv | sqlite-utils memory --load-csv -

  But that's verbose compared to the version without the long `--load-x` option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861987651
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTk4NzY1MQ==
created_at: 2021-06-16T02:27:20Z
updated_at: 2021-06-16T02:27:20Z
author_association: OWNER
body:
  Solution: `sqlite-utils memory -` attempts to detect the input based on if it starts with a `{` or `[` (likely JSON) or if it doesn't use the `csv.Sniffer()` mechanism. Or you can use `sqlite-utils memory -:csv` to specifically indicate the type of input.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861989987
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MTk4OTk4Nw==
created_at: 2021-06-16T02:34:21Z
updated_at: 2021-06-16T02:34:21Z
author_association: OWNER
body:
  The documentation already covers this
  ```
  $ sqlite-utils :memory: "select sqlite_version()"
  [{"sqlite_version()": "3.29.0"}]
  ```
  https://sqlite-utils.datasette.io/en/latest/cli.html#running-queries-and-returning-json

  `sqlite-utils memory "select sqlite_version()"` is a little bit more intuitive than that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjAxODkzNw==
created_at: 2021-06-16T03:59:28Z
updated_at: 2021-06-16T04:00:05Z
author_association: OWNER
body:
  Mainly for debugging purposes it would be useful to be able to save the created in-memory database back to a file again later. This could be done with:

      sqlite-utils memory blah.csv --save saved.db

  Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump

  Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862040906
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjA0MDkwNg==
created_at: 2021-06-16T05:02:47Z
updated_at: 2021-06-16T05:02:47Z
author_association: OWNER
body:
  Got a prototype working!
  ```
   % curl -s 'https://fivethirtyeight.datasettes.com/polls/president_approval_polls.csv?_size=max&_stream=1' | sqlite-utils memory - 'select * from t limit 5' --nl 
  {"rowid": "1", "question_id": "139304", "poll_id": "74225", "state": "", "politician_id": "11", "politician": "Donald Trump", "pollster_id": "568", "pollster": "YouGov", "sponsor_ids": "352", "sponsors": "Economist", "display_name": "YouGov", "pollster_rating_id": "391", "pollster_rating_name": "YouGov", "fte_grade": "B", "sample_size": "1500", "population": "a", "population_full": "a", "methodology": "Online", "start_date": "1/16/21", "end_date": "1/19/21", "sponsor_candidate": "", "tracking": "", "created_at": "1/20/21 10:18", "notes": "", "url": "https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf", "source": "538", "yes": "42.0", "no": "53.0"}
  {"rowid": "2", "question_id": "139305", "poll_id": "74225", "state": "", "politician_id": "11", "politician": "Donald Trump", "pollster_id": "568", "pollster": "YouGov", "sponsor_ids": "352", "sponsors": "Economist", "display_name": "YouGov", "pollster_rating_id": "391", "pollster_rating_name": "YouGov", "fte_grade": "B", "sample_size": "1155", "population": "rv", "population_full": "rv", "methodology": "Online", "start_date": "1/16/21", "end_date": "1/19/21", "sponsor_candidate": "", "tracking": "", "created_at": "1/20/21 10:18", "notes": "", "url": "https://docs.cdn.yougov.com/y9zsit5bzd/weeklytrackingreport.pdf", "source": "538", "yes": "44.0", "no": "55.0"}
  {"rowid": "3", "question_id": "139306", "poll_id": "74226", "state": "", "politician_id": "11", "politician": "Donald Trump", "pollster_id": "23", "pollster": "American Research Group", "sponsor_ids": "", "sponsors": "", "display_name": "American Research Group", "pollster_rating_id": "9", "pollster_rating_name": "American Research Group", "fte_grade": "B", "sample_size": "1100", "population": "a", "population_full": "a", "methodology": "Live Phone", "start_date": "1/16/21", "end_date": "1/19/21", "sponsor_candidate": "", "tracking": "", "created_at": "1/20/21 10:18", "notes": "", "url": "https://americanresearchgroup.com/economy/", "source": "538", "yes": "30.0", "no": "66.0"}
  {"rowid": "4", "question_id": "139307", "poll_id": "74226", "state": "", "politician_id": "11", "politician": "Donald Trump", "pollster_id": "23", "pollster": "American Research Group", "sponsor_ids": "", "sponsors": "", "display_name": "American Research Group", "pollster_rating_id": "9", "pollster_rating_name": "American Research Group", "fte_grade": "B", "sample_size": "990", "population": "rv", "population_full": "rv", "methodology": "Live Phone", "start_date": "1/16/21", "end_date": "1/19/21", "sponsor_candidate": "", "tracking": "", "created_at": "1/20/21 10:18", "notes": "", "url": "https://americanresearchgroup.com/economy/", "source": "538", "yes": "29.0", "no": "67.0"}
  {"rowid": "5", "question_id": "139298", "poll_id": "74224", "state": "", "politician_id": "11", "politician": "Donald Trump", "pollster_id": "1528", "pollster": "AtlasIntel", "sponsor_ids": "", "sponsors": "", "display_name": "AtlasIntel", "pollster_rating_id": "546", "pollster_rating_name": "AtlasIntel", "fte_grade": "B/C", "sample_size": "5188", "population": "a", "population_full": "a", "methodology": "Online", "start_date": "1/15/21", "end_date": "1/19/21", "sponsor_candidate": "", "tracking": "", "created_at": "1/19/21 21:52", "notes": "", "url": "https://projects.fivethirtyeight.com/polls/20210119_US_Atlas2.pdf", "source": "538", "yes": "44.6", "no": "53.9"}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862042110
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDg2MjA0MjExMA==
created_at: 2021-06-16T05:05:51Z
updated_at: 2021-06-16T05:06:11Z
author_association: OWNER
body:
  Initial documentation is here: https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#running-queries-directly-against-csv-data

  It only talks about CSV at the moment - needs to be updated to mention JSON too once that is implemented.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862043974
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDg2MjA0Mzk3NA==
created_at: 2021-06-16T05:10:12Z
updated_at: 2021-06-16T05:10:12Z
author_association: OWNER
body:
  I can stop promoting `:memory:` here and promote `memory` instead:

  https://github.com/simonw/sqlite-utils/blob/c7234cae8336b8525034e8f917d82dd0699abd42/docs/cli.rst#L83-L86

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045438
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDg2MjA0NTQzOA==
created_at: 2021-06-16T05:14:00Z
updated_at: 2021-06-16T05:14:00Z
author_association: OWNER
body:
  I should probably refactor the CSV/JSON/loading stuff into a function in `utils.py` in order to share some of the implementation with the existing `sqlite-utils insert` code: https://github.com/simonw/sqlite-utils/blob/287cdcae8908916687f2ecccc87c38549d004ac6/sqlite_utils/cli.py#L691-L734

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862045639
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDg2MjA0NTYzOQ==
created_at: 2021-06-16T05:14:38Z
updated_at: 2021-06-16T05:14:38Z
author_association: OWNER
body:
  Can't share much code though since a bunch of that `insert` stuff is specific to that command - showing progress bars, returning errors on illegal option combinations etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862046009
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDg2MjA0NjAwOQ==
created_at: 2021-06-16T05:15:38Z
updated_at: 2021-06-16T05:15:38Z
author_association: OWNER
body:
  I'm going to add a `--encoding` option - it will affect ALL CSV input files, so if you have CSV files with different encodings you'll need to sort that mess out yourself (likely by importing each CSV file separately into a database using `sqlite-utils insert` with different `--encoding` values).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862475685
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ3NTY4NQ==
created_at: 2021-06-16T15:26:19Z
updated_at: 2021-06-16T15:29:38Z
author_association: OWNER
body:
  Here's a radical idea: what if I combined `sqlite-utils memory` into `sqlite-utils query`?

  The trick here would be to detect if the arguments passed on the command-line refer to SQLite databases or if they refer to CSV/JSON data that should be imported into temporary tables.

  Detecting a SQLite database file is actually really easy - they all start with the same binary string:
  ```pycon
  >>> open("my.db", "rb").read(100)
  b'SQLite format 3\x00...
  ```
  (Need to carefully check that a CSV file with`SQLite format 3` as the first column name doesn't accidentally get interpreted as a SQLite DB though).

  So then what would the semantics of `sqlite-utils query` (which is also the default command) be?

  - `sqlite-utils mydb.db "select * from x"`
  - `sqlite-utils my.csv "select * from my"`
  - `sqlite-utils mydb.db my.csv "select * from mydb.x join my on ..."` - this is where it gets weird. We can't import the CSV data directly into `mpdb.db` - it's suppose to go into the in-memory database - so now we need to start using database aliases like `mydb.x` because we passed at least one other file?

  The complexity here is definitely in the handling of a combination of SQLite database files and CSV filenames. Also, `sqlite-utils query` doesn't accept multiple filenames at the moment, so that will change.

  I'm not 100% sold on this as being better than having a separate `sqlite-utils memory` command, as seen in #273.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862478881
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ3ODg4MQ==
created_at: 2021-06-16T15:30:24Z
updated_at: 2021-06-16T15:30:24Z
author_association: OWNER
body:
  But... `sqlite-utils my.csv "select * from my"` is a much more compelling initial experience than `sqlite-utils memory my.csv "select * from my"`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862479704
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ3OTcwNA==
created_at: 2021-06-16T15:31:31Z
updated_at: 2021-06-16T15:31:31Z
author_association: OWNER
body:
  Plus, could I make this change to `sqlite-utils query` without breaking backwards compatibility? Adding a new `sqlite-utils memory` command is completely safe from that perspective.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862484557
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ4NDU1Nw==
created_at: 2021-06-16T15:37:51Z
updated_at: 2021-06-16T15:38:34Z
author_association: OWNER
body:
  I wonder if there's a better name for this than `sqlite-utils memory`?

  - `sqlite-utils memory hello.csv "select * from hello"`
  - `sqlite-utils mem hello.csv "select * from hello"`
  - `sqlite-utils temp hello.csv "select * from hello"`
  - `sqlite-utils adhoc hello.csv "select * from hello"`
  - `sqlite-utils scratch hello.csv "select * from hello"`

  I think `memory` is best. I don't like the others, except for `scratch` which is OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862485408
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ4NTQwOA==
created_at: 2021-06-16T15:38:58Z
updated_at: 2021-06-16T15:39:28Z
author_association: OWNER
body:
  Also `sqlite-utils memory` reflects the existing `sqlite-utils :memory:` mechanism, which is a point in its favour.

  And it helps emphasize that the file you are querying will be loaded into memory, so probably don't try this against a 1GB CSV file.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862491016
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/272
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ5MTAxNg==
created_at: 2021-06-16T15:46:13Z
updated_at: 2021-06-16T15:46:13Z
author_association: OWNER
body:
  Columns from data imported from CSV in this way is currently treated as `TEXT`, which means numeric sorts and suchlike won't work as people might expect. It would be good to do automatic type detection here, see #179.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862491721
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/270
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ5MTcyMQ==
created_at: 2021-06-16T15:47:06Z
updated_at: 2021-06-16T15:47:06Z
author_association: OWNER
body:
  SQLite doesn't have a JSON column type - it has JSON processing functions, but they operate against TEXT columns - so there's nothing I can do here unfortunately.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862493179
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/267
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ5MzE3OQ==
created_at: 2021-06-16T15:49:13Z
updated_at: 2021-06-16T15:49:13Z
author_association: OWNER
body:
  The big challenge here is that the rows returned  by this library aren't objects, they are Python dictionaries - so adding methods to them isn't possible without changing the type that is returned by these methods.

  Part of the philosophy of the library is that it should make it as easy as possible to round-trip between Python dictionaries and SQLite table data, so I don't think adding methods like this is going to fit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/267#issuecomment-862494864
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/267
node_id: MDEyOklzc3VlQ29tbWVudDg2MjQ5NDg2NA==
created_at: 2021-06-16T15:51:28Z
updated_at: 2021-06-16T16:26:15Z
author_association: OWNER
body:
  I did add a slightly clumsy mechanism recently to help a bit here though: the `pks_and_rows_where()` method: https://sqlite-utils.datasette.io/en/stable/python-api.html#listing-rows-with-their-primary-keys

  More details in the issue for that feature: #240

  The idea here is that if you want to call update you need the primary key for the row - so you can do this:

  ```python
  for pk, row in db["sometable"].pks_and_rows_where():
      db["sometable"].update(pk, {"modified": 1}")
  ```
  The `pk` may end up as a single value or a tuple depending on if the table has a compound primary key - but you don't need to worry about that if you use this method as it will return the correct primary key value for you.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/270#issuecomment-862574390
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/270
node_id: MDEyOklzc3VlQ29tbWVudDg2MjU3NDM5MA==
created_at: 2021-06-16T17:34:49Z
updated_at: 2021-06-16T17:34:49Z
author_association: NONE
body:
  Sorry, I got confused because SQLite has a JSON column type, even if it is treated as TEXT, and I though automatic facets were available for JSON arrays stored as JSON only :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862605436
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDg2MjYwNTQzNg==
created_at: 2021-06-16T18:19:05Z
updated_at: 2021-06-16T18:19:05Z
author_association: OWNER
body:
  `--attach` documentation: https://github.com/simonw/sqlite-utils/blob/192dc2c5b73bd836ab8e2e5fed4b36c6ea02f250/docs/cli.rst#joining-in-memory-data-against-existing-databases-using-attach

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/275#issuecomment-862617165
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/275
node_id: MDEyOklzc3VlQ29tbWVudDg2MjYxNzE2NQ==
created_at: 2021-06-16T18:34:51Z
updated_at: 2021-06-16T18:34:51Z
author_association: OWNER
body:
  Also use this: https://github.com/simonw/datasette/blob/83e9c8bc7585dcc62f200e37c2daefcd669ee05e/codecov.yml

  And add a badge, as seen on https://github.com/simonw/asgi-csrf

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/273#issuecomment-862817185
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/273
node_id: MDEyOklzc3VlQ29tbWVudDg2MjgxNzE4NQ==
created_at: 2021-06-17T00:15:34Z
updated_at: 2021-06-17T00:15:34Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > :exclamation: No coverage uploaded for pull request base (`main@78aebb6`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#section-missing-base-commit).
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/273/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #273   +/-   ##
  =======================================
    Coverage        ?   96.10%           
  =======================================
    Files           ?        4           
    Lines           ?     1873           
    Branches        ?        0           
  =======================================
    Hits            ?     1800           
    Misses          ?       73           
    Partials        ?        0           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [78aebb6...df7a37b](https://codecov.io/gh/simonw/sqlite-utils/pull/273?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/277#issuecomment-863205049
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/277
node_id: MDEyOklzc3VlQ29tbWVudDg2MzIwNTA0OQ==
created_at: 2021-06-17T12:40:49Z
updated_at: 2021-06-17T12:40:49Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#277](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (abbd324) into [main](https://codecov.io/gh/simonw/sqlite-utils/commit/a19ce1a4d0048d389411cfe11a5dbe4c503720e1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a19ce1a) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/sqlite-utils/pull/277/graphs/tree.svg?width=650&height=150&src=pr&token=O0X3703L9P&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #277   +/-   ##
  =======================================
    Coverage   96.06%   96.06%           
  =======================================
    Files           4        4           
    Lines        1828     1829    +1     
  =======================================
  + Hits         1756     1757    +1     
    Misses         72       72           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/277/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `94.03% <100.00%> (+<0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a19ce1a...abbd324](https://codecov.io/gh/simonw/sqlite-utils/pull/277?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1378#issuecomment-863230355
issue_url: https://api.github.com/repos/simonw/datasette/issues/1378
node_id: MDEyOklzc3VlQ29tbWVudDg2MzIzMDM1NQ==
created_at: 2021-06-17T13:16:17Z
updated_at: 2021-06-17T13:16:17Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  > Merging [#1378](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0c132d1) into [main](https://codecov.io/gh/simonw/datasette/commit/83e9c8bc7585dcc62f200e37c2daefcd669ee05e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (83e9c8b) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1378/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1378   +/-   ##
  =======================================
    Coverage   91.68%   91.68%           
  =======================================
    Files          34       34           
    Lines        4340     4340           
  =======================================
    Hits         3979     3979           
    Misses        361      361           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [83e9c8b...0c132d1](https://codecov.io/gh/simonw/datasette/pull/1378?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864103005
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDg2NDEwMzAwNQ==
created_at: 2021-06-18T15:04:15Z
updated_at: 2021-06-18T15:04:15Z
author_association: OWNER
body:
  To detect JSON, check to see if the stream starts with `[` or `{` - maybe do something more sophisticated than that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864129273
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDg2NDEyOTI3Mw==
created_at: 2021-06-18T15:47:47Z
updated_at: 2021-06-18T15:47:47Z
author_association: OWNER
body:
  Detecting valid JSON is tricky - just because a stream starts with `[` or `{` doesn't mean the entire stream is valid JSON. You need to parse the entire stream to determine that for sure.

  One way to solve this would be with a custom state machine. Another would be to use the `ijson` streaming parser - annoyingly it throws the same exception class for invalid JSON for different reasons, but the `e.args[0]` for that exception includes human-readable text about the error - if it's anything other than `parse error: premature EOF` then it probably means the JSON was invalid.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864206308
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/279
node_id: MDEyOklzc3VlQ29tbWVudDg2NDIwNjMwOA==
created_at: 2021-06-18T18:25:04Z
updated_at: 2021-06-18T18:25:04Z
author_association: OWNER
body:
  Or... since I'm not using a streaming JSON parser at the moment, if I think something is JSON I can load the entire thing into memory to validate it.

  I still need to detect newline-delimited JSON. For that I can consume the first line of the input to see if it's a valid JSON object, then maybe sniff the second line too?

  This does mean that if the input is a single line of GIANT JSON it will all be consumed into memory at once, but that's going to happen anyway.

  So I need a function which, given a file pointer, consumes from it, detects the type, then returns that type AND a file pointer to the beginning of the file again. I can use `io.BufferedReader` for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


