# issues

node_id: MDU6SXNzdWU4MDgwMzY3NzQ=
title: Run tests against Windows in GitHub Actions
state: closed
created_at: 2021-02-14T20:09:45Z
updated_at: 2021-02-14T20:39:55Z
closed_at: 2021-02-14T20:39:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'm going to try and get the test suite to run in Windows on GitHub Actions.

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/225#issuecomment-778834504_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/232/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTczMTQ3MTY4
title: Run tests against Ubuntu, macOS and Windows
state: closed
created_at: 2021-02-14T20:11:02Z
updated_at: 2021-02-14T20:39:54Z
closed_at: 2021-02-14T20:39:54Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/233
body: Refs #232
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/233/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDgwNDY1OTc=
title: .insert_all() fails if subsequent chunks contain additional columns
state: closed
created_at: 2021-02-14T21:01:51Z
updated_at: 2021-02-14T21:03:40Z
closed_at: 2021-02-14T21:03:40Z
author_association: OWNER
body: Reported by @nieuwenhoven in #225 along with a proposed fix.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/234/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDg3NzE2OTA=
title: More flexible formatting of records with CSS grid
state: open
created_at: 2021-02-15T19:28:17Z
updated_at: 2021-02-15T19:28:35Z
author_association: NONE
type: issue
body:
  In several applications I've been experimenting with alternate formatting of datasette query results. Lately I've found that CSS grids work very well and seem quite general for formatting rows. In CSS I use grid templates to define the layout of each record and the regions for each field, hiding the fields I don't want. It's pretty flexible and looks good. It's also a great basis for highly responsive layout.

  I initially thought I'd only use this feature for record detail views, but now I use it for index views as well. 

  However, there are some limitations:
  * With the existing table templates, it seems that you can change the `display` property on the enclosing `table`, `tbody`, and `tr` to make them be grid-like, but that seems hacky (convert `table` and `tbody` to be `display: block` and `tr` to be `display: grid`).
  * More significantly, it's very nice to have the column name available when rendering each record to display headers/field labels. The existing templates don't do that, so a custom `_table` template is necessary.
  * I don't know if any plugins are sensitive to whether data is rendered as a table or not since I'm not completely clear how plugins get their data. 
  * Regardless, you need custom CSS to take full advantage of grids. I don't have a proposal on how to integrate them more deeply.

  It would be helpful to at least have an official example or test that used a grid layout for records to make sure nothing in datasette breaks with it. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1225/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDg4NDM0MDE=
title: --port option should validate port is between 0 and 65535
state: closed
created_at: 2021-02-15T22:01:33Z
updated_at: 2021-02-18T18:41:27Z
closed_at: 2021-02-18T18:41:27Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Currently throws an ugly error message:
  ```
  (datasette-graphql) datasette-graphql % datasette fivethirtyeight.db -p 80094
  INFO:     Started server process [45497]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/datasette-graphql-n1OSJCS8/bin/datasette", line 8, in <module>
      sys.exit(cli())
  ...
      server = await loop.create_server(
    File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py", line 1461, in create_server
      sock.bind(sa)
  OverflowError: bind(): port must be 0-65535.
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1226/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTAzOTQ2MTY=
title: Configure sphinx.ext.extlinks for issues
state: closed
created_at: 2021-02-17T17:38:02Z
updated_at: 2021-08-02T21:38:39Z
closed_at: 2021-02-18T01:20:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Spotted this in the aspw documentation: https://github.com/rogerbinns/apsw/blob/3.34.0-r1/doc/conf.py#L29-L36

  ```python
  extlinks={
      'cvstrac': ('https://sqlite.org/cvstrac/tktview?tn=%s',
                  'SQLite ticket #'),
      'sqliteapi': ('https://sqlite.org/c3ref/%s.html', 'XXYouShouldNotSeeThisXX'),
      'issue': ('https://github.com/rogerbinns/apsw/issues/%s',
                'APSW issue '),
      'source': ('https://github.com/rogerbinns/apsw/blob/master/%s',
                 ''),
      }
  ```
  Which lets you link to issues like this:

      :issue:`268`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1227/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTAzOTcwMjU=
title: 500 error caused by faceting if a column called `n` exists
state: closed
created_at: 2021-02-17T17:41:20Z
updated_at: 2022-03-19T06:44:40Z
closed_at: 2022-03-19T01:38:04Z
author_association: NONE
type: issue
state_reason: completed
body:
  I recently discovered `datasette` thanks to your great talk at FOSDEM and would like to use it for some projects. However, when trying to use it on databases created from some csv ot tsv files, I am sometimes getting this issue when going to http://127.0.0.1:8001/databasetest/databasetest and I don't exactly understand what it refers to.

  So far, I couldn't find anything relevant when reviewing the raw text files that could explain this issue, nor could I find something obvious between the files that generate this issue and those that don't. Does the error ring a bell and, if so, could you please point me to the right direction?

  ```
  $ datasette databasetest.db 
  INFO:     Started server process [1408482]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
  INFO:     127.0.0.1:56394 - "GET / HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56394 - "GET /-/static/app.css?4e362c HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56396 - "GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56398 - "GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1" 200 OK
  Traceback (most recent call last):
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/app.py", line 1099, in route_path
      response = await view(request, send)
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 147, in view
      request, **request.scope["url_route"]["kwargs"]
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 121, in dispatch_request
      return await handler(request, *args, **kwargs)
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 260, in get
      request, database, hash, correct_hash_provided, **kwargs
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/base.py", line 434, in view_get
      request, database, hash, **kwargs
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/views/table.py", line 782, in data
      suggested_facets.extend(await facet.suggest())
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/facets.py", line 168, in suggest
      and any(r["n"] > 1 for r in distinct_values)
    File "/home/kabouik/.local/lib/python3.7/site-packages/datasette/facets.py", line 168, in <genexpr>
      and any(r["n"] > 1 for r in distinct_values)
  TypeError: '>' not supported between instances of 'str' and 'int'
  INFO:     127.0.0.1:56402 - "GET /databasetest/databasetest HTTP/1.1" 500 Internal Server Error
  INFO:     127.0.0.1:56402 - "GET /-/static/app.css?4e362c HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56404 - "GET / HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56404 - "GET /-/static/app.css?4e362c HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56406 - "GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56408 - "GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56408 - "GET /databasetest HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56408 - "GET /-/static/app.css?4e362c HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56404 - "GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56406 - "GET /-/static/codemirror-5.57.0.min.css HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56410 - "GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56414 - "GET /-/static/codemirror-5.57.0-sql.min.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56412 - "GET /-/static/codemirror-5.57.0.min.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56408 - "GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56408 - "GET /databasetest?sql=select+*+from+databasetest HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56410 - "GET /-/static/app.css?4e362c HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56408 - "GET /-/static-plugins/datasette_vega/main.2acbb312.css HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56412 - "GET /-/static/codemirror-5.57.0.min.css HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56404 - "GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56406 - "GET /-/static/codemirror-5.57.0.min.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56414 - "GET /-/static-plugins/datasette_vega/main.08f5d3d8.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56408 - "GET /-/static/codemirror-5.57.0-sql.min.js HTTP/1.1" 200 OK
  INFO:     127.0.0.1:56410 - "GET /databasetest.json?sql=select+*+from+databasetest&_shape=array&_shape=array HTTP/1.1" 200 OK
  ^CINFO:     Shutting down
  INFO:     Waiting for application shutdown.
  INFO:     Application shutdown complete.
  INFO:     Finished server process [1408482]
  ```

  Note that there is no error if I go to http://127.0.0.1:8001/databasetest and then click on `Run SQL`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1228/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTc1MTg3NDU3
title: ensure immutable databses when starting in configuration directory mode with
state: closed
created_at: 2021-02-17T20:18:26Z
updated_at: 2022-04-22T13:16:36Z
closed_at: 2021-03-29T00:17:32Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1229
type: pull
body:
  fixes #1224 

  This PR ensures all databases found in a configuration directory that match the files in `inspect-data.json` will be set to `immutable` as outlined in https://docs.datasette.io/en/latest/settings.html#configuration-directory-mode

  specifically on building the `datasette` instance it checks:
  - if `immutables` is an empty tuple - as passed by the cli code
  - if `immutables` is the default function value `None` - when it's not explicitly set

  And correctly builds the immutable database list from the `inspect-data[file]` keys.

  Note for this to work the `inspect-data.json` file must contain `file` paths which are relative to the configuration directory otherwise the file paths won't match and the dbs won't be set to immutable. 

  I couldn't find an easy way to test this due to the way `make_app_client` works, happy to take directions on adding a test for this. 

  I've updated the relevant docs as well, i.e. use the `inspect` cli cmd from the config directory path to create the relevant file
  ```
  cd $config_dir
  datasette inspect *.db --inspect-file=inspect-data.json
  ```
  https://docs.datasette.io/en/latest/performance.html#using-datasette-inspect

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1229/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTA2MTg0OTU=
state: closed
created_at: 2021-02-17T23:33:23Z
updated_at: 2023-06-26T01:47:01Z
closed_at: 2023-06-25T23:25:53Z
author_association: NONE
type: issue
state_reason: completed
title:
  Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified

body:
  Thanks for what seems like a truly great suite of libraries. I wanted to try out Datasette, but never got more than half way through your YouTube video with the SF tree dataset. Whenever I try to extract a column, I get a `sqlite3.OperationalError: table sqlite_master may not be modified` error from Python. This snippet reproduces the error on my system, Python 3.9.1 and sqlite-utils 3.5 on an M1 Macbook Pro running in rosetta mode:
  ```
  curl "https://data.nasa.gov/resource/y77d-th95.json" | \
      sqlite-utils insert meteorites.db meteorites - --pk=id
  sqlite-utils extract meteorites.db meteorites  recclass
  ```

  I have tried googling the problem, but all I've found is that this *might* be a problem with the sqlite3 database running in defensive mode, but I definitely can't know for sure. Does the problem seem familiar to you?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235/reactions", "total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTEwNTQwMDA=
state: open
created_at: 2021-02-18T12:27:02Z
updated_at: 2021-02-18T15:22:15Z
author_association: NONE
type: issue
title:
  Vega charts are plotted only for rows on the visible page, cluster maps only for rows in the remaining pages

body:
  I filtered a data set on some criteria and obtain 265 results, split over three pages (100, 100, 65), and reazlized that Vega plots are only applied to the results displayed on the current page, instead of the whole filtered data, _e.g._, 100 on page 1, 100 on page 2, 65 on page 3. Is there a way to force the graphs to consider all results instead of just the page, considering that pages rarely represent sensible information?

  Likewise, while the cluster map does show all results on the first page, if you go to next pages, it will show all remaining results except the previous page(s), _e.g._, 265 on page 1, 165 on page 2, 65 on page 3.

  In both cases, I don't see many situations where one would like to represent the data this way, and it might even lead to interpretation errors when viewing the data. Am I missing some cases where this would be best? Perhaps a clickable option to subset visual representations according visible pages _vs._ display all search results would do?

  [Edit] Oh, I just saw the "Load all" button under the cluster map as well as the [setting to alter the max number or results](https://docs.datasette.io/en/stable/settings.html#max-returned-rows). So I guess this issue only is about the Vega charts.


node_id: MDU6SXNzdWU4MTEzNjcyNTc=
title: Race condition errors in new refresh_schemas() mechanism
state: closed
created_at: 2021-02-18T18:49:54Z
updated_at: 2021-07-16T19:44:59Z
closed_at: 2021-07-16T19:44:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I tried running a Locust load test against Datasette and hit an error message about a failure to create tables because they already existed. I think this means there are race conditions in the new `refresh_schemas()` mechanism added in #1150.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1231/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTc1OTQwMTkz
title: --crossdb option for joining across databases
state: closed
created_at: 2021-02-18T19:48:50Z
updated_at: 2021-02-18T22:09:13Z
closed_at: 2021-02-18T22:09:12Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1232
type: pull
body:
  Refs #283. Still needs:

  - [x] Unit test for --crossdb queries
  - [x] Show warning on console if it truncates at ten databases (or on web interface)
  - [x] Show connected databases on the `/_memory` database page
  - [x] Documentation
  - [x] https://latest.datasette.io/ demo should demonstrate this feature

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1232/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTE0NTg0NDY=
title: "datasette publish cloudrun" cannot publish files with spaces in their name
state: open
created_at: 2021-02-18T21:08:31Z
updated_at: 2021-02-18T21:10:08Z
author_association: OWNER
type: issue
body:
  Got this error:
  ```
  Step 6/9 : RUN datasette inspect fixtures.db extra database.db --inspect-file inspect-data.json
   ---> Running in db9da0068592
  Usage: datasette inspect [OPTIONS] [FILES]...
  Try 'datasette inspect --help' for help.

  Error: Invalid value for '[FILES]...': Path 'extra' does not exist.
  The command '/bin/sh -c datasette inspect fixtures.db extra database.db --inspect-file inspect-data.json' returned a non-zero code: 2
  ERROR
  ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 2
  ```
  While working on the demo for #1232, using this deploy command:
  ```
  GITHUB_SHA=crossdb datasette publish cloudrun fixtures.db 'extra database.db' \
              -m fixtures.json \
              --plugins-dir=plugins \
              --branch=$GITHUB_SHA \
              --version-note=$GITHUB_SHA \
              --extra-options="--setting template_debug 1 --crossdb" \
              --install=pysqlite3-binary \
              --service=datasette-latest-crossdb
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1233/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTE1MDU2Mzg=
title: Runtime support for ATTACHing multiple databases
state: open
created_at: 2021-02-18T22:06:47Z
updated_at: 2021-02-22T21:06:28Z
author_association: OWNER
type: issue
body:
  > The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases.
  >
  > It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you.
  >
  > So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/283#issuecomment-781665560_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1234/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTE1ODkzNDQ=
title: Upgrade Python version used by official Datasette Docker image
state: closed
created_at: 2021-02-19T00:47:40Z
updated_at: 2021-02-19T01:48:31Z
closed_at: 2021-02-19T01:48:30Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Currently uses 3.7.2:

  https://github.com/simonw/datasette/blob/73bed175631a79e13a521eee82f8451dd0477eb3/Dockerfile#L1

  There's a security fix for Python which it would be good to ship in this image (even though I'm reasonably confident it doesn't affect Datasette): https://bugs.python.org/issue42938

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1235/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTE2ODA1MDI=
title: --attach command line option for attaching extra databases
state: closed
created_at: 2021-02-19T04:38:30Z
updated_at: 2021-02-19T05:10:41Z
closed_at: 2021-02-19T05:08:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This will enable cross-database joins, as seen in https://github.com/simonw/datasette/issues/283

  Also refs #113

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/236/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTIyMjgzMTQ=
title: Ability to increase size of the SQL editor window
state: closed
created_at: 2021-02-19T18:09:27Z
updated_at: 2021-05-18T03:28:25Z
closed_at: 2021-02-22T21:05:21Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1236/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTI3MDQ4Njk=
title: ?_pretty=1 option for pretty-printing JSON output
state: open
created_at: 2021-02-20T20:54:40Z
updated_at: 2021-11-16T18:28:33Z
author_association: OWNER
type: issue
body:
  Suggested by @frankieroberto in https://github.com/simonw/datasette/issues/782#issuecomment-782746755

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1237/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTc3OTUzNzI3
title: WIP: Add Gmail takeout mbox import
state: open
created_at: 2021-02-22T21:30:40Z
updated_at: 2021-07-28T07:18:56Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/google-takeout-to-sqlite/pulls/5
type: pull
body:
  WIP

  This PR adds the ability to import emails from a Gmail mbox export from Google Takeout.

  This is my first PR to a datasette/dogsheep repo. I've tested this on my personal Google Takeout mbox with ~520,000 emails going back to 2004. This took around ~20 minutes to process.

  To provide some feedback on the progress of the import I added the "rich" python module. I'm happy to remove that if adding a dependency is discouraged. However, I think it makes a nice addition to give feedback on the progress of a long import.

  Do we want to log emails that have errors when trying to import them?

  Dealing with encodings with emails is a bit tricky. I'm very open to feedback on how to deal with those better. As well as any other feedback for improvements.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTM4OTk0NzI=
title: Custom pages don't work with base_url setting
state: closed
created_at: 2021-02-22T21:58:58Z
updated_at: 2021-06-05T18:59:55Z
closed_at: 2021-06-05T18:59:55Z
author_association: NONE
type: issue
state_reason: completed
body:
  It seems that custom pages aren't routing properly when the `base_url` setting is used.

  To reproduce, with Datasette 0.55.

  Create a `templates/pages/custom.html` with some text.
  ```
  mkdir -p templates/pages/
  echo "Hello, world!" > templates/pages/custom.html
  ```

  Start Datasette.

  ```
  datasette --template-dir templates/
  ```

  Visit http://localhost:8001/custom and see "Hello, world!".

  Start Datasette with a `base_url`.

  ```
  datasette --template-dir templates/ --setting base_url /prefix/
  ```

  Visit http://localhost:8001/prefix/custom and see a "Database not found: custom" 404.

  Note that like all routes, http://localhost:8001/custom still works when run with `base_url`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1238/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTM5Nzg4NTg=
title: JSON filter fails if column contains spaces
state: closed
created_at: 2021-02-23T00:18:07Z
updated_at: 2021-02-23T00:22:53Z
closed_at: 2021-02-23T00:22:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Got this exception:

  `ERROR: conn=<sqlite3.Connection object at 0x10ea68e40>, sql = 'select Address, Affiliation, County, [Has Report], [Latest report notes], [Latest report yes], Latitude, [Location Type], Longitude, Name, id, [Appointment scheduling instructions], [Availability Info], [Latest report] from locations where rowid in (\n            select locations.rowid from locations, json_each(locations.Availability Info) j\n            where j.value = :p0\n        ) and "Latest report yes" = :p1 order by id limit 101', params = {'p0': 'Yes: appointment required', 'p1': '1'}: near "Info": syntax error`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1239/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTQ1OTE5NjI=
title: Allow facetting on custom queries
state: closed
created_at: 2021-02-23T15:52:19Z
updated_at: 2021-02-26T18:19:46Z
closed_at: 2021-02-26T18:18:18Z
author_association: NONE
type: issue
state_reason: completed
body:
  Facets are a tremendously useful feature, especially for people peeking at the database for the first time and still having little knowledge about the details of the data. It is of great assistance to discover interesting features to explore futher in advanced queries.

  Yet, it seems it's impossible to use facets when running a custom SQL query, be it from the little gear icons in column names, the facet suggestions at the top (hidden when performing a custom query), or by appending a facet code to the URL. 

  Is there a technical limitation, or is this something that could be unlocked easily?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1240/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTQ1OTUwMjE=
title: Share button for copying current URL
state: open
created_at: 2021-02-23T15:55:40Z
updated_at: 2023-08-24T20:09:52Z
author_association: NONE
type: issue
body:
  I use datasette in an `iframe` inside another HTML file that contains other ways to represent my data (mostly leaflets maps built with R on summarized data), and the datasette `iframe` is a tab in that page. 

  This particular use prevents users to access the full URLs of their datasette views and queries, which is a shame because the way datasette handles URLs to make every view or query easy to share is awesome. I know how to get the URL from the context menu of my browser, but I don't think many visitors would do it or even notice that datasette uses permalinks for pretty much every action they do. Would it be possible to add a "Share link" button to the interface, either in datasette itself or in a plugin?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1241/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTU1NTQzODU=
state: closed
created_at: 2021-02-24T14:55:06Z
updated_at: 2021-02-25T17:11:41Z
closed_at: 2021-02-25T17:11:41Z
author_association: NONE
type: issue
state_reason: completed
title:
  db["my_table"].drop(ignore=True) parameter, plus sqlite-utils drop-table --ignore and drop-view --ignore

body:
  When I'm generating a derived table in python, I often drop the table and create it from scratch. However, the first time I generate the table, it doesn't exist, so the drop raises an exception.  That means more boilerplate.

  I was going to submit a pull request that adds an "if_exists" option to the `drop` method of tables and views. 

  However, for a utility like sqlite_utils, perhaps the "IF EXISTS" SQL semantics is what you want most of the time, and thus should be the default.

  What do you think?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/237/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTc5Njk3ODMz
title: fix small typo
state: closed
created_at: 2021-02-25T00:22:34Z
updated_at: 2021-03-04T05:46:10Z
closed_at: 2021-03-04T05:46:10Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1243
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1243/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTY1MjM3NjM=
title: .add_foreign_key() corrupts database if column contains a space
state: closed
created_at: 2021-02-25T15:07:20Z
updated_at: 2021-02-25T16:54:02Z
closed_at: 2021-02-25T16:54:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I ran this:

      db["Reports"].add_foreign_key("Reported by ID", "Reporters", "id")

  And got this:

  ```
  ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in add_foreign_keys(self, foreign_keys)
      616         # Have to VACUUM outside the transaction to ensure .foreign_keys property
      617         # can see the newly created foreign key.
  --> 618         self.vacuum()
      619 
      620     def index_foreign_keys(self):

  ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in vacuum(self)
      629 
      630     def vacuum(self):
  --> 631         self.execute("VACUUM;")
      632 
      633 

  ~/jupyter-venv/lib/python3.9/site-packages/sqlite_utils/db.py in execute(self, sql, parameters)
      234             return self.conn.execute(sql, parameters)
      235         else:
  --> 236             return self.conn.execute(sql)
      237 
      238     def executescript(self, sql):

  DatabaseError: database disk image is malformed
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/238/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTY1MjY1Mzg=
title: sqlite-utils extract could handle nested objects
state: open
created_at: 2021-02-25T15:10:28Z
updated_at: 2022-09-03T23:46:02Z
author_association: OWNER
type: issue
body:
  Imagine a table (imported from a nested JSON file) where one of the columns contains values that look like this:

      {"email": "anonymous@noreply.airtable.com", "id": "usrROSHARE0000000", "name": "Anonymous"}

  The `sqlite-utils extract` command already uses single text values in a column to populate a new table. It would not be much of a stretch for it to be able to use JSON instead, including specifying which of those values should be used as the primary key in the new table.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/239/reactions", "total_count": 6, "+1": 5, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTY1NjA4MTk=
title: table.pks_and_rows_where() method returning primary keys along with the rows
state: closed
created_at: 2021-02-25T15:49:28Z
updated_at: 2021-02-25T16:39:23Z
closed_at: 2021-02-25T16:28:23Z
author_association: OWNER
type: issue
state_reason: completed
body:
  *Original title: Easier way to update a row returned from .rows*

  Here's a surprisingly hard problem I ran into while trying to implement #239 - given a row returned by `db[table].rows` how can you update that row?

  The problem is that the `db[table].update(...)` method requires a primary key. But if you have a row from the `db[table].rows` iterator it might not even contain the primary key - provided the table is a `rowid` table.

  Instead, currently, you need to introspect the table and, if `rowid` is a primary key, explicitly include that in the `select=`  argument to `table.rows_where(...)` - otherwise it will not be returned.

  A utility mechanism to make this easier would be very welcome.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/240/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTgwMjM1NDI3
title: Extract expand - work in progress
state: open
created_at: 2021-02-25T16:36:38Z
updated_at: 2021-02-25T16:36:38Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/241
body: Refs #239. Still needs documentation and CLI implementation.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/241/reactions", "total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTc1Mjg0NTI=
title: Plugin tip: look at the examples linked from the hooks page
state: closed
created_at: 2021-02-26T17:18:27Z
updated_at: 2021-02-26T17:30:38Z
closed_at: 2021-02-26T17:27:15Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Someone asked "what are good example plugins I can look at?" and I realized that the answer is to look through the example links on https://docs.datasette.io/en/stable/plugin_hooks.html - but that tip should be written down somewhere on the https://docs.datasette.io/en/stable/writing_plugins.html page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1244/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTc1NDQyNTE=
title: Sticky table column headers would be useful, especially on the query page
state: open
created_at: 2021-02-26T17:42:51Z
updated_at: 2021-04-02T20:53:35Z
author_association: OWNER
body: Suggestion from office hours.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1245/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTc1OTcyNjg=
title: Suggest for ArrayFacet possibly confused by blank values
state: closed
created_at: 2021-02-26T19:11:52Z
updated_at: 2021-03-01T03:46:11Z
closed_at: 2021-03-01T03:46:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I sometimes don't get the suggestion for facet-by-array for columns that contain arrays. I think it may be because they have empty spaces in them - or perhaps it's because the null detection doesn't actually work.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1246/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTc5ODk0MzY=
title: Async support
state: open
created_at: 2021-02-27T18:29:38Z
updated_at: 2021-10-28T14:37:56Z
author_association: CONTRIBUTOR
type: issue
body:
  Following our conversation last week, want to note this here before I forget.

  I've had a couple situations where I'd like to do a bunch of updates in an async event loop, but I run into SQLite's issues with concurrent writes. This feels like something sqlite-utils could help with.

  PeeWee ORM has a [SQLite write queue](http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#sqliteq) that might be a good model. It's using threads or gevent, but I _think_ that approach would translate well enough to asyncio. 

  Happy to help with this, too.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/242/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTg0MzA0MDU=
title: datasette.add_memory_database() method
state: closed
created_at: 2021-03-01T03:48:38Z
updated_at: 2021-03-01T04:02:26Z
closed_at: 2021-03-01T04:02:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I just wrote this code:

  https://github.com/simonw/datasette/blob/47eb885cc2c3aafa03645c330c6f597bee9b3b25/tests/test_facets.py#L334-L335

  It would be nice if you didn't have to separately instantiate a database object here.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1247/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MTg2ODQ5Nzg=
title: How can i use this utils to deal with fts on column meta of tables ?
state: open
created_at: 2021-03-01T09:45:05Z
updated_at: 2021-03-01T09:45:05Z
author_association: NONE
type: issue
body:
  Thank you to release this bravo project.
  When i use this project on multi table db, I want to implement 
  convenient search on column name from different tables.
  I want to develop a meta table to save the meta data of different columns
  of different tables and search on this meta table to get rows from the
  data table (which the meta table describes)
  does this project provide some simple function on it ?

  You can think a have a knowledge graph about the table in the db, 
  and i save this knowledge graph into the db with fts enabled.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/243/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTgzNDA3OTg5
title: Typo in upsert example
state: closed
created_at: 2021-03-02T23:14:14Z
updated_at: 2021-05-19T02:58:21Z
closed_at: 2021-05-19T02:58:21Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/244
body: Remove extra `[`
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/244/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MjE4NDEwNDY=
title: Upgrade to latest sqlite-utils
state: open
created_at: 2021-03-04T07:21:54Z
updated_at: 2021-03-04T07:22:51Z
author_association: MEMBER
body: This is pinned to v1 at the moment.
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MjMwMzUwODA=
title: duckdb database (very low performance in SQLite)
state: closed
created_at: 2021-03-05T12:20:29Z
updated_at: 2021-03-08T00:25:27Z
closed_at: 2021-03-08T00:25:27Z
author_association: NONE
type: issue
state_reason: completed
body:
  My sqlite is getting too big to be processed by datasette (more than 10 minutes waiting to load) so I am working with duckdb and is waaaaay faster. I think the fastest embeddable database actually.

  https://duckdb.org/

  Taking into account DuckDb is SQLite based it would be GREAT to use it with datasette.

  is that possible?

  Regards and thanks for a superb job

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1248/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MjQwNjQwNjk=
title: Updated Dockerfile with SpatiaLite version 5.0
state: closed
created_at: 2021-03-08T00:17:36Z
updated_at: 2022-01-20T21:29:43Z
closed_at: 2021-03-29T00:57:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The version bundled in Datasette's Docker image right now is 4.4.0-RC0

  https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/Dockerfile#L16-L17

  5 has been out for a couple of months and has a bunch of big improvements, most notable stable KNN support.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1249/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MjQwNjc2MDQ=
title: Research: Plugin hook for alternative database connections
state: closed
created_at: 2021-03-08T00:28:15Z
updated_at: 2021-03-12T01:01:25Z
closed_at: 2021-03-12T01:01:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The `Database` class is a natural looking fit for a plugin hook to load custom database connections... potentially even databases other than SQLite. DuckDB (refs #968) could make for a great starting point, since it looks very compatible with the existing SQLite code.

  The real win would be if this could lead to running Datasette against PostgreSQL. I made some initial explorations in that direction a while ago in #670.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1250/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MjQ3NTAxMzQ=
title: facet option not appearing when table is big
state: open
created_at: 2021-03-08T16:54:04Z
updated_at: 2021-03-08T16:54:16Z
author_association: NONE
type: issue
body:
  I have a big table with more than 500.000 rows.
  Trying to facet by one of my columns, the options are not available as for the other smaller tables.

  I have tried to set it in URL as:

  `&_facet=city_id`

  to no avail.

  is there any limit?  how can I force the option "facet" to appear for big tables?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1251/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTg3MzMyNDcz
title: Add back styling to lists within table cells (fixes #1141)
state: closed
created_at: 2021-03-09T03:00:57Z
updated_at: 2021-03-29T00:14:04Z
closed_at: 2021-03-29T00:14:04Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1252
type: pull
body:
  This overrides the Datasette reset - see https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/static/app.css#L35-L38 - to add back the default styling of list items displayed within Datasette table cells.

  Following this change, the same content as in the original issue looks like this:

  ![2021-03-09_02:57:32](https://user-images.githubusercontent.com/7476523/110411982-63e5ae80-8083-11eb-9b5c-e5dc825073e2.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1252/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MjYwNjQ1NTI=
title: Capture "Ctrl + Enter" or "⌘ + Enter" to send SQL query?
state: open
created_at: 2021-03-09T15:00:50Z
updated_at: 2021-10-30T16:00:42Z
author_association: NONE
type: issue
body:
  It appears as though "Shift + Enter" triggers the form submit action to submit SQL, but could that action be bound to the "Ctrl + Enter" or "⌘ + Enter" action?

  I feel like that pattern already exists in a number of similar tools and could improve usability of the editor.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1253/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTg4NjAxNjI3
title: Update Docker Spatialite version to 5.0.1 + add support for Spatialite topology functions
state: closed
created_at: 2021-03-09T20:49:08Z
updated_at: 2021-03-10T18:27:45Z
closed_at: 2021-03-09T22:04:23Z
author_association: NONE
pull_request: simonw/datasette/pulls/1254
type: pull
body:
  This requires adding the RT Topology library (Spatialite changed to RT Topology from LWGEOM between 4.4 and 5.0), as well as upgrading the GEOS version (which is the reason for switching to `python:3.7.10-slim-buster` as the base image.)

  `autoconf` and `libtool` are added to build RT Topology, and Spatialite is now built with `--disable-minizip` (minizip wasn't an option in 4.4 and I didn't want to add another dependency) and `--disable-dependency-tracking` which, according to Spatialite, "speeds up one-time builds"

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1254/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MjY3MDAwOTU=
title: Facets timing out but work when filtering
state: open
created_at: 2021-03-09T22:01:39Z
updated_at: 2021-04-02T20:50:08Z
author_association: NONE
type: issue
body:
  System info:

  Windows 10
  Datasette 0.55 installed via pip
  Python 3.8.5 in a conda environment

  I'm getting the message `These facets timed out` on any faceting operation. However, when I apply a filter, the facets appear in the filtered view. The error returns when the filter is removed. My data only has 38,450 rows.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1255/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTg5MjYzMjk3
title: Minor type in IP adress
state: closed
created_at: 2021-03-10T08:28:22Z
updated_at: 2021-03-10T18:26:46Z
closed_at: 2021-03-10T18:26:40Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1256
body: 127.0.01 replaced by 127.0.0.1
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1256/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4Mjg4MTE2MTg=
title: Table names containing single quotes break things
state: closed
created_at: 2021-03-11T06:29:38Z
updated_at: 2021-06-02T03:28:29Z
closed_at: 2021-06-02T03:28:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. I found a table called `Yesterday's ELRs by County`

  It threw an error inside the `detect_fts()` function attempting to run this SQL query:

  ```sql
          select name from sqlite_master
              where rootpage = 0
              and (
                  sql like '%VIRTUAL TABLE%USING FTS%content="Yesterday's ELRs by County"%'
                  or sql like '%VIRTUAL TABLE%USING FTS%content=[Yesterday's ELRs by County]%'
                  or (
                      tbl_name = "Yesterday's ELRs by County"
                      and sql like '%VIRTUAL TABLE%USING FTS%'
                  )
              )
  ```
  Here's the code at fault: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/__init__.py#L534-L548

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1257/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4Mjg4NTg0MjE=
title: Allow canned query params to specify default values
state: open
created_at: 2021-03-11T07:19:02Z
updated_at: 2023-02-20T23:39:58Z
author_association: NONE
type: issue
body:
  If I call a canned query that includes named parameters, without passing any parameters, datasette runs the query anyway, resulting in an HTTP status code 400, and a visible error in the browser, with only a link back to home.  This means that one of the default links on https://site/database/ will lead to a broken page with no apparent way out.

  ![image](https://user-images.githubusercontent.com/1385831/110748683-13e72300-820e-11eb-855c-32e03dfef5bf.png)

  Is there any way to skip performing the query when parameters aren't supplied, but otherwise render the usual canned query page?  Alternatively, can I supply default values for my parameters, either when defining my canned queries or when linking to the canned query page from the default database template.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1258/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzAyODM0NDc=
title: bucket name
state: open
created_at: 2021-03-12T16:40:57Z
updated_at: 2021-03-12T16:40:57Z
author_association: NONE
type: issue
body:
  I followed the instructions to setup credentials but I am getting a invalid bucket name. Can you put a sample auth.json file in the base that shows the correct format for this?  Thanks

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/34/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzA1NjcyNzU=
title: Research using CTEs for faster facet counts
state: open
created_at: 2021-03-12T22:19:49Z
updated_at: 2021-03-21T22:55:31Z
author_association: OWNER
type: issue
body:
  https://www.sqlite.org/changes.html#version_3_35_0

  > Add support for the [MATERIALIZED](https://www.sqlite.org/lang_with.html#mathint) and [NOT MATERIALIZED](https://www.sqlite.org/lang_with.html#mathint) hints when specifying [common table expressions](https://www.sqlite.org/lang_with.html). The default behavior was formerly NOT MATERIALIZED, but is now changed to MATERIALIZED for CTEs that are used more than once.

  If a CTE creates a table that is used multiple time in that query, SQLite will now default to creating a materialized table for the duration of that query.

  This could be a big performance boost when applying faceting multiple times against the same query. Consider this example query:
  ```sql
  WITH data as (
    select
      *
    from
      [global-power-plants]
  ),
  country_long as (select 
    'country_long' as col, country_long as value, count(*) as c from data group by country_long
    order by c desc limit 10
  ),
  primary_fuel as (
  select
    'primary_fuel' as col, primary_fuel as value, count(*) as c from data group by primary_fuel
    order by c desc limit 10
  )
  select * from primary_fuel union select * from country_long order by col, c desc
  ```
  https://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+10%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+10%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long+order+by+col%2C+c+desc

  Outputs:

  col | value | c
  -- | -- | --
  country_long | United States of America | 8688
  country_long | China | 4235
  country_long | United Kingdom | 2603
  country_long | Brazil | 2360
  country_long | France | 2155
  country_long | India | 1590
  country_long | Germany | 1309
  country_long | Canada | 1159
  country_long | Spain | 829
  country_long | Russia | 545
  primary_fuel | Solar | 9662
  primary_fuel | Hydro | 7155
  primary_fuel | Wind | 5188
  primary_fuel | Gas | 3922
  primary_fuel | Coal | 2390
  primary_fuel | Oil | 2290
  primary_fuel | Biomass | 1396
  primary_fuel | Waste | 1087
  primary_fuel | Nuclear | 198
  primary_fuel | Geothermal | 189

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1259/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTkyMjg5MzI0
title: Correct some typos
state: closed
created_at: 2021-03-13T04:26:56Z
updated_at: 2021-05-19T02:58:04Z
closed_at: 2021-05-19T02:58:04Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/245
type: pull
body:
  Noticed a typo in the docs and followed that up with a spellcheck. Had to bite my tongue at some of the British spellings.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/245/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTkyMzY0MjU1
title: Add a fallback ID, print if no ID found
state: open
created_at: 2021-03-13T13:38:29Z
updated_at: 2021-03-13T14:44:04Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/healthkit-to-sqlite/pulls/16
body: Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/14
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/16/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTkyNTQ4MTAz
title: Fix: code quality issues
state: closed
created_at: 2021-03-14T13:56:10Z
updated_at: 2021-03-29T00:22:41Z
closed_at: 2021-03-29T00:22:41Z
author_association: NONE
pull_request: simonw/datasette/pulls/1260
type: pull
body:
  ### Description
  Hi :wave: I work at [DeepSource](https://deepsource.io), I ran DeepSource analysis on the forked copy of this repo and found some interesting [code quality issues](https://deepsource.io/gh/withshubh/datasette/issues/?category=recommended) in the codebase, opening this PR so you can assess if our platform is right and helpful for you.

  ### Summary of changes

  - Replaced ternary syntax with if expression
  - Removed redundant `None` default
  - Used `is` to compare type of objects
  - Iterated dictionary directly
  - Removed unnecessary lambda expression
  - Refactored unnecessary `else` / `elif` when `if` block has a `return` statement
  - Refactored unnecessary `else` / `elif` when `if` block has a `raise` statement
  - Added .deepsource.toml to continuously analyze and detect code quality issues

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1260/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzE3NTEzNjc=
title: Escaping FTS search strings
state: closed
created_at: 2021-03-15T12:15:09Z
updated_at: 2021-08-18T18:57:13Z
closed_at: 2021-08-18T18:43:12Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Thanks for the excellent library, it's very nice to use!

  I've been building some in memory search functionality for a data annotation tool i'm making, and I got tripped up a little bit with escaping the full text search queries. First I tried using `db.quote(q)`, which doesn't work, because sqlite FTS has it's own (separate)[ query syntax](https://www2.sqlite.org/fts5.html#full_text_query_syntax). You can see this happening here also:

  http://search-24ways.herokuapp.com/24ways-f8f455f/articles?_search=acces%2A

  I got around this by aggressively escaping quotes inside the query string like this:

  ```python
          quoted = q.replace('"', '""')
          quoted = f'"{quoted}"'
          print(quoted)
          results = db["data"].search(quoted, columns=["id"])
          return [x["id"] for x in results]

  ```

  This works in the sense it doesn't crash, but it also removes access to the search query syntax. Given the well specified definition, it might be possible for sqlite-utils to provide a `db.quote_query(q)` which would intelligently escape a query whilst leaving the syntax intact. This would be very nice!

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/246/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzIwOTIzMjE=
title: Some links aren't properly URL encoded.
state: closed
created_at: 2021-03-15T18:43:59Z
updated_at: 2021-03-21T02:06:44Z
closed_at: 2021-03-20T21:36:06Z
author_association: NONE
type: issue
state_reason: completed
body:
  It seems like a percent sign in the query causes some links to end invalid.

  The json and CSV links on this page don't behave like expected: https://honeypot-brimston3.vercel.app/honeypot?sql=select+time%2C+count%28time%29+as+count+from+%28select+strftime%28%22%25Y-%25m-%25d%22%2C+_etime%29+as+time+from+ssh+%29+group+by+time+order+by+time%3B

  I can take a swing at trying to fix this, but my python isn't strong and I need a pointer at the right approach and files to change.

  Thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1261/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTkzODA1ODA0
title: FTS quote functionality from datasette
state: closed
created_at: 2021-03-16T11:17:34Z
updated_at: 2021-08-18T18:43:12Z
closed_at: 2021-08-18T18:43:12Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/247
type: pull
body:
  Addresses #246 - this is a bit of a kludge because it doesn't actually *validate* the FTS string, just makes sure that it will not crash when executed, but I figured that building a query parser is a bit out of the scope of sqlite-utils and if you actually want to use the query language, you probably need to parse that yourself.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/247/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzQ2MDIyOTk=
title: Plugin hook that could support 'order by random()' for table view
state: open
created_at: 2021-03-18T10:02:01Z
updated_at: 2021-03-18T17:55:01Z
author_association: NONE
type: issue
body:
  I am frequently using Datasette to quickly get a visual impression for a table without reviewing it in its entirety. Because I have some groups of similar records, the default sorting options mean that each page is very similar and not representative of the full dataset. The current interface allows sorting by columns, but random sorting is only available via custom SQL.

  Maybe this could be a button or link.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1262/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzYwNjMzODk=
title: Datetime columns are not properly formatted to be recognizes as datetime
state: open
created_at: 2021-03-19T14:33:04Z
updated_at: 2021-03-19T14:33:04Z
author_association: NONE
type: issue
body:
  Currently, the datetimes are formatted in a way that is not recognized by datasette-vega for plotting with a `Date/time` type for the axis. 

  For example, if you have datasette running locally with `datasette-vega` installed and have a database that includes resting heart rate:

  ```
  http://localhost:8001/healthkit/rRestingHeartRate#g.mark=line&g.x_column=startDate&g.x_type=temporal&g.y_column=value&g.y_type=quantitative
  ```

  The plot is blank unless you choose `Label` as the type for the date data.

  The `startDate` (and `creationDate` and `endDate`) columns appear like: `2019-11-14 18:22:18 -0700`

  If instead the format for this column is changed slightly: `2019-11-14T18:22:18-07:00` they are recognized as proper dates and the charting works as expected.

  I have a PR that addresses this issue, will submit shortly.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTk2NjI3Nzgw
title: Add datetime parsing
state: open
created_at: 2021-03-19T14:34:22Z
updated_at: 2021-03-19T14:34:22Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/healthkit-to-sqlite/pulls/18
type: pull
body:
  Parses the datetime columns so they are subsequently properly recognized as
  datetime.

  Fixes https://github.com/dogsheep/healthkit-to-sqlite/issues/17

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzYxMjMwMzA=
title: Support for HTTP Basic Authentication
state: closed
created_at: 2021-03-19T15:31:09Z
updated_at: 2021-03-19T22:05:12Z
closed_at: 2021-03-19T21:03:09Z
author_association: NONE
type: issue
state_reason: completed
body:
  It would be nice if datasette could support [HTTP Basic Authentication](https://en.wikipedia.org/wiki/Basic_access_authentication).

  For now I could ofcourse leverage Nginx for basic authentication, but it would be nice to have support for this in datasette by default or via a plugin like datasette-auth-github.

  My main usecase is to put the whole datasette instance behind a username/password prompt via Basic Auth and not specific urls.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1265/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzYyNzM4OTE=
title: Documentation for Response.asgi_send(send) method
state: closed
created_at: 2021-03-19T18:52:49Z
updated_at: 2021-03-20T21:35:00Z
closed_at: 2021-03-20T21:32:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I found myself wanting to use this method for https://github.com/simonw/datasette-auth-passwords/issues/15 - but it's not documented. It should be documented.

  https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/asgi.py#L320-L340

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1266/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzY4Mjk1NjA=
title: support for Apache Arrow / parquet files I/O
state: open
created_at: 2021-03-20T14:59:30Z
updated_at: 2021-10-28T23:46:48Z
author_association: NONE
type: issue
body:
  I just started looking at Apache Arrow using pyarrow for import and export of tabular datasets, and it looks quite compelling. It might be worth looking at for sqlite-utils and/or datasette.

  As a test, I took a random jsonl data dump of a dataset I have with floats, strings, and ints and converted it to arrow's parquet format using the naive `pyarrow.parquet.write_file()` command, which has automatic type inferrence. It compressed down to 7% of the original size. Conversion of a 26MB JSON file and serializing it to parquet was eyeblink instantaneous. Parquet files are portable and can be directly imported into pandas and other analytics software. 

  The only hangup is the automatic type inference of the naive reader. It's great for general laziness and for parsing JSON columns (it correctly interpreted a table of mine with a JSON array). However, I did get an exception for a string column where most entries looked integer-like but had a couple values that weren't -- the reader tried to coerce all of them for some reason, even though the JSON type is string. Since the writer optionally takes a schema, it shouldn't be too hard to grab the sqlite header types. With some additional hinting, you might get datetime columns and JSON, which are native Arrow types. 

  Somewhat tangentially, someone even wrote an sqlite vfs extension for Parquet: https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/248/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzY5MjMxOTQ=
title: JSON API for search results
state: open
created_at: 2021-03-20T22:21:36Z
updated_at: 2021-03-20T22:21:36Z
author_association: MEMBER
body: Refs https://github.com/simonw/datasette/issues/878
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzY5NjM4NTA=
title: Full text search possibly broken?
state: closed
created_at: 2021-03-21T02:03:44Z
updated_at: 2021-03-21T02:43:32Z
closed_at: 2021-03-21T02:43:32Z
author_association: NONE
type: issue
state_reason: completed
body:
  I'm not quite sure if this is an issue with sqlite-utils or datasette.

  **Background**
  I was previously using sqlite-utils version < 3.6. I have a bunch of csv files that have some data scraped from a website.

  ```
  sqlite-utils create-table mydb.db post \
      posted_date text \
      url text \
      title text \
      raw_text text \
      --not-null posted_date \
      --not-null url \
      --pk=url
  ```
  FTS is enabled via
  `sqlite-utils enable-fts ./mydb.db post title raw_text`

  Data is loaded to the table via
  `sqlite-utils insert ./mydb.db post ${filename} --csv`

  Note that the data contains text in my language Tamil.

  Loading happens just fine.
  datasette serves the db file just fine. It recognizes FTS and shows the "search" box. However, none of the queries work. Whatever text I supply, it always returns 0 rows. I literally copy paste words from the row listing on the screen and paste it on the search box.

  Interestingly, only thing  I can remember is switching to sqlite-utils 3.6. I had to do this because the prior version had an issue with column size.

  I have attached one of the csv files that can be loaded to the table. Substitute "${filename}" with that file for the sqlite-utils insert command.
  [posts_20200417-20201231.csv.zip](https://github.com/simonw/sqlite-utils/files/6176697/posts_20200417-20201231.csv.zip)

  Interestingly, the FTS based search from datasette worked just fine before this version upgrade. That is, the queries returned results. I will try to downgrade just to see if the theory is correct. 

  I appreciate any help here. Thanks.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/249/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzcyMDg5MDE=
title: Update Datasette alternativeto listening with details
state: closed
created_at: 2021-03-21T23:20:20Z
updated_at: 2021-03-22T04:37:26Z
closed_at: 2021-03-22T04:37:26Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hello,

  I recently learned about Datasette from an old hackernews post. It seems like an awesome project and I actually have use case I might be trying out in the coming months.

  Alas, to get a better understanding of your project I looked it up on alternativeto to see what it is similar too. I promise it's not spam, it's reputable enough to have a [Wikipedia](https://en.wikipedia.org/wiki/AlternativeTo) page. There was no listing on the website so I went ahead and created a listing that is now approved.

  I encourage anyone who likes this project and hopes to spread the word to help update the listing by:
  1. Adding to the list of software it compares to
  2. Uploading screenshots
  3. Writing a review
  4. Adding "features"

  I know this may seem spammy but I promise I have no affiliation with alternativeto I'm just a happy user and know it's a popular site for discovering software.

  Here is the listing for datasette:
  https://alternativeto.net/software/datasette/about/

  Cheers

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1267/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzczMDg3MDM=
title: Figure out why SpatiaLite 5.0 hangs the database page on Linux
state: closed
created_at: 2021-03-22T04:44:16Z
updated_at: 2022-01-20T21:29:43Z
closed_at: 2021-03-22T17:41:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See detailed notes in https://github.com/simonw/datasette/issues/1249 - for some reason SpatiaLite 5.0 hangs the `/dbname` page on Linux (inside Docker containers, both with a custom compiled SpatiaLite and one installed from the Ubuntu 20.10 package repository). This doesn't happen on macOS with SpatiaLite 5 installed using Homebrew.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1268/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzczNDg0Nzk=
title: Don't attempt to run count(*) against virtual tables
state: closed
created_at: 2021-03-22T05:57:43Z
updated_at: 2021-03-22T17:40:42Z
closed_at: 2021-03-22T17:40:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Counting the rows in a virtual table doesn't seem very interesting to me, and it's the cause of at least one crashing bug with SpatiaLite 5.0 on Linux, see https://github.com/simonw/datasette/issues/1268

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1269/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzczNTAwOTI=
title: Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler()
state: open
created_at: 2021-03-22T06:00:17Z
updated_at: 2021-03-23T16:45:39Z
author_association: OWNER
type: issue
body:
  > Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?
  >
  > https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1268#issuecomment-803764919_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1270/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTk4MjEzNTY1
title: Use SQLite conn.interrupt() instead of sqlite_timelimit()
state: open
created_at: 2021-03-22T17:34:20Z
updated_at: 2021-03-22T21:49:27Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1271
type: pull
body:
  Refs #1270, #1268, #1249

  Before merging this I need to do some more testing (to make sure that expensive queries really are properly cancelled). I also need to delete a bunch of code relating to the old mechanism of cancelling queries.

  [See comment below: this doesn't actually cancel the query due to a thread-local confusion]

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1271/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzgxNDgwODc=
title: Handle byte order marks (BOMs) in CSV files
state: closed
created_at: 2021-03-22T22:13:18Z
updated_at: 2021-05-29T05:34:21Z
closed_at: 2021-05-29T05:34:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I often find `sqlite-utils insert ... --csv` creates a first column with a weird character at the start of it - which it turns out is the UTF-8 BOM. Fix that.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/250/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzgyNDUzMzg=
title: Unit tests for the Dockerfile
state: open
created_at: 2021-03-23T01:36:29Z
updated_at: 2022-07-29T10:22:59Z
author_association: OWNER
type: issue
body:
  Working on the Dockerfile in #1249 made me wish for automated tests - to confirm that it boots up correctly, can run SpatiaLite and doesn't have weird bugs like the `/db` page hanging.

  These could run in CI too, but maybe only if the `Dockerfile` is updated.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1272/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzgzODI4OTA=
title: Refresh SpatiaLite documentation
state: open
created_at: 2021-03-23T06:05:55Z
updated_at: 2022-01-20T21:28:50Z
author_association: OWNER
type: issue
body:
  https://docs.datasette.io/en/0.55/spatialite.html was written before I had tools like [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite) and [shapefile-to-sqlite](https://datasette.io/tools/shapefile-to-sqlite).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1273/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzkwMDgzNzE=
title: Might there be some way to comment metadata.json?
state: closed
created_at: 2021-03-23T18:33:00Z
updated_at: 2021-03-23T20:14:54Z
closed_at: 2021-03-23T20:14:54Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  I don't know what license to use... Would be nice to be able to add a comment regarding that uncertainty in my metadata.json file

  I like laktak's little video comment in favor of Human json (Hjson)
  https://stackoverflow.com/questions/244777/can-comments-be-used-in-json

  Hmmm... one of the commenters there said comments are allowed in yaml... so that's a good argument for yaml.

  Anyhow, just came to mind, and thought I'd mention it here. Looks like https://hjson.github.io/ has the details.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1274/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MzkzNjc0NTE=
title: Idea: long-running query mode
state: open
created_at: 2021-03-24T05:23:20Z
updated_at: 2021-03-24T05:23:20Z
author_association: OWNER
type: issue
body:
  It would be cool if you could run Datasette in a long-running query mode, for use with trusted users - something like this:

      datasette --unlimited my.db

  This would disable the query limit, but would also enable a feature where if a query takes longer than e.g. 1s to return Datasette returns an HTML page to the browser with a progress indicator and polls the server until the query is complete.... but also provides the user with a "cancel" button. This relates to the `.interrupt()` research in #1270.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1275/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDEzNzc3MDI=
title: "sqlite-utils convert" command to replace the separate "sqlite-transform" tool
state: closed
created_at: 2021-03-25T22:36:36Z
updated_at: 2021-08-02T22:39:46Z
closed_at: 2021-08-02T04:47:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See https://github.com/simonw/sqlite-transform/issues/11 - I built a separate `sqlite-transform` tool a while ago that uses the word "transform" to means something entirely different from `sqlite-utils transform` - I'd like to resolve this by merging the two tools.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/251/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDE0NTYzMDY=
title: Invalid SQL: "no such table: pragma_database_list" on database page
state: closed
created_at: 2021-03-26T00:03:53Z
updated_at: 2021-03-31T16:27:27Z
closed_at: 2021-03-28T23:52:31Z
author_association: NONE
type: issue
state_reason: completed
body:
  Don't think this has been covered here yet. I'm a little stumped with this one and can't tell if it's a bug or I have something misconfigured. 

  Oddly, when running locally the usual list of tables populates (i.e. at /charts a list of tables in charts.db). But when on the web server it throws an Invalid SQL error and "no such table: pragma_database_list" below. 

  All the url endpoints seem to work fine aside from this - individual tables (/charts/chart_one), as well as stored queries (/charts/query_one). 

  Not sure if this has anything to do with upgrading to Datasette 0.55, or something to do with our setup, which uses a metadata build script similar to [the one for the 538 server](https://github.com/simonw/fivethirtyeight-datasette/blob/main/make_metadata.py), or something else. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1276/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDIwNjI5NDk=
title: Support json-line files
state: closed
created_at: 2021-03-26T15:19:39Z
updated_at: 2021-03-26T15:21:38Z
closed_at: 2021-03-26T15:21:38Z
author_association: NONE
type: issue
state_reason: completed
body:
  It's common for many processes to create flat files where each line is a JSON object.

  So the file isn't a json array.

  Many tools (like jq) support this natively, it'd be great for sqlite-utils to also!

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/252/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDIyMTI1ODY=
title: Facet by array breaks if table name contains a space
state: closed
created_at: 2021-03-26T18:38:19Z
updated_at: 2021-03-27T03:49:38Z
closed_at: 2021-03-27T03:49:34Z
author_association: OWNER
body: It breaks when you try to select a filtered item.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1277/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDI0MTYxMTA=
title: SpatiaLite timezones demo is broken
state: closed
created_at: 2021-03-27T04:45:27Z
updated_at: 2022-01-20T21:29:43Z
closed_at: 2021-03-27T16:17:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/blob/5fd02890650db790b2ffdb90eb9f78f8e0639c37/docs/spatialite.rst#L96

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1278/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjAyMTA3OTM1
title: Minor Docs Update. Added `--app` to fly install command.
state: closed
created_at: 2021-03-27T16:58:08Z
updated_at: 2021-03-29T00:11:55Z
closed_at: 2021-03-29T00:11:55Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1279
type: pull
body:
  Without this flag, there's an error locally. 

  ```
  > datasette publish fly bigmac.db

  Usage: datasette publish fly [OPTIONS] [FILES]...
  Try 'datasette publish fly --help' for help.

  Error: Missing option '-a' / '--app'.
  ```

  I also got an error message which later turned out to be because I hadn't added my credit card information yet to `fly`. I wasn't sure if I should add that mention to the docs here, or to submit a bug-report over at https://github.com/simonw/datasette-publish-fly. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1279/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDI2OTUzNzQ=
title: Support to annotate photos on other than macOS OSes
state: open
created_at: 2021-03-28T09:01:25Z
updated_at: 2021-04-05T07:37:57Z
author_association: NONE
type: issue
body:
  dogsheep-photos allows to annotate photos using Apple Photo's db. It would be nice to have such ability on other OSes too. For example using trained local model or using Google Vision API (see #14).

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjAyMjYxMDky
title: Add testres-db tool
state: closed
created_at: 2021-03-28T15:43:23Z
updated_at: 2022-02-16T05:12:05Z
closed_at: 2022-02-16T05:12:05Z
author_association: NONE
pull_request: dogsheep/dogsheep.github.io/pulls/6
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDI4NjI3MDg=
title: Ability to run CI against multiple SQLite versions
state: open
created_at: 2021-03-28T23:54:50Z
updated_at: 2021-05-10T19:07:46Z
author_association: OWNER
type: issue
body:
  Issue #1276 happened because I didn't run tests against a SQLite version prior to 3.16.0 (released 2017-01-02).

  Glitch is a deployment target and runs SQLite 3.11.0 from 2016-02-15.

  If CI ran against that version of SQLite this bug could have been avoided.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1280/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDI4ODEyMjE=
title: Latest Datasette tags missing from Docker Hub
state: closed
created_at: 2021-03-29T00:58:30Z
updated_at: 2021-03-29T01:41:48Z
closed_at: 2021-03-29T01:41:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Spotted this while testing https://github.com/simonw/datasette/issues/1249#issuecomment-808998719_

  https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated isn't showing the tags for any version more recent than 0.54.1 - we are up to 0.56 now.

  But the `:latest` tag is for the new 0.56 release.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1281/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjAzMDgyMjgw
title: Fix little typo
state: closed
created_at: 2021-03-29T19:45:28Z
updated_at: 2021-03-29T19:57:34Z
closed_at: 2021-03-29T19:57:34Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1282
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1282/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDM4ODQ3NDU=
title: advanced #export causes unexpected scrolling
state: open
created_at: 2021-03-29T22:46:57Z
updated_at: 2021-03-29T22:46:57Z
author_association: CONTRIBUTOR
type: issue
body:
  1. Visit a datasette table page
  2. Click on the "(advanced)" link. This adds a fragment identifier "#export" to the URL, and scrolls down to the "Advanced export" div with the "export" id.
  3. Manually scroll back up, and click on a suggested facet. The fragment identifier is still present, and the app scrolls back down to the "Advanced export" div. I think this is unwanted behavior.

  The user remedy seems to be to manually remove the "#export" from the URL.

  This behavior happens in my project, and in:
  https://covid-19.datasettes.com/covid/economist_excess_deaths  (for instance) 
  but not in this table: 
  https://global-power-plants.datasettes.com/global-power-plants/global-power-plants

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1283/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDU3OTQ0MzY=
title: Feature or Documentation Request: Individual table as home page template
state: open
created_at: 2021-03-31T03:56:17Z
updated_at: 2021-11-04T03:15:01Z
author_association: CONTRIBUTOR
type: issue
body:
  It would be great to have a sample showing how to move a single database that has a single table, to the index page. I'm trying it now, and find there is a real depth of Datasette and Python understanding that's required to be successful. 

  I've got all the basic jinja concepts down... variables, template control structures, template inheritance, template overrides, css, html, the --template-dir and --static arguments, etc. 

  But copying the table.html file to index.html doesn't work. There are undocumented functions and filters... I can figure some of them out (yay, url_builder.py and utils/__init__.py!) but it's a slog better handled by a much stronger Python developer. 

  One sample would make a world of difference. The ideal form of this documentation would be a diff between the default table.html and how that would look if essentially moved to index.html. The use case is for everyone who wants to create a public-facing website to explore a single table at the root directory. (Maybe a second bit of documentation for people who have a single database with multiple tables.)

  (Hmm... might be cool to have a setting for that, where it happens automagically! If only one table, then home page is at the table level. if only one database, then home page is at the database level.... as an option.)

  I suppose I could ignore this, and somehow do this in the DNS settings once I hook up Vercel to a domain name, maybe.. and remove the breadcrumbs in table.html... but for now, a documentation request in the form of a diff... for viewing a single table (or a single database) at the root.

  (Actually, there's probably room for a whole expanded section on templates. Noticed some nice table metadata in one of the datasette examples, for instance...  Hmm... maybe a whole library of solutions in one place... maybe a documentation hackathon! If that's of interest, of course it's a separate issue. )

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1284/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDc0MjM1NTk=
title: fixtures.db example error in sql-utils blog post
state: closed
created_at: 2021-03-31T22:07:36Z
updated_at: 2021-05-19T03:31:48Z
closed_at: 2021-05-19T03:31:47Z
author_association: NONE
type: issue
state_reason: completed
body:
  En route to trying to understand column order transform documentation, I tried the instructions here:
  https://simonwillison.net/2020/Sep/23/sqlite-advanced-alter-table/
  I get a malformed database schema syntax error.

  ```
  $ wget https://latest.datasette.io/fixtures.db
  --2021-03-31 18:00:23--  https://latest.datasette.io/fixtures.db
  Resolving latest.datasette.io (latest.datasette.io)... 2607:f8b0:4004:801::2013, 142.250.73.211
  Connecting to latest.datasette.io (latest.datasette.io)|2607:f8b0:4004:801::2013|:443... connected.
  HTTP request sent, awaiting response... 200 OK
  Length: unspecified [application/octet-stream]
  Saving to: ‘fixtures.db’

  fixtures.db                                                             [ <=>                                                                                                                                                              ] 260.00K  --.-KB/s    in 0.1s

  2021-03-31 18:00:23 (2.41 MB/s) - ‘fixtures.db’ saved [266240]

  $ sqlite3 fixtures.db '.schema facetable'
  Error: malformed database schema (generated_columns) - near "AS": syntax error

  $ sqlite3 fixtures.db
  SQLite version 3.28.0 2019-04-15 14:49:49
  Enter ".help" for usage hints.
  sqlite> .schema
  Error: malformed database schema (generated_columns) - near "AS": syntax error
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/253/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDc3MDA3MjY=
title: Feature Request or Plugin Request: Numeric Range Facets
state: open
created_at: 2021-04-01T01:50:20Z
updated_at: 2021-04-01T02:28:19Z
author_association: CONTRIBUTOR
type: issue
body:
  It would be great to offer facets for numeric data ranges. 

  The ranges could pull from typical GIS methods of creating choropleth maps. 
  https://gisgeography.com/choropleth-maps-data-classification/
  Of the following, for mapping, I've always preferred a Jenks Natural Breaks, or a cross between Jenks and Pretty breaks.

  - Equal Intervals 
  - Quantile (equal count) 
  - Standard Deviation
  - Natural Breaks (Jenks) Classification 
  - Pretty Breaks
  - Some sort of Aggregate Jenks Classification (this isn't standard, but it would be nice to be able to set classification ranges that work across tables.)

  Here are some links for Natural Breaks, in case this method is unfamiliar.

  - https://en.wikipedia.org/wiki/Jenks_natural_breaks_optimization
  - http://wiki.gis.com/wiki/index.php/Jenks_Natural_Breaks_Classification
  - https://medium.com/analytics-vidhya/jenks-natural-breaks-best-range-finder-algorithm-8d1907192051

  Per that last link, there is a Jenks Python module...  They also describe it as data-intensive for larger datasets. Maybe this is a good plugin idea.

  An example of equal Intervals would be 
  0 – < 10
  10 – < 20
  20 – < 30
  30 – < 40

  It's kind of confusing to have that less-than sign in there. it could also be displayed as:
  0 – 10
  10 – 20
  20 – 30
  30 – 40

  But then it's not completely clear which category 10 is in, for instance.

  (Best to right-justify.. and use an "en dash" between numbers.)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1285/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDkyMjAxNTQ=
title: Better default display of arrays of items
state: open
created_at: 2021-04-02T13:31:40Z
updated_at: 2021-06-12T12:36:15Z
author_association: CONTRIBUTOR
type: issue
body:
  Would be great to have template filters that convert array fields to bullets and/or delimited lists upon table display:
  ```
  |to_bullets
  |to_comma_delimited
  |to_semicolon_delimited
  ```
  or maybe: 
  ```
  |join_array("bullet")
  |join_array("bullet","square")
  |join_array(";")
  |join_array(",")
  ```
  Keeping in mind that bullets show up in html as \<li\> while other delimiting characters appear after the value.

  Of course, the fields themselves would remain as facetable arrays. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1286/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDkzOTY3NTg=
title: Upgrade to Python 3.9.4
state: open
created_at: 2021-04-02T18:43:15Z
updated_at: 2021-04-03T22:38:39Z
author_association: OWNER
type: issue
body:
  Has some security fixes https://pythoninsider.blogspot.com/2021/04/python-393-and-389-are-now-available.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1287/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDk1MTI4NDA=
title: Facets: show counts for null
state: open
created_at: 2021-04-02T22:33:44Z
updated_at: 2021-04-02T22:33:44Z
author_association: NONE
type: issue
body:
  Hi,

  Thank you for Datasette and being a fan of SQLite!

  Not all rows in a record will always contain data.
  So when using a facet on a column where some records have data and others don't, you don't get an accurate count of the results.

  Please consider also counting and showing null records with facets.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1288/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDk1NDM1MDI=
title: Speed up tests with pytest-xdist
state: closed
created_at: 2021-04-03T00:47:39Z
updated_at: 2021-04-03T03:42:28Z
closed_at: 2021-04-03T03:42:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I think I can get this working for almost every test, then use the pattern in https://github.com/pytest-dev/pytest-xdist/issues/385#issuecomment-444545641 to opt specific tests out of being run in parallel.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1289/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjA4MzIzMDI4
title: Use pytest-xdist to speed up tests
state: closed
created_at: 2021-04-03T03:34:36Z
updated_at: 2021-04-03T03:42:29Z
closed_at: 2021-04-03T03:42:28Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1290
body: Closes #1289, refs #1212.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1290/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjA4MzM0MDk2
title: Update docs: explain allow_download setting
state: closed
created_at: 2021-04-03T05:28:33Z
updated_at: 2021-06-05T19:48:51Z
closed_at: 2021-06-05T19:48:51Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1291
type: pull
body:
  This fixes one possible source of confusion seen in #502 and clarifies
  when database downloads will be shown and allowed.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1291/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDk5NzU4MTA=
title: Research ctypes.util.find_library('spatialite')
state: open
created_at: 2021-04-04T22:36:59Z
updated_at: 2022-01-20T21:28:50Z
author_association: OWNER
type: issue
body:
  Spotted this in the Django SpatiaLite backend: https://github.com/django/django/blob/8f6a7a0e9e7c5404af6520ae606927e32415eb00/django/contrib/gis/db/backends/spatialite/base.py#L24-L36

  ```python
  ctypes.util.find_library('spatialite')
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1292/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NDk5Nzg5NjQ=
title: Show column metadata plus links for foreign keys on arbitrary query results
state: open
created_at: 2021-04-04T22:59:42Z
updated_at: 2022-09-02T17:34:09Z
author_association: OWNER
type: issue
state_reason: reopened
body:
  Related to #620. It would be _really_ cool if Datasette could magically detect the source of the data displayed in an arbitrary query and, if that data represents a foreign key, display it as a hyperlink.

  Compare https://latest.datasette.io/fixtures/facetable

  <img width="1202" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/113523748-a7d2b300-955e-11eb-904d-8cf2639b0b5f.png">

  To https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+order+by+pk+limit+101

  <img width="998" alt="fixtures__select_pk__created__planet_int__on_earth__state__city_id__neighborhood__tags__complex_array__distinct_some_null_from_facetable_order_by_pk_limit_101" src="https://user-images.githubusercontent.com/9599/113523761-be790a00-955e-11eb-8c82-36b9d0226bc8.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1293/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NTM2NzIyMjQ=
title: "You can check out any time you like. But you can never leave!"
state: open
created_at: 2021-04-08T17:02:15Z
updated_at: 2021-04-08T18:35:50Z
author_association: CONTRIBUTOR
type: issue
body:
  (Feel free to rename this one.)

  - The column gear lets you "Show not-blank rows." Then it places a parameter in the URL, which a web developer would notice, but a lot of users won't notice, or know to delete it. Would be good to toggle "Show not-blank rows" with "Show all rows." (Also would be quite helpful to have a "Show blank rows | Show all rows" option)
  - The column gear lets you "Sort ascending" and "Sort descending" but then you're stuck with some sort of sorted version thereafter, unless you know to sort the ID column, or to remove the full _sort parameter and its value in the URL. Would be good to offer a "Remove sort" option in the gear.
  - These requests are in the same camp as: https://github.com/simonw/datasette-vega/issues/36
  - I suspect there are other url parameter instances where similar analysis would be helpful, but the three above are the use cases I've run across. 

  UPDATE:
  - It would be helpful to have a "Previous page" available for all but the first table page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1294/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NTUyOTY5Mzc=
title: Errors should have links to further information
state: open
created_at: 2021-04-11T12:39:12Z
updated_at: 2022-12-14T23:28:49Z
author_association: OWNER
type: issue
body:
  Inspired by this tweet:
  https://twitter.com/willmcgugan/status/1381186384510255104

  > While I am thinking about faqs. I’d also like to add short URLs to Rich exceptions.
  >
  > I loath cryptic error messages, and I’ve created a fair few myself. In Rich I’ve tried to make them as plain English as possible. But...
  >
  >  would be great if every error message linked to a page that explains the error in detail and offers fixes.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1295/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjEzMTc4OTY4
title: Dockerfile: use Ubuntu 20.10 as base
state: open
created_at: 2021-04-12T00:23:32Z
updated_at: 2021-07-20T08:52:13Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/datasette/pulls/1296
type: pull
body:
  This PR changes the main Dockerfile to use ubuntu:20.10 as base image instead of python:3.9.2-slim-buster (itself based on debian:buster-slim).

  The Dockerfile is essentially the one from https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 with some additional cleanups to slim it down.

  This fixes a couple of issues:
  1. The SQLite version in Debian Buster (2.6.0) doesn't support generated columns
  2. Installing SpatiaLite from the Debian sid repositories has the side effect of also installing updates to libc and libstdc++ from sid.

  As a bonus, the Docker image becomes smaller:


  ```
  $ docker image ls
  REPOSITORY                   TAG           IMAGE ID       CREATED       SIZE
  datasette                    0.56-ubuntu   f7aca255140a   5 hours ago   212MB
  datasetteproject/datasette   0.56          efb3b282f390   13 days ago   258MB
  ```

  ### Reproduction of the first issue

  ```
  $ curl -O https://latest.datasette.io/fixtures.db
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  100  260k    0  260k    0     0   489k      0 --:--:-- --:--:-- --:--:--  489k

  $ docker run -v `pwd`:/mnt datasetteproject/datasette:0.56 datasette /mnt/fixtures.db
  Traceback (most recent call last):
    File "/usr/local/bin/datasette", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/datasette/cli.py", line 544, in serve
      asyncio.get_event_loop().run_until_complete(check_databases(ds))
    File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
      return future.result()
    File "/usr/local/lib/python3.9/site-packages/datasette/cli.py", line 584, in check_databases
      await database.execute_fn(check_connection)
    File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 155, in execute_fn
      return await asyncio.get_event_loop().run_in_executor(
    File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
      result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 153, in in_thread
      return fn(conn)
    File "/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py", line 892, in check_connection
      for r in conn.execute(
  sqlite3.DatabaseError: malformed database schema (generated_columns) - near "AS": syntax error
  ```

  Here is the SQLite version:

  ```
  $ docker run -v `pwd`:/mnt -it datasetteproject/datasette:0.56 /bin/bash
  root@d9220d3b95dd:/# python3
  Python 3.9.2 (default, Mar 27 2021, 02:50:26) 
  [GCC 8.3.0] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sqlite3
  >>> sqlite3.version
  '2.6.0'
  ```

  ### Reproduction of the second issue

  ```
  $ docker build . -t datasette --build-arg VERSION=0.55
  [...snip...]
  The following packages will be upgraded:
    libc-bin libc6 libstdc++6
  [...snip...]
  Unpacking libc6:amd64 (2.31-11) over (2.28-10) ...
  [...snip...]
  Unpacking libstdc++6:amd64 (10.2.1-6) over (8.3.0-6) ...
  [...snip...]
  ```

  Both libc and libstdc++ are backwards compatible, so the image still works, but it will result in a combination of libraries and Python versions that exists only in the Datasette image, so it's likely untested. In addition, since Debian sid is an always-changing rolling-release, the versions of libc, libstdc++, Spatialite, and their dependencies change frequently, so the library versions in the Datasette image will depend on the day when it was built.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1296/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NTU0NTE0NjA=
title: Documentation: json1, and introspection endpoints
state: open
created_at: 2021-04-12T00:38:00Z
updated_at: 2021-04-12T01:29:33Z
author_association: CONTRIBUTOR
type: issue
body:
  https://docs.datasette.io/en/stable/facets.html notes that:
  > If your SQLite installation provides the json1 extension (you can check using /-/versions) Datasette will automatically detect columns that contain JSON arrays...

  When I check -/versions I see two sections relevant to json1:
  ```
          "extensions": {
              "json1": null
          },
          "compile_options": [
              ...
              "ENABLE_JSON1",
  ```
          
     The ENABLE_JSON1 makes me think json1 is likely available. But the  `"json1": null` made me think it wasn't available (because of the `null`). It would help if the documentation provided clarity about how to know if json1 is installed. It would also be helpful if the `/-/versions` information signalled somehow that that is to be appended to the hostname or domain name (or whatever you want to call it, or simply show it, using `example.com/-/versions` instead of `/-/versions`. Likewise on that last point, for https://docs.datasette.io/en/stable/introspection.html#introspection , at least at some point on that page detailing where those introspection endpoints go. (Sometimes documentation can be so abbreviated that it's hard for new users to figure out what's going on.)
     

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1297/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NTU0NzY1MDE=
title: improve table horizontal scroll experience
state: open
created_at: 2021-04-12T01:55:16Z
updated_at: 2022-08-30T21:11:49Z
author_association: CONTRIBUTOR
type: issue
body:
  Wide tables aren't a huge problem if you know to click and drag right. But it's not at all obvious to do that. (it also tends to blue-select any content as it's dragging.) Depending on column widths, public users might entirely miss all the columns to the right. 

  There is a scrollbar at the bottom of the table, but I'm displaying ALL my records because it's the only way for datasette-vega to make accurate charts. So that bottom scrollbar is likely to be missed. I wonder if some sort of javascript-y mouseover to an arrow might help, similar to those seen in image carousels. Ah: here's a perfect example:

  1. Visit http://google.com
  2. Search for: animals endangered
  3. Note the 'g-right-button' (in the code) that looks like a right-facing caret in a circle. 
  4. Click on that and the carousel scrolls right (and 'g-left-button' appears on the left).

  Might be tricky to do that on a table, rather than a one-row carousel, but it's worth experimenting with.

  Another option is just to put the scrollbars at the top of the table, too. 

  Meantime, I'm trying to build a button like the "View/hide all columns on https://salaries.news.baltimoresun.com/salaries-be494cf/2019+Maryland+state+salaries
  Might be nice to have that available by default, with settings in the metadata showing which are on by default.

  (I saw some other closed issues related to horizontal scrolling, and admit I don't entirely understand them. For instance, the animated gif at https://github.com/simonw/datasette/issues/998#issuecomment-714117534 confuses me. )

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1298/reactions", "total_count": 4, "+1": 4, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NTY4OTUyOTE=
title: Design better empty states
state: open
created_at: 2021-04-13T12:06:12Z
updated_at: 2021-04-13T12:06:12Z
author_association: OWNER
body: Inspiration here: https://emptystat.es/
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1299/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjE0NzI3MDM2
title: Fix incorrect create-table cli description
state: closed
created_at: 2021-04-13T20:03:15Z
updated_at: 2021-05-19T04:43:46Z
closed_at: 2021-05-19T02:57:26Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/254
body: The description for `create-table` was duplicated from `create-index`.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/254/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NTg1MDEwNzk=
title: transform --help should tell you the available types
state: closed
created_at: 2021-04-15T05:24:48Z
updated_at: 2021-05-29T03:55:52Z
closed_at: 2021-05-29T03:55:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  Usage: sqlite-utils transform [OPTIONS] PATH TABLE

    Transform a table beyond the capabilities of ALTER TABLE

  Options:
    --type <TEXT TEXT>...     Change column type to X
  ```
  This should specify that the possible types are 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/255/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjA2MjU4MzM=
title: Make row available to `render_cell` plugin hook
state: closed
created_at: 2021-04-18T10:14:37Z
updated_at: 2022-07-07T16:34:05Z
closed_at: 2022-07-07T16:31:22Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  *Original title: **Generating URL for a row inside `render_cell` hook***

  Hey,
  I am using Datasette to view a database that contains video metadata. It has BLOB columns that contain video thumbnails in JPG format (around 100-500KB per row). 

  I've registered an output formatter that extends `datasette.blob_renderer.render_blob` function and serves the column with `image/jpeg` content type.

  ```python
  from datasette.blob_renderer import render_blob

  async def render_jpg(datasette, database, rows, columns, request, table, view_name):
      response = await render_blob(datasette, database, rows, columns, request, table, view_name)
      response.content_type = "image/jpeg"
      response.headers["Content-Disposition"] = f'inline; filename="image.jpg"'
      return response


  @hookimpl
  def register_output_renderer():
      return {
          "extension": "jpg",
          "render": render_jpg,
          "can_render": lambda: True,
      }
  ```

  This works well. I can visit `http://localhost:8001/mydb/videos/1.jpg?_blob_column=thumbnail` and view the image.

  I want to display the image directly with an `<img>` tag (lazy-loaded of course). So, I need a URL, because embedding base64 would increase the page size too much (each image > 100KB). 

  Datasette generates a link with `.blob` extension for blob columns. It does this by calling `datasette.urls.row_blob`

  https://github.com/simonw/datasette/blob/7a2ed9f8a119e220b66d67c7b9e07cbab47b1196/datasette/views/table.py#L169-L179

  But I have no way of getting the row inside the `render_cell` hook. 

  ```python
  @hookimpl
  def render_cell(value, column, table, database, datasette):
      if isinstance(value, bytes) and imghdr.what(None, value):
          # generate url
          return '$renderedLink'
  ```

  Any pointers?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1300/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjA3MjI3MTE=
title: Publishing to cloudrun with immutable mode?
state: open
created_at: 2021-04-18T17:51:46Z
updated_at: 2022-10-07T02:38:04Z
author_association: CONTRIBUTOR
type: issue
body:
  I'm a bit confused about immutable mode and publishing to cloudrun. (I want to publish with immutable mode so that I can support database downloads.)

  Running `datasette publish cloudrun --extra-options="-i example.db"` leads to an error:
  > Error: Invalid value for '-i' / '--immutable': Path 'example.db' does not exist. 

  However, running `datasette publish cloudrun example.db` not only works but seems to publish in immutable mode anyway! I'm seeing this both with `/-/databases.json` and the fact that downloads are working.

  When I just `datasette serve` locally, this succeeds both ways and works as expected.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1301/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjA3MzQ3MjI=
title: Fix disappearing facets
state: open
created_at: 2021-04-18T18:42:33Z
updated_at: 2021-04-20T07:40:15Z
author_association: CONTRIBUTOR
type: issue
body:
  1. Clone https://github.com/mroswell/list-N
  2. Run `datasette disinfectants.db -o`
  3. Select the `Safer_or_Toxic` facet.
  4. Select `Toxic`.
  5. Close out the `Safer_or_Toxic` facet.
  6. Examine `Suggested facets` list. `Safer_or_Toxic` is GONE.
  7. Try some other facets. When you select an element, and then close the list, in some cases, the facet properly returns to the `Suggested facet` list... Arrays and dates properly return to the list, but fields with strings don't return to the list. 

  Since my site is devoted to whether disinfectants are Safer or Toxic, having the suggested facet disappear from the suggested facet list is very confusing* to end-users. This, along with a few other issues, unfortunately proved beyond my own programming ability to address. So I hired a Senior-level developer to address a number of issues, including this disappearing act.

  8. Open a new terminal. Run `datasette disinfectants.db -m metadata.json --static static:static/  --template-dir templates/ --plugins-dir plugins/ -p 8001 -o`
  9. Repeat steps 3-6, but this time, the Safer_or_Toxic facet returns to the list (and the related URL parameters are removed).

  I'm not sure how to do a pull request for this, because the plugin contains other functionality that goes beyond this bug. I wanted the facets sorted in a certain order (both in the suggested facet list, and the detail lists) (... the detail lists were hopping around all over the place before...) I wanted the duplicate facets removed (leaving only the one where you can facet by individual item in an array.) I wanted the arrays to be presented in a prettier fashion (I did that in the template... That could be moved over to the plugin at some point)

  I'm thinking it'll be very helpful if applicable parts of my project's plugin (sort_suggested_facets_plugin.py) will be able to be incorporated back into datasette, but I leave that to you to consider.

  (* The disappearing facet bug was especially confusing because I'm removing the filters and sql from the table page, at the request of the organization. The filters and sql detail created a lot of confusion for end users who try to find disinfectants used by Hospitals, for instance, as an '=' won't find them, since they are part of the Use_site array.) My disappearing-facet confusion was documented in my own issue: https://github.com/mroswell/list-N/issues/57 (addressed by the plugin). Other facet-related issues here: https://github.com/mroswell/list-N/issues/54 (addressed by the plugin); https://github.com/mroswell/list-N/issues/15 (addressed by template); https://github.com/mroswell/list-N/issues/53 (not yet addressed). 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1302/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjE4MDExOTc3
title: Update pytest-asyncio requirement from <0.15,>=0.10 to >=0.10,<0.16
state: closed
created_at: 2021-04-19T13:49:12Z
updated_at: 2021-04-19T18:18:17Z
closed_at: 2021-04-19T18:18:17Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1303
type: pull
body:
  Updates the requirements on [pytest-asyncio](https://github.com/pytest-dev/pytest-asyncio) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/7aa574af1830665be23b983407fdff8f5fcf3ccc"><code>7aa574a</code></a> 0.15.0</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/dadff91937e03c9cc08b290d14c8bc72aa793f20"><code>dadff91</code></a> Remove poetry</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/d4b79ea6481cb498a2fd3feb8a05f48249559d55"><code>d4b79ea</code></a> Lint</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/22d91b2a6df1b3557cfc0370736d7cad4ba8bef4"><code>22d91b2</code></a> Black linting</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/5c47b78ebfab7a42a9450b88e4a13bf5b135e228"><code>5c47b78</code></a> Covecov</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/22e338a4dbf67a08146757cc3a2320b7e88c7eae"><code>22e338a</code></a> Fix CI</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/70cbdeff9f9b04635714b2347ebdbea71acd93a6"><code>70cbdef</code></a> Travis -&gt; GitHub Actions</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/740af1843f1dafb47ba6db236c1e354731993949"><code>740af18</code></a> Close event loops when replacing them</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/a5161343442fb64b7fcb91dfc5841b9f79698989"><code>a516134</code></a> plugin: Set unused_tcp_port_factory scope to 'session'</li>
  <li><a href="https://github.com/pytest-dev/pytest-asyncio/commit/1bb7f300f4f5c31d47675ed018e9a46ecd4d8496"><code>1bb7f30</code></a> refactor: Removed the &quot;_35&quot; and &quot;_36&quot; suffixes from test modules.</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-asyncio/compare/v0.10.0...v0.15.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1303/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjE2MjI4Mzk=
state: closed
created_at: 2021-04-19T18:01:03Z
updated_at: 2021-05-19T03:26:54Z
closed_at: 2021-05-19T03:26:54Z
author_association: NONE
type: issue
state_reason: completed
title:
  inserting with --nl errors with: sqlite3.OperationalError: table <table> has no column named <column>

body:
  I have a `jsonl` file, it is 10,000 lines long.

  Inserting from the cli with `sqlite-utils insert db table file --nl --batch-size 10000` fails with this missing column error, even though I'm telling it to use the whole file in the first batch.

  This seems similar to #18 and #139, but maybe it's unique to `--nl`?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/256/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjM4ODQ4MDU=
title: Document how to send multiple values for "Named parameters"
state: open
created_at: 2021-04-21T13:19:06Z
updated_at: 2021-12-08T03:23:14Z
author_association: NONE
type: issue
body:
  https://docs.datasette.io/en/stable/sql_queries.html#named-parameters

  I thought that I had seen an example of how to do this example below, but I can't seem to find it

  ```sql
  select
    *
  from
    bib
  where
    bib.bib_record_num in (1008088,1008092)
  ```

  ```sql
  select
    *
  from
    bib
  where
    bib.bib_record_num in (:bib_record_numbers)
  ```
  ![image](https://user-images.githubusercontent.com/9308268/115558839-2333a480-a281-11eb-85e6-ce3bada79140.png)

  https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-204d100?sql=select%0D%0A++*%0D%0Afrom%0D%0A++bib%0D%0Awhere%0D%0A++bib.bib_record_num+in+%28%3Abib_record_numbers%29&bib_record_numbers=1008088%2C1008092

  Or, maybe this isn't a fully supported feature.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1304/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjQ5Njk2ODM=
title: Index view crashes when any database table is not accessible to actor
state: closed
created_at: 2021-04-22T13:44:22Z
updated_at: 2021-06-02T04:26:29Z
closed_at: 2021-06-02T04:26:29Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Because of https://github.com/simonw/datasette/blob/main/datasette/views/index.py#L63, the ```tables``` dict built does not include invisible tables; however, if https://github.com/simonw/datasette/blob/main/datasette/views/index.py#L80 is reached (because table_counts was not successfully initialized, e.g. due to a very large database) then as db.get_all_foreign_keys() returns ALL tables, a KeyError will be raised.

  This error can be recreated with the fixtures.db if any table is hidden, e.g. by adding something like                 ```"foreign_key_references": {
                      "allow": {}
                  }``` to fixtures-metadata.json and deleting ```or not table_counts``` from https://github.com/simonw/datasette/blob/main/datasette/views/index.py#L77.

  I'm not sure how to fix this error; perhaps by testing if the table is in the aforementions ```tables``` dict.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1305/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjIxMTE3OTc4
title: Avoid error sorting by relationships if related tables are not allowed
state: closed
created_at: 2021-04-22T13:53:17Z
updated_at: 2021-06-02T04:27:00Z
closed_at: 2021-06-02T04:25:28Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1306
body: Refs #1305
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1306/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjY2Njg0MTU=
title: Columns named "link" display in bold
state: closed
created_at: 2021-04-24T05:58:11Z
updated_at: 2021-04-24T06:07:49Z
closed_at: 2021-04-24T06:07:49Z
author_association: OWNER
body: Reported in office hours today.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1308/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NjgxODgwNjg=
state: closed
created_at: 2021-04-26T20:46:25Z
updated_at: 2021-05-19T02:57:05Z
closed_at: 2021-05-19T02:57:05Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
title:
  Insert from JSON containing strings with non-ascii characters are escaped as unicode for lists, tuples, dicts.

body:
  JSON Test File (test.json):

  ```json
  [
      {
          "id": 123,
          "text": "FR Théâtre"
      },
      {
          "id": 223,
          "text": [
              "FR Théâtre"
          ]
      }
  ]
  ```

  Command to import:

  ```bash
  sqlite-utils insert test.db text test.json --pk=id
  ```

  Resulting table view from datasette:

  ![image](https://user-images.githubusercontent.com/6586811/116147833-cdf2fb00-a6a5-11eb-8412-0aae81b6e6dd.png)

  Original, db.py line 2225:

  ```python
          return json.dumps(value, default=repr)
  ```

  Fix, db.py line 2225:

  ```python
          return json.dumps(value, default=repr, ensure_ascii=False)
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/257/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjIzNzU1NzIz
title: Fixing insert from JSON containing strings with non-ascii characters …
state: closed
created_at: 2021-04-26T20:50:00Z
updated_at: 2021-05-19T02:47:44Z
closed_at: 2021-05-19T02:47:44Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/258
body:
  …are escaped aps unicode for lists, tuples, dicts

  Fix of #257

type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/258/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjI0NjM1NDQw
title: Bump black from 20.8b1 to 21.4b0
state: closed
created_at: 2021-04-27T20:28:11Z
updated_at: 2021-04-28T18:26:06Z
closed_at: 2021-04-28T18:26:04Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1309
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 20.8b1 to 21.4b0.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.4b0</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fixed a rare but annoying formatting instability created by the combination of
  optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
  pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href="https://github-redirect.dependabot.com/psf/black/issues/1629">#1629</a> and all of its many many
  duplicates. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2126">#2126</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
  and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1740">#1740</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2092">#2092</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
  docstring quotes (<a href="https://github-redirect.dependabot.com/psf/black/issues/1637">#1637</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer removes all empty lines between non-function code and decorators
  when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1646">#1646</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
  in if/while statements (<a href="https://github-redirect.dependabot.com/psf/black/issues/1655">#1655</a>)</p>
  </li>
  <li>
  <p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
  to split lines (<a href="https://github-redirect.dependabot.com/psf/black/issues/1824">#1824</a>)</p>
  </li>
  <li>
  <p>fixed a crash when PWD=/ on POSIX (<a href="https://github-redirect.dependabot.com/psf/black/issues/1631">#1631</a>)</p>
  </li>
  <li>
  <p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href="https://github-redirect.dependabot.com/psf/black/issues/1664">#1664</a>)</p>
  </li>
  <li>
  <p>Prevent coloured diff output being interleaved with multiple files (<a href="https://github-redirect.dependabot.com/psf/black/issues/1673">#1673</a>)</p>
  </li>
  <li>
  <p>Added support for PEP 614 relaxed decorator syntax on python 3.9 (<a href="https://github-redirect.dependabot.com/psf/black/issues/1711">#1711</a>)</p>
  </li>
  <li>
  <p>Added parsing support for unparenthesized tuples and yield expressions in annotated
  assignments (<a href="https://github-redirect.dependabot.com/psf/black/issues/1835">#1835</a>)</p>
  </li>
  <li>
  <p>use lowercase hex strings (<a href="https://github-redirect.dependabot.com/psf/black/issues/1692">#1692</a>)</p>
  </li>
  <li>
  <p>added <code>--extend-exclude</code> argument (PR <a href="https://github-redirect.dependabot.com/psf/black/issues/2005">#2005</a>)</p>
  </li>
  <li>
  <p>speed up caching by avoiding pathlib (<a href="https://github-redirect.dependabot.com/psf/black/issues/1950">#1950</a>)</p>
  </li>
  <li>
  <p><code>--diff</code> correctly indicates when a file doesn't end in a newline (<a href="https://github-redirect.dependabot.com/psf/black/issues/1662">#1662</a>)</p>
  </li>
  <li>
  <p>Added <code>--stdin-filename</code> argument to allow stdin to respect <code>--force-exclude</code> rules
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/1780">#1780</a>)</p>
  </li>
  <li>
  <p>Lines ending with <code>fmt: skip</code> will now be not formatted (<a href="https://github-redirect.dependabot.com/psf/black/issues/1800">#1800</a>)</p>
  </li>
  <li>
  <p>PR <a href="https://github-redirect.dependabot.com/psf/black/issues/2053">#2053</a>: Black no longer relies on typed-ast for Python 3.8 and higher</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/master/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h3>21.4b0</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fixed a rare but annoying formatting instability created by the combination of
  optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
  pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href="https://github-redirect.dependabot.com/psf/black/issues/1629">#1629</a> and all of its many many
  duplicates. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2126">#2126</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
  and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1740">#1740</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2092">#2092</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
  docstring quotes (<a href="https://github-redirect.dependabot.com/psf/black/issues/1637">#1637</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer removes all empty lines between non-function code and decorators
  when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1646">#1646</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
  in if/while statements (<a href="https://github-redirect.dependabot.com/psf/black/issues/1655">#1655</a>)</p>
  </li>
  <li>
  <p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
  to split lines (<a href="https://github-redirect.dependabot.com/psf/black/issues/1824">#1824</a>)</p>
  </li>
  <li>
  <p>fixed a crash when PWD=/ on POSIX (<a href="https://github-redirect.dependabot.com/psf/black/issues/1631">#1631</a>)</p>
  </li>
  <li>
  <p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href="https://github-redirect.dependabot.com/psf/black/issues/1664">#1664</a>)</p>
  </li>
  <li>
  <p>Prevent coloured diff output being interleaved with multiple files (<a href="https://github-redirect.dependabot.com/psf/black/issues/1673">#1673</a>)</p>
  </li>
  <li>
  <p>Added support for PEP 614 relaxed decorator syntax on python 3.9 (<a href="https://github-redirect.dependabot.com/psf/black/issues/1711">#1711</a>)</p>
  </li>
  <li>
  <p>Added parsing support for unparenthesized tuples and yield expressions in annotated
  assignments (<a href="https://github-redirect.dependabot.com/psf/black/issues/1835">#1835</a>)</p>
  </li>
  <li>
  <p>added <code>--extend-exclude</code> argument (PR <a href="https://github-redirect.dependabot.com/psf/black/issues/2005">#2005</a>)</p>
  </li>
  <li>
  <p>speed up caching by avoiding pathlib (<a href="https://github-redirect.dependabot.com/psf/black/issues/1950">#1950</a>)</p>
  </li>
  <li>
  <p><code>--diff</code> correctly indicates when a file doesn't end in a newline (<a href="https://github-redirect.dependabot.com/psf/black/issues/1662">#1662</a>)</p>
  </li>
  <li>
  <p>Added <code>--stdin-filename</code> argument to allow stdin to respect <code>--force-exclude</code> rules
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/1780">#1780</a>)</p>
  </li>
  <li>
  <p>Lines ending with <code>fmt: skip</code> will now be not formatted (<a href="https://github-redirect.dependabot.com/psf/black/issues/1800">#1800</a>)</p>
  </li>
  <li>
  <p>PR <a href="https://github-redirect.dependabot.com/psf/black/issues/2053">#2053</a>: Black no longer relies on typed-ast for Python 3.8 and higher</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b0)](https://dependabot.com/compatibility-score/?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b0)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1309/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NzAxMjUxMjY=
title: I'm creating a plugin to export a spreadsheet file (.ods or .xlsx)
state: closed
created_at: 2021-04-28T16:20:11Z
updated_at: 2021-04-30T07:26:11Z
closed_at: 2021-04-30T06:58:46Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,

  I have started developing a plugin to export records as a spreadsheet file. It could be ods or xlsx, whatever is easier.

  I have spotted the following packages:

  - ods files: https://pypi.org/project/odswriter/
  - xlsx files: https://openpyxl.readthedocs.io/en/stable/index.html (quite powerful) or https://xlsxwriter.readthedocs.io/ (faster)

  This is the code I have so far, I test it with the `--plugins-dir` option:

  ```python
  from datasette import hookimpl
  from datasette.utils.asgi import Response
  import odswriter as ods

  def render_spreadsheet(rows):
      with ods.writer(open("test.ods","wb")) as odsfile:
          for row in rows:
              odsfile.writerow(["String", "ABCDEF123456", "123456"])
          return Response(odsfile, content_type="application/vnd.oasis.opendocument.spreadsheet", status=200)


  @hookimpl
  def register_output_renderer():
      return {"extension": "ods", "render": render_spreadsheet}

  ``` 

  I get the following error:

  ```
  Traceback (most recent call last):
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/app.py", line 1128, in route_path
      await response.asgi_send(send)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py", line 339, in asgi_send
      body = body.encode("utf-8")
  AttributeError: 'ODSWriter' object has no attribute 'encode'
  ERROR:    Exception in ASGI application
  Traceback (most recent call last):
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/app.py", line 1128, in route_path
      await response.asgi_send(send)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py", line 339, in asgi_send
      body = body.encode("utf-8")
  AttributeError: 'ODSWriter' object has no attribute 'encode'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/colin/.local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
      result = await app(self.scope, self.receive, self.send)
    File "/home/colin/.local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
      return await self.app(scope, receive, send)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py", line 161, in __call__
      await self.app(scope, receive, send)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/tracer.py", line 75, in __call__
      await self.app(scope, receive, send)
    File "/home/colin/.local/lib/python3.8/site-packages/asgi_csrf.py", line 107, in app_wrapped_with_csrf
      await app(scope, receive, wrapped_send)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/app.py", line 1086, in __call__
      return await self.route_path(scope, receive, send, path)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/app.py", line 1133, in route_path
      return await self.handle_500(request, send, exception)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/app.py", line 1267, in handle_500
      await asgi_send_html(
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py", line 217, in asgi_send_html
      await asgi_send(
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py", line 237, in asgi_send
      await asgi_start(send, status, headers, content_type)
    File "/home/colin/.local/lib/python3.8/site-packages/datasette/utils/asgi.py", line 246, in asgi_start
      await send(
    File "/home/colin/.local/lib/python3.8/site-packages/asgi_csrf.py", line 103, in wrapped_send
      await send(event)
    File "/home/colin/.local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 482, in send
      raise RuntimeError(msg % message_type)
  RuntimeError: Expected ASGI message 'http.response.body', but got 'http.response.start'.
  ```

  I tried with `AsgiFileDownload` like in [DatabaseDownload](https://github.com/simonw/datasette/blob/main/datasette/views/database.py#L150) to deal with the binary nature of the ods file, but the renderer expects a Response:

  > <datasette.utils.asgi.AsgiFileDownload object at 0x7f5b9bd503d0> should be dict or Response

  However, the `Response` class only supports the following methods, not binary:

  - html
  - text
  - json
  - redirect

  How would you suggest me to proceed to have my ods file downloaded?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1310/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjI1NDU3NTc5
title: Bump black from 20.8b1 to 21.4b1
state: closed
created_at: 2021-04-28T18:25:58Z
updated_at: 2021-04-29T13:58:11Z
closed_at: 2021-04-29T13:58:09Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1311
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 20.8b1 to 21.4b1.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.4b1</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2142">#2142</a>)</p>
  </li>
  <li>
  <p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2120">#2120</a>)</p>
  </li>
  <li>
  <p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
  in the name of the cache file. Without this fix, changes in these flags would not take
  effect if the cache had already been populated. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2131">#2131</a>)</p>
  </li>
  <li>
  <p>Don't remove necessary parentheses from assignment expression containing assert /
  return statements. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2143">#2143</a>)</p>
  </li>
  </ul>
  <h4><em>Packaging</em></h4>
  <ul>
  <li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
  </ul>
  <h2>21.4b0</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fixed a rare but annoying formatting instability created by the combination of
  optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
  pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href="https://github-redirect.dependabot.com/psf/black/issues/1629">#1629</a> and all of its many many
  duplicates. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2126">#2126</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
  and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1740">#1740</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2092">#2092</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
  docstring quotes (<a href="https://github-redirect.dependabot.com/psf/black/issues/1637">#1637</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer removes all empty lines between non-function code and decorators
  when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1646">#1646</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
  in if/while statements (<a href="https://github-redirect.dependabot.com/psf/black/issues/1655">#1655</a>)</p>
  </li>
  <li>
  <p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
  to split lines (<a href="https://github-redirect.dependabot.com/psf/black/issues/1824">#1824</a>)</p>
  </li>
  <li>
  <p>fixed a crash when PWD=/ on POSIX (<a href="https://github-redirect.dependabot.com/psf/black/issues/1631">#1631</a>)</p>
  </li>
  <li>
  <p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href="https://github-redirect.dependabot.com/psf/black/issues/1664">#1664</a>)</p>
  </li>
  <li>
  <p>Prevent coloured diff output being interleaved with multiple files (<a href="https://github-redirect.dependabot.com/psf/black/issues/1673">#1673</a>)</p>
  </li>
  <li>
  <p>Added support for PEP 614 relaxed decorator syntax on python 3.9 (<a href="https://github-redirect.dependabot.com/psf/black/issues/1711">#1711</a>)</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/master/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h3>21.4b1</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2142">#2142</a>)</p>
  </li>
  <li>
  <p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2120">#2120</a>)</p>
  </li>
  <li>
  <p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
  in the name of the cache file. Without this fix, changes in these flags would not take
  effect if the cache had already been populated. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2131">#2131</a>)</p>
  </li>
  <li>
  <p>Don't remove necessary parentheses from assignment expression containing assert /
  return statements. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2143">#2143</a>)</p>
  </li>
  </ul>
  <h4><em>Packaging</em></h4>
  <ul>
  <li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
  </ul>
  <h3>21.4b0</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fixed a rare but annoying formatting instability created by the combination of
  optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
  pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href="https://github-redirect.dependabot.com/psf/black/issues/1629">#1629</a> and all of its many many
  duplicates. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2126">#2126</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
  and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1740">#1740</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2092">#2092</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
  docstring quotes (<a href="https://github-redirect.dependabot.com/psf/black/issues/1637">#1637</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer removes all empty lines between non-function code and decorators
  when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1646">#1646</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer adds an incorrect space after a parenthesized assignment expression
  in if/while statements (<a href="https://github-redirect.dependabot.com/psf/black/issues/1655">#1655</a>)</p>
  </li>
  <li>
  <p>Added <code>--skip-magic-trailing-comma</code> / <code>-C</code> to avoid using trailing commas as a reason
  to split lines (<a href="https://github-redirect.dependabot.com/psf/black/issues/1824">#1824</a>)</p>
  </li>
  <li>
  <p>fixed a crash when PWD=/ on POSIX (<a href="https://github-redirect.dependabot.com/psf/black/issues/1631">#1631</a>)</p>
  </li>
  <li>
  <p>fixed &quot;I/O operation on closed file&quot; when using --diff (<a href="https://github-redirect.dependabot.com/psf/black/issues/1664">#1664</a>)</p>
  </li>
  <li>
  <p>Prevent coloured diff output being interleaved with multiple files (<a href="https://github-redirect.dependabot.com/psf/black/issues/1673">#1673</a>)</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b1)](https://dependabot.com/compatibility-score/?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b1)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1311/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NzA5NDY3NjQ=
title: how to query many-to-many relationship via json API?
state: open
created_at: 2021-04-29T12:09:49Z
updated_at: 2021-04-29T12:09:49Z
author_association: NONE
type: issue
body:
  Hi,

  Firstly thanks for Datasette, it's great!

  I'm trying to use the JSON API to query data from a Datasette instance. I have a simple 3 table many-to-many relationship, like so:

  `category` - list of categories
  `document` - list of documents
  `document_category` - join table (a category contains many documents, and a document can be a member of multiple categories)

  the `document_category` table foreign keys to the other two using their respective row_ids.

  Now I want to return "all documents within category X" but I cannot see a way to do this without executing two queries; the first to lookup the row_id of category X, and the second to join `document` with `document_category` where category ID is <id>.

  I could easily write this in SQL, but this makes programmatic handling of pagination much more difficult (we'd have to dynamically modify the SQL to select the row_id and include the correct where and limit clauses).

  Is there a way to achieve this using the JSON API?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1312/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjI2MTMwMTM1
title: Bump black from 20.8b1 to 21.4b2
state: closed
created_at: 2021-04-29T13:58:06Z
updated_at: 2021-04-29T15:47:50Z
closed_at: 2021-04-29T15:47:49Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1313
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 20.8b1 to 21.4b2.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.4b2</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fix crash if the user configuration directory is inaccessible. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2158">#2158</a>)</p>
  </li>
  <li>
  <p>Clarify
  <a href="https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism">circumstances</a>
  in which <em>Black</em> may change the AST (<a href="https://github-redirect.dependabot.com/psf/black/issues/2159">#2159</a>)</p>
  </li>
  </ul>
  <h4><em>Packaging</em></h4>
  <ul>
  <li>Install <code>primer.json</code> (used by <code>black-primer</code> by default) with black. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2154">#2154</a>)</li>
  </ul>
  <h2>21.4b1</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2142">#2142</a>)</p>
  </li>
  <li>
  <p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2120">#2120</a>)</p>
  </li>
  <li>
  <p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
  in the name of the cache file. Without this fix, changes in these flags would not take
  effect if the cache had already been populated. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2131">#2131</a>)</p>
  </li>
  <li>
  <p>Don't remove necessary parentheses from assignment expression containing assert /
  return statements. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2143">#2143</a>)</p>
  </li>
  </ul>
  <h4><em>Packaging</em></h4>
  <ul>
  <li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
  </ul>
  <h2>21.4b0</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fixed a rare but annoying formatting instability created by the combination of
  optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
  pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href="https://github-redirect.dependabot.com/psf/black/issues/1629">#1629</a> and all of its many many
  duplicates. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2126">#2126</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
  and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1740">#1740</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2092">#2092</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
  docstring quotes (<a href="https://github-redirect.dependabot.com/psf/black/issues/1637">#1637</a>)</p>
  </li>
  <li>
  <p><code>Black</code> no longer removes all empty lines between non-function code and decorators
  when formatting typing stubs. Now <code>Black</code> enforces a single empty line. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1646">#1646</a>)</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/master/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h3>21.4b2</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fix crash if the user configuration directory is inaccessible. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2158">#2158</a>)</p>
  </li>
  <li>
  <p>Clarify
  <a href="https://github.com/psf/black/blob/master/docs/the_black_code_style.md#pragmatism">circumstances</a>
  in which <em>Black</em> may change the AST (<a href="https://github-redirect.dependabot.com/psf/black/issues/2159">#2159</a>)</p>
  </li>
  </ul>
  <h4><em>Packaging</em></h4>
  <ul>
  <li>Install <code>primer.json</code> (used by <code>black-primer</code> by default) with black. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2154">#2154</a>)</li>
  </ul>
  <h3>21.4b1</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fix crash on docstrings ending with &quot;\ &quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2142">#2142</a>)</p>
  </li>
  <li>
  <p>Fix crash when atypical whitespace is cleaned out of dostrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2120">#2120</a>)</p>
  </li>
  <li>
  <p>Reflect the <code>--skip-magic-trailing-comma</code> and <code>--experimental-string-processing</code> flags
  in the name of the cache file. Without this fix, changes in these flags would not take
  effect if the cache had already been populated. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2131">#2131</a>)</p>
  </li>
  <li>
  <p>Don't remove necessary parentheses from assignment expression containing assert /
  return statements. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2143">#2143</a>)</p>
  </li>
  </ul>
  <h4><em>Packaging</em></h4>
  <ul>
  <li>Bump pathspec to &gt;= 0.8.1 to solve invalid .gitignore exclusion handling</li>
  </ul>
  <h3>21.4b0</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>
  <p>Fixed a rare but annoying formatting instability created by the combination of
  optional trailing commas inserted by <code>Black</code> and optional parentheses looking at
  pre-existing &quot;magic&quot; trailing commas. This fixes issue <a href="https://github-redirect.dependabot.com/psf/black/issues/1629">#1629</a> and all of its many many
  duplicates. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2126">#2126</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now processes one-line docstrings by stripping leading and trailing spaces,
  and adding a padding space when needed to break up &quot;&quot;&quot;&quot;. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1740">#1740</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now cleans up leading non-breaking spaces in comments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2092">#2092</a>)</p>
  </li>
  <li>
  <p><code>Black</code> now respects <code>--skip-string-normalization</code> when normalizing multiline
  docstring quotes (<a href="https://github-redirect.dependabot.com/psf/black/issues/1637">#1637</a>)</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://api.dependabot.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b2)](https://dependabot.com/compatibility-score/?dependency-name=black&package-manager=pip&previous-version=20.8b1&new-version=21.4b2)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1313/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjI2MjIyNjc2
title: Upgrade to GitHub-native Dependabot
state: closed
created_at: 2021-04-29T15:36:41Z
updated_at: 2021-04-29T15:47:22Z
closed_at: 2021-04-29T15:47:21Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1314
type: pull
body:
  _Dependabot Preview will be shut down on August 3rd, 2021. In order to keep getting Dependabot updates, please merge this PR and migrate to GitHub-native Dependabot before then._

  Dependabot has been fully integrated into GitHub, so you no longer have to install and manage a separate app. This pull request migrates your configuration from Dependabot.com to a config file, using the [new syntax][new_syntax]. When merged, we'll swap out `dependabot-preview` (me) for a new `dependabot` app, and you'll be all set!

  With this change, you'll now use the [Dependabot page in GitHub][dependabot_page], rather than the [Dependabot dashboard][dashboard], to monitor your version updates, and you'll configure Dependabot through the new config file rather than a UI.







  If you've got any questions or feedback for us, please let us know by creating an issue in the [dependabot/dependabot-core][issues] repository.

  [Learn more about migrating to GitHub-native Dependabot][learn]

  Please note that regular `@dependabot` commands do not work on this pull request.

  [dashboard]: https://app.dependabot.com/
  [dependabot_page]: https://github.com/simonw/datasette/network/updates
  [issues]: https://github.com/dependabot/dependabot-core/issues/new?assignees=%40dependabot%2Fpreview-migration-reviewers&labels=E%3A+preview-migration&template=migration-issue.md
  [learn]: http://docs.github.com/code-security/supply-chain-security/upgrading-from-dependabotcom-to-github-native-dependabot
  [new_syntax]: https://help.github.com/en/github/administering-a-repository/configuration-options-for-dependency-updates
  [org_secrets_url]: https://github.com/settings/secrets/dependabot
  [repo_secrets_url]: https://github.com/simonw/datasette/settings/secrets/dependabot

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1314/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4NzEzMDQ5Njc=
title: settings.json should be picked up by "datasette publish cloudrun"
state: open
created_at: 2021-04-29T18:16:41Z
updated_at: 2021-04-29T18:16:41Z
author_association: OWNER
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1315/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjMwNTc4NzM1
title: Bump black from 21.4b2 to 21.5b0
state: closed
created_at: 2021-05-05T13:07:51Z
updated_at: 2021-05-11T13:12:32Z
closed_at: 2021-05-11T13:12:31Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1318
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 21.4b2 to 21.5b0.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.5b0</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href="https://github-redirect.dependabot.com/psf/black/issues/2169">#2169</a>)</li>
  <li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
  being called but with no arguments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2182">#2182</a>)</li>
  </ul>
  <h4><em>Black-Primer</em></h4>
  <ul>
  <li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href="https://github-redirect.dependabot.com/psf/black/issues/2187">#2187</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/master/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h3>21.5b0</h3>
  <h4><em>Black</em></h4>
  <ul>
  <li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href="https://github-redirect.dependabot.com/psf/black/issues/2169">#2169</a>)</li>
  <li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
  being called but with no arguments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2182">#2182</a>)</li>
  </ul>
  <h4><em>Black-Primer</em></h4>
  <ul>
  <li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href="https://github-redirect.dependabot.com/psf/black/issues/2187">#2187</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.4b2&new-version=21.5b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1318/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjM0ODIxMDY1
title: Add Docker multi-arch support with Buildx
state: closed
created_at: 2021-05-08T19:35:03Z
updated_at: 2021-05-27T16:49:24Z
closed_at: 2021-05-27T16:49:24Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1319
type: pull
body:
  This adds Docker support to extra CPU architectures (like arm) using [Docker's Buildx action](https://github.com/marketplace/actions/docker-setup-buildx)

  You can see [what that looks like on Dockerhub](https://hub.docker.com/r/blairdrummond/datasette/tags?page=1&ordering=last_updated)

  And how it lets Datasette run on a Raspberry Pi (top is my dockerhub, bottom is upstream)

  ![Screenshot from 2021-05-08 15-32-25](https://user-images.githubusercontent.com/10801138/117551210-a17a9f80-b012-11eb-966b-10e1590dd4a9.png)

  The workflow log [here](https://github.com/blairdrummond/datasette/runs/2535743398?check_suite_focus=true) (I subbed `blairdrummond` for datasetteproject in my branch) 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1319/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4ODQ5NTIxNzk=
title: Can't use apt-get in Dockerfile when using datasetteproj/datasette as base
state: closed
created_at: 2021-05-10T19:37:27Z
updated_at: 2021-05-24T18:15:56Z
closed_at: 2021-05-24T18:07:08Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  The datasette base Docker image is super convenient, but there's one problem: if any of the plugins you install require additional system dependencies (e.g., xz, git, curl) then any attempt to use apt in said Dockerfile results in an explosion:

  ```
  $ docker-compose build
  Building server
  [+] Building 9.9s (7/9)
   => [internal] load build definition from Dockerfile                                                                                                                                                                                                     0.0s
   => => transferring dockerfile: 666B                                                                                                                                                                                                                     0.0s
   => [internal] load .dockerignore                                                                                                                                                                                                                        0.0s
   => => transferring context: 34B                                                                                                                                                                                                                         0.0s
   => [internal] load metadata for docker.io/datasetteproject/datasette:latest                                                                                                                                                                             0.6s
   => [base 1/4] FROM docker.io/datasetteproject/datasette@sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a                                                                                                                         0.0s
   => [internal] load build context                                                                                                                                                                                                                        1.8s
   => => transferring context: 2.44MB                                                                                                                                                                                                                      1.8s
   => CACHED [base 2/4] WORKDIR /datasette                                                                                                                                                                                                                 0.0s
   => ERROR [base 3/4] RUN apt-get update     && apt-get install --no-install-recommends -y git ssh curl xz-utils                                                                                                                                          9.2s
  ------
   > [base 3/4] RUN apt-get update     && apt-get install --no-install-recommends -y git ssh curl xz-utils:
  #6 0.446 Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
  #6 0.449 Get:2 http://deb.debian.org/debian buster InRelease [121 kB]
  #6 0.459 Get:3 http://httpredir.debian.org/debian sid InRelease [157 kB]
  #6 0.784 Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
  #6 0.790 Get:5 http://httpredir.debian.org/debian sid/main amd64 Packages [8626 kB]
  #6 1.003 Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
  #6 1.180 Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [286 kB]
  #6 7.095 Get:8 http://deb.debian.org/debian buster-updates/main amd64 Packages [10.9 kB]
  #6 8.058 Fetched 17.2 MB in 8s (2243 kB/s)
  #6 8.058 Reading package lists...
  #6 9.166 E: flAbsPath on /var/lib/dpkg/status failed - realpath (2: No such file or directory)
  #6 9.166 E: Could not open file  - open (2: No such file or directory)
  #6 9.166 E: Problem opening
  #6 9.166 E: The package lists or status file could not be parsed or opened.
  ```

  The problem seems to be from completely wiping out `/var/lib/dpkg` in the upstream Dockerfile:

  https://github.com/simonw/datasette/blob/1b697539f5b53cec3fe13c0f4ada13ba655c88c7/Dockerfile#L18

  I've tested without removing the directory and apt works as expected.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1320/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjQwNDg0OTY2
title: Bump black from 21.4b2 to 21.5b1
state: closed
created_at: 2021-05-11T13:12:28Z
updated_at: 2021-05-22T23:55:39Z
closed_at: 2021-05-22T23:55:39Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1321
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 21.4b2 to 21.5b1.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.5b1</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>Refactor <code>src/black/__init__.py</code> into many files (<a href="https://github-redirect.dependabot.com/psf/black/issues/2206">#2206</a>)</li>
  </ul>
  <h3>Documentation</h3>
  <ul>
  <li>Replaced all remaining references to the <a href="https://github.com/psf/black/tree/main"><code>master</code></a> branch with the <a href="https://github.com/psf/black/tree/main"><code>main</code></a> branch. Some additional changes in the source code were also made. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2210">#2210</a>)</li>
  <li>Sigificantly reorganized the documentation to make much more sense. Check them out by heading over to <a href="https://black.readthedocs.io/en/stable/">the stable docs on RTD</a>. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2174">#2174</a>)</li>
  </ul>
  <h2>21.5b0</h2>
  <h4><em>Black</em></h4>
  <ul>
  <li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href="https://github-redirect.dependabot.com/psf/black/issues/2169">#2169</a>)</li>
  <li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
  being called but with no arguments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2182">#2182</a>)</li>
  </ul>
  <h4><em>Black-Primer</em></h4>
  <ul>
  <li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href="https://github-redirect.dependabot.com/psf/black/issues/2187">#2187</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/main/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h2>21.5b1</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>Refactor <code>src/black/__init__.py</code> into many files (<a href="https://github-redirect.dependabot.com/psf/black/issues/2206">#2206</a>)</li>
  </ul>
  <h3>Documentation</h3>
  <ul>
  <li>Replaced all remaining references to the
  <a href="https://github.com/psf/black/tree/main"><code>master</code></a> branch with the
  <a href="https://github.com/psf/black/tree/main"><code>main</code></a> branch. Some additional changes in
  the source code were also made. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2210">#2210</a>)</li>
  <li>Sigificantly reorganized the documentation to make much more sense. Check them out by
  heading over to <a href="https://black.readthedocs.io/en/stable/">the stable docs on RTD</a>.
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/2174">#2174</a>)</li>
  </ul>
  <h2>21.5b0</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>Set <code>--pyi</code> mode if <code>--stdin-filename</code> ends in <code>.pyi</code> (<a href="https://github-redirect.dependabot.com/psf/black/issues/2169">#2169</a>)</li>
  <li>Stop detecting target version as Python 3.9+ with pre-PEP-614 decorators that are
  being called but with no arguments (<a href="https://github-redirect.dependabot.com/psf/black/issues/2182">#2182</a>)</li>
  </ul>
  <h3><em>Black-Primer</em></h3>
  <ul>
  <li>Add <code>--no-diff</code> to black-primer to suppress formatting changes (<a href="https://github-redirect.dependabot.com/psf/black/issues/2187">#2187</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.4b2&new-version=21.5b1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1321/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjQzMTQ5Mjcz
title: Update click requirement from ~=7.1.1 to >=7.1.1,<8.1.0
state: closed
created_at: 2021-05-12T13:08:56Z
updated_at: 2021-05-22T23:54:48Z
closed_at: 2021-05-22T23:54:48Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1323
type: pull
body:
  Updates the requirements on [click](https://github.com/pallets/click) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/click/releases">click's releases</a>.</em></p>
  <blockquote>
  <h2>8.0.0</h2>
  <p>New major versions of all the core Pallets libraries, including Click 8.0, have been released! :tada:</p>
  <ul>
  <li>Read the announcement on our blog: <a href="https://palletsprojects.com/blog/flask-2-0-released/">https://palletsprojects.com/blog/flask-2-0-released/</a></li>
  <li>Read the full list of changes: <a href="https://click.palletsprojects.com/changes/#version-8-0-0">https://click.palletsprojects.com/changes/#version-8-0-0</a></li>
  <li>Retweet the announcement on Twitter: <a href="https://twitter.com/PalletsTeam/status/1392266507296514048">https://twitter.com/PalletsTeam/status/1392266507296514048</a></li>
  <li>Follow our blog, Twitter, or GitHub to see future announcements.</li>
  </ul>
  <p>This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.</p>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/click/blob/main/CHANGES.rst">click's changelog</a>.</em></p>
  <blockquote>
  <h2>Version 8.0.0</h2>
  <p>Released 2021-05-11</p>
  <ul>
  <li>Drop support for Python 2 and 3.5.</li>
  <li>Colorama is always installed on Windows in order to provide style
  and color support. :pr:<code>1784</code></li>
  <li>Adds a repr to Command, showing the command name for friendlier
  debugging. :issue:<code>1267</code>, :pr:<code>1295</code></li>
  <li>Add support for distinguishing the source of a command line
  parameter. :issue:<code>1264</code>, :pr:<code>1329</code></li>
  <li>Add an optional parameter to <code>ProgressBar.update</code> to set the
  <code>current_item</code>. :issue:<code>1226</code>, :pr:<code>1332</code></li>
  <li><code>version_option</code> uses <code>importlib.metadata</code> (or the
  <code>importlib_metadata</code> backport) instead of <code>pkg_resources</code>.
  :issue:<code>1582</code></li>
  <li>If validation fails for a prompt with <code>hide_input=True</code>, the value
  is not shown in the error message. :issue:<code>1460</code></li>
  <li>An <code>IntRange</code> or <code>FloatRange</code> option shows the accepted range in
  its help text. :issue:<code>1525</code>, :pr:<code>1303</code></li>
  <li><code>IntRange</code> and <code>FloatRange</code> bounds can be open (<code>&lt;</code>) instead
  of closed (<code>&lt;=</code>) by setting <code>min_open</code> and <code>max_open</code>. Error
  messages have changed to reflect this. :issue:<code>1100</code></li>
  <li>An option defined with duplicate flag names (<code>&quot;--foo/--foo&quot;</code>)
  raises a <code>ValueError</code>. :issue:<code>1465</code></li>
  <li><code>echo()</code> will not fail when using pytest's <code>capsys</code> fixture on
  Windows. :issue:<code>1590</code></li>
  <li>Resolving commands returns the canonical command name instead of the
  matched name. This makes behavior such as help text and
  <code>Context.invoked_subcommand</code> consistent when using patterns like
  <code>AliasedGroup</code>. :issue:<code>1422</code></li>
  <li>The <code>BOOL</code> type accepts the values &quot;on&quot; and &quot;off&quot;. :issue:<code>1629</code></li>
  <li>A <code>Group</code> with <code>invoke_without_command=True</code> will always invoke
  its result callback. :issue:<code>1178</code></li>
  <li><code>nargs == -1</code> and <code>nargs &gt; 1</code> is parsed and validated for
  values from environment variables and defaults. :issue:<code>729</code></li>
  <li>Detect the program name when executing a module or package with
  <code>python -m name</code>. :issue:<code>1603</code></li>
  <li>Include required parent arguments in help synopsis of subcommands.
  :issue:<code>1475</code></li>
  <li>Help for boolean flags with <code>show_default=True</code> shows the flag
  name instead of <code>True</code> or <code>False</code>. :issue:<code>1538</code></li>
  <li>Non-string objects passed to <code>style()</code> and <code>secho()</code> will be
  converted to string. :pr:<code>1146</code></li>
  <li><code>edit(require_save=True)</code> will detect saves for editors that exit
  very fast on filesystems with 1 second resolution. :pr:<code>1050</code></li>
  <li>New class attributes make it easier to use custom core objects
  throughout an entire application. :pr:<code>938</code></li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pallets/click/commit/9da166957f5848b641231d485467f6140bca2bc0"><code>9da1669</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/click/issues/1877">#1877</a> from pallets/release-8.0.0</li>
  <li><a href="https://github.com/pallets/click/commit/dfa63691631e733712d8a7d706e154f3d7b7cd5d"><code>dfa6369</code></a> release version 8.0.0</li>
  <li><a href="https://github.com/pallets/click/commit/b862cb17cb99e748bf0160583d00874aaabe2f04"><code>b862cb1</code></a> update requirements</li>
  <li><a href="https://github.com/pallets/click/commit/f51584cf506bec6624435b408ce3cf2f7a10baf6"><code>f51584c</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/click/issues/1876">#1876</a> from pallets/pre-commit-ci-schedule</li>
  <li><a href="https://github.com/pallets/click/commit/804c71c4dbf22cb5fa5bda6c2a5fb902b790f229"><code>804c71c</code></a> update pre-commit monthly</li>
  <li><a href="https://github.com/pallets/click/commit/ac655f8ac8a13cd45ce96bdc296bd857654088f0"><code>ac655f8</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/click/issues/1872">#1872</a> from janLuke/fix/formatter_write_text</li>
  <li><a href="https://github.com/pallets/click/commit/dcd991d986d43c4f12cc838c4008dfafc015c21d"><code>dcd991d</code></a> HelpFormatter.write_text uses full width</li>
  <li><a href="https://github.com/pallets/click/commit/5215fc1030ed911c0c67b18e2fcad95fc173dd0e"><code>5215fc1</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/click/issues/1870">#1870</a> from AdrienPensart/allow_colors_in_metavar</li>
  <li><a href="https://github.com/pallets/click/commit/e3e1691620c4791e662e4d8846b18295c1a018a2"><code>e3e1691</code></a> repr is erasing ANSI escapes codes</li>
  <li><a href="https://github.com/pallets/click/commit/482e6e6ae7e4da1712c16baf28c2cb239ee35b8a"><code>482e6e6</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/click/issues/1875">#1875</a> from pallets/pre-commit-ci-update-config</li>
  <li>Additional commits viewable in <a href="https://github.com/pallets/click/compare/7.1.1...8.0.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1323/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjQzMTQ5MzIw
title: Update jinja2 requirement from <2.12.0,>=2.10.3 to >=2.10.3,<3.1.0
state: closed
created_at: 2021-05-12T13:08:59Z
updated_at: 2021-05-17T17:19:41Z
closed_at: 2021-05-17T17:19:40Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1324
type: pull
body:
  Updates the requirements on [jinja2](https://github.com/pallets/jinja) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/jinja/releases">jinja2's releases</a>.</em></p>
  <blockquote>
  <h2>3.0.0</h2>
  <p>New major versions of all the core Pallets libraries, including Jinja 3.0, have been released! :tada:</p>
  <ul>
  <li>Read the announcement on our blog: <a href="https://palletsprojects.com/blog/flask-2-0-released/">https://palletsprojects.com/blog/flask-2-0-released/</a></li>
  <li>Read the full list of changes: <a href="https://jinja.palletsprojects.com/changes/#version-3-0-0">https://jinja.palletsprojects.com/changes/#version-3-0-0</a></li>
  <li>Retweet the announcement on Twitter: <a href="https://twitter.com/PalletsTeam/status/1392266507296514048">https://twitter.com/PalletsTeam/status/1392266507296514048</a></li>
  <li>Follow our blog, Twitter, or GitHub to see future announcements.</li>
  </ul>
  <p>This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.</p>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/jinja/blob/main/CHANGES.rst">jinja2's changelog</a>.</em></p>
  <blockquote>
  <h2>Version 3.0.0</h2>
  <p>Released 2021-05-11</p>
  <ul>
  <li>Drop support for Python 2.7 and 3.5.</li>
  <li>Bump MarkupSafe dependency to &gt;=1.1.</li>
  <li>Bump Babel optional dependency to &gt;=2.1.</li>
  <li>Remove code that was marked deprecated.</li>
  <li>Add type hinting. :pr:<code>1412</code></li>
  <li>Use :pep:<code>451</code> API to load templates with
  :class:<code>~loaders.PackageLoader</code>. :issue:<code>1168</code></li>
  <li>Fix a bug that caused imported macros to not have access to the
  current template's globals. :issue:<code>688</code></li>
  <li>Add ability to ignore <code>trim_blocks</code> using <code>+%}</code>. :issue:<code>1036</code></li>
  <li>Fix a bug that caused custom async-only filters to fail with
  constant input. :issue:<code>1279</code></li>
  <li>Fix UndefinedError incorrectly being thrown on an undefined variable
  instead of <code>Undefined</code> being returned on
  <code>NativeEnvironment</code> on Python 3.10. :issue:<code>1335</code></li>
  <li>Blocks can be marked as <code>required</code>. They must be overridden at
  some point, but not necessarily by the direct child. :issue:<code>1147</code></li>
  <li>Deprecate the <code>autoescape</code> and <code>with</code> extensions, they are
  built-in to the compiler. :issue:<code>1203</code></li>
  <li>The <code>urlize</code> filter recognizes <code>mailto:</code> links and takes
  <code>extra_schemes</code> (or <code>env.policies[&quot;urlize.extra_schemes&quot;]</code>) to
  recognize other schemes. It tries to balance parentheses within a
  URL instead of ignoring trailing characters. The parsing in general
  has been updated to be more efficient and match more cases. URLs
  without a scheme are linked as <code>https://</code> instead of <code>http://</code>.
  :issue:<code>522, 827, 1172</code>, :pr:<code>1195</code></li>
  <li>Filters that get attributes, such as <code>map</code> and <code>groupby</code>, can
  use a false or empty value as a default. :issue:<code>1331</code></li>
  <li>Fix a bug that prevented variables set in blocks or loops from
  being accessed in custom context functions. :issue:<code>768</code></li>
  <li>Fix a bug that caused scoped blocks from accessing special loop
  variables. :issue:<code>1088</code></li>
  <li>Update the template globals when calling
  <code>Environment.get_template(globals=...)</code> even if the template was
  already loaded. :issue:<code>295</code></li>
  <li>Do not raise an error for undefined filters in unexecuted
  if-statements and conditional expressions. :issue:<code>842</code></li>
  <li>Add <code>is filter</code> and <code>is test</code> tests to test if a name is a
  registered filter or test. This allows checking if a filter is
  available in a template before using it. Test functions can be
  decorated with <code>@pass_environment</code>, <code>@pass_eval_context</code>,
  or <code>@pass_context</code>. :issue:<code>842</code>, :pr:<code>1248</code></li>
  <li>Support <code>pgettext</code> and <code>npgettext</code> (message contexts) in i18n
  extension. :issue:<code>441</code></li>
  <li>The <code>|indent</code> filter's <code>width</code> argument can be a string to</li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pallets/jinja/commit/417f822196f66155e8c121e5229cc12a6b02ce14"><code>417f822</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/jinja/issues/1417">#1417</a> from pallets/release-3.0.0</li>
  <li><a href="https://github.com/pallets/jinja/commit/b15fd1c94c696c0e17e2a2668ce70571d613d721"><code>b15fd1c</code></a> release version 3.0.0</li>
  <li><a href="https://github.com/pallets/jinja/commit/eed1546e195338458784c242e2aace220d0559e8"><code>eed1546</code></a> update requirements</li>
  <li><a href="https://github.com/pallets/jinja/commit/9627e73aa7024ba8823eac05e69038c5a1c9dda3"><code>9627e73</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/jinja/issues/1416">#1416</a> from pallets/pre-commit-ci-schedule</li>
  <li><a href="https://github.com/pallets/jinja/commit/6630044b574577d86e26aa0823d3548acb948743"><code>6630044</code></a> update pre-commit monthly</li>
  <li><a href="https://github.com/pallets/jinja/commit/e1bf3bc3625c5e74f080594eead801ac3b0341af"><code>e1bf3bc</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/jinja/issues/1415">#1415</a> from pallets/pre-commit-ci-update-config</li>
  <li><a href="https://github.com/pallets/jinja/commit/0798a6616e0de2bb848c8e5e9582a7b6dafbac39"><code>0798a66</code></a> [pre-commit.ci] pre-commit autoupdate</li>
  <li><a href="https://github.com/pallets/jinja/commit/53fea337d1b817aa12f5833df745987427e40e6a"><code>53fea33</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/jinja/issues/1414">#1414</a> from pallets/typing-marker</li>
  <li><a href="https://github.com/pallets/jinja/commit/ba1f27e11b55cb6f6a0fd64a8f3d078369831691"><code>ba1f27e</code></a> fix py.typed in manifest</li>
  <li><a href="https://github.com/pallets/jinja/commit/1a3342b27ec09b547cd238ab0fb35c658116f822"><code>1a3342b</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/jinja/issues/1412">#1412</a> from pallets/typing</li>
  <li>Additional commits viewable in <a href="https://github.com/pallets/jinja/compare/2.10.3...3.0.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1324/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjQzMTQ5MzY0
title: Update itsdangerous requirement from ~=1.1 to >=1.1,<3.0
state: closed
created_at: 2021-05-12T13:09:03Z
updated_at: 2021-05-22T23:54:25Z
closed_at: 2021-05-22T23:54:25Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1325
type: pull
body:
  Updates the requirements on [itsdangerous](https://github.com/pallets/itsdangerous) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/itsdangerous/releases">itsdangerous's releases</a>.</em></p>
  <blockquote>
  <h2>2.0.0</h2>
  <p>New major versions of all the core Pallets libraries, including ItsDangerous 2.0, have been released! :tada:</p>
  <ul>
  <li>Read the announcement on our blog: <a href="https://palletsprojects.com/blog/flask-2-0-released/">https://palletsprojects.com/blog/flask-2-0-released/</a></li>
  <li>Read the full list of changes: <a href="https://itsdangerous.palletsprojects.com/changes/#version-2-0-0">https://itsdangerous.palletsprojects.com/changes/#version-2-0-0</a></li>
  <li>Retweet the announcement on Twitter: <a href="https://twitter.com/PalletsTeam/status/1392266507296514048">https://twitter.com/PalletsTeam/status/1392266507296514048</a></li>
  <li>Follow our blog, Twitter, or GitHub to see future announcements.</li>
  </ul>
  <p>This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.</p>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pallets/itsdangerous/blob/main/CHANGES.rst">itsdangerous's changelog</a>.</em></p>
  <blockquote>
  <h2>Version 2.0.0</h2>
  <p>Released 2021-05-11</p>
  <ul>
  <li>Drop support for Python 2 and 3.5.</li>
  <li>JWS support (<code>JSONWebSignatureSerializer</code>,
  <code>TimedJSONWebSignatureSerializer</code>) is deprecated. Use a dedicated
  JWS/JWT library such as authlib instead. :issue:<code>129</code></li>
  <li>Importing <code>itsdangerous.json</code> is deprecated. Import Python's
  <code>json</code> module instead. :pr:<code>152</code></li>
  <li>Simplejson is no longer used if it is installed. To use a different
  library, pass it as <code>Serializer(serializer=...)</code>. :issue:<code>146</code></li>
  <li><code>datetime</code> values are timezone-aware with <code>timezone.utc</code>. Code
  using <code>TimestampSigner.unsign(return_timestamp=True)</code> or
  <code>BadTimeSignature.date_signed</code> may need to change. :issue:<code>150</code></li>
  <li>If a signature has an age less than 0, it will raise
  <code>SignatureExpired</code> rather than appearing valid. This can happen if
  the timestamp offset is changed. :issue:<code>126</code></li>
  <li><code>BadTimeSignature.date_signed</code> is always a <code>datetime</code> object
  rather than an <code>int</code> in some cases. :issue:<code>124</code></li>
  <li>Added support for key rotation. A list of keys can be passed as
  <code>secret_key</code>, oldest to newest. The newest key is used for
  signing, all keys are tried for unsigning. :pr:<code>141</code></li>
  <li>Removed the default SHA-512 fallback signer from
  <code>default_fallback_signers</code>. :issue:<code>155</code></li>
  <li>Add type information for static typing tools. :pr:<code>186</code></li>
  </ul>
  <h2>Version 1.1.0</h2>
  <p>Released 2018-10-26</p>
  <ul>
  <li>Change default signing algorithm back to SHA-1. :pr:<code>113</code></li>
  <li>Added a default SHA-512 fallback for users who used the yanked 1.0.0
  release which defaulted to SHA-512. :pr:<code>114</code></li>
  <li>Add support for fallback algorithms during deserialization to
  support changing the default in the future without breaking existing
  signatures. :pr:<code>113</code></li>
  <li>Changed capitalization of packages back to lowercase as the change
  in capitalization broke some tooling. :pr:<code>113</code></li>
  </ul>
  <h2>Version 1.0.0</h2>
  <p>Released 2018-10-18</p>
  <p>YANKED</p>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pallets/itsdangerous/commit/d101100c395958d67368b8c37d95a9c404598c2e"><code>d101100</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/itsdangerous/issues/235">#235</a> from pallets/release-2.0.0</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/ca0f59ac73507014729a5857e985229604e5e83b"><code>ca0f59a</code></a> release version 2.0.0</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/d1ed89f6ac50d58fa43ce37f92db371bebc20bc5"><code>d1ed89f</code></a> update requirements</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/d1722ea35b4239b6d08e8d418edc74d6594eebd6"><code>d1722ea</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/itsdangerous/issues/234">#234</a> from pallets/pre-commit-ci-schedule</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/d1eb7aa76756268061f2d31551f79da7882b45a9"><code>d1eb7aa</code></a> update pre-commit monthly</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/acbc456c91d4a9e9f63b9230be5a1cf3d5a1d767"><code>acbc456</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/itsdangerous/issues/233">#233</a> from pallets/pre-commit-ci-update-config</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/04e485a0b0b00d8b2dc73a287852d642d3366c72"><code>04e485a</code></a> [pre-commit.ci] pre-commit autoupdate</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/c0e6b484a6cdc92c38bec74ffcf350bc42f20e2b"><code>c0e6b48</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/itsdangerous/issues/232">#232</a> from pallets/pre-commit-ci-update-config</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/6a9df8338ce39b780cebd8a7f54b8a7cb75eeab4"><code>6a9df83</code></a> [pre-commit.ci] pre-commit autoupdate</li>
  <li><a href="https://github.com/pallets/itsdangerous/commit/477f42c3e74f1a823deb3d7c3ab94a5b534d545b"><code>477f42c</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pallets/itsdangerous/issues/231">#231</a> from pallets/dependabot/pip/pre-commit-2.12.1</li>
  <li>Additional commits viewable in <a href="https://github.com/pallets/itsdangerous/compare/1.1.0...2.0.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1325/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTE5NjkwMzc=
title: How to limit fields returned from the JSON API?
state: closed
created_at: 2021-05-14T14:27:41Z
updated_at: 2021-05-23T02:55:06Z
closed_at: 2021-05-23T02:55:00Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,

  I have quite wide tables, and in many cases only want a subset of the data (to save on network bandwidth). I need to use the JSON API as handling pagination is so much easier, but I can't see a way to select specific columns.

  Is there a way to do this, or is it a feature request?

  Thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1326/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjQ1MTAwODQ4
title: Recovering of malformed ENEX file
state: open
created_at: 2021-05-15T07:49:31Z
updated_at: 2021-05-15T19:57:50Z
author_association: FIRST_TIMER
pull_request: dogsheep/evernote-to-sqlite/pulls/12
type: pull
body:
  Hey .. Awesome work developing this project, that I found very useful to me and saved me some work.. Thanks.. :)

  Some background to this PR... 
  I've been searching around for a tool allowing me to transforming my personal collection of Evernote notes to a format easier to search and potentially easier import to future services. 

  Now I discovered problem processing my large data ~5GB using the existing source using Pythons builtin xml-parser that unfortunately was unable to succeed without exception breaking the process. 

  My first attempt I tried to adapt to more robust lxml package allowing huge data and with "recover", but even if it worked better it also failed processing the whole data. Even using the memory efficient etree.iterparse() it also unfortunately got into trouble.

  And with no luck finding any other libraries successfully parsing this enormous file I instead chose to build a "hugexmlparser" module that allows parsing this huge file using yield (on a byte-to-byte-level) and allows you to set a maximum size for <note> to cater for potential malformed or undesirable large attachments to export, should succeed covering potential exceptions. Some cases found where the parses discover malformed XML within <content> so also in those cases try to save as much as possible by escaping (to be dealt at a later stage, better than nothing), and if a missing end </note> before new (malformed?) it would add this after encounter a new start-tag.

  The code for the recovery process is a bit rough and for certain room for refactoring, but at the moment is seem to achieve what I wanted.

  Now with the above we pass this a minor changed version of save_note_recovery() assure the existing works.
  Also adding this as a new recover-enex command to click and kept the original options. 
  A couple of new tests was added as well to check against using this command.

  Now this currently works to me, but thought I might share a PR in such as you find use for this yourself or found useful to others finding this repository.

  As a second step .. When the time allows it would have been nice to also be able to easily export from SQLite to formatted HTML/MD and attachments saved... but that might perhaps be better a separate project ... or if you or someone else have something that might shared to save some trouble, I would be interested ;-) 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTI0NTcyMDg=
title: Support Unicode characters in metadata.json
state: closed
created_at: 2021-05-15T14:33:58Z
updated_at: 2021-05-24T19:10:21Z
closed_at: 2021-05-24T19:10:21Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hello , when I used Burmese (Unicode) characters in metadata.json like below - 
  ![image](https://user-images.githubusercontent.com/20846286/118364978-cba70100-b5c0-11eb-967c-7dc3b62478f2.png)

  It gave wrong results when I run datasette - 
  ![image](https://user-images.githubusercontent.com/20846286/118365025-fc873600-b5c0-11eb-97ce-19541b8cc6d8.png)

  It would be great & helpful for us if metadata.json can support in Unicode supported Asian Languages. 
  Thanks & Regards. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1327/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjQ1ODQ5MDI3
title: Update aiofiles requirement from <0.7,>=0.4 to >=0.4,<0.8
state: closed
created_at: 2021-05-17T13:07:31Z
updated_at: 2021-05-22T23:53:57Z
closed_at: 2021-05-22T23:53:56Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1330
type: pull
body:
  Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/Tinche/aiofiles/commit/469c1a8b94ccccf25516706c88dde2e69ea45f3c"><code>469c1a8</code></a> v0.7.0</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/26decd33a86f1fa93599a1fad117cd8dead28c5a"><code>26decd3</code></a> Add PyPy</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/734d182bb97e1185eeb71a0f9dd92a4645779795"><code>734d182</code></a> README update</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/b0bb11077d9cedf2527422b855dc4f11acbb07cf"><code>b0bb110</code></a> More coverage</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/e28714d2c74cee0f5b0cd54e659467f9e8bf6f70"><code>e28714d</code></a> Fix CI</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/6765e3de2bfa5939548f478ce186269771174d91"><code>6765e3d</code></a> Switch to poetry some more</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/165ba880eaed010a15b590327cd8a7f396fc6f5b"><code>165ba88</code></a> Switch to poetry</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/aba1c10453f174391b1a40fc8c769a8e813c2907"><code>aba1c10</code></a> Try isolated_build</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/f8b0e2710e53f57368be5579b4826b51a2c02846"><code>f8b0e27</code></a> passenv</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/d291390e8b93b3585236233f49ad2a3b91744523"><code>d291390</code></a> Tweak Tox</li>
  <li>Additional commits viewable in <a href="https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.7.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1330/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTM1Mzc3NDQ=
title: Add support for Jinja2 version 3.0
state: closed
created_at: 2021-05-17T17:14:36Z
updated_at: 2021-05-23T00:57:39Z
closed_at: 2021-05-23T00:57:39Z
author_association: NONE
type: issue
state_reason: completed
body:
  A week ago, [The Pallets Project](https://github.com/pallets) released [new major versions of several of its projects](https://palletsprojects.com/blog/flask-2-0-released/). Among those updates is one for Jinja2, which bumps it to version 3.0.0.

  I'd like for datasette to support Jinaj2 version 3.0.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1331/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTM4OTA0OTY=
title: ?_facet_size=X to increase number of facets results on the page
state: closed
created_at: 2021-05-18T02:40:16Z
updated_at: 2021-05-27T16:13:07Z
closed_at: 2021-05-23T00:34:37Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Is there a way to add a parameter to the URL to modify default_facet_size?

  LIkewise, a way to produce a link on the three dots to expand to all items (or match previous number of items, or add x more)?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1332/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTQ5NDgxMDA=
title: Suggest the --alter option if a new column cannot be added
state: closed
created_at: 2021-05-19T03:17:38Z
updated_at: 2021-05-19T03:27:33Z
closed_at: 2021-05-19T03:26:26Z
author_association: OWNER
body: Refs #256.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/259/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjQ3NTUyMTQx
title: Fix small typo
state: closed
created_at: 2021-05-19T11:17:04Z
updated_at: 2021-05-22T23:53:34Z
closed_at: 2021-05-22T23:53:34Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1335
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1335/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTU2ODYwMzk=
title: Document turning on WAL for live served SQLite databases
state: open
created_at: 2021-05-19T17:08:58Z
updated_at: 2022-01-13T21:55:59Z
author_association: OWNER
type: issue
body:
  Datasette docs don't talk about WAL yet, which allows you to safely serve reads from a database file while it is accepting writes.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1336/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTcyMTI0NTg=
title: Ability to fetch commits from branches other than the default
state: open
created_at: 2021-05-20T17:58:08Z
updated_at: 2021-05-20T17:58:08Z
author_association: MEMBER
type: issue
body:
  This tool is currently almost entirely ignorant of the concept of branches. One example: you can't retrieve commits from any branch other than the default (usually main).

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/63/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTg5MDQ0MDI=
title: "More" link for facets that shows _facet_size=max results
state: closed
created_at: 2021-05-23T00:08:51Z
updated_at: 2021-05-27T16:14:14Z
closed_at: 2021-05-27T16:01:03Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Original title:  "More" link for facets that shows the full set of results_

  The simplest way to do this will be to have it link to a generated SQL query.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1332#issuecomment-846479062_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1337/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4OTkxNjkzMDc=
title: Fix jinja2 warnings
state: closed
created_at: 2021-05-24T01:38:23Z
updated_at: 2021-05-24T01:41:55Z
closed_at: 2021-05-24T01:41:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Lots of these in the test suite now, after the Jinja upgrade in #1331:
  ```
  tests/test_plugins.py::test_hook_render_cell_link_from_json
    datasette/tests/plugins/my_plugin_2.py:45: DeprecationWarning: 'jinja2.escape' is deprecated and will be removed in Jinja 3.1. Import 'markupsafe.escape' instead.
      label=jinja2.escape(data["label"] or "") or "&nbsp;",

  tests/test_plugins.py::test_hook_render_cell_link_from_json
    datasette/tests/plugins/my_plugin_2.py:41: DeprecationWarning: 'jinja2.Markup' is deprecated and will be removed in Jinja 3.1. Import 'markupsafe.Markup' instead.
      return jinja2.Markup(
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1338/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjUxNDkyODg4
title: ?_col=/?_nocol= to show/hide columns on the table page
state: closed
created_at: 2021-05-24T17:15:20Z
updated_at: 2021-05-27T04:17:44Z
closed_at: 2021-05-27T04:17:43Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1339
type: pull
body:
  See #615. Still to do:

  - [x] Allow combination of `?_col=` and `?_nocol=` (`_nocol` wins)
  - [x] Deduplicate same column if passed in `?_col=` multiple times
  - [x] Validate that user did not try to remove a primary key
  - [x] Add tests
  - [x] Ensure this works correctly for SQL views
  - [x] Add documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1339/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDEwMDk3ODc=
title: Research: Cell action menu (like column action but for individual cells)
state: open
created_at: 2021-05-25T15:49:16Z
updated_at: 2021-05-26T18:59:58Z
author_association: OWNER
type: issue
body:
  Had an idea today that it might be useful to select an individual cell and say things like "show me all other rows with the same value" - maybe even a set of other menu options against cells as well.

  Mocked up a show-on-hover ellipses demo using the CSS inspector:

  ![idea](https://user-images.githubusercontent.com/9599/119528316-f0744480-bd35-11eb-8eb4-1deea6d60cce.gif)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1340/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDMyMDAzMjg=
title: "Show all columns" cog menu item should show if ?_col= is used
state: closed
created_at: 2021-05-27T04:28:17Z
updated_at: 2021-05-27T04:31:16Z
closed_at: 2021-05-27T04:31:16Z
author_association: OWNER
type: issue
state_reason: completed
body:
  On https://latest.datasette.io/fixtures/sortable?_col=sortable the "Show all columns" item (from #615) is not shown (it should be):

  <img width="630" alt="fixtures__sortable__201_rows" src="https://user-images.githubusercontent.com/9599/119766097-44337a80-be69-11eb-8674-8031e72cb17d.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1341/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDM5MDI0OTU=
title: Improve `path_with_replaced_args()` and friends and document them
state: open
created_at: 2021-05-27T15:18:28Z
updated_at: 2021-05-27T15:23:02Z
author_association: OWNER
type: issue
body:
  > In order to cleanly implement this I need to expose the `path_with_replaced_args` utility function to Datasette's template engine. This is the first time this will become an exposed (and hence should-by-documented) API and I don't like its shape much.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1337#issuecomment-849721280_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1342/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDM5NzgxMzM=
title: Figure out how to publish alpha/beta releases to Docker Hub
state: closed
created_at: 2021-05-27T16:42:17Z
updated_at: 2021-05-27T16:46:37Z
closed_at: 2021-05-27T16:45:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the `latest` tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1319#issuecomment-849780481_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1343/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDM5ODYxNzg=
title: Test Datasette Docker images built for different architectures
state: open
created_at: 2021-05-27T16:52:29Z
updated_at: 2022-09-06T00:07:58Z
author_association: OWNER
type: issue
body:
  Continuing on from #1319 - now that we have the ability to build Datasette's Docker image against multiple architectures we should test that it works.

  We can do this with QEMU emulation, see https://twitter.com/nevali/status/1397958044571602945

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1344/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDQwNzE5Mzg=
title: ?_nocol= does not interact well with default facets
state: closed
created_at: 2021-05-27T18:39:55Z
updated_at: 2021-05-31T02:40:44Z
closed_at: 2021-05-31T02:31:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Clicking "Hide this column" on `fips` on https://covid-19.datasettes.com/covid/ny_times_us_counties shows this error:

  https://covid-19.datasettes.com/covid/ny_times_us_counties?_nocol=fips

  > ## Invalid SQL
  > no such column: fips

  The reason is that https://covid-19.datasettes.com/-/metadata sets up the following:

  ```json
    "ny_times_us_counties": {
        "sort_desc": "date",
        "facets": [
            "state",
            "county",
            "fips"
        ],
  ```
  It's setting `fips` as a default facet, which breaks if you attempt to remove the column using `?_nocol`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1345/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjU1Njg0NDc3
title: Re-display user's query with an error message if an error occurs
state: closed
created_at: 2021-05-28T02:04:20Z
updated_at: 2021-06-02T03:46:21Z
closed_at: 2021-06-02T03:46:21Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1346
body: Refs #619
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1346/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjU1NzI2Mzg3
title: Test docker platform blair only
state: closed
created_at: 2021-05-28T02:47:09Z
updated_at: 2021-05-28T02:47:28Z
closed_at: 2021-05-28T02:47:28Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1347
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1347/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjU1NzQxNDI4
title: DRAFT: add test and scan for docker images
state: open
created_at: 2021-05-28T03:02:12Z
updated_at: 2021-05-28T03:06:16Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1348
type: pull
body:
  **NOTE: I don't think this PR is ready, since the arm/v6 and arm/v7 images are failing pytest due to missing dependencies (gcc and friends). But it's pretty close.**

  Closes https://github.com/simonw/datasette/issues/1344 . Using a build-matrix for the platforms and [this test](https://github.com/simonw/datasette/issues/1344#issuecomment-849820019), we test all the platforms in parallel. I also threw in container scanning.

  ### Switch `pip install` to use either tags or commit shas

  Notably! This also [changes the Dockerfile](https://github.com/blairdrummond/datasette/blob/7fe5315d68e04fce64b5bebf4e2d7feec44f8546/Dockerfile#L20) so that it accepts tags or commit-shas.

  ```
  # It's backwards compatible with tags, but also lets you use shas
  root@712071df17af:/# pip install git+git://github.com/simonw/datasette.git@0.56                                                                                                                                            
  Collecting git+git://github.com/simonw/datasette.git@0.56                                                                                                                                                                  
    Cloning git://github.com/simonw/datasette.git (to revision 0.56) to /tmp/pip-req-build-u6dhm945                                                                                                                          
    Running command git clone -q git://github.com/simonw/datasette.git /tmp/pip-req-build-u6dhm945                                                                                                                           
    Running command git checkout -q af5a7f1c09f6a902bb2a25e8edf39c7034d2e5de                                                                                                                                                 
  Collecting Jinja2<2.12.0,>=2.10.3                                                                            
    Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB) 
  ```

  This lets you build the containers in CI every push for testing, which maybe resolves [this problem](https://github.com/simonw/datasette/issues/1272#issuecomment-808648974)?

  # Workflow run example

  You can see the results in my workflow [here](https://github.com/blairdrummond/datasette/pull/2/checks?check_run_id=2690570717). The commit history is different because I squashed this branch, also in the testing branch I had to change `github.com/simonw` to `github.com/blairdrummond` for the CI to pick up my git_sha.

  ## Why did the builds fail?

  **NOTE:** The results of all the tests fail, but for different reasons! A few fail to install Rust, the amd64 passes the tests (phew!) but has critical CVEs which fail the container scan, the Arm/v6 and Arm/v7 seem to fail to install the test dependencies due to missing programs like `gcc`. (`gcc` is not sufficient though, as [this run](https://github.com/blairdrummond/datasette/pull/3/checks?check_run_id=2690672982) indicates) 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1348/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDYzMzAxODc=
title: Support creating descending order indexes
state: closed
created_at: 2021-05-29T03:42:59Z
updated_at: 2021-05-29T05:01:39Z
closed_at: 2021-05-29T05:01:39Z
author_association: OWNER
type: issue
state_reason: completed
body:
  SQLite lets you create indexes in reverse order, which can have a surprisingly big impact on performance, see https://github.com/simonw/covid-19-datasette/issues/27

  I tried doing this using `sqlite-utils` like so, but it's didn't work:

  ```python
  db["ny_times_us_counties"].create_index(["date desc"])
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/260/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDYzNDU4OTk=
title: `table.xindexes` using `PRAGMA index_xinfo(table)`
state: closed
created_at: 2021-05-29T04:23:48Z
updated_at: 2021-06-03T03:54:14Z
closed_at: 2021-06-03T03:51:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > `PRAGMA index_xinfo(table)` DOES return that data:
  > ```
  > (Pdb) [c[0] for c in fresh_db.execute("PRAGMA > index_xinfo('idx_dogs_age_name')").description]
  > ['seqno', 'cid', 'name', 'desc', 'coll', 'key']
  > (Pdb) fresh_db.execute("PRAGMA index_xinfo('idx_dogs_age_name')").fetchall()
  > [(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)]
  > ```
  > See https://sqlite.org/pragma.html#pragma_index_xinfo
  > 
  > Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/261/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjU3MzczNzI2
title: Ability to add descending order indexes
state: closed
created_at: 2021-05-29T04:51:04Z
updated_at: 2021-05-29T05:01:42Z
closed_at: 2021-05-29T05:01:39Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/262
body: Refs #260
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/262/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDYzNTYzMzE=
title: `sqlite-utils indexes` command
state: closed
created_at: 2021-05-29T04:52:34Z
updated_at: 2021-06-03T04:34:38Z
closed_at: 2021-06-03T04:34:38Z
author_association: OWNER
type: issue
state_reason: completed
body:
  While working on #260 I realized there's no command to show indexes in a database, even though there is one for showing tables and one for triggers.

  I should implement #261 first.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/263/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDYzODU5OTE=
title: CSV ?_stream=on redundantly calculates facets for every page
state: closed
created_at: 2021-05-29T06:11:23Z
updated_at: 2021-06-17T18:12:32Z
closed_at: 2021-06-01T15:52:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm trying to figure out why a full CSV export from https://covid-19.datasettes.com/covid/ny_times_us_counties runs unbearably slowly.

  It's because the streaming endpoint works by scrolling through every page, and it turns out every page calculates facets and suggested facets!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1349/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDY5Nzc3MTk=
title: ?_nofacets=1 query string argument for disabling facets and suggested facets
state: closed
created_at: 2021-05-31T02:22:29Z
updated_at: 2021-06-01T16:19:38Z
closed_at: 2021-05-31T02:39:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This is needed as an internal option for #1349. `datasette-graphql` can benefit from this too - maybe can even use it so that if you pass `?_shape=array` it gets automatically added, fixing #263.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1350/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDY5OTM3MzE=
title: Get `?_trace=1` working with CSV and streaming CSVs
state: closed
created_at: 2021-05-31T03:02:15Z
updated_at: 2021-06-17T18:12:32Z
closed_at: 2021-06-01T15:50:09Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1349#issuecomment-851133125_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1351/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDc2NDI1NDY=
title: Supporting additional output formats, like GeoJSON
state: closed
created_at: 2021-05-31T18:03:32Z
updated_at: 2021-06-03T05:12:21Z
closed_at: 2021-06-03T05:12:21Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  I have a project going where it would be useful to do some spatial processing in SQLite (instead of large files) and then output GeoJSON. So my workflow would be something like this:

  1. Read Shapefiles, GeoJSON, CSVs into a SQLite database
  2. Join, filter, prune as needed
  3. Export GeoJSON for just the stuff I need at that moment, while still having a database of things that will be useful later

  I'm wondering if this is worth adding to SQLite-utils itself (GeoJSON, at least), or if it's better to make a counterpart to the ecosystem of `*-to-sqlite` tools, say a suite of `sqlite-to-*` things. Or would it be crazy to have a plugin system?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/264/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDc2NDU4MTM=
title: Error: Use either --since or --since_id, not both
state: closed
created_at: 2021-05-31T18:11:04Z
updated_at: 2021-08-20T00:01:31Z
closed_at: 2021-08-20T00:01:31Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  I'm using the following command:

  ```
  twitter-to-sqlite user-timeline -a twitter-auth.json twitter/tweets.db --since
  ```

  Which gives the following error:
  ```
  Error: Use either --since or --since_id, not both
  ```

  Running without `--since`.

  ```
  Traceback (most recent call last):
    File "/usr/local/bin/twitter-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
      return __callback(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/cli.py", line 317, in user_timeline
      for tweet in bar:
    File "/usr/local/lib/python3.9/site-packages/click/_termui_impl.py", line 328, in generator
      for rv in self.iter:
    File "/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/utils.py", line 234, in fetch_user_timeline
      yield from fetch_timeline(
    File "/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/utils.py", line 202, in fetch_timeline
      raise Exception(str(tweets["errors"]))
  Exception: [{'code': 44, 'message': 'since_id parameter is invalid.'}]
  ```

  ```
  Python 3.9.5
  twitter-to-sqlite, version 0.21.3
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57/reactions", "total_count": 4, "+1": 4, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDc3OTU1NjI=
title: Using enable_fts before search term
state: open
created_at: 2021-06-01T01:43:34Z
updated_at: 2023-04-01T17:27:18Z
author_association: NONE
type: issue
body:
  Many thanks for the sqlite-utils suite of utilities. Has made my life much much easier. 
  I used this to create a table and enable FTS. All works fine. The datasette utility detects FTS and shows a text box. Searching for a term using that interface works well.

  However, when I start to use features by following https://www.sqlite.org/fts5.html section **"3. Full-text Query Syntax"** I seem to run into issues that I suspect is due to `escape_fts` wrapper function.

  As an example, if i search for the term `"^குகை" `on the text box in datasette it produces 140 results. However, when i tweak the query produced by datasette to not use "escape_fts" it produces 5 results.

  Similarly, when I try to restrict the search to a single column in FTS using a spec like `{title : ^குகை}` it returns no rows. The same thing pulls results when used without `escape_fts`. The text in the table is in Tamil language and the search term is a Tamil word.

  ```
     ...
      where
        posts_fts match escape_fts(:search)
  ```
  vs

  ```
      ...
      where
        posts_fts match (:search)
  ```

  Any ideas why? How can I get the benefits of both escaping as well as utilizing different facets of providing / controlling search terms? Thanks.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/265/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjU4OTkxNDA0
title: Bump black from 21.5b1 to 21.5b2
state: closed
created_at: 2021-06-01T13:08:52Z
updated_at: 2021-06-02T02:56:45Z
closed_at: 2021-06-02T02:56:44Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1352
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 21.5b1 to 21.5b2.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.5b2</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>A space is no longer inserted into empty docstrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2249">#2249</a>)</li>
  <li>Fix handling of .gitignore files containing non-ASCII characters on Windows (<a href="https://github-redirect.dependabot.com/psf/black/issues/2229">#2229</a>)</li>
  <li>Respect <code>.gitignore</code> files in all levels, not only <code>root/.gitignore</code> file (apply
  <code>.gitignore</code> rules like <code>git</code> does) (<a href="https://github-redirect.dependabot.com/psf/black/issues/2225">#2225</a>)</li>
  <li>Restored compatibility with Click 8.0 on Python 3.6 when LANG=C used (<a href="https://github-redirect.dependabot.com/psf/black/issues/2227">#2227</a>)</li>
  <li>Add extra uvloop install + import support if in python env (<a href="https://github-redirect.dependabot.com/psf/black/issues/2258">#2258</a>)</li>
  <li>Fix --experimental-string-processing crash when matching parens are not found (<a href="https://github-redirect.dependabot.com/psf/black/issues/2283">#2283</a>)</li>
  <li>Make sure to split lines that start with a string operator (<a href="https://github-redirect.dependabot.com/psf/black/issues/2286">#2286</a>)</li>
  <li>Fix regular expression that black uses to identify f-expressions (<a href="https://github-redirect.dependabot.com/psf/black/issues/2287">#2287</a>)</li>
  </ul>
  <h3><em>Blackd</em></h3>
  <ul>
  <li>Add a lower bound for the <code>aiohttp-cors</code> dependency. Only 0.4.0 or higher is
  supported. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2231">#2231</a>)</li>
  </ul>
  <h3><em>Packaging</em></h3>
  <ul>
  <li>Release self-contained x86_64 MacOS binaries as part of the GitHub release pipeline
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/2198">#2198</a>)</li>
  <li>Always build binaries with the latest available Python (<a href="https://github-redirect.dependabot.com/psf/black/issues/2260">#2260</a>)</li>
  </ul>
  <h3>Documentation</h3>
  <ul>
  <li>Add discussion of magic comments to FAQ page (<a href="https://github-redirect.dependabot.com/psf/black/issues/2272">#2272</a>)</li>
  <li><code>--experimental-string-processing</code> will be enabled by default in the future (<a href="https://github-redirect.dependabot.com/psf/black/issues/2273">#2273</a>)</li>
  <li>Fix typos discovered by codespell (<a href="https://github-redirect.dependabot.com/psf/black/issues/2228">#2228</a>)</li>
  <li>Fix Vim plugin installation instructions. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2235">#2235</a>)</li>
  <li>Add new Frequently Asked Questions page (<a href="https://github-redirect.dependabot.com/psf/black/issues/2247">#2247</a>)</li>
  <li>Fix encoding + symlink issues preventing proper build on Windows (<a href="https://github-redirect.dependabot.com/psf/black/issues/2262">#2262</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/main/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h2>21.5b2</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>A space is no longer inserted into empty docstrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2249">#2249</a>)</li>
  <li>Fix handling of .gitignore files containing non-ASCII characters on Windows (<a href="https://github-redirect.dependabot.com/psf/black/issues/2229">#2229</a>)</li>
  <li>Respect <code>.gitignore</code> files in all levels, not only <code>root/.gitignore</code> file (apply
  <code>.gitignore</code> rules like <code>git</code> does) (<a href="https://github-redirect.dependabot.com/psf/black/issues/2225">#2225</a>)</li>
  <li>Restored compatibility with Click 8.0 on Python 3.6 when LANG=C used (<a href="https://github-redirect.dependabot.com/psf/black/issues/2227">#2227</a>)</li>
  <li>Add extra uvloop install + import support if in python env (<a href="https://github-redirect.dependabot.com/psf/black/issues/2258">#2258</a>)</li>
  <li>Fix --experimental-string-processing crash when matching parens are not found (<a href="https://github-redirect.dependabot.com/psf/black/issues/2283">#2283</a>)</li>
  <li>Make sure to split lines that start with a string operator (<a href="https://github-redirect.dependabot.com/psf/black/issues/2286">#2286</a>)</li>
  <li>Fix regular expression that black uses to identify f-expressions (<a href="https://github-redirect.dependabot.com/psf/black/issues/2287">#2287</a>)</li>
  </ul>
  <h3><em>Blackd</em></h3>
  <ul>
  <li>Add a lower bound for the <code>aiohttp-cors</code> dependency. Only 0.4.0 or higher is
  supported. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2231">#2231</a>)</li>
  </ul>
  <h3>Integrations</h3>
  <ul>
  <li>The official Black action now supports choosing what version to use, and supports the
  major 3 OSes. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1940">#1940</a>)</li>
  </ul>
  <h3>Packaging</h3>
  <ul>
  <li>Release self-contained x86_64 MacOS binaries as part of the GitHub release pipeline
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/2198">#2198</a>)</li>
  <li>Always build binaries with the latest available Python (<a href="https://github-redirect.dependabot.com/psf/black/issues/2260">#2260</a>)</li>
  </ul>
  <h3>Documentation</h3>
  <ul>
  <li>Add discussion of magic comments to FAQ page (<a href="https://github-redirect.dependabot.com/psf/black/issues/2272">#2272</a>)</li>
  <li><code>--experimental-string-processing</code> will be enabled by default in the future (<a href="https://github-redirect.dependabot.com/psf/black/issues/2273">#2273</a>)</li>
  <li>Fix typos discovered by codespell (<a href="https://github-redirect.dependabot.com/psf/black/issues/2228">#2228</a>)</li>
  <li>Fix Vim plugin installation instructions. (<a href="https://github-redirect.dependabot.com/psf/black/issues/2235">#2235</a>)</li>
  <li>Add new Frequently Asked Questions page (<a href="https://github-redirect.dependabot.com/psf/black/issues/2247">#2247</a>)</li>
  <li>Fix encoding + symlink issues preventing proper build on Windows (<a href="https://github-redirect.dependabot.com/psf/black/issues/2262">#2262</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.5b1&new-version=21.5b2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1352/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDg0NDY5OTc=
title: ?_nocount=1 for opting out of table counts
state: closed
created_at: 2021-06-01T15:53:27Z
updated_at: 2021-06-01T16:18:54Z
closed_at: 2021-06-01T16:17:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Running a trace against a CSV streaming export with the new `_trace=1` feature from #1351 shows that the following code is executing a `select count(*) from table` for every page of results returned: https://github.com/simonw/datasette/blob/d1d06ace49606da790a765689b4fbffa4c6deecb/datasette/views/table.py#L700-L705

  This is inefficient - a new `?_nocount=1` option would let us disable this count in the same way as #1349: https://github.com/simonw/datasette/blob/d1d06ace49606da790a765689b4fbffa4c6deecb/datasette/views/base.py#L264-L276

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1353/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MDg0NjU3NDc=
title: Update help in tests for latest Click
state: closed
created_at: 2021-06-01T16:14:31Z
updated_at: 2021-06-01T16:17:04Z
closed_at: 2021-06-01T16:17:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Now that Uvicorn 0.14 is out with an unpinned Click dependency - https://github.com/encode/uvicorn/pull/1033 - our test suite runs against Click 8.0 - which subtly changes the output of `--help` causing test failures: https://github.com/simonw/datasette/runs/2720383031?check_suite_focus=true
  ```
      def test_help_includes(name, filename):
          expected = (docs_path / filename).read_text()
          runner = CliRunner()
          result = runner.invoke(cli, name.split() + ["--help"], terminal_width=88)
          actual = f"$ datasette {name} --help\n\n{result.output}"
          # actual has "Usage: cli package [OPTIONS] FILES"
          # because it doesn't know that cli will be aliased to datasette
          expected = expected.replace("Usage: datasette", "Usage: cli")
  >       assert expected == actual
  E       AssertionError: assert '$ datasette ...e and exit.\n' == '$ datasette ...e and exit.\n'
  E         Skipping 848 identical leading characters in diff, use -v to show
  E           nt_id xxx
  E         + 
  E             --version-note TEXT             Additional note to show on /-/versions
  E             --secret TEXT                   Secret used for signing secure values, such as signed
  E                                             cookies
  E         + 
  E             --title TEXT                    Title for metadata
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1354/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTAwODg5MzY=
title: datasette --get should efficiently handle streaming CSV
state: open
created_at: 2021-06-03T04:40:40Z
updated_at: 2022-03-20T22:38:53Z
author_association: OWNER
type: issue
body:
  It would be great if you could use `datasette --get` to run queries that return streaming CSV data without running out of RAM.

  Current implementation looks like it loads the entire result into memory first: https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/cli.py#L546-L552

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1355/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTAwOTI1Nzc=
title: Research: syntactic sugar for using --get with SQL queries, maybe "datasette query"
state: open
created_at: 2021-06-03T04:49:42Z
updated_at: 2022-01-20T01:06:37Z
author_association: OWNER
type: issue
body:
  Inspired by https://github.com/simonw/sqlite-utils/issues/264 - in particular this example:
  ```
  datasette covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 1' 
  - date: '2020-01-21'
    county: Snohomish
    state: Washington
    fips: 53061
    cases: 1
    deaths: 0
  ```
  Having to construct that URL - including potentially URL escaping the SQL query - isn't a great developer experience.

  Imagine if you could do this instead:

      datasette covid.db --query "select * from ny_times_us_counties limit 1" --format yaml

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1356/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjYyNTU3MjQw
title: Make custom pages compatible with base_url setting
state: closed
created_at: 2021-06-05T18:54:39Z
updated_at: 2021-06-05T18:59:54Z
closed_at: 2021-06-05T18:59:54Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1357
body: Refs #1238.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1357/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTI0MTgwOTQ=
title: Release Datasette 0.57
state: closed
created_at: 2021-06-05T19:56:13Z
updated_at: 2021-06-05T22:20:07Z
closed_at: 2021-06-05T22:20:07Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Need release notes. Changes are here: https://github.com/simonw/datasette/compare/0.56...368aa5f1b16ca35f82d90ff747023b9a2bfa27c1

  Partial release notes already exist for the two alphas, https://github.com/simonw/datasette/releases/tag/0.57a0 and https://github.com/simonw/datasette/releases/tag/0.57a1

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1358/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTI0MTkzNDk=
title: `?_trace=1` should only be available with a new `trace_debug` setting
state: closed
created_at: 2021-06-05T19:59:27Z
updated_at: 2021-06-05T20:18:46Z
closed_at: 2021-06-05T20:18:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Just like template debug mode is controlled by this off-by-default setting: https://github.com/simonw/datasette/blob/368aa5f1b16ca35f82d90ff747023b9a2bfa27c1/datasette/app.py#L160-L164

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1359/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTI0NjQ0NDM=
title: Security flaw, to be fixed in 0.56.1 and 0.57
state: closed
created_at: 2021-06-05T21:53:51Z
updated_at: 2021-06-05T22:23:23Z
closed_at: 2021-06-05T22:22:06Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See security advisory here for details: https://github.com/simonw/datasette/security/advisories/GHSA-xw7c-jx9m-xh5g - the `?_trace=1` debugging option was not correctly escaping its JSON output, resulting in a [reflected cross-site scripting](https://owasp.org/www-community/attacks/xss/#reflected-xss-attacks) vulnerability.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1360/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTI0ODUwNDA=
title: Intermittent CI failure: restore_working_directory FileNotFoundError
state: closed
created_at: 2021-06-05T22:48:13Z
updated_at: 2021-06-05T23:16:24Z
closed_at: 2021-06-05T23:16:24Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. in https://github.com/simonw/datasette/runs/2754772233 - this is an intermittent error:
  ```
  __________ ERROR at setup of test_hook_register_routes_render_message __________
  [gw0] linux -- Python 3.8.10 /opt/hostedtoolcache/Python/3.8.10/x64/bin/python

  tmpdir = local('/tmp/pytest-of-runner/pytest-0/popen-gw0/test_hook_register_routes_rend0')
  request = <SubRequest 'restore_working_directory' for <Function test_hook_register_routes_render_message>>

      @pytest.fixture
      def restore_working_directory(tmpdir, request):
  >       previous_cwd = os.getcwd()
  E       FileNotFoundError: [Errno 2] No such file or directory
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1361/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTI4NjQ5MzY=
title: Consider using CSP to protect against future XSS
state: open
created_at: 2021-06-06T15:32:20Z
updated_at: 2022-10-08T18:42:09Z
author_association: OWNER
type: issue
body:
  The XSS in #1360 would have been a lot less damaging if Datasette used CSP to protect against such vulnerabilities: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1362/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTI5NTkyNjQ=
title: Don't truncate columns on the list of databases
state: closed
created_at: 2021-06-06T22:01:56Z
updated_at: 2021-06-06T22:07:50Z
closed_at: 2021-06-06T22:07:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://covid-19.datasettes.com/covid currently truncates at 9 database columns:

  <img width="860" alt="covid" src="https://user-images.githubusercontent.com/9599/120941536-11467d80-c6d8-11eb-970a-ce469623f92c.png">

  Django SQL Dashboard showed me that this is a bad idea - having the full list of columns is actually really useful documentation for crafting custom SQL queries.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1364/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTMwMTc1Nzc=
title: pathlib.Path breaks internal schema
state: closed
created_at: 2021-06-07T01:40:37Z
updated_at: 2021-06-21T15:57:39Z
closed_at: 2021-06-21T15:57:39Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Ran into an issue while trying to build a plugin to render GeoJSON. I'm using pytest's `tmp_path` fixture, which is a `pathlib.Path`, to get a temporary database path. I was getting a weird error involving writes, but I was doing reads. Turns out it's the internal database trying to insert a `Path` where it wants a string.

  My test looked like this:

  ```python
  @pytest.mark.asyncio
  async def test_render_feature_collection(tmp_path):
      database = tmp_path / "test.db"
      datasette = Datasette([database])

      # this will break with a path
      await datasette.refresh_schemas()

      # build a url
      url = datasette.urls.table(database.stem, TABLE_NAME, format="geojson")

      response = await datasette.client.get(url)
      fc = response.json()

      assert 200 == response.status_code
  ```

  I only ran into this while running tests, because passing in database paths from the CLI uses strings, but it's a weird error and probably something other people have run into.

  The fix is easy enough: Convert the path to a string and everything works. So this:

  ```python
  @pytest.mark.asyncio
  async def test_render_feature_collection(tmp_path):
      database = tmp_path / "test.db"
      datasette = Datasette([str(database)])

      # this is fine now
      await datasette.refresh_schemas()
  ```

  This could (probably, haven't tested) be fixed [here](https://github.com/simonw/datasette/blob/03ec71193b9545536898a4bc7493274fec48bdd7/datasette/app.py#L357) by calling `str(db.path)` or by doing that conversion earlier.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1365/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTMxMzU3MjM=
title: Add some types, enforce with mypy
state: closed
created_at: 2021-06-07T06:05:56Z
updated_at: 2021-08-18T22:25:38Z
closed_at: 2021-08-18T22:25:38Z
author_association: OWNER
type: issue
state_reason: completed
body:
  A good starting point would be adding type information to the members of these named tuples and the introspection methods that return them:

  https://github.com/simonw/sqlite-utils/blob/9dff7a38831d471b1dff16d40d89eb5c3b4e84d6/sqlite_utils/db.py#L51-L75

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/266/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTM4MDk4MDI=
title: Get rid of this `restore_working_directory` hack entirely
state: open
created_at: 2021-06-07T18:01:21Z
updated_at: 2021-06-07T18:03:03Z
author_association: OWNER
type: issue
body:
  > That seems to have fixed it. I'd love to get rid of this `restore_working_directory` hack entirely.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1361#issuecomment-855308811_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1366/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTM4MjM4ODk=
title: Navigation menu display bug
state: closed
created_at: 2021-06-07T18:18:08Z
updated_at: 2021-06-07T18:24:19Z
closed_at: 2021-06-07T18:24:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  With Datasette 0.57 the navigation menu looks like this:

  <img width="991" alt="Datasette_Fixtures___memory___internal__fixtures__extra_database" src="https://user-images.githubusercontent.com/9599/121069137-ffb9b000-c781-11eb-962a-b250a2772590.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1367/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjYzODM2OTY1
title: DRAFT: A new plugin hook for dynamic metadata
state: closed
created_at: 2021-06-07T18:56:00Z
updated_at: 2021-06-26T22:24:54Z
closed_at: 2021-06-26T22:24:54Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1368
type: pull
body:
  Note that this is a WORK IN PROGRESS!

  This PR adds the following plugin hook:

      get_metadata(
        datasette=self, key=key, database=database, table=table,
        fallback=fallback
      )

  This gets called when we're building our metdata for the rest of the system to use. Datasette merges whatever the plugins return with any local metadata (from metadata.yml/yaml/json) allowing for a live-editable dynamic Datasette. __A major design consideration is this: should Datasette perform the metadata merge? Or should Datasette allow plugins to perform any modifications themselves?__

  As a security precation, local meta is *not* overwritable by plugin hooks. The workflow for transitioning to live-meta would be to load the plugin with the full metadata.yaml and save. Then remove the parts of the metadata that you want to be able to change from the file.

  I have a WIP dynamic configuration plugin here, for reference: https://github.com/next-LI/datasette-live-config/

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1368/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTM5MDAzNzQ=
title: Don't show foreign key IDs twice if no label
state: open
created_at: 2021-06-07T19:47:02Z
updated_at: 2021-06-07T19:47:24Z
author_association: OWNER
type: issue
body:
  ![B5B54D94-A768-4544-A88D-CDCAB417CD3C](https://user-images.githubusercontent.com/9599/121078979-6e9d0600-c78e-11eb-8b70-20e6d29b48b1.jpeg)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1369/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjY0MDcyMDQ2
title: Ensure db.path is a string before trying to insert into internal database
state: closed
created_at: 2021-06-08T01:16:48Z
updated_at: 2021-06-21T15:57:39Z
closed_at: 2021-06-21T15:57:39Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1370
type: pull
body:
  Fixes #1365 

  This is the simplest possible fix, with a test that will fail without it. There are a bunch of places where `db.path` is getting converted to and from a `Path` type, so this fix errs on the side of calling `str(db.path)` right before it's inserted.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1370/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTU0MjE0OTk=
title: row.update() or row.pk
state: open
created_at: 2021-06-08T19:56:00Z
updated_at: 2021-06-22T17:27:27Z
author_association: NONE
type: issue
body:
  Hi,

  fantastic framework for working with Sqlite3 databases!!!

  I tried to update spezific rows in a table and used

  for row in db[tablename]:
      newValue = row["counter"] * row["prize"]    
      row.update({"Fieldname": newValue})
      print(row)


  This updates the value in the printet row, but not in the database. So I switched to

  db[tablename].update(id, {"Filedname": newValue})

  This works fine. But row.update would be nicer, because no need for the id (its that row), no need for the tablename and the db (all defined in the for row ... loop).

  Thx

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/267/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTU0NTUyMjg=
title: Menu plugin hooks should include the request
state: closed
created_at: 2021-06-08T20:23:35Z
updated_at: 2021-06-10T04:46:01Z
closed_at: 2021-06-10T04:46:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://docs.datasette.io/en/stable/plugin_hooks.html#menu-links-datasette-actor

  - `menu_links(datasette, actor)`
  - `table_actions(datasette, actor, database, table)`
  - `database_actions(datasette, actor, database)`

  All three of these should optionally also accept the `request` object. This would allow them to take into account additional cookies, `Authorization` headers or the current request URL (including the domain/subdomain) - or even access `request.scope` for extra context that might have been passed down from ASGI middleware.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1371/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTU0ODgyNDQ=
title: Add section to "writing plugins" about security, e.g. avoiding XSS
state: open
created_at: 2021-06-08T20:49:33Z
updated_at: 2021-06-08T20:49:46Z
author_association: OWNER
type: issue
body:
  https://docs.datasette.io/en/stable/writing_plugins.html should have tips on writing secure plugins.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1372/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjY1ODkyMzEz
title: Update trustme requirement from <0.8,>=0.7 to >=0.7,<0.9
state: closed
created_at: 2021-06-09T13:09:44Z
updated_at: 2021-06-13T15:38:47Z
closed_at: 2021-06-13T15:38:47Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1373
type: pull
body:
  Updates the requirements on [trustme](https://github.com/python-trio/trustme) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/python-trio/trustme/commit/f9e13e03963260b32cab03f4e4c25cc515ddfad3"><code>f9e13e0</code></a> Release 0.8.0</li>
  <li><a href="https://github.com/python-trio/trustme/commit/4ae4435f4477dc8433f8f6d51cdb891652a5a228"><code>4ae4435</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/304">#304</a> from python-trio/dependabot/add-v2-config-file</li>
  <li><a href="https://github.com/python-trio/trustme/commit/87679026732cc441ac87c4415cf1a2bd6547116c"><code>8767902</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/327">#327</a> from graingert/test-on-py310</li>
  <li><a href="https://github.com/python-trio/trustme/commit/6abfddd5f3139bde9ee4ba0f848fb7c84c9ac54c"><code>6abfddd</code></a> Merge branch 'master' of github.com:python-trio/trustme into test-on-py310</li>
  <li><a href="https://github.com/python-trio/trustme/commit/51d3bdfb0d874ba52d59a84e10f2c12741de1046"><code>51d3bdf</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/python-trio/trustme/issues/328">#328</a> from tiran/correct_ku_eku</li>
  <li><a href="https://github.com/python-trio/trustme/commit/034fb3aad7890ac6e4cb24d821413bd790a84b35"><code>034fb3a</code></a> retry codecov more</li>
  <li><a href="https://github.com/python-trio/trustme/commit/53e121d4f2746333cedb412cabd510549e677ba6"><code>53e121d</code></a> try codecov harder</li>
  <li><a href="https://github.com/python-trio/trustme/commit/c1e7923784e3282aa63c28a1afe2c7960d80390f"><code>c1e7923</code></a> require codecov in ci</li>
  <li><a href="https://github.com/python-trio/trustme/commit/e3ac2d61a5ab902cdeac1c95dfc0a49c81c5d290"><code>e3ac2d6</code></a> Update tests/test_trustme.py</li>
  <li><a href="https://github.com/python-trio/trustme/commit/496dca6f2146268058b300a7fc24d4bdb90c5c33"><code>496dca6</code></a> close the wrapped sockets to prevent Unraisable ResourceWarnings</li>
  <li>Additional commits viewable in <a href="https://github.com/python-trio/trustme/compare/v0.7.0...v0.8.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1373/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjY4MTI5NDQx
title: Bump black from 21.5b2 to 21.6b0
state: closed
created_at: 2021-06-11T13:07:39Z
updated_at: 2021-06-13T15:33:23Z
closed_at: 2021-06-13T15:33:22Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1374
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 21.5b2 to 21.6b0.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.6b0</h2>
  <h2><em>Black</em></h2>
  <ul>
  <li>Fix failure caused by <code>fmt: skip</code> and indentation (<a href="https://github-redirect.dependabot.com/psf/black/issues/2281">#2281</a>)</li>
  <li>Account for += assignment when deciding whether to split string (<a href="https://github-redirect.dependabot.com/psf/black/issues/2312">#2312</a>)</li>
  <li>Correct max string length calculation when there are string operators (<a href="https://github-redirect.dependabot.com/psf/black/issues/2292">#2292</a>)</li>
  <li>Fixed option usage when using the <code>--code</code> flag (<a href="https://github-redirect.dependabot.com/psf/black/issues/2259">#2259</a>)</li>
  <li>Do not call <code>uvloop.install()</code> when <em>Black</em> is used as a library (<a href="https://github-redirect.dependabot.com/psf/black/issues/2303">#2303</a>)</li>
  <li>Added <code>--required-version</code> option to require a specific version to be running (<a href="https://github-redirect.dependabot.com/psf/black/issues/2300">#2300</a>)</li>
  <li>Fix incorrect custom breakpoint indices when string group contains fake f-strings
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/2311">#2311</a>)</li>
  <li>Fix regression where <code>R</code> prefixes would be lowercased for docstrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2285">#2285</a>)</li>
  <li>Fix handling of named escapes (<code>\N{...}</code>) when <code>--experimental-string-processing</code> is
  used (<a href="https://github-redirect.dependabot.com/psf/black/issues/2319">#2319</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/main/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h2>21.6b0</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>Fix failure caused by <code>fmt: skip</code> and indentation (<a href="https://github-redirect.dependabot.com/psf/black/issues/2281">#2281</a>)</li>
  <li>Account for += assignment when deciding whether to split string (<a href="https://github-redirect.dependabot.com/psf/black/issues/2312">#2312</a>)</li>
  <li>Correct max string length calculation when there are string operators (<a href="https://github-redirect.dependabot.com/psf/black/issues/2292">#2292</a>)</li>
  <li>Fixed option usage when using the <code>--code</code> flag (<a href="https://github-redirect.dependabot.com/psf/black/issues/2259">#2259</a>)</li>
  <li>Do not call <code>uvloop.install()</code> when <em>Black</em> is used as a library (<a href="https://github-redirect.dependabot.com/psf/black/issues/2303">#2303</a>)</li>
  <li>Added <code>--required-version</code> option to require a specific version to be running (<a href="https://github-redirect.dependabot.com/psf/black/issues/2300">#2300</a>)</li>
  <li>Fix incorrect custom breakpoint indices when string group contains fake f-strings
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/2311">#2311</a>)</li>
  <li>Fix regression where <code>R</code> prefixes would be lowercased for docstrings (<a href="https://github-redirect.dependabot.com/psf/black/issues/2285">#2285</a>)</li>
  <li>Fix handling of named escapes (<code>\N{...}</code>) when <code>--experimental-string-processing</code> is
  used (<a href="https://github-redirect.dependabot.com/psf/black/issues/2319">#2319</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.5b2&new-version=21.6b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1374/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTkxODE1NTk=
title: db.schema property and sqlite-utils schema command
state: closed
created_at: 2021-06-11T20:25:47Z
updated_at: 2021-06-11T20:51:56Z
closed_at: 2021-06-11T20:51:56Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `table.schema` returns the schema for a table. `db.schema` should return the schema for the whole databes.

  Can do this using `select sql from sqlite_master where sql is not null`:

  https://latest.datasette.io/fixtures?sql=select+sql+from+sqlite_master+where+sql+is+not+null

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/268/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTkyNTA2MjE=
title: bool type not supported
state: closed
created_at: 2021-06-11T22:00:36Z
updated_at: 2021-06-15T01:34:10Z
closed_at: 2021-06-15T01:34:10Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi! Thank you for sharing this very nice tool :)
  It would be nice to have support for more types, like `bool`: it is not possible to convert to boolean at the moment. My suggestion would be to handle it as `bool(int(value))`, like csvkit does.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/269/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTkzMTQ4MDY=
title: Cannot set type JSON
state: closed
created_at: 2021-06-11T23:53:22Z
updated_at: 2021-06-16T17:34:49Z
closed_at: 2021-06-16T15:47:06Z
author_association: NONE
type: issue
state_reason: completed
body:
  It would be great if the column type could be set to JSON. That would not be different from handling a regular string. It would be something like `repr(value)` and it would work with both JSON and CSV inputs, no matter if `value` is a real list or just a string representing a list.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/270/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTk1MDg0OTg=
title: JSON export dumps JSON fields as TEXT
state: closed
created_at: 2021-06-12T09:45:08Z
updated_at: 2021-06-14T09:41:59Z
closed_at: 2021-06-13T15:37:58Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi!
  When a user tries to export data as JSON, I would expect to see the value of JSON columns represented as JSON instead of being rendered as a string. What do you think?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1375/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTk3MDI0NTE=
title: table.upsert_all() fails if input has a single column that should be a primary key
state: closed
created_at: 2021-06-13T02:50:27Z
updated_at: 2021-06-13T02:57:29Z
closed_at: 2021-06-13T02:57:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This works:
  ```pycon
  >>> db['foo'].insert_all([{"name": "hello"}], pk="name")
  <Table foo (name)>
  ```
  But this fails:
  ```
  >>> db['foo3'].upsert_all([{"name": "hello"}], pk="name")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1837, in upsert_all
      return self.insert_all(
    File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1778, in insert_all
      self.insert_chunk(
    File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 1588, in insert_chunk
      result = self.db.execute(query, params)
    File "/Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py", line 213, in execute
      return self.conn.execute(sql, parameters)
  sqlite3.OperationalError: near "WHERE": syntax error
  ```
  With the debugger:
  ```
  >>> import pdb; pdb.pm()
  > /Users/simon/.local/share/virtualenvs/datasette.io-TK86ygSO/lib/python3.9/site-packages/sqlite_utils/db.py(213)execute()
  -> return self.conn.execute(sql, parameters)
  (Pdb) print(sql, parameters)
  UPDATE [foo3] SET  WHERE [name] = ? ['hello']
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/271/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTk3MzMyMTM=
title: Searching for whitespace throws an error
state: closed
created_at: 2021-06-13T06:57:57Z
updated_at: 2021-06-13T14:36:39Z
closed_at: 2021-06-13T14:36:39Z
author_association: MEMBER
body:
  https://datasette.io/-/beta?q=+ returns a 500

  > fts5: syntax error near ""

type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/33/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MTk4MjI4MTc=
title: Official Datasette Docker image should use SQLite >= 3.31.0 (for generated columns)
state: open
created_at: 2021-06-13T15:25:51Z
updated_at: 2021-06-13T15:39:37Z
author_association: NONE
type: issue
body:
  Trying to run datasette via the Docker container doesn't seem to work:

  ```
  $ docker run -p 8001:8001 -v `pwd`:/mnt     datasetteproject/datasette     datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  Traceback (most recent call last):
    File "/usr/local/bin/datasette", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/datasette/cli.py", line 544, in serve
      asyncio.get_event_loop().run_until_complete(check_databases(ds))
    File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
      return future.result()
    File "/usr/local/lib/python3.9/site-packages/datasette/cli.py", line 584, in check_databases
      await database.execute_fn(check_connection)
    File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 155, in execute_fn
      return await asyncio.get_event_loop().run_in_executor(
    File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
      result = self.fn(*self.args, **self.kwargs)
    File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 153, in in_thread
      return fn(conn)
    File "/usr/local/lib/python3.9/site-packages/datasette/utils/__init__.py", line 892, in check_connection
      for r in conn.execute(
  sqlite3.DatabaseError: malformed database schema (generated_columns) - near "AS": syntax error
  ```

  I have confirmed that the downloaded `fixtures.db` database is fine:

  ```
  [skia-public] jcgregorio@jcgregorio840 ~/Downloads 
  $ sqlite3 fixtures.db 
  SQLite version 3.34.1 2021-01-20 14:10:07
  Enter ".help" for usage hints.
  sqlite> pragma integrity_check;
  ok
  sqlite> 

  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1376/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjA2MzYyMTY=
title: feature: support "events"
state: open
created_at: 2021-06-14T17:42:49Z
updated_at: 2021-06-15T00:48:37Z
author_association: NONE
type: issue
body:
  the GitHub API provides the ability to fetch all events for a given user, organization, or repository: https://docs.github.com/en/rest/reference/activity#list-events-for-the-authenticated-user

  this would allow users to export all of the issue comments, new issues, etc. that they created. something which is currently missing from the GitHub takeout exports.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/64/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjA4ODQwODU=
title: Mechanism for plugins to exclude certain paths from CSRF checks
state: closed
created_at: 2021-06-15T00:48:20Z
updated_at: 2021-06-23T22:51:33Z
closed_at: 2021-06-23T22:51:33Z
author_association: OWNER
body: I need this for a plugin I'm building that offers a POST API.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1377/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjE4Nzg3MzM=
title: Idea: import CSV to memory, run SQL, export in a single command
state: closed
created_at: 2021-06-15T23:02:48Z
updated_at: 2021-06-19T23:36:48Z
closed_at: 2021-06-18T15:05:03Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I quite often load a CSV file into a SQLite DB, then do stuff with it (like export results back out again as a new CSV) without any intention of keeping the CSV file around afterwards.

  What if `sqlite-utils` could do this for me? Something like this:

      sqlite-utils --csv blah.csv --csv baz.csv "select * from blah join baz ..."

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/272/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjcxMDE0NzUx
title: sqlite-utils memory command for directly querying CSV/JSON data
state: closed
created_at: 2021-06-16T05:04:58Z
updated_at: 2021-06-18T15:01:17Z
closed_at: 2021-06-18T15:00:52Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/273
type: pull
body:
  Refs #272. Initial implementation only does CSV data, still needs:

  - [x] Implement `--save`
  - [x] Add `--dump` to the documentation
  - [x] Add `--attach` example to the documentation
  - [x] Replace `:memory:` in documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/273/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjI4MzIxMTM=
title: sqlite-utils dump my.db command
state: closed
created_at: 2021-06-16T16:30:14Z
updated_at: 2021-06-16T23:51:54Z
closed_at: 2021-06-16T23:51:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Inspired by the `--dump` mechanism I added to `sqlite-utils memory` here: https://github.com/simonw/sqlite-utils/issues/272#issuecomment-862018937

  > Can use `.iterdump()` to implement this: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.iterdump
  >
  > Maybe instead (or as-well-as) offer `--dump` which dumps out the SQL from that.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/274/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjI5NTU2OTc=
title: Enable code coverage
state: closed
created_at: 2021-06-16T18:33:49Z
updated_at: 2021-06-17T00:12:12Z
closed_at: 2021-06-17T00:12:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://app.codecov.io/gh/simonw/sqlite-utils

  Same mechanism as Datasette. Need to copy across the token from that page and add an equivalent of this workflow: https://github.com/simonw/datasette/blob/main/.github/workflows/test-coverage.yml

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/275/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjcyMDUzODEx
title: basic support for events
state: open
created_at: 2021-06-17T00:51:30Z
updated_at: 2022-10-03T22:35:03Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/github-to-sqlite/pulls/65
type: pull
body:
  a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64

  testing instructions:

  ```
  $ github-to-sqlite events events.db user/khimaros
  ```

  if the specified user is the authenticated user, it will also include private events.

  caveat: pagination appears to be broken (i don't see `next` in the response JSON from GitHub)

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/65/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjM2MDI2OTM=
title: support small help flag -h
state: closed
created_at: 2021-06-17T07:59:31Z
updated_at: 2021-06-18T14:56:59Z
closed_at: 2021-06-18T14:56:59Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/276/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjcyMzU5NjA5
title: add -h support closes #276
state: closed
created_at: 2021-06-17T08:08:26Z
updated_at: 2021-06-18T14:56:59Z
closed_at: 2021-06-18T14:56:59Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/277
type: pull
body:
  This appears to be the [canonical solution](https://click.palletsprojects.com/en/7.x/documentation/#help-parameter-customization).

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/277/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjM2OTc4ODg=
title: Support db as first parameter before subcommand, or as environment variable
state: closed
created_at: 2021-06-17T09:26:29Z
updated_at: 2021-06-20T22:39:57Z
closed_at: 2021-06-18T15:43:19Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/278/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjcyNjIwMTgw
title: Update pytest-xdist requirement from <2.3,>=2.2.1 to >=2.2.1,<2.4
state: closed
created_at: 2021-06-17T13:11:56Z
updated_at: 2021-06-20T00:17:07Z
closed_at: 2021-06-20T00:17:06Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1378
type: pull
body:
  Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version.
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst">pytest-xdist's changelog</a>.</em></p>
  <blockquote>
  <h1>pytest-xdist 2.3.0 (2021-06-16)</h1>
  <h2>Deprecations and Removals</h2>
  <ul>
  <li><code>[#654](https://github.com/pytest-dev/pytest-xdist/issues/654) &lt;https://github.com/pytest-dev/pytest-xdist/issues/654&gt;</code>_: Python 3.5 is no longer supported.</li>
  </ul>
  <h2>Features</h2>
  <ul>
  <li>
  <p><code>[#646](https://github.com/pytest-dev/pytest-xdist/issues/646) &lt;https://github.com/pytest-dev/pytest-xdist/issues/646&gt;</code>_: Add <code>--numprocesses=logical</code> flag, which automatically uses the number of logical CPUs available, instead of physical CPUs with <code>auto</code>.</p>
  <p>This is very useful for test suites which are not CPU-bound.</p>
  </li>
  <li>
  <p><code>[#650](https://github.com/pytest-dev/pytest-xdist/issues/650) &lt;https://github.com/pytest-dev/pytest-xdist/issues/650&gt;</code>_: Added new <code>pytest_handlecrashitem</code> hook to allow handling and rescheduling crashed items.</p>
  </li>
  </ul>
  <h2>Bug Fixes</h2>
  <ul>
  <li>
  <p><code>[#421](https://github.com/pytest-dev/pytest-xdist/issues/421) &lt;https://github.com/pytest-dev/pytest-xdist/issues/421&gt;</code>_: Copy the parent process sys.path into local workers, to work around execnet's python -c adding the current directory to sys.path.</p>
  </li>
  <li>
  <p><code>[#638](https://github.com/pytest-dev/pytest-xdist/issues/638) &lt;https://github.com/pytest-dev/pytest-xdist/issues/638&gt;</code>_: Fix issue caused by changing the branch name of the pytest repository.</p>
  </li>
  </ul>
  <h2>Trivial Changes</h2>
  <ul>
  <li>
  <p><code>[#592](https://github.com/pytest-dev/pytest-xdist/issues/592) &lt;https://github.com/pytest-dev/pytest-xdist/issues/592&gt;</code>_: Replace master with controller where ever possible.</p>
  </li>
  <li>
  <p><code>[#643](https://github.com/pytest-dev/pytest-xdist/issues/643) &lt;https://github.com/pytest-dev/pytest-xdist/issues/643&gt;</code>_: Use 'main' to refer to pytest default branch in tox env names.</p>
  </li>
  </ul>
  <h1>pytest-xdist 2.2.1 (2021-02-09)</h1>
  <h2>Bug Fixes</h2>
  <ul>
  <li><code>[#623](https://github.com/pytest-dev/pytest-xdist/issues/623) &lt;https://github.com/pytest-dev/pytest-xdist/issues/623&gt;</code>_: Gracefully handle the pending deprecation of Node.fspath by using config.rootpath for topdir.</li>
  </ul>
  <h1>pytest-xdist 2.2.0 (2020-12-14)</h1>
  <h2>Features</h2>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/fe57b39563aa80be7605302baa090069a172037a"><code>fe57b39</code></a> fixup: add release title underline for 2.3.0</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/26e7d953f41c38c6aca209c63e2f82140def9255"><code>26e7d95</code></a> prepare release 2.3.0</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/b02a6db771ef2ca2e6df2f6c94143cbeeefc7f1b"><code>b02a6db</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/667">#667</a> from graingert/fix-sys-path</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/b0722675e033b346382475b1ab2e0e32cdb07ae6"><code>b072267</code></a> add newsfile</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/881cc4827171e382cf2254687bc63df0ecb8c189"><code>881cc48</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/672">#672</a> from pytest-dev/pre-commit-ci-update-config</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/958679e9a717fe4cea93aac989f072deb4962f08"><code>958679e</code></a> [pre-commit.ci] pre-commit autoupdate</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/7f07d50808eec48febcae8308a8194cc247eb40f"><code>7f07d50</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/646">#646</a> from kroeschl/numprocesses-logical</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/fb518def5523b4ae0ae81f1dcea31aee8f6b3dea"><code>fb518de</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest-xdist/issues/669">#669</a> from pytest-dev/pre-commit-ci-update-config</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/0b14d9289482cdbfe85d73c47b58ecc89ecf1c06"><code>0b14d92</code></a> [pre-commit.ci] pre-commit autoupdate</li>
  <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/02f971d45ff40313d4b31c58396bbd546a120a2b"><code>02f971d</code></a> swap docstring</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-xdist/compare/v2.2.1...v2.3.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1378/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjQyMDM3ODM=
title: Idea: ?_end=1 option for streaming CSV responses
state: open
created_at: 2021-06-17T18:11:21Z
updated_at: 2021-06-17T18:11:30Z
author_association: OWNER
type: issue
body:
  As discussed in this thread: https://twitter.com/simonw/status/1405554676993433605 - one of the disadvantages of Datasette's streaming CSV feature is that it's hard to tell if you got the whole file or if the connection ended early - or if an error occurred.

  Idea: offer an optional `?_end=1` parameter which, if enabled, adds a single row to the end of the CSV file that looks like this:

  `END,,,,,,,,,`

  For however many columns the CSV file usually has.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1379/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjQ3NDg5NTU=
title: Serve all db files in a folder
state: open
created_at: 2021-06-18T10:03:32Z
updated_at: 2021-11-13T08:09:11Z
author_association: NONE
type: issue
body:
  I tried to get the `serve` command to serve all the .db files in the `/mnt` folder but is seems that the server does not refresh the list of files.

  In more detail:

  * Starting datasette as a docker container with: 
  ```
  docker run -p 8001:8001 -v `pwd`:/mnt \
      datasetteproject/datasette \
      datasette -p 8001 -h 0.0.0.0 /mnt
  ```

  * Datasette correctly serves all the *.db files found in the /mnt folder 
  * When the server is running, if I copy a new file in the $PWD folder, Datasette does not seem to see the new files, forcing me to restart Docker.

  Is there an option/setting that I overlooked, or is this something missing?

  BTW, the `--reload` setting, although at first glance is what you think you need, does not seem to do anything in regards of seeing all *.db files.

  Thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1380/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjQ5OTA2Nzc=
title: sqlite-utils memory should handle TSV and JSON in addition to CSV
state: closed
created_at: 2021-06-18T15:02:54Z
updated_at: 2021-06-19T03:11:59Z
closed_at: 2021-06-19T03:11:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  - Use sniff to detect CSV or TSV (if `:tsv` or `:csv` was not specified) and delimiters

  Follow-on from #272

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/279/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjQ5OTExOTQ=
title: Add --encoding option to sqlite-utils memory
state: closed
created_at: 2021-06-18T15:03:32Z
updated_at: 2021-06-18T15:29:46Z
closed_at: 2021-06-18T15:29:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Follow-on from #272 - this will work like `--encoding` on `sqlite-utils insert` and will affect all CSV files processed by `sqlite-utils memory`.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/280/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjQ5OTIzMTg=
title: Mechanism for explicitly stating CSV or JSON or TSV for sqlite-utils memory
state: closed
created_at: 2021-06-18T15:04:53Z
updated_at: 2021-06-19T03:11:59Z
closed_at: 2021-06-19T03:11:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  - Implement `filename.json:json` and `-:nl` and suchlike options for specifying the format rather than guessing it - see https://github.com/simonw/sqlite-utils/issues/272#issuecomment-861985944

  Follows #272

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/281/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjUzMDUxODY=
title: Automatic type detection for CSV data
state: closed
created_at: 2021-06-19T03:33:21Z
updated_at: 2021-06-19T04:42:03Z
closed_at: 2021-06-19T04:38:00Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I've touched on this before in #179 - but now that I've added `sqlite-utils memory` this is much more important - because unlike with `sqlite-utils insert` the in-memory command doesn't give you the opportunity to fix any types you imported from CSV, so queries like `select * from stdin where age > 3` are never going to work correctly against these temporary in-memory tables.

  Teaching `sqlite-utils insert` to detect types for columns in a CSV file would be a backwards-compatibility breaking change. Teaching `sqlite-utils memory` that trick would not be, since it hasn't been included in a release yet.

  It's a little inconsistent, but I'm going to have `sqlite-utils memory` default to detecting types while `sqlite-utils insert` does not. In each case this can be controlled by a new command-line option:

      cat file.csv | sqlite-utils memory - --no-detect-types

  To opt-in for `sqlite-utils insert`:

      cat file.csv | sqlite-utils insert blah.db blah - --detect-types

  I'll have short options for these too: `-n` for `--no-detect-types` and `-d` for `--detect-types`.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/282/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjUzMTkyMTQ=
title: memory: Shouldn't detect types for JSON
state: closed
created_at: 2021-06-19T05:17:35Z
updated_at: 2021-06-19T14:52:48Z
closed_at: 2021-06-19T14:52:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/sqlite-utils/blob/ec5174ed40fa283cb06f25ee0c0136297ec313ae/sqlite_utils/cli.py#L1244-L1251

  This runs against JSON as well as CSV/TSV - which isn't necessary and In fact throws errors if there is any nested data.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/283/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjUzMjAxNjc=
title: .transform(types=) turns rowid into a concrete column
state: closed
created_at: 2021-06-19T05:25:27Z
updated_at: 2021-06-19T15:28:30Z
closed_at: 2021-06-19T15:28:30Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Noticed this in the tests for `sqlite-utils memory` in #282 - is it possible to fix this?

  https://github.com/simonw/sqlite-utils/commit/ec5174ed40fa283cb06f25ee0c0136297ec313ae

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/284/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjczODcyOTc0
title: Add instagram-to-sqlite
state: open
created_at: 2021-06-19T12:26:16Z
updated_at: 2021-07-28T07:58:59Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/dogsheep.github.io/pulls/7
type: pull
body:
  The tool covers only chat imports at the time of opening this PR but I'm planning to import everything else that I feel inquisitive about

  ref: https://github.com/gavindsouza/instagram-to-sqlite

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjU0MDY5NjQ=
title: Datasette with Glitch - is it possible to use CSV with ISO-8859-1 encoding?
state: closed
created_at: 2021-06-19T14:37:20Z
updated_at: 2021-06-20T00:21:02Z
closed_at: 2021-06-20T00:20:06Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi
  Please, I used Remix on Glitch to create a project on Glitch and uploaded a CSV
  But it's a CSV with ISO-8859-1 encoding (https://en.wikipedia.org/wiki/ISO/IEC_8859-1)
  Is it possible for me to change the encoding to correctly visualize the data?
  Example: https://emphasized-carpal-pillow.glitch.me/data/Emendas
  Best

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1382/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjU0MTAzMDU=
title: Introspection property for telling if a table is a rowid table
state: closed
created_at: 2021-06-19T14:56:16Z
updated_at: 2021-06-19T15:12:33Z
closed_at: 2021-06-19T15:12:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/284#issuecomment-864416785_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/285/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjU0ODc5NDY=
title: Add installation instructions
state: closed
created_at: 2021-06-19T23:55:36Z
updated_at: 2021-06-20T18:47:13Z
closed_at: 2021-06-20T18:47:13Z
author_association: OWNER
body: `pip install sqlite-utils`, `pipx install sqlite-utils` and `brew install sqlite-utils`
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/286/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjU0OTE4NTc=
title: Improve test coverage for `inspect.py`
state: open
created_at: 2021-06-20T00:22:43Z
updated_at: 2021-06-20T00:22:49Z
author_association: OWNER
type: issue
body:
  https://codecov.io/gh/simonw/datasette/src/main/datasette/inspect.py shows only 36% coverage for that module at the moment.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1383/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjU1NDQwNzA=
title: Update rowid examples in the docs
state: closed
created_at: 2021-06-20T08:03:00Z
updated_at: 2021-06-20T18:26:21Z
closed_at: 2021-06-20T18:26:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Changed in #284 - a couple of examples need updating on https://github.com/simonw/sqlite-utils/blob/3.10/docs/cli.rst.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/287/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjU1NDU0Njg=
title: sqlite-utils memory blah.json --schema
state: closed
created_at: 2021-06-20T08:10:40Z
updated_at: 2021-06-20T18:26:21Z
closed_at: 2021-06-20T18:26:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Like `--dump` but only outputs the schema - useful for understanding what you are about to run queries against.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/288/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjU2NzcxOTE=
title: Mypy fixes for rows_from_file()
state: closed
created_at: 2021-06-20T20:34:59Z
updated_at: 2021-06-22T18:44:36Z
closed_at: 2021-06-22T18:13:26Z
author_association: NONE
type: issue
state_reason: completed
body:
  Following https://github.com/simonw/sqlite-utils/issues/279#issuecomment-864328927

  You had two mypy errors.

  The first:

  > sqlite_utils/utils.py:157: error: Argument 1 to "BufferedReader" has incompatible type "BinaryIO"; expected "RawIOBase"

  Looking at the `BufferedReader` docs, it seems to expect a `RawIOBase`, and this [has been copied into typeshed](https://github.com/python/typeshed/blob/9ec2f8712480c57353cea097a65d75a2c4ec1846/stdlib/io.pyi#L100). There may be scope to change how `BufferedReader` is documented and typed upstream, but for now it wouldn't be too bad to use a `typing.cast()`:

  ```
  # Detect the format, then call this recursively
  buffered = io.BufferedReader(
      cast(io.RawIOBase, fp),  # Undocumented BufferedReader support for BinaryIO
      buffer_size=4096,
  )
  ```

  The second error seems to be flagging a legitimate bug in your code:

  > sqlite_utils/utils.py:163: error: Argument 1 to "decode" of "bytes" has incompatible type "Optional[str]"; expected "str"

  From your type hints, `encoding` may be `None`. In the CSV format block, you use `encoding or "utf-8-sig"` to set a default, maybe that's desirable in this case too?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/289/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjY3NzczMTA=
title: `db.query()` method (renamed `db.execute_returning_dicts()`)
state: closed
created_at: 2021-06-22T03:03:54Z
updated_at: 2021-06-24T23:17:38Z
closed_at: 2021-06-24T22:54:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Most of this library deals with lists of Python dictionaries - `.insert_all()`, `.rows`, `.rows_where()`, `.search()`.

  The `db.execute()` method is the only thing that returns a `sqlite3` cursor.

  There is a clumsily named `db.execute_returning_dicts(sql)` method but it's not currently mentioned in the documentation.

  It needs a better name, and needs to be properly documented.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/290/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MjczODU1NDA=
title: any guidance / experience on imessage-to-sqlite ?
state: open
created_at: 2021-06-22T15:46:16Z
updated_at: 2021-06-22T15:46:16Z
author_association: NONE
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5Mjc3NjYyOTY=
title: Adopt flake8
state: closed
created_at: 2021-06-23T01:19:37Z
updated_at: 2021-06-24T17:50:27Z
closed_at: 2021-06-24T17:50:27Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/291/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5Mjc3ODk4MTE=
title: Add contributing documentation
state: closed
created_at: 2021-06-23T02:13:05Z
updated_at: 2021-06-25T17:53:51Z
closed_at: 2021-06-25T17:53:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Like https://docs.datasette.io/en/latest/contributing.html (but simpler) - should cover how to run `black` and `flake8` and `mypy` and how to run the tests.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/292/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Njc3NTU0OTI5
title: Test against Python 3.10-dev
state: closed
created_at: 2021-06-25T01:40:39Z
updated_at: 2021-10-13T21:49:33Z
closed_at: 2021-10-13T21:49:33Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/293
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/293/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MzA4MDcxMzU=
title: Plugin hook for dynamic metadata
state: open
created_at: 2021-06-26T22:36:03Z
updated_at: 2022-03-14T00:36:42Z
author_association: OWNER
type: issue
body:
  @brandonrobertz contributed an implementation of this in PR #1368, which I just merged. Opening this ticket to track further work on this before it goes out in a Datasette release (likely preceded by an alpha).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1384/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Njc4NDU5NTU0
title: Fix + improve get_metadata plugin hook docs
state: closed
created_at: 2021-06-27T05:43:20Z
updated_at: 2021-09-13T18:53:11Z
closed_at: 2021-09-13T18:53:11Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1385
type: pull
body:
  This fixes documentation inaccuracies and adds a disclaimer about the signature of the `get_metadata` hook.

  Addresses the following comments:
  - https://github.com/simonw/datasette/issues/1384#issuecomment-869069926
  - https://github.com/simonw/datasette/issues/1384#issuecomment-869075368

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1385/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MzA5NDY4MTc=
title: KeyError: 'accuracy' when processing Location History
state: open
created_at: 2021-06-27T14:39:43Z
updated_at: 2021-06-27T14:39:43Z
author_association: NONE
type: issue
body:
  I'm new to both the dogsheep tools and datasette but have been experimenting a bit the last few days and these are really cool tools!

  I encountered a problem running my Google location history through this tool running the latest release in a docker container:

  ```
  Traceback (most recent call last):
    File "/usr/local/bin/google-takeout-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/cli.py", line 49, in my_activity
      utils.save_location_history(db, zf)
    File "/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py", line 27, in save_location_history
      db["location_history"].upsert_all(
    File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1105, in upsert_all
      return self.insert_all(
    File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 990, in insert_all
      chunk = list(chunk)
    File "/usr/local/lib/python3.9/site-packages/google_takeout_to_sqlite/utils.py", line 33, in <genexpr>
      "accuracy": row["accuracy"],
  KeyError: 'accuracy'
  ```

  It looks like the tool assumes the `accuracy` key will be in every location history entry. 

  My first attempt at a local patch to get myself going was to convert accessing the `accuracy` key to a `.get` instead to hopefully make the row nullable but I wasn't quite sure what `sqlite_utils` would do there. That did work in that the import happened and so I was going to propose a patch that made that change but in updating the existing test to include an entry with a missing accuracy entry, I noticed the expected type of the field appeared to be changing to a string in the test (and from a quick scan through the sqlite_utils code, probably TEXT in the database). Given this change in column type, it seemed that opening an issue first before proposing a fix seemed warranted. It seems the schema would need to be explicitly specified if you wanted a nullable integer column.

  Now that I've done a successful import run using my initial fix of calling `.get` on the row dict, I can see with datasette that I only have 7 data points (out of ~250k) that have a null accuracy column. They are all from 2011-2012 in an import that includes points spanning ~2010-2016 so perhaps another approach might be to filter those entries out during import if it really is that infrequent?

  I'm happy to provide a PR for a fix but figured I'd ask about which direction is preferred first.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Njc5MDM1ODQ3
title: Update asgiref requirement from <3.4.0,>=3.2.10 to >=3.2.10,<3.5.0
state: closed
created_at: 2021-06-28T13:13:07Z
updated_at: 2021-07-11T01:36:19Z
closed_at: 2021-07-11T01:36:18Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1386
type: pull
body:
  Updates the requirements on [asgiref](https://github.com/django/asgiref) to permit the latest version.
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/django/asgiref/blob/main/CHANGELOG.txt">asgiref's changelog</a>.</em></p>
  <blockquote>
  <h2>3.4.0 (2021-06-27)</h2>
  <ul>
  <li>
  <p>Calling sync_to_async directly from inside itself (which causes a deadlock
  when in the default, thread-sensitive mode) now has deadlock detection.</p>
  </li>
  <li>
  <p>asyncio usage has been updated to use the new versions of get_event_loop,
  ensure_future, wait and gather, avoiding deprecation warnings in Python 3.10.
  Python 3.6 installs continue to use the old versions; this is only for 3.7+</p>
  </li>
  <li>
  <p>sync_to_async and async_to_sync now have improved type hints that pass
  through the underlying function type correctly.</p>
  </li>
  <li>
  <p>All Websocket* types are now spelled WebSocket, to match our specs and the
  official spelling. The old names will work until release 3.5.0, but will
  raise deprecation warnings.</p>
  </li>
  <li>
  <p>The typing for WebSocketScope and HTTPScope's <code>extensions</code> key has been
  fixed.</p>
  </li>
  </ul>
  <h2>3.3.4 (2021-04-06)</h2>
  <ul>
  <li>The async_to_sync type error is now a warning due the high false negative
  rate when trying to detect coroutine-returning callables in Python.</li>
  </ul>
  <h2>3.3.3 (2021-04-06)</h2>
  <ul>
  <li>The sync conversion functions now correctly detect functools.partial and other
  wrappers around async functions on earlier Python releases.</li>
  </ul>
  <h2>3.3.2 (2021-04-05)</h2>
  <ul>
  <li>
  <p>SyncToAsync now takes an optional &quot;executor&quot; argument if you want to supply
  your own executor rather than using the built-in one.</p>
  </li>
  <li>
  <p>async_to_sync and sync_to_async now check their arguments are functions of
  the correct type.</p>
  </li>
  <li>
  <p>Raising CancelledError inside a SyncToAsync function no longer stops a future
  call from functioning.</p>
  </li>
  <li>
  <p>ThreadSensitive now provides context hooks/override options so it can be
  made to be sensitive in a unit smaller than threads (e.g. per request)</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  <p>... (truncated)</p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/django/asgiref/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1386/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MzE3NTI3NzM=
title: Add a `sqlite-utils memory` example to the README
state: closed
created_at: 2021-06-28T16:35:59Z
updated_at: 2021-08-18T21:40:03Z
closed_at: 2021-08-18T21:40:03Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/294/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MzQxMjM0NDg=
title: Insert with --tsv and --no-headers give error about --nl arguments
state: closed
created_at: 2021-06-30T21:01:01Z
updated_at: 2021-08-18T20:19:04Z
closed_at: 2021-08-18T20:18:57Z
author_association: NONE
type: issue
state_reason: completed
body:
  Not quite sure if this is a bug, or just an assumption I made but I thought `--tsv` and `--no-headers` would work together when inserting from a file, and currently they seem not to (sqlite-utils, version 3.12, installed on Mac OS X via brew)

  Instead it says:

  `Error: Use just one of --nl, --csv or --tsv`

  As if it has interpreted the --no-headers as --nl.

  The --help does specifically say CSV:
   `--no-headers              CSV file has no header row`
   
  And this heading in the documentation also only refers to CSV, but the text does mention TSV in passing, and I'd generally expect them to behave the same in most cases.
  https://sqlite-utils.datasette.io/en/stable/cli.html#csv-files-without-a-header-row

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/295/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MzU5MzA4MjA=
title: absolute_url() behind a proxy assembles incorrect http://127.0.0.1:8001/ URLs
state: closed
created_at: 2021-07-02T16:58:25Z
updated_at: 2021-07-02T17:58:23Z
closed_at: 2021-07-02T17:33:05Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Reported in the wild on https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber - the "next page" link links to https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib?_facet=bib_level_callnumber&_next=100

  That installation uses `"base_url": "/collection-analysis/"`

  Weirdly all of the other links on that page - to facet results, sort orders, row permalinks etc - work fine. It's JUST the `next_url` one that is broken.

  Also broken in their JSON: https://ilsweb.cincinnatilibrary.org/collection-analysis/current_collection-3d4a4b7/bib.json?_size=1 returns

  ```json
    "suggested_facets": [],
    "next": "1",
    "next_url": "https://127.0.0.1:8010/collection-analysis/current_collection-3d4a4b7/bib.json?_size=1&_next=1",
    "private": false,
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1387/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5MzkwNTE1NDk=
title: Serve using UNIX domain socket
state: closed
created_at: 2021-07-07T16:13:37Z
updated_at: 2021-07-11T01:18:38Z
closed_at: 2021-07-10T23:38:32Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Would it be possible to make datasette serve using UNIX domain socket similar to Uvicorn's ``--uds``?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1388/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDAwNzcxNjg=
title: "searchmode": "raw" in table metadata
state: closed
created_at: 2021-07-08T17:32:10Z
updated_at: 2021-07-10T18:33:13Z
closed_at: 2021-07-10T18:33:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > http://localhost:8001/index/summary?_search=language%3Aeng&_sort=title&_searchmode=raw
  >
  > But I'm not able to manage it in the metadata file. Here is mine (note that the sort column is taken into account)
  > Here it is:
  >
  > ```
  > {
  >     "databases": {
  >       "index": {
  >         "tables": {
  >             "summary": {
  >                 "sort": "title",
  >                 "searchmode": "raw"
  >             }
  >         }
  >       }
  >     }
  >   }

  _Originally posted by @Krazybug in https://github.com/simonw/datasette/issues/759#issuecomment-624860451_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1389/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDA4OTE2OTg=
title: Mention restarting systemd in documentation
state: closed
created_at: 2021-07-09T16:05:15Z
updated_at: 2021-07-09T16:32:57Z
closed_at: 2021-07-09T16:32:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://docs.datasette.io/en/stable/deploying.html#running-datasette-using-systemd

  Need to clarify that if you add a new database or change metadata you need to restart systemd.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1390/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDEzMDA5NDY=
title: Stop using generated columns in fixtures.db
state: closed
created_at: 2021-07-10T18:26:11Z
updated_at: 2021-07-10T19:26:58Z
closed_at: 2021-07-10T19:26:00Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Refs #1376 - but I also keep running into this myself, where I try to run something against `fixtures.db` and get this confusing error:

      sqlite3.DatabaseError: malformed database schema (generated_columns) - near "AS": syntax error

  I'm going to stop using generated columns in `fixtures.db` and instead dynamically generate the generated column table for the duration of the relevant test.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1391/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Njg3Mjk4MTEy
title: Update deploying.rst
state: closed
created_at: 2021-07-11T08:43:19Z
updated_at: 2021-07-13T17:42:31Z
closed_at: 2021-07-13T17:42:27Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1392
body: Use same base url for Apache as in the example
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1392/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Njg3MzA0MjQy
title: Update deploying.rst
state: closed
created_at: 2021-07-11T09:32:16Z
updated_at: 2021-07-13T18:32:49Z
closed_at: 2021-07-13T18:32:49Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1393
type: pull
body:
  Example on how to use Unix domain socket option on Apache. Not testet.

  (Usually I would have used [`ProxyPassReverse`](https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypassreverse) in combination with `ProxyPass` , i.e.

  ```apache
  ProxyPass /my-datasette/ http://127.0.0.1:8009/my-datasette/
  ProxyPassReverse /my-datasette/ http://127.0.0.1:8009/my-datasette/
  ```

  and

  ```apache
  ProxyPass /my-datasette/ unix:/tmp/datasette.sock|http://localhost/my-datasette/
  ProxyPassReverse /my-datasette/ unix:/tmp/datasette.sock|http://localhost/my-datasette/
  ```
  )

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1393/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDQzMjY1MTI=
title: `table.search(..., quote=True)` parameter and `sqlite-utils search --quote` option
state: closed
created_at: 2021-07-14T11:26:47Z
updated_at: 2021-08-18T20:13:12Z
closed_at: 2021-08-18T20:10:48Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,
  Recently got this error:
  ```
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/home/ethan/git/music-metadata-indexer/src/mmindexer/__init__.py", line 38, in <module>
      start("/home/ethan/git/music-metadata-indexer/sample", "/home/ethan/git/music-metadata-indexer/test.db")
    File "/home/ethan/git/music-metadata-indexer/src/mmindexer/__init__.py", line 23, in start
      scanner.build_database()
    File "/home/ethan/git/music-metadata-indexer/src/mmindexer/scan.py", line 79, in build_database
      _import_song(self.db, Path(dirpath).joinpath(f), self.logger) 
    File "/home/ethan/git/music-metadata-indexer/src/mmindexer/scan.py", line 23, in _import_song
      db.add_song(filepath)
    File "/home/ethan/git/music-metadata-indexer/src/mmindexer/index.py", line 166, in add_song
      for match in self.search("albums", album): 
    File "/home/ethan/git/music-metadata-indexer/env/lib/python3.9/site-packages/sqlite_utils/db.py", line 1625, in search
      cursor = self.db.execute(
    File "/home/ethan/git/music-metadata-indexer/env/lib/python3.9/site-packages/sqlite_utils/db.py", line 243, in execute
      return self.conn.execute(sql, parameters)
  sqlite3.OperationalError: fts5: syntax error near "." 
  ```
  So, the error seems to suggest there was a "." character somewhere in the SQL command that was causing the error. I did a little digging and found this in the docs: https://www.sqlite.org/fts5.html#fts5_strings. "." is one of the many prohibited characters.

  My solution was to just strip these out of the query using this line
  `query = query.translate({e: None for e in itertools.chain(range(0,26), range(27, 48), range(58,65), range(91,95), [96], range(123,128))})`

  Perhaps this could be included into the `table.search()` function?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/296/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDQ4NDY3NzY=
title: Option for importing CSV data using the SQLite .import mechanism
state: open
created_at: 2021-07-14T22:36:41Z
updated_at: 2023-09-22T20:49:52Z
author_association: OWNER
type: issue
body:
  As seen in https://til.simonwillison.net/sqlite/import-csv - `.mode csv` and then `.import school.csv schools` is hugely faster than importing via `sqlite-utils insert` and doing the work in Python - but it can only be implemented by shelling out to the `sqlite3` CLI tool, it's not functionality that is exposed to the Python `sqlite3` module.

  An option to use this would be useful - maybe something like this:

      sqlite-utils insert blah.db blah blah.csv --fast

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/297/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDQ4NzA3OTk=
title: Big performance boost on faceting: skip the inner order by
state: closed
created_at: 2021-07-14T23:32:29Z
updated_at: 2021-07-16T02:23:32Z
closed_at: 2021-07-15T00:05:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I just noticed something that could make for a huge performance improvement in faceting.

  The default query used by Datasette when faceting looks like this:
  ```sql
  select
    country_long,
    count(*)
  from (
    select * from [global-power-plants] order by rowid
  )
  where
    country_long is not null
  group by
    country_long
  order by
    count(*) desc
  ```
  Here it takes 53ms: https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D+order+by+rowid%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc

  Note that there's a `order by rowid` in there which isn't necessary - the order on that inner query doesn't matter since we're grouping and counting.

  I had assumed SQLite would optimize this away - but it turns out it doesn't! Consider this version of the query, with that pointless order by removed:
  ```
  select
    country_long,
    count(*)
  from (
    select * from [global-power-plants]
  )
  where
    country_long is not null
  group by
    country_long
  order by
    count(*) desc
  ```
  https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++country_long%2C%0D%0A++count%28*%29%0D%0Afrom+%28%0D%0A++select+*+from+%5Bglobal-power-plants%5D%0D%0A%29%0D%0Awhere%0D%0A++country_long+is+not+null%0D%0Agroup+by%0D%0A++country_long%0D%0Aorder+by%0D%0A++count%28*%29+desc runs in 7.2ms!

  I tried this optimization on a table with 2.5m rows in it - without the optimization it took 5 seconds, with the optimization it took 450ms. So this is a very significant improvement!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1394/reactions", "total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDQ5MDM4ODE=
title: "invalid reference format" publishing Docker image
state: closed
created_at: 2021-07-15T01:02:07Z
updated_at: 2021-10-19T08:10:26Z
closed_at: 2021-07-15T19:47:25Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Error ocurred at the end of the publish flow for Datasette 0.58: https://github.com/simonw/datasette/runs/3072216421
  ```
  Removing intermediate container cf32b9440907
   ---> dfd6985b2afc
  Successfully built dfd6985b2afc
  Successfully tagged ***/datasette:0.58
  invalid reference format
  Error: Process completed with exit code 1.
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1396/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjkxNzA3NDA5
title: Fix for race condition in refresh_schemas(), closes #1231
state: closed
created_at: 2021-07-16T19:44:43Z
updated_at: 2021-07-16T19:45:00Z
closed_at: 2021-07-16T19:44:58Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1397
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1397/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NDcwNDQ2Njc=
title: Documentation on using Datasette as a library
state: open
created_at: 2021-07-18T14:15:27Z
updated_at: 2021-07-30T03:21:49Z
author_association: OWNER
type: issue
body:
  Instantiating `Datasette()` directly is an increasingly interesting pattern. I do it in tests all the time, but thanks to `datasette.client` there are plenty of neat things you can do with it in a library context.

  Maybe support `from datasette import Datasette` for this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1398/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjkyNTU3Mzgx
title: Multiple sort
state: open
created_at: 2021-07-19T12:20:14Z
updated_at: 2021-07-19T12:20:14Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/datasette/pulls/1399
type: pull
body:
  Closes #197.
  I have added support for sorting by multiple parameters as mentioned in the issue above, and together with that, a suggestion on how to implement such sorting in the user interface.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1399/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NjkyNTk2MDA2
title: Bump black from 21.6b0 to 21.7b0
state: closed
created_at: 2021-07-19T13:13:41Z
updated_at: 2021-08-25T01:29:56Z
closed_at: 2021-08-25T01:29:55Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1400
type: pull
body:
  Bumps [black](https://github.com/psf/black) from 21.6b0 to 21.7b0.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/releases">black's releases</a>.</em></p>
  <blockquote>
  <h2>21.7b0</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>Configuration files using TOML features higher than spec v0.5.0 are now supported
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/2301">#2301</a>)</li>
  <li>Add primer support and test for code piped into black via STDIN (<a href="https://github-redirect.dependabot.com/psf/black/issues/2315">#2315</a>)</li>
  <li>Fix internal error when <code>FORCE_OPTIONAL_PARENTHESES</code> feature is enabled (<a href="https://github-redirect.dependabot.com/psf/black/issues/2332">#2332</a>)</li>
  <li>Accept empty stdin (<a href="https://github-redirect.dependabot.com/psf/black/issues/2346">#2346</a>)</li>
  <li>Provide a more useful error when parsing fails during AST safety checks (<a href="https://github-redirect.dependabot.com/psf/black/issues/2304">#2304</a>)</li>
  </ul>
  <h3>Docker</h3>
  <ul>
  <li>Add new <code>latest_release</code> tag automation to follow latest black release on docker
  images (<a href="https://github-redirect.dependabot.com/psf/black/issues/2374">#2374</a>)</li>
  </ul>
  <h3>Integrations</h3>
  <ul>
  <li>The vim plugin now searches upwards from the directory containing the current buffer
  instead of the current working directory for pyproject.toml. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1871">#1871</a>)</li>
  <li>The vim plugin now reads the correct string normalization option in pyproject.toml
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/1869">#1869</a>)</li>
  <li>The vim plugin no longer crashes Black when there's boolean values in pyproject.toml
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/1869">#1869</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/psf/black/blob/main/CHANGES.md">black's changelog</a>.</em></p>
  <blockquote>
  <h2>21.7b0</h2>
  <h3><em>Black</em></h3>
  <ul>
  <li>Configuration files using TOML features higher than spec v0.5.0 are now supported
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/2301">#2301</a>)</li>
  <li>Add primer support and test for code piped into black via STDIN (<a href="https://github-redirect.dependabot.com/psf/black/issues/2315">#2315</a>)</li>
  <li>Fix internal error when <code>FORCE_OPTIONAL_PARENTHESES</code> feature is enabled (<a href="https://github-redirect.dependabot.com/psf/black/issues/2332">#2332</a>)</li>
  <li>Accept empty stdin (<a href="https://github-redirect.dependabot.com/psf/black/issues/2346">#2346</a>)</li>
  <li>Provide a more useful error when parsing fails during AST safety checks (<a href="https://github-redirect.dependabot.com/psf/black/issues/2304">#2304</a>)</li>
  </ul>
  <h3>Docker</h3>
  <ul>
  <li>Add new <code>latest_release</code> tag automation to follow latest black release on docker
  images (<a href="https://github-redirect.dependabot.com/psf/black/issues/2374">#2374</a>)</li>
  </ul>
  <h3>Integrations</h3>
  <ul>
  <li>The vim plugin now searches upwards from the directory containing the current buffer
  instead of the current working directory for pyproject.toml. (<a href="https://github-redirect.dependabot.com/psf/black/issues/1871">#1871</a>)</li>
  <li>The vim plugin now reads the correct string normalization option in pyproject.toml
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/1869">#1869</a>)</li>
  <li>The vim plugin no longer crashes Black when there's boolean values in pyproject.toml
  (<a href="https://github-redirect.dependabot.com/psf/black/issues/1869">#1869</a>)</li>
  </ul>
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li>See full diff in <a href="https://github.com/psf/black/commits">compare view</a></li>
  </ul>
  </details>
  <br />


  [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=21.6b0&new-version=21.7b0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1400/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTA2NjQ5NzE=
title: unordered list is not rendering bullet points in description_html on database page
state: open
created_at: 2021-07-22T13:24:18Z
updated_at: 2021-10-23T13:09:10Z
author_association: CONTRIBUTOR
type: issue
body:
  Thanks for this tremendous package, @simonw!

  In the `description_html` for a database, I [have an unordered list](https://github.com/labordata/warehouse/blob/fcea4502e5b615b0eb3e0bdcb45ec634abe20bb6/warehouse_metadata.yml#L19-L22).

  However, on the database page on the deployed site, it is not rendering this as a bulleted list.

  ![Screenshot 2021-07-22 at 09-21-51 nlrb](https://user-images.githubusercontent.com/536941/126645923-2777b7f1-fd4c-4d2d-af70-a35e49a07675.png)

  Page here: https://labordata-warehouse.herokuapp.com/nlrb-9da4ae5

  The documentation gives an [example of using an unordered list](https://docs.datasette.io/en/stable/metadata.html#using-yaml-for-metadata) in a `description_html`, so I expected this will work.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1401/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTExODU0MTE=
title: feature request: social meta tags
state: open
created_at: 2021-07-23T01:57:23Z
updated_at: 2021-07-26T19:31:41Z
author_association: CONTRIBUTOR
type: issue
body:
  it would be very nice if the twitter, slack, and other social media could make rich cards when people post a link to a datasette instance

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1402/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTE1ODE3NjM=
title: Read lines with JSON object
state: closed
created_at: 2021-07-23T13:28:52Z
updated_at: 2021-08-03T06:50:47Z
closed_at: 2021-08-02T21:55:16Z
author_association: NONE
type: issue
state_reason: completed
body:
  I found this posted on HN a while ago and love it -- thank you!

  As a minor improvement, it would be great to have the ability to parse a file with line-separated JSON objects. Currently the parser obviously requires an array wrapping all these objects.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/298/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTE4MTczMjg=
title: 403 when getting token
state: open
created_at: 2021-07-23T18:43:26Z
updated_at: 2021-10-12T18:31:57Z
author_association: NONE
type: issue
body:
  I tried to use https://your-foursquare-oauth-token.glitch.me/ to get my Swarm auth token and got a 403 after I clicked the Allow button:

  ![image](https://user-images.githubusercontent.com/285352/126826478-60e53614-263d-40bb-9f1d-c1a676644eb0.png)

  I'm not sure if this is the right repo to report this in

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTIxNTQ0Njg=
title: Ability to see just specific table schemas with `sqlite-utils schema`
state: closed
created_at: 2021-07-24T22:00:05Z
updated_at: 2021-07-24T22:12:01Z
closed_at: 2021-07-24T22:08:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It currently accepts no arguments. Allowing for optional arguments specifying tables would be useful:

      sqlite-utils schema fixtures.db facetable searchable

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/299/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTIxNzk4MzA=
title: Command for fetching Hacker News threads from the search API
state: open
created_at: 2021-07-25T02:00:45Z
updated_at: 2021-07-25T03:12:57Z
author_association: MEMBER
type: issue
body:
  I want to be able to fetch every item for a domain, e.g. https://news.ycombinator.com/from?site=simonwillison.net

reactions:
  {"url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTIxODkxNzM=
title: Use HN algolia endpoint to retrieve trees
state: open
created_at: 2021-07-25T03:35:27Z
updated_at: 2021-07-25T18:41:17Z
author_association: MEMBER
type: issue
body:
  The `trees` command currently has to make a request for every single comment. Algolia have an endpoint that bundles the entire thread together into a single request.

  `https://hn.algolia.com/api/v1/items/ID`

  Here's an example that loads quickly, with about 50 comments: https://hn.algolia.com/api/v1/items/27941108

  It doesn't appear to use pagination at all - if a thread is big then the response is big.

  I ran this search to find some stories with more than 1000 comments: https://hn.algolia.com/api/v1/search?tags=story&numericFilters=num_comments%3E=1000

  Here's one: https://news.ycombinator.com/item?id=25015967 with 4759 comments. Hitting the API takes 41s and returns 3.7 MB of JSON!
  ```
  wget 'https://hn.algolia.com/api/v1/items/25015967'  0.03s user 0.04s system 0% cpu 41.368 total
  /tmp % ls -lah 25015967 
  -rw-r--r--  1 simon  wheel   3.7M Jul 24 20:31 25015967
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTMyMTgwNDM=
title: Labels explaining what hidden tables are for
state: open
created_at: 2021-07-26T19:29:22Z
updated_at: 2022-03-21T22:20:37Z
author_association: OWNER
type: issue
body:
  A reasonable question: "What are those hidden tables for?"

  This could be answered by adding a small piece of explanatory text to each table - based on if it's related to FTS or to SpatiaLite or configured to be hidden for some other reason.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1403/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTMzNTIwMTU=
title: `register_routes()` hook should take `datasette` argument
state: closed
created_at: 2021-07-26T23:00:33Z
updated_at: 2021-07-26T23:27:07Z
closed_at: 2021-07-26T23:26:00Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Currently that plugin hook takes no arguments at all. This means it's not possible to conditionally register routes based on Datasette plugin configuration.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1404/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Njk4NDIzNjY3
title: Add Gmail takeout mbox import (v2)
state: open
created_at: 2021-07-28T07:05:32Z
updated_at: 2023-09-08T01:22:49Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/google-takeout-to-sqlite/pulls/8
type: pull
body:
  WIP

  This PR builds on #5 to continue implementing gmail import support.

  Building on @UtahDave's work, these commits add a few performance and bug fixes:

  * Decreased memory overhead for import by manually parsing mbox headers.
  * Fixed error where some messages in the mbox would yield a row with NULL in all columns.

  I will send more commits to fix any errors I encounter as I run the importer on my personal takeout data.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTUzMTYyNTA=
title: utils.parse_metadata() should be a documented internal function
state: closed
created_at: 2021-07-28T23:51:39Z
updated_at: 2021-07-29T23:33:30Z
closed_at: 2021-07-29T23:30:24Z
author_association: OWNER
body: Because it's used by this plugin: https://github.com/simonw/datasette-remote-metadata
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1405/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTYzMDM0NzA=
title: Tests failing with FileNotFoundError in runner.isolated_filesystem
state: closed
created_at: 2021-07-30T00:39:00Z
updated_at: 2021-07-31T18:56:35Z
closed_at: 2021-07-31T18:56:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. https://github.com/simonw/datasette/runs/3197141955

  I've seen this error before, but I don't yet have a good workaround for it.

  ```
      @contextlib.contextmanager
      def isolated_filesystem(
          self, temp_dir: t.Optional[t.Union[str, os.PathLike]] = None
      ) -> t.Iterator[str]:
          """A context manager that creates a temporary directory and
          changes the current working directory to it. This isolates tests
          that affect the contents of the CWD to prevent them from
          interfering with each other.
      
          :param temp_dir: Create the temporary directory under this
              directory. If given, the created directory is not removed
              when exiting.
      
          .. versionchanged:: 8.0
              Added the ``temp_dir`` parameter.
          """
  >       cwd = os.getcwd()
  E       FileNotFoundError: [Errno 2] No such file or directory

  /opt/hostedtoolcache/Python/3.6.14/x64/lib/python3.6/site-packages/click/testing.py:466: FileNotFoundError
  =========================== short test summary info ============================
  FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_apt_get_install
  FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[---setting force_https_urls on]
  FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[--setting base_url /foo---setting base_url /foo --setting force_https_urls on]
  FAILED tests/test_publish_cloudrun.py::test_publish_cloudrun_extra_options[--setting force_https_urls off---setting force_https_urls off]
  FAILED tests/test_publish_heroku.py::test_publish_heroku_requires_heroku - Fi...
  FAILED tests/test_publish_heroku.py::test_publish_heroku_installs_plugin - Fi...
  FAILED tests/test_publish_heroku.py::test_publish_heroku - FileNotFoundError:...
  FAILED tests/test_publish_heroku.py::test_publish_heroku_plugin_secrets - Fil...
  ================== 8 failed, 920 passed in 188.22s (0:03:08) ===================
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1406/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTY4MzI4MzY=
title: Returning underlying cause for User Defined Functions
state: closed
created_at: 2021-07-30T15:08:21Z
updated_at: 2021-08-02T21:53:50Z
closed_at: 2021-08-02T21:53:50Z
author_association: NONE
type: issue
state_reason: completed
body:
  The sqlite3 client takes user defined functions and replaces the text with "user-defined function raised exception`" so it's not apparent what's gone wrong:

  ```
  Unexpected error: user-defined function raised exception
  ```

  As mentioned in https://code.djangoproject.com/ticket/29500 and https://stackoverflow.com/questions/45824209/how-to-get-an-error-kind-from-sqlite-create-function/45834923#45834923 the workaround for this is to enable callback tracebacks:

  ```
  sqlite3.enable_callback_tracebacks(True)
  ```

  It would be nice if https://sqlite-utils.datasette.io/en/stable/python-api.html#registering-custom-sql-functions either included a reference to `enable_callback_tracebacks` or if registering a user defined function set this flag automatically.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/300/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTcyOTg0NzU=
title: OSError: AF_UNIX path too long in ds_unix_domain_socket_server
state: closed
created_at: 2021-07-31T18:36:06Z
updated_at: 2021-07-31T19:03:44Z
closed_at: 2021-07-31T19:03:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Got this exception while working on #1406.

  ```
      @pytest.fixture(scope="session")
      def ds_unix_domain_socket_server(tmp_path_factory):
          socket_folder = tmp_path_factory.mktemp("uds")
          uds = str(socket_folder / "datasette.sock")
          ds_proc = subprocess.Popen(
              ["datasette", "--memory", "--uds", uds],
              stdout=subprocess.PIPE,
              stderr=subprocess.STDOUT,
              cwd=tempfile.gettempdir(),
          )
          # Give the server time to start
          time.sleep(1.5)
          # Check it started successfully
  >       assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
  E       AssertionError: INFO:     Started server process [48453]
  E         INFO:     Waiting for application startup.
  E         INFO:     Application startup complete.
  E         Traceback (most recent call last):
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 33, in <module>
  E             sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')())
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
  E             return self.main(*args, **kwargs)
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1062, in main
  E             rv = self.invoke(ctx)
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
  E             return _process_result(sub_ctx.command.invoke(sub_ctx))
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
  E             return ctx.invoke(self.callback, **ctx.params)
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 763, in invoke
  E             return __callback(*args, **kwargs)
  E           File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 583, in serve
  E             uvicorn.run(ds.app(), **uvicorn_kwargs)
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/main.py", line 393, in run
  E             server.run()
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/server.py", line 50, in run
  E             loop.run_until_complete(self.serve(sockets=sockets))
  E           File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
  E             return future.result()
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/server.py", line 67, in serve
  E             await self.startup(sockets=sockets)
  E           File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/uvicorn/server.py", line 133, in startup
  E             server = await asyncio.start_unix_server(
  E           File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/streams.py", line 132, in start_unix_server
  E             return await loop.create_unix_server(factory, path, **kwds)
  E           File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/asyncio/unix_events.py", line 296, in create_unix_server
  E             sock.bind(path)
  E         OSError: AF_UNIX path too long
  E         
  E       assert not 1
  E        +  where 1 = <bound method Popen.poll of <subprocess.Popen object at 0x106924af0>>()
  E        +    where <bound method Popen.poll of <subprocess.Popen object at 0x106924af0>> = <subprocess.Popen object at 0x106924af0>.poll
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1407/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTczMDIwODU=
title: Review places in codebase that use os.chdir(), in particularly relating to tests
state: open
created_at: 2021-07-31T18:57:06Z
updated_at: 2021-07-31T19:00:32Z
author_association: OWNER
type: issue
body:
  > To clarify: the core problem here is that an error is thrown any time you call `os.getcwd()` but the directory you are currently in has been deleted.
  >
  > `runner.isolated_filesystem()` assumes that the current directory in has not been deleted. But the various temporary directory utilities in `pytest` work by creating directories and then deleting them.
  >
  > Maybe there's a larger problem here that I play a bit fast and loose with `os.chdir()` in both the test suite and in various lines of code in Datasette itself (in particular in the publish commands)?

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1406#issuecomment-890390198_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1408/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTczMTAyNzg=
title: `default_allow_sql` setting (a re-imagining of the old `allow_sql` setting)
state: closed
created_at: 2021-07-31T19:48:56Z
updated_at: 2023-01-07T18:06:01Z
closed_at: 2023-01-05T00:51:31Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In 49d6d2f7b0f6cb02e25022e1c9403811f1fa0a7c as part of #813 I removed the `allow_sql` setting - on the basis that users could disable the ability to execute custom SQL queries using the new permission system instead.

  I don't think this was the right decision. Disabling custom SQL is an important security capability, and explaining how to do it using permissions is significantly  more complex than letting people know they can add `--setting allow_sql off`.

  So I want to bring that setting back - maybe with a different, better name - and have it modify the default for that option if the permissions system doesn't have an opinion.

  That way people can still use the setting but then use permissions to allow specific signed-in users access to execute SQL.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1409/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTczMTU2ODQ=
state: open
created_at: 2021-07-31T20:27:12Z
updated_at: 2021-07-31T20:27:49Z
author_association: OWNER
type: issue
title:
  Rename settings to `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`

body:
  > If I was prone to over-thinking (which I am) I'd note that `allow_facet` and `allow_download` and `allow_csv_stream` are all settings that do NOT have an equivalent in the newer permissions system, which is itself a little weird and inconsistent.
  >
  > So maybe there's a future task where I introduce those as both permissions and metadata `"allow_x"` blocks, then rename the settings themselves to be called `default_allow_facet` and `default_allow_download` and `default_allow_csv_stream`.
  >
  > If I was going to do that I should get it in before Datasette 1.0.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1409#issuecomment-890400425_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1410/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTczNDU0NzY=
title: Canned query ?sql= is pointlessly echoed in query string starting from hidden mode
state: closed
created_at: 2021-08-01T00:17:13Z
updated_at: 2021-08-01T03:27:30Z
closed_at: 2021-08-01T00:58:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Example: https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1

  Submitting that form again results in this:

  https://latest.datasette.io/fixtures/neighborhood_search?sql=%0D%0Aselect+neighborhood%2C+facet_cities.name%2C+state%0D%0Afrom+facetable%0D%0A++++join+facet_cities%0D%0A++++++++on+facetable.city_id+%3D+facet_cities.id%0D%0Awhere+neighborhood+like+%27%25%27+%7C%7C+%3Atext+%7C%7C+%27%25%27%0D%0Aorder+by+neighborhood%3B%0D%0A&_hide_sql=1&text=cork

  Because the HTML on https://latest.datasette.io/fixtures/neighborhood_search?text=cork&_hide_sql=1 includes this:

  ```html
      <h3>Custom SQL query returning 1 row <span class="show-hide-sql">(<a href="/fixtures/neighborhood_search?text=cork">show</a>)</span></h3>
      
      
          <input type="hidden" name="sql" value="
  select neighborhood, facet_cities.name, state
  from facetable
      join facet_cities
          on facetable.city_id = facet_cities.id
  where neighborhood like &#39;%&#39; || :text || &#39;%&#39;
  order by neighborhood;
  ">
          <input type="hidden" name="_hide_sql" value="1">
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1411/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTczNDc4NjA=
title: Mention WAL mode in the documentation (plus backup tips)
state: open
created_at: 2021-08-01T00:27:11Z
updated_at: 2021-08-01T00:27:11Z
author_association: OWNER
type: issue
body:
  This is useful for people who are deploying Datasette with any write functionality, especially if they might be using something like EFS.

  I can add a section about this to the bottom of https://docs.datasette.io/en/stable/deploying.html and then link to that section from both https://docs.datasette.io/en/stable/sql_queries.html#writable-canned-queries and https://docs.datasette.io/en/stable/internals.html#await-db-execute-write-sql-params-none-block-false

  Also useful: mention that just copying a SQLite database file while it is being written to may not get a consistent file, so tell people to use one of the SQLite backup mechanisms instead.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1412/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTczODM4MTQ=
title: insert-files should get a --silent option
state: closed
created_at: 2021-08-01T04:11:03Z
updated_at: 2021-08-02T19:12:21Z
closed_at: 2021-08-02T19:12:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The new `sqlite-utils convert` command I'm adding in #251 will have a `--silent` option for turning off the progress bars. The only other command that has progress bars right now is `insert-files` so it should get this option too, for consistency.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/301/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTc1MjkyNDg=
title: Python library version of `sqlite-utils convert`
state: closed
created_at: 2021-08-01T16:11:02Z
updated_at: 2021-08-02T04:47:40Z
closed_at: 2021-08-02T04:47:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Spin off from #251. The ability to execute Python functions to convert and split columns should be part of the library too, not just the CLI.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/302/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NzAwOTQ0NjQ0
title: sqlite-utils convert command and db[table].convert(...) method
state: closed
created_at: 2021-08-01T16:52:42Z
updated_at: 2021-08-02T04:47:42Z
closed_at: 2021-08-02T04:47:39Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/303
type: pull
body:
  Refs #251, #302.

  - [x] Get recipes working
  - [x] Document recipes
  - [x] Implement `db[table].convert(...)` method
  - [x] Add tests for recipes that use the new Python method
  - [x] Implement `db[table].convert(..., multi=True)` mechanism
  - [x] Documentation for `db[table].convert(...)`
  - [x] Refactor `sqlite-utils convert` to use the new method

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/303/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTc3MzExNzg=
title: `table.convert(..., where=)` and `sqlite-utils convert ... --where=`
state: closed
created_at: 2021-08-02T04:27:23Z
updated_at: 2021-08-02T19:00:00Z
closed_at: 2021-08-02T18:58:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  For applying the conversion to a subset of rows selected using the where clause.

  Should also take optional arguments, as seen in `db["dogs"].delete_where("age < ?", [3])`.

  Follows #302 and #251. This was originally https://github.com/simonw/sqlite-transform/issues/9

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/304/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTc3NDE4MjA=
title: Python: need a way to execute a count with an extra where clause
state: closed
created_at: 2021-08-02T04:52:02Z
updated_at: 2021-08-02T05:08:22Z
closed_at: 2021-08-02T05:08:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I need this for #304. I'll probably add this to the `.execute_count()` method as `where=` and `where_args=`.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/305/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTg1MTY3NDM=
title: Configure sphinx.ext.extlinks for issues
state: closed
created_at: 2021-08-02T21:19:19Z
updated_at: 2021-08-02T21:39:34Z
closed_at: 2021-08-02T21:29:22Z
author_association: OWNER
body: As seen in Datasette: https://github.com/simonw/datasette/issues/1227
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/306/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTkxMzcxNDM=
title: feature request: document minimum permissions for service account for cloudrun
state: open
created_at: 2021-08-03T13:48:43Z
updated_at: 2023-11-05T16:46:59Z
author_association: CONTRIBUTOR
type: issue
body:
  Thanks again for such a powerful project.

  For deploying to cloudrun from github actions, I'd like to create a service account with minimal permissions.

  It would be great to document what those minimum permission that need to be set in the IAM.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1415/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTkyNzY2Mjk=
title: Use rich to render tracebacks on errors, if available
state: closed
created_at: 2021-08-03T16:12:08Z
updated_at: 2021-08-03T16:12:51Z
closed_at: 2021-08-03T16:12:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Now thinking I should try adding Rich as an optional dependency to Datasette - if it's there, show tracebacks using it. Could be really handy for development
  > https://twitter.com/simonw/status/1422576091055616003

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1416/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTkyNzg0NzI=
title: Use codespell in CI to spot spelling errors
state: closed
created_at: 2021-08-03T16:14:15Z
updated_at: 2021-08-03T16:36:40Z
closed_at: 2021-08-03T16:36:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I noticed Rich is using this: https://github.com/willmcgugan/rich/commit/9c12a4537499797c43725fff5276ef0da62423ef#diff-ce84a1b2c9eb4ab3ea22f610cad7111cb9a2f66365c3b24679901376a2a73ab2

  Ran it against the Datasette docs and found a bunch of obvious fixes, surprisingly with no false positives.

  ```
  datasette % codespell docs/*.rst
  docs/authentication.rst:63: perfom ==> perform
  docs/authentication.rst:76: perfom ==> perform
  docs/changelog.rst:429: repsonse ==> response
  docs/changelog.rst:503: permissons ==> permissions
  docs/changelog.rst:717: compatibilty ==> compatibility
  docs/changelog.rst:1172: browseable ==> browsable
  docs/deploying.rst:191: similiar ==> similar
  docs/internals.rst:434: Respons ==> Response, respond
  docs/internals.rst:440: Respons ==> Response, respond
  docs/internals.rst:717: tha ==> than, that, the
  docs/performance.rst:42: databse ==> database
  docs/plugin_hooks.rst:667: utilites ==> utilities
  docs/publish.rst:168: countainer ==> container
  docs/settings.rst:352: inalid ==> invalid
  docs/sql_queries.rst:406: preceeded ==> preceded, proceeded
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1417/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NzAyNDIyMjYz
title: Spelling corrections plus CI job for codespell
state: closed
created_at: 2021-08-03T16:21:19Z
updated_at: 2021-08-03T16:36:39Z
closed_at: 2021-08-03T16:36:38Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1418
body: Refs #1417.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1418/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTkzMDUyMDk=
title: codespell to spell check documentation
state: closed
created_at: 2021-08-03T16:48:19Z
updated_at: 2021-08-03T16:48:53Z
closed_at: 2021-08-03T16:48:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  As seen in https://github.com/simonw/datasette/issues/1417 and https://til.simonwillison.net/python/codespell

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/307/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTk3MTAwMDg=
title: `publish cloudrun` should deploy a more recent SQLite version
state: open
created_at: 2021-08-04T00:45:55Z
updated_at: 2021-08-05T03:23:24Z
author_association: CONTRIBUTOR
type: issue
body:
  I recently changed from deploying a datasette using `datasette publish heroku` to `datasette publish cloudrun`. [A query that ran on the heroku site](https://odpr.bunkum.us/odpr-6c2f4fc?sql=with+pivot_members+as+%28%0D%0A++select%0D%0A++++f_num%2C%0D%0A++++max%28union_name%29+as+union_name%2C%0D%0A++++max%28aff_abbr%29+as+abbreviation%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2007%0D%0A++++%29+as+%222007%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2008%0D%0A++++%29+as+%222008%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2009%0D%0A++++%29+as+%222009%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2010%0D%0A++++%29+as+%222010%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2011%0D%0A++++%29+as+%222011%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2012%0D%0A++++%29+as+%222012%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2013%0D%0A++++%29+as+%222013%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2014%0D%0A++++%29+as+%222014%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2015%0D%0A++++%29+as+%222015%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2016%0D%0A++++%29+as+%222016%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2017%0D%0A++++%29+as+%222017%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2018%0D%0A++++%29+as+%222018%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2019%0D%0A++++%29+as+%222019%22%0D%0A++from%0D%0A++++lm_data%0D%0A++++left+join+ar_membership+using+%28rpt_id%29%0D%0A++where%0D%0A++++members+%21%3D+%27%27%0D%0A++++and+trim%28desig_name%29+%3D+%27NHQ%27%0D%0A++++and+union_name+not+in+%28%0D%0A++++++%27AFL-CIO%27%2C%0D%0A++++++%27CHANGE+TO+WIN%27%2C%0D%0A++++++%27FOOD+ALLIED+SVC+TRADES+DEPT+AFL-CIO%27%0D%0A++++%29%0D%0A++++and+f_num+not+in+%28387%2C+296%2C+123%2C+347%2C+531897%2C+30410%2C+49%29%0D%0A++++and+lower%28ar_membership.category%29+IN+%28%0D%0A++++++%27active+members%27%2C%0D%0A++++++%27active+professional%27%2C%0D%0A++++++%27regular+members%27%2C%0D%0A++++++%27full+time+member%27%2C%0D%0A++++++%27full+time+members%27%2C%0D%0A++++++%27active+education+support+professional%27%2C%0D%0A++++++%27active%27%2C%0D%0A++++++%27active+member%27%2C%0D%0A++++++%27members%27%2C%0D%0A++++++%27see+item+69%27%2C%0D%0A++++++%27regular%27%2C%0D%0A++++++%27dues+paying+members%27%2C%0D%0A++++++%27building+trades+journeyman%27%2C%0D%0A++++++%27full+per+capita+tax+payers%27%2C%0D%0A++++++%27active+postal%27%2C%0D%0A++++++%27active+membership%27%2C%0D%0A++++++%27full-time+member%27%2C%0D%0A++++++%27regular+active+member%27%2C%0D%0A++++++%27journeyman%27%2C%0D%0A++++++%27member%27%2C%0D%0A++++++%27journeymen%27%2C%0D%0A++++++%27one+half+per+capita+tax+payers%27%2C%0D%0A++++++%27schedule+1%27%2C%0D%0A++++++%27active+memebers%27%2C%0D%0A++++++%27active+members+-+us%27%2C%0D%0A++++++%27dues+paying+membership%27%0D%0A++++%29%0D%0A++GROUP+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++pivot_members%0D%0Aorder+by%0D%0A++%222019%22+desc%3B), now throws a syntax error on the [cloudrun site](https://labordata.bunkum.us/odpr-6c2f4fc?sql=with+pivot_members+as+%28%0D%0A++select%0D%0A++++f_num%2C%0D%0A++++max%28union_name%29+as+union_name%2C%0D%0A++++max%28aff_abbr%29+as+abbreviation%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2007%0D%0A++++%29+as+%222007%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2008%0D%0A++++%29+as+%222008%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2009%0D%0A++++%29+as+%222009%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2010%0D%0A++++%29+as+%222010%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2011%0D%0A++++%29+as+%222011%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2012%0D%0A++++%29+as+%222012%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2013%0D%0A++++%29+as+%222013%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2014%0D%0A++++%29+as+%222014%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2015%0D%0A++++%29+as+%222015%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2016%0D%0A++++%29+as+%222016%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2017%0D%0A++++%29+as+%222017%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2018%0D%0A++++%29+as+%222018%22%2C%0D%0A++++sum%28number%29+filter+%28%0D%0A++++++where%0D%0A++++++++yr_covered+%3D+2019%0D%0A++++%29+as+%222019%22%0D%0A++from%0D%0A++++lm_data%0D%0A++++left+join+ar_membership+using+%28rpt_id%29%0D%0A++where%0D%0A++++members+%21%3D+%27%27%0D%0A++++and+trim%28desig_name%29+%3D+%27NHQ%27%0D%0A++++and+union_name+not+in+%28%0D%0A++++++%27AFL-CIO%27%2C%0D%0A++++++%27CHANGE+TO+WIN%27%2C%0D%0A++++++%27FOOD+ALLIED+SVC+TRADES+DEPT+AFL-CIO%27%0D%0A++++%29%0D%0A++++and+f_num+not+in+%28387%2C+296%2C+123%2C+347%2C+531897%2C+30410%2C+49%29%0D%0A++++and+lower%28ar_membership.category%29+IN+%28%0D%0A++++++%27active+members%27%2C%0D%0A++++++%27active+professional%27%2C%0D%0A++++++%27regular+members%27%2C%0D%0A++++++%27full+time+member%27%2C%0D%0A++++++%27full+time+members%27%2C%0D%0A++++++%27active+education+support+professional%27%2C%0D%0A++++++%27active%27%2C%0D%0A++++++%27active+member%27%2C%0D%0A++++++%27members%27%2C%0D%0A++++++%27see+item+69%27%2C%0D%0A++++++%27regular%27%2C%0D%0A++++++%27dues+paying+members%27%2C%0D%0A++++++%27building+trades+journeyman%27%2C%0D%0A++++++%27full+per+capita+tax+payers%27%2C%0D%0A++++++%27active+postal%27%2C%0D%0A++++++%27active+membership%27%2C%0D%0A++++++%27full-time+member%27%2C%0D%0A++++++%27regular+active+member%27%2C%0D%0A++++++%27journeyman%27%2C%0D%0A++++++%27member%27%2C%0D%0A++++++%27journeymen%27%2C%0D%0A++++++%27one+half+per+capita+tax+payers%27%2C%0D%0A++++++%27schedule+1%27%2C%0D%0A++++++%27active+memebers%27%2C%0D%0A++++++%27active+members+-+us%27%2C%0D%0A++++++%27dues+paying+membership%27%0D%0A++++%29%0D%0A++GROUP+by%0D%0A++++f_num%0D%0A%29%0D%0Aselect%0D%0A++*%0D%0Afrom%0D%0A++pivot_members%0D%0Aorder+by%0D%0A++%222019%22+desc%3B).

  I suspect this is because they are running different versions of sqlite3.

  - Heroku: sqlite3 3.31.1 ([-/versions](https://odpr.bunkum.us/-/versions))
  - Cloudrun: sqlite3 3.27.2 ([-/versions](https://labordata.bunkum.us/-/versions))

  If so, it would be great to

  1. harmonize the sqlite3 versions across platforms
  2. update the docker files so as to update the sqlite3 version for cloudrun

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1419/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTk4OTgxNjY=
title: `datasette publish cloudrun --cpu X` option
state: closed
created_at: 2021-08-04T05:04:31Z
updated_at: 2021-08-05T00:54:59Z
closed_at: 2021-08-04T05:33:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  For setting the number of vCPUs - current valid values are 1, 2 or 4: https://cloud.google.com/run/docs/configuring/cpu

  Pass that through to `gcloud run deploy --image IMAGE_URL --cpu CPU`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1420/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NTk5OTkwOTU=
title: "Query parameters" form shows wrong input fields if query contains "03:31" style times
state: closed
created_at: 2021-08-04T07:29:04Z
updated_at: 2021-08-09T03:41:07Z
closed_at: 2021-08-09T03:33:02Z
author_association: NONE
type: issue
state_reason: completed
body:
  Datasette version `0.58.1`.

  I'm guessing this is a bug in the code that looks for `:param`-style query parameters..

  <img width="543" alt="image" src="https://user-images.githubusercontent.com/6988/128139832-ef9c5291-f3d7-4402-8625-b45d26b4e5bc.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1421/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NjEwMDg1MDc=
title: Add an interactive tutorial as a Jupyter notebook
state: open
created_at: 2021-08-04T20:34:22Z
updated_at: 2021-08-04T21:30:59Z
author_association: OWNER
body: Can show people how to open this up in Binder.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/308/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NjEzNjc4NDM=
title: Ability to default to hiding the SQL for a canned query
state: closed
created_at: 2021-08-05T02:51:39Z
updated_at: 2021-08-07T05:32:29Z
closed_at: 2021-08-07T05:32:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm working on a project with some HUGE (400+ lines of SQL) canned queries right now.

  Any time you land on the canned query page you have to scroll down a long distance to get to the results!

  Would be useful to be able to default to https://latest.datasette.io/fixtures/magic_parameters?_hide_sql=1 without needing the parameter.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1422/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NjIzOTEzMjU=
title: Show count of facet values if ?_facet_size=max
state: closed
created_at: 2021-08-06T04:42:20Z
updated_at: 2021-12-15T17:48:40Z
closed_at: 2021-08-16T18:56:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I sometimes want to get a count of the values in a facet - if it's a facet of US states for example I want to know if all 50 are represented.

  Idea: if `?_facet_size=max` is present, add a count to the facet heading. So on:

  https://latest.datasette.io/fixtures/compound_three_primary_keys?_facet=content&_facet_size=max&_facet=pk1&_facet=pk2#facet-pk2

  It could have something like this:

  <img width="658" alt="fixtures__compound_three_primary_keys__1_001_rows_and_NY_629_755_•_Untitled-1_—_datasette" src="https://user-images.githubusercontent.com/9599/128456981-1f3d5348-90f1-45cc-9287-0f7bb0015257.png">

  Note that the first column shows >1000 - because in that case we've truncated the facet calculation since the maximum allowed returned rows is 1000.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1423/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NjM1MjcwNDU=
title: Document exceptions that can be raised by db.execute() and friends
state: open
created_at: 2021-08-08T22:23:25Z
updated_at: 2021-08-08T22:27:31Z
author_association: OWNER
body: Not currently covered here: https://docs.datasette.io/en/stable/internals.html#await-db-execute-sql
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1424/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NjM1Mjg0NTc=
title: render_cell() hook should support returning an awaitable
state: closed
created_at: 2021-08-08T22:32:29Z
updated_at: 2021-08-09T07:14:35Z
closed_at: 2021-08-09T03:00:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Many of the plugin hooks can return an awaitable - e.g. https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hook-extra-template-vars - but `render_cell()` doesn't support this.

  I recently found myself wanting to execute an additional SQL query from that hook, but it wasn't possible to do that since I couldn't use `await`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1425/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU5NjM4OTcxMTE=
title: sqlite-utils insert errors should show SQL and parameters, if possible
state: closed
created_at: 2021-08-09T11:24:14Z
updated_at: 2021-08-09T23:40:29Z
closed_at: 2021-08-09T22:25:58Z
author_association: NONE
type: issue
state_reason: completed
body:
  I've tried several approaches, but this is the current one:

  ```sh
  echo $json-line | sqlite-utils insert json.db jsontable --truncate --alter --detect-types -
  ```
  In all cases, I get this error:

  ```sh
  OverflowError: Python int too large to convert to SQLite INTEGER
  Traceback (most recent call last):
    File "/home/sean/.local/bin/sqlite-utils", line 8, in <module>
      sys.exit(cli())
    File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 841, in insert
      insert_upsert_implementation(
    File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 780, in insert_upsert_implementation
      db[table].insert_all(
    File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 2145, in insert_all
      self.insert_chunk(
    File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 1957, in insert_chunk
      result = self.db.execute(query, params)
    File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 257, in execute
      return self.conn.execute(sql, parameters)
  ```

  I googled the error and checked SO answers and advice, all good. I changed my JSON file to not use integers so I no longer get this error. Of course, that makes using the database a bit harder, so I also tried to solve the problem by modifying DB structure (while using integers in JSON).

  If change all `INTEGER` Data Types to something else (`STRING`, `TEXT`) and try to import again using `--truncate`, I still get this error. I suppose I should tell sqlite-utils which columns should use non-INTEGER Data Type rather than rely on it to check my SQL table configuration.

  If that is the case, can this error be a bit more specific for easier troubleshooting - maybe tell us which which record caused the problem when that error is thrown? 

  My table has 60+ columns, many of which use 64-bit integers (not all records are large or known in advance), so while I can modify JSON to use strings instead of integers, it decreases usability and finding out which records have values for which SQLite integers aren't sufficient requires some work (I'm thinking about parsing all integers with `jq` and sorting output by length to identify those columns, but I'd prefer if sqlite-utils could tell me that). 

  My environment:

  - Python 3.8.10
    - sqlite-utils            3.14 
    - pandas                  1.3.1               
    - numpy                   1.21.1              
    - sqlite-fts4             1.0.1
  - sqlite 3.31.1-4ubuntu0.2

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/309/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


