# issues

node_id: MDU6SXNzdWU2OTE1NTc1NDc=
title: Category 3: received
state: closed
created_at: 2020-09-03T01:40:36Z
updated_at: 2020-09-03T17:38:51Z
closed_at: 2020-09-03T17:38:51Z
author_association: MEMBER
body: A category for things that were sent to me: DMs, emails etc. Follows #7.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTIxMjUxMTA=
title: Public / Private mechanism
state: closed
created_at: 2020-09-03T16:47:03Z
updated_at: 2020-09-03T17:33:52Z
closed_at: 2020-09-03T17:33:52Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Some of the data in Dogsheep is stuff that was written publicly - tweets, blog posts, GitHub commits to public repos.

  Some of it is private data - emails, photos, direct messages, Swarm checkins, commits to private repos.

  Being able to filter for just one or the other (or both) would be useful. Especially when giving demos!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTIyMDI0MDg=
title: Idea: maps and GeoJSON support
state: open
created_at: 2020-09-03T18:47:10Z
updated_at: 2020-09-04T01:45:03Z
author_association: MEMBER
type: issue
body:
  It would be cool if the `display_sql` could return a column populated with GeoJSON which would the automatically be displayed on a map in the results (or maybe default JS would look for a `class="geojson"` element output by the `display` template) - ala https://github.com/simonw/datasette-leaflet-geojson

  Then I could render workout routes on a map, or Swarm checkin points.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTIzODY2MjU=
title: Support advanced FTS queries
state: closed
created_at: 2020-09-03T21:29:56Z
updated_at: 2020-09-03T21:40:51Z
closed_at: 2020-09-03T21:40:51Z
author_association: MEMBER
body: `simon willison NOT screenshot` for example.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTMzMTgwOTU=
title: On FTS exception rerun the query with quoting
state: closed
created_at: 2020-09-04T15:44:18Z
updated_at: 2020-09-05T16:23:01Z
closed_at: 2020-09-05T16:23:01Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Searching for eg `#dogfest` currently throws an FTS exception - but I want to support advanced FTS query tricks as seen in #13.

  https://dogsheep.simonwillison.net/-/beta?q=%23dogfest

  > fts5: syntax error near "#"

  Idea: catch that error and re-run the query with FTS escaping applied!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTQxMzY0OTA=
title: Add a bunch of config examples
state: open
created_at: 2020-09-05T17:58:43Z
updated_at: 2020-09-18T23:17:39Z
author_association: MEMBER
body: I can bring these over from my personal Dogsheep.
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTQ0OTM1NjY=
title: Timeline view
state: open
created_at: 2020-09-06T19:13:58Z
updated_at: 2020-09-21T02:42:29Z
author_association: MEMBER
body: Ability to browse (and facet) by date.
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTQ1MDA2Nzk=
title: Rename "table" to "type"
state: closed
created_at: 2020-09-06T19:34:41Z
updated_at: 2020-09-09T03:03:22Z
closed_at: 2020-09-09T03:03:22Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I think "table" is the wrong name for the concept I'm using it for here.

  Two reasons: firstly, `table` is a reserved word in SQLite. More importantly, it turns out there's not a direct mapping from tables to types of search result. In particular, for GitHub I ended up having two different "tables" of repositories - one for repos created by me, another for repos that I have starred.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTUyNzYzMjg=
title: More attractive indentation of created FTS table schema
state: closed
created_at: 2020-09-07T16:49:30Z
updated_at: 2020-09-07T18:12:50Z
closed_at: 2020-09-07T18:12:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  On https://github-to-sqlite.dogsheep.net/github/licenses_fts the create table SQL is displayed as:
  ```sql
  CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (
                  [name],
                  content=[licenses]
              );
  ```
  It would be more aesthetically pleasing if it looked like this:
  ```sql
  CREATE VIRTUAL TABLE [licenses_fts] USING FTS5 (
      [name],
      content=[licenses]
  );
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/148/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTUzMTkyNTg=
title: FTS table with 7 rows has _fts_docsize table with 9,141 rows
state: closed
created_at: 2020-09-07T18:06:16Z
updated_at: 2020-09-07T21:16:34Z
closed_at: 2020-09-07T21:16:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm seeing a weird issue with some of the SQLite databases that I am using with the FTS5 module.

  I have a database with a `licenses` table that contains 7 rows: <https://github-to-sqlite.dogsheep.net/github/licenses>

  The FTS table also has 7 rows: <https://github-to-sqlite.dogsheep.net/github/licenses_fts>

  Somehow the accompanying `licenses_fts_docsize` shadow table now has 9,141 rows in it! <https://github-to-sqlite.dogsheep.net/github/licenses_fts_docsize>

  And `licenses_fts_data` has 41 rows - should I expect that to have 7 rows? <https://github-to-sqlite.dogsheep.net/github/licenses_fts_data>

  I have a hunch that it might be a problem with the triggers. These are the triggers that are updating that FTS table: <https://github-to-sqlite.dogsheep.net/github?sql=select+*+from+sqlite_master+where+type+%3D+%27trigger%27+and+tbl_name+%3D+%27licenses%27>

  | type | name | tbl_name | rootpage | sql |
  | --- | --- | --- | --- | --- |
  | trigger | licenses_ai | licenses | 0 | `CREATE TRIGGER [licenses_ai] AFTER INSERT ON [licenses] BEGIN INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |
  | trigger | licenses_ad | licenses | 0 | `CREATE TRIGGER [licenses_ad] AFTER DELETE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); END` |
  | trigger | licenses_au | licenses | 0 | `CREATE TRIGGER [licenses_au] AFTER UPDATE ON [licenses] BEGIN INSERT INTO [licenses_fts] ([licenses_fts], rowid, [name]) VALUES('delete', old.rowid, old.[name]); INSERT INTO [licenses_fts] (rowid, [name]) VALUES (new.rowid, new.[name]); END` |

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTUzNTk2MDc=
title: Feature for tracing SQL queries
state: closed
created_at: 2020-09-07T19:43:08Z
updated_at: 2020-09-07T21:57:01Z
closed_at: 2020-09-07T21:57:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Debugging `sqlite-utils` when something weird happens (e.g. #149) can be a bit tricky since it runs a bunch of different SQL statements behind the scenes.

  An optional "tracing" mechanism for seeing what SQL is being executed would be useful.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/150/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDgxNjE2NzA0
title: Tracer mechanism for seeing underlying SQL
state: closed
created_at: 2020-09-07T19:46:43Z
updated_at: 2020-09-07T21:57:00Z
closed_at: 2020-09-07T21:57:00Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/151
type: pull
body:
  Refs #150. Needs tests and documentation, including for the new `db.execute()` and `db.executescript()` methods.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/151/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTUzNzYwNTQ=
title: Turn on recursive_triggers by default
state: closed
created_at: 2020-09-07T20:26:36Z
updated_at: 2020-09-07T21:17:48Z
closed_at: 2020-09-07T20:45:14Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://www.sqlite.org/pragma.html#pragma_recursive_triggers says:

  > Prior to SQLite [version 3.6.18](https://www.sqlite.org/releaselog/3_6_18.html) (2009-09-11), recursive triggers were not supported. The behavior of SQLite was always as if this pragma was set to OFF. Support for recursive triggers was added in version 3.6.18 but was initially turned OFF by default, for compatibility. Recursive triggers may be turned on by default in future versions of SQLite.

  So I think the fix for the complex issue in #149 is to turn on `recursive_triggers` globally by default for `sqlite-utils`.

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688499924_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/152/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTUzNzc4MDQ=
title: table.optimize() should delete junk rows from *_fts_docsize
state: closed
created_at: 2020-09-07T20:31:09Z
updated_at: 2020-09-24T20:35:46Z
closed_at: 2020-09-07T21:16:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > The second challenge here is cleaning up all of those junk rows in existing `*_fts_docsize` tables. Doing that just to the demo database from https://github-to-sqlite.dogsheep.net/github.db dropped its size from 22MB to 16MB! Here's the SQL:
  > ```sql
  > DELETE FROM [licenses_fts_docsize] WHERE id NOT IN (
  >   SELECT rowid FROM [licenses_fts]);
  > ```
  > I can do that as part of the existing `table.optimize()` method, which optimizes FTS tables.
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688501064_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/153/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTU0NDE1MzA=
title: OperationalError: cannot change into wal mode from within a transaction
state: open
created_at: 2020-09-07T23:42:44Z
updated_at: 2020-09-07T23:47:10Z
author_association: OWNER
type: issue
body:
  I'm getting this error when running:

      sqlite-utils enable-wal beta.db

  `OperationalError: cannot change into wal mode from within a transaction`

  I'm worried that maybe that's because of this new code from #152:

  https://github.com/simonw/sqlite-utils/blob/deb2eb013ff85bbc828ebc244a9654f0d9c3139e/sqlite_utils/db.py#L128-L129

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/154/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTU1NTM1MjI=
title: Deleted records stay in the search index
state: open
created_at: 2020-09-08T05:14:23Z
updated_at: 2020-09-08T05:15:51Z
author_association: MEMBER
type: issue
body:
  Here's why: https://github.com/dogsheep/dogsheep-beta/blob/24f7898d41a39218058f174c75ba62f7c0fcfff6/dogsheep_beta/utils.py#L44-L53

  That should probably do `DELETE FROM index1.search_index WHERE [table] = ?` first.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTU1NTY2ODE=
title: Figure out incremental re-indexing
state: open
created_at: 2020-09-08T05:23:31Z
updated_at: 2020-09-08T05:27:07Z
author_association: MEMBER
type: issue
body:
  As tables get bigger reindexing everything on a schedule (essentially recreating the entire index from scratch) will start to become a performance bottleneck.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTYwNDU1ODE=
title: rebuild-fts command and table.rebuild_fts() method
state: closed
created_at: 2020-09-08T17:19:26Z
updated_at: 2020-09-24T20:35:46Z
closed_at: 2020-09-08T23:16:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://sqlite.org/forum/forumpost/fa777fff86

  > Easiest thing would be to run a 'rebuild' to rebuild the FTS index from scratch based on the contents of the content table. i.e.
  >
  >     INSERT INTO licenses_fts(licenses_fts) VALUES('rebuild');
  >
  > https://www.sqlite.org/fts5.html#the_rebuild_command

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/155/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTY5MDgzODk=
title: Verification checks for metadata.json on startup
state: open
created_at: 2020-09-09T15:21:53Z
updated_at: 2020-09-09T15:24:31Z
author_association: OWNER
type: issue
body:
  I lost a bunch of time yesterday trying to figure out why a Datasette instance wasn't starting up - it turned out it was because I had a `facets:` reference that mentioned a column that did not exist.

  Catching these on startup would be good.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/961/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDgzMDI3NTg3
title: Typos in tests
state: closed
created_at: 2020-09-09T18:00:58Z
updated_at: 2020-09-09T18:24:50Z
closed_at: 2020-09-09T18:21:23Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/156
type: pull
body:
  One of these is my fault, and the other is one I just happened to come across.  They're harmless, but might as well be fixed.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/156/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTcxNjI5Mzk=
title: Add more tags so people can find your project.
state: open
created_at: 2020-09-09T21:14:09Z
updated_at: 2020-09-09T21:14:09Z
author_association: NONE
body:
  quantified-self habit-tracking google-fit time-tracking wearables quantifiedself
  for example

type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/20/reactions", "total_count": 1, "+1": 0, "-1": 1, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTcxNzk4MDY=
title: sqlite-utils add-foreign-keys command
state: closed
created_at: 2020-09-09T21:44:30Z
updated_at: 2020-09-24T20:34:50Z
closed_at: 2020-09-20T20:14:30Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Like `add-foreign-key` but can do multiple foreign keys at once. Inspired by https://github.com/simonw/calands-datasette/blob/99de39dd80a906f5c1f16724467b0cd55ba4ef36/build.sh which does this:

  ```
  sqlite-utils add-foreign-key calands.db units_with_maps ACCESS_TYP
  sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_NAME
  sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_LEV
  sqlite-utils add-foreign-key calands.db units_with_maps AGNCY_TYP
  sqlite-utils add-foreign-key calands.db units_with_maps LAYER
  sqlite-utils add-foreign-key calands.db units_with_maps MNG_AGENCY
  sqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_LEV
  sqlite-utils add-foreign-key calands.db units_with_maps MNG_AG_TYP
  sqlite-utils add-foreign-key calands.db units_with_maps COUNTY
  sqlite-utils add-foreign-key calands.db units_with_maps DES_TP
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/157/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDgzMTc1NTA5
title: Fix accidental mega long line in docs
state: closed
created_at: 2020-09-09T22:31:23Z
updated_at: 2020-09-16T06:21:43Z
closed_at: 2020-09-16T06:21:43Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/158
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/158/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTg3OTEyMTg=
title: favorites --stop_after=N stops after min(N, 200)
state: open
created_at: 2020-09-11T03:38:14Z
updated_at: 2020-09-13T05:11:14Z
author_association: CONTRIBUTOR
type: issue
body:
  For any number greater than 200, `favorites --stop_after` stops after getting 200 tweets, e.g.
  ```
  $ twitter-to-sqlite favorites tweets.db --stop_after=300
  Importing favorites  [####################################]  199
  $
  ```
  I don't _think_ this is a limitation of the API (if you omit `--stop_after` you get some very large number, possibly all of them), so I _think_ this is a bug.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/50/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTk2MjIwNDY=
title: datasette --pdb option for debugging errors
state: closed
created_at: 2020-09-11T18:33:10Z
updated_at: 2020-09-11T21:34:24Z
closed_at: 2020-09-11T18:38:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I needed to debug an exception from deep inside a Jinja template the other day. I hacked this together and it helped.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/962/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU2OTk5NDc1NzQ=
title: Currently selected array facets are not correctly persisted through hidden form fields
state: closed
created_at: 2020-09-12T01:49:17Z
updated_at: 2020-09-12T21:54:29Z
closed_at: 2020-09-12T21:54:09Z
author_association: NONE
type: issue
state_reason: completed
body:
  Faceted search uses JSON array elements as facets rather than the arrays. However, if a search is "Apply"ed (using the Apply button), the array itself rather than its elements used. 

  To reproduce:
  https://latest.datasette.io/fixtures/facetable?_sort=pk&_facet=created&_facet=tags&_facet_array=tags

  Press "Apply", which might be done when removing a filter. Notice that the "tags" facet values are now arrays, not array elements. It appears the "&_facet_array=tags" element of the query string is dropped.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/963/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDA3MjgyMTc=
title: raise_404 mechanism for custom templates
state: closed
created_at: 2020-09-14T03:22:15Z
updated_at: 2020-09-14T17:49:44Z
closed_at: 2020-09-14T17:39:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Having tried this out I think it does need a `raise_404()` mechanism - which needs to be smart enough to trigger the default 404 handler without accidentally going into an infinite loop.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/944#issuecomment-691788478_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/964/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDEyOTQ3Mjc=
title: Documentation for 404.html, 500.html templates
state: closed
created_at: 2020-09-14T17:36:59Z
updated_at: 2020-09-14T18:49:49Z
closed_at: 2020-09-14T18:47:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This mechanism is not documented: https://github.com/simonw/datasette/blob/30b98e4d2955073ca2bca92ca7b3d97fcd0191bf/datasette/app.py#L1119-L1129

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/965/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDE1ODQ0NDg=
title: Remove _request_ip example from canned queries documentation
state: closed
created_at: 2020-09-15T03:51:33Z
updated_at: 2020-09-15T03:52:45Z
closed_at: 2020-09-15T03:52:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `_request_ip` isn't valid, so it shouldn't be in the example: https://github.com/simonw/datasette/blob/cb515a9d75430adaf5e545a840bbc111648e8bfd/docs/sql_queries.rst#L320-L322

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/966/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDIwNjk0Mjk=
title: Writable canned queries with magic parameters fail if POST body is empty
state: closed
created_at: 2020-09-15T16:14:43Z
updated_at: 2020-09-15T20:13:10Z
closed_at: 2020-09-15T20:13:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  When I try to use the new `?_json=1` feature from #880 with magic parameters from #842 I get this error:

  > Incorrect number of bindings supplied. The current statement uses 1, and there are 0 supplied

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/967/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDIzODY5NDg=
title: .delete_where() does not auto-commit (unlike .insert() or .upsert())
state: open
created_at: 2020-09-16T01:55:52Z
updated_at: 2023-04-01T17:21:05Z
author_association: NONE
body:
  When you use the delete_where() function on a table, it never commits....

  Is that intentional?

type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/159/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDMyMTYwNDQ=
title: Feature: gists and starred gists
state: open
created_at: 2020-09-17T02:30:52Z
updated_at: 2020-09-17T02:30:52Z
author_association: MEMBER
body: https://developer.github.com/v3/gists/#list-starred-gists
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/49/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDMyMTg0NDg=
title: Documentation for twitter-to-sqlite fetch
state: open
created_at: 2020-09-17T02:38:10Z
updated_at: 2020-09-17T02:38:10Z
author_association: MEMBER
type: issue
body:
  It's mentioned in passing in the README but it deserves its own section:
  ```
  $ twitter-to-sqlite fetch \
      "https://api.twitter.com/1.1/account/verify_credentials.json" \
      | grep '"id"' | head -n 1
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/51/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDMyMTg3NTY=
title: Commands for making authenticated API calls
state: open
created_at: 2020-09-17T02:39:07Z
updated_at: 2020-10-19T05:01:29Z
author_association: MEMBER
body: Similar to `twitter-to-sqlite fetch`, see https://github.com/dogsheep/twitter-to-sqlite/issues/51
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDMyNDYwMzE=
title: github-to-sqlite should handle rate limits better
state: open
created_at: 2020-09-17T04:01:50Z
updated_at: 2022-10-14T16:34:07Z
author_association: MEMBER
type: issue
body:
  From #50 - right now it will crash with an error of it hits the rate limit. Since the rate limit information (including reset time) is available in the headers it could automatically sleep and try again instead.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDM5NTE5MTg=
title: Option to sort search results by date
state: closed
created_at: 2020-09-17T22:32:39Z
updated_at: 2020-09-17T22:55:35Z
closed_at: 2020-09-17T22:55:35Z
author_association: MEMBER
body: Sometimes I want to sort by date, not by relevance.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDM5NjI5MTc=
title: Bug: UI says sorted by relevance in timeline view
state: closed
created_at: 2020-09-17T23:02:07Z
updated_at: 2020-09-17T23:13:14Z
closed_at: 2020-09-17T23:13:14Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  In regular timeline view sort defaults to newest, not relevance - so this UI is incorrect:

  <img width="837" alt="Dogsheep_Beta_and_The_latest_news_from_the_Datasette_ecosystem_of_tools_-_Datasette_Newsletter_and_Ability_to_invite_people_to_a_team_who_don_t_have_acounts_yet_·_Issue__307_·_simonw_datasettecloud" src="https://user-images.githubusercontent.com/9599/93536956-1facf900-f8ff-11ea-889b-bc8356e366df.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDM5NzA3MTM=
title: Sort option should persist between multiple searches
state: closed
created_at: 2020-09-17T23:21:26Z
updated_at: 2020-09-18T22:39:12Z
closed_at: 2020-09-18T22:39:12Z
author_association: MEMBER
body: Following #21
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDM5NzA4MTQ=
title: the JSON object must be str, bytes or bytearray, not 'Undefined'
state: closed
created_at: 2020-09-17T23:21:41Z
updated_at: 2020-09-18T22:33:32Z
closed_at: 2020-09-18T22:33:32Z
author_association: MEMBER
body: Got this on a search results page.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDQ2ODU4OTA=
title: template_debug mechanism
state: closed
created_at: 2020-09-18T22:11:09Z
updated_at: 2020-09-18T22:12:21Z
closed_at: 2020-09-18T22:12:03Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  > I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.
  _Originally posted by @simonw in https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584_

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDUwNTc5NTU=
title: Add --tar option to "datasette publish heroku"
state: closed
created_at: 2020-09-20T06:54:53Z
updated_at: 2020-10-08T23:55:59Z
closed_at: 2020-10-08T23:30:59Z
author_association: NONE
type: issue
state_reason: completed
body:
  This issue is about how best to pass additional options to tools used for publishing datasettes. A concrete example is wanting to pass the `--tar` flag to the heroku CLI tool. I think there are at least two options for doing this: documentation for each publishing tool to explain how to set flags via env variables (if possible) or building a mechanism that lets users pass additional flags through datasette.

  When using `datasette publish heroku binder-launches.db --extra-options="--config facet_time_limit_ms:35000 --config sql_time_limit_ms:35000" --name=binderlytics --install=datasette-vega` to publish https://binderlytics.herokuapp.com/ the following error happens:

  ```
   ›   Warning: heroku update available from 7.42.1 to 7.43.0.
   ›   Warning: heroku update available from 7.42.1 to 7.43.0.
   ›   Warning: heroku update available from 7.42.1 to 7.43.0.
  Setting WEB_CONCURRENCY and restarting ⬢ binderlytics... done, v13
  WEB_CONCURRENCY: 1
   ›   Warning: heroku update available from 7.42.1 to 7.43.0.
   ▸    Couldn't detect GNU tar. Builds could fail due to decompression errors
   ▸    See https://devcenter.heroku.com/articles/platform-api-deploying-slugs#create-slug-archive
   ▸    Please install it, or specify the '--tar' option
   ▸    Falling back to node's built-in compressor
  buffer.js:358
      throw new ERR_INVALID_OPT_VALUE.RangeError('size', size);
      ^

  RangeError [ERR_INVALID_OPT_VALUE]: The value "3303763968" is invalid for option "size"
      at Function.alloc (buffer.js:367:3)
      at new Buffer (buffer.js:281:19)
      at Readable.<anonymous> (/Users/thead/.local/share/heroku/node_modules/archiver-utils/index.js:39:15)
      at Readable.emit (events.js:322:22)
      at endReadableNT (/Users/thead/.local/share/heroku/node_modules/readable-stream/lib/_stream_readable.js:1010:12)
      at processTicksAndRejections (internal/process/task_queues.js:84:21) {
    code: 'ERR_INVALID_OPT_VALUE'
  }
  ```

  After installing GNU tar with `brew install gnu-tar` and modifying `datasette/publish/heroku.py` to include the `--tar=/path/to/gnu-tar` publishing works.

  I think the problem occurs once your heroku slug reaches a certain size. At least when I add only a few 100 entries to the datasette then the error does not occcur.

  datasette version 0.49.1
  OSX 10.14.6 (18G103)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/969/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDUxMDg0OTI=
title: request an "-o" option on "datasette server" to open the default browser at the running url
state: closed
created_at: 2020-09-20T13:16:34Z
updated_at: 2020-10-08T23:54:27Z
closed_at: 2020-09-22T14:27:04Z
author_association: NONE
type: issue
state_reason: completed
body:
  This is a request for a "convenience" feature, and only a nice to have. It's based on seeing this feature in several little command line hypertext server apps.

  If you run, for example:

      datasette.exe serve --open "mydb.s3db" 

  I would like it if default browser is launched, at the URL that is being served.

  The angular cli does this, for example 

    ng serve <project> --open  #see https://angular.io/cli/serve

  ...as does my usual mini web server of choice when inspecting local static files....

    npx http-server -o # see https://www.npmjs.com/package/http-server

  Just a tiny thing. Love your work!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/970/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDUxOTA3MjM=
title: table.enable_fts(..., replace=True)
state: closed
created_at: 2020-09-20T21:36:23Z
updated_at: 2020-09-24T20:35:47Z
closed_at: 2020-09-20T22:05:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I noticed that https://til.simonwillison.net/ search doesn't use porter stemming. I'd like to add that, but since [the build script](https://github.com/simonw/til/blob/9d3f0fca30e94df3970df52b0447907a077e4673/build_database.py) always operates on an existing database (to avoid re-rendering markdown and re-building image thumbnails) I'd like it to only add porter stemming if it's not there already.

  So I'd like to be able to say "set up FTS to look like this, and fix it if it doesn't".

  I think the neatest way to do that is with a `replace=True` argument to `.enable_fts()`, for consistency with `def .create_view(self, name, sql, replace=True)`.

  So the `replace=True` argument would check and see if the configured FTS exists already with the correct options (columns, stemming, triggers) - and if any of those are incorrect it would call `.disable_fts()` and then create a new FTS configuration with the correct options.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/160/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDUyMTUyMzA=
title: Pagination
state: open
created_at: 2020-09-21T00:14:37Z
updated_at: 2020-09-21T02:55:54Z
author_association: MEMBER
type: issue
body:
  Useful for #16 (timeline view) since you can now filter to just the items on a specific day - but if there are more than 50 items you can't see them all.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDU4Mjc0NTc=
title: Support the dbstat table
state: closed
created_at: 2020-09-21T18:38:53Z
updated_at: 2020-09-21T19:00:02Z
closed_at: 2020-09-21T18:59:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `dbstat` is a table that is usually available on SQLite giving statistics about the database. For example:

  https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=SELECT+*+FROM+%22dbstat%22+WHERE+name%3D%27bachelorette%2Fbachelorette%27%3B

  | name                      | path   |   pageno | pagetype   |   ncell |   payload |   unused |   mx_payload |   pgoffset |   pgsize |
  |---------------------------|--------|----------|------------|---------|-----------|----------|--------------|------------|----------|
  | bachelorette/bachelorette | /      |       89 | internal   |      13 |         0 |     3981 |            0 |     360448 |     4096 |
  | bachelorette/bachelorette | /000/  |       91 | leaf       |      66 |      3792 |       32 |           74 |     368640 |     4096 |
  | bachelorette/bachelorette | /001/  |       92 | leaf       |      67 |      3800 |       14 |           74 |     372736 |     4096 |
  | bachelorette/bachelorette | /002/  |       93 | leaf       |      65 |      3717 |       46 |           70 |     376832 |     4096 |
  | bachelorette/bachelorette | /003/  |       94 | leaf       |      68 |      3742 |        6 |           71 |     380928 |     4096 |
  | bachelorette/bachelorette | /004/  |       95 | leaf       |      70 |      3696 |       42 |           66 |     385024 |     4096 |
  | bachelorette/bachelorette | /005/  |       96 | leaf       |      69 |      3721 |       22 |           71 |     389120 |     4096 |
  | bachelorette/bachelorette | /006/  |       97 | leaf       |      70 |      3737 |        1 |           72 |     393216 |     4096 |
  | bachelorette/bachelorette | /007/  |       98 | leaf       |      69 |      3728 |       15 |           69 |     397312 |     4096 |
  | bachelorette/bachelorette | /008/  |       99 | leaf       |      73 |      3715 |        8 |           64 |     401408 |     4096 |
  | bachelorette/bachelorette | /009/  |      100 | leaf       |      73 |      3705 |       18 |           62 |     405504 |     4096 |
  | bachelorette/bachelorette | /00a/  |      101 | leaf       |      75 |      3681 |       32 |           62 |     409600 |     4096 |
  | bachelorette/bachelorette | /00b/  |      102 | leaf       |      77 |      3694 |        9 |           62 |     413696 |     4096 |
  | bachelorette/bachelorette | /00c/  |      103 | leaf       |      74 |      3673 |       45 |           62 |     417792 |     4096 |
  | bachelorette/bachelorette | /00d/  |      104 | leaf       |       5 |       228 |     3835 |           48 |     421888 |     4096 |

  Other than direct `select * from dbsat` queries it is completely invisible.

  It would be cool if https://fivethirtyeight.datasettes.com/fivethirtyeight/dbstat didn't 404 (on databases for which that table was available).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/971/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDU4NDA2NzM=
title: Support faceting against arbitrary SQL queries
state: open
created_at: 2020-09-21T19:00:43Z
updated_at: 2021-12-15T18:02:20Z
author_association: OWNER
type: issue
body:
  > ... support for running facets against arbitrary custom SQL queries is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646
  > 
  > https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/971#issuecomment-696307922_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/972/reactions", "total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 3, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDkwNjA3OTQ5
title: table.transform() method
state: closed
created_at: 2020-09-21T23:16:59Z
updated_at: 2020-09-22T07:48:24Z
closed_at: 2020-09-22T04:20:02Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/161
type: pull
body:
  Refs #114

  - [x] Ability to change the primary key
  - [x] Support for changing default value for columns
  - [x] Support for changing `NOT NULL` status of columns
  - [x] Support for copying existing foreign keys and removing them
  - <strike>Support for `conversions=` parameter</strike>
  - [x] Detailed documentation
  - [x] `PRAGMA foreign_keys` stuff

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/161/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDU5OTU3MjI=
title: A decorator for registering custom SQL functions
state: closed
created_at: 2020-09-22T00:18:32Z
updated_at: 2020-09-22T00:40:44Z
closed_at: 2020-09-22T00:32:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Syntactic sugar for `db.conn.create_function` - it would work something like this:

  ```python
  db = sqlite_utils.Database("mydb.db")

  @db.register_function
  def scramble(text):
      chars = list(text)
      random.shuffle(chars)
      return "".join(chars)
  ```
  The decorator would inspect the function to find its name and arity (number of arguments). Having run the above you could then do:
  ```python
  db.execute("select scramble('hello')").fetchall()
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/162/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDYwMDE1MTc=
title: Idea: conversions= could take Python functions
state: open
created_at: 2020-09-22T00:37:12Z
updated_at: 2021-12-20T00:56:52Z
author_association: OWNER
type: issue
body:
  Right now you use `conversions=` like this:

  ```python
  db["example"].insert({
      "name": "The Bigfoot Discovery Museum"
  }, conversions={"name": "upper(?)"})
  ```
  How about if you could optionally provide a Python function (or a lambda) like this?
  ```python
  db["example"].insert({
      "name": "The Bigfoot Discovery Museum"
  }, conversions={"name": lambda s: s.upper()})
  ```
  This would work by creating a random name for that function, registering it (similar to #162), executing the SQL and then un-registering the custom function at the end.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/163/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDYwMTc0MTY=
title: sqlite-utils transform sub-command
state: closed
created_at: 2020-09-22T01:32:20Z
updated_at: 2020-09-24T20:34:50Z
closed_at: 2020-09-22T07:48:05Z
author_association: OWNER
body: The `.transform()` method in #114 warrants an equivalent CLI tool.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/164/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDYwOTEwNDY=
title: Make .transform() a keyword arguments only function
state: closed
created_at: 2020-09-22T05:37:29Z
updated_at: 2020-09-24T20:35:47Z
closed_at: 2020-09-22T06:39:12Z
author_association: OWNER
body: And rename the first argument from `columns=` to `types=`
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/165/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDkwNzAzMTcz
title: Keyword only arguments for transform()
state: closed
created_at: 2020-09-22T05:41:44Z
updated_at: 2020-09-22T06:39:11Z
closed_at: 2020-09-22T06:39:11Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/166
body: Refs #165
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/166/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDYwOTgwMDU=
title: Review the foreign key pragma stuff
state: closed
created_at: 2020-09-22T05:55:20Z
updated_at: 2020-09-23T00:13:02Z
closed_at: 2020-09-23T00:13:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > It is not possible to enable or disable foreign key constraints in the middle of a multi-statement transaction (when SQLite is not in autocommit mode). Attempting to do so does not return an error; it simply has no effect.

  https://sqlite.org/foreignkeys.html

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/167/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDYxNjc0NTY=
title: Automate (as much as possible) updates published to Homebrew
state: closed
created_at: 2020-09-22T08:08:37Z
updated_at: 2020-11-09T07:43:30Z
closed_at: 2020-11-09T07:43:30Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'd like to get new `sqlite-utils` (and Datasette) releases submitted to Homebrew as painlessly as possible.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/168/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDY0ODYzMjM=
title: 'bool' object is not callable error
state: closed
created_at: 2020-09-22T15:30:54Z
updated_at: 2020-10-08T23:54:32Z
closed_at: 2020-09-22T15:40:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm getting this when latest is deployed to Cloud Run:
  ```
  Traceback (most recent call last):
    File "/usr/local/bin/datasette", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/datasette/cli.py", line 406, in serve
      inspect_data = json.load(open(inspect_file))
  TypeError: 'bool' object is not callable
  ```
  I think I may have broken things in #970 - a980199e61fe7ccf02c2123849d86172d2ae54ff

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/973/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDY3NTc4OTE=
title: Progress bar for "sqlite-utils extract"
state: closed
created_at: 2020-09-22T23:40:21Z
updated_at: 2020-09-24T20:34:50Z
closed_at: 2020-09-23T00:02:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Since these operations could take a long time against large tables, it would be neat if there was a progress bar option for the CLI command.
  >
  > The operations are full table scans so calculating progress shouldn't be too difficult.
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513246831_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/169/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDY3Njg3OTg=
title: Release notes for 2.20
state: closed
created_at: 2020-09-23T00:13:22Z
updated_at: 2020-09-23T00:31:25Z
closed_at: 2020-09-23T00:31:25Z
author_association: OWNER
body: https://github.com/simonw/sqlite-utils/compare/2.19...b8e004
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/170/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDc0MDc1Njc=
title: Idea: transitive closure tables for tree structures
state: closed
created_at: 2020-09-23T14:17:33Z
updated_at: 2020-10-22T04:38:35Z
closed_at: 2020-10-22T04:07:14Z
author_association: NONE
type: issue
state_reason: completed
body:
  I just read that sqlite has a transitive closure table extension using a virtual table in order to represent trees:

  https://charlesleifer.com/blog/querying-tree-structures-in-sqlite-using-python-and-the-transitive-closure-extension/

  Even without this extension, though, a util to build a transitive closure table would allow trees to be queried easily. Since it relies on self-referential foreign keys, the relationships might even be able to be automatically detected. 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/171/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDc0MjcyMDA=
title: Improve performance of extract operations
state: closed
created_at: 2020-09-23T14:40:50Z
updated_at: 2020-09-24T15:43:57Z
closed_at: 2020-09-24T15:43:57Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This command took about 12 minutes (against a 150MB file with 680,000 rows in it):
  ```
  sqlite-utils extract salaries.db salaries \
     'Organization Group Code' 'Organization Group' \
    --table 'organization_groups' \
    --fk-column 'organization_group_id' \
    --rename 'Organization Group Code' code \
    --rename 'Organization Group' name
  ```
  I'm pretty confident we can do better than that.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/172/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDc0Nzg2NDk=
title: Progress bar for sqlite-utils insert
state: closed
created_at: 2020-09-23T15:43:56Z
updated_at: 2021-11-01T08:42:24Z
closed_at: 2020-10-27T18:16:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It would be nice if `sqlite-utils insert` had a progress bar, for when it's churning through huge CSV files.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/173/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDc4NDkxNzU=
title: static assets and favicon aren't cached by the browser
state: open
created_at: 2020-09-24T04:44:55Z
updated_at: 2022-01-13T22:21:28Z
author_association: NONE
type: issue
body:
  Using datasette to solve some frustrating problems with our fulfillment provider today, I was surprised to see repeated requests for assets under /-/static and the favicon.  While it won't likely be a huge performance bottleneck, I bet datasette would feel a bit zippier if you had Uvicorn serving up some caching-related headers telling the browser it was safe to cache static assets.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/974/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDkyMjU3NDA1
title: Much, much faster extract() implementation
state: closed
created_at: 2020-09-24T07:52:31Z
updated_at: 2020-09-24T15:44:00Z
closed_at: 2020-09-24T15:43:56Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/174
body: Takes my test down from ten minutes to four seconds. Refs #172.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/174/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDgxODU0MDU=
title: Dependabot couldn't authenticate with https://pypi.python.org/simple/
state: closed
created_at: 2020-09-24T13:44:40Z
updated_at: 2020-09-25T13:34:34Z
closed_at: 2020-09-25T13:34:34Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Dependabot couldn't authenticate with https://pypi.python.org/simple/.

  You can provide authentication details in your [Dependabot dashboard](https://app.dependabot.com/accounts/simonw) by clicking into the account menu (in the top right) and selecting 'Config variables'.

  [View the update logs](https://app.dependabot.com/accounts/simonw/update-logs/48611311).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/975/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDgyNjE3NzU=
title: Add docs for .transform(column_order=)
state: closed
created_at: 2020-09-24T15:19:04Z
updated_at: 2020-09-24T20:35:48Z
closed_at: 2020-09-24T16:00:56Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Need to update docs for `.transform()` now that `column_order=` is available.
  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/174#discussion_r494403327_

  Maybe also add this as an option to `sqlite-utils transform` - since reordering columns is actually a pretty nice capability.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/175/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDgyODk3ODM=
title: Idea: -o could open to a more convenient location
state: closed
created_at: 2020-09-24T15:56:35Z
updated_at: 2020-10-26T05:07:10Z
closed_at: 2020-10-26T05:06:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Idea: if a database only has a single table, this could open straight to `/db/table`. If it has multiple tables but a single database it could open straight to `/db`.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/970#issuecomment-698434236_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/976/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDgyOTMxMTQ=
title: sqlite-utils transform column order option
state: closed
created_at: 2020-09-24T16:01:21Z
updated_at: 2020-09-24T20:34:51Z
closed_at: 2020-09-24T16:11:59Z
author_association: OWNER
body: Split from #175
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/176/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDgzMDE4MTA=
title: Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key
state: closed
created_at: 2020-09-24T16:13:50Z
updated_at: 2020-09-24T20:35:03Z
closed_at: 2020-09-24T16:19:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  These both currently require you to provide three strings, for `column`, `other_table`, `other_column`.

  Just providing `column` should be enough information.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/177/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDkzMTYyNzY3
title: Update README.md
state: closed
created_at: 2020-09-25T15:52:11Z
updated_at: 2020-10-01T14:18:30Z
closed_at: 2020-09-30T20:29:28Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/178
type: pull
body:
  The `sqlite-utils insert releases.db releases - --pk` is missing the pk field name, added ` "id"` to fix it.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/178/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDk1Nzc2MjU=
title: sqlite-utils transform/insert --detect-types
state: closed
created_at: 2020-09-26T17:28:55Z
updated_at: 2021-06-19T03:36:16Z
closed_at: 2021-06-19T03:36:05Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Idea from https://github.com/simonw/datasette-edit-tables/issues/13 - provide Python utility methods and accompanying CLI options for detecting the likely types of TEXT columns.

  So if you have a text column that actually contained exclusively integer string values, it can let you know and let you run transform against it.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/179/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDk3ODk2MzQ=
title: Sort order is not persisted by facet filter links
state: open
created_at: 2020-09-27T18:22:07Z
updated_at: 2020-09-27T18:22:07Z
author_association: MEMBER
type: issue
body:
  A link to `/-/beta?category=1&timestamp__date=2018-08-01&q=swedish` should be to `/-/beta?category=1&timestamp__date=2018-08-01&q=swedish&sort=newest`

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDk4NjExOTQ=
title: Try running some tests using Hypothesis
state: closed
created_at: 2020-09-28T01:11:30Z
updated_at: 2020-10-19T04:51:55Z
closed_at: 2020-10-19T04:51:55Z
author_association: OWNER
body: Inspired by this Twitter conversation: https://twitter.com/simonw/status/1310386009465479168
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/180/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MDk5MjAwMjc=
title: pk=["id"] should have same effect as pk="id"
state: closed
created_at: 2020-09-28T04:28:07Z
updated_at: 2020-10-14T21:59:47Z
closed_at: 2020-10-14T21:59:47Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  In [11]: db['one'].insert({"id": 1, "name": "oentuh"}, pk="id")
  Out[11]: <Table one (id, name)>

  In [12]: db['two'].insert({"id": 1, "name": "oentuh"}, pk=["id"])
  Out[12]: <Table two (id, name)>

  In [13]: db['one'].schema
  Out[13]: 'CREATE TABLE [one] (\n   [id] INTEGER PRIMARY KEY,\n   [name] TEXT\n)'

  In [14]: db['two'].schema
  Out[14]: 'CREATE TABLE [two] (\n   [id] INTEGER,\n   [name] TEXT\n)'
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/181/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDk0MTQ2MDQz
title: Update pytest requirement from <6.1.0,>=5.2.2 to >=5.2.2,<6.2.0
state: closed
created_at: 2020-09-28T13:33:05Z
updated_at: 2020-09-28T22:16:36Z
closed_at: 2020-09-28T22:16:35Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/977
type: pull
body:
  Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p>
  <blockquote>
  <h2>6.1.0</h2>
  <h1>pytest 6.1.0 (2020-09-26)</h1>
  <h2>Breaking Changes</h2>
  <ul>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5585">#5585</a>: As per our policy, the following features which have been deprecated in the 5.X series are now
  removed:</p>
  <ul>
  <li>The <code>funcargnames</code> read-only property of <code>FixtureRequest</code>, <code>Metafunc</code>, and <code>Function</code> classes. Use <code>fixturenames</code> attribute.</li>
  <li><code>@pytest.fixture</code> no longer supports positional arguments, pass all arguments by keyword instead.</li>
  <li>Direct construction of <code>Node</code> subclasses now raise an error, use <code>from_parent</code> instead.</li>
  <li>The default value for <code>junit_family</code> has changed to <code>xunit2</code>. If you require the old format, add <code>junit_family=xunit1</code> to your configuration file.</li>
  <li>The <code>TerminalReporter</code> no longer has a <code>writer</code> attribute. Plugin authors may use the public functions of the <code>TerminalReporter</code> instead of accessing the <code>TerminalWriter</code> object directly.</li>
  <li>The <code>--result-log</code> option has been removed. Users are recommended to use the <a href="https://github.com/pytest-dev/pytest-reportlog">pytest-reportlog</a> plugin instead.</li>
  </ul>
  <p>For more information consult
  <a href="https://docs.pytest.org/en/stable/deprecations.html">Deprecations and Removals</a> in the docs.</p>
  </li>
  </ul>
  <h2>Deprecations</h2>
  <ul>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/6981">#6981</a>: The <code>pytest.collect</code> module is deprecated: all its names can be imported from <code>pytest</code> directly.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7097">#7097</a>: The <code>pytest._fillfuncargs</code> function is deprecated. This function was kept
  for backward compatibility with an older plugin.</p>
  <p>It's functionality is not meant to be used directly, but if you must replace
  it, use function._request._fillfixtures() instead, though note this is not
  a public API and may break in the future.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7210">#7210</a>: The special <code>-k '-expr'</code> syntax to <code>-k</code> is deprecated. Use <code>-k 'not expr'</code>
  instead.</p>
  <p>The special <code>-k 'expr:'</code> syntax to <code>-k</code> is deprecated. Please open an issue
  if you use this and want a replacement.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7255">#7255</a>: The pytest_warning_captured &lt;_pytest.hookspec.pytest_warning_captured&gt; hook is deprecated in favor
  of pytest_warning_recorded &lt;_pytest.hookspec.pytest_warning_recorded&gt;, and will be removed in a future version.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7648">#7648</a>: The <code>gethookproxy()</code> and <code>isinitpath()</code> methods of <code>FSCollector</code> and <code>Package</code> are deprecated;
  use <code>self.session.gethookproxy()</code> and <code>self.session.isinitpath()</code> instead.
  This should work on all pytest versions.</p>
  </li>
  </ul>
  <h2>Features</h2>
  <ul>
  <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7667">#7667</a>: New <code>--durations-min</code> command-line flag controls the minimal duration for inclusion in the slowest list of tests shown by <code>--durations</code>. Previously this was hard-coded to <code>0.005s</code>.</li>
  </ul>
  <h2>Improvements</h2>
  <!-- raw HTML omitted -->
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst">pytest's changelog</a>.</em></p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest/commit/868bc003ecb0af65548c14de997205cc0e37c3c6"><code>868bc00</code></a> Prepare release version 6.1.0</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/0b327cc75b5f7452eb80c604ce51c032da316a4b"><code>0b327cc</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7796">#7796</a> from bluetech/changelog-cleanups</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/d3c746eb8eb96e68e1c2624a46d99b9e60d455ec"><code>d3c746e</code></a> changelog: some consistency cleanups</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/d3f47bf3468c964d4e41258c55db6577d99c7992"><code>d3f47bf</code></a> Improved 'Declaring new hooks' section in docs. (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7782">#7782</a>)</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/3db24893b4d6ba44d6d94a836cedb9d21724916b"><code>3db2489</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7784">#7784</a> from nicoddemus/use-new-pip-solver-7783</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/821562513526dcd439b253f25677cc2c998d8fef"><code>8215625</code></a> Use new pip resolver in plugins tox env</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/5cfd7c0ddd5b1ffc111399e2d118b243896827b0"><code>5cfd7c0</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7780">#7780</a> from bluetech/final</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/a99ca879e7c9db0ac91324e701275e9439cf7b73"><code>a99ca87</code></a> Mark some public and to-be-public classes as <code>@final</code></li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/050c2df737f7c6c3b1de7a6f375de107edcb6873"><code>050c2df</code></a> Use multiple issue template types and mention Discussions (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7739">#7739</a>)</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/cdfdb3a25d1033f732072d9a4d2fb700feb73a09"><code>cdfdb3a</code></a> Add docs about reusing fixtures from other projects (<a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7772">#7772</a>)</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest/compare/5.2.2...6.1.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/977/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTA1MDY3MDg=
title: Rendering glitch with column headings on mobile
state: closed
created_at: 2020-09-28T19:04:45Z
updated_at: 2020-10-08T23:54:40Z
closed_at: 2020-09-28T22:43:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ![6348FED5-1310-47DA-BF54-810E63C96C6F](https://user-images.githubusercontent.com/9599/94474607-bc368d00-0182-11eb-9bb1-9a0e455353f1.jpeg)

  https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_parse%28%2210+october+2020+3pm%22%29%2C%0D%0A++dateutil_easter%28%222020%22%29%2C%0D%0A++dateutil_parse_fuzzy%28%22This+is+due+10+september%22%29%2C%0D%0A++dateutil_parse%28%221%2F2%2F2020%22%29%2C%0D%0A++dateutil_parse%28%222020-03-04%22%29%2C%0D%0A++dateutil_parse_dayfirst%28%222020-03-04%22%29%2C%0D%0A++dateutil_easter%282020%29

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/978/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTA2NTA2MzM=
title: Default table view JSON should include CREATE TABLE
state: closed
created_at: 2020-09-28T23:54:58Z
updated_at: 2023-01-09T15:32:39Z
closed_at: 2023-01-09T15:32:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://latest.datasette.io/fixtures/facetable.json doesn't currently include the CREATE TABLE statement for the page, even though it's available on the HTML version at https://latest.datasette.io/fixtures/facetable

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/979/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTA4MTkwMjA=
title: Another rendering glitch with column headers on mobile
state: closed
created_at: 2020-09-29T06:53:13Z
updated_at: 2020-10-08T23:54:49Z
closed_at: 2020-09-29T19:21:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Similar to #978.

  https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule(%27FREQ%3DHOURLY%3BCOUNT%3D5%27)%2C%0D%0A++dateutil_rrule_date(%0D%0A++++%27FREQ%3DDAILY%3BCOUNT%3D3%27%2C%0D%0A++++%271st+jan+2020%27%0D%0A++)%3B

  <img width="452" alt="fixtures__select_dateutil_rrule__FREQ_HOURLY_COUNT_5____dateutil_rrule_date___FREQ_DAILY_COUNT_3____1st_jan_2020____" src="https://user-images.githubusercontent.com/9599/94523237-c0e05d00-01e5-11eb-880d-5535f43f07a5.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/980/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTE2Mjc2Mjg=
title: Action menu for table columns
state: closed
created_at: 2020-09-30T04:45:38Z
updated_at: 2020-10-08T23:55:00Z
closed_at: 2020-09-30T23:58:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  At the very least I'd like a menu on each table column that lets me select sort-asc v.s. sort-desc without having to click twice.

  I'd also like to be able to indicate that a column should be used for faceting (possibly only for columns that are not floating point and do not have a unique index on them).

  This needs to be built with accessibility in mind - I don't want screenreaders to read out the contents of a menu as the "th" label for any given cell.

  Related: #690

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/981/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTE2NDkzMjU=
title: Better handling of encodings other than utf-8 for "sqlite-utils insert"
state: closed
created_at: 2020-09-30T05:43:48Z
updated_at: 2020-10-16T17:20:41Z
closed_at: 2020-10-16T17:18:52Z
author_association: NONE
type: issue
state_reason: completed
body:
  Makefile:
  ```
  data.db:
          curl -O http://maps.natalian.org/data.txt
          go run csv-write.go > data.csv
          sqlite-utils insert data.db travels data.csv --csv

  clean:
          rm data*
  ```
  [csv-write.go](https://gist.github.com/kaihendry/dff2442de20d73f900026d13bf7a11d9)


  Error message is:

  ```
  sqlite-utils insert data.db travels data.csv --csv
  Traceback (most recent call last):
    File "/home/hendry/.local/bin/sqlite-utils", line 8, in <module>
      sys.exit(cli())
    File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/home/hendry/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 614, in insert
      insert_upsert_implementation(
    File "/home/hendry/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 553, in insert_upsert_implementation
      headers = next(reader)
    File "/usr/lib/python3.8/codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe3 in position 1234: invalid continuation byte
  make: *** [Makefile:4: data.db] Error 1
  [hendry@t14s datasette-map]$ sqlite-utils --version
  sqlite-utils, version 2.19
  ```

  Little bit surprised if Go is spewing out bad Unicode, but I'm not sure how to grok `position 1234`..

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/182/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTIyMDIzMzM=
title: SQL editor should allow execution of write queries, if you have permission
state: open
created_at: 2020-09-30T19:04:35Z
updated_at: 2022-01-13T22:21:29Z
author_association: OWNER
type: issue
body:
  The `datasette-write` plugin provides this at the moment https://github.com/simonw/datasette-write - but it feels like it should be a built-in capability, protected by a default permission.

  UI concept: if you have write permission then the existing SQL editor gets an "execute write" checkbox underneath it.

  JavaScript can spot if you appear to be trying to execute an UPDATE or INSERT or DELETE query and check that checkbox for you.

  If you link to a query page with a non-SELECT then that query will be displayed in the box ready for you to POST submit it. The page will also then get "cannot be embedded" headers to protect against clickjacking.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/982/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTIyNjA0Mjk=
title: JavaScript plugin hooks mechanism similar to pluggy
state: open
created_at: 2020-09-30T20:32:43Z
updated_at: 2021-01-25T04:43:58Z
author_association: OWNER
type: issue
body:
  > It would be neat to provide a JavaScript plugin hook that plugins can use to add their own options to this menu. No idea what that would look like though.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/981#issuecomment-701616922_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/983/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTIzMTY5NTk=
title: Try out GitHub code scanning
state: closed
created_at: 2020-09-30T22:16:14Z
updated_at: 2020-09-30T22:23:44Z
closed_at: 2020-09-30T22:23:44Z
author_association: OWNER
body: https://github.blog/2020-09-30-code-scanning-is-now-available/
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/183/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTIzNjg0MzI=
title: Review accessibility of new column action menus
state: open
created_at: 2020-09-30T23:56:44Z
updated_at: 2020-10-01T00:01:36Z
author_association: OWNER
body: Feature added in #981
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/984/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTI4MzkzODM=
title: Column actions should support facet by compound primary keys
state: closed
created_at: 2020-10-01T13:21:57Z
updated_at: 2020-10-08T23:55:11Z
closed_at: 2020-10-01T16:50:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  On https://latest.datasette.io/fixtures/compound_three_primary_keys the column action menu doesn't display for the pk1, pk2 and pk3 columns (because they are primary keys) even though faceting by them is actually useful.

  Refs #98

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/985/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDk2Mjk4MTgw
title: Allow facet by primary keys, fixes #985
state: closed
created_at: 2020-10-01T14:18:55Z
updated_at: 2020-10-01T16:51:45Z
closed_at: 2020-10-01T16:51:45Z
author_association: NONE
pull_request: simonw/datasette/pulls/986
type: pull
body:
  Hello! This PR makes it possible to facet by primary keys.
  Did I get it right that just removing the condition on UI side is enough? From testing it works fine with primary keys, just as with normal keys.
  If so, should I also remove unused `data-is-pk`?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/986/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTI5ODQ3Mzg=
title: Documented HTML hooks for JavaScript plugin authors
state: open
created_at: 2020-10-01T16:10:14Z
updated_at: 2021-01-25T04:00:03Z
author_association: OWNER
type: issue
body:
  In #981 I added `data-column=` attributes to the `<th>` on the table page. These should become part of Datasette's documented API so JavaScript plugin authors can use them to derive things about the tables shown on a page (`datasette-cluster-map uses them as-of https://github.com/simonw/datasette-cluster-map/issues/18).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/987/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTMyMDk0MDQ=
title: Mechanism for plugins to construct URLs that respect base_url
state: closed
created_at: 2020-10-01T21:54:15Z
updated_at: 2020-10-23T19:44:05Z
closed_at: 2020-10-15T23:01:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Had a thought: this is likely to break in plugins too, such as `datasette-edit-schema` which constructs URLs for redirects e.g. here: https://github.com/simonw/datasette-edit-schema/blob/dbd0abee6dd3385b114cfe9671f7ead1c4855b60/datasette_edit_schema/__init__.py#L46-L48

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/865#issuecomment-702418045_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/988/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTMzMDQ0MTc=
title: Column action sort descending/ascending links should remove _next= pagination
state: closed
created_at: 2020-10-02T02:33:48Z
updated_at: 2020-10-08T23:55:15Z
closed_at: 2020-10-04T18:05:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  On page https://latest.datasette.io/fixtures/sortable?_next=15%2Cg%2Cz&_sort=sortable clicking on `sortable_with_nulls > sort_ascending` links to https://latest.datasette.io/fixtures/sortable?_next=15%2Cg%2Cz&_sort_desc=sortable_with_nulls - which doesn't make sense.

  Changing the sort order needs to reset to the first page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/989/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTQzNzcyNjg=
title: Redesign application homepage
state: open
created_at: 2020-10-04T18:48:45Z
updated_at: 2021-01-26T19:06:36Z
author_association: OWNER
type: issue
body:
  Most Datasette instances only host a single database, but the current homepage design assumes that it should leave plenty of space for multiple databases:

  <img width="878" alt="Datasette_Fixtures__fixtures" src="https://user-images.githubusercontent.com/9599/95024344-5b51fd80-0637-11eb-8a11-40bad16f6907.png">

  Reconsider this design - should the default show more information?

  The Covid-19 Datasette homepage looks particularly sparse I think: https://covid-19.datasettes.com/

  <img width="782" alt="COVID-19_cases__using_data_from_Johns_Hopkins_CSSE__the_New_York_Times_and_the_LA_Times__covid" src="https://user-images.githubusercontent.com/9599/95024391-876d7e80-0637-11eb-8f19-ef38e4c87d2a.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/991/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTQ0NDk4Nzk=
title: Change "--config foo:bar" to "--setting foo bar"
state: closed
created_at: 2020-10-05T01:27:45Z
updated_at: 2020-11-24T20:01:54Z
closed_at: 2020-11-24T20:01:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I designed the config format before I had a good feel for CLI design using Click. `--config max_page_size 2000` is better than `--config max_page_size:2000`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/992/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTUwNzI5MzU=
title: Column action menu should show column type
state: closed
created_at: 2020-10-05T18:40:49Z
updated_at: 2020-10-08T23:55:19Z
closed_at: 2020-10-06T00:33:15Z
author_association: OWNER
type: issue
state_reason: completed
body:
  <img width="616" alt="data__rent_burden_households_and_median_gross_rent_percent__97_rows" src="https://user-images.githubusercontent.com/9599/95118702-97509580-06ff-11eb-8523-39eb1b389acd.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/993/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDk4MTA0ODMw
title: Run tests against Python 3.9
state: closed
created_at: 2020-10-05T20:40:13Z
updated_at: 2020-10-09T16:22:51Z
closed_at: 2020-10-09T16:22:50Z
author_association: OWNER
pull_request: simonw/datasette/pulls/994
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/994/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDk4NjMwNzA4
title: Document setting Google Cloud SDK properties
state: closed
created_at: 2020-10-06T15:18:01Z
updated_at: 2020-10-08T23:55:30Z
closed_at: 2020-10-06T16:25:38Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/995
type: pull
body:
  Document setting Google Cloud SDK properties to avoid having to respond to interactive prompts when running `datasette publish cloudrun`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/995/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTY3NTYwODI=
title: Better handling of multiple matching template wildcard paths
state: closed
created_at: 2020-10-07T18:25:40Z
updated_at: 2020-10-08T23:55:41Z
closed_at: 2020-10-07T22:51:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I tried building this:

      templates/pages/{topic}.html
      templates/pages/{topic}/{slug}.html

  And it didn't work - hits to /foo/bar which should have been rendered by the `{slug}.html` template were instead rendered by the top level `{topic.html}` template.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/996/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NDk5NjAzMzU5
title: Test against Python 3.9
state: closed
created_at: 2020-10-08T01:37:05Z
updated_at: 2020-10-08T01:44:06Z
closed_at: 2020-10-08T01:44:06Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/184
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/184/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTY5ODg0Nzg=
title: Documentation covering buildpack deployment
state: closed
created_at: 2020-10-08T03:21:52Z
updated_at: 2020-10-08T23:56:03Z
closed_at: 2020-10-08T23:32:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  A tidied up version of https://til.simonwillison.net/til/til/digitalocean_datasette-on-digitalocean-app-platform.md - but mention that you can deploy to Heroku using the same mechanism.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/997/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTc2OTk4ODQ=
title: Wide tables should scroll horizontally within the page
state: closed
created_at: 2020-10-08T22:13:27Z
updated_at: 2020-12-11T09:25:09Z
closed_at: 2020-10-22T01:12:26Z
author_association: OWNER
body: Wrap the main table in `<div style="overflow-x: auto">`
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/998/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTc3MjkwNTY=
title: Datasette should default to running Uvicorn with workers=1
state: closed
created_at: 2020-10-08T23:07:03Z
updated_at: 2020-10-08T23:55:46Z
closed_at: 2020-10-08T23:21:36Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Uvicorn uses the `WEB_CONCURRENCY` variable, if set, to specify the number of workers to use.

  Datasette does not work with options for this other than 1:

  ```
  WEB_CONCURRENCY=2 datasette .
  WARNING:  You must pass the application as an import string to enable 'reload' or 'workers'.
  ```
  This was the cause of the Heroku bug in #627.

  I fixed that issue by setting `WEB_CONCURRENCY=1` in `datasette publish heroku`, but a better fix would be to hard-code `workers=1` in Datasette itself.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/999/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTAwMjU2NDg1
title: datasette.client internal requests mechanism
state: closed
created_at: 2020-10-08T23:58:25Z
updated_at: 2020-10-09T16:11:26Z
closed_at: 2020-10-09T16:11:25Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1000
body: Refs #943
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1000/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTc3Njg0NDE=
title: OPTIONS requests return a 500 error
state: closed
created_at: 2020-10-09T00:57:13Z
updated_at: 2020-10-09T01:44:41Z
closed_at: 2020-10-09T01:43:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  % curl -vv -XOPTIONS https://latest.datasette.io/
  *   Trying 216.58.195.83:443...
  > OPTIONS / HTTP/1.1
  > Host: latest.datasette.io
  > User-Agent: curl/7.70.0
  > Accept: */*
  > 
  * Mark bundle as not supporting multiuse
  < HTTP/1.1 500 Internal Server Error
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1001/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTc3ODM2OTI=
title: Release notes for Datasette 0.50
state: closed
created_at: 2020-10-09T01:45:00Z
updated_at: 2020-10-09T17:52:54Z
closed_at: 2020-10-09T17:52:53Z
author_association: OWNER
body: https://github.com/simonw/datasette/compare/0.49.1...c12b7a5def7028845a54a9fdac4052a87a0a8bb8
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1002/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTgyMzg5Njc=
title: from_json jinja2 filter
state: open
created_at: 2020-10-09T15:30:58Z
updated_at: 2020-10-09T17:17:07Z
author_association: NONE
type: issue
body:
  When JSON fields are rendered in a jinja2 template, it is handy to be able to manipulate them as data (e.g., iterate over an array of values). 

  Ansible has a "from_json" function, which just called json.loads. It's a trivial as a datasette plugin, but it seems generally useful. Does it makes sense to add it directly into the app?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1003/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTgyNTU4MDM=
title: Replace MockRequest with Request.fake()
state: closed
created_at: 2020-10-09T15:55:28Z
updated_at: 2020-10-09T16:26:24Z
closed_at: 2020-10-09T16:26:24Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This code: https://github.com/simonw/datasette/blob/7249ac5ca04b5ddc6517750326ee7e522cc49145/tests/utils.py#L1-L8

  Predates the introduction of this class method: https://github.com/simonw/datasette/blob/7249ac5ca04b5ddc6517750326ee7e522cc49145/datasette/utils/asgi.py#L108-L121

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1004/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTgyNTkyMDI=
title: Remove xfail tests when new httpx is released
state: closed
created_at: 2020-10-09T16:00:19Z
updated_at: 2021-02-28T22:41:08Z
closed_at: 2021-02-28T22:41:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.
  >
  > I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing.
  >
  _Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706263157_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1005/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTgyNjQ4MTE=
title: Documentation for datasette.client
state: closed
created_at: 2020-10-09T16:09:02Z
updated_at: 2020-10-09T17:22:31Z
closed_at: 2020-10-09T17:20:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'm going to document this in a separate issue.

  _Originally posted by @simonw in https://github.com/simonw/datasette/pull/1000#issuecomment-706269271_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1006/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTgyNzI1OTM=
title: set-env and add-path commands have been deprecated
state: open
created_at: 2020-10-09T16:21:18Z
updated_at: 2020-10-09T16:23:51Z
author_association: OWNER
type: issue
body:
  https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/

  > Starting today runner version 2.273.5 will begin to warn you if you use the `add-path` or `set-env` commands. We are monitoring telemetry for the usage of these commands and plan to fully disable them in the future.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1007/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTAwNzk4MDkx
title: Add json_loads and json_dumps jinja2 filters
state: open
created_at: 2020-10-09T20:11:34Z
updated_at: 2020-12-15T02:30:28Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/datasette/pulls/1008
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1008/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg0ODQwODI=
title: json / CSV links are broken in Datasette 0.50
state: closed
created_at: 2020-10-10T00:07:42Z
updated_at: 2020-10-10T02:32:03Z
closed_at: 2020-10-10T02:32:03Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. on https://latest.datasette.io/fixtures/sortable

  That export link block is  broken. The HTML is:

  ```html
  <p class="export-links">
    This data as <a href="//fixtures/sortable.json">json</a>,
    <a href="//fixtures/sortable.csv?_size=max">CSV</a> (<a href="#export">advanced</a>)
  </p>
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1010/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg1MjE0Njk=
title: column name links broken in 0.50.1
state: closed
created_at: 2020-10-10T03:37:51Z
updated_at: 2020-10-10T04:09:32Z
closed_at: 2020-10-10T03:52:07Z
author_association: NONE
type: issue
state_reason: completed
body:
  I just upgraded from 0.49 to 0.50.1 and found that the links on column headers are broken. 

  If I inspect the source, they have a leading "//" (without host or port) rather than including base_url like other links on the page do. The links in the "gears" menu for each column do work.

   I don't have custom templates for my project. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1011/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg1NDA3NTE=
title: For 1.0 update trove classifier in setup.py
state: open
created_at: 2020-10-10T05:52:08Z
updated_at: 2021-11-16T13:18:36Z
author_association: OWNER
body: Development Status :: 5 - Production/Stable
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1012/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg3MjM1NDM=
title: Add Link: pagination HTTP headers
state: closed
created_at: 2020-10-10T23:42:40Z
updated_at: 2020-10-23T19:44:05Z
closed_at: 2020-10-11T00:18:51Z
author_association: OWNER
body: Spun off from #782. These can go on all of the JSON endpoints that support pagination.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1014/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5MTAzMTg=
title: Research: could Datasette install its own plugins?
state: open
created_at: 2020-10-11T19:33:06Z
updated_at: 2020-10-11T19:35:04Z
author_association: OWNER
type: issue
body:
  It would be cool if Datasette could offer a plugin browsing interface where users could install plugins by clicking "Install" on them - similar to how VS Code extensions work.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1015/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5MzQ5NDI=
title: Documentation on how to use this with Datasette
state: open
created_at: 2020-10-11T21:56:27Z
updated_at: 2020-10-11T22:14:00Z
author_association: MEMBER
body: In particular how to use `datasette-render-images` to see the images.
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5MzgwNDY=
title: Convert dates to a better format
state: closed
created_at: 2020-10-11T22:12:33Z
updated_at: 2020-10-11T23:15:03Z
closed_at: 2020-10-11T23:15:03Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  They currently look like this:

  https://github.com/dogsheep/evernote-to-sqlite/blob/9d8efd17580f6ddf76745c145d1e69dd24e52b64/tests/test_evernote_to_sqlite.py#L35-L36

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5MzgzMjE=
title: Use a content hash for the note IDs
state: closed
created_at: 2020-10-11T22:13:46Z
updated_at: 2020-10-11T23:15:04Z
closed_at: 2020-10-11T23:15:04Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Without a GUID note IDs are pretty ineffective, but using a hash of the contents will at least avoid creating identical duplicates in the future.

  https://sqlite-utils.readthedocs.io/en/stable/python-api.html#setting-an-id-based-on-the-hash-of-the-row-contents

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5Mzg1MDg=
title: Configure FTS + add an index on the date columns
state: closed
created_at: 2020-10-11T22:14:40Z
updated_at: 2020-10-11T23:41:29Z
closed_at: 2020-10-11T23:41:29Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Sort by date descending is likely the most common way of sorting, so that column should be indexed.

  Also add FTS configuration for both notes and the OCR column on resources.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5Mzg4ODk=
title: Figure out how to display images from <en-media> tags inline in Datasette
state: open
created_at: 2020-10-11T22:17:03Z
updated_at: 2020-10-16T20:16:28Z
author_association: MEMBER
type: issue
body:
  Relates to #1. Evernote XML looks like this:

  ```xml
  <?xml version="1.0"?>
  <en-note>
    <div>This note includes two images.</div>
    <div>
      <b>The Python logo</b>
    </div>
    <div>
      <en-media hash="61098c2c541de7f0a907c301dd6542da" type="image/svg+xml" width="125"/>
    </div>
    <div>
      <b>The Evernote logo</b>
    </div>
    <div>
      <en-media hash="91bd26175acac0b2ffdb6efac199f8ca" type="image/svg+xml" width="125"/>
    </div>
  </en-note>
  ```
  That hash is the md5 we use to store resources. It should be possible to turn these into embedded image tags, especially if done in conjunction with the https://github.com/simonw/datasette-media plugin.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5NDkxODI=
title: Better handling of OCR data
state: closed
created_at: 2020-10-11T23:20:52Z
updated_at: 2020-10-12T00:04:10Z
closed_at: 2020-10-12T00:04:10Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  > I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.

  _Originally posted by @simonw in https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028_

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5NTIxMDc=
title: Use db[table] consistently in documentation
state: closed
created_at: 2020-10-11T23:39:12Z
updated_at: 2020-10-12T00:13:41Z
closed_at: 2020-10-12T00:13:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The Python docs have a bunch of examples like this: https://sqlite-utils.readthedocs.io/en/stable/python-api.html
  ```python
  dogs.enable_fts(["name", "twitter"], create_triggers=True)
  ```
  This would be easier for people to understand if it looked like this instead:
  ```python
  db["dogs"].enable_fts(["name", "twitter"], create_triggers=True)
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/185/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MTg5NTM2Njk=
title: Add a "delete" icon next to filters (in addition to "remove filter")
state: closed
created_at: 2020-10-11T23:49:53Z
updated_at: 2020-10-23T19:44:06Z
closed_at: 2020-10-12T03:01:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The "remove filter" option in the select box is not very discoverable.

  <img width="549" alt="evernote__notes__85_rows_where_where_source-url_is_not_blank_sorted_by_id" src="https://user-images.githubusercontent.com/9599/95693532-b8265880-0be1-11eb-971e-038dede14cdb.png">

  It would be good to have an additional remove icon, pointed to by the pink arrow, which removes a specific selected filter.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1016/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTAxNTc5MDg4
title: Update janus requirement from <0.6,>=0.4 to >=0.4,<0.7
state: closed
created_at: 2020-10-12T13:29:46Z
updated_at: 2020-10-14T21:52:08Z
closed_at: 2020-10-14T21:52:07Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1017
type: pull
body:
  Updates the requirements on [janus](https://github.com/aio-libs/janus) to permit the latest version.
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/aio-libs/janus/blob/master/CHANGES.rst">janus's changelog</a>.</em></p>
  <blockquote>
  <h1>Changes</h1>
  <h2>0.5.0 (2020-04-23)</h2>
  <ul>
  <li>Remove explicit loop arguments and forbid creating queues outside event loops <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/246">#246</a></li>
  </ul>
  <h2>0.4.0 (2018-07-28)</h2>
  <ul>
  <li>Add <code>py.typed</code> macro <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/89">#89</a></li>
  <li>Drop python 3.4 support and fix minimal version python3.5.3 <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/88">#88</a></li>
  <li>Add property with that indicates if queue is closed <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/86">#86</a></li>
  </ul>
  <h2>0.3.2 (2018-07-06)</h2>
  <ul>
  <li>Fixed python 3.7 support <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/97">#97</a></li>
  </ul>
  <h2>0.3.1 (2018-01-30)</h2>
  <ul>
  <li>Fixed bug with join() in case tasks are added by sync_q.put() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/75">#75</a></li>
  </ul>
  <h2>0.3.0 (2017-02-21)</h2>
  <ul>
  <li>Expose unfinished_tasks property <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/34">#34</a></li>
  </ul>
  <h2>0.2.4 (2016-12-05)</h2>
  <ul>
  <li>Restore tarball deploying</li>
  </ul>
  <h2>0.2.3 (2016-07-12)</h2>
  <ul>
  <li>Fix exception type</li>
  </ul>
  <h2>0.2.2 (2016-07-11)</h2>
  <ul>
  <li>Update asyncio.async() to use asyncio.ensure_future() <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/6">#6</a></li>
  </ul>
  <h2>0.2.1 (2016-03-24)</h2>
  <ul>
  <li>Fix python setup.py test command <a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/4">#4</a></li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/aio-libs/janus/commit/d1867245cc803c2b9540b252928343415c16a1d7"><code>d186724</code></a> Fix yaml</li>
  <li><a href="https://github.com/aio-libs/janus/commit/dbb2d7bcd5a3b916c12ade55a9af3a27ff44ec50"><code>dbb2d7b</code></a> Fix deploy script</li>
  <li><a href="https://github.com/aio-libs/janus/commit/18df625c1c8ce0876db57370dea43c623657b8ba"><code>18df625</code></a> Bump to 0.6.0</li>
  <li><a href="https://github.com/aio-libs/janus/commit/a50b7ec0b9bf04d5c86a55fd255969984f10d93c"><code>a50b7ec</code></a> Test on ubuntu only, the library has no platform specific dependencies</li>
  <li><a href="https://github.com/aio-libs/janus/commit/b599d94fbfd17c179feea025b7c068f7454b968a"><code>b599d94</code></a> Fix workflow</li>
  <li><a href="https://github.com/aio-libs/janus/commit/9897fca5e2196c8036324c313f486616850c3eb2"><code>9897fca</code></a> Setup github workflows</li>
  <li><a href="https://github.com/aio-libs/janus/commit/cde69186efea8ae76f522ea0655070f110bffacf"><code>cde6918</code></a> Drop Python 3.5, test on Python 3.9, format with black/isort</li>
  <li><a href="https://github.com/aio-libs/janus/commit/5f04d79a6a51bd285bba10494f5e0ed2196339d7"><code>5f04d79</code></a> Support Python 3.9 officially</li>
  <li><a href="https://github.com/aio-libs/janus/commit/ac23eb74d2a15286449de6cafaa854086550394a"><code>ac23eb7</code></a> janus: remove unused type ignores (<a href="https://github-redirect.dependabot.com/aio-libs/janus/issues/287">#287</a>)</li>
  <li><a href="https://github.com/aio-libs/janus/commit/0da8f954ad87634093588163c2a40d7400882afb"><code>0da8f95</code></a> Make all tests non-skipped again</li>
  <li>Additional commits viewable in <a href="https://github.com/aio-libs/janus/compare/v0.4.0...v0.6.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1017/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTAxNTc5MzE1
title: Update asgiref requirement from ~=3.2.10 to >=3.2.10,<3.4.0
state: closed
created_at: 2020-10-12T13:30:09Z
updated_at: 2020-10-14T21:51:36Z
closed_at: 2020-10-14T21:51:35Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1018
type: pull
body:
  Updates the requirements on [asgiref](https://github.com/django/asgiref) to permit the latest version.
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/django/asgiref/blob/master/CHANGELOG.txt">asgiref's changelog</a>.</em></p>
  <blockquote>
  <h2>3.3.0 (2020-10-09)</h2>
  <ul>
  <li>sync_to_async now defaults to thread-sensitive mode being on</li>
  <li>async_to_sync now works inside of forked processes</li>
  <li>WsgiToAsgi now correctly clamps its response body when Content-Length is set</li>
  </ul>
  <h2>3.2.10 (2020-08-18)</h2>
  <ul>
  <li>Fixed bugs due to bad WeakRef handling introduced in 3.2.8</li>
  </ul>
  <h2>3.2.9 (2020-06-16)</h2>
  <ul>
  <li>Fixed regression with exception handling in 3.2.8 related to the contextvars fix.</li>
  </ul>
  <h2>3.2.8 (2020-06-15)</h2>
  <ul>
  <li>Fixed small memory leak in local.Local</li>
  <li>contextvars are now persisted through AsyncToSync</li>
  </ul>
  <h2>3.2.7 (2020-03-24)</h2>
  <ul>
  <li>Bug fixed in local.Local where deleted Locals would occasionally inherit
  their storage into new Locals due to memory reuse.</li>
  </ul>
  <h2>3.2.6 (2020-03-23)</h2>
  <ul>
  <li>local.Local now works in all threading situations, no longer requires
  periodic garbage collection, and works with libraries that monkeypatch
  threading (like gevent)</li>
  </ul>
  <h2>3.2.5 (2020-03-11)</h2>
  <ul>
  <li><strong>self</strong> is now preserved on methods by async_to_sync</li>
  </ul>
  <h2>3.2.4 (2020-03-10)</h2>
  <!-- raw HTML omitted -->
  </blockquote>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/django/asgiref/commit/7dba5ff61313dfd1535de200590b9155abde20f0"><code>7dba5ff</code></a> Releasing 3.3.0</li>
  <li><a href="https://github.com/django/asgiref/commit/e1e0dd9a900966b29345e6069c0fdbc9a449e97f"><code>e1e0dd9</code></a> Added ZeroCopy extension</li>
  <li><a href="https://github.com/django/asgiref/commit/3834d13a8f5da921bb5f8ca16ee66d353fdbdadb"><code>3834d13</code></a> Added rpc.py to Implementations (<a href="https://github-redirect.dependabot.com/django/asgiref/issues/198">#198</a>)</li>
  <li><a href="https://github.com/django/asgiref/commit/03b0dbb518e3bbf6c4083ae1e51bd6e2062fa503"><code>03b0dbb</code></a> Clamped WsgiToAsgi response body using Content-Length value</li>
  <li><a href="https://github.com/django/asgiref/commit/cfd82e48a2059ff93ce20b94e1755c455e3c3d29"><code>cfd82e4</code></a> Fix linting with unused import removal</li>
  <li><a href="https://github.com/django/asgiref/commit/cc1877efc8bcf5b6e606eb5c50cfdcca1213c83a"><code>cc1877e</code></a> Fix import sorting in previous commit.</li>
  <li><a href="https://github.com/django/asgiref/commit/7becc9daca2628c46af1cb7e46b4c47c1ea27adf"><code>7becc9d</code></a> Making thread_sensitive=True the default</li>
  <li><a href="https://github.com/django/asgiref/commit/66a6e68af258d626f5004ef305255b3154e552d4"><code>66a6e68</code></a> Fixed <a href="https://github-redirect.dependabot.com/django/asgiref/issues/194">#194</a>: Made async_to_sync work inside a fork</li>
  <li><a href="https://github.com/django/asgiref/commit/4ab9d8ea79bc202aa52ab4e71f315a65d025a377"><code>4ab9d8e</code></a> Fixed <a href="https://github-redirect.dependabot.com/django/asgiref/issues/193">#193</a>: Bumped docs version to 3.0</li>
  <li><a href="https://github.com/django/asgiref/commit/1c9d06329dbd679a7c564b0652147855a721edb4"><code>1c9d063</code></a> Clarified &quot;Optional&quot; meaning (<a href="https://github-redirect.dependabot.com/django/asgiref/issues/190">#190</a>)</li>
  <li>Additional commits viewable in <a href="https://github.com/django/asgiref/compare/3.2.10...3.3.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1018/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTAxNzkxNjYz
title: Update utils.py to fix sqlite3.OperationalError
state: closed
created_at: 2020-10-12T20:17:53Z
updated_at: 2020-10-12T20:25:10Z
closed_at: 2020-10-12T20:25:09Z
author_association: CONTRIBUTOR
pull_request: dogsheep/swarm-to-sqlite/pulls/10
type: pull
body:
  Fixes the errors:
  - sqlite3.OperationalError: table posts has no column named text
  - sqlite3.OperationalError: table photos has no column named hasSticker

  That will cause sqlite-utils to notice if there's a missing column and add it. As recommended by @simonw

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjEwNTA4MTU=
title: "Edit SQL" button on canned queries
state: closed
created_at: 2020-10-14T00:51:39Z
updated_at: 2020-10-23T19:44:06Z
closed_at: 2020-10-14T03:44:23Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Feature request: Would it be possible to add an "edit this query" button on canned queries? Clicking it would open the canned query as an editable sql query. I think the intent is to have named parameters to allow this, but sometimes you just gotta rewrite it? 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1019/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjEwNjg5Mjk=
title: Method for datasette.client() to forward on authentication
state: open
created_at: 2020-10-14T01:47:49Z
updated_at: 2020-10-19T22:45:01Z
author_association: OWNER
type: issue
body:
  I stumbled into this while working on Dogsheep Beta: the requests it re-dispatched through `TableView` did not carry authentication cookies, and since this was against a private instance they were thus denied.

  https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L223-L231

  This made me think that `datasette.client.get()` (which Dogsheep Beta will start using shortly) could benefit from some kind of utility mechanism for passing through the cookies and general authenticated state from the current request.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1020/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTAzNjg1MDc3
title: Fix table name in spatialite example command
state: closed
created_at: 2020-10-14T22:19:34Z
updated_at: 2020-10-14T23:46:46Z
closed_at: 2020-10-14T23:46:46Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1022
type: pull
body:
  The example query for creating a new point geometry seems to be using a table called 'museums' but at one point it instead uses 'events'. I *believe* it is intended to be museums (the example makes more sense if so). 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1022/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjI2NzM4MTg=
title: Fix issues relating to base_url
state: closed
created_at: 2020-10-15T21:02:06Z
updated_at: 2020-11-24T19:51:44Z
closed_at: 2020-10-31T20:51:01Z
author_association: OWNER
body: Lots of `base_url` bugs that I'd like to solve at once.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1023/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjI2NzQ3MDg=
title: Figure out how to run an environment that exercises the base_url proxy setting
state: closed
created_at: 2020-10-15T21:03:39Z
updated_at: 2020-10-23T19:44:06Z
closed_at: 2020-10-15T22:34:04Z
author_association: OWNER
body: Refs #1023.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1024/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjI3MjQwODY=
title: Fix last remaining links to "/" that do not respect base_url
state: closed
created_at: 2020-10-15T22:46:38Z
updated_at: 2020-10-23T19:44:06Z
closed_at: 2020-10-20T05:21:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Refs #1023
  ```
  datasette % git grep '"/"' -- '*.html'
  datasette/templates/error.html:        <a href="/">home</a>
  datasette/templates/patterns.html:        <a href="/">home</a> /
  datasette/templates/query.html:        <a href="/">home</a> /
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1025/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjI3Mzg5ODg=
title: How should datasette.client interact with base_url
state: closed
created_at: 2020-10-15T23:07:11Z
updated_at: 2020-10-31T19:29:52Z
closed_at: 2020-10-31T19:29:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Refs #1023. If Datasette is running with a `base_url` setting and a plugin calls e.g. `datasette.client.get("/-/plugins.json")` what should happen?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1026/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjI3NTgxMzI=
title: Add documentation on serving Datasette behind a proxy using base_url
state: closed
created_at: 2020-10-15T23:46:29Z
updated_at: 2020-10-31T21:14:05Z
closed_at: 2020-10-31T21:14:05Z
author_association: OWNER
body:
  This can go on this page: https://docs.datasette.io/en/stable/deploying.html

  Refs #1023, #865

type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1027/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjI4MTY0MzY=
title: .extract() shouldn't extract null values
state: open
created_at: 2020-10-16T02:41:08Z
updated_at: 2021-08-12T12:32:14Z
author_association: OWNER
type: issue
body:
  This almost works, but it creates a rogue `type` record with a value of None.
  ```
  In [1]: import sqlite_utils
  In [2]: db = sqlite_utils.Database(memory=True)
  In [5]: db["creatures"].insert_all([
      {"id": 1, "name": "Simon", "type": None},
      {"id": 2, "name": "Natalie", "type": None},
      {"id": 3, "name": "Cleo", "type": "dog"}], pk="id")
  Out[5]: <Table creatures (id, name, type)>
  In [7]: db["creatures"].extract("type")
  Out[7]: <Table creatures (id, name, type_id)>
  In [8]: list(db["creatures"].rows)
  Out[8]: 
  [{'id': 1, 'name': 'Simon', 'type_id': None},
   {'id': 2, 'name': 'Natalie', 'type_id': None},
   {'id': 3, 'name': 'Cleo', 'type_id': 2}]
  In [9]: db["type"]
  Out[9]: <Table type (id, type)>
  In [10]: list(db["type"].rows)
  Out[10]: [{'id': 1, 'type': None}, {'id': 2, 'type': 'dog'}]
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/186/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjM0NjAxMDc=
title: Maybe: Utility method / CLI tool for initializing SpatiaLite
state: closed
created_at: 2020-10-16T19:04:03Z
updated_at: 2022-02-05T00:04:26Z
closed_at: 2020-10-16T19:15:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I think this should initialize SpatiaLite against the current database if it has not been initialized already.
  >
  > Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/187/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA1MDc2NDE4
title: Add fitbit-to-sqlite
state: open
created_at: 2020-10-16T20:04:05Z
updated_at: 2020-10-16T20:04:05Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/dogsheep.github.io/pulls/5
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjM3MDgzMTA=
title: About loading spatialite
state: closed
created_at: 2020-10-17T08:47:02Z
updated_at: 2022-02-05T00:04:26Z
closed_at: 2020-10-17T08:52:58Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi @simonw ,
  If I run

  ```
  sqlite3
  .load /usr/local/lib/mod_spatialite.so
  select spatialite_version();
  ```

  I have `5.0.0`.

  ![image](https://user-images.githubusercontent.com/30607/96332706-d8cd3300-1065-11eb-906b-daf99963198e.png)


  If I run

  ```
  sqlite-utils :memory: "select spatialite_version()" --load-extension=spatialite
  ```

  I have

  ```
  Traceback (most recent call last):
    File "/home/aborruso/.local/bin/sqlite-utils", line 8, in <module>
      sys.exit(cli())
    File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/home/aborruso/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 936, in query
      _load_extensions(db, load_extension)
    File "/home/aborruso/.local/lib/python3.8/site-packages/sqlite_utils/cli.py", line 1326, in _load_extensions
      db.conn.load_extension(ext)
  TypeError: argument 1 must be str, not None
  ```

  How to load properly spatialite extension in sqlite-utils?

  Thank you very muc

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/188/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjM4MDM3Nzc=
title: --load-extension=spatialite shortcut
state: closed
created_at: 2020-10-17T17:02:08Z
updated_at: 2022-01-20T21:29:41Z
closed_at: 2020-10-19T22:37:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I added this to `sqlite-utils` in https://github.com/simonw/sqlite-utils/issues/136 and I really like it: pass a special value of `spatialite` and Datasette should attempt to load it from known likely installation locations.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1028/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA1MzM5NTE1
title: fix(docs): broken link
state: closed
created_at: 2020-10-17T20:03:20Z
updated_at: 2020-10-17T20:05:04Z
closed_at: 2020-10-17T20:05:04Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1029
body: This PR fixes a broken markdown link in the `Publish` docs page.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1029/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjM4MzgzMzE=
title: export.xml file name varies with different language settings
state: closed
created_at: 2020-10-17T20:07:18Z
updated_at: 2020-10-17T21:39:15Z
closed_at: 2020-10-17T21:14:10Z
author_association: NONE
type: issue
state_reason: completed
body:
  The XML file exported from my phone has a Norwegian file name  – `eksport.xml` 🙄  

  I can work around this by unpacking the zip and using `--xml`, but then I lose the workout points. 

  Perhaps this could be solved by `--localized-xml eksport.xml`? Alternatively just fall back to the first XML file in the root folder of the zip.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjM4NjE2ODM=
title: Switch to using datasette.client
state: closed
created_at: 2020-10-17T22:42:26Z
updated_at: 2020-10-17T23:00:47Z
closed_at: 2020-10-17T23:00:47Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  `datasette.client` is designed for this kind of thing, to replace this code: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L223-L232

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA1NDUzOTAw
title: Make `package` command deal with a configuration directory argument
state: open
created_at: 2020-10-18T11:07:02Z
updated_at: 2020-10-19T08:01:51Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/datasette/pulls/1030
type: pull
body:
  Currently if we run `datasette package` on a configuration directory we'll get an exception when we try to hard link to the directory. This PR copies the tree and makes the Dockerfile run inspect on all *.db files.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1030/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjQyNjQ1NzQ=
title: Option to fetch README and/or HTML-rendered README for repos
state: closed
created_at: 2020-10-19T05:10:24Z
updated_at: 2020-10-19T05:33:42Z
closed_at: 2020-10-19T05:33:42Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I'm thinking:

      github-to-sqlite repos ... --readme # Populates readme column with raw text
      github-to-sqlite repos ... --readme-html # Populates readme_html column with raw HTML

  https://developer.github.com/v3/repos/contents/#get-a-repository-readme

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/52/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA1NzY5NDYy
title: Fallback to databases in inspect-data.json when no -i options are passed
state: closed
created_at: 2020-10-19T07:51:06Z
updated_at: 2021-03-29T01:46:45Z
closed_at: 2021-03-29T00:23:41Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/datasette/pulls/1031
type: pull
body:
  Currenlty `Datasette.__init__` checks immutables against None to decide whether to fallback to inspect-data.json. This patch modifies the serve command to pass None when no -i options are passed so this fallback works correctly.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1031/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjQ3NTk1ODg=
title: Add search highlighting snippets
state: open
created_at: 2020-10-19T16:00:48Z
updated_at: 2021-08-26T20:23:11Z
author_association: MEMBER
body: Like on https://til.simonwillison.net/til/search?q=Snippet
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjQ4NzgxNTE=
title: Bring date parsing into Datasette core
state: open
created_at: 2020-10-19T18:30:45Z
updated_at: 2020-10-19T19:37:55Z
author_association: OWNER
type: issue
body:
  Currently this is mainly handled by a plugin - https://github.com/simonw/datasette-dateutil - but I realise now that this really needs to be core functionality.

  See also Twitter thread: https://twitter.com/simonw/status/1318234808653213696

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1032/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjUwOTk3Nzc=
title: datasette.urls.static_plugins(...) method
state: closed
created_at: 2020-10-20T01:16:32Z
updated_at: 2020-10-24T22:58:33Z
closed_at: 2020-10-24T20:03:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Follow-on from #904. For constructing URLs like this:

      /-/static-plugins/NAME_OF_PLUGIN_PACKAGE/yourfile.js

  Should be documented on https://docs.datasette.io/en/latest/writing_plugins.html#static-assets and https://docs.datasette.io/en/latest/internals.html#datasette-urls

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1033/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjUxODQ2NDU=
title: Better way of representing binary data in .csv output
state: closed
created_at: 2020-10-20T04:28:58Z
updated_at: 2021-06-17T18:13:21Z
closed_at: 2020-10-29T22:47:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I just noticed this: https://latest.datasette.io/fixtures/binary_data.csv

  ```csv
  rowid,data
  1,b'\x15\x1c\x02\xc7\xad\x05\xfe'
  2,b'\x15\x1c\x03\xc7\xad\x05\xfe'
  ```
  There's no good way to represent binary data in a CSV file, but this seems like one of the more-bad options.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1034/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjU3NDM3NTU=
title: datasette.urls.table(..., format="json") argument
state: closed
created_at: 2020-10-20T16:09:34Z
updated_at: 2020-10-31T18:16:43Z
closed_at: 2020-10-31T18:16:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > That `datasette.urls.table("db", "table") + ".json"` example is bad because if the table name contains a `.` it should be `?_format=json` instead.
  > 
  > Maybe `.table()` should have a `format="json"` option that knows how to do this.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1026#issuecomment-712962517_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1035/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjU5OTY1MDc=
title: Make it possible to download BLOB data from the Datasette UI
state: closed
created_at: 2020-10-20T22:47:56Z
updated_at: 2021-01-18T17:45:00Z
closed_at: 2020-10-25T00:14:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Currently you can only extract binary BLOB data as base64-encoded JSON, which is not user friendly at all. It should always be possible for end-users to get the binary data out.

  I'm worried about XSS vulnerabilities here, but hopefully sending `Content-Type: application/octet-stream` helps there? Need to research that.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1036/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjYwOTQ3NTQ=
title: Add horizontal scrollbar to tables
state: closed
created_at: 2020-10-21T03:13:34Z
updated_at: 2020-10-27T20:52:04Z
closed_at: 2020-10-21T03:16:36Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Currently you have to scroll the entire page sideways if a table is wide. Make the table `overflow-x: auto` instead.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1037/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA3MjY3MDg3
title: DOC: Fix syntax error
state: closed
created_at: 2020-10-21T05:45:38Z
updated_at: 2020-10-21T22:57:21Z
closed_at: 2020-10-21T22:44:17Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1038
type: pull
body:
  If I understand https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes correctly, `register_routes` should return a `List[Tuple[str, Callable]]`. I believe the current code in documentation has a syntax error (extra `)`). 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1038/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjY2ODc1NzI=
title: Add an animation to the column actions menu
state: closed
created_at: 2020-10-21T16:56:28Z
updated_at: 2020-10-23T19:44:07Z
closed_at: 2020-10-21T17:02:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Inspired by the animation on some of GitHub's dropdown menus:

  https://github.com/primer/css/blob/da8ee54248e6d76c15c18e53684a15a6516b5b7f/src/utilities/animations.scss#L114-L131

  ```css
  /* Fade in an element and scale it fast */
  .anim-scale-in {
    animation-name: scale-in;
    animation-duration: 0.15s;
    animation-timing-function: cubic-bezier(0.2, 0, 0.13, 1.5);
  }

  @keyframes scale-in {
    0% {
      opacity: 0;
      transform: scale(0.5);
    }

    100% {
      opacity: 1;
      transform: scale(1);
    }
  }
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1039/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA3OTAzMzky
title: /db/table/-/blob/pk/column.blob download URL
state: closed
created_at: 2020-10-21T22:39:15Z
updated_at: 2020-10-24T23:09:20Z
closed_at: 2020-10-24T23:09:19Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1040
type: pull
body:
  Refs #1036. Still needs:

  - [x] Comprehensive tests across all of the code branches, plus permissions
  - [x] A bit more refactoring to share logic cleanly with `RowView`
  - ~~A configuration option to disable this feature (probably)~~

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1040/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjc2Mjc5MjM=
title: extra_js_urls and extra_css_urls should respect base_url setting
state: closed
created_at: 2020-10-22T18:34:33Z
updated_at: 2020-10-31T20:49:28Z
closed_at: 2020-10-31T20:48:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1033#issuecomment-714681365_

  Refs #1023

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1041/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjc4MDIwODE=
title: Plugin hook for loading templates
state: closed
created_at: 2020-10-23T00:18:39Z
updated_at: 2020-10-30T17:47:21Z
closed_at: 2020-10-30T17:47:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This can work with the Jinja template loaders. It would unlock things like storing templates in SQLite.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1042/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjc4NDg2MjU=
title: Some workout columns should be float, not text
state: open
created_at: 2020-10-23T02:47:02Z
updated_at: 2022-06-23T04:35:02Z
author_association: MEMBER
type: issue
body:
  Columns `duration`, `totalDistance` and `totalEnergyBurned` should be converted to float.

  https://github.com/dogsheep/healthkit-to-sqlite/blob/71e36e1cf034b96de2a8e6652265d782d3fdf63b/healthkit_to_sqlite/utils.py#L50-L57

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA4NzE5NTY3
title: Include LICENSE in sdist
state: closed
created_at: 2020-10-23T05:04:12Z
updated_at: 2020-10-26T00:14:57Z
closed_at: 2020-10-23T20:54:35Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1043
type: pull
body:
  Hi, thanks for `datasette`! 

  This PR adds the `LICENSE` to source distributions, which seems the norm for Apache-2.0 stuff.

  I noticed the [0.50.2 sdist](https://files.pythonhosted.org/packages/f2/ba/1b5f182c3f1769c0863bcaa77406bdcb81c92e31bb579959c01b1d8951c0/datasette-0.50.2.tar.gz) doesn't ship `LICENSE`, but the 0.5.2 `whl` does, so I'm assuming the intent _is_ to ship... and it's a one-liner!

  Motivation: 

  It might be a bit of a slog, but I'm looking to see about getting `datasette` (and friends!) available on conda-forge. There are a few missing upstreams (`asgi-csrf`, `python-basecov`, `mergedeep`) and some of the plugins don't even appear to _have_ tarballs (just `whl`!), but the little stuff like licenses are nice to get out handled upstream vs separately grabbing them.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1043/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA4NzIwNjYw
title: Add minimum supported python
state: closed
created_at: 2020-10-23T05:08:03Z
updated_at: 2020-10-23T20:53:08Z
closed_at: 2020-10-23T20:53:08Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1044
type: pull
body:
  Thanks for `datasette`!

  This PR adds `python_requires` to formally signal the [minimum supported python version](https://packaging.python.org/guides/dropping-older-python-versions/#specify-the-version-ranges-for-supported-python-distributions) (which is pointed out with classifiers, so seems pretty straightforward).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1044/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjg2MDAwNDg=
title: Document that datasette.render_template(template, ...) also accepts a list of templates
state: closed
created_at: 2020-10-23T23:37:12Z
updated_at: 2020-10-24T00:22:10Z
closed_at: 2020-10-24T00:22:09Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none

  `await .render_template(template, context=None, request=None)`

  This currently only accepts a single template. If it accepted a list of templates (where the first available template gets rendered) it could be more widely used by Datasette internally.

  Spotted this while researching #1042.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1045/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjg4OTUxOTM=
title: Link to blob downloads in the right places
state: closed
created_at: 2020-10-24T23:00:41Z
updated_at: 2020-10-25T00:13:21Z
closed_at: 2020-10-25T00:13:21Z
author_association: OWNER
body: Split from #1040, refs #1036.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1046/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjg4OTUyMzM=
title: A new section in the docs about how Datasette handles BLOB columns
state: closed
created_at: 2020-10-24T23:01:02Z
updated_at: 2020-10-31T22:11:25Z
closed_at: 2020-10-31T21:38:05Z
author_association: OWNER
body: Split from #1040, refs #1036.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1047/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjg5MDUwOTg=
title: Documentation and unit tests for urls.row() urls.row_blob() methods
state: open
created_at: 2020-10-25T00:13:53Z
updated_at: 2022-07-10T16:23:57Z
author_association: OWNER
type: issue
body:
  https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/app.py#L1307-L1313

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1048/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTA5NTkwMjA1
title: Add template block prior to extra URL loaders
state: closed
created_at: 2020-10-25T13:08:55Z
updated_at: 2020-10-29T09:20:52Z
closed_at: 2020-10-29T09:20:34Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1049
type: pull
body:
  To handle packages that require Javascript state setting prior to loading a package (eg [`thebelab`](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html), provide a template block before the URLs are loaded.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1049/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjkwNTczODg=
title: Switch to .blob render extension for BLOB downloads
state: closed
created_at: 2020-10-25T16:26:21Z
updated_at: 2020-10-29T22:01:39Z
closed_at: 2020-10-29T22:01:39Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This may require a complete rethink of the `/db/table/-/blob/row/column.blob` mechanism I just built for #1036.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1050/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjkwOTY1OTU=
title: Better display of binary data on arbitrary query results page
state: closed
created_at: 2020-10-25T19:38:06Z
updated_at: 2020-10-29T22:12:16Z
closed_at: 2020-10-29T22:01:39Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101

  <img width="718" alt="fixtures__select_rowid__data_from_binary_data_order_by_rowid_limit_101_and_Switch_to__blob_render_extension_for_BLOB_downloads_·_Issue__1050_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/97117089-c2a32080-16be-11eb-8e4b-63821715b98e.png">

  Problem: if these were larger fields that HTML page could have multiple megabytes of Python binary string representations on it.

  It should behave more like the regular table view does:

  <img width="593" alt="fixtures__binary_data__2_rows" src="https://user-images.githubusercontent.com/9599/97117113-ea928400-16be-11eb-8730-3c1c8d6c3491.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1051/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MjkxODMzMzI=
title: Column action menu overlapped by Leaflet maps
state: closed
created_at: 2020-10-26T02:17:29Z
updated_at: 2020-10-27T20:52:04Z
closed_at: 2020-10-26T02:19:36Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Using `datasette-leaflet-geojson`:

  <img width="913" alt="Cursor_and_benches__benches__17_387_rows" src="https://user-images.githubusercontent.com/9599/97127361-a4f1ad80-16f6-11eb-9c81-91ba7c990982.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1052/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mjk2MDQ4Mzg=
title: Document recommendations for plugin authors to design URLs
state: closed
created_at: 2020-10-26T14:19:21Z
updated_at: 2020-10-29T19:37:58Z
closed_at: 2020-10-29T19:35:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See thread: https://twitter.com/kanedr/status/1320653434895347713

  > The process and API for making a plugin is great btw. One question I had was the best format for URLs. I've created an url like /<database>/<table>/reconcile as a json endpoint, but that could conflict with the row-level URLs. Is there a recommended pattern to use?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1053/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTEwMjM1OTA5
title: Allow iterables other than Lists in m2m records
state: closed
created_at: 2020-10-26T18:47:44Z
updated_at: 2020-10-27T16:28:37Z
closed_at: 2020-10-27T16:24:21Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/189
type: pull
body:
  I was playing around with sqlite-utils, creating a Roam Research dogsheep-style importer for Datasette, and ran into a slight snag.

  I wanted to use a generator to add an order column in an importer. It looked something like:

  ```
  def order_generator(iterable, attr=None):
      if attr is None:
          attr = "order"
      order: int = 0

      for i in iterable:
          i[attr] = order
          order += 1
          yield i
  ```

  When I used this with `insert_all` and other things, it worked fine--but it didn't work as the `records` argument to `m2m`.  I dug into it, and sqlite-utils is explicitly checking if the records argument is a list or a tuple.  I flipped the check upside down, and now it checks if the argument is a mapping.  If it's a mapping, it wraps it in a list, otherwise it leaves it alone.

  (I get that it might not really make sense to put the order column on the second table.  I changed my import schema a bit, and no longer have a real example, but maybe this change still makes sense.)

  The automated tests still pass, but I did not add any new ones.

  Let me know what you think! I'm really loving Datasette and its ecosystem; thanks for everything!

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/189/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzAxOTk0NjQ=
title: Switch from versioneer to concrete version in setup.py
state: closed
created_at: 2020-10-27T07:38:08Z
updated_at: 2020-10-29T03:38:18Z
closed_at: 2020-10-29T03:38:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The new PyPI resolver keeps on showing me warnings like this one when I install Datasette directly from GitHub using `pip install https://github.com/simonw/datasette/archive/main.zip`:
  ```
  Successfully built datasette
  Installing collected packages: datasette
    Attempting uninstall: datasette
      Found existing installation: datasette 0.50.2
      Uninstalling datasette-0.50.2:
        Successfully uninstalled datasette-0.50.2
  ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

  We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

  datasette-upload-csvs 0.5 requires datasette>=0.47, but you'll have datasette 0+unknown which is incompatible.
  datasette-publish-vercel 0.8 requires datasette>=0.44, but you'll have datasette 0+unknown which is incompatible.
  datasette-psutil 0.2 requires datasette>=0.44, but you'll have datasette 0+unknown which is incompatible.
  datasette-leaflet-geojson 0.6 requires datasette>=0.48, but you'll have datasette 0+unknown which is incompatible.
  datasette-edit-schema 0.3 requires datasette>=0.44, but you'll have datasette 0+unknown which is incompatible.
  datasette-cluster-map 0.13 requires datasette>=0.48, but you'll have datasette 0+unknown which is incompatible.
  Successfully installed datasette-0+unknown
  ```
  This is because we use versioneer. I'm going to drop that in favour of embedding the version directly in `setup.py`, like I do in other projects such as `sqlite-utils`.

  I'll use a `.dev` suffix in the development version, as suggested by https://www.python.org/dev/peps/pep-0440/#developmental-releases

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1054/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzAyMTA4ODA=
title: query.html and table.html should share the same table implementation
state: open
created_at: 2020-10-27T07:58:21Z
updated_at: 2020-10-27T07:58:29Z
author_association: OWNER
type: issue
body:
  In #998 I made a change that affected the table page but didn't affect the query page because I incorrectly assumed they shared rendering logic.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1055/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTEwOTU2MTM0
title: Progress bar for sqlite-utils insert command
state: closed
created_at: 2020-10-27T18:08:53Z
updated_at: 2020-10-27T18:16:03Z
closed_at: 2020-10-27T18:16:03Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/190
body: Refs #173
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/190/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTExMDA1NTQy
title: Radical new colour scheme and base styles, courtesy of @natbat
state: closed
created_at: 2020-10-27T19:31:49Z
updated_at: 2020-10-27T19:39:57Z
closed_at: 2020-10-27T19:39:56Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1056
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1056/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzA3OTc3ODc=
title: --cors should enable /fixtures.db CORS access
state: closed
created_at: 2020-10-27T20:38:34Z
updated_at: 2020-10-27T20:52:05Z
closed_at: 2020-10-27T20:51:09Z
author_association: OWNER
body: So Datasette can work with `SQL.js` as seen in https://observablehq.com/@mbostock/sqlite
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1057/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzA4MDI5OTQ=
title: Database download should implement cascading permissions
state: closed
created_at: 2020-10-27T20:43:27Z
updated_at: 2020-10-28T03:15:47Z
closed_at: 2020-10-28T03:15:47Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L130-L136

  Should be updated for #832 cascading permissions. Example commit: https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1058/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTExNTQ5Mzc0
title: Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7
state: closed
created_at: 2020-10-28T13:32:40Z
updated_at: 2020-10-28T17:08:29Z
closed_at: 2020-10-28T17:08:28Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1059
type: pull
body:
  Updates the requirements on [aiofiles](https://github.com/Tinche/aiofiles) to permit the latest version.
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/Tinche/aiofiles/commit/b4482aeef5312d4bf0c1abda5892df80957fa01f"><code>b4482ae</code></a> 0.6.0</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/eaccd3e43013b843f606665a68d3db395437e537"><code>eaccd3e</code></a> Update Changelog</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/2de90cf95e48db9178ef5bb246b5866892e8ef8e"><code>2de90cf</code></a> Test on 3.9</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/62b4c27096a5437f0608886e98cbe2681992ca57"><code>62b4c27</code></a> Update .travis.yml</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/4083c7371a2e667a316efa55613cfe6b826304b2"><code>4083c73</code></a> Removed pypy3 for ppc64le</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/26e795b5dafe7d227147f1663ffeca6a76efe453"><code>26e795b</code></a> Update .travis.yml</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/bd154cf3bf60f03aa22b3b264578a987d136be85"><code>bd154cf</code></a> Update .travis.yml</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/7ae0e2ace9193e4e2417fc76ea51caf84ba0e656"><code>7ae0e2a</code></a> Update .travis.yml</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/84eeb6c686ca6ba0ac700e93a078547cbaac625d"><code>84eeb6c</code></a> Update .travis.yml</li>
  <li><a href="https://github.com/Tinche/aiofiles/commit/c48f5b0a825f8c68c2238f098585c1f1c6ba2c16"><code>c48f5b0</code></a> Proxy name and mode properties</li>
  <li>Additional commits viewable in <a href="https://github.com/Tinche/aiofiles/compare/v0.4.0...v0.6.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1059/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzE3NDA0NTg=
title: Idea: @db.register_function(deterministic=True)
state: closed
created_at: 2020-10-28T19:45:18Z
updated_at: 2020-10-28T21:31:06Z
closed_at: 2020-10-28T21:31:06Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Python 3.8 added a `deterministic` parameter to `db.create_function()`: https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.create_function

  `sqlite-utils` could expose this in the decorator, only actually applying it if the Python version supports it (using feature detection) - since nothing will break if it's not applied.

  https://sqlite-utils.readthedocs.io/en/stable/python-api.html#registering-custom-sql-functions

  ```python
  db = Database(memory=True)

  @db.register_function(deterministic=True)
  def reverse_string(s):
      return "".join(reversed(list(s)))
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/191/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTExODY4MTUz
title: New explicit versioning mechanism
state: closed
created_at: 2020-10-28T22:14:55Z
updated_at: 2020-10-29T03:38:17Z
closed_at: 2020-10-29T03:38:16Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1060
type: pull
body:
  - Remove all references to versioneer
  - Re-implement versioning to use a static string baked into the repo
  - Ensure that string is output by `datasette --version` and `/-/versions`

  Refs #1054

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1060/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTEyNTQ1MzY0
title: .blob output renderer
state: closed
created_at: 2020-10-29T20:25:08Z
updated_at: 2020-10-29T22:01:40Z
closed_at: 2020-10-29T22:01:39Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1061
type: pull
body:
  - [x] Remove the `/-/...blob/...` route I added in #1040 in place of the new `.blob` renderer URLs
  - [x] Link to new `.blob` download links on the arbitrary query page (using `_blob_hash=...`) - plus tests for this

  Closes #1050, Closes #1051

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1061/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzI2NzQxNDg=
title: Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows
state: open
created_at: 2020-10-29T21:25:02Z
updated_at: 2022-09-28T14:09:54Z
author_association: OWNER
type: issue
body:
  This can drive the upgrade of the `register_output_renderer` hook to be able to handle streaming all rows in a large query.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1062/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzI2ODU2NDM=
title: .csv should link to .blob downloads
state: closed
created_at: 2020-10-29T21:45:58Z
updated_at: 2021-06-17T18:12:30Z
closed_at: 2020-10-29T22:47:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  - [x] Update `.csv` output to link to these things (and get that `xfail` test to pass)
  - ~~Add a `.csv?_blob_base64=1` argument that causes them to be output in base64 in the CSV~~

  > Moving the CSV work to a separate ticket.
  _Originally posted by @simonw in https://github.com/simonw/datasette/pull/1061#issuecomment-719042601_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1063/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzI3OTg5MTM=
title: Navigation menu plus plugin hook
state: closed
created_at: 2020-10-30T00:49:36Z
updated_at: 2020-10-30T03:45:16Z
closed_at: 2020-10-30T03:45:16Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Needed for #690. Prototype in https://github.com/simonw/datasette/commit/0d7ac764861d84be24d661cf4104ce61ea11a82a

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1064/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTEyNzM2NzA1
title: Nav menu plus menu_links() hook
state: closed
created_at: 2020-10-30T03:40:18Z
updated_at: 2020-10-30T03:45:17Z
closed_at: 2020-10-30T03:45:16Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1065
body: Closes #1064, refs #690.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1065/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzI4NTkwMzA=
title: Table actions menu plus plugin hook
state: closed
created_at: 2020-10-30T03:46:54Z
updated_at: 2020-10-30T05:18:36Z
closed_at: 2020-10-30T05:16:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > For the table actions: attaching it to a cog icon next to the table name could make sense.
  >
  > <img width="503" alt="data__faa-wildlife-strikes__26_rows_where_where_AIRPORT____OIL_RIG__sorted_by_rowid_and_Fix_table_name_in_spatialite_example_command_by_jsfenfen_·_Pull_Request__1022_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/96168896-45192c80-0ed6-11eb-9d90-6172a8346468.png">
  >
  > This is the column action icon at twice the size, color `#666`.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/690#issuecomment-709497595_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1066/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzI5MDUzNjA=
title: Table actions menu on view pages, not on query pages
state: closed
created_at: 2020-10-30T05:56:39Z
updated_at: 2020-10-31T17:51:31Z
closed_at: 2020-10-31T17:40:14Z
author_association: OWNER
body: Follow-on from #1066.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1067/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzI5Mzk5MjE=
title: Default menu links should check a real permission
state: closed
created_at: 2020-10-30T07:08:34Z
updated_at: 2020-10-30T15:44:13Z
closed_at: 2020-10-30T15:42:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_menu_links.py#L4-L6

  This should check a named permission so that it can be customized by permission plugins.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1068/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTEzMTA2MDI2
title: load_template() plugin hook
state: closed
created_at: 2020-10-30T15:59:45Z
updated_at: 2020-10-30T17:47:20Z
closed_at: 2020-10-30T17:47:19Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1069
body: Refs #1042
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1069/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzMzOTA4ODQ=
title: load_template() example in documentation showing loading from a database
state: closed
created_at: 2020-10-30T17:45:03Z
updated_at: 2020-10-31T16:22:51Z
closed_at: 2020-10-31T16:22:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I should include an example in the documentation that shows loading templates from a database table.
  _Originally posted by @simonw in https://github.com/simonw/datasette/pull/1069#issuecomment-719664530_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1070/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM0ODU0MjM=
title: Messages should be displayed full width
state: closed
created_at: 2020-10-30T20:11:35Z
updated_at: 2020-10-30T20:20:02Z
closed_at: 2020-10-30T20:13:05Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In the pattern portfolio:

  <img width="907" alt="Datasette__Pattern_Portfolio_and_datasette_—_pipenv_shell_▸_Python_—_142×48" src="https://user-images.githubusercontent.com/9599/97752509-5a36b380-1ab1-11eb-9a26-700cb295fe11.png">

  But they're currently showing like this:

  <img width="901" alt="Datasette___memory_" src="https://user-images.githubusercontent.com/9599/97752544-69b5fc80-1ab1-11eb-80c2-c88aabe6a832.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1071/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM0OTk5MzA=
title: load_template hook doesn't work for include/extends
state: closed
created_at: 2020-10-30T20:33:44Z
updated_at: 2020-10-31T20:48:18Z
closed_at: 2020-10-30T22:50:57Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Includes like this one always go to disk, without hitting the `load_template` plugin hook:
  ```html+jinja
  <footer class="ft">{% block footer %}{% include "_footer.html" %}{% endblock %}</footer>
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1072/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM1NjA0MTc=
title: Remove load_template plugin hook
state: closed
created_at: 2020-10-30T22:51:52Z
updated_at: 2020-10-31T16:22:00Z
closed_at: 2020-10-31T16:22:00Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I couldn't get it working correctly with async (necessary for include/extend to function), and on deeper investigation it appears that I can build something equivalent to what I wanted using the existing `prepare_jinja2_environment` hook.

  > I'm going to remove the `load_template` plugin hook and see if it's possible to build the edit templates extension against `prepare_jinja2_environment` instead.
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1072#issuecomment-719833744_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1073/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM3NjgwMzc=
title: latest.datasette.io should include plugins from fixtures
state: closed
created_at: 2020-10-31T17:23:23Z
updated_at: 2020-10-31T19:47:47Z
closed_at: 2020-10-31T19:47:47Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > It bothers me that these aren't visible in any public demos. Maybe `latest.datasette.io` should include the `my_plugins.py` and `my_plugins2.py` plugins?
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1067#issuecomment-719961701_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1074/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM3OTY5NDI=
title: PrefixedUrlString mechanism broke everything
state: closed
created_at: 2020-10-31T19:58:05Z
updated_at: 2020-10-31T20:48:51Z
closed_at: 2020-10-31T20:48:51Z
author_association: OWNER
body: Added in 7a67bc7a569509d65b3a8661e0ad2c65f0b09166 refs #1026. Lots of tests are failing now.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1075/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM4MDUwODk=
title: Release notes for 0.51
state: closed
created_at: 2020-10-31T20:51:21Z
updated_at: 2020-10-31T22:27:00Z
closed_at: 2020-10-31T22:27:00Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Start by combining release notes from https://github.com/simonw/datasette/releases/tag/0.51a0 and https://github.com/simonw/datasette/releases/tag/0.51a1 and https://github.com/simonw/datasette/releases/tag/0.51a2

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1076/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM4MjkzODU=
title: database_actions plugin hook
state: closed
created_at: 2020-10-31T23:48:12Z
updated_at: 2020-11-02T18:43:25Z
closed_at: 2020-11-02T18:29:50Z
author_association: OWNER
body: Like `column_actions` but adds a cog menu to the database page.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1077/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzM5OTk2MTU=
title: Handle long breadcrumbs better with new menu
state: open
created_at: 2020-11-01T15:57:41Z
updated_at: 2022-01-13T22:21:29Z
author_association: OWNER
type: issue
body:
  On this page when signed in as root: https://latest.datasette.io/fixtures/roadside_attraction_characteristics/1

  ![EF921CB1-625F-4D04-A850-490B812A72B3](https://user-images.githubusercontent.com/9599/97807807-db0fbf80-1c17-11eb-9c77-ae5169b12c3d.jpeg)

  ![A49D8B76-5ACF-4F71-A8B4-21A44F5C8D51](https://user-images.githubusercontent.com/9599/97807809-dea34680-1c17-11eb-9511-a49af56a4bd2.jpeg)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1079/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzQ3Nzc2MzE=
state: open
created_at: 2020-11-02T19:55:06Z
updated_at: 2022-02-04T06:25:18Z
author_association: OWNER
body: Can use `/database/-/...` namespace from #296
type: issue
title:
  "View all" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1080/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzU1MzI3NTE=
title: sqlite-utils search command
state: closed
created_at: 2020-11-03T18:07:59Z
updated_at: 2020-11-08T17:07:01Z
closed_at: 2020-11-08T17:07:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  A command that knows how to run a search against a FTS enabled table and return results ranked by relevance.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/192/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzU2NDQ1MTM=
title: Fixtures should use FTS4 or FTS5, not FTS3
state: closed
created_at: 2020-11-03T21:24:13Z
updated_at: 2020-11-12T00:03:00Z
closed_at: 2020-11-12T00:02:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Just spotted that `fixtures.db` uses FTS3, which is pretty much obsolete these days.

  https://github.com/simonw/datasette/blob/13d1228d80c91d382a05b1a9549ed02c300ef851/tests/fixtures.py#L488-L489

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1081/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzU2NDgyMDk=
title: --tsv output format option
state: closed
created_at: 2020-11-03T21:31:18Z
updated_at: 2020-11-07T00:09:52Z
closed_at: 2020-11-07T00:09:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  We already support `--csv` for output, and the `insert` command accepts `--tsv`. The output format options should accept `--tsv` too.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/193/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzU2NTA4NjQ=
title: 3.0 release with some minor breaking changes
state: closed
created_at: 2020-11-03T21:36:31Z
updated_at: 2020-11-08T17:19:35Z
closed_at: 2020-11-08T17:19:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  While working on search (#192) I've spotted a few small changes I would like to make that would break backwards compatibility in minor ways, hence requiring a 3.x release.

  `db[table].search()` - I would like this to default to sorting by rank

  Also I'd like to free up the `-c` and `-f` options for other purposes from the standard output formats here:

  https://github.com/simonw/sqlite-utils/blob/43eae8b193d362f2b292df73e087ed6f10838144/sqlite_utils/cli.py#L48-L58

  I'd like `-f` to be used to indicate a full-text search column during an insert and `-c` to indicate a column (so you can specify which columns you want to output).

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/194/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTE1MDE0ODgz
title: table.search() improvements plus sqlite-utils search command
state: closed
created_at: 2020-11-03T22:02:08Z
updated_at: 2020-11-06T18:30:49Z
closed_at: 2020-11-06T18:30:42Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/195
body: Refs #192. Still needs tests.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/195/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzU4NTIyNzQ=
title: DigitalOcean buildpack memory errors for large sqlite db?
state: open
created_at: 2020-11-04T06:35:32Z
updated_at: 2020-11-04T19:35:44Z
author_association: NONE
type: issue
body:
  1. Have a sqlite db stored in Dropbox
  2. Previously tried the Digital Ocean build pack minimal approach (e.g. Procfile, requirements.txt, bin/post_compile)
  3. bin/post_compile with wget from Dropbox
  4. download of large sqlite db is successful 
  5. log reveals that when building Docker container, Digital Ocean runs out of memory for 5gb+ sqlite db but works fine for 2gb+ sqlite db

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1082/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzYzNjUzMDY=
title: Advanced CSV export for arbitrary queries
state: open
created_at: 2020-11-04T19:23:05Z
updated_at: 2021-06-17T18:12:31Z
author_association: OWNER
type: issue
body:
  There's no link to download the CSV file - the table page has that as an advanced export option, but this is missing from the query page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1083/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzY1MjAzMTA=
title: Introspect if table is FTS4 or FTS5
state: closed
created_at: 2020-11-05T00:45:50Z
updated_at: 2020-11-05T03:54:07Z
closed_at: 2020-11-05T03:54:07Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I want `.search()` to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5.
  >
  > This means I need to be able to introspect and tell if a table is FTS4 or FTS5.

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/196/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzcxNTM5Mjc=
title: Rethink how table.search() method works
state: closed
created_at: 2020-11-05T18:04:34Z
updated_at: 2020-11-08T17:07:37Z
closed_at: 2020-11-08T17:07:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I need to improve this method to help build `sqlite-utils search` in #192 (PR is #195).

  The challenge is deciding how it should handle sorting by relevance - especially since that is easy in FTS5 but not at all easy in FTS4.

  > Latest test failure:
  > ```
  > 114  ->	        assert [("racoons are biting trash pandas", "USA", "bar")] == table.search(
  > 115  	            "bite", order="rowid"
  > 116  	        )
  > 117  	
  > 118  	
  > 119  	def test_optimize_fts(fresh_db):
  > (Pdb) table.search("bite")
  > [(2, 'racoons are biting trash pandas', 'USA', 'bar', -9.641434262948206e-07)]
  > ```
  > The problem here is that the `table.search()` method now behaves differently for FTS4 v.s. FTS5 tables.
  > 
  > With FTS4 you get back just the table columns.
  >
  > With FTS5 you also get back the `rowid` as the first column and the `rank` score as the last column.
  > 
  > This is weird. It also makes me question whether having `.search()` return a list of tuples is the right API design.

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/195#issuecomment-722542895_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/197/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzczOTQ0NzA=
title: Table/database action menu cut off if too short
state: closed
created_at: 2020-11-06T01:55:23Z
updated_at: 2020-11-21T23:45:59Z
closed_at: 2020-11-21T23:45:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ![3CC0C181-959E-4B20-BE39-806ED93E833E](https://user-images.githubusercontent.com/9599/98316836-03891800-1f90-11eb-9e52-5266baf33296.jpeg)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1084/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mzc0NzY0MjM=
title: Support order by relevance against FTS4
state: closed
created_at: 2020-11-06T05:36:31Z
updated_at: 2020-11-06T18:30:44Z
closed_at: 2020-11-06T18:30:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  For #192 and #197 I've decided I want to be able to order by relevance in FTS4 as well as FTS5.

  This means I need to port over my work on bm25() from https://github.com/simonw/sqlite-fts4 (since I don't want to add a full dependency).

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/198/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mzc4NTU3MzE=
title: @db.register_function(..., replace=False) to avoid double-registering custom functions
state: closed
created_at: 2020-11-06T15:39:21Z
updated_at: 2020-11-06T18:30:44Z
closed_at: 2020-11-06T18:30:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'd like a mechanism to optionally avoid registering a custom function if it has already been registered.

  SQLite doesn't seem to offer a way to introspect registered custom functions so I'll need to track what has already been registered in `sqlite-utils` instead.

  > Should I register the custom `rank_bm25` SQLite function for every connection, or should I register it against the connection just the first time the user attempts an FTS4 search? I think I'd rather register it only if it is needed.

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/198#issuecomment-723145383_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/199/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzgxMTUxNjU=
title: sqlite-utils rows -c option
state: closed
created_at: 2020-11-07T00:22:12Z
updated_at: 2020-11-07T00:28:48Z
closed_at: 2020-11-07T00:28:47Z
author_association: OWNER
type: issue
state_reason: completed
body:
  To let you specify the exact columns you want. Based on the `-c` option to `sqlite-utils search` in #192.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/200/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3MzgxMjg5MTM=
title: .search(columns=) and sqlite-utils search -c ... bug
state: closed
created_at: 2020-11-07T01:27:26Z
updated_at: 2020-11-08T16:54:15Z
closed_at: 2020-11-08T16:54:15Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Both `table.search(columns=)` and the `sqlite-utils search -c` option do not work as expected - they always return both the `rowid` and the `rank` columns even if those have not been requested.

  This should be fixed before the 3.0 non-alpha release.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/201/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Mzg1MTQzNjc=
title: sqlite-utils insert -f colname - for configuring full-text search
state: closed
created_at: 2020-11-08T17:30:09Z
updated_at: 2021-01-03T05:00:36Z
closed_at: 2021-01-03T05:00:27Z
author_association: OWNER
type: issue
state_reason: completed
body:
  A mechanism for specifying columns that should be configured for full-text search as part of the initial data import:

      sqlite-utils insert mydb.db articles articles.csv --csv -f title -f body

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/202/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTE4OTg4ODc5
title: Use FTS4 in fixtures
state: closed
created_at: 2020-11-11T06:44:30Z
updated_at: 2020-11-12T00:02:59Z
closed_at: 2020-11-12T00:02:58Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1085
body: Refs #1081
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1085/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDEwMjEzNDI=
title: Foreign keys with blank titles result in non-clickable links
state: closed
created_at: 2020-11-11T19:41:09Z
updated_at: 2020-11-28T23:28:29Z
closed_at: 2020-11-11T23:46:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  <img width="851" alt="index__core_snapshot_tags__3_rows_and_swarm__select_venues_name__checkins_id__createdAt__type__timeZoneOffset__editableUntil___like___isMayor__source__venue__createdBy__event__sticker__created__comments_count__entities__shout__checkins_priv" src="https://user-images.githubusercontent.com/9599/98856656-bafbaf80-2412-11eb-8070-4f3a47511973.png">

  The HTML looks like this:

  ```html
  <td class="col-tag_id type-int"><a href="/index/core_tag/1"></a>&nbsp;<em>1</em></td>
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1086/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDEyMzE4NDk=
title: Idea: ?_extra=urls for getting back URLs to useful things
state: open
created_at: 2020-11-12T02:55:41Z
updated_at: 2021-12-15T18:06:16Z
author_association: OWNER
type: issue
body:
  Working on https://github.com/simonw/datasette-search-all/issues/10 made me realize that sometimes it can be difficult to calculate the URL for a database, table or row within Datasette.

  It would be useful to have an optional extra JSON extension (using `?_extra=` from #262) that can help with this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1087/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDEyNjg5NTY=
title: OperationalError('interrupted') can 500 on row page
state: closed
created_at: 2020-11-12T04:29:55Z
updated_at: 2020-11-28T23:28:35Z
closed_at: 2020-11-12T04:36:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I got this on my (private) https://dogsheep.simonwillison.net/twitter/tweets/1188612004572880896 page:

  <img width="1059" alt="Error_500" src="https://user-images.githubusercontent.com/9599/98895899-a6440980-245c-11eb-98e8-fb97d10ec2fd.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1088/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDE2NjU3MjY=
title: Sweep documentation for words that minimize involved difficulty
state: closed
created_at: 2020-11-12T14:53:05Z
updated_at: 2020-11-28T23:28:43Z
closed_at: 2020-11-12T20:07:26Z
author_association: OWNER
body: Inspired by https://github.com/django/django/pull/11482
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1089/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDE4NjIzNjQ=
title: Custom widgets for canned query forms
state: open
created_at: 2020-11-12T19:21:07Z
updated_at: 2021-03-27T16:25:25Z
author_association: OWNER
type: issue
body:
  This is an idea that was cut from the first version of writable canned queries:

  > I really want the option to use a `<textarea>` for a specific value.
  > 
  > Idea: metadata syntax like this:

  > 
  > ```json
  > {
  >     "databases": {
  >        "my-database": {
  >            "queries": {
  >                "add_twitter_handle": {
  >                    "sql": "insert into twitter_handles (username) values (:username)",
  >                    "write": true,
  >                    "params": {
  >                        "username": {
  >                             "widget": "textarea"
  >                        }
  >                    }
  >                }
  >            }
  >        }
  >     }
  > }
  > ```
  > 
  > I can ship with some default widgets and provide a plugin hook for registering extra widgets.
  >
  > This opens up some really exciting possibilities for things like map widgets that let you draw polygons.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-608125928_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1090/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDIwMTEwNDk=
title: .json and .csv exports fail to apply base_url
state: closed
created_at: 2020-11-12T23:45:16Z
updated_at: 2021-01-24T21:20:24Z
closed_at: 2021-01-09T22:19:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Just tested with the latest Docker image, and it works pretty much everywhere!  THANK YOU!
  > 
  > I did notice that if I try to export json or csv, the base is not applied.  Not sure if I should reopen this issue or open a new one.
  > 
  > To see this, go here: https://corpora.tika.apache.org/datasette/corpora-metadata/REF_PARSE_EXCEPTION_TYPES
  > 
  > Click/hover over json or CSV and you'll see that the 'datasette' base is not included.

  _Originally posted by @tballison in https://github.com/simonw/datasette/issues/865#issuecomment-726385422_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1091/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDIwNDE2Njc=
title: Make cascading permission checks available to plugins
state: closed
created_at: 2020-11-13T01:02:55Z
updated_at: 2023-08-30T22:17:42Z
closed_at: 2023-08-30T22:17:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The `BaseView` class has a method for cascading permission checks, but it's not easily accessible to plugins.

  https://github.com/simonw/datasette/blob/5eb8e9bf250b26e30b017d39a392c33973997656/datasette/views/base.py#L75-L99

  This leaves plugins like `datasette-graphql` having to implement their own versions of this logic, which is bad: https://github.com/simonw/datasette-graphql/issues/65

  > First check `view-database` - if that says `False` then disallow access, if it says `True` then allow access. If it says `None` check `view-instance`.

  This should become a supported API that plugins are encouraged to use.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1092/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDMwMTEzOTc=
title: import EX_CANTCREAT means datasette fails to work on Windows
state: closed
created_at: 2020-11-14T14:17:11Z
updated_at: 2020-12-05T19:35:04Z
closed_at: 2020-12-05T19:35:04Z
author_association: NONE
type: issue
state_reason: completed
body:
  Trying to use datasette 0.51.1 gives the following error:

  ```
  ImportError: cannot import name 'EX_CANTCREAT' from 'os' (C:\Users\drkan\AppData\Local\Programs\Python\Python39\lib\os.py)
  ```

  Looks like that code is only available on unix: https://docs.python.org/3/library/os.html#os.EX_CANTCREAT

  Removing the line makes it work fine (`EX_CANTCREAT` doesn't seem to be used anywhere?)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1094/reactions", "total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTIxMDU0NjEy
title: SQLite does not have case sensitive columns
state: open
created_at: 2020-11-14T20:12:32Z
updated_at: 2021-08-24T13:28:26Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/healthkit-to-sqlite/pulls/13
type: pull
body:
  This solves a weird issue when there is record with metadata key
  that is only different in letter cases.

  See the test for details.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDMyOTc1ODI=
state: closed
created_at: 2020-11-15T16:57:28Z
updated_at: 2021-02-11T22:13:17Z
closed_at: 2021-02-11T22:13:17Z
author_association: NONE
type: issue
state_reason: completed
title:
  evernote-to-sqlite on windows 10 give this error: TypeError: insert() got an unexpected keyword argument 'replace'

body:
  running evernote-to-sqlite 0.2 on windows 10. Command:

  evernote-to-sqlite enex evernote.db MyNotes.enex

  I get the followinng error:

  File "C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\evernote_to_sqlite\utils.py", line 46, in save_note
      note_id = db["notes"].insert(row, hash_id="id", replace=True, alter=True).last_pk
  TypeError: insert() got an unexpected keyword argument 'replace'

  Removing replace=True,

  Leads to below error:

    note_id = db["notes"].insert(row, hash_id="id", alter=True).last_pk
    File "C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\sqlite_utils\db.py", line 924, in insert
      return self.insert_all(
    File "C:\Users\marti\AppData\Roaming\Python\Python38\site-packages\sqlite_utils\db.py", line 1046, in insert_all
      result = self.db.conn.execute(sql, values)
  sqlite3.IntegrityError: UNIQUE constraint failed: notes.id

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDMzNTk2NDY=
title: TSV should be a default export option
state: open
created_at: 2020-11-15T22:24:02Z
updated_at: 2021-06-17T18:12:31Z
author_association: OWNER
body: Refs #1095
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1096/reactions", "total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTIxMjc2Mjk2
title: Use f-strings
state: closed
created_at: 2020-11-15T23:12:36Z
updated_at: 2020-11-15T23:24:24Z
closed_at: 2020-11-15T23:24:23Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1097
type: pull
body:
  Since Datasette now requires Python 3.6, how about some f-strings?

  I ran this in the `datasette` root checkout:
  ```
  pip install flynt
  flynt .
  black .
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1097/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDMzNzA5MDA=
title: Foreign key links break for compound foreign keys
state: closed
created_at: 2020-11-15T23:22:14Z
updated_at: 2020-11-29T19:50:31Z
closed_at: 2020-11-29T19:30:23Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Reported on Twitter here: https://twitter.com/ZaneSelvans/status/1328093641395548161

  > Maybe I'm doing something wrong here but the automatically generated links based foreign key relationships seem to be working here for utility_id_eia, but not for plant_id_eia & generator_id which seems odd: https://pudl-datasette-xl7xwcpe2a-uc.a.run.app/pudl/generators_eia860
  >
  > Right now it seems like they're trying to, but with only one of the two keys, so it gives "Error 500. You did not supply a value for binding 2." Maybe only create the links when it's a simple foreign key?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1098/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDMzNzExMDM=
title: Support linking to compound foreign keys
state: open
created_at: 2020-11-15T23:23:17Z
updated_at: 2023-01-25T00:58:26Z
author_association: OWNER
type: issue
body:
  Reported as a bug in #1098 because they caused 500 errors - but it would be even better if Datasette could hyperlink to related rows via compound foreign keys.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1099/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTIxMjg3OTk0
title: changes to allow for compound foreign keys
state: open
created_at: 2020-11-16T00:30:10Z
updated_at: 2023-01-25T18:47:18Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/203
type: pull
body:
  Add support for compound foreign keys, as per issue #117 

  Not sure if this is the right approach. In particular I'm unsure about:

   - the new `ForeignKey` class, which replaces the namedtuple in order to ensure that `column` and `other_column` are forced into tuples. The class does the job, but doesn't feel very elegant.
   - I haven't rewritten `guess_foreign_table` to take account of multiple columns, so it just checks for the first column in the foreign key definition. This isn't ideal.
   - I haven't added any ability to the CLI to add compound foreign keys, it's only in the python API at the moment.

  The PR also contains a minor related change that columns and tables are always quoted in foreign key definitions.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/203/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDM0MDAyMTY=
title: Error thrown: sqlite3.OperationalError: table users has no column named lastName
state: closed
created_at: 2020-11-16T01:21:18Z
updated_at: 2021-01-18T04:35:22Z
closed_at: 2021-01-18T04:35:22Z
author_association: NONE
type: issue
state_reason: completed
body:
  Just installed `swarm-to-sqlite-0.3.2` and tried according to the docs:

  ```
  Traceback (most recent call last):
    File "/usr/local/bin/swarm-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/cli.py", line 73, in cli
      save_checkin(checkin, db)
    File "/usr/local/lib/python3.9/site-packages/swarm_to_sqlite/utils.py", line 82, in save_checkin
      checkins_table.m2m("users", user, m2m_table="likes", pk="id")
    File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1914, in m2m
      id = other_table.insert(record, pk=pk, replace=True).last_pk
    File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1647, in insert
      return self.insert_all(
    File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1765, in insert_all
      self.insert_chunk(
    File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 1575, in insert_chunk
      result = self.db.execute(query, params)
    File "/usr/local/lib/python3.9/site-packages/sqlite_utils/db.py", line 200, in execute
      return self.conn.execute(sql, parameters)
  sqlite3.OperationalError: table users has no column named lastName
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDUzOTMyOTg=
title: Discussion: Adding support for fetching only fresh tweets
state: closed
created_at: 2020-11-18T07:01:48Z
updated_at: 2020-11-18T07:12:45Z
closed_at: 2020-11-18T07:12:45Z
author_association: NONE
type: issue
state_reason: completed
body:
  I think it'd be very useful if this tool has an option like `--incremental` to fetch only newer tweets. This way operations could complete very fast in sequential runs. I'd want to try to implement this feature if it seems OK for this tool's purpose. 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/52/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDc3MDIxNDQ=
title: Error on OPTIONS request to database
state: closed
created_at: 2020-11-20T18:16:43Z
updated_at: 2020-12-03T00:57:35Z
closed_at: 2020-12-03T00:50:17Z
author_association: NONE
type: issue
state_reason: completed
body:
  When I perform an OPTIONS request against a database or table datasette fails with an internal error.

  All these tests result in the traceback below.

  ```
  curl -XOPTIONS http://127.0.0.1:8001/test-db
  curl -XOPTIONS http://127.0.0.1:8001/test-db/table1
  curl -XOPTIONS http://127.0.0.1:8001/test-db/table1\?_search\=test
  ```

  ```
  Traceback (most recent call last):
    File "[path-to-python]/site-packages/datasette/app.py", line 1033, in route_path
      response = await view(request, send)
    File "[path-to-python]/site-packages/datasette/views/base.py", line 146, in view
      request, **request.scope["url_route"]["kwargs"]
    File "[path-to-python]/site-packages/datasette/views/base.py", line 118, in dispatch_request
      return await handler(request, *args, **kwargs)
  TypeError: object Response can't be used in 'await' expression
  ```

  Making the `options` function in the `DataView` class async fixed it for me.

  ```python
      async def options(self, request, *args, **kwargs):
          r = Response.text("ok")
          if self.ds.cors:
              r.headers["Access-Control-Allow-Origin"] = "*"
          return r
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1100/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTI1MzcxMDI5
title: fix import error if note has no "updated" element
state: closed
created_at: 2020-11-22T22:51:05Z
updated_at: 2021-02-11T22:34:06Z
closed_at: 2021-02-11T22:34:06Z
author_association: CONTRIBUTOR
pull_request: dogsheep/evernote-to-sqlite/pulls/8
type: pull
body:
  I got the following error when executing evernote-to-sqlite enex evernote.db evernote.enex
  ```   
  ... 
    File "evernote_to_sqlite/cli.py", line 31, in enex
      save_note(db, note)
    File "evernote_to_sqlite/utils.py", line 28, in save_note
      updated = note.find("updated").text
  AttributeError: 'NoneType' object has no attribute 'text'
  ```    

  Seems that in some cases the updated element is not added to the note, this is a part of the problematic note:

  ```    
  <created>20201019T074518Z</created>
  <note-attributes>
     <source>web.clip7</source>
     <source-application>webclipper.evernote</source-application>
  </note-attributes>
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDgzNzI0Njk=
title: ParseError: undefined entity &scaron;
state: closed
created_at: 2020-11-22T23:04:35Z
updated_at: 2021-02-11T22:10:55Z
closed_at: 2021-02-11T22:10:55Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  I encountered a parse error if the enex file contained &scaron; or &nbsp; 

  Run command: 
  evernote-to-sqlite enex evernote.db evernote.enex

  ```
  Traceback (most recent call last):
  ...
    File "evernote_to_sqlite/cli.py", line 31, in enex
      save_note(db, note)
    File "evernote_to_sqlite/utils.py", line 35, in save_note
      content = ET.tostring(ET.fromstring(content_xml)).decode("utf-8")
    File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1320, in XML
      parser.feed(text)
  xml.etree.ElementTree.ParseError: undefined entity &scaron;: line 3, column 35
  ```

  Workaround:
  ```
  sed -i 's/&scaron;//g' evernote.enex
  sed -i 's/&nbsp;//g' evernote.enex
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDkyODMwMzI=
title: register_output_renderer() should support streaming data
state: open
created_at: 2020-11-24T02:17:09Z
updated_at: 2023-01-21T22:07:19Z
author_association: OWNER
type: issue
body:
  > I'd like to implement this by first extending the `register_output_renderer()` hook to support streaming huge responses, then switching CSV to use the plugin hook in addition to TSV using it.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1096#issuecomment-732542285_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1101/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDkyODk2MTE=
title: Plugin testing docs should show datasette.client
state: closed
created_at: 2020-11-24T02:34:46Z
updated_at: 2020-11-29T07:47:22Z
closed_at: 2020-11-29T07:44:58Z
author_association: OWNER
body: https://docs.datasette.io/en/stable/testing_plugins.html currently shows how to use HTTPX directly.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1102/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDk5Nzk0NTQ=
title: Rename /-/config to /-/settings
state: closed
created_at: 2020-11-24T19:31:00Z
updated_at: 2020-11-24T20:19:20Z
closed_at: 2020-11-24T20:19:19Z
author_association: OWNER
body: As part of rebranding config to settings, see also #992.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1103/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDk5ODE2NjM=
title: config.json in directory config mode should be settings.json
state: closed
created_at: 2020-11-24T19:34:38Z
updated_at: 2020-11-24T20:37:42Z
closed_at: 2020-11-24T20:37:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Another knock-on effect of #992.

  https://github.com/simonw/datasette/blob/4bac9f18f9d04e5ed10f072502bcc508e365438e/docs/config.rst#L51-L55

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1104/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDk5ODIwMjI=
title: Rebrand config as settings
state: closed
created_at: 2020-11-24T19:35:12Z
updated_at: 2020-11-24T21:40:28Z
closed_at: 2020-11-24T21:40:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I realized I need a tracking ticket for this.

  I want to start splitting things like plugin configuration and default facets / sort order out of `metadata.json` - so I want to start calling those things configuration. But the term configuration is already used for the `--config` family of global settings. So I'm rebranding that type of configuration as settings to free up the name "configuration" for more run-time concerns (default sort order) and plugin configuration.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1105/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NDk5ODM4NTc=
title: Rebrand and redirect config.rst as settings.rst
state: closed
created_at: 2020-11-24T19:38:17Z
updated_at: 2020-11-24T21:39:58Z
closed_at: 2020-11-24T21:39:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'd like to redirect https://docs.datasette.io/en/stable/config.html to a new https://docs.datasette.io/en/stable/settings.html page too. I can use https://docs.readthedocs.io/en/stable/user-defined-redirects.html for that.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1105#issuecomment-733190827_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1106/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTAwNzkwODU=
title: Rename datasette.config() method to datasette.setting()
state: closed
created_at: 2020-11-24T21:24:11Z
updated_at: 2020-11-24T22:09:11Z
closed_at: 2020-11-24T22:06:38Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Part of #1105. Thankfully this isn't yet part of the documented public API on https://docs.datasette.io/en/stable/internals.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1107/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTAwODczNTA=
title: Configure /en/stable/config.html redirect when I ship 0.52
state: closed
created_at: 2020-11-24T21:39:19Z
updated_at: 2020-11-29T02:42:42Z
closed_at: 2020-11-29T02:42:42Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Like this:

  <img width="654" alt="Edit_Redirects___Read_the_Docs" src="https://user-images.githubusercontent.com/9599/100154418-60bf0d80-2e5a-11eb-92bf-527a14edb5c5.png">

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1106#issuecomment-733248437_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1108/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTAwODk4NDc=
title: Deprecate --config in Datasette 1.0 (in favour of --setting)
state: open
created_at: 2020-11-24T21:43:57Z
updated_at: 2020-12-17T22:07:49Z
author_association: OWNER
body: I added a deprecation warning to this in #992.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1109/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTI2ODQ3ODIz
title: Fixed conflicting CLI flags
state: closed
created_at: 2020-11-24T23:25:12Z
updated_at: 2022-08-21T21:11:56Z
closed_at: 2022-08-21T21:11:56Z
author_association: CONTRIBUTOR
pull_request: dogsheep/pocket-to-sqlite/pulls/7
type: pull
body:
  The `-a` used for the auth credentials and the shortened form of the `--all` flags were in conflict on the `fetch` command. To be consistent with other `-to-sqlite` libraries in the Dogsheep ecosystem, I removed the shortened form of the `--all` flag.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTAzMzAwMjk=
title: datasette publish option for installing extra apt-get packages
state: closed
created_at: 2020-11-25T03:03:43Z
updated_at: 2020-11-28T23:28:56Z
closed_at: 2020-11-25T03:05:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I ran into a need for this while playing with https://github.com/simonw/datasette-ripgrep - I need to install the `ripgrep` Ubuntu package when I deploy the plugin using Cloud Run.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1110/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTExOTUwMTc=
title: Accessing a database's `.json` is slow for very large SQLite files
state: open
created_at: 2020-11-26T00:27:27Z
updated_at: 2021-01-04T19:57:53Z
author_association: CONTRIBUTOR
type: issue
body:
  I have a SQLite DB that's pretty large, 23GB and something like 300 million rows. I expect that most queries I run on it will be slow, which is fine, but there are some things that Datasette does that makes working with the DB very slow. Specifically, when I access the `.json` metadata for a table (which I believe it comes from `datasette/views/database.py`, it takes 43 seconds for the request to come in:

  ```bash
  $ time curl localhost:9999/out.json
  {"database": "out", "size": 24291454976, "tables": [{"name": "PageviewsHour", "columns": ["file", "code", "page", "pageviews"], "primary_keys": [], "count": null, "hidden": false, "fts_table": null, "foreign_keys": {"incoming": [], "outgoing": [{"other_table": "PageviewsHourFiles", "column": "file", "other_column": "file_id"}]}, "private": false}, {"name": "PageviewsHourFiles", "columns": ["file_id", "filename", "sha256", "size", "day", "hour"], "primary_keys": ["file_id"], "count": null, "hidden": false, "fts_table": null, "foreign_keys": {"incoming": [{"other_table": "PageviewsHour", "column": "file_id", "other_column": "file"}], "outgoing": []}, "private": false}, {"name": "sqlite_sequence", "columns": ["name", "seq"], "primary_keys": [], "count": 1, "hidden": false, "fts_table": null, "foreign_keys": {"incoming": [], "outgoing": []}, "private": false}], "hidden_count": 0, "views": [], "queries": [], "private": false, "allow_execute_sql": true, "query_ms": 43340.23213386536}
  real	0m43.417s
  user	0m0.006s
  sys	0m0.016s
  ```

  I suspect this is because a `COUNT(*)` is happening under the hood, which, when I run it through sqlite directly, does take around the same time:

  ```bash
  $ time sqlite3 out.db < <(echo "select count(*) from PageviewsHour;")
  362794272

  real	0m44.523s
  user	0m2.497s
  sys	0m6.703s
  ```

  I'm using the `.json` request in the [Observable Datasette Client](https://observablehq.com/@asg017/datasette-client) to 1) verify that a link passed in is a reachable Datasette instance, and 2) a quick way to look at metadata for a db. A few different solutions I can think of:

  1. Have some other endpoint, like `/-/datasette.json` that the Observable Datasette client can fetch from to verify that the passed in URL is a valid Datasette (doesnt solve the slow problem, feel free to split this issue into 2)
  2. Have a way to turn off table counts when accessing a database's `.json` view, like `?no_count=1` or something
  3. Maybe have a timeout on the `table_counts()` function if it takes too long. which is odd, because it seems like it already does that (I think?), I can debug a little more if that's the case

  More than happy to debug further, or send a PR if you like one of the proposals above!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1111/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTI4OTk3NjE0
title: Fix --metadata doc usage
state: closed
created_at: 2020-11-28T19:19:51Z
updated_at: 2020-11-28T23:28:21Z
closed_at: 2020-11-28T19:53:48Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1112
type: pull
body:
  I stumbled on this while trying to figure out how to configure datasette-ripgrep via https://github.com/simonw/datasette-ripgrep/issues/15

  You may not want to update the changelog (those are annoying) so I added two commits in case that's easier. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1112/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTI3ODkxNTk=
title: 500 error on row page if query against foreign keys hits time limit
state: closed
created_at: 2020-11-28T23:20:08Z
updated_at: 2020-11-29T02:40:01Z
closed_at: 2020-11-28T23:23:31Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This page exhibited the following error: https://data.catalyst.coop/ferc1/f1_respondent_id/145

  `(OperationalError('interrupted'), 'select (select count(*) from f1_acb_epda where respondent_id=:id), (select count(*) from f1_accumdepr_prvsn where respondent_id=:id), (select count(*) from f1_accumdfrrdtaxcr where respondent_id=:id), (select count(*) from f1_adit_190_detail where respondent_id=:id), (select count(*) from f1_adit_190_notes where respondent_id=:id), (select count(*) from f1_adit_amrt_prop where respondent_id=:id), (select count(*) from f1_adit_other where respondent_id=:id), (select count(*) from f1_adit_other_prop where respondent_id=:id), (select count(*) from f1_allowances where respondent_id=:id), (select count(*) from f1_bal_sheet_cr where respondent_id=:id), (select count(*) from f1_capital_stock where respondent_id=:id), (select count(*) from f1_cash_flow where respondent_id=:id), (select count(*) from f1_cmmn_utlty_p_e where respondent_id=:id), (select count(*) from f1_comp_balance_db where respondent_id=:id), (select count(*) from f1_construction where respondent_id=:id), (select count(*) from f1_control_respdnt where respondent_id=:id), (select count(*) from f1_co_directors where respondent_id=:id), (select count(*) from f1_cptl_stk_expns where respondent_id=:id), (select count(*) from f1_csscslc_pcsircs where respondent_id=:id), (select count(*) from f1_dacs_epda where respondent_id=:id), (select count(*) from f1_dscnt_cptl_stk where respondent_id=:id), (select count(*) from f1_edcfu_epda where respondent_id=:id), (select count(*) from f1_elctrc_erg_acct where respondent_id=:id), (select count(*) from f1_elctrc_oper_rev where respondent_id=:id), (select count(*) from f1_elc_oper_rev_nb where respondent_id=:id), (select count(*) from f1_elc_op_mnt_expn where respondent_id=:id), (select count(*) from f1_electric where respondent_id=:id), (select count(*) from f1_envrnmntl_expns where respondent_id=:id), (select count(*) from f1_envrnmntl_fclty where respondent_id=:id), (select count(*) from f1_fuel where respondent_id=:id), (select count(*) from f1_general_info where respondent_id=:id), (select count(*) from f1_gnrt_plant where respondent_id=:id), (select count(*) from f1_important_chg where respondent_id=:id), (select count(*) from f1_incm_stmnt_2 where respondent_id=:id), (select count(*) from f1_income_stmnt where respondent_id=:id), (select count(*) from f1_miscgen_expnelc where respondent_id=:id), (select count(*) from f1_misc_dfrrd_dr where respondent_id=:id), (select count(*) from f1_mthly_peak_otpt where respondent_id=:id), (select count(*) from f1_mtrl_spply where respondent_id=:id), (select count(*) from f1_nbr_elc_deptemp where respondent_id=:id), (select count(*) from f1_nonutility_prop where respondent_id=:id), (select count(*) from f1_nuclear_fuel where respondent_id=:id), (select count(*) from f1_officers_co where respondent_id=:id), (select count(*) from f1_othr_dfrrd_cr where respondent_id=:id), (select count(*) from f1_othr_pd_in_cptl where respondent_id=:id), (select count(*) from f1_othr_reg_assets where respondent_id=:id), (select count(*) from f1_othr_reg_liab where respondent_id=:id), (select count(*) from f1_overhead where respondent_id=:id), (select count(*) from f1_pccidica where respondent_id=:id), (select count(*) from f1_plant_in_srvce where respondent_id=:id), (select count(*) from f1_pumped_storage where respondent_id=:id), (select count(*) from f1_purchased_pwr where respondent_id=:id), (select count(*) from f1_reconrpt_netinc where respondent_id=:id), (select count(*) from f1_reg_comm_expn where respondent_id=:id), (select count(*) from f1_respdnt_control where respondent_id=:id), (select count(*) from f1_retained_erng where respondent_id=:id), (select count(*) from f1_r_d_demo_actvty where respondent_id=:id), (select count(*) from f1_sales_by_sched where respondent_id=:id), (select count(*) from f1_sale_for_resale where respondent_id=:id), (select count(*) from f1_sbsdry_totals where respondent_id=:id), (select count(*) from f1_schedules_list where respondent_id=:id), (select count(*) from f1_security_holder where respondent_id=:id), (select count(*) from f1_slry_wg_dstrbtn where respondent_id=:id), (select count(*) from f1_substations where respondent_id=:id), (select count(*) from f1_taxacc_ppchrgyr where respondent_id=:id), (select count(*) from f1_unrcvrd_cost where respondent_id=:id), (select count(*) from f1_utltyplnt_smmry where respondent_id=:id), (select count(*) from f1_work where respondent_id=:id), (select count(*) from f1_xmssn_adds where respondent_id=:id), (select count(*) from f1_xmssn_elc_bothr where respondent_id=:id), (select count(*) from f1_xmssn_elc_fothr where respondent_id=:id), (select count(*) from f1_xmssn_line where respondent_id=:id), (select count(*) from f1_xtraordnry_loss where respondent_id=:id), (select count(*) from f1_audit_log where respondent_id=:id), (select count(*) from f1_load_file_names where respondent_id=:id), (select count(*) from f1_privilege where respondent_id=:id), (select count(*) from f1_sys_error_log where respondent_id=:id), (select count(*) from f1_hydro where respondent_id=:id), (select count(*) from f1_ident_attsttn where respondent_id=:id), (select count(*) from f1_steam where respondent_id=:id), (select count(*) from f1_leased where respondent_id=:id), (select count(*) from f1_sbsdry_detail where respondent_id=:id), (select count(*) from f1_plant where respondent_id=:id), (select count(*) from f1_long_term_debt where respondent_id=:id), (select count(*) from f1_106_2009 where respondent_id=:id), (select count(*) from f1_106a_2009 where respondent_id=:id), (select count(*) from f1_106b_2009 where respondent_id=:id), (select count(*) from f1_208_elc_dep where respondent_id=:id), (select count(*) from f1_231_trn_stdycst where respondent_id=:id), (select count(*) from f1_324_elc_expns where respondent_id=:id), (select count(*) from f1_325_elc_cust where respondent_id=:id), (select count(*) from f1_331_transiso where respondent_id=:id), (select count(*) from f1_338_dep_depl where respondent_id=:id), (select count(*) from f1_397_isorto_stl where respondent_id=:id), (select count(*) from f1_398_ancl_ps where respondent_id=:id), (select count(*) from f1_399_mth_peak where respondent_id=:id), (select count(*) from f1_400_sys_peak where respondent_id=:id), (select count(*) from f1_400a_iso_peak where respondent_id=:id), (select count(*) from f1_429_trans_aff where respondent_id=:id), (select count(*) from f1_allowances_nox where respondent_id=:id), (select count(*) from f1_cmpinc_hedge_a where respondent_id=:id), (select count(*) from f1_cmpinc_hedge where respondent_id=:id), (select count(*) from f1_email where respondent_id=:id), (select count(*) from f1_rg_trn_srv_rev where respondent_id=:id), (select count(*) from f1_s0_filing_log where respondent_id=:id), (select count(*) from f1_security where respondent_id=:id)', {'id': '145'})`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1113/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTI5MDkwNTYw
title: use jsonify_if_need for sql updates
state: closed
created_at: 2020-11-29T10:49:00Z
updated_at: 2020-12-08T17:49:42Z
closed_at: 2020-12-08T17:49:42Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/204
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/204/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTI5NjY0NzY=
title: --load-extension=spatialite not working with datasetteproject/datasette docker image
state: closed
created_at: 2020-11-29T17:35:20Z
updated_at: 2022-01-20T21:29:42Z
closed_at: 2020-11-29T17:37:45Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/commit/6aa5886379dd9017215904fb28567b80018902f9 added the `--load-extension=spatialite` shortcut looking for the extension in these places:

  https://github.com/simonw/datasette/blob/12877d7a48e2aa28bb5e780f929a218f7265d849/datasette/utils/__init__.py#L56-L60

  However, in the datasetteproject/datasette docker image the file is at `/usr/local/lib/mod_spatialite.so`.

  This results in the example command [here](https://docs.datasette.io/en/stable/installation.html#loading-spatialite) failing:

  ```
  % docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=spatialite
  Error: Could not find SpatiaLite extension
  ```

  But it does work when given an explicit path:

  ```
  % docker run --rm -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette datasette -p 8001 -h 0.0.0.0 /mnt/data.db --load-extension=/usr/local/lib/mod_spatialite.so
  INFO:     Started server process [1]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
  ...
  ```

  Perhaps `SPATIALITE_PATHS` should include `/usr/local/lib/mod_spatialite.so`?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1114/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTI5OTUyMjc=
title: SpatiaLite error could suggest --load-extension=spatialite
state: closed
created_at: 2020-11-29T20:05:07Z
updated_at: 2022-01-20T21:29:42Z
closed_at: 2020-11-29T20:13:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/cli.py#L533-L548

  This could use the `find_spatialite()` function and, if it finds something, suggest the user use `--load-extension=spatialite`

  https://github.com/simonw/datasette/blob/242bc89fdf2e775e340d69a4e851b3a9accb31c6/datasette/utils/__init__.py#L1015-L1019

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1115/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTMwMDA0MDU=
title: Command for fetching file contents
state: open
created_at: 2020-11-29T20:31:04Z
updated_at: 2020-11-30T00:36:09Z
author_association: MEMBER
type: issue
body:
  Something like this:

      github-to-sqlite files github.db simonw/datasette

  This would fetch all files from the `main` branch into a `files` table.

  Additional options could handle things like pulling files from a branch or tag, or just pulling files that match a specific glob or that exist in a specific directory.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/53/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTMwMjYwMDM=
title: github-to-sqlite workflows command
state: closed
created_at: 2020-11-29T21:56:42Z
updated_at: 2020-11-29T22:08:46Z
closed_at: 2020-11-29T21:57:17Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  A command that fetches the YAML workflows for different repos, parses them and stores them in relational tables would be really useful for maintaining larger numbers of workflows.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/54/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTMwMjYzODg=
title: github-to-sqlite workflows does not correctly replace existing records
state: closed
created_at: 2020-11-29T21:58:43Z
updated_at: 2020-11-29T23:48:50Z
closed_at: 2020-11-29T23:48:50Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Following #54 - see this TODO: https://github.com/dogsheep/github-to-sqlite/blob/1b23ce11953f9f59c0161ea1f99188b55b5ea11c/github_to_sqlite/utils.py#L700

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/55/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTMxMjIwODI=
title: Link to example tables from the README
state: closed
created_at: 2020-11-30T04:01:51Z
updated_at: 2020-11-30T04:10:27Z
closed_at: 2020-11-30T04:10:27Z
author_association: MEMBER
body: Would help demonstrate how the tool works.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/56/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTM2NjgxNzc=
title: GENERATED column support
state: closed
created_at: 2020-11-30T17:33:47Z
updated_at: 2020-11-30T21:29:59Z
closed_at: 2020-11-30T21:29:59Z
author_association: NONE
type: issue
state_reason: completed
body:
  I think this is a feature request... perhaps I should just try to contribute it myself, but thought I'd check in case support is planned already.

  For a table with the following schema, datasette 0.51.1 doesn't pick up the GENERATED columns and the column list only contains `(rowid, body)`  If I edit the SQL and select the generated columns, it will happily show them.

  At first glance it appears that [`def table_column_details(conn, table):`](https://github.com/simonw/datasette/blob/4777362bf2692bc72b221ec47c3e6216151d1b89/datasette/utils/__init__.py#L575) would have to be refactored to use a different methodology to get the columns, since `PRAGMA table_info(deeds);` returns just `0|body|TEXT|0||0` so maybe it wouldn't be worth it.

  ```
  CREATE TABLE deeds (
      body TEXT,
      id INT GENERATED ALWAYS AS (json_extract(body, '$.id')) STORED,
      consideration INT GENERATED ALWAYS AS (json_extract(body, '$.consideration')) STORED
  );
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1116/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTI5NzgzMjc1
title: Support for generated columns
state: closed
created_at: 2020-11-30T20:10:46Z
updated_at: 2020-11-30T22:23:19Z
closed_at: 2020-11-30T21:29:58Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1117
type: pull
body:
  Refs #1116. My first attempt at this worked on my laptop but broke in CI, so I'm going to iterate on it in a pull request instead.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1117/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTM3ODgyNjE=
title: messagge_is_html typo
state: closed
created_at: 2020-11-30T20:43:22Z
updated_at: 2020-11-30T21:24:28Z
closed_at: 2020-11-30T21:24:28Z
author_association: OWNER
body: https://ripgrep.datasette.io/-/ripgrep?pattern=messagge_is_html
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1118/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTM4NzY4MDg=
title: Include generated columns in fixtures.db, if SQLite version supports it
state: closed
created_at: 2020-11-30T23:25:31Z
updated_at: 2020-12-01T00:41:14Z
closed_at: 2020-12-01T00:28:04Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In #1117 I added a one-off test that creates a DB with generated columns in it:

  https://github.com/simonw/datasette/blob/461670a0b87efa953141b449a9a261919864ceb3/tests/test_api.py#L1933-L1964

  If this table was conditionally added to `fixtures.db` (if SQLite supports the feature) it would be available in the demo at https://latest.datasette.io/

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1119/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTI5ODg3ODYx
title: generated_columns table in fixtures.py
state: closed
created_at: 2020-12-01T00:17:19Z
updated_at: 2020-12-01T00:28:03Z
closed_at: 2020-12-01T00:28:02Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1120
body: Refs #1119
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1120/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTQxNzg3ODA=
title: Table actions cog is misaligned
state: closed
created_at: 2020-12-01T08:41:25Z
updated_at: 2020-12-03T01:03:19Z
closed_at: 2020-12-03T00:33:36Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  At the moment it looks like this
  https://datasette-graphql-demo.datasette.io/github/repos

  ![image](https://user-images.githubusercontent.com/3243482/100716533-e6e2d300-33c9-11eb-866e-1e83ba228bf5.png)

  Adding a few flex statements fixes the alignment and centers `h1` text and the cog icon vertically.
  ![image](https://user-images.githubusercontent.com/3243482/100716605-f8c47600-33c9-11eb-8d69-0e37499cf641.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1121/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTMwMTI1Njk1
title: Fix misaligned table actions cog
state: closed
created_at: 2020-12-01T08:41:46Z
updated_at: 2020-12-03T10:56:40Z
closed_at: 2020-12-03T00:33:37Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1122
body: Fixes https://github.com/simonw/datasette/issues/1121
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1122/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTU3MjEyNzU=
title: Table actions hook are order dependent, should not be
state: closed
created_at: 2020-12-03T00:47:12Z
updated_at: 2020-12-03T00:50:17Z
closed_at: 2020-12-03T00:50:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Got this error: https://github.com/simonw/datasette/runs/1489770800
  ```
  >       assert get_table_actions_links(response_2.text) == [
              {"label": "From async", "href": "/"},
              {"label": "Database: fixtures", "href": "/"},
              {"label": f"Table: {table_or_view}", "href": "/"},
          ]
  E       AssertionError: assert [{'href': '/'...'From async'}] == [{'href': '/'...: facetable'}]
  E         At index 0 diff: {'label': 'Database: fixtures', 'href': '/'} != {'label': 'From async', 'href': '/'}
  E         Use -v to get the full diff
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1123/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTY0Mzk1MTY=
title: Datasette on Amazon Linux on ARM returns 404 for static assets
state: closed
created_at: 2020-12-03T18:20:37Z
updated_at: 2020-12-03T21:42:02Z
closed_at: 2020-12-03T21:10:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Very weird bug this one. Steps to reproduce:
  ```
  # I started a amzn2-ami-hvm-2.0.20201126.0-arm64-gp2 t4g.micro instance
  ec2 % ssh -i simonw-ec2.pem ec2-user@ec2-18-219-238-192.us-east-2.compute.amazonaws.com
  [ec2-user@ip-172-31-30-7 ~]$ sudo yum install python3
  Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
  ...
  Is this ok [y/d/N]: y
  Downloading packages:
  (1/4): python3-3.7.9-1.amzn2.0.1.aarch64.rpm                                                                                        |  72 kB  00:00:00     
  (2/4): python3-pip-9.0.3-1.amzn2.0.2.noarch.rpm                                                                                     | 1.9 MB  00:00:00     
  (3/4): python3-setuptools-38.4.0-3.amzn2.0.6.noarch.rpm                                                                             | 617 kB  00:00:00     
  (4/4): python3-libs-3.7.9-1.amzn2.0.1.aarch64.rpm                                                                                   | 9.1 MB  00:00:00     
  -----------------------------------------------------------------------------------------------------------------------------------------------------------
  Total                                                                                                                       68 MB/s |  12 MB  00:00:00     
  Running transaction check
  Running transaction test
  Transaction test succeeded
  Running transaction
    Installing : python3-pip-9.0.3-1.amzn2.0.2.noarch                                                                                                    1/4 
    Installing : python3-3.7.9-1.amzn2.0.1.aarch64                                                                                                       2/4 
    Installing : python3-setuptools-38.4.0-3.amzn2.0.6.noarch                                                                                            3/4 
    Installing : python3-libs-3.7.9-1.amzn2.0.1.aarch64                                                                                                  4/4 
    Verifying  : python3-setuptools-38.4.0-3.amzn2.0.6.noarch                                                                                            1/4 
    Verifying  : python3-pip-9.0.3-1.amzn2.0.2.noarch                                                                                                    2/4 
    Verifying  : python3-3.7.9-1.amzn2.0.1.aarch64                                                                                                       3/4 
    Verifying  : python3-libs-3.7.9-1.amzn2.0.1.aarch64                                                                                                  4/4 

  Installed:
    python3.aarch64 0:3.7.9-1.amzn2.0.1                                                                                                                      

  Dependency Installed:
    python3-libs.aarch64 0:3.7.9-1.amzn2.0.1          python3-pip.noarch 0:9.0.3-1.amzn2.0.2          python3-setuptools.noarch 0:38.4.0-3.amzn2.0.6         

  Complete!
  [ec2-user@ip-172-31-30-7 ~]$ python3 -m pip install --user pipx
  Collecting pipx
    Downloading https://files.pythonhosted.org/packages/42/8a/8447ec14562c5d97afbee54ec390864718bccce9dfb0506c8c77852489d3/pipx-0.15.6.0-py3-none-any.whl (43kB)
      100% |████████████████████████████████| 51kB 3.8MB/s 
  Collecting packaging>=20.0 (from pipx)
    Downloading https://files.pythonhosted.org/packages/28/87/8edcf555adaf60d053ead881bc056079e29319b643ca710339ce84413136/packaging-20.7-py2.py3-none-any.whl
  Collecting argcomplete<2.0,>=1.9.4 (from pipx)
    Downloading https://files.pythonhosted.org/packages/e3/d0/ee7fc6ceac8957196def9bfa3b955d02163058defd3edd51ef7b1ff2769e/argcomplete-1.12.2-py2.py3-none-any.whl
  Collecting userpath>=1.4.1 (from pipx)
    Downloading https://files.pythonhosted.org/packages/fb/f1/faca8a3cc86bd2223aaf72edd222bc31d6ae2eea5feaf17a144634a92e6d/userpath-1.4.1-py2.py3-none-any.whl
  Collecting pyparsing>=2.0.2 (from packaging>=20.0->pipx)
    Downloading https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl (67kB)
      100% |████████████████████████████████| 71kB 8.8MB/s 
  Collecting importlib-metadata<4,>=0.23; python_version == "3.7" (from argcomplete<2.0,>=1.9.4->pipx)
    Downloading https://files.pythonhosted.org/packages/99/c7/4ccf2baa455613aa9e61372365aba8594ff2806c82189c31a6c65e7c679e/importlib_metadata-3.1.1-py3-none-any.whl
  Collecting click (from userpath>=1.4.1->pipx)
    Downloading https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl (82kB)
      100% |████████████████████████████████| 92kB 10.5MB/s 
  Collecting distro; platform_system == "Linux" (from userpath>=1.4.1->pipx)
    Downloading https://files.pythonhosted.org/packages/25/b7/b3c4270a11414cb22c6352ebc7a83aaa3712043be29daa05018fd5a5c956/distro-1.5.0-py2.py3-none-any.whl
  Collecting zipp>=0.5 (from importlib-metadata<4,>=0.23; python_version == "3.7"->argcomplete<2.0,>=1.9.4->pipx)
    Downloading https://files.pythonhosted.org/packages/41/ad/6a4f1a124b325618a7fb758b885b68ff7b058eec47d9220a12ab38d90b1f/zipp-3.4.0-py3-none-any.whl
  Installing collected packages: pyparsing, packaging, zipp, importlib-metadata, argcomplete, click, distro, userpath, pipx
  Successfully installed argcomplete-1.12.2 click-7.1.2 distro-1.5.0 importlib-metadata-3.1.1 packaging-20.7 pipx-0.15.6.0 pyparsing-2.4.7 userpath-1.4.1 zipp-3.4.0
  [ec2-user@ip-172-31-30-7 ~]$ python3 -m pipx ensurepath
  /home/ec2-user/.local/bin is already in PATH.
  /home/ec2-user/.local/bin is already in PATH.

  All pipx binary directories have been added to PATH. If you are sure
  you want to proceed, try again with the '--force' flag.

  Otherwise pipx is ready to go! ✨ 🌟 ✨
  [ec2-user@ip-172-31-30-7 ~]$ pipx install datasette
    installed package datasette 0.52.2, Python 3.7.9
    These apps are now globally available
      - datasette
  done! ✨ 🌟 ✨
  [ec2-user@ip-172-31-30-7 ~]$ datasette --version
  datasette, version 0.52.2
  [ec2-user@ip-172-31-30-7 ~]$ datasette --get /-/static/app.css
  404
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1124/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTY2MjI2NDg=
title: Show pysqlite3 version on /-/versions
state: closed
created_at: 2020-12-03T21:57:23Z
updated_at: 2020-12-04T04:16:57Z
closed_at: 2020-12-04T04:16:57Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This code can use `pysqlite3` if available. The version should be exposed on `/-/versions`.

  https://github.com/simonw/datasette/blob/4cce5516661b24afeddaf35bee84b00fbf5c7f89/datasette/utils/sqlite.py#L1-L4

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1125/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTY3NjE5NjM=
title: Switch to google-github-actions/setup-gcloud for demo deploy
state: closed
created_at: 2020-12-04T03:11:24Z
updated_at: 2020-12-04T03:51:38Z
closed_at: 2020-12-04T03:51:38Z
author_association: OWNER
type: issue
state_reason: completed
body:
  That workflow is showing warnings: https://github.com/simonw/datasette/actions/runs/399400077

  > Thank you for using setup-gcloud Action. GoogleCloudPlatform/github-actions/setup-gcloud has been deprecated, please switch to google-github-actions/setup-gcloud.

  https://github.com/google-github-actions/setup-gcloud has a Cloud Run example here: https://github.com/google-github-actions/setup-gcloud/blob/master/example-workflows/cloud-run/.github/workflows/cloud-run.yml

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1126/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTY4MTgyNTA=
title: Make the custom SQL query text box larger or resizable
state: closed
created_at: 2020-12-04T05:37:11Z
updated_at: 2021-06-02T04:29:06Z
closed_at: 2021-06-02T04:28:55Z
author_association: NONE
type: issue
state_reason: completed
body:
  The text entry field for custom SQL queries is too small to display a moderately complex query, especially when it's been formatted. Would it be easy to make the textbox resizable by the user rather than having a fixed height?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1127/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTMyMzQyMDI1
title: Fix startup error on windows
state: closed
created_at: 2020-12-04T07:12:26Z
updated_at: 2020-12-06T08:41:45Z
closed_at: 2020-12-05T19:35:04Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1128
type: pull
body:
  Fixes https://github.com/simonw/datasette/issues/1094

  This import isn't used at all, and causes error on startup on Windows.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1128/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTY4NzU4Mjc=
title: Fix footer to the bottom of the page
state: open
created_at: 2020-12-04T07:28:07Z
updated_at: 2020-12-04T16:04:29Z
author_association: CONTRIBUTOR
type: issue
body:
  Footer doesn't stick to the bottom if the body content isn't long enough to reach the end of viewport.

  ![before & after](https://user-images.githubusercontent.com/3243482/101134785-f6595a80-361b-11eb-81ce-b8b5cb9c5bc2.png)


  This can be fixed using flexbox.

  ```css
  body {
    min-height: 100vh;
    display: flex;
    flex-direction: column;
  }

  .content {
    flex-grow: 1;
  }
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1129/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTMyMzQ4OTE5
title: Fix footer not sticking to bottom in short pages
state: open
created_at: 2020-12-04T07:29:01Z
updated_at: 2021-06-15T13:27:48Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1130
body: Fixes https://github.com/simonw/datasette/issues/1129
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1130/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTc0ODE5NDk=
title: "datasette inspect" outputs invalid JSON if an error is logged
state: closed
created_at: 2020-12-05T00:00:45Z
updated_at: 2020-12-05T20:48:34Z
closed_at: 2020-12-05T05:21:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See https://github.com/simonw/register-of-members-interests/issues/6:
  ```
  % datasette inspect regmem.db                       
  ERROR: conn=<sqlite3.Connection object at 0x10d2a9c60>, sql = 'select count(*) from [items_fts]', params = None: SQL logic error
  {
    "regmem": {
      "hash": "6fde27e3dea80d6b65f2ac7f89cd8448980fee8c91b505ba29c311ba0393317f",
      "size": 936198144,
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1131/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTg4OTk1ODE=
title: New filter: array does not contain
state: closed
created_at: 2020-12-07T22:28:20Z
updated_at: 2020-12-07T22:50:36Z
closed_at: 2020-12-07T22:41:09Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I want to see all of my GitHub repos that are tagged `datasette-plugin` but are NOT tagged `datasette-io`:

  https://github-to-sqlite.dogsheep.net/github/repos?_facet=owner&owner=9599&_facet_array=topics&topics__arraycontains=datasette-plugin&topics__arraycontains=datasette-io#facet-topics

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1132/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTg5NDQwMDY=
title: --readme throws 404 error if README does not exist in repo
state: closed
created_at: 2020-12-07T23:58:49Z
updated_at: 2020-12-16T18:17:54Z
closed_at: 2020-12-16T18:17:54Z
author_association: MEMBER
body: It should fail silently (populate the column with a null) instead.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/57/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NTk2OTU3ODA=
title: Option to omit header row in CSV export
state: closed
created_at: 2020-12-08T18:54:46Z
updated_at: 2021-06-17T18:12:31Z
closed_at: 2020-12-10T23:28:51Z
author_association: OWNER
body: `?_header=off` - for symmetry with existing option `?_nl=on`.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1133/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjAzMTI1Nzk=
title: "_searchmode=raw" throws an index out of range error when combined with "_search_COLUMN"
state: closed
created_at: 2020-12-09T13:05:37Z
updated_at: 2020-12-10T05:57:17Z
closed_at: 2020-12-09T19:56:55Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon!
  Maybe it's just me, but when [using _searchmode=raw (trying to enable wildcard-searching) in combination with the "_search_COLUMN"-table argument](https://byraadsarkivet.aarhus.dk/db/cases?_searchmode=raw&_search_title=sundhedsfrem*), I get a list index out of range error. [When combining with the simpler "_search"-argument everything works, including wildcard-seaches.](https://byraadsarkivet.aarhus.dk/db/cases?_search=sundhedsfrem*&_searchmode=raw). Here's the traceback:

  ```
  Traceback (most recent call last):
    File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py", line 122, in route_path
      return await view(new_scope, receive, send)
    File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/utils/asgi.py", line 196, in view
      request, **scope["url_route"]["kwargs"]
    File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py", line 204, in get
      request, database, hash, correct_hash_provided, **kwargs
    File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/base.py", line 342, in view_get
      request, database, hash, **kwargs
    File "/Users/cjk/.local/share/virtualenvs/minutes-jMDZ8Ssk/lib/python3.7/site-packages/datasette/views/table.py", line 393, in data
      search_col = key.split("_search_", 1)[1]
  IndexError: list index out of range

  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1134/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjA2MDU4ODI=
title: Feature: --create option to create database file if it does not yet exist
state: closed
created_at: 2020-12-09T19:23:58Z
updated_at: 2021-01-24T21:19:39Z
closed_at: 2020-12-09T19:45:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'd like to be able to tell people to run the following in the Datasette documentation to get started:

      brew install datasette
      datasette install datasette-upload-csvs
      datasette data.db --create --root --open

  This would give them a local Datasette instance with the ability to drag-and-drop CSV files directly into it.

  Just one catch: I don't want to have to talk them through creating an empty SQLite database file. So I want to add a new `--create` option which means "If any of the database files passed on the command-line do not yet exist, create them".

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1135/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjA2MjEzNTY=
title: Establish pattern for release branches to support bug fixes
state: closed
created_at: 2020-12-09T19:48:18Z
updated_at: 2020-12-09T20:17:02Z
closed_at: 2020-12-09T20:14:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I want to fix the bug in #1134 and ship it as Datasette 0.52.5 - but the `main` branch now has a feature in it (4c25b035b2370983c8dd5e0c8762e9154e379774 added `arraynotcontains`, #1132).

  I'm not ready for a feature release, so instead I want to release 0.52.5 with just that bug fix.

  This is the first time I will have shipped a release from a branch. I need to establish that pattern and add it to the documentation in https://docs.datasette.io/en/stable/contributing.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1136/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjA5NjA1NTk=
title: sqlite3.OperationalError: near "(": syntax error
state: closed
created_at: 2020-12-10T06:44:40Z
updated_at: 2020-12-10T19:18:22Z
closed_at: 2020-12-10T07:24:23Z
author_association: NONE
type: issue
state_reason: completed
body:
  The sqlite version is 3.22.0 2018-01-22 18:45:57 0c55d179733b46d8d0ba4d88e01a25e10677046ee3da1d5b1581e86726f2alt1
  sqlite-utils, version 3.0

  It fails here: https://github.com/kaihendry/aws-partners-datasette/runs/1528432635?check_suite_focus=true

  I'm not sure where the problem is, since it works _fine locally_ on Archlinux system running 3.34.0 2020-12-01 16:14:00 a26b6597e3ae272231b96f9982c3bcc17ddec2f2b6eb4df06a224b91089fed5b


  https://github.com/kaihendry/aws-partners-datasette/blob/main/create-summary-view.sh

  Maybe I need to bump up from ubuntu-latest to ?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/205/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjE3MDY4NTg=
title: Update README to reflect new datasette.io site
state: closed
created_at: 2020-12-10T23:22:06Z
updated_at: 2020-12-10T23:28:50Z
closed_at: 2020-12-10T23:28:50Z
author_association: OWNER
body: Can finally close #659.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1137/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjE3MTMwNzk=
title: "Powered by Datasette" should link to new datasette.io site
state: closed
created_at: 2020-12-10T23:33:41Z
updated_at: 2020-12-15T02:28:10Z
closed_at: 2020-12-10T23:37:14Z
author_association: OWNER
body: https://datasette.io/
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1138/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjE5MTU3OTA=
title: sqlite-utils should suggest --csv if JSON parsing fails
state: closed
created_at: 2020-12-11T05:17:56Z
updated_at: 2021-10-30T15:52:17Z
closed_at: 2021-01-03T18:42:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  ~ % gsutil cat gs://ossf-criticality-score/python_top_200.csv | sqlite-utils insert /tmp/crit.db crit -
  ...
    File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
      raise JSONDecodeError("Expecting value", s, err.value) from None
  json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  ```
  A nicer error message here would be one that says the JSON is invalid but suggests that maybe you could try `--csv`.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/206/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjMyMDc5NDg=
title: Default styling for bullet point lists
state: closed
created_at: 2020-12-12T02:49:33Z
updated_at: 2021-03-29T00:14:05Z
closed_at: 2021-03-29T00:14:05Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I just noticed that https://datasette.io/content/recent_releases (which uses `datasette-render-markdown`) is missing its bullet points:

  <img width="864" alt="content__recent_releases__399_rows" src="https://user-images.githubusercontent.com/9599/101970823-95b0bb80-3be1-11eb-9512-9274cef64cf3.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1141/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjMyODM2MTY=
title: sqlite-utils analyze-tables command
state: closed
created_at: 2020-12-12T04:33:12Z
updated_at: 2020-12-13T07:25:23Z
closed_at: 2020-12-13T07:20:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  A command which analyzes a table (potentially taking quite a while if the table is large) and outputs information for each column - things like:

  - How many unique values does this column have?
  - How many null rows?
  - How many blank rows? (defined as empty string)
  - What are the 10 most common values?
  - What are the 10 least common values?

  The command can output this information to the terminal, but it should also provide an option for writing the information to a database table so it can be explored later.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/207/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTM3NzkxNjc1
title: sqlite-utils analyze-tables command and table.analyze_column() method
state: closed
created_at: 2020-12-12T05:27:49Z
updated_at: 2020-12-13T07:20:16Z
closed_at: 2020-12-13T07:20:12Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/208
type: pull
body:
  Refs #207

  - [x] Improve design of CLI output
  - [x] Truncate long values in least/most common
  - [x] Add a `-c` column selection option
  - [x] Tests
  - [x] Documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/208/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjMzNjE0NTg=
title: "Stream all rows" is not at all obvious
state: open
created_at: 2020-12-12T06:24:57Z
updated_at: 2021-06-17T18:12:31Z
author_association: OWNER
type: issue
body:
  Got a question about how to download all rows - the current option isn't at all clear.

  <img width="668" alt="loans__ppp_loans__9_511_rows_where_where_search_matches__tech__sorted_by_rowid" src="https://user-images.githubusercontent.com/9599/101977057-ac660b00-3bff-11eb-88f4-c93ffd03d3e0.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1142/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjQwNTkyMzU=
title: More flexible CORS support in core, to encourage good security practices
state: open
created_at: 2020-12-12T17:06:35Z
updated_at: 2022-02-13T17:41:17Z
author_association: NONE
type: issue
body:
  It would be nice if the `--cors` option accepted an origin regex to more securely allow secure local development.

  As an example, Observable notebooks namespace every user's notebooks by their username and user content is served from username.observableusercontent.com, so you would set `--cors-origin username.observableusercontent.com` to restrict access to a local development Datasette instance to only your own notebooks, rather than exposing the data to any website that makes a request.

  Thank you for all of your work on Datasette!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1143/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjU2MzczMjQ=
title: JavaScript to help plugins interact with the fragment part of the URL
state: open
created_at: 2020-12-13T20:36:06Z
updated_at: 2020-12-14T14:47:11Z
author_association: OWNER
type: issue
body:
  Suggested by Markus Holtermann on Twitter, who is building https://github.com/MarkusH/datasette-chartjs

  > I've been looking at datasette-vega for how you persist chart settings between form submissions. I've adopted that for datasette-chartjs. Any thoughts on adding a public JS API to #datasette itself, that plugins can rely on?
  >
  > I'm talking about functions like onFragmentChange, serialize, unserialize, ... That turn an object into a URL encoded string and put it into the location's hash. And also updating all links/forms automatically.
  >
  > Essentially, a plugins could do something like `document.datasette.setConfigValue('prefix', 'foo', 'bar')` and `.getConfigValue('prefix', 'foo')`. And the functions would take care of updating document.location.hash, all (necessary) a.href and form.action

  https://twitter.com/m_holtermann/status/1338183973311295492

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1144/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjYxNTY4NzU=
title: Test failure with sqlite 3.34 in test_cli.py::test_optimize
state: closed
created_at: 2020-12-14T08:58:18Z
updated_at: 2021-01-01T23:52:46Z
closed_at: 2021-01-01T23:52:46Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  pytest output:
  ```
  ...
  ============================== short test summary info ===============================
  FAILED tests/test_cli.py::test_optimize[tables0] - assert 1662976 < 1662976
  FAILED tests/test_cli.py::test_optimize[tables1] - assert 1667072 < 1662976
  ===================== 2 failed, 538 passed, 3 skipped in 34.32s ======================
  ```

  Came across this while packaging `sqlite-utils` for NixOS, but it can be recreated it using the `alpine:edge` docker image as well as follows:

  ```
  docker run --rm -it alpine:edge /bin/sh
  # apk update && apk add git sqlite python3 gcc python3-dev musl-dev && python3 -m ensurepip
  # git clone https://github.com/simonw/sqlite-utils.git
  # cd sqlite-utils/
  # pip3 install -e .[test]
  # pytest
  ```

  This definitely works on sqlite v3.33.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/209/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTM5NDg5NTI1
title: Update pytest requirement from <6.2.0,>=5.2.2 to >=5.2.2,<6.3.0
state: closed
created_at: 2020-12-14T14:22:16Z
updated_at: 2021-01-24T21:20:29Z
closed_at: 2020-12-16T21:44:39Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1145
type: pull
body:
  Updates the requirements on [pytest](https://github.com/pytest-dev/pytest) to permit the latest version.
  <details>
  <summary>Release notes</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/releases">pytest's releases</a>.</em></p>
  <blockquote>
  <h2>6.2.0</h2>
  <h1>pytest 6.2.0 (2020-12-12)</h1>
  <h2>Breaking Changes</h2>
  <ul>
  <li><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7808">#7808</a>: pytest now supports python3.6+ only.</li>
  </ul>
  <h2>Deprecations</h2>
  <ul>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7469">#7469</a>: Directly constructing/calling the following classes/functions is now deprecated:</p>
  <ul>
  <li><code>_pytest.cacheprovider.Cache</code></li>
  <li><code>_pytest.cacheprovider.Cache.for_config()</code></li>
  <li><code>_pytest.cacheprovider.Cache.clear_cache()</code></li>
  <li><code>_pytest.cacheprovider.Cache.cache_dir_from_config()</code></li>
  <li><code>_pytest.capture.CaptureFixture</code></li>
  <li><code>_pytest.fixtures.FixtureRequest</code></li>
  <li><code>_pytest.fixtures.SubRequest</code></li>
  <li><code>_pytest.logging.LogCaptureFixture</code></li>
  <li><code>_pytest.pytester.Pytester</code></li>
  <li><code>_pytest.pytester.Testdir</code></li>
  <li><code>_pytest.recwarn.WarningsRecorder</code></li>
  <li><code>_pytest.recwarn.WarningsChecker</code></li>
  <li><code>_pytest.tmpdir.TempPathFactory</code></li>
  <li><code>_pytest.tmpdir.TempdirFactory</code></li>
  </ul>
  <p>These have always been considered private, but now issue a deprecation warning, which may become a hard error in pytest 7.0.0.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7530">#7530</a>: The <code>--strict</code> command-line option has been deprecated, use <code>--strict-markers</code> instead.</p>
  <p>We have plans to maybe in the future to reintroduce <code>--strict</code> and make it an encompassing flag for all strictness
  related options (<code>--strict-markers</code> and <code>--strict-config</code> at the moment, more might be introduced in the future).</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7988">#7988</a>: The <code>@pytest.yield_fixture</code> decorator/function is now deprecated. Use pytest.fixture instead.</p>
  <p><code>yield_fixture</code> has been an alias for <code>fixture</code> for a very long time, so can be search/replaced safely.</p>
  </li>
  </ul>
  <h2>Features</h2>
  <ul>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/5299">#5299</a>: pytest now warns about unraisable exceptions and unhandled thread exceptions that occur in tests on Python&gt;=3.8.
  See unraisable for more information.</p>
  </li>
  <li>
  <p><a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/7425">#7425</a>: New pytester fixture, which is identical to testdir but its methods return pathlib.Path when appropriate instead of <code>py.path.local</code>.</p>
  <p>This is part of the movement to use pathlib.Path objects internally, in order to remove the dependency to <code>py</code> in the future.</p>
  <p>Internally, the old Testdir &lt;_pytest.pytester.Testdir&gt; is now a thin wrapper around Pytester &lt;_pytest.pytester.Pytester&gt;, preserving the old interface.</p>
  </li>
  </ul>
  <!-- raw HTML omitted -->
  </blockquote>
  </details>
  <details>
  <summary>Changelog</summary>
  <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst">pytest's changelog</a>.</em></p>
  </details>
  <details>
  <summary>Commits</summary>
  <ul>
  <li><a href="https://github.com/pytest-dev/pytest/commit/e7073afe6e2373175130511135020af8b4e3a670"><code>e7073af</code></a> Prepare release version 6.2.0</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/683f29f84df61e9dab92970a57190d20ca58385e"><code>683f29f</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8129">#8129</a> from bluetech/docs-pygments-workaround</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/0feeddf8edb87052402fafe690d019e3eb75dfa4"><code>0feeddf</code></a> doc: temporary workaround for pytest-pygments lexing error</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/b4782757772373a2ea00802606407e9399b98031"><code>b478275</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8128">#8128</a> from bluetech/skip-reason-empty</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/3302ff994959964c3ec2ab35ec1fe98c038684c0"><code>3302ff9</code></a> terminal: when the skip/xfail is empty, don't show it as &quot;()&quot;</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/59bd0f6912fd9ec2b4c3e3bbc989636bb5f54dbf"><code>59bd0f6</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8126">#8126</a> from bluetech/tox-regen-pretend-scm2</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/6298ff1f4e21307627b59aa86437145ddcf2bafe"><code>6298ff1</code></a> tox: use pip legacy resolver for regen job</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/d51ecbd44d7e2f4f8c21f246fe9ed034bc8bae5b"><code>d51ecbd</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8125">#8125</a> from bluetech/tox-rm-pip-req</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/f237b077fc83fe834589e619be1e898f28026eb2"><code>f237b07</code></a> tox: remove requires: pip&gt;=20.3.1</li>
  <li><a href="https://github.com/pytest-dev/pytest/commit/95e0e19b8d5b55b0462d420af7454f3bdcbe7fbc"><code>95e0e19</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pytest-dev/pytest/issues/8124">#8124</a> from bluetech/s0undt3ch-feature/skip-context-hook</li>
  <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest/compare/5.2.2...6.2.0">compare view</a></li>
  </ul>
  </details>
  <br />


  Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

  [//]: # (dependabot-automerge-start)
  [//]: # (dependabot-automerge-end)

  ---

  <details>
  <summary>Dependabot commands and options</summary>
  <br />

  You can trigger Dependabot actions by commenting on this PR:
  - `@dependabot rebase` will rebase this PR
  - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
  - `@dependabot merge` will merge this PR after your CI passes on it
  - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
  - `@dependabot cancel merge` will cancel a previously requested merge and block automerging
  - `@dependabot reopen` will reopen this PR if it is closed
  - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
  - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
  - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
  - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
  - `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme

  Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
  - Update frequency (including time of day and day of week)
  - Pull request limits (per update run and/or open at any time)
  - Out-of-range updates (receive only lockfile updates, if desired)
  - Security updates (receive only security updates, if desired)



  </details>

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1145/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Njc1NjE4ODY=
title: Syntax error with + symbol when deployed to Vercel
state: closed
created_at: 2020-12-15T12:53:12Z
updated_at: 2020-12-16T21:57:42Z
closed_at: 2020-12-16T21:51:54Z
author_association: NONE
type: issue
state_reason: completed
body:
  Works locally:
  ```
  (ins)[hendry@t14s aws-partners]$ sqlite3 partners.db < test.sql
  5|{"href":"https://partners.amazonaws.com/partners/001E000000NaBI0IAN","label":"Slalom"}
  6|{"href":"https://partners.amazonaws.com/partners/001E000000VHBQIIA5","label":"Accenture"}
  7|{"href":"https://partners.amazonaws.com/partners/001E000000Rp588IAB","label":"Druva"}
  8|{"href":"https://partners.amazonaws.com/partners/001E0000013FeQXIA0","label":"Palo Alto Networks"}
  9|{"href":"https://partners.amazonaws.com/partners/001E000000Rl12lIAB","label":"New Relic"}
  10|{"href":"https://partners.amazonaws.com/partners/001E000000NaBHWIA3","label":"Deloitte"}
  11|{"href":"https://partners.amazonaws.com/partners/001E000000Rp5GDIAZ","label":"MegazoneCloud"}
  12|{"href":"https://partners.amazonaws.com/partners/001E000000NaBHMIA3","label":"iret, Inc."}
  (ins)[hendry@t14s aws-partners]$ cat test.sql
  select A.launch_rank, A.partner_info from summary A INNER JOIN summary B
      ON A.launch_rank>=B.launch_rank-3 AND A.launch_rank<=B.launch_rank+4
     WHERE B."partner_info" LIKE '%Palo Alto%'
  ```

  Copy the SQL into https://aws-partners-singapore.vercel.app/partners and get syntax error:

  ![image](https://user-images.githubusercontent.com/765871/102217393-78e4f280-3f17-11eb-9e13-ca79ed62ec34.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1148/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Njc2ODU5NjE=
title: Support of RData files
state: closed
created_at: 2020-12-15T15:04:14Z
updated_at: 2021-01-02T00:02:40Z
closed_at: 2021-01-02T00:02:40Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon,

  Would be great if you could ingest RData files!

  I could do this in a few lines of code but I am too lazy - sorry!

  Peter

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/210/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjkxNTAzOTQ=
title: Readme HTML has broken internal links
state: closed
created_at: 2020-12-16T17:58:11Z
updated_at: 2020-12-16T19:20:14Z
closed_at: 2020-12-16T19:20:14Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  From https://github.com/simonw/datasette.io/issues/46
  ```html
  <li><a href="#filtering-tables">Filtering tables</a></li>
  ...
  <h3><a id="user-content-filtering-tables" class="anchor" aria-hidden="true" href="#filtering-tables"><svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path fill-rule="evenodd" d="M7.775 3.275a.75.75 0 001.06 1.06l1.25-1.25a2 2 0 112.83 2.83l-2.5 2.5a2 2 0 01-2.83 0 .75.75 0 00-1.06 1.06 3.5 3.5 0 004.95 0l2.5-2.5a3.5 3.5 0 00-4.95-4.95l-1.25 1.25zm-4.69 9.64a2 2 0 010-2.83l2.5-2.5a2 2 0 012.83 0 .75.75 0 001.06-1.06 3.5 3.5 0 00-4.95 0l-2.5 2.5a3.5 3.5 0 004.95 4.95l1.25-1.25a.75.75 0 00-1.06-1.06l-1.25 1.25a2 2 0 01-2.83 0z"></path></svg></a>Filtering tables</h3>
  ```
  So this is a bug in GitHub's API, but we need to work around it.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/58/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjkyODIyMDY=
title: Upgrade to sqlite-utils 3.0 (tests are failing)
state: closed
created_at: 2020-12-16T21:25:15Z
updated_at: 2020-12-16T21:27:11Z
closed_at: 2020-12-16T21:27:10Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
          results = beta_db["search_index"].search("run")
          if use_porter:
              assert results == [
                  (
                      "dogs.db/dogs",
                      "1",
                      "Cleo",
                      "2020-08-22 04:41:33",
                      1,
                      0,
                      "running",
                      None,
                      None,
                  )
              ]
          else:
  >           assert results == []
  E           assert <generator ob...x7f3c080a3e08> == []
  E             +<generator object Table.search at 0x7f3c080a3e08>
  E             -[]
  E             Full diff:
  E             - []
  E             + <generator object Table.search at 0x7f3c080a3e08>
  ```
  This was caused by a backwards incompatible change in sqlite-utils 3.0: https://sqlite-utils.readthedocs.io/en/stable/changelog.html#v3-0

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjkzNzY0NDc=
title: killed by oomkiller on large location-history
state: open
created_at: 2020-12-17T00:32:24Z
updated_at: 2020-12-17T00:48:32Z
author_association: NONE
type: issue
body:
  memory seems to grow unbounded and is oom-killed after about 20GB memory usage.

  this is happening while loading a ~1GB uncompressed location history.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NjkzOTc3NDI=
title: sqlite-utils error on takeout import
state: open
created_at: 2020-12-17T01:18:48Z
updated_at: 2020-12-17T01:19:04Z
author_association: NONE
type: issue
body:
  ```
  $ google-takeout-to-sqlite my-activity takeout.db /path/to/zip
  ...
  sqlite3.OperationalError: no such table: main.my_activity
  ```

  there is no table create in `utils.py`, unlike other importers such as github-to-sqlite

  additionally, this package and hackernews-to-sqlite have conflicting `sqlite-utils` dep with datasette and dogsheep-beta

reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/3/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Njk1MjA5Mzk=
title: Make it easier to theme Datasette with CSS
state: open
created_at: 2020-12-17T05:01:26Z
updated_at: 2021-03-22T21:43:16Z
author_association: OWNER
type: issue
body:
  I want to theme https://datasette.io/ so that when you visit https://datasette.io/content (the Datasette UI part of it) the navigation from the parent site is used.

  I tried dropping in a `base.html` template like this:

  ```html
  {% extends "page_base.html" %}

  {% block base_extra_head %}

  <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
  {% for url in extra_css_urls %}
      <link rel="stylesheet" href="{{ url.url }}"{% if url.sri %} integrity="{{ url.sri }}" crossorigin="anonymous"{% endif %}>
  {% endfor %}
  {% for url in extra_js_urls %}
      <script src="{{ url.url }}"{% if url.sri %} integrity="{{ url.sri }}" crossorigin="anonymous"{% endif %}></script>
  {% endfor %}
  {% block extra_head %}{% endblock %}
  {% endblock %}

  {% block extra_body_end %}
  {% include "_close_open_menus.html" %}

  {% for body_script in body_scripts %}
      <script>{{ body_script }}</script>
  {% endfor %}
  {% endblock %}
  ```
  But this resulted in pages looking like this:

  <img width="1067" alt="content__categories__3_rows" src="https://user-images.githubusercontent.com/9599/102446045-c168e280-3fe1-11eb-94d6-e7350798eb96.png">

  Note that the cog menu is broken and the filter UI is unstyled. To get these working correctly I would need to copy over a whole lot of Datasette's default CSS - and that means that when Datasette changes in the future those pages could break in subtle ways.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1149/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzA0MzY4NzY=
title: Maintain an in-memory SQLite table of connected databases and their tables
state: closed
created_at: 2020-12-17T23:02:13Z
updated_at: 2020-12-27T14:51:39Z
closed_at: 2020-12-18T22:34:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I want Datasette to have its own internal metadata about connected tables, to power features like a paginated searchable homepage in #461. I want this to be a SQLite table.

  This could also be part of the directory scanning mechanism prototyped in #672 - where Datasette can be set to continually scan a directory for new database files that it can serve.

  Also relevant to the Datasette Library concept in #417.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1150/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzA0NDg2MjI=
title: Database class mechanism for cross-connection in-memory databases
state: closed
created_at: 2020-12-17T23:25:43Z
updated_at: 2021-01-26T19:07:44Z
closed_at: 2020-12-18T01:01:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Next challenge: figure out how to use the `Database` class from https://github.com/simonw/datasette/blob/0.53/datasette/database.py for an in-memory database which persists data for the duration of the lifetime of the server, and allows access to that in-memory database from multiple threads in a way that lets them see each other's changes.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747768112_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1151/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzA1OTgwMjQ=
title: Efficiently calculate list of databases/tables a user can view
state: open
created_at: 2020-12-18T06:13:01Z
updated_at: 2021-12-27T23:04:31Z
author_association: OWNER
type: issue
body:
  > The homepage currently performs a massive flurry of permission checks - one for each, database, table and view: https://github.com/simonw/datasette/blob/0.53/datasette/views/index.py#L21-L75
  > 
  > A paginated version of this is a little daunting as the permission checks would have to be carried out in every single table just to calculate the count that will be paginated.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-747864831_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1152/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTQyNDA2OTEw
title: BugFix for encoding and not update info.
state: closed
created_at: 2020-12-18T08:58:54Z
updated_at: 2021-02-11T22:37:56Z
closed_at: 2021-02-11T22:37:56Z
author_association: NONE
pull_request: dogsheep/evernote-to-sqlite/pulls/10
type: pull
body:
  Bugfix 1:

  Traceback (most recent call last):
    File "d:\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "d:\anaconda3\lib\runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "D:\Anaconda3\Scripts\evernote-to-sqlite.exe\__main__.py", line 7, in <module>
    File "d:\anaconda3\lib\site-packages\click\core.py", line 829, in __call__
    File "d:\anaconda3\lib\site-packages\click\core.py", line 782, in main
      rv = self.invoke(ctx)
    File "d:\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
      return ctx.invoke(self.callback, **ctx.params)
    File "d:\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "d:\anaconda3\lib\site-packages\evernote_to_sqlite\cli.py", line 30, in enex
      for tag, note in find_all_tags(fp, ["note"], progress_callback=bar.update):
    File "d:\anaconda3\lib\site-packages\evernote_to_sqlite\utils.py", line 11, in find_all_tags
      chunk = fp.read(1024 * 1024)
  UnicodeDecodeError: 'gbk' codec can't decode byte 0xa4 in position 383: illegal multibyte sequence

  Bugfix 2:

  Traceback (most recent call last):
    File "D:\Anaconda3\Scripts\evernote-to-sqlite-script.py", line 33, in <module>
      sys.exit(load_entry_point('evernote-to-sqlite==0.3', 'console_scripts', 'evernote-to-sqlite')())
    File "D:\Anaconda3\lib\site-packages\click\core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "D:\Anaconda3\lib\site-packages\click\core.py", line 782, in main
      rv = self.invoke(ctx)
    File "D:\Anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "D:\Anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "D:\Anaconda3\lib\site-packages\click\core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "D:\Anaconda3\lib\site-packages\evernote_to_sqlite-0.3-py3.8.egg\evernote_to_sqlite\cli.py", line 31, in enex
    File "D:\Anaconda3\lib\site-packages\evernote_to_sqlite-0.3-py3.8.egg\evernote_to_sqlite\utils.py", line 28, in save_note
  AttributeError: 'NoneType' object has no attribute 'text'

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzEyMDI0NTQ=
title: Use YAML examples in documentation by default, not JSON
state: closed
created_at: 2020-12-18T22:20:15Z
updated_at: 2023-07-08T20:09:48Z
closed_at: 2023-07-08T20:08:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  YAML configuration is much better for multi-line strings, and I'm increasingly adding configuration options to Datasette that benefit from that - fragments of HTML in `description_html` or SQL queries used to configure things like https://github.com/simonw/datasette-atom for example.

  Rather than confusing things by showing both in the documentation, I should switch all of the default examples to use YAML instead.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1153/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzEyMDgwMDk=
title: Documentation for new _internal database and tables
state: closed
created_at: 2020-12-18T22:34:52Z
updated_at: 2021-01-25T00:09:22Z
closed_at: 2021-01-25T00:08:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Needs documentation, but I can wait to write that until I've tested out the feature a bit more.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1150#issuecomment-748352106_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1154/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzEyMTYyOTM=
title: Better internal database_name for _internal database
state: closed
created_at: 2020-12-18T22:47:27Z
updated_at: 2020-12-22T20:04:35Z
closed_at: 2020-12-22T20:04:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://latest.datasette.io/login-as-root then https://latest.datasette.io/_internal

  <img width="1627" alt="_schemas__columns__132_rows" src="https://user-images.githubusercontent.com/9599/102668086-e70bfe00-413f-11eb-927d-e37ec9238e12.png">

  That `database_name` is ugly. It should just be `_internal`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1155/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzEzMTYzMDE=
title: Searching for "github-to-sqlite" throws an error
state: closed
created_at: 2020-12-19T06:07:20Z
updated_at: 2020-12-19T06:18:07Z
closed_at: 2020-12-19T06:18:07Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://datasette.io/-/beta?q=github-to-sqlite&sort=relevance&type=blog.db%2Fentries - "no such column: to"

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzEzMjQ4Mzc=
title: --since support for favorites
state: closed
created_at: 2020-12-19T07:08:23Z
updated_at: 2020-12-19T07:47:11Z
closed_at: 2020-12-19T07:47:11Z
author_association: NONE
type: issue
state_reason: completed
body:
  Having support for `--since` for updating your favorites would be ideal as the api is both slow and it only returns ~3k most recent favorites.

  https://twittercommunity.com/t/cant-get-all-favorite-tweets-by-rest-api/22007/3

  The api seems to take an optional `since_id` parameter - https://developer.twitter.com/en/docs/twitter-api/v1/tweets/post-and-engage/api-reference/get-favorites-list

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/53/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTQzMDE1ODI1
title: Update for Big Sur
state: open
created_at: 2020-12-20T04:36:45Z
updated_at: 2023-08-08T15:52:52Z
author_association: CONTRIBUTOR
pull_request: dogsheep/dogsheep-photos/pulls/31
type: pull
body:
  Refactored out the SQL for extracting aesthetic scores to use osxphotos -- adds compatbility for Big Sur via osxphotos which has been updated for new table names in Big Sur.  Have not yet refactored the SQL for extracting  labels which is still compatible with Big Sur.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzE2MDg2OTI=
title: UNIQUE constraint failed: workouts.id
state: open
created_at: 2020-12-20T15:11:20Z
updated_at: 2023-07-10T14:46:52Z
author_association: NONE
type: issue
body:
  I'm getting an error on my initial attempt to import data:

  ```console
  $ healthkit-to-sqlite 20201119\ healthkit\ export.zip healthkit.db
  Importing from HealthKit  [###################################-]   98%  00:00:01
  Traceback (most recent call last):
    File "venv/bin/healthkit-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "venv/lib/python3.9/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli
      convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
    File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py", line 34, in convert_xml_to_sqlite
      workout_to_db(el, db, zipfile)
    File "venv/lib/python3.9/site-packages/healthkit_to_sqlite/utils.py", line 57, in workout_to_db
      pk = db["workouts"].insert(record, alter=True, hash_id="id").last_pk
    File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1660, in insert
      return self.insert_all(
    File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1778, in insert_all
      self.insert_chunk(
    File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 1588, in insert_chunk
      result = self.db.execute(query, params)
    File "venv/lib/python3.9/site-packages/sqlite_utils/db.py", line 213, in execute
      return self.conn.execute(sql, parameters)
  sqlite3.IntegrityError: UNIQUE constraint failed: workouts.id
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTQzMjQ2NTM1
title: Remove unneeded exists=True for -a/--auth flag.
state: closed
created_at: 2020-12-21T06:03:55Z
updated_at: 2021-05-22T14:06:19Z
closed_at: 2021-05-19T16:08:12Z
author_association: CONTRIBUTOR
pull_request: dogsheep/github-to-sqlite/pulls/59
body: The file does not need to exist when using an environment variable.
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzI0MDg3NTA=
title: Rename _schemas to _internal
state: closed
created_at: 2020-12-21T19:27:58Z
updated_at: 2021-01-24T21:20:39Z
closed_at: 2020-12-21T19:51:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I like `_internal` as the name for the in-memory Datasette schema database better.

  Refs #1154 #1150 #1155

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1156/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzI0MzgyNzM=
title: Use time.perf_counter() instead of time.time() to measure performance
state: closed
created_at: 2020-12-21T20:21:41Z
updated_at: 2021-01-24T21:20:42Z
closed_at: 2020-12-21T21:49:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I do that in a bunch of places: https://ripgrep.datasette.io/-/ripgrep?pattern=time%28%29&literal=on&glob=datasette%2F%2A%2A

  https://docs.python.org/3/library/time.html#time.perf_counter

  > `time.``perf_counter`() → float
  > 
  > Return the value (in fractional seconds) of a performance counter, i.e. a clock with the highest available resolution to measure a short duration. It does include time elapsed during sleep and is system-wide. The reference point of the returned value is undefined, so that only the difference between the results of consecutive calls is valid.
  > 
  > _New in version 3.3._

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1157/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTQ0OTIzNDM3
title: Modernize code to Python 3.6+
state: closed
created_at: 2020-12-23T16:21:38Z
updated_at: 2021-01-24T21:20:50Z
closed_at: 2020-12-23T17:04:32Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1158
type: pull
body:
  - compact dict and set building
  - remove redundant parentheses
  - simplify chained conditions
  - change method name to lowercase
  - use triple double quotes for docstrings

  please feel free to accept/reject any of these independent commits

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1158/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTQ1MjY0NDM2
title: Improve the display of facets information
state: open
created_at: 2020-12-24T11:01:47Z
updated_at: 2023-07-31T18:57:59Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: simonw/datasette/pulls/1159
type: pull
body:
  This PR changes the display of facets to hopefully make them more readable.

  Before | After
  ---|---
  ![image](https://user-images.githubusercontent.com/552629/103084609-b1ec2980-45df-11eb-85bc-68ab8df3e8d9.png) | ![image](https://user-images.githubusercontent.com/552629/103085220-620e6200-45e1-11eb-8189-5dd5d3e2569e.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1159/reactions", "total_count": 4, "+1": 4, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzU2NjYyOTY=
title: "datasette insert" command and plugin hook
state: open
created_at: 2020-12-29T02:37:03Z
updated_at: 2021-06-17T18:12:32Z
author_association: OWNER
type: issue
body:
  Tools for loading data into Datasette currently mostly exist as separate utilities - `yaml-to-sqlite` and `csvs-to-sqlite` and suchlike.

  Bringing these into Datasette could have some interesting properties:

  - A `datasette insert` command could be extended with plugins to handle more formats
  - Any format that can be inserted on the command-line could also be inserted using a web UI or web API - which would benefit from new format plugin hooks
  - If Datasette ever grows beyond SQLite (see #670) a built-in import mechanism could work for those other databases as well - without me needing to write `yaml-to-postgresql` and suchlike

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1160/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzYxMDExMDE=
title: Update a whole bunch of links to datasette.io instead of datasette.readthedocs.io
state: open
created_at: 2020-12-29T21:47:31Z
updated_at: 2020-12-29T21:49:57Z
author_association: OWNER
body: https://ripgrep.datasette.io/-/ripgrep?pattern=%28datasette%7Csqlite-utils%29%5C.readthedocs%5C.io
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1161/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzYxMjgyNjk=
title: First working version of "datasette insert data.db file.csv"
state: open
created_at: 2020-12-29T23:20:11Z
updated_at: 2021-06-17T18:12:32Z
author_association: OWNER
body: Refs #1160
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1162/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzYxMjg1NjU=
title: "datasette insert data.db url-to-csv"
state: open
created_at: 2020-12-29T23:21:21Z
updated_at: 2021-06-17T18:12:32Z
author_association: OWNER
body: Refs #1160 - get filesystem imports working first for #1162, then add import-from-URL.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1163/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzY2MzQzMTg=
title: Mechanism for minifying JavaScript that ships with Datasette
state: open
created_at: 2020-12-30T20:59:06Z
updated_at: 2022-01-13T22:21:29Z
author_association: OWNER
type: issue
body:
  > If I'm going to minify it I'll need to figure out a build step in Datasette itself so that I can easily work on that minified version.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752748496_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1164/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzY2MzU0MjY=
title: Mechanism for executing JavaScript unit tests
state: open
created_at: 2020-12-30T21:02:34Z
updated_at: 2022-01-13T22:21:29Z
author_association: OWNER
type: issue
body:
  > I'm going to need to add JavaScript unit tests for this new plugin system.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/983#issuecomment-752757289_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1165/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzcxNDA3OTk=
title: Adopt Prettier for JavaScript code formatting
state: open
created_at: 2020-12-31T21:25:27Z
updated_at: 2022-01-13T22:22:18Z
author_association: OWNER
body: https://prettier.io/ - I'm going to go with 2 spaces.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1166/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzcxNDU5NTQ=
title: Add Prettier to contributing documentation
state: closed
created_at: 2020-12-31T22:00:55Z
updated_at: 2021-01-25T02:01:19Z
closed_at: 2021-01-25T01:58:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Following #1166 - the docs at https://docs.datasette.io/en/stable/contributing.html should include a section about JavaScript, and it should document how to run Prettier.

  I run it in VS Code but it can be run on the command-line too:

      npx prettier 'datasette/static/*[!.min].js' --write

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1167/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzczMzMzODg=
title: Mechanism for storing metadata in _metadata tables
state: open
created_at: 2021-01-01T18:47:27Z
updated_at: 2023-09-28T18:29:05Z
author_association: OWNER
type: issue
body:
  _Original title: Perhaps metadata should all live in a `_metadata` in-memory database_

  Inspired by #1150 - metadata should be exposed as an API, and for large Datasette instances that API may need to be paginated. So why not expose it through an in-memory database table?

  One catch to this: plugins. #860 aims to add a plugin hook for metadata. But if the metadata comes from an in-memory table, how do the plugins interact with it?

  The need to paginate over metadata does make a plugin hook that returns metadata for an individual table seem less wise, since we don't want to have to do 10,000 plugin hook invocations to show a list of all metadata.

  If those plugins write directly to the in-memory table how can their contributions survive the server restarting?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1168/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzczODY0NjU=
title: table.triggers_dict introspection property
state: closed
created_at: 2021-01-02T02:04:00Z
updated_at: 2021-01-02T02:10:10Z
closed_at: 2021-01-02T02:10:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `table.triggers` currently returns a list of `Trigger` values. A `table.triggers_dict` property could behave like `columns_dict`, returning a dictionary mapping trigger names to their SQL definitions for that table.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/211/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzczOTIwMjA=
title: Mechanism for maintaining cache of table counts using triggers
state: closed
created_at: 2021-01-02T02:58:53Z
updated_at: 2021-01-02T21:40:27Z
closed_at: 2021-01-02T21:40:27Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Counting all of the rows in a large table is expensive - this is one of the main causes of performance problems in Datasette when running against large databases.

  Carefully constructed SQL triggers could be used to maintain accurate cached counts for a table, by incrementing and decrementing a counter every time a row is inserted or deleted.

  `sqlite-utils` already has a mechanism for creating triggers for FTS - the `table.enable_fts(..., create_triggers=True)` method. How about a similar mechanism for setting up triggers to maintain a count of table rows?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/212/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc1Mjk5Nzk=
title: db.enable_counts() method
state: closed
created_at: 2021-01-02T21:44:55Z
updated_at: 2021-01-02T22:04:02Z
closed_at: 2021-01-02T22:04:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Following #212 it would be useful if there was a utility method for enabling counts for ALL tables in a database:

  ```python
  db.enable_counts()
  ```

  Open question: should this setup triggers for virtual tables such as FTS tables? Could that break things?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/213/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc1MzAxMDc=
title: sqlite-utils enable-counts command
state: closed
created_at: 2021-01-02T21:45:48Z
updated_at: 2021-01-03T04:26:44Z
closed_at: 2021-01-03T04:26:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The CLI version of #212 and #213.

      # Enable counts for all tables:
      sqlite-utils enable-counts data.db

      # Enable counts for specific tables:
      sqlite-utils enable-counts data.db table1 table2

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/214/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc1MzU0MDI=
title: Use _counts to speed up counts
state: closed
created_at: 2021-01-02T22:30:17Z
updated_at: 2021-01-03T20:19:40Z
closed_at: 2021-01-03T20:19:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Utility mechanism for taking advantage of the new `_counts` table from #212 would be nice.

  These can trigger automatically if the `_counts` table exists, but since `sqlite-utils` needs to work against any existing database there should be a way of opting out of this optimization.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/215/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc1NDAzNTI=
title: database.triggers_dict introspection property
state: closed
created_at: 2021-01-02T23:13:00Z
updated_at: 2021-01-03T04:27:14Z
closed_at: 2021-01-03T04:25:36Z
author_association: OWNER
body: Following #211
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/216/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc1NDMzMzY=
title: Rename .escape() to .quote()
state: closed
created_at: 2021-01-02T23:40:52Z
updated_at: 2021-01-03T04:27:38Z
closed_at: 2021-01-03T04:15:23Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `.quote()` is a better name because it reflects that the method adds quotes around the value.

  This method has never been documented so I'm going to rename it without a major version bump. 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/217/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc1NjA0NzQ=
title: "sqlite-utils triggers" command
state: closed
created_at: 2021-01-03T02:34:50Z
updated_at: 2021-01-03T03:49:51Z
closed_at: 2021-01-03T03:03:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  A command to list the triggers in the database.

      sqlite-utils triggers my.db

  Can optionally take one or more tables:

      sqlite-utils triggers my.db table1 table2

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/218/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc2Nzc2NzE=
title: Prettier package not actually being cached
state: closed
created_at: 2021-01-03T17:04:41Z
updated_at: 2021-01-04T19:52:34Z
closed_at: 2021-01-04T19:52:33Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  With the current configuration Prettier seems to be installed on every run - which can been [seen from the output](https://github.com/simonw/datasette/runs/1631686028?check_suite_focus=true#step:4:4):

  ```
  npx: installed 1 in 5.166s
  ```

  Prettier isn't explicitly being installed (it's surprising that actually installing the dependencies isn't included in the [actions/cache docs](https://github.com/actions/cache/blob/main/examples.md#macos-and-ubuntu)) but it turns out that `npx` will automatically install the package for the specified command (it actually _guesses_ the package name from the name of the command). I'm not sure where Prettier ends up being installed but it doesn't appear to be in `~/.npm` according to the [post-cache output](https://github.com/simonw/datasette/runs/1631686028#step:7:2) (or `./node_modules` when I tested locally):

  ```
  Cache hit occurred on the primary key Linux-npm-565329898f77080e58b14d45cf816ab94877e6f2ece9d395c369c533548a7ee7, not saving cache.
  ```

  I think there are a couple of approaches to tackling this, you could manually install/cache Prettier within the action, or add a `package.json` with Prettier. I would go with the latter because it's a more standard and maintainable approach and it will also ensure that, along with CI, anyone working on the project will run the same version of Prettier (you'll also get Dependabot JavaScript updates).

  I've tested the [`package.json` approach on a branch](https://github.com/simonw/datasette/compare/main...benpickles:cache-prettier) and am happy to turn it into a pull request if you fancy.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1169/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzc3MDc1NDQ=
title: reset_counts() method and command
state: closed
created_at: 2021-01-03T20:08:28Z
updated_at: 2021-01-03T20:59:37Z
closed_at: 2021-01-03T20:59:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Thought: maybe there should be a `.reset_counts()` method too, for if the table gets out of date with the triggers.
  >
  > One way that could happen is if a table is dropped and recreated - the counts in the `_counts` table would likely no longer match the number of rows in that table.

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/219/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTQ4MjcxNDcy
title: Install Prettier via package.json
state: closed
created_at: 2021-01-04T14:18:03Z
updated_at: 2021-01-24T21:21:01Z
closed_at: 2021-01-04T19:52:34Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1170
type: pull
body:
  This adds a package.json with Prettier and means that developers/CI will use the same version. It also ensures that NPM packages are cached on GitHub Actions which fixes #1169.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1170/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzgzODA4MzY=
title: Feature Request: Gmail
state: open
created_at: 2021-01-04T21:31:09Z
updated_at: 2021-03-04T20:54:44Z
author_association: NONE
type: issue
body:
  From takeout, I only exported my Gmail account.  Ideally I could parse this into sqlite via this tool.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzg0NTA0ODY=
title: GitHub Actions workflow to build and sign macOS binary executables
state: open
created_at: 2021-01-04T23:36:59Z
updated_at: 2021-01-07T19:36:00Z
author_association: OWNER
type: issue
body:
  Using PyInstaller, as explored in #93 and https://til.simonwillison.net/python/packaging-pyinstaller

  The bigger challenge will be the code signing bit. I'll need a Apple Developer account ($99/year) and some extensive CI fiddling.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1171/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzg1MzA1MjM=
title: /-/static should be excluded from auth and permission checks
state: open
created_at: 2021-01-05T02:53:41Z
updated_at: 2021-01-05T02:53:41Z
author_association: OWNER
type: issue
body:
  I want to set far future / immutable cache headers on everything served from `/-/static` and `/-/static-plugins`

  This has security implications since it will be possible to see what plugins are installed by checking for known static URLs. I'm fine with that - performance is more important here.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1172/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzg2ODIzMTc=
title: GitHub Actions workflow to build manylinux binary
state: open
created_at: 2021-01-05T07:41:11Z
updated_at: 2021-01-05T07:41:43Z
author_association: OWNER
body: Refs #1171 and #93
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1173/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzkwODgwNzE=
title: Archive import appears to be broken on recent exports
state: open
created_at: 2021-01-05T14:18:01Z
updated_at: 2023-01-04T11:06:55Z
author_association: CONTRIBUTOR
type: issue
body:
  I requested a Twitter export yesterday, and unfortunately they seem to have changed it such that `twitter-to-sqlite import` can't handle it anymore 😢 

  So far I've ran into two issues. The first was easy to work around, but the second will take more investigation. If I can find the time I'll keep working on it and update this issue accordingly.

  The issues (so far):

  ### 1. Data seems to have moved to a `data/` subdirectory

  Running `twitter-to-sqlite import` on the raw zip file reports a bunch of "not yet implemented" errors, and then exits without actually importing anything:

  ```
  ❯ twitter-to-sqlite import tarchive.db twitter.zip
  ...
  data/manifest: not yet implemented
  data/account-creation-ip: not yet implemented
  data/account-suspension: not yet implemented
  ... (dozens of more lines like this, including critical stuff like data/tweets) ...
  ```

  (`tarchive.db` now exists, but is empty)

  Workaround: unpack the zip file, and run `twitter-to-sqlite import tarchive.db path/to/archive/data`

  That gets further, but:

  ### 2. Some schema(s?) have changed

  At least, the `blocks` schema seems different now:

  ```
  ❯ twitter-to-sqlite import tarchive.db archive/data
  direct-messages-group: not yet implemented
  branch-links: not yet implemented
  periscope-expired-broadcasts: not yet implemented
  direct-messages: not yet implemented
  mute: not yet implemented
  Traceback (most recent call last):
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/bin/twitter-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/cli.py", line 772, in import_
      archive.import_from_file(db, filepath.name, open(filepath, "rb").read())
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py", line 215, in import_from_file
      to_insert = transformer(data)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py", line 115, in lists_member
      return {"lists-member": _list_from_common(data)}
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py", line 200, in _list_from_common
      for url in block["userListInfo"]["urls"]:
  KeyError: 'urls'
  ```

  That's as far as I got before I needed to work on something else. I'll report back if I get further!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3NzkxNTY1MjA=
title: Use structlog for logging
state: open
created_at: 2021-01-05T15:11:36Z
updated_at: 2022-07-26T12:52:10Z
author_association: OWNER
body: To solve #241 JSON logging.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1175/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTQ5MjA0MDYz
title: Fix archive imports
state: closed
created_at: 2021-01-05T15:54:48Z
updated_at: 2021-08-20T00:02:49Z
closed_at: 2021-08-20T00:02:49Z
author_association: CONTRIBUTOR
pull_request: dogsheep/twitter-to-sqlite/pulls/55
body: This fixes the issues discussed in #54
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3Nzk2OTE3Mzk=
title: Policy on documenting "public" datasette.utils functions
state: closed
created_at: 2021-01-05T22:55:25Z
updated_at: 2022-02-07T06:43:32Z
closed_at: 2022-02-07T06:42:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette-css-properties starts [like this](https://github.com/simonw/datasette-css-properties/blob/0.1/datasette_css_properties/__init__.py#L1-L3):

  ```python
  from datasette import hookimpl
  from datasette.utils.asgi import Response
  from datasette.utils import escape_css_string, to_css_class
  ```
  `escape_css_string` and `to_css_class` are not documented, which means relying on them is risky since there's no promise that they won't change.

  Would be good to figure out a policy on this, and maybe promote some of them to "documented" status.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1176/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODAxNTM1NjI=
title: Ability to stream all rows as newline-delimited JSON
state: open
created_at: 2021-01-06T07:10:48Z
updated_at: 2022-03-21T15:08:52Z
author_association: OWNER
type: issue
body:
  > Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:
  > 
  >     pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True)
  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1101#issuecomment-755128038_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1177/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODAyNjc4NTc=
title: Use force_https_urls on when deploying with Cloud Run
state: closed
created_at: 2021-01-06T08:20:55Z
updated_at: 2021-01-24T21:21:05Z
closed_at: 2021-01-06T18:24:47Z
author_association: OWNER
type: issue
state_reason: completed
body:
  _Original title: datasette.absolute_url() should return https:// not http:// on Cloud Run_

  https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on currently provides `http://` links, which break in Observable since it won't load `http://` content.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1178/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODAyNzg1NTA=
title: Make original path available to render hooks
state: open
created_at: 2021-01-06T08:31:45Z
updated_at: 2021-01-25T04:44:33Z
author_association: OWNER
type: issue
body:
  https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py

  ```python
  async def render_notebook(datasette, request):
      return Response.html(
          await datasette.render_template(
              "export_notebook.html",
              {
                  "csv_stream_url": datasette.absolute_url(
                      request,
                      path_with_format(
                          request=request, format="csv", extra_qs={"_stream": "on"}
                      ),
                  ),
                  "json_url": datasette.absolute_url(
                      request,
                      path_with_format(
                          request=request, format="json", extra_qs={"_shape": "array"}
                      ),
                  ),
                  "json": json,
              },
          )
      )
  ```
  This results in https://latest-with-plugins.datasette.io/github/issue_comments.Notebook showing `http://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_format=json&_shape=array`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1179/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODA3Njc1NDI=
title: Lazily evaluated arguments for call_with_supported_arguments
state: open
created_at: 2021-01-06T18:43:34Z
updated_at: 2021-01-07T18:56:24Z
author_association: OWNER
type: issue
body:
  While building https://github.com/simonw/datasette-export-notebook I thought it would be nice to be able to show a count of exported records on the page "This will stream 10,422 records to your notebook".

  None of the documented arguments on https://docs.datasette.io/en/0.53/plugin_hooks.html#register-output-renderer-datasette expose the count. The closest is `sql` which could be executed as `select count(*) from ({sql})` but that's a bit inelegant.

  So, idea: if your defined render function takes a `total` argument, the caller runs a `count(*)` query and passes you that total.

  To implement this I would need to teach the `call_with_supported_arguments` that some arguments are lazy - they should execute a function (or an async function) but only if they are needed.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1180/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODEyNjI1MTA=
title: Certain database names results in 404: "Database not found: None"
state: closed
created_at: 2021-01-07T12:01:16Z
updated_at: 2021-12-21T18:25:15Z
closed_at: 2021-01-25T05:13:19Z
author_association: NONE
type: issue
state_reason: completed
body:
  I have a file named `test-database (1).sqlite`. When requesting the home route `/`, I see datasette is able to read it correctly:

  <img width="409" alt="Screenshot 2021-01-07 at 12 54 46" src="https://user-images.githubusercontent.com/1470389/103889964-9442b800-50e7-11eb-9548-fbf10ed3b1ea.png">

  However, if I click any of the links, datasette replies with: `Error 404 Database not found: None`

  It seems the hash is crucial, as renaming the file to `database (1).sqlite` makes the error go away.

  This lines checks for a single dash:
  https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184

  ```
  $ datasette test-database\ \(1\).sqlite 
  INFO:     Started server process [68314]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
  INFO:     127.0.0.1:54043 - "GET /favicon.ico HTTP/1.1" 200 OK
  INFO:     127.0.0.1:54043 - "GET / HTTP/1.1" 200 OK
  ...
  INFO:     127.0.0.1:54044 - "GET /favicon.ico HTTP/1.1" 200 OK
  INFO:     127.0.0.1:54044 - "GET /test-database (1) HTTP/1.1" 404 Not Found

  ```
  Version:
  ```
  $ datasette --version
  datasette, version 0.53
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1181/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODI2OTIxNTk=
title: Retire "Ecosystem" page in favour of datasette.io/plugins and /tools
state: closed
created_at: 2021-01-09T21:54:47Z
updated_at: 2021-01-24T21:21:09Z
closed_at: 2021-01-09T22:17:28Z
author_association: OWNER
body: https://docs.datasette.io/en/stable/ecosystem.html is no longer needed.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1182/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODI3MDg0Njk=
title: Take advantage of sqlite-utils cached table counts, if available
state: open
created_at: 2021-01-09T23:51:48Z
updated_at: 2021-01-12T02:42:08Z
author_association: OWNER
type: issue
body:
  sqlite-utils 3.2 now has a mechanism for creating a `_counts` table with triggers to maintain counts for individual tables. Datasette could look for this table and use it for counts, dramatically speeding up places that currently suffer from slow counts. Refs #859.

  https://sqlite-utils.datasette.io/en/stable/python-api.html#cached-table-counts-using-triggers

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1183/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODM3MTQwNzY=
title: request.full_path property
state: closed
created_at: 2021-01-11T21:21:58Z
updated_at: 2021-01-24T21:21:16Z
closed_at: 2021-01-11T21:34:47Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'll also add `request.full_path` for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1179#issuecomment-755495387_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1184/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODM3Nzg2NzI=
title: Better error message for *_fts methods against views
state: closed
created_at: 2021-01-11T23:24:00Z
updated_at: 2021-02-22T20:44:51Z
closed_at: 2021-02-14T22:34:26Z
author_association: NONE
type: issue
state_reason: completed
body:
  enable_fts and its related methods only work on tables, not views. 

  Could those methods and possibly others move up to the Queryable superclass?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/220/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODM5MTA5MDE=
title: .add_missing_columns() does not take case insensitivity into account
state: closed
created_at: 2021-01-12T05:01:00Z
updated_at: 2021-01-12T23:17:33Z
closed_at: 2021-01-12T23:17:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  SQLite columns are case insensitive - but the `.add_missing_columns()` method doesn't know that. This means that it can crash if it identifies a column that is a case-insensitive duplicate of an existing column. https://github.com/simonw/sqlite-utils/blob/4cc82fd0bccc9d2eeb3510beb4e691d7da099f84/sqlite_utils/db.py#L1974-L1980

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/221/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODQ2MjgxNjM=
title: "Statement may not contain PRAGMA" error is not strictly true
state: closed
created_at: 2021-01-12T22:07:10Z
updated_at: 2021-01-24T21:21:37Z
closed_at: 2021-01-12T22:26:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Consider https://latest.datasette.io/fixtures?sql=select+%27select%0D%0A%27+%7C%7C+group_concat%28%27++++case+when+%5B%27+%7C%7C+name+%7C%7C+%27%5D+is+not+null+then+%27+%7C%7C+quote%28name+%7C%7C+%27%2C+%27%29+%7C%7C+%27+else+%27%27%27%27+end%27%2C+%27+%7C%7C%0D%0A%27%29+%7C%7C+%27%0D%0A++as+columns%2C%0D%0A++count%28*%29+as+num_rows%0D%0Afrom%0D%0A++%5B%27+%7C%7C+%3Atable+%7C%7C+%27%5D%0D%0Agroup+by%0D%0A++columns%0D%0Aorder+by%0D%0A++num_rows+desc%27+as+query+from+pragma_ytable_info%28%3Atable%29&table=facetable

  It says "Statement may not contain PRAGMA" - but that's not actually true. Datasette has an allow-list of PRAGMA that are OK - in this case there was a typo in `pragma_ytable_info` which caused the error, but pragma_table_info` would have been OK.

  So the error message is misleading.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1185/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODU1NzM3OTM=
title: script type="module" support
state: closed
created_at: 2021-01-14T01:17:47Z
updated_at: 2021-01-24T21:21:41Z
closed_at: 2021-01-14T01:50:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Custom JavaScript can be loaded in `metadata.json` like this:
  ```json
  {
      "extra_js_urls": [
          {
              "url": "https://code.jquery.com/jquery-3.2.1.slim.min.js",
              "sri": "sha256-k2WSCIexGzOj3Euiig+TlR8gA0EmPjuc79OEeY5L45g="
          }
      ]
  }
  ```
  Add a `"module": true` option which causes the resulting script element to use `<script type="module">`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1186/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODU1ODg5NDI=
title: extra_body_script() support for script type="module"
state: closed
created_at: 2021-01-14T02:01:47Z
updated_at: 2021-01-24T21:21:44Z
closed_at: 2021-01-14T02:14:39Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Follows #1186. The `extra_body_script()` plugin hook should provide a mechanism for specifying that the script should use `<script type="module">`.

  Relevant docs: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1187/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODcwOTgxNDY=
title: `datasette publish upload` mechanism for uploading databases to an existing Datasette instance
state: closed
created_at: 2021-01-15T18:18:42Z
updated_at: 2023-08-30T22:16:39Z
closed_at: 2023-08-30T22:16:38Z
author_association: NONE
type: issue
state_reason: completed
body:
  If I have a self-hosted instance of Datasette up and running, I'd like to be able to the use the CLI to publish databases to that instance, not only Google or Heroku. Ideally there'd be a `url` parameter or something similar to which one could point the publish command to their instance. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1190/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODcwOTgzNDU=
title: Ability for plugins to collaborate when adding extra HTML to blocks in default templates
state: open
created_at: 2021-01-15T18:18:51Z
updated_at: 2023-09-18T06:55:52Z
author_association: OWNER
type: issue
body:
  Sometimes a plugin may want to add content to an existing default template - for example `datasette-search-all` adds a new search box at the top of `index.html`. I also want `datasette-upload-csvs` to add a CTA on the `database.html` page: https://github.com/simonw/datasette-upload-csvs/issues/18

  Currently plugins can do this by providing a new version of the `index.html` template - but if multiple plugins try to do that only one of them will succeed.

  It would be better if there were known areas of those templates which plugins could add additional content to, such that multiple plugins can use the same spot.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1191/reactions", "total_count": 4, "+1": 4, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODcxMDQ4NTA=
title: Form Plugin for in-depth Datasette Querying
state: open
created_at: 2021-01-15T18:24:50Z
updated_at: 2021-01-15T18:24:50Z
author_association: NONE
type: issue
body:
  I envision a sort of easy-to-build form plugin that would be able to map a user's inputs to different fields/columns in a Datasette database. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1192/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODcxNzMyNzY=
title: Research plugin hook for alternative database backends
state: open
created_at: 2021-01-15T20:27:50Z
updated_at: 2021-03-12T01:01:54Z
author_association: OWNER
type: issue
body:
  I started exploring what Datasette would like running against PostgreSQL in #670 and @dazzag24 did some work on Parquet described in #657.

  I had initially thought this was WAY too much additional complexity, but I'm beginning to think that the `Database` class may be small enough that having it abstract away the details of running queries against alternative database backends could be feasible.

  A bigger issue is SQL generation, but I realized that most of Datasette's SQL generation code exists just in the `TableView` class that runs the table page. If this was abstracted into some kind of SQL builder that could be then customized per-database it might be reasonable to get it working.

  Very unlikely for this to make it into Datasette 1.0, but maybe this would be the defining feature of Datasette 2.0?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1193/reactions", "total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODc5MDA0MTI=
title: .m2m() should accept alter=True parameter
state: closed
created_at: 2021-01-18T04:15:43Z
updated_at: 2021-01-18T04:26:10Z
closed_at: 2021-01-18T04:26:10Z
author_association: OWNER
body: Needed by https://github.com/dogsheep/swarm-to-sqlite/issues/11
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/222/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODg0NDc3ODc=
title: ?_size= argument is not persisted by hidden form fields in the table filters
state: closed
created_at: 2021-01-18T17:41:52Z
updated_at: 2021-01-25T03:10:23Z
closed_at: 2021-01-25T03:10:23Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Click "Apply" on https://covid-19.datasettes.com/covid/ny_times_us_counties?_size=1000&county__exact=San+Francisco&state__exact=California&_sort_desc=date#g.mark=line&g.x_column=date&g.x_type=temporal&g.y_column=cases&g.y_type=quantitative and the `?_size=1000` parameter from the URL will no longer apply on the reloaded page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1194/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODg1Mjc5MzI=
title: --delimiter option for CSV import
state: closed
created_at: 2021-01-18T20:25:03Z
updated_at: 2021-02-06T01:39:47Z
closed_at: 2021-02-06T01:34:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://bruxellesdata.opendatasoft.com/explore/dataset/dog-toilets/export/?location=12,50.85802,4.38054 says:

  > CSV uses semicolon (;) as a separator.

  Would be useful to be able to do this:

      sqlite-utils insert places.db places places.csv --delimiter ';'

  `--delimiter` could imply `--csv`

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/223/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3ODkzMzY1OTI=
title: view_name = "query" for the query page
state: open
created_at: 2021-01-19T20:21:36Z
updated_at: 2021-01-25T04:40:08Z
author_association: OWNER
body: It uses `view_name` of `database` at the moment which isn't as useful.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1195/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTEyMzc3OTk=
title: Access Denied Error in Windows
state: open
created_at: 2021-01-21T15:40:40Z
updated_at: 2021-04-14T19:28:38Z
author_association: NONE
type: issue
body:
  I am trying to publish a db to vercel. But while issuing the below command throwing `Access Denied` error which is leading to `RecursionError: maximum recursion depth exceeded while calling a Python object`.

  I am using PyCharm and Python 3.9. I have reinstalled both and launched PyCharm as Admin in Windows 10. But still the issue persists.

  Issued command `datasette publish vercel jmeter.db --project jmeter --install datasette-vega`

  PS: localhost is working fine.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1196/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTEzODE2MjM=
title: DB size limit for publishing with Heroku
state: closed
created_at: 2021-01-21T18:08:43Z
updated_at: 2021-01-24T20:53:44Z
closed_at: 2021-01-24T20:53:44Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hello,
  I tried searching for this, but can't seem to get a great answer: Does anybody know the size limit for databases deploying to Heroku? The files I'm working with are pretty large, but I might be able to pare them down if I have a limit in mind.

   I'm getting the following error when running `datasette heroku publish`:

  `RangeError [ERR_INVALID_OPT_VALUE]: The value "14504095744" is invalid for option "size"`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1197/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTYwMjA0MzA2
title: Add fts offset docs.
state: closed
created_at: 2021-01-22T20:50:58Z
updated_at: 2021-02-14T19:31:06Z
closed_at: 2021-02-14T19:31:06Z
author_association: NONE
pull_request: simonw/sqlite-utils/pulls/224
type: pull
body:
  The limit can be passed as a string to the query builder to have an offset. I have tested it using the shorthand `limit=f"15, 30"`, the standard syntax should work too.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/224/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTI2MjU4MTI=
title: Plugin testing documentation on using pytest-httpx
state: closed
created_at: 2021-01-23T18:46:16Z
updated_at: 2021-01-24T20:40:38Z
closed_at: 2021-01-24T20:38:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I keep on having to figure this out: if you use the https://pypi.org/project/pytest-httpx/ fixture to write tests against mocked external APIs, they will fail because that module will break Datasette's own `datasette.client` testing mechanism.

  You can fix this using:

  ```python
  @pytest.fixture
  def non_mocked_hosts():
      return ["localhost"]
  ```

  See https://github.com/simonw/datasette-indieauth/blob/1.2/tests/test_indieauth.py

  I can add this tip to the https://docs.datasette.io/en/stable/testing_plugins.html page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1198/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTI2NTIzOTE=
title: Experiment with PRAGMA mmap_size=N
state: open
created_at: 2021-01-23T21:24:09Z
updated_at: 2021-07-17T17:39:17Z
author_association: OWNER
type: issue
body:
  https://sqlite.org/mmap.html - SQLite supports memory-mapped I/O but it's disabled by default. The `PRAGMA mmap_size=N` option can be used to enable it.

  It would be very interesting to understand the impact this could have on Datasette performance for various different shapes of data.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1199/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTI4NTE0NDQ=
title: XML parse error
state: closed
created_at: 2021-01-24T17:38:54Z
updated_at: 2021-02-11T21:18:58Z
closed_at: 2021-02-11T21:18:48Z
author_association: NONE
type: issue
state_reason: completed
body:
  I am on Windows 10 using Windows Subsystem for Linux, Python 3.8. I installed evernote-to-sqlite via pipx (in a venv). I tried using enex files from the latest version of Evernote for Windows (10.6.9 which only lets you export 50 notes at a time) and from Legacy Evernote (6.25.2.9198 which lets you export all your notes at once). The enex file from latest evernote gives this error:

      File "/usr/lib/python3.8/xml/etree/ElementTree.py", line 1320, in XML parser.feed(text)
      xml.etree.ElementTree.ParseError: XML or text declaration not at start of entity: line 2, column 6

  The enex file from Legacy Evernote gives this error:

      File "/home/david/.local/pipx/venvs/evernote-to-sqlite/lib/python3.8/site-packages/evernote_to_sqlite/utils.py", line 28, in save_note
      updated = note.find("updated").text
      AttributeError: 'NoneType' object has no attribute 'text'

reactions:
  {"url": "https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTI4OTA3NjU=
title: ?_size=10 option for the arbitrary query page would be useful
state: open
created_at: 2021-01-24T20:55:35Z
updated_at: 2021-02-11T03:13:59Z
author_association: OWNER
type: issue
body:
  https://latest.datasette.io/fixtures?sql=select+*+from+compound_three_primary_keys&_size=10 - `_size=10` does not do anything at the moment. It would be useful if it did.

  Would also be good if it persisted in a hidden form field.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1200/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTI5MDQ1OTU=
title: Release notes for Datasette 0.54
state: closed
created_at: 2021-01-24T21:22:28Z
updated_at: 2021-01-25T17:42:21Z
closed_at: 2021-01-25T17:42:21Z
author_association: OWNER
type: issue
state_reason: completed
body:
  These will incorporate the release notes from the alpha, much expanded: https://github.com/simonw/datasette/releases/tag/0.54a0

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1201/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTI5MzEyNDQ=
title: Documentation convention for marking unstable APIs.
state: closed
created_at: 2021-01-24T23:47:18Z
updated_at: 2021-01-25T00:01:02Z
closed_at: 2021-01-25T00:01:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/1154#issuecomment-766462197_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1202/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTYwNzI1NzE0
title: Easier way to run Prettier locally
state: closed
created_at: 2021-01-25T01:39:06Z
updated_at: 2021-01-25T01:41:46Z
closed_at: 2021-01-25T01:41:46Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1203
body: Refs #1167
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1203/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTYwNzYwMTQ1
title: WIP: Plugin includes
state: open
created_at: 2021-01-25T03:59:06Z
updated_at: 2021-12-17T07:10:49Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1204
type: pull
body:
  Refs #1191

  Next steps:

  - [ ] Get comfortable that this pattern is the right way to go
  - [ ] Implement it for all of the other pages, not just the table page
  - [ ] Add a new set of plugin tests that exercise ALL of these new hook locations
  - [ ] Document, then ship

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1204/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTMwMjc4Mzc=
title: Rename /:memory: to /_memory
state: closed
created_at: 2021-01-25T05:04:56Z
updated_at: 2021-01-28T22:55:02Z
closed_at: 2021-01-28T22:51:42Z
author_association: OWNER
type: issue
state_reason: completed
body:
  For consistency with `/_internal` - and because then we don't need to escape the `:` characters.

  This change would need to be in before Datasette 1.0. I could land it earlier and set up redirects from the old URLs though.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1205/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTYwODMxNjM4
title: Release 0.54
state: closed
created_at: 2021-01-25T06:45:47Z
updated_at: 2021-01-25T17:33:30Z
closed_at: 2021-01-25T17:33:29Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1206
body: Refs #1201
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1206/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTM4ODE3NTY=
title: Document the Datasette(..., pdb=True) testing pattern
state: closed
created_at: 2021-01-26T02:48:10Z
updated_at: 2021-01-29T02:37:19Z
closed_at: 2021-01-29T02:12:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  If you're writing tests for a Datasette plugin and you get a 500 error from inside Datasette, you can cause Datasette to open a PDB session within the application server code by doing this:

  ```python
  ds = Datasette([db_path], pdb=True)
  response = await ds.client.get("/")
  ```

  You'll need to run `pytest -s` to interact with the debugger, otherwise you'll get an error.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1207/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTYxNTEyNTAz
title: added try / except to write_records
state: open
created_at: 2021-01-26T03:56:21Z
updated_at: 2021-01-26T03:56:21Z
author_association: FIRST_TIME_CONTRIBUTOR
pull_request: dogsheep/healthkit-to-sqlite/pulls/15
type: pull
body:
  to keep the data write from failing if it came across an error during processing. In particular when trying to convert my HealthKit zip file (and that of my wife's) it would consistently error out with the following:

  ```
  db.py 1709 insert_chunk
  result = self.db.execute(query, params)

  db.py 226 execute
  return self.conn.execute(sql, parameters)

  sqlite3.OperationalError:
  too many SQL variables

  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
  db.py 1709 insert_chunk
  result = self.db.execute(query, params)

  db.py 226 execute
  return self.conn.execute(sql, parameters)

  sqlite3.OperationalError:
  too many SQL variables

  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
  db.py 1709 insert_chunk
  result = self.db.execute(query, params)

  db.py 226 execute
  return self.conn.execute(sql, parameters)

  sqlite3.OperationalError:
  table rBodyMass has no column named metadata_HKWasUserEntered

  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
  healthkit-to-sqlite 8 <module>
  sys.exit(cli())

  core.py 829 __call__
  return self.main(*args, **kwargs)

  core.py 782 main
  rv = self.invoke(ctx)

  core.py 1066 invoke
  return ctx.invoke(self.callback, **ctx.params)

  core.py 610 invoke
  return callback(*args, **kwargs)

  cli.py 57 cli
  convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)

  utils.py 42 convert_xml_to_sqlite
  write_records(records, db)

  utils.py 143 write_records
  db[table].insert_all(

  db.py 1899 insert_all
  self.insert_chunk(

  db.py 1720 insert_chunk
  self.insert_chunk(

  db.py 1720 insert_chunk
  self.insert_chunk(

  db.py 1714 insert_chunk
  result = self.db.execute(query, params)

  db.py 226 execute
  return self.conn.execute(sql, parameters)

  sqlite3.OperationalError:
  table rBodyMass has no column named metadata_HKWasUserEntered
  ```

  Adding the try / except in the `write_records` seems to fix that issue. 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTQ1NTQ4ODE=
state: closed
created_at: 2021-01-26T20:56:28Z
updated_at: 2021-03-11T16:15:49Z
closed_at: 2021-03-11T16:15:49Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
title:
  A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper

body:
  Your code is full of open files that are never closed, especially when you deal with reading/writing json/yaml files.

  If you run python with warnings enabled this problem becomes evident.
  This probably contributes to some memory leaks in long running datasettes if the GC will not 'collect' those resources properly.

  This is easily fixed by using a context manager instead of just using open:
  ```python
  with open('some_file', 'w') as opened_file:
      opened_file.write('string')
  ```

  In some newer parts of the code you use Path objects 'read_text' and 'write_text' functions which close the file properly and are prefered in some cases.


  If you want I can create a PR for all places i found this pattern in.


  Bellow is a fraction of places where i found a ResourceWarning:
  ```python

  update-docs-help.py:
    20          actual = actual.replace("Usage: cli ", "Usage: datasette ")
    21:         open(docs_path / filename, "w").write(actual)
    22  

  datasette\app.py:
    210          ):
    211:             inspect_data = json.load((config_dir / "inspect-data.json").open())
    212              if immutables is None:

    266          if config_dir and (config_dir / "settings.json").exists() and not config:
    267:             config = json.load((config_dir / "settings.json").open())
    268          self._settings = dict(DEFAULT_SETTINGS, **(config or {}))

    445              self._app_css_hash = hashlib.sha1(
    446:                 open(os.path.join(str(app_root), "datasette/static/app.css"))
    447                  .read()

  datasette\cli.py:
    130      else:
    131:         out = open(inspect_file, "w")
    132      loop = asyncio.get_event_loop()

    459      if inspect_file:
    460:         inspect_data = json.load(open(inspect_file))
    461  

  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1208/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTUzNjc0MDI=
title: v0.54 500 error from sql query in custom template; code worked in v0.53; found a workaround
state: open
created_at: 2021-01-27T19:08:13Z
updated_at: 2021-01-28T23:00:27Z
author_association: NONE
type: issue
body:
  v0.54 500 error in sql query template; code worked in v0.53; found a workaround

  **schema:**   
  CREATE TABLE "talks" ("talk" TEXT,"series" INTEGER, "talkdate" TEXT)  
  CREATE TABLE "series" ("id" INTEGER PRIMARY KEY, "series" TEXT, talks_list TEXT default '', website TEXT default '');

  **Live example of correctly rendered template in v.053:** https://cosmotalks-cy6xkkbezq-uw.a.run.app/cosmotalks/talks/1

  **Description of problem:** I needed 'sql select' code in a custom row-mydatabase-mytable.html template to lookup the series name for a foreign key integer value in the talks table.  So `metadata.json` specifies the `datasette-template-sql` plugin.

  The code below worked perfectly in v0.53 (just the relevant sql statement part is shown; full code is [here](https://github.com/jrdmb/cosmotalks-datasette/blob/main/templates/row-cosmotalks-talks.html)):

  ```
  {# custom addition #}  
  {% for row in display_rows %}  
      ...  
      {% set sname = sql("select series from series where id = ?", [row.series]) %}  
      <strong>Series name: {{ sname[0].series }}  
      ...  
  {% endfor %}  
  {# End of custom addition #}  
  ```

  **In v0.54, that code resulted in a 500 error with a 'no such table series' message.**  A second query in that template also did not work but the above is fully illustrative of the problem.

  All templates were up-to-date along with datasette v0.54.

  **Workaround:** After fiddling around with trying different things, what worked was the syntax from [Querying a different database from the datasette-template-sql github repo](https://github.com/simonw/datasette-template-sql#querying-a-different-database) to add the database name to the sql statement:

  `{% set sname = sql("select series from series where id = ?", [row.series], database="cosmotalks") %}`

  Though this was found to work, it should not be necessary to add `database="cosmotalks"` since per the `datasette-template-sql` README, it's only needed when querying a different database, but here it's a table within the same database.


node_id: MDU6SXNzdWU3OTYyMzQzMTM=
title: Immutable Database w/ Canned Queries
state: closed
created_at: 2021-01-28T18:08:29Z
updated_at: 2021-02-05T11:30:34Z
closed_at: 2021-02-05T11:30:34Z
author_association: NONE
type: issue
state_reason: completed
body:
  I have a database that I only want to read from; when instructing datasette to treat the database as immutable my defined canned queries disappear. Are these two features incompatible or have I hit an unintended bug? Thanks for datasette in any way, it's a joy to use!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1210/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTY3MzY2MDc=
title: Not all quoted statuses get fetched?
state: closed
created_at: 2021-01-29T09:48:44Z
updated_at: 2021-02-03T10:36:36Z
closed_at: 2021-02-03T10:36:36Z
author_association: NONE
type: issue
state_reason: completed
body:
  ![image](https://user-images.githubusercontent.com/42315895/106259325-5f75dc80-621f-11eb-8311-db8f2fe2a257.png)

  In my database I have 13300 quote tweets, but eta 3600 have `quoted_status` empty.

  I fetched some of them using `https://api.twitter.com/1.1/statuses/show.json?id=xx` and they did have ids of quoted tweets.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTcwOTcxNDA=
title: Use Data from SQLite in other commands
state: open
created_at: 2021-01-29T18:35:52Z
updated_at: 2021-02-12T18:29:43Z
author_association: CONTRIBUTOR
type: issue
body:
  As a total beginner here how could you access data from the sqlite table to run other commands.

  What I am thinking is I want to get all the repos in an organization then using the repo list pull all the commit messages for each repo. 

  I love this project by the way!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTY0MTcyMTQw
title: fixing typo in get cli help text
state: closed
created_at: 2021-01-29T18:57:04Z
updated_at: 2021-05-19T16:07:09Z
closed_at: 2021-05-19T16:07:09Z
author_association: CONTRIBUTOR
pull_request: dogsheep/github-to-sqlite/pulls/61
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTY0MjE1MDEx
title: fix for problem in Table.insert_all on search for columns per chunk of rows
state: closed
created_at: 2021-01-29T20:16:07Z
updated_at: 2021-02-14T21:04:13Z
closed_at: 2021-02-14T21:04:13Z
author_association: NONE
pull_request: simonw/sqlite-utils/pulls/225
type: pull
body:
  Hi,

  I ran into a problem when trying to create a database from my Apple Healthkit data using [healthkit-to-sqlite](https://github.com/dogsheep/healthkit-to-sqlite). The program crashed because of an invalid insert statement that was generated for table `rDistanceCycling`. 

  The actual problem turned out to be in [sqlite-utils](https://github.com/simonw/sqlite-utils). `Table.insert_all` processes the data to be inserted in chunks of rows and checks for every chunk which columns are used, and it will collect all column names in the variable `all_columns`.  The collection of columns is done using a nested list comprehension that is not completely correct. 

  I'm using a Windows machine and had to make a few adjustments to the tests in order to be able to run them because they had a posix dependency.

  Thanks, kind regards,

  Frans

  ```
  # this is a (condensed) chunk of data from my Apple healthkit export that caused the problem.
  # the 3 last items in the chunk have additional keys: metadata_HKMetadataKeySyncVersion and metadata_HKMetadataKeySyncIdentifier

  chunk = [{'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '7.0.1',
            'device': '<<HKDevice: 0x281cf6c70>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>',
            'unit': 'km', 'creationDate': '2020-10-10 12:29:09 +0100', 'startDate': '2020-10-10 12:29:06 +0100',
            'endDate': '2020-10-10 12:29:07 +0100', 'value': '0.00518016'},
           {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '7.0.1',
            'device': '<<HKDevice: 0x281cf6c70>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:7.0.1>',
            'unit': 'km', 'creationDate': '2020-10-10 12:29:10 +0100', 'startDate': '2020-10-10 12:29:07 +0100',
            'endDate': '2020-10-10 12:29:08 +0100', 'value': '0.00544049'},
           {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6',
            'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',
            'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:40:50 +0100',
            'endDate': '2020-07-15 16:42:49 +0100', 'value': '0.952092', 'metadata_HKMetadataKeySyncVersion': '1',
            'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520450.99823:616520569.99360:119'},
           {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6',
            'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',
            'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:42:49 +0100',
            'endDate': '2020-07-15 16:44:51 +0100', 'value': '0.848983', 'metadata_HKMetadataKeySyncVersion': '1',
            'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520569.99360:616520691.98826:119'},
           {'sourceName': 'AppleÂ\xa0Watch van Frans', 'sourceVersion': '6.2.6',
            'device': '<<HKDevice: 0x281cf83e0>, name:Apple Watch, manufacturer:Apple Inc., model:Watch, hardware:Watch3,4, software:6.2.6>',
            'unit': 'km', 'creationDate': '2020-10-14 05:54:12 +0100', 'startDate': '2020-07-15 16:44:51 +0100',
            'endDate': '2020-07-15 16:46:50 +0100', 'value': '0.834403', 'metadata_HKMetadataKeySyncVersion': '1',
            'metadata_HKMetadataKeySyncIdentifier': '3:674DBCDB-3FE8-40D1-9FC1-E54A2B413805:616520691.98826:616520810.98305:119'}]



  def all_columns_old():
      all_columns = [col for col in chunk[0]]
      all_columns += [column for record in chunk
                             for column in record if column not in all_columns]
      return all_columns


  def all_columns_new():
      all_columns = [col for col in chunk[0]]
      for record in chunk:
          all_columns += [column for column in record if column not in all_columns]
      return all_columns



  if __name__ == '__main__':
      from pprint import pprint

      print('problem: ')
      pprint(all_columns_old())
      print('\nfix: ')
      pprint(all_columns_new())

  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/225/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTY0NjA4MjY0
title: Use context manager instead of plain open
state: closed
created_at: 2021-01-31T07:58:10Z
updated_at: 2021-03-11T16:15:50Z
closed_at: 2021-03-11T16:15:50Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1211
type: pull
body:
  Context manager with open closes the files after usage. Fixes: https://github.com/simonw/datasette/issues/1208

  When the object is already a pathlib.Path i used read_text
  write_text functions

  In some cases pathlib.Path.open were used in context manager,
  it is basically the same as builtin open.

  Tests are passing: 850 passed, 5 xfailed, 10 xpassed

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1211/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTc2NTE4MzE=
title: Tests are very slow.
state: closed
created_at: 2021-01-31T08:06:16Z
updated_at: 2021-02-19T22:54:13Z
closed_at: 2021-02-19T22:54:13Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Working on my PR i noticed that tests are very slow.

  The plain pytest run took about 37 minutes for me.
  However i could shave of about 10 minutes from that if i used pytest-xdist to parallelize execution.
  `pytest -n 8` is run only in 28 minutes on my machine.

  I can create a PR to mention that in your documentation.
  This will be a simple change to add pytest-xdist to requirements and change a command to run pytest in documentation.

  Does that make sense to you?

  After a bit more investigation it looks like python-xdist is not an answer. It creates a race condition for tests that try to clead temp dir before run.

  Profiling shows that most time is spent on conn.executescript(TABLES) in make_app_client function. Which makes sense.

  Perhaps the better approach would be look at the app_client fixture which is already session scoped, but not used by all test cases.
  And/or use conn = sqlite3.connect(":memory:") which is much faster.
  And/or truncate tables after each TC instead of deleting the file and re-creating them.

  I can take a look which is the best approach if you give the go-ahead. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1212/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTc3Mjg5Mjk=
title: QUESTION: extract full text
state: open
created_at: 2021-01-31T14:50:10Z
updated_at: 2021-01-31T14:50:10Z
author_association: NONE
type: issue
body:
  This may be solved or a feature already, but I couldn't figure it out, is it possible to extract and store also full text from the saved pages? The same way that Pocket parses the text, it'd be amazing to be able to store (and thus make searchable later) the text.

  Thank you very much for the project, it's such an amazing idea! 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTc3ODQwODA=
title: Stargazers and workflows commands always require an auth file when using GITHUB_TOKEN
state: open
created_at: 2021-01-31T18:56:05Z
updated_at: 2021-01-31T18:56:05Z
author_association: CONTRIBUTOR
type: issue
body:
  Requested fix in https://github.com/dogsheep/github-to-sqlite/pull/59

  The stargazers and workflows commands always require an auth file, even when using a `GITHUB_TOKEN`.  Other commands don't require the auth file.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/62/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTk2NjM5NTk=
title: gzip support for HTML (and JSON) responses
state: open
created_at: 2021-02-02T20:36:28Z
updated_at: 2021-02-02T20:41:55Z
author_association: OWNER
type: issue
body:
  This page https://datasette-tiles-demo.datasette.io/San_Francisco/tiles is 2MB because of all of the base64 images. Gzipped it's 1.5MB.

  Since Datasette is usually deployed without a frontend gzipping proxy, Datasette itself needs to solve for this.

  Gzipping everything won't work because some endpoints - the all-rows CSV endpoint and the download-database endpoint - are streaming and hence can't be buffered-and-gzipped.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1213/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU3OTk2OTM3Nzc=
title: Re-submitting filter form duplicates _x querystring arguments
state: closed
created_at: 2021-02-02T21:13:35Z
updated_at: 2021-02-02T21:28:53Z
closed_at: 2021-02-02T21:21:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Really nasty bug, caused by #1194 fix in 07e163561592c743e4117f72102fcd350a600909

  Navigate to this page: https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id

  Click "Apply" to submit the form and the resulting URL is https://github-to-sqlite.dogsheep.net/github/labels?_search=help&_sort=id&_search=help&_sort=id

  That's because the (truncated) HTML for the form looks like this:

  ```html
      ... <input id="_search" type="search" name="_search" value="help">
      ...
              <div class="select-wrapper small-screen-only">
                  <select name="_sort" id="sort_by">
                      <option value="">Sort...</option>
                              <option value="id" selected>Sort by id</option>
                              <option value="node_id">Sort by node_id</option>
                              ...
                  </select>
              </div>
              ...
              <input type="hidden" name="_search" value="help">
              <input type="hidden" name="_sort" value="id">
          <input type="submit" value="Apply">
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1214/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDA2NjkzNDc=
title: /-/databases should reflect connection order, not alphabetical order
state: closed
created_at: 2021-02-03T20:20:23Z
updated_at: 2021-06-02T03:10:19Z
closed_at: 2021-06-02T03:10:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The order in which databases are attached to Datasette matters - it affects the homepage, and it's beginning to influence how certain plugins work (see https://github.com/simonw/datasette-tiles/issues/8).

  Two years ago in cccea85be6aaaeadb31f3b588ec7f732628815f5 I made `/-/databases` return things in alphabetical order, to fix a test failure in Python 3.5.

  Python 3.5 is no longer supported, so this is no longer necessary - and this behaviour should now be treated as a bug.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1216/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDE3ODA2MjU=
title: SSL Error
state: open
created_at: 2021-02-05T02:12:56Z
updated_at: 2021-02-07T18:45:04Z
author_association: NONE
type: issue
body:
  Here's the error I get when running `pip install pocket-to-sqlite`: 

  ```
    Could not fetch URL https://pypi.python.org/simple/pocket-to-sqlite/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661) - skipping
    Could not find a version that satisfies the requirement pocket-to-sqlite (from versions: )
  No matching distribution found for pocket-to-sqlite
  ```

  Does this require python 3? 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDI1MTMzNTk=
title: Possible to deploy as a python app (for Rstudio connect server)?
state: open
created_at: 2021-02-05T22:21:24Z
updated_at: 2022-11-04T11:37:52Z
author_association: NONE
type: issue
body:
  Is it possible to deploy a `datasette` application as a python web app?

  In my enterprise, I have option to deploy python apps via [Rstudio Connect](https://github.com/rstudio/rsconnect-python), and I would like to publish a `datasette` dashboard for sharing.

  I welcome any pointers to converting `datasette serve` into a python app that can be run as something like `python datasette.py --my_data.db`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1217/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDI1ODM0NTA=
title: 3.4 release is broken - includes a rogue line
state: closed
created_at: 2021-02-06T02:08:01Z
updated_at: 2021-02-06T02:10:26Z
closed_at: 2021-02-06T02:10:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I started seeing weird errors, caused by this line: https://github.com/simonw/sqlite-utils/blob/f8010ca78fed8c5fca6cde19658ec09fdd468420/sqlite_utils/cli.py#L1-L3

  That was added by accident in 1b666f9315d4ea6bb332b2e75e48480c26100199

  I'm surprised the tests didn't catch this!

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/226/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDMzMzM3Njk=
title: KeyError: 'Contents' on running upload
state: open
created_at: 2021-02-08T08:36:37Z
updated_at: 2021-07-22T06:40:25Z
author_association: NONE
type: issue
body:
  Following the readme, on big sur, and having entered my auth creds via `dogsheep-photos s3-auth`:

  ```
  (venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos upload photos.db     ~/Pictures/Photos\ /Users/robin/Pictures/Library.photoslibrary --dry-run

  Fetching existing keys from S3...
  Traceback (most recent call last):
    File "/Users/robin/datasette/venv/bin/dogsheep-photos", line 8, in <module>
      sys.exit(cli())
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py", line 96, in upload
      key.split(".")[0] for key in get_all_keys(client, creds["photos_s3_bucket"])
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/utils.py", line 46, in get_all_keys
      for row in page["Contents"]:
  KeyError: 'Contents'
  ```

  Possibly since the bucket is in `EU (London) eu-west-2` and this into is not requested?

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDMzMzg3Mjk=
title: photo-to-sqlite: command not found
state: open
created_at: 2021-02-08T08:42:57Z
updated_at: 2021-02-12T15:00:44Z
author_association: NONE
type: issue
body:
  Having installed in a venv I get:
  ```
  (venv) (base) Robins-MacBook:datasette robin$ photo-to-sqlite apple-photos photos.db

  -bash: photo-to-sqlite: command not found
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDMzNTY5NDI=
title: /usr/local/opt/python3/bin/python3.6: bad interpreter: No such file or directory
state: open
created_at: 2021-02-08T09:07:00Z
updated_at: 2021-02-23T12:12:17Z
author_association: NONE
type: issue
body:
  Error as above, however I do have python3.8 and the readme indicates this is supported.

  ```
  (venv) (base) Robins-MacBook:datasette robin$ ls /usr/local/opt/python3/bin/

  .. pip3 python3 python3.8
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1218/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDM5Mjk2OTQ=
title: Try profiling Datasette using scalene
state: open
created_at: 2021-02-08T20:37:06Z
updated_at: 2021-02-08T22:13:00Z
author_association: OWNER
body: https://github.com/emeryberger/scalene looks like an interesting profiling tool.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1219/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDY3NDMxMTY=
title: Installing datasette via docker: Path 'fixtures.db' does not exist
state: closed
created_at: 2021-02-11T21:09:14Z
updated_at: 2021-02-12T21:35:17Z
closed_at: 2021-02-12T21:35:17Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi,
  If I run

  ```
  docker run -p 8001:8001 -v `pwd`:/mnt \                                                                         1 ↵
      datasetteproject/datasette \
      datasette -p 8001 -h 0.0.0.0 fixtures.db
  ```

  I have 

  ```
  Error: Invalid value for '[FILES]...': Path 'fixtures.db' does not exist.
  ```

  If I run `test -f fixtures.db && echo "it exists."` I have `it exists.`.

  What's my error?

  Thank you

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1220/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDY4NDk0MjQ=
title: Support SSL/TLS directly
state: closed
created_at: 2021-02-12T00:18:29Z
updated_at: 2022-12-18T02:39:04Z
closed_at: 2021-02-12T00:52:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This should be pretty easy because Uvicorn supports them already. Need a good mechanism for testing it - https://pypi.org/project/trustme/ looks ideal.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1221/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTcyMjA5MjQz
title: --ssl-keyfile and --ssl-certfile, refs #1221
state: closed
created_at: 2021-02-12T00:45:58Z
updated_at: 2021-02-12T00:52:18Z
closed_at: 2021-02-12T00:52:17Z
author_association: OWNER
pull_request: simonw/datasette/pulls/1222
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1222/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0NTcyMjU0MTAz
title: Add compile option to Dockerfile to fix failing test (fixes #696)
state: closed
created_at: 2021-02-12T03:38:05Z
updated_at: 2021-03-07T12:01:12Z
closed_at: 2021-03-07T07:41:17Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/1223
type: pull
body:
  This test was failing when run inside the Docker container: `test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]`,

  with this error:

  ```
      def test_searchable(app_client, path, expected_rows):
          response = app_client.get(path)
  >       assert expected_rows == response.json["rows"]
  E       AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
  E         Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
  E         Full diff:
  E         + []
  E         - [[1, 'barry cat', 'terry dog', 'panther'],
  E         -  [2, 'terry dog', 'sara weasel', 'puma']]
  ```

  The issue was that the version of sqlite3 built inside the Docker container was built with FTS3 and FTS4 enabled, but without the
  `SQLITE_ENABLE_FTS3_PARENTHESIS` compile option passed, which adds support for using `AND` and `NOT` within `match` expressions (see https://sqlite.org/fts3.html#compiling_and_enabling_fts3_and_fts4 and https://www.sqlite.org/compile.html).

  Without this, the `AND` used in the search in this test was being interpreted as a literal string, and so no matches were found. Adding this compile option fixes this.

  ---

  I actually ran into this issue because the same test was failing when I ran the test suite on my own machine, outside of Docker, and so I eventually tracked this down to my system sqlite3 also being compiled without this option.

  I wonder if this is a sign of a slightly deeper issue, that Datasette can silently behave differently based on the version and compilation of sqlite3 it is being used with. On my own system I fixed the test suite by running `pip install pysqlite3-binary`, so that this would be picked up instead of the `sqlite` package, as this seems to be compiled using this option, . Maybe using `pysqlite3-binary` could be installed/recommended by default so a more deterministic version of sqlite is used? Or there could be some feature detection done on the available sqlite version, to know what features are available and can be used/tested?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1223/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDcxNzQxNjE=
title: Error reading csv files with large column data
state: closed
created_at: 2021-02-12T11:51:47Z
updated_at: 2021-02-16T11:48:03Z
closed_at: 2021-02-14T21:17:19Z
author_association: NONE
type: issue
state_reason: completed
body:
  *Feel free to close this issue - I mostly added it for reference for future folks that run into this :)*

  I have a CSV file with one column that has very long strings.  When i try to import this file via the `insert` command I get the following error: 
  ```
  sqlite-utils insert database.db table_name file_with_large_column.csv

  Traceback (most recent call last):
    File "/usr/local/bin/sqlite-utils", line 10, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 774, in insert
      default=default,
    File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 705, in insert_upsert_implementation
      docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs
    File "/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py", line 1852, in insert_all
      first_record = next(records)
    File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 703, in <genexpr>
      docs = (decode_base64_values(doc) for doc in docs)
    File "/usr/local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 681, in <genexpr>
      docs = (dict(zip(headers, row)) for row in reader)
  _csv.Error: field larger than field limit (131072)
  ```
  Built with the docker image `datasetteproject/datasette:0.54` with the following versions:
  ```
  # sqlite-utils --version
  sqlite-utils, version 3.4.1

  # datasette --version
  datasette, version 0.54
  ```
  It appears this is a [known issue](https://stackoverflow.com/a/54517228/2761423) reading in csv files in python and [doesn't look to be modifiable](https://github.com/python/cpython/blob/ea46579067fd2d4e164d6605719ffec690c4d621/Modules/_csv.c#L1685) through system / env vars (i may be very wrong on this).

  Noting that using sqlite3 `import` command work without error (not using the python csv reader)
  ```
  sqlite3 database.db
  sqlite> .mode csv
  sqlite> .import file_with_large_column.csv table_name
  ```
  Sadly I couldn't see an easy way around this while using the cli as it appears this value needs to be changed in python code. FWIW I've switched to using https://datasette.io/tools/csvs-to-sqlite for importing csv data and it's working well. 

  Finally, I'm loving https://datasette.io/ thank you very much for an amazing tool and data ecosytem 🙇‍♀️ 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/227/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDc0MzMxODE=
title: can't start immutable databases from configuration dir mode
state: closed
created_at: 2021-02-12T17:50:13Z
updated_at: 2021-03-29T00:17:31Z
closed_at: 2021-03-29T00:17:31Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Say I have a `/databases/` directory with multiple sqlite db files in that dir (`1.db` & `2.db`) and an `inspect-data.json` file.

  If I start datasette via `datasette -h 0.0.0.0 /databases/` then the resulting databases are set to `is_mutable: true` as inspected via http://127.0.0.1:8001/-/databases.json

  I don't want to have to list out the databases by name, e.g. `datasette -i /databases/1.db -i /databases/2.db` as i want the system to autodetect the sqlite dbs i have in the configuration directory 

  According to the docs outlined in https://docs.datasette.io/en/latest/settings.html?highlight=immutable#configuration-directory-mode this should be possible
  > `inspect-data.json` the result of running datasette inspect - any database files listed here will be treated as immutable, so they should not be changed while Datasette is running
   
  I believe that if the `inspect-json.json` file present, then in theory the databases will be automatically set to immutable via this code https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/app.py#L211-L216

  However it appears the Click Multiple Options will return a tuple via https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L311-L317

  The resulting tuple is passed to the Datasette app via `kwargs` and overrides the behaviour to set the databases to immutable via this arg https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/app.py#L182

  If you think this is a bug and needs fixing, I am willing to make a PR to check for the empty `immutable` tuple before calling the Datasette class initializer as I think leaving that class interface alone is the best path here.

  Thoughts?

  Also - i'm loving Datasette, it truly is a wonderful tool, thank you :)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/1224/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDc0MzcwODk=
title: --no-headers option for CSV and TSV
state: closed
created_at: 2021-02-12T17:56:51Z
updated_at: 2021-12-26T07:01:31Z
closed_at: 2021-02-14T22:25:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c has a fascinating CSV file that doesn't have a header row - it starts like this:

  ```csv
  Computation and measurement of turbulent flow through idealized turbine blade passages,,"Loizou, Panos A.",https://isni.org/isni/0000000136122593,,University of Manchester,https://isni.org/isni/0000000121662407,1989,Thesis (Ph.D.),,Physical Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232781,
  "Prolactin and growth hormone secretion in normal, hyperprolactinaemic and acromegalic man",,"Prescott, R. W. G.",https://isni.org/isni/0000000134992122,,University of Newcastle upon Tyne,https://isni.org/isni/0000000104627212,1983,Thesis (Ph.D.),,Biological Sciences,,,https://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.232784,
  ```

  It would be useful if `sqlite-utils insert ... --csv` had a mechanism for importing files like this one.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/228/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDc4MTcxOTc=
title: Hitting `_csv.Error: field larger than field limit (131072)`
state: closed
created_at: 2021-02-13T19:52:44Z
updated_at: 2021-02-14T21:33:33Z
closed_at: 2021-02-14T21:33:33Z
author_association: NONE
type: issue
state_reason: completed
body:
  I have a csv file where one of the fields is so large it is throwing an exception with this error and stops loading:
  	```
  	_csv.Error: field larger than field limit (131072)
  	```

  The stack trace occurs here: https://github.com/simonw/sqlite-utils/blob/3.1/sqlite_utils/cli.py#L633


  There is a way to handle this that helps:
  https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072

  One issue I had with this problem was sqlite-utils only provides limited context as to where the problem line is.
  There is the progress bar, but that is by percent rather than by line number.  It would have been helpful if it could have provided a line number.

  Also, it would have been useful if it had allowed the loading to continue with later lines.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/229/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDgwMDgzMDU=
title: --sniff option for sniffing delimiters
state: closed
created_at: 2021-02-14T17:43:54Z
updated_at: 2021-02-14T21:15:33Z
closed_at: 2021-02-14T19:24:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/230/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU4MDgwMjg3NTc=
title: limit=X, offset=Y parameters for more Python methods
state: closed
created_at: 2021-02-14T19:31:23Z
updated_at: 2021-02-14T20:03:08Z
closed_at: 2021-02-14T20:03:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I'm going to add a `offset=` parameter to support this case. Thanks for the suggestion!

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/224#issuecomment-778828495_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/231/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


