# issues

node_id: MDU6SXNzdWU0NDMwMzQwMDM=
title: Write release notes for 0.28
state: closed
created_at: 2019-05-11T20:36:56Z
updated_at: 2019-05-19T21:24:44Z
closed_at: 2019-05-19T21:24:20Z
author_association: OWNER
body:
  So much new stuff!

  https://github.com/simonw/datasette/compare/0.27...master

type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/463/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDMwMzQyMTg=
title: Add Glitch to Getting Started docs section
state: closed
created_at: 2019-05-11T20:39:39Z
updated_at: 2019-05-16T05:04:35Z
closed_at: 2019-05-16T05:03:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Glitch is by far the easiest way to start trying out Datasette. Add a section to https://datasette.readthedocs.io/en/latest/getting_started.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/464/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDMwMzg1ODQ=
title: Decide what to do about /-/inspect
state: closed
created_at: 2019-05-11T21:39:46Z
updated_at: 2019-06-28T16:34:33Z
closed_at: 2019-06-28T16:34:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It's not clear to me what this endpoint should do now as a result of #419 - it's still useful to be able to introspect databases for tools like datasette-registry, but since we aren't pre-calculating introspection data any more I need to rethink the approach.

  For one thing, this endpoint may need to be paginated. Or maybe it should be split up into separate endpoints for each connected database? Those should probably be paginated too seeing as fivethirtyeight has 400+ tables.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/465/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDMwNDA2NjU=
title: Move "no such module: VirtualSpatialIndex" code elsewhere
state: closed
created_at: 2019-05-11T22:09:00Z
updated_at: 2022-01-20T21:29:41Z
closed_at: 2019-05-11T22:57:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  We currently show a useful warning (from #331) when the user tries to open a spatialite database without first loading the module:

  https://github.com/simonw/datasette/blob/c692cd291111050483a32bea1ee08e994a0b781b/datasette/app.py#L547-L554

  This code is part of `.inspect()` which is going away - see #462 - so I need to find somewhere else for it to live.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/466/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDQ3MTEyNTQ=
title: Index page row counts only for DBs with < 30 tables (10ms count limit per table)
state: closed
created_at: 2019-05-16T01:21:36Z
updated_at: 2019-05-16T03:03:45Z
closed_at: 2019-05-16T03:03:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Split out from #460.

  If a database is mutable, calculating row counts gets expensive. I'm only going to calculate row counts for the index page if it has less than X tables (both hidden and non-hidden) AND each table can be counted in less than 10ms.

  If any count takes longer than 10ms I'll cancel the counting entirely. We currently show an inaccurate count if this happens, which is just confusing.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/467/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDQ3NDYwMjE=
title: Pagination for the database index page
state: closed
created_at: 2019-05-16T04:13:56Z
updated_at: 2020-10-16T23:20:26Z
closed_at: 2020-10-16T23:20:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Some databases have a LOT of tables. Now that we often calculate table row counts dynamically we could really speed things up by paginating the database index page, e.g. http://fivethirtyeight-datasette.herokuapp.com/fivethirtyeight

  If we're paginating, having a filter-search-for-table widget (similar to the search-for-database widget I'm planning for the homepage) would make sense.

  Related: pagination for homepage #461 and Datasette Library #417

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/468/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDQ3NDkzNzM=
title: publish commands should use new -i option
state: closed
created_at: 2019-05-16T04:31:40Z
updated_at: 2019-05-19T22:53:41Z
closed_at: 2019-05-19T22:53:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I can make this change only after releasing 0.28 - if I make the change earlier than that `publish heroku` etc will break because they will install the latest release of Datasette which will not understand the `-i` option.

  This is a one-line fix: replace this:

  https://github.com/simonw/datasette/blob/2ad9d15cd6901654e6801e2faa29e6fc08bae5fa/datasette/utils.py#L489

  With this: (need to do it for other publishers too though)

  ```
  quoted_files = " ".join(
      ["-i {}".format(shlex.quote(file_name)) for file_name in file_names]
  )
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/469/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDQ5OTc5Mzc=
title: /-/databases showing currently attached database details
state: closed
created_at: 2019-05-16T14:45:18Z
updated_at: 2019-05-19T19:28:44Z
closed_at: 2019-05-16T14:50:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Split from #419. Mainly useful to see what is connected as mutable v.s. immutable.

  Also helps fill the gap left by `/-/inspect` until #465 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/470/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDUwMDMwMjk=
title: ?_hash=1 and --config hash_urls:1 should only work for immutable databases
state: closed
created_at: 2019-05-16T14:54:25Z
updated_at: 2019-05-16T15:11:03Z
closed_at: 2019-05-16T15:11:03Z
author_association: OWNER
body: Split from #419.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/471/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDUyMzAwNzc=
title: Rename "publish now" to "publish nowv1"
state: closed
created_at: 2019-05-17T01:58:52Z
updated_at: 2019-05-19T18:07:39Z
closed_at: 2019-05-19T18:07:39Z
author_association: OWNER
body: This will help clarify that you need a nowv1 account use it.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/472/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDU4NTA5MzQ=
title: Plugin hook: filters_from_request
state: closed
created_at: 2019-05-19T18:44:33Z
updated_at: 2021-12-17T23:11:30Z
closed_at: 2021-12-17T19:02:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I meant to add this as part of the facets plugin mechanism but didn't quite get to it. Original idea was to allow plugins to register extra filters, as seen in `datasette/filters.py`:

  https://github.com/simonw/datasette/blob/260085838887ee343f4d3b177c422e7aef5ade9d/datasette/filters.py#L83-L98

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/473/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDU4NTU3ODk=
title: Do not allow downloads of mutable databases
state: closed
created_at: 2019-05-19T19:35:32Z
updated_at: 2019-05-19T20:41:17Z
closed_at: 2019-05-19T20:41:16Z
author_association: OWNER
type: issue
state_reason: completed
body:
  If the file changes during download it will probably result in a corrupt download. Safer not to allow downloads at all of mutable databases.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/474/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDU4NTU5MTA=
title: Documentation for about and about_url metadata
state: closed
created_at: 2019-05-19T19:36:59Z
updated_at: 2019-05-19T20:13:36Z
closed_at: 2019-05-19T20:13:36Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Added in https://github.com/simonw/datasette/commit/bf6b0f918de4aeee7c1036ac975ce2fb23237da7 without docs.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/475/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDU4NTg0OTE=
title: Remove "datasette skeleton"
state: closed
created_at: 2019-05-19T20:04:11Z
updated_at: 2019-05-19T20:06:06Z
closed_at: 2019-05-19T20:06:06Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It doesn't work any more, and it's not a particularly useful feature - I've hardly used it since I added it.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/476/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDU4NjI1MDE=
title: Documentation for ArrayFacet (facet by JSON array)
state: closed
created_at: 2019-05-19T20:47:27Z
updated_at: 2019-05-29T21:39:12Z
closed_at: 2019-05-19T21:19:43Z
author_association: OWNER
body: This is missing from https://datasette.readthedocs.io/en/0.27.1/facets.html right now
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/477/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDU4NjgyMzQ=
title: Make it so Docker build doesn't delay PyPI release
state: closed
created_at: 2019-05-19T21:52:10Z
updated_at: 2019-07-08T03:30:41Z
closed_at: 2019-07-07T20:03:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Datasette automated releases currently include building a Docker image that has a full custom-compiled version of SQLite and SpatiaLite. This takes ages!

  I still want to publish this Docker image (to https://hub.docker.com/r/datasetteproject/datasette/tags ) but I'd like it if this wasn't a blocker on pushing the new package to PyPI. Ideally PyPI publish would happen first.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/478/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjgwMjA0Mjc2
title: doc typo fix
state: closed
created_at: 2019-05-19T22:54:25Z
updated_at: 2019-05-20T16:42:29Z
closed_at: 2019-05-20T16:42:29Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/479
body: Fix typo in performance doc page
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/479/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjgwMjA1NTAy
title: Split pypi and docker travis tasks
state: closed
created_at: 2019-05-19T23:14:37Z
updated_at: 2019-07-07T20:03:20Z
closed_at: 2019-07-07T20:03:20Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/480
type: pull
body:
  Resolves #478 

  This *should* work, but because this is a change that'll only really be testable on a) this repo, b) master branch, this might fail fast if I didn't get the configurations right. 


  Looking at #478 it should just be as simple as splitting out the docker and pypi processes into separate jobs, but it might end up being more complicated than that, depending on what pre-processes the pypi deployment needs, and how travisci treats deployment steps without scripts in general. 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/480/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDY0Mjk0MjE=
title: Facet by date
state: closed
created_at: 2019-05-21T05:55:54Z
updated_at: 2019-05-29T21:39:12Z
closed_at: 2019-05-21T06:09:49Z
author_association: OWNER
body: Ability to facet on datetime fields by their date.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/481/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDY0MzM3MzU=
title: Example of a custom facet plugin is incorrect
state: closed
created_at: 2019-05-21T06:12:47Z
updated_at: 2019-07-07T23:19:10Z
closed_at: 2019-07-07T23:19:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The function signatures are wrong on https://datasette.readthedocs.io/en/0.28/plugins.html#register-facet-classes

  The new signatures are: `async def suggest(self)` and `async def facet_results(self)` - the `sql` and `params` are now passed to the class constructor.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/482/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDc0MDg1Mjc=
title: Option to facet by date using month or year
state: open
created_at: 2019-05-23T01:25:29Z
updated_at: 2019-05-29T21:38:27Z
author_association: OWNER
type: issue
body:
  Facet by date (from #481) can take datetimes and facet them by the day component.

  https://latest.datasette.io/fixtures/facetable?_facet_date=created

  I'd like to also be able to facet by month or year.

  I'm not sure what the best way to achieve this is. Could be two more Facet classes (YearFacet and MonthFacet) but I think it might be nicer if the existing DateFacet could take an optional argument that changed its behaviour. But... if I do that, do I expose it in the UI somewhere or is it only available to URL-hackers?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/483/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDc0NTE0OTI=
title: Mechanism for displaying summary of m2m relationships in rows on table view
state: open
created_at: 2019-05-23T05:02:41Z
updated_at: 2019-05-23T06:34:05Z
author_association: OWNER
type: issue
body:
  Part of #354 (m2m support)

  It would be fantastic if rows that are part of a m2m relationship could display it in an additional column in the table view.

  It might look something like this: https://russian-ira-facebook-ads.datasettes.com/russian-ads-919cbfd/display_ads?_search=black+lives+matter

  <img width="1475" alt="russian-ads__display_ads__50_rows_where_where_search_matches__black_lives_matter_" src="https://user-images.githubusercontent.com/9599/58226788-12690580-7cdd-11e9-9925-73153b27a413.png">

  That example [was achieved](https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/daf51a8c50a78e8bc7971c211005fd85e66ccf64/russian-ads-metadata.yaml#L72-L77) using a custom SQL query and [datasette-json-html](https://github.com/simonw/datasette-json-html) - but I'd like this to be a built-in feature instead.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/484/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDc0NjkyNTM=
title: Improvements to table label detection
state: open
created_at: 2019-05-23T06:19:49Z
updated_at: 2022-10-03T00:04:42Z
author_association: OWNER
type: issue
body:
  Label detection doesn't work if the primary key is called pk rather than id,  so this page doesn't work: https://latest.datasette.io/fixtures/roadside_attraction_characteristics

  Code is here: 

  https://github.com/simonw/datasette/blob/cccea85be6aaaeadb31f3b588ec7f732628815f5/datasette/app.py#L644-L653

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/485/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDgxODkyOTg=
title: Ability to add extra routes and related templates
state: closed
created_at: 2019-05-24T14:04:25Z
updated_at: 2019-05-24T14:43:28Z
closed_at: 2019-05-24T14:43:09Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon

  Thank for an excellent job! Datasette is such an obviously good idea (once you have that idea!) and so well done. The only thing that I miss, is the ability to add extras routes (with associated jinja2-templates). For most of the datasets, that I would like to publish, I would also like at least a page, that describes the data (semantics, provenance, biases...) and a page explaining our cookie- and privacy-policies (which would allows us to use something like Goggle Analytics).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/486/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDgzOTE0OTI=
title: Option to ignore inserts if primary key exists already
state: closed
created_at: 2019-05-25T00:17:12Z
updated_at: 2019-05-29T05:09:01Z
closed_at: 2019-05-29T04:18:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > I've just noticed that SQLite lets you IGNORE inserts that collide with a pre-existing key. This can be quite handy if you have a dataset that keeps changing in part, and you don't want to upsert and replace pre-existing PK rows but you do want to ignore collisions to existing PK rows.
  > 
  > Do `sqlite_utils` support such (cavalier!) behaviour?

  _Originally posted by @psychemedia in https://github.com/simonw/sqlite-utils/issues/18#issuecomment-480621924_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDgzOTU2NjU=
title: Release notes for 1.0
state: closed
created_at: 2019-05-25T00:58:03Z
updated_at: 2019-05-25T01:18:27Z
closed_at: 2019-05-25T01:06:52Z
author_association: OWNER
body: https://github.com/simonw/sqlite-utils/compare/0.14...251e473
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDg2NjQ3OTI=
title: Refactor database methods off Datasette class
state: closed
created_at: 2019-05-27T04:52:41Z
updated_at: 2019-05-27T20:05:34Z
closed_at: 2019-05-27T05:08:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Methods like this one:

  https://github.com/simonw/datasette/blob/182a3017c24e3fa3af60e4ac0c91c7e48f8736fd/datasette/app.py#L497-L503

  Should live on the `ConnectedDatabase` class instead.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/487/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDg2NjgyMDQ=
title: Move detect_primary_keys to Database class method
state: closed
created_at: 2019-05-27T05:11:51Z
updated_at: 2019-05-27T20:05:34Z
closed_at: 2019-05-27T18:29:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g.

  https://github.com/simonw/datasette/blob/026c84db30bd0a75ecde146a80a5d142078dc299/datasette/views/table.py#L73-L75

  Should be

  ```
      pks = await db.primary_keys(table)
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/488/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDg5Nzc0NDQ=
title: Pagination breaks when combined with expanded foreign keys
state: closed
created_at: 2019-05-27T19:56:56Z
updated_at: 2019-05-28T02:48:57Z
closed_at: 2019-05-28T02:23:27Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Consider https://edb3662.datasette.io/fixtures/roadside_attraction_characteristics?_sort=attraction_id&_size=2

  The "Next page" link goes here, which returns 0 rows:

  https://edb3662.datasette.io/fixtures/roadside_attraction_characteristics?_size=2&_next=%257B%2527value%2527%253A%2B2%252C%2B%2527label%2527%253A%2B%2527Winchester%2BMystery%2BHouse%2527%257D%2C2&_sort=attraction_id

  That's because if you double-url-decode that `_next` link you get this:

  `_next={'value': 2, 'label': 'Winchester Mystery House'},2`

  It should be `_next=2,2`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/489/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDg5Nzg5MDc=
title: Rename InterruptedError exception class
state: closed
created_at: 2019-05-27T20:04:25Z
updated_at: 2019-05-28T00:16:45Z
closed_at: 2019-05-28T00:16:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://github.com/simonw/datasette/blob/edb36629e7356f70f42b9d37fea5dfe9cc3c364a/datasette/utils.py#L49-L50

  Python has a built-in exception called this, so we should call ours something else.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/490/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDk0NDU3MTU=
title: Figure out how to use Firebase with cloudrun to enable vanity URLs and CDN caching
state: open
created_at: 2019-05-28T19:48:06Z
updated_at: 2019-05-28T19:48:35Z
author_association: OWNER
type: issue
body:
  It looks like Firebase can solve a couple of problems with the existing `datasette publish cloudrun` hosting mechanism:

  * The URLs it produces aren't pretty enough. Firebase offers more control over vanity URLs.
  * CDN caching (as seen in `datasette publish now`) is great for improving performance and saving money on Cloud Run execution time.

  https://firebase.google.com/docs/hosting/cloud-run looks like it can help with both of these.

  Lots of interesting questions:

  * Should this be a new `datasette publish firebase` command or should it instead be implemented as additional custom options to `datasette publish cloudrun`?
  * How much harder does it become to do account setup?
  * How much will this option cost users?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/491/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDk1NjUyMDQ=
title: Syntactic sugar for creating m2m records
state: closed
created_at: 2019-05-29T02:17:48Z
updated_at: 2019-08-04T03:54:58Z
closed_at: 2019-08-04T03:37:34Z
author_association: OWNER
body: Python library only. What would be a syntactically pleasant way of creating a m2m record?
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDk4MTg4OTc=
title: Additional Column Constraints?
state: closed
created_at: 2019-05-29T13:47:03Z
updated_at: 2019-06-13T06:47:17Z
closed_at: 2019-06-13T06:30:26Z
author_association: NONE
type: issue
state_reason: completed
body:
  I'm looking to import data from XML with a pre-defined schema that maps fairly closely to a relational database.
  In particular, it has explicit annotations for when fields are required, optional, or when a default value should be inferred.

  Would there be value in adding the ability to define `NOT NULL` and `DEFAULT` column constraints to sqlite-utils?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDk4NDg4MDM=
title: Allow .insert(..., foreign_keys=()) to auto-detect table and primary key
state: closed
created_at: 2019-05-29T14:39:22Z
updated_at: 2019-06-13T05:32:32Z
closed_at: 2019-06-13T05:32:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The `foreign_keys=` argument currently takes a list of triples:
  ```python
  db["usages"].insert_all(
      usages_to_insert,
      foreign_keys=(
          ("line_id", "lines", "id"),
          ("definition_id", "definitions", "id"),
      ),
  )
  ```
  As of #16 we have a mechanism for detecting the primary key column (the third item in this triple) - we should use that here too, so foreign keys can be optionally defined as a list of pairs.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDk4NTQ2MDQ=
title: Facets not correctly persisted in hidden form fields
state: closed
created_at: 2019-05-29T14:49:39Z
updated_at: 2020-09-15T20:12:29Z
closed_at: 2020-09-15T20:12:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Steps to reproduce: visit https://2a4b892.datasette.io/fixtures/roadside_attractions?_facet_m2m=attraction_characteristic and click "Apply"

  Result is a 500: `no such column: attraction_characteristic`

  The error occurs because of this hidden HTML input:

      <input type="hidden" name="_facet" value="attraction_characteristic">

  This should be:

      <input type="hidden" name="_facet_m2m" value="attraction_characteristic">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/492/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDk4ODYzMTk=
title: Rename metadata.json to config.json
state: closed
created_at: 2019-05-29T15:48:03Z
updated_at: 2023-08-23T01:29:21Z
closed_at: 2023-08-23T01:29:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It is increasingly being useful configuration options, when it started out as purely metadata.

  Could cause confusion with the `--config` mechanism though - maybe that should be called "settings" instead?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/493/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NDk5MzE4OTk=
title: --reload should only trigger for -i databases
state: closed
created_at: 2019-05-29T17:28:43Z
updated_at: 2020-02-24T19:45:05Z
closed_at: 2020-02-24T19:45:05Z
author_association: OWNER
body: Right now it's triggering any time a mutable database changes.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/494/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTAwMzIxMzQ=
title: facet_m2m gets confused by multiple relationships
state: open
created_at: 2019-05-29T21:37:28Z
updated_at: 2020-12-17T05:08:22Z
author_association: OWNER
type: issue
body:
  I got this for a database I was playing with:

  <img width="794" alt="hackathon__hacks_project__24_rows" src="https://user-images.githubusercontent.com/9599/58593179-12a65b00-821f-11e9-8621-ec0a62d41b46.png">

  I think this is because of these three tables:

  <img width="733" alt="hackathon__select___from_sqlite_master_" src="https://user-images.githubusercontent.com/9599/58593271-471a1700-821f-11e9-8d23-b27bfc3dec73.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/495/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTA4NjI1Nzc=
title: Additional options to gcloud build command in cloudrun - timeout
state: closed
created_at: 2019-05-31T15:43:55Z
updated_at: 2019-05-31T23:05:05Z
closed_at: 2019-05-31T23:05:05Z
author_association: NONE
type: issue
state_reason: completed
body:
  I am trying to deploy a 3.1 GB dataset to cloudrun with datasette. Currrently the docker build times out. Would be nice to have a timeout flag or additional gcloud commands that could be specified. 

  Here is the line https://github.com/simonw/datasette/blob/f825e2012109247fa246e2b938f8174069e574f1/datasette/publish/cloudrun.py#L78

  I would be happy to submit a PR to allow for a timeout option. What are your ideas of allowing the user additional build publishing flag options?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/496/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjg0MzkwMTk3
title: Upgrade pytest to 4.6.1
state: closed
created_at: 2019-06-03T01:45:34Z
updated_at: 2019-06-03T02:06:32Z
closed_at: 2019-06-03T02:06:27Z
author_association: OWNER
pull_request: simonw/datasette/pulls/497
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/497/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTE1MTM1NDE=
title: Full text search of all tables at once?
state: closed
created_at: 2019-06-03T14:24:43Z
updated_at: 2020-05-30T17:26:02Z
closed_at: 2020-05-30T17:26:02Z
author_association: NONE
type: issue
state_reason: completed
body:
  Does datasette have a built-in way, in a browser, to do a full-text search of all columns, in all databases and tables, that have full-text search enabled? Is there a plugin that does this?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/498/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTE1ODU3NjQ=
title: Accessibility for non-techie newsies?
state: open
created_at: 2019-06-03T16:49:37Z
updated_at: 2019-06-05T21:22:55Z
author_association: NONE
type: issue
body:
  Hi again, I'm having fun uploading datasets to Heroku via datasette. I'd like to set up datasette so that it's easy for other newsroom workers, who don't use Linux and aren't programmers, to upload datasets. Does datsette provide this out-of-the-box, or as a plugin? 

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/499/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjg0NzQzNzk0
title: Fix typo in install step: should be install -e
state: closed
created_at: 2019-06-03T21:50:51Z
updated_at: 2019-06-11T18:48:43Z
closed_at: 2019-06-11T18:48:40Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/500
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/500/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjg1Njk4MzEw
title: Test against Python 3.8-dev using Travis
state: closed
created_at: 2019-06-06T08:37:53Z
updated_at: 2019-11-11T03:23:29Z
closed_at: 2019-11-11T03:23:29Z
author_association: OWNER
pull_request: simonw/datasette/pulls/501
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/501/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTMxMzE5MTc=
title: Exporting sqlite database(s)?
state: closed
created_at: 2019-06-06T16:39:53Z
updated_at: 2021-04-03T05:16:54Z
closed_at: 2019-06-11T18:50:42Z
author_association: NONE
type: issue
state_reason: completed
body:
  I'm working on datasette from one computer. But if I want to work on it from another computer and want to copy the SQLite database(s) already on the Heroku datasette instance, how to I copy the database(s) to the second computer so that I can then update it and push to online via datasette's command line code that pushes code to Heroku?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/502/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTMyNDM0NTk=
title: Handle SQLite databases with spaces in their names?
state: closed
created_at: 2019-06-06T21:20:59Z
updated_at: 2019-11-04T23:16:30Z
closed_at: 2019-11-04T23:16:30Z
author_association: NONE
type: issue
state_reason: completed
body:
  I named my SQLite database "Government workers" and published it to Heroku. When I clicked the "Government workers" database online it lead to a 404 page: `Database not found: Government%20workers`.

  I believe this is because the database name has a space.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/503/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTM2MzkxOTY=
title: Remove TableView ?_group_count= feature
state: closed
created_at: 2019-06-07T18:25:18Z
updated_at: 2019-11-06T05:13:10Z
closed_at: 2019-11-06T05:13:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This feature really doesn't warrant continuing to exist. For reference: #150 and #44

  Don't forget to remove it from the docs: https://github.com/simonw/datasette/blob/172da009d890aa029cff7138b4dcfd4f60948525/docs/json_api.rst#L322-L324

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/504/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTM4Mjk5MTA=
title: Add white-space: pre-wrap to SQL create statement
state: closed
created_at: 2019-06-08T19:59:56Z
updated_at: 2019-07-07T20:26:55Z
closed_at: 2019-07-07T20:26:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now a super-long CREATE TABLE statement causes the table page to be even wider than the table itself:

  <img width="1067" alt="many-photos-tables__RKMaster__1_row_where_where_modelId___18479_and_New_Issue_·_simonw_datasette_and_many-photos-tables__RKPlace__1_274_rows" src="https://user-images.githubusercontent.com/9599/59151726-27d47400-89ed-11e9-9ff7-76c129127329.png">

  Adding `white-space: pre-wrap` to that `<pre>` element is an easy fix:

  <img width="1070" alt="many-photos-tables__RKMaster__28_260_rows_and_New_Issue_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/59151732-476b9c80-89ed-11e9-8c1b-c9af98bad5b4.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/505/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTM4NDYyMTc=
title: Option to display binary data
state: closed
created_at: 2019-06-08T23:44:12Z
updated_at: 2019-06-11T15:48:27Z
closed_at: 2019-06-09T16:07:39Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In #442 we suppressed rendering of binary data:

  <img width="358" alt="many-photos-tables__RKAlbumVersion_albumId_RidIndex__36_rows" src="https://user-images.githubusercontent.com/9599/59153367-865d1a80-8a0c-11e9-9384-27dcba0b61b6.png">

  It turns out there is one use-case where displaying binary data is useful: when you're poking around looking at random SQLite databases you find in `~/Library` trying to figure out what they are for.

  So, a mechanism for opting in to ugly display of binary data again would be useful.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/506/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTU0ODYyODY=
title: Mechanism for turning nested JSON into foreign keys / many-to-many
state: open
created_at: 2019-06-13T00:52:06Z
updated_at: 2022-06-29T23:35:29Z
author_association: OWNER
type: issue
body:
  The GitHub JSON APIs have a really interesting convention with respect to related objects.

  Consider https://api.github.com/repos/simonw/sqlite-utils/issues - here's a truncated subset:
  ```json
    {
      "id": 449818897,
      "node_id": "MDU6SXNzdWU0NDk4MTg4OTc=",
      "number": 24,
      "title": "Additional Column Constraints?",
      "user": {
        "login": "IgnoredAmbience",
        "id": 98555,
        "node_id": "MDQ6VXNlcjk4NTU1",
        "avatar_url": "https://avatars0.githubusercontent.com/u/98555?v=4",
        "gravatar_id": ""
      },
      "labels": [
        {
          "id": 993377884,
          "node_id": "MDU6TGFiZWw5OTMzNzc4ODQ=",
          "url": "https://api.github.com/repos/simonw/sqlite-utils/labels/enhancement",
          "name": "enhancement",
          "color": "a2eeef",
          "default": true
        }
      ],
      "state": "open"
    }
  ```
  The `user` column lists a complete user. The `labels` column has a list of labels.

  Since both user and label have populated `id` field this is actually enough information for us to create records for them AND set up the corresponding foreign key (for user) and m2m relationships (for labels).

  It would be really neat if `sqlite-utils` had some kind of mechanism for correctly processing these kind of patterns.

  Thanks to `jq` there's not much need for extra customization of the shape here - if we support a narrowly defined structure users can use `jq` to reshape arbitrary JSON to match.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/26/reactions", "total_count": 4, "+1": 4, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTU0OTY1MDQ=
title: sqlite-utils create-table command
state: closed
created_at: 2019-06-13T01:43:30Z
updated_at: 2020-05-03T15:26:15Z
closed_at: 2020-05-03T15:26:15Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Spun off from #24 - it would be useful if CLI users could create new tables (with explicit column types, not null rules and defaults) without having to insert an example record.

  - [x] Get it working
  - [x] Support `--pk`
  - [x] Support `--not-null`
  - [x] Support `--default`
  - [x] Support `--fk colname othertable othercol`
  - [x] Support `--replace` and `--ignore`
  - [x] Documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTU4NTI4MDE=
title: Every datasette plugin on the ecosystem page should have a screenshot
state: open
created_at: 2019-06-13T17:02:51Z
updated_at: 2020-09-17T02:47:35Z
author_association: OWNER
body: https://github.com/simonw/datasette/blob/master/docs/ecosystem.rst
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/507/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTU5NjUxNzQ=
title: Ability to set default sort order for a table or view in metadata.json
state: closed
created_at: 2019-06-13T21:40:51Z
updated_at: 2020-05-28T18:53:03Z
closed_at: 2020-05-28T18:53:02Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It can go here in the documentation: https://datasette.readthedocs.io/en/stable/metadata.html#setting-which-columns-can-be-used-for-sorting

  Also need to fix this sentence which is no longer true:

  > By default, database views in Datasette do not support sorting

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/508/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTU5OTY4MDk=
title: Rearrange the docs by area, not CLI vs Python
state: closed
created_at: 2019-06-13T23:33:35Z
updated_at: 2019-07-15T02:37:20Z
closed_at: 2019-07-15T02:37:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The docs for eg inserting data should live on the same page, rather than being split across the API and CLI pages.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTY1Njg4ODA=
title: Support opening multiple databases with the same stem
state: closed
created_at: 2019-06-15T19:32:00Z
updated_at: 2020-12-22T20:04:35Z
closed_at: 2020-12-22T20:04:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. I should be able to do this:

      datasette App/data.db Other_App/data.db

  This currently errors because you can't have two databases taking the `/data` URL path.

  Instead, how about in this particular case assigning the second database `/data-1`?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/509/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTY1NjkwNjc=
title: Ability to facet by delimiter (e.g. comma separated fields)
state: open
created_at: 2019-06-15T19:34:41Z
updated_at: 2019-07-08T15:44:51Z
author_association: OWNER
type: issue
body:
  E.g. if a field contains "Tags,With,Commas" be able to facet them in the same way as `_facet_array=` lets you facet `["Tags", "With", "Commas"]`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/510/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTY1Nzg0NzQ=
title: Get Datasette tests passing on Windows in GitHub Actions
state: open
created_at: 2019-06-15T21:41:58Z
updated_at: 2021-07-11T17:23:05Z
author_association: OWNER
type: issue
body:
  This should almost happen as a side-effect or moving from Sanic to Uvicorn during the port to ASGI: #272 

  Additional steps:

  - test it manually
  - update documentation
  - set up some form of Windows CI

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/511/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTcxNDc5MzY=
title: "about" parameter in metadata does not appear when alone
state: open
created_at: 2019-06-17T21:04:20Z
updated_at: 2019-10-11T15:49:13Z
author_association: NONE
type: issue
body:
  Here's an example of metadata I have for one database on datasette.

  ```
  "Records-requests": {
  	"tables": {
  		"Some table": {
  			"about": "This table has data."
  		}
  	}
  }
  ```

  The text in `about` does not show up when I publish the data. But it shows up after I add a `"source"` parameter in the metadata.

  Is this intended?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/512/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTcyMDE5MDc=
title: Is it possible to publish to Heroku despite slug size being too large?
state: closed
created_at: 2019-06-18T00:12:02Z
updated_at: 2019-06-21T22:35:54Z
closed_at: 2019-06-21T22:35:54Z
author_association: NONE
type: issue
state_reason: completed
body:
  I'm trying to push more than 1.5GB worth of SQLite databases -- 535MB compressed -- to Heroku but I get this error when I run the `datasette publish heroku` command.

      Compiled slug size: 535.5M is too large (max is 500M).

  Can I publish the databases and make datasette work on Heroku despite the large slug size?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/513/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTg5NDEyMDM=
title: Prevent accidental add-foreign-key with invalid column
state: closed
created_at: 2019-06-20T23:57:24Z
updated_at: 2019-06-20T23:58:26Z
closed_at: 2019-06-20T23:58:26Z
author_association: OWNER
type: issue
state_reason: completed
body:
  You can corrupt your database by running:

      $ sqlite-utils add-foreign-key my.db table non_existent_column other_table other_column

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/29/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTkzOTc2MjU=
title: Documentation with recommendations on running Datasette in production without using Docker
state: closed
created_at: 2019-06-21T22:48:12Z
updated_at: 2020-10-08T23:55:53Z
closed_at: 2020-10-08T23:33:05Z
author_association: NONE
type: issue
state_reason: completed
body:
  I've got some SQLite databases too big to push to Heroku or the other services with built-in support in datasette. 

  So instead I moved my datasette code and databases to a remote server on Kimsufi. In the folder containing the SQLite databases I run the following code.

  `nohup datasette serve -h 0.0.0.0 *.db --cors --port 8000 --metadata metadata.json > output.log 2>&1 &`.

  When I go to `http://my-remote-server.com:8000`, the site loads. But I know this is not a good long-term solution to running datasette on this server. 

  What is the "correct" way to have this site run, preferably on server port 80?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/514/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk0NjkyNzg=
title: Try shrinking official image with docker-slim
state: open
created_at: 2019-06-22T12:25:37Z
updated_at: 2019-06-22T12:25:37Z
author_association: OWNER
type: issue
body:
  This looks really promising: https://github.com/docker-slim/docker-slim

  If it can shave substantial size from our official container reliably we could add it to the automated build process.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/515/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk1MDkxMjY=
title: Enforce import sort order with isort
state: open
created_at: 2019-06-22T20:35:50Z
updated_at: 2023-08-23T02:15:36Z
author_association: OWNER
type: issue
body:
  I want to use isort to order imports. A few steps here:

  - [x] Add a .isort.cfg file (see below)
  - [x] Use `isort -rc` to reformat existing code
  - [ ] Commit this change
  - [x] Add a unit test that ensures future changes remain isort compatible

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/516/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk1MzcwNDc=
title: Add unit test for "static" mechanism in plugins
state: closed
created_at: 2019-06-23T05:03:31Z
updated_at: 2021-01-04T20:15:19Z
closed_at: 2021-01-04T20:15:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Split out from #272 - this is actually quite tricky. Here's the relevant code:

  https://github.com/simonw/datasette/blob/35429f90894321eda7f2db31b9ea7976f31f73ac/datasette/utils.py#L602-L614

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/517/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkwODk3MTA0
title: Port Datasette from Sanic to ASGI + Uvicorn
state: closed
created_at: 2019-06-23T15:18:42Z
updated_at: 2019-06-24T13:42:50Z
closed_at: 2019-06-24T03:13:09Z
author_association: OWNER
pull_request: simonw/datasette/pulls/518
type: pull
body:
  Most of the code here was fleshed out in comments on #272 (Port Datasette to ASGI) - this pull request will track the final pieces:

  - [x] Update test harness to more correctly simulate the `raw_path` issue
  - [x] Use `raw_path` so table names containing `/` can work correctly
  - [x] Bug: JSON not served with correct content-type
  - [x] Get ?_trace=1 working again
  - [x] Replacement for `@app.listener("before_server_start")`
  - [x] Bug: `/fixtures/table%2Fwith%2Fslashes.csv?_format=json` downloads as CSV
  - [x] Replace Sanic request and response objects with my own classes, so I can remove Sanic dependency
  - [x] Final code tidy-up before merging to master

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/518/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk1OTAwMjE=
title: Decide what goes into Datasette 1.0
state: closed
created_at: 2019-06-23T15:47:41Z
updated_at: 2021-11-15T23:26:11Z
closed_at: 2021-11-15T23:26:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Datasette ASGI #272 is a big part of it... but 1.0 will generally be an indicator that Datasette is a stable platform for developers to write plugins and custom templates against. So lots to think about.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/519/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk1OTgwODA=
title: asgi_wrapper plugin hook
state: closed
created_at: 2019-06-23T17:16:45Z
updated_at: 2019-07-03T04:40:34Z
closed_at: 2019-07-03T04:06:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  After #272 we can finally add this hook. It will allow plugins to wrap their own ASGI middleware around Datasette. Potential use-cases include:

  * adding authentication
  * custom CORS headers (see #454)
  * maybe gzip support?
  * possibly defining entirely new routes, though that may be better handled by a separate hook

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/520/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk2MjE2ODM=
title: Easier way of creating custom row templates
state: closed
created_at: 2019-06-23T21:49:27Z
updated_at: 2019-07-03T03:23:56Z
closed_at: 2019-07-03T03:23:56Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I was messing around with a custom `_rows_and_columns.html` template and ended up with this:
  ```html
  {% for row in display_rows %}
    <div> 
      <hr>
      {% for cell in row %}
        {% if cell.column == "First_Name" %}
          <h2 class="scientist">{{ cell.value }}
        {% elif cell.column == "Last_Name" %}
          {{ cell.value }}</h1>
        {% elif cell.column == "Short_Description" %}
          <p><strong>{{ cell.column }}: </strong>{{ cell.value }}</p>
          <p>
        {% else %}
          <strong>{{ cell.column }}: </strong>{{ cell.value }}&nbsp;&nbsp;&nbsp;
        {% endif %}
      {% endfor %}
    </div>
  {% endfor %}
  ```
  This is nasty. I'd like to be able to do something like this instead:
  ```
  {% for row in display_rows %}
    <h2 class="scientist">{{ row["First_Name"] }} {{ row["Last_Name"] }}</h2>
    ...
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/521/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk2MjIzOTA=
title: Handle case-insensitive headers in a nicer way
state: open
created_at: 2019-06-23T21:56:34Z
updated_at: 2019-06-26T18:48:53Z
author_association: OWNER
body: Spun out from https://github.com/simonw/datasette/pull/518#discussion_r296486289
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/522/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk2Mjc1NDk=
title: Show total/unfiltered row count when filtering
state: closed
created_at: 2019-06-23T22:56:48Z
updated_at: 2019-06-24T01:38:14Z
closed_at: 2019-06-24T01:38:14Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  When I'm seeing a filtered view of a table, I'd like to be able to see something like '2 rows where status != "closed" (of 1000 total)' to have a context for the data I'm seeing – e.g. currently my database is being filled by an importer, so this information would be super helpful.

  Since this information would be a performance hit, maybe something like '12 rows where status != "closed" (of ??? total)' with lazy-loading on-click(?) could be applied (Or via a "How many total?" tooltip, or …)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/523/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkwOTcxMjk1
title: Sort commits using isort, refs #516
state: closed
created_at: 2019-06-24T05:04:48Z
updated_at: 2023-08-23T01:31:08Z
closed_at: 2023-08-23T01:31:08Z
author_association: OWNER
pull_request: simonw/datasette/pulls/524
body: Also added a lint unit test to ensure they stay sorted. #516
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/524/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk3MTQ5NDM=
title: Add section on sqite-utils enable-fts to the search documentation
state: closed
created_at: 2019-06-24T06:39:16Z
updated_at: 2019-06-24T16:36:35Z
closed_at: 2019-06-24T16:29:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://datasette.readthedocs.io/en/stable/full_text_search.html already has a section about csvs-to-sqlite, sqlite-utils is even more relevant.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/525/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk4ODI5MDI=
title: Stream all results for arbitrary SQL and canned queries
state: open
created_at: 2019-06-24T13:09:45Z
updated_at: 2022-09-28T04:01:25Z
author_association: NONE
type: issue
body:
  I think that there is a difficulty with canned queries.

  When I want to stream all results of a canned query TwoDays I get only first 1.000 records.

  Example:
  `http://myserver/history_sample/two_days.csv?_stream=on`

  returns only first 1.000 records.

  If I do the same with the whole database i.e.
  `http://myserver/history_sample/database.csv?_stream=on`

  I get correctly all records.

  Any ideas?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/526/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NTk5MzY1ODU=
title: Unable to use rank when fts-table generated with csvs-to-sqlite
state: closed
created_at: 2019-06-24T14:49:48Z
updated_at: 2019-06-24T15:21:18Z
closed_at: 2019-06-24T15:09:10Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon.

  If i generate a fts-table with the csvs-to-sqlite f-option, I'm unable to use (in datasette's GUI) the internal ranking of the table for sorting or viewing, but if I generate the fts-table with the enable-fts argument from sqlite-utils, everyrthing works ok. Eg.:

  datasette, version 0.28
  sqlite-utils, version 1.2.1
  csvs-to-sqlite, version 0.9

  No column named rank with these commands:
  $ csvs-to-sqlite minutes.csv minutes.db -f text_data
  $ datasette -i minutes.db
  select rank, * from minutes_fts where minutes_fts match 'dog'

  Everything ok with these commands:
  $ csvs-to-sqlite minutes.csv minutes.db
  $ sqlite-utils enable-fts minutes.db text_data
  $ datasette -i minutes.db
  select rank, * from minutes_fts where minutes_fts match 'dog'

  Am I doing something wrong?

  Thank you for a great application!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/527/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjAwOTU5Mjg=
title: Establish a pattern for Datasette plugins built on top of Pandas
state: open
created_at: 2019-06-24T21:05:52Z
updated_at: 2019-06-24T21:05:52Z
author_association: OWNER
type: issue
body:
  The Pandas ecosystem is huge, varied and full of tools that are really good at doing interesting analysis on top of tabular data.

  Pandas should not be a dependency of Datasette core, but I think there is a lot of potential in having plugins which use Pandas to apply interesting analysis to data sucked out of Datasette's SQLite tables.

  One example ([thanks, Tony](https://twitter.com/psychemedia/status/1143259809715752962)): https://github.com/ResidentMario/missingno could form the basis of a fantastic plugin for getting a high-level overview of how complete each column in a table is.

  Some thought is needed here about what shape these kind of plugins might take, and what plugin hooks they would use.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/528/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkxNTM0NTk2
title: Use keyed rows - fixes #521
state: closed
created_at: 2019-06-25T12:33:48Z
updated_at: 2019-06-25T12:35:07Z
closed_at: 2019-06-25T12:35:07Z
author_association: NONE
pull_request: simonw/datasette/pulls/529
type: pull
body:
  Supports template syntax like this:

  ```
  {% for row in display_rows %}
    <h2 class="scientist">{{ row["First_Name"] }} {{ row["Last_Name"] }}</h2>
    ...
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/529/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjA1NDAzMjE=
title: Extract codemirror SQL editor out into a plugin
state: closed
created_at: 2019-06-25T17:07:51Z
updated_at: 2020-10-01T00:42:08Z
closed_at: 2020-10-01T00:42:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now codemirror (used for the SQL editor on https://latest.datasette.io/fixtures?sql=select+*+from+%5B123_starts_with_digits%5D  ) is the only JavaScript in Datasette.

  It's also the only vendored dependency.

  I'd like to move it out to a plugin. But... ideally I would like that plugin to be part of the default "pip install datasette" experience.

  I don't know what the best pattern for optional dependencies is. I don't want to have to tell people to run `pip install datasette[full]`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/530/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjEyMTUxMTg=
title: Option to open database in read-only mode
state: closed
created_at: 2019-06-26T22:50:38Z
updated_at: 2020-05-11T19:17:17Z
closed_at: 2020-05-11T19:17:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Would this make it 100% safe to run reads against a database file that is being written to by another process?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjEyMzc2MTg=
title: Mechanism for adding multiple foreign key constraints at once
state: closed
created_at: 2019-06-27T00:04:30Z
updated_at: 2019-06-29T06:27:40Z
closed_at: 2019-06-29T06:27:40Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Needed by [db-to-sqlite](https://github.com/simonw/db-to-sqlite). It currently works by collecting all of the foreign key relationships it can find and then applying them at the end of the process.

  The problem is, the `add_foreign_key()` method looks like this:

  https://github.com/simonw/sqlite-utils/blob/86bd2bba689e25f09551d611ccfbee1e069e5b66/sqlite_utils/db.py#L498-L516

  That means it's doing a full `VACUUM` for every single relationship it sets up - and if you have hundreds of foreign key relationships in your database this can take hours.

  I think the right solution is to have a `.add_foreign_keys(list_of_args)` method which does the bulk operation and then a single `VACUUM`. `.add_foreign_key(...)` can then call the bulk action with a single list item.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkyODc5MjA0
title: db.add_foreign_keys() method
state: closed
created_at: 2019-06-28T15:40:33Z
updated_at: 2019-06-29T06:27:39Z
closed_at: 2019-06-29T06:27:39Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/32
body:
  Refs #31. Still TODO:

  - [x] Unit tests
  - [x] Documentation

type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjIxMTczMTE=
title: /database/-/inspect
state: open
created_at: 2019-06-28T16:33:41Z
updated_at: 2019-07-08T15:43:57Z
author_association: OWNER
type: issue
body:
  Build `/database/-/inspect` which shows tables, columns, column types and foreign keys

  It won't show table counts. Or maybe it will include them optionally but only for `-i` databases, in a special area of the JSON reserved for immutable-only inspect details.

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/465#issuecomment-506797086_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/531/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjI0MjM4Mzk=
title: index_foreign_keys / index-foreign-keys utilities
state: closed
created_at: 2019-06-30T16:42:03Z
updated_at: 2019-06-30T23:54:11Z
closed_at: 2019-06-30T23:50:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Sometimes it's good to have indices on all columns that are foreign keys, to allow for efficient reverse lookups.

  This would be a useful utility:

      $ sqlite-utils index-foreign-keys database.db

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/33/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkzMTE3MTgz
title: sqlite-utils index-foreign-keys / db.index_foreign_keys()
state: closed
created_at: 2019-06-30T16:43:40Z
updated_at: 2019-06-30T23:50:55Z
closed_at: 2019-06-30T23:50:55Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/34
type: pull
body:
  Refs #33

  - [x] `sqlite-utils index-foreign-keys` command
  - [x] `db.index_foreign_keys()` method
  - [x] unit tests
  - [x] documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/34/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjI0MzA5MjA=
title: table.update(...) method
state: closed
created_at: 2019-06-30T18:06:15Z
updated_at: 2019-07-28T15:43:52Z
closed_at: 2019-07-28T15:43:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Spun off from #23 - this method will allow a user to update a specific row.

  Currently the only way to do that it is to call `.upsert({full record})` with the primary key field matching an existing record - but this does not support partial updates.
  ```python
  db["events"].update(3, {"name": "Renamed"})
  ```
  This method only works on an existing table, so there's no need for a `pk="id"` specifier - it can detect the primary key by looking at the table.

  If the primary key is compound the first argument can be a tuple:
  ```python
  db["events_venues"].update((3, 2), {"custom_label": "Label"})
  ```
  The method can be called without the second dictionary argument. Doing this selects the row specified by the primary key (throwing an error if it does not exist) and remembers it so that chained operations can be carried out - see proposal in https://github.com/simonw/sqlite-utils/issues/23#issuecomment-507055345

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjI4MTc1ODk=
title: Support compound primary keys
state: closed
created_at: 2019-07-01T17:00:07Z
updated_at: 2019-07-15T04:28:52Z
closed_at: 2019-07-15T04:28:52Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This should work:
  ```python
  table = db["dog_breeds"].insert({
      "dog_id": 1,
      "breed_id": 2
  }, pk=("dog_id", "breed_id"))
  ```
  Needed for m2m work in #23

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/36/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjI5MjgwMzg=
title: Switch setup.py to using ~= for dependencies
state: closed
created_at: 2019-07-01T21:53:48Z
updated_at: 2019-07-03T04:32:58Z
closed_at: 2019-07-03T04:32:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  `~=` means "compatible release" https://www.python.org/dev/peps/pep-0440/#compatible-release

  See also https://stackoverflow.com/questions/39590187/in-requirements-txt-what-does-tilde-equals-mean

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/532/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkzOTYyNDA1
title: Support cleaner custom templates for rows and tables, closes #521
state: closed
created_at: 2019-07-03T00:40:18Z
updated_at: 2019-07-03T03:23:06Z
closed_at: 2019-07-03T03:23:06Z
author_association: OWNER
pull_request: simonw/datasette/pulls/533
body:
  - [x] Rename `_rows_and_columns.html` to `_table.html`
  - [x] Unit test
  - [x] Documentation

type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/533/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjM0OTI4MTU=
title: 500 error on m2m facet detection
state: open
created_at: 2019-07-03T00:42:42Z
updated_at: 2020-12-17T05:08:22Z
author_association: OWNER
type: issue
body:
  This may help debug:
  ```
  diff --git a/datasette/facets.py b/datasette/facets.py
  index 76d73e5..07a4034 100644
  --- a/datasette/facets.py
  +++ b/datasette/facets.py
  @@ -499,11 +499,14 @@ class ManyToManyFacet(Facet):
                   "outgoing"
               ]
               if len(other_table_outgoing_foreign_keys) == 2:
  -                destination_table = [
  -                    t
  -                    for t in other_table_outgoing_foreign_keys
  -                    if t["other_table"] != self.table
  -                ][0]["other_table"]
  +                try:
  +                    destination_table = [
  +                        t
  +                        for t in other_table_outgoing_foreign_keys
  +                        if t["other_table"] != self.table
  +                    ][0]["other_table"]
  +                except IndexError:
  +                    import pdb; pdb.pm()
                   # Only suggest if it's not selected already
                   if ("_facet_m2m", destination_table) in args:
                       continue
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/534/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkzOTkyMzgy
title: Added asgi_wrapper plugin hook, closes #520
state: closed
created_at: 2019-07-03T03:58:00Z
updated_at: 2019-07-03T04:06:26Z
closed_at: 2019-07-03T04:06:26Z
author_association: OWNER
pull_request: simonw/datasette/pulls/535
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/535/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MjkzOTk0NDQz
title: Switch to ~= dependencies, closes #532
state: closed
created_at: 2019-07-03T04:12:16Z
updated_at: 2019-07-03T04:32:55Z
closed_at: 2019-07-03T04:32:55Z
author_association: OWNER
pull_request: simonw/datasette/pulls/536
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/536/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjM1NDQyMDY=
title: Populate "endpoint" key in ASGI scope
state: open
created_at: 2019-07-03T04:54:47Z
updated_at: 2019-07-22T06:03:18Z
author_association: OWNER
type: issue
body:
  This is a trick used by Starlette so that other layers of ASGI middleware can see which route was selected.

  They added it here: https://github.com/encode/starlette/commit/34d0097feb6f057bd050d5057df5a2f96b97384e

  If Datasette supports it as well we can benefit from it if we integrate this sentry_asgi middleware (probably as a `datasette-sentry` plugin): https://github.com/encode/sentry-asgi/blob/c6a42d44d31f85885b79e4ee898683ecf8104971/sentry_asgi/middleware.py#L34-L35

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/537/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjM5MTU4NjM=
title: Mechanism for secrets in plugin configuration
state: closed
created_at: 2019-07-03T19:23:34Z
updated_at: 2019-07-04T05:47:54Z
closed_at: 2019-07-04T05:47:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See https://github.com/simonw/datasette-auth-github/issues/1

  We need a mechanism where by plugins can tap into "secret" config options without exposing them in the visible metadata.json (where plugin configs currently live, see https://datasette.readthedocs.io/en/stable/plugins.html#plugin-configuration )

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/538/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk0NDAwNDQ2
title: Secret plugin configuration options
state: closed
created_at: 2019-07-04T03:21:20Z
updated_at: 2019-07-04T05:36:45Z
closed_at: 2019-07-04T05:36:45Z
author_association: OWNER
pull_request: simonw/datasette/pulls/539
body: Refs #538
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/539/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjQ0NDk1NzA=
title: Add a universal navigation bar which can be modified by plugins
state: closed
created_at: 2019-07-05T03:50:33Z
updated_at: 2019-07-06T23:13:29Z
closed_at: 2019-07-06T23:11:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Needed by https://github.com/simonw/datasette-auth-github/issues/5

  We already have a navigation breadcrumbs header on some pages, I can extend that to be present on every page and make it easy to modify with custom templates.

  <img width="768" alt="fixtures__facetable" src="https://user-images.githubusercontent.com/9599/60697075-5aaf4200-9e9d-11e9-9009-8f794971f5e5.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/540/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjQ3Nzk4MTA=
title: Plugin hook for adding extra template context variables
state: closed
created_at: 2019-07-05T21:37:05Z
updated_at: 2019-07-06T00:05:59Z
closed_at: 2019-07-06T00:05:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It turns out I need this for https://github.com/simonw/datasette-auth-github/issues/5

  It can be modelled on the `extra_body_script` hook: https://datasette.readthedocs.io/en/stable/plugins.html#extra-body-script-template-database-table-view-name-datasette

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/541/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk0OTkyNTc4
title: extra_template_vars plugin hook
state: closed
created_at: 2019-07-05T22:19:17Z
updated_at: 2019-07-06T00:05:57Z
closed_at: 2019-07-06T00:05:56Z
author_association: OWNER
pull_request: simonw/datasette/pulls/542
body: Refs #541
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/542/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjQ4Njg4NDQ=
title: datasette publish option for setting plugin configuration secrets
state: closed
created_at: 2019-07-06T16:21:23Z
updated_at: 2019-07-08T02:06:34Z
closed_at: 2019-07-08T02:06:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Follow-on from #538 - the `datasette publish` command needs a way of passing secrets which will be made available to plugin configuration but will not be exposed in `/-/metadata.json`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/543/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk1MDY1Nzk2
title: --plugin-secret option
state: closed
created_at: 2019-07-06T22:18:20Z
updated_at: 2019-07-08T02:06:31Z
closed_at: 2019-07-08T02:06:31Z
author_association: OWNER
pull_request: simonw/datasette/pulls/544
type: pull
body:
  Refs #543

  - [x] Zeit Now v1 support
  - [x] Solve escaping of ENV in Dockerfile
  - [x] Heroku support
  - [x] Unit tests
  - [x] Cloud Run support
  - [x] Documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/544/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjQ5MDU4OTQ=
title: Fix header on 404 page
state: closed
created_at: 2019-07-07T01:47:40Z
updated_at: 2019-07-07T20:26:55Z
closed_at: 2019-07-07T20:26:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  <img width="707" alt="Error_404" src="https://user-images.githubusercontent.com/9599/60762932-83127a00-a01e-11e9-853a-6ea6aa5e8f6d.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/545/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk1MTI3MjEz
title: Facet by delimiter
state: open
created_at: 2019-07-07T20:06:05Z
updated_at: 2019-11-18T23:46:01Z
author_association: OWNER
pull_request: simonw/datasette/pulls/546
body: Refs #510
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/546/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjQ5OTAxODQ=
title: Release notes for 0.29
state: closed
created_at: 2019-07-07T20:30:28Z
updated_at: 2019-07-08T03:31:59Z
closed_at: 2019-07-08T03:31:59Z
author_association: OWNER
body: There's a lot of stuff... https://github.com/simonw/datasette/compare/0.28...master
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/547/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjQ5OTQxMDU=
title: Add datasette-cors and datasette-auth-github plugins to Ecosystem page
state: closed
created_at: 2019-07-07T21:14:14Z
updated_at: 2019-07-08T02:02:36Z
closed_at: 2019-07-08T02:02:36Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/548/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjUwMDExODU=
title: Send pull request to the repo that the _table.html template will break
state: closed
created_at: 2019-07-07T22:45:17Z
updated_at: 2019-07-08T03:36:46Z
closed_at: 2019-07-08T03:36:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Bump this to 0.29 https://github.com/simonw/salaries-datasette/blob/master/requirements/base.txt
  And rename https://github.com/simonw/salaries-datasette/blob/master/templates/_rows_and_columns.html to _table.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/549/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjUwMDI5Nzg=
title: Pull m2m faceting out of master so we can ship a release without it
state: closed
created_at: 2019-07-07T23:10:48Z
updated_at: 2019-07-07T23:21:22Z
closed_at: 2019-07-07T23:21:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  After spending some time with #495 I believe I need to make some pretty major changes to how m2m faceting works. I don't want it to block the release of ASGI Datasette so I'm going to revert it back out of master for the moment and merge it back in after the release has gone out.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/550/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjUwMDMwNzA=
title: Ship many-to-many faceting support (and facet-by-delimiter)
state: open
created_at: 2019-07-07T23:11:45Z
updated_at: 2019-07-08T15:45:23Z
author_association: OWNER
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/551/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjUwMTk4ODI=
title: Add --plugin-secret support to "datasette package"
state: open
created_at: 2019-07-08T01:46:47Z
updated_at: 2019-07-08T01:47:30Z
author_association: OWNER
type: issue
body:
  Split out from #544.

  I think I should combine this with #347 (renaming `datasette package` to `datasette publish docker`).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/552/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjUzMjc4NDQ=
title: Potential improvements to facet-by-date
state: open
created_at: 2019-07-08T15:37:53Z
updated_at: 2019-07-08T15:41:55Z
author_association: OWNER
type: issue
body:
  In addition to #483 Tobias had some useful suggestions on Twitter:

  https://twitter.com/rixxtr/status/1148253926476701696
  > I think for date facets, it might be more meaningful to order them by date, rather than by size? Or offer both? I'm *definitely* often interested in size-over-time, so https://data.rixx.de/django_tickets/tickets?_facet_date=created#facet-created … isn't all that helpful!

  Screenshot of that link:

  <img width="1092" alt="django_tickets__tickets__29_846_rows" src="https://user-images.githubusercontent.com/9599/60823090-9f680100-a15b-11e9-84e9-52b9d666e90f.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/553/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk1NzExNTA0
title: Fix static mounts using relative paths and prevent traversal exploits
state: closed
created_at: 2019-07-09T11:32:02Z
updated_at: 2019-07-11T16:29:26Z
closed_at: 2019-07-11T16:13:19Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/554
type: pull
body:
  While debugging why my static mounts using a relative path (`--static mystatic:rel/path/to/dir`) not working, I noticed that the requests fail no matter what, returning 404 errors. 

  The reason is that datasette tries to prevent traversal exploits by checking if the path is relative to its registered directory. This check fails when the mount is a relative directory, because `/abs/dir/file` obviously not under `dir/file`. 

  https://github.com/simonw/datasette/blob/81fa8b6cdc5457b42a224779e5291952314e8d20/datasette/utils/asgi.py#L303-L306

  This also has the consequence of returning any requested file, because when `/abs/dir/../../evil.file` resolves `aiofiles` happily returns it to the client after it resolves the path itself. The solution is to make sure we're checking relativity of paths after they're fully resolved.

  I've implemented the mentioned changes and also updated the tests.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/554/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjU3MzEwNjI=
title: Static mounts with relative paths not working
state: closed
created_at: 2019-07-09T11:38:35Z
updated_at: 2019-07-11T16:13:22Z
closed_at: 2019-07-11T16:13:22Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Datasette fails to serve files from static mounts that are created using relative paths `datasette --static mystatic:rel/path/to/static/dir`.  
  I've explained the problem and the solution in the pull request: https://github.com/simonw/datasette/pull/554

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/555/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk1NzQ4MjY4
title: Add support for running datasette as a module
state: closed
created_at: 2019-07-09T13:13:30Z
updated_at: 2019-07-11T16:07:45Z
closed_at: 2019-07-11T16:07:44Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/556
type: pull
body:
  This PR allows running datasette using `python -m datasette` command in addition to just running the executable.

  This function is quite useful when debugging a plugin in a project because IDEs like PyCharm can easily start a debug session when datasette is run as a module in contrast to trying to attach a debugger to a running process.

  ![image](https://user-images.githubusercontent.com/3243482/60890448-fc4ede80-a263-11e9-8b42-d2a3db8d1a59.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/556/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjU4MTUzNzI=
title: Experiment with type hints
state: closed
created_at: 2019-07-09T14:30:34Z
updated_at: 2021-08-18T21:48:57Z
closed_at: 2021-08-18T21:48:57Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Since it's designed to be used in Jupyter or for rapid prototyping in an IDE (and it's still pretty small) `sqlite-utils` feels like a great candidate for me to finally try out Python type hints.

  https://veekaybee.github.io/2019/07/08/python-type-hints/ is good.

  It suggests the mypy docs for getting started: https://mypy.readthedocs.io/en/latest/existing_code.html plus this tutorial: https://pymbook.readthedocs.io/en/latest/typehinting.html

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/37/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk2NzM1MzIw
title: Get tests running on Windows using Travis CI
state: closed
created_at: 2019-07-11T16:36:57Z
updated_at: 2021-07-10T23:39:48Z
closed_at: 2021-07-10T23:39:48Z
author_association: OWNER
pull_request: simonw/datasette/pulls/557
body: Refs #511
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/557/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NjcyMTgyNzA=
title: Support unicode in url
state: closed
created_at: 2019-07-12T04:43:24Z
updated_at: 2019-07-15T01:29:30Z
closed_at: 2019-07-14T02:49:33Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi, I defined some custom queries in my `metadata.json`. There are Chinese characters in the names of the queries. So the urls are like `http://127.0.0.1:8001/mydb/测试查询`.
  When opening such urls, datasette will throw an exception.
  ```
  Traceback (most recent call last):
    File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/asgi.py", line 100, in __call__
      return await view(new_scope, receive, send)
    File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/asgi.py", line 172, in view
      request, **scope["url_route"]["kwargs"]
    File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/views/base.py", line 267, in get
      request, database, hash, correct_hash_provided, **kwargs
    File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/views/base.py", line 471, in view_get
      for key in self.ds.renderers.keys()
    File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/views/base.py", line 471, in <dictcomp>
      for key in self.ds.renderers.keys()
    File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/__init__.py", line 655, in path_with_format
      path = request.path
    File "/home/zhe/miniconda3/lib/python3.7/site-packages/datasette/utils/asgi.py", line 49, in path
      self.scope.get("raw_path", self.scope["path"].encode("latin-1"))
  UnicodeEncodeError: 'latin-1' codec can't encode characters in position 9-11: ordinal not in range(256)
  ```
  This used to work when datasette was based on sanic.

  Btw, thanks for the great work!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/558/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk3MjQzMDcz
title: Bump to uvicorn 0.8.4
state: closed
created_at: 2019-07-12T22:30:29Z
updated_at: 2019-07-13T22:34:58Z
closed_at: 2019-07-13T22:34:58Z
author_association: OWNER
pull_request: simonw/datasette/pulls/559
type: pull
body:
  https://github.com/encode/uvicorn/commits/0.8.4

  Query strings will now be included in log files: https://github.com/encode/uvicorn/pull/384

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/559/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0Njc3OTA2NDY=
title: CodeMirror fails to load on database page
state: closed
created_at: 2019-07-14T03:31:00Z
updated_at: 2019-09-03T01:03:02Z
closed_at: 2019-07-14T03:38:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It's not loading on https://latest.datasette.io/fixtures

  But it does load on https://latest.datasette.io/fixtures?sql=select+*+from+facetable

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/560/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk3NDEyNDY0
title: table.update() method
state: closed
created_at: 2019-07-14T17:03:49Z
updated_at: 2019-07-28T15:43:51Z
closed_at: 2019-07-28T15:43:51Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/38
type: pull
body:
  Refs #35

  Still to do:

  - [x] Unit tests
  - [x] Switch to using `.get()`
  - [x] Better exceptions, plus unit tests for what happens if pk does not exist
  - [x] Documentation
  - [x] Ensure compound primary keys work properly
  - [x] `alter=True` support

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/38/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0Njc4NjQwNzE=
title: table.get(...) method
state: closed
created_at: 2019-07-14T17:20:51Z
updated_at: 2019-07-15T04:28:53Z
closed_at: 2019-07-15T04:28:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Utility method for fetching a record by its primary key.

  Accepts a single value (for primary key / rowid tables) or a list/tuple of values (for compound primary keys, refs #36).

  Raises a `NotFoundError` if the record cannot be found.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/39/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk3NDU5Nzk3
title: .get() method plus support for compound primary keys
state: closed
created_at: 2019-07-15T03:43:13Z
updated_at: 2019-07-15T04:28:57Z
closed_at: 2019-07-15T04:28:52Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/40
type: pull
body:
  - [x] Tests for the `NotFoundError` exception
  - [x] Documentation for `.get()` method
  - [x] Support `--pk` multiple times to define CLI compound primary keys
  - [x] Documentation for compound primary keys

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/40/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mjk4OTYyNTUx
title: Fix typos
state: closed
created_at: 2019-07-18T15:13:35Z
updated_at: 2019-07-26T10:25:45Z
closed_at: 2019-07-26T10:25:45Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/561
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/561/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzAxMzE1Mzc=
title: sqlite-utils insert --tsv option
state: closed
created_at: 2019-07-19T04:27:21Z
updated_at: 2019-07-19T04:50:47Z
closed_at: 2019-07-19T04:50:47Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now we only support ingesting CSV, but sometimes interesting data is released as TSV.

  https://www.washingtonpost.com/national/2019/07/18/how-download-use-dea-pain-pills-database/ for example.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/41/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzAzNDU5Mjk=
title: table.extract(...) method and "sqlite-utils extract" command
state: closed
created_at: 2019-07-19T14:09:36Z
updated_at: 2020-09-22T23:39:31Z
closed_at: 2020-09-22T23:37:49Z
author_association: OWNER
type: issue
state_reason: completed
body:
  One of my favourite features of [csvs-to-sqlite](https://github.com/simonw/csvs-to-sqlite) is that it can "extract" columns into a separate lookup table - for example:

      csvs-to-sqlite big_csv_file.csv -c country output.db

  This will turn the `country` column in the resulting table into a integer foreign key against a new `country` table. You can see an example of what that looks like here: https://san-francisco.datasettes.com/registered-business-locations-3d50679/Business+Corridor was extracted from https://san-francisco.datasettes.com/registered-business-locations-3d50679/Registered_Business_Locations_-_San_Francisco?Business%20Corridor=1

  I'd like to have the same capability in `sqlite-utils` - but with the ability to run it against an existing SQLite table rather than just against a CSV.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/42/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA1NDI5Mzg=
title: Facet by array shouldn't suggest for arrays that are not arrays-of-strings
state: closed
created_at: 2019-07-19T20:51:29Z
updated_at: 2019-11-01T19:42:10Z
closed_at: 2019-11-01T19:37:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It's triggering for arrays that look like this at the moment:
  ```json
  [
      {
          "type": "HKWorkoutEventTypeSegment",
          "date": "2019-05-21 09:43:50 -0700",
          "duration": "12.2780519704024",
          "durationUnit": "min"
      },
      {
          "type": "HKWorkoutEventTypeSegment",
          "date": "2019-05-21 09:43:50 -0700",
          "duration": "19.467273102204",
          "durationUnit": "min"
      }
  ]
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/562/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA2MzcwNjg=
title: Use XML Analyser to figure out the structure of the export XML
state: closed
created_at: 2019-07-20T05:19:02Z
updated_at: 2019-07-20T05:20:09Z
closed_at: 2019-07-20T05:20:09Z
author_association: MEMBER
body: https://github.com/simonw/xml_analyser
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA2MzcxNTI=
title: Import workouts
state: closed
created_at: 2019-07-20T05:20:21Z
updated_at: 2019-07-20T06:21:41Z
closed_at: 2019-07-20T06:21:41Z
author_association: MEMBER
body: From #1
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA2MzcyMDY=
title: Import ActivitySummary
state: closed
created_at: 2019-07-20T05:21:00Z
updated_at: 2019-07-20T05:58:07Z
closed_at: 2019-07-20T05:58:07Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  From #1

  ```python
    'ActivitySummary': {'attr_counts': {'activeEnergyBurned': 980,
                                       'activeEnergyBurnedGoal': 980,
                                       'activeEnergyBurnedUnit': 980,
                                       'appleExerciseTime': 980,
                                       'appleExerciseTimeGoal': 980,
                                       'appleStandHours': 980,
                                       'appleStandHoursGoal': 980,
                                       'dateComponents': 980},
                       'child_counts': {},
                       'count': 980,
                       'parent_counts': {'HealthData': 980}},
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA2NDA1MDU=
title: Import Records
state: closed
created_at: 2019-07-20T06:11:20Z
updated_at: 2019-07-20T06:21:41Z
closed_at: 2019-07-20T06:21:41Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  From #1:
  ```python
   'Record': {'attr_counts': {'creationDate': 2672233,
                              'device': 2665111,
                              'endDate': 2672233,
                              'sourceName': 2672233,
                              'sourceVersion': 2671779,
                              'startDate': 2672233,
                              'type': 2672233,
                              'unit': 2650012,
                              'value': 2672232},
              'child_counts': {'HeartRateVariabilityMetadataList': 2318,
                               'MetadataEntry': 287974},
              'count': 2672233,
              'parent_counts': {'Correlation': 2, 'HealthData': 2672231}},
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA2OTE2MjI=
title: Add progress bar
state: closed
created_at: 2019-07-20T16:29:07Z
updated_at: 2019-07-22T03:30:13Z
closed_at: 2019-07-22T02:49:22Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Showing a progress bar would be nice, using Click.

  The easiest way to do this would probably be be to hook it up to the length of the compressed content, and update it as this code pushes more XML bytes through the parser:

  https://github.com/dogsheep/healthkit-to-sqlite/blob/d64299765064501f4efdd9a0b21dbdba9ec4287f/healthkit_to_sqlite/utils.py#L6-L10

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA2OTE5OTk=
title: .add_column() doesn't match indentation of initial creation
state: closed
created_at: 2019-07-20T16:33:10Z
updated_at: 2019-07-23T13:09:11Z
closed_at: 2019-07-23T13:09:05Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I spotted a table which was created once and then had columns added to it and the formatted SQL looks like this:

  ```sql
  CREATE TABLE [records] (
     [type] TEXT,
     [sourceName] TEXT,
     [sourceVersion] TEXT,
     [unit] TEXT,
     [creationDate] TEXT,
     [startDate] TEXT,
     [endDate] TEXT,
     [value] TEXT,
     [metadata_Health Mate App Version] TEXT,
     [metadata_Withings User Identifier] TEXT,
     [metadata_Modified Date] TEXT,
     [metadata_Withings Link] TEXT,
     [metadata_HKWasUserEntered] TEXT
  , [device] TEXT, [metadata_HKMetadataKeyHeartRateMotionContext] TEXT, [metadata_HKDeviceManufacturerName] TEXT, [metadata_HKMetadataKeySyncVersion] TEXT, [metadata_HKMetadataKeySyncIdentifier] TEXT, [metadata_HKSwimmingStrokeStyle] TEXT, [metadata_HKVO2MaxTestType] TEXT, [metadata_HKTimeZone] TEXT, [metadata_Average HR] TEXT, [metadata_Recharge] TEXT, [metadata_Lights] TEXT, [metadata_Asleep] TEXT, [metadata_Rating] TEXT, [metadata_Energy Threshold] TEXT, [metadata_Deep Sleep] TEXT, [metadata_Nap] TEXT, [metadata_Edit Slots] TEXT, [metadata_Tags] TEXT, [metadata_Daytime HR] TEXT)
  ```

  It would be nice if the columns that were added later matched the indentation of the initial columns.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/43/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzA4NTY3ODI=
title: Break up records into different tables for each type
state: closed
created_at: 2019-07-22T01:54:59Z
updated_at: 2019-07-22T03:28:55Z
closed_at: 2019-07-22T03:28:50Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I don't think there's much benefit to having all of the different record types stored in the same enormous table. Here's what I get when I use `_facet=type`:

  <img width="358" alt="hello2__records__2_672_233_rows" src="https://user-images.githubusercontent.com/9599/61601118-e2f54d00-abe8-11e9-8bf6-3df2ef969112.png">

  I'm going to try splitting these up into separate tables - so `HKQuantityTypeIdentifierBodyMassIndex` becomes a table called `rBodyMassIndex` - and see if that's nicer to work with.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzEyOTIwNTA=
title: incorrect json url for row-level data?
state: closed
created_at: 2019-07-22T19:59:38Z
updated_at: 2019-10-21T02:03:09Z
closed_at: 2019-10-21T02:03:09Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  While visiting [this example page](https://register-of-members-interests.datasettes.com/regmem-98dc8b7/people/uk.org.publicwhip%2Fperson%2F10001) (linked from Datasette documentation), manually clicking on [the link](https://register-of-members-interests.datasettes.com/regmem-98dc8b7/people/uk.org.publicwhip%2Fperson%2F10001?_format=json) ("This data as .json") to the json data results in an error 500 `data() got an unexpected keyword argument 'as_format'`

  The [JSON page linked to from the documentation](https://register-of-members-interests.datasettes.com/regmem-d22c12c/people/uk.org.publicwhip%2Fperson%2F10001.json) however is correct (the page address ends in `.json` rather than using a query string `?format=json`)

  This particular datasette demo page is now a few versions behind, but I was able to reproduce the issue using v0.29.2 and a downloaded copy of the demo database (and also with the current HEAD).

  Here is a stack trace:

  ```
  Traceback (most recent call last):
    File "/home/romain/miniconda3/envs/dsbug/lib/python3.7/site-packages/datasette/utils/asgi.py", line 101, in __call__
      return await view(new_scope, receive, send)
    File "/home/romain/miniconda3/envs/dsbug/lib/python3.7/site-packages/datasette/utils/asgi.py", line 173, in view
      request, **scope["url_route"]["kwargs"]
    File "/home/romain/miniconda3/envs/dsbug/lib/python3.7/site-packages/datasette/views/base.py", line 267, in get
      request, database, hash, correct_hash_provided, **kwargs
    File "/home/romain/miniconda3/envs/dsbug/lib/python3.7/site-packages/datasette/views/base.py", line 399, in view_get
      request, database, hash, **kwargs
  TypeError: data() got an unexpected keyword argument 'as_format'

  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/563/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzE2Mjg0ODM=
title: Utilities for building lookup tables
state: closed
created_at: 2019-07-23T10:59:58Z
updated_at: 2019-07-23T13:07:01Z
closed_at: 2019-07-23T13:07:01Z
author_association: OWNER
type: issue
state_reason: completed
body:
  While building https://github.com/dogsheep/healthkit-to-sqlite I found a need for a neat mechanism for easily building lookup tables - tables where each unique value in a column is replaced by a foreign key to a separate table.

  csvs-to-sqlite currently creates those with its "extract" mechanism - but that's written as custom code against Pandas. I'd like to eventually replace Pandas with sqlite-utils there.

  See also #42 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/44/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzAwMjg2NTM1
title: Implemented table.lookup(...), closes #44
state: closed
created_at: 2019-07-23T13:03:30Z
updated_at: 2019-07-23T13:07:00Z
closed_at: 2019-07-23T13:07:00Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/45
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/45/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzE3ODA0NDM=
title: extracts= option for insert/update/etc
state: closed
created_at: 2019-07-23T15:55:46Z
updated_at: 2020-03-01T16:53:40Z
closed_at: 2019-07-23T17:00:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Relates to #42 and #44. I want the ability to extract values out into lookup tables during bulk insert/upsert operations.

  `db.insert_all(rows, extracts=["species"])`

  - creates species table for values in the species column

  `db.insert_all(rows, extracts={"species": "Species"})`

  - as above but the new table is called `Species`.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/46/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzAwMzc3NTk5
title: extracts= table parameter
state: closed
created_at: 2019-07-23T16:30:29Z
updated_at: 2019-07-23T17:00:43Z
closed_at: 2019-07-23T17:00:43Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/47
body: Still needs docs. Refs #46
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/47/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzE4MTg5Mzk=
title: Jupyter notebook demo of the library, launchable on Binder
state: closed
created_at: 2019-07-23T17:05:05Z
updated_at: 2022-01-26T02:08:46Z
closed_at: 2022-01-26T02:08:39Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/48/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzIwOTcyMjA=
title: Script uses a lot of RAM
state: closed
created_at: 2019-07-24T06:11:11Z
updated_at: 2019-07-24T06:35:52Z
closed_at: 2019-07-24T06:35:52Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I'm using an XML pull parser which should avoid the need to slurp the whole XML file into memory, but it's not working - the script still uses over 1GB of RAM when it runs according to Activity Monitor.

  I think this is because I'm still causing the full root element to be incrementally loaded into memory just in case I try and access it later.

  http://effbot.org/elementtree/iterparse.htm says I should use `elem.clear()` as I go. It also says:

  > The above pattern has one drawback; it does not clear the root element, so you will end up with a single element with lots of empty child elements. If your files are huge, rather than just large, this might be a problem. To work around this, you need to get your hands on the root element.

  So I will try that recipe and see if it helps.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzAwNTgwMjIx
title: Use less RAM
state: closed
created_at: 2019-07-24T06:35:01Z
updated_at: 2019-07-24T06:35:52Z
closed_at: 2019-07-24T06:35:52Z
author_association: MEMBER
pull_request: dogsheep/healthkit-to-sqlite/pulls/8
body: Closes #7
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzIxMTUzODE=
title: extracts= should support multiple-column extracts
state: open
created_at: 2019-07-24T07:06:41Z
updated_at: 2020-10-16T19:18:19Z
author_association: OWNER
type: issue
body:
  Lookup tables can be constructed on compound columns, but the `extracts=` option doesn't currently support that.

  Right now extracts can be defined in two ways:
  ```python
  # Extract these columns into tables with the same name:
  dogs = db.table("dogs", extracts=["breed", "most_recent_trophy"])

  # Same as above but with custom table names:
  dogs = db.table("dogs", extracts={"breed": "Breeds", "most_recent_trophy": "Trophies"})
  ```
  Need some kind of syntax for much more complicated extractions, like when two columns (say "source" and "source_version") are extracted into a single table.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/49/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzI0MjkwNDg=
title: Too many SQL variables
state: closed
created_at: 2019-07-24T18:24:17Z
updated_at: 2019-07-26T10:01:05Z
closed_at: 2019-07-26T10:01:05Z
author_association: NONE
type: issue
state_reason: completed
body:
  Decided to try importing my data, and ran into this:

  ```
  Traceback (most recent call last):
    File "/Users/tholo/Source/health/bin/healthkit-to-sqlite", line 10, in <module>
      sys.exit(cli())
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py", line 50, in cli
      convert_xml_to_sqlite(fp, db, progress_callback=bar.update)
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 41, in convert_xml_to_sqlite
      write_records(records, db)
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 80, in write_records
      column_order=["startDate", "endDate", "value", "unit"],
    File "/Users/tholo/Source/health/lib/python3.7/site-packages/sqlite_utils/db.py", line 911, in insert_all
      result = self.db.conn.execute(sql, values)
  sqlite3.OperationalError: too many SQL variables
  ```

  Added some debug output in sqlite_utils/db.py, which resulted in:

  ```
                  INSERT INTO [rBodyMassIndex] ([creationDate], [endDate], [metadata_HKWasUserEntered], [metadata_Health Mate App Version], [metadata_Modified Date], [metadata_Withings Link], [metadata_Withings User Identifier], [sourceName], [sourceVersion], [startDate], [unit], [value]) VALUES
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ,
                      (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                  ;
  ```

  with the attached data:

  ```
  ['2019-06-27 22:55:10 -0700', '2011-06-22 21:05:53 -0700', '0', '4.4.2', '2011-06-23 04:05:53 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308801953&type=1', '301293', 'Health Mate', '4040200', '2011-06-22 21:05:53 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-06-23 09:36:27 -0700', '0', '4.4.2', '2011-06-23 16:36:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308846987&type=1', '301293', 'Health Mate', '4040200', '2011-06-23 09:36:27 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-06-23 23:54:07 -0700', '0', '4.4.2', '2011-06-24 06:55:19 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308898447&type=1', '301293', 'Health Mate', '4040200', '2011-06-23 23:54:07 -0700', 'count', '30.679', '2019-06-27 22:55:10 -0700', '2011-06-24 09:13:40 -0700', '0', '4.4.2', '2011-06-24 16:14:35 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1308932020&type=1', '301293', 'Health Mate', '4040200', '2011-06-24 09:13:40 -0700', 'count', '30.3549', '2019-06-27 22:55:10 -0700', '2011-06-25 08:30:08 -0700', '0', '4.4.2', '2011-06-25 15:30:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309015808&type=1', '301293', 'Health Mate', '4040200', '2011-06-25 08:30:08 -0700', 'count', '30.3395', '2019-06-27 22:55:10 -0700', '2011-06-26 07:47:51 -0700', '0', '4.4.2', '2011-06-26 14:48:27 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309099671&type=1', '301293', 'Health Mate', '4040200', '2011-06-26 07:47:51 -0700', 'count', '30.2315', '2019-06-27 22:55:10 -0700', '2011-06-28 08:48:26 -0700', '0', '4.4.2', '2011-06-28 15:49:13 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309276106&type=1', '301293', 'Health Mate', '4040200', '2011-06-28 08:48:26 -0700', 'count', '30.0617', '2019-06-27 22:55:10 -0700', '2011-06-29 09:21:16 -0700', '0', '4.4.2', '2011-06-29 16:21:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309364476&type=1', '301293', 'Health Mate', '4040200', '2011-06-29 09:21:16 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-06-30 08:41:46 -0700', '0', '4.4.2', '2011-06-30 15:42:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309448506&type=1', '301293', 'Health Mate', '4040200', '2011-06-30 08:41:46 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2011-07-01 09:05:28 -0700', '0', '4.4.2', '2011-07-01 16:06:24 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309536328&type=1', '301293', 'Health Mate', '4040200', '2011-07-01 09:05:28 -0700', 'count', '29.8611', '2019-06-27 22:55:10 -0700', '2011-07-02 08:58:50 -0700', '0', '4.4.2', '2011-07-02 15:59:40 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309622330&type=1', '301293', 'Health Mate', '4040200', '2011-07-02 08:58:50 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2011-07-04 09:33:43 -0700', '0', '4.4.2', '2011-07-04 16:34:19 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309797223&type=1', '301293', 'Health Mate', '4040200', '2011-07-04 09:33:43 -0700', 'count', '30.0309', '2019-06-27 22:55:10 -0700', '2011-07-06 09:40:23 -0700', '0', '4.4.2', '2011-07-06 16:41:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1309970423&type=1', '301293', 'Health Mate', '4040200', '2011-07-06 09:40:23 -0700', 'count', '30.1852', '2019-06-27 22:55:10 -0700', '2011-07-08 08:08:48 -0700', '0', '4.4.2', '2011-07-08 15:09:51 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310137728&type=1', '301293', 'Health Mate', '4040200', '2011-07-08 08:08:48 -0700', 'count', '30.0309', '2019-06-27 22:55:10 -0700', '2011-07-09 08:31:05 -0700', '0', '4.4.2', '2011-07-09 15:31:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310225465&type=1', '301293', 'Health Mate', '4040200', '2011-07-09 08:31:05 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-07-10 08:14:36 -0700', '0', '4.4.2', '2011-07-10 15:15:12 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310310876&type=1', '301293', 'Health Mate', '4040200', '2011-07-10 08:14:36 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2011-07-12 07:55:21 -0700', '0', '4.4.2', '2011-07-12 14:55:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310482521&type=1', '301293', 'Health Mate', '4040200', '2011-07-12 07:55:21 -0700', 'count', '30.108', '2019-06-27 22:55:10 -0700', '2011-07-13 08:48:05 -0700', '0', '4.4.2', '2011-07-13 15:48:42 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310572085&type=1', '301293', 'Health Mate', '4040200', '2011-07-13 08:48:05 -0700', 'count', '30', '2019-06-27 22:55:10 -0700', '2011-07-14 09:05:16 -0700', '0', '4.4.2', '2011-07-14 16:05:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310659516&type=1', '301293', 'Health Mate', '4040200', '2011-07-14 09:05:16 -0700', 'count', '29.9074', '2019-06-27 22:55:10 -0700', '2011-07-15 07:09:56 -0700', '0', '4.4.2', '2011-07-15 14:10:35 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310738996&type=1', '301293', 'Health Mate', '4040200', '2011-07-15 07:09:56 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-07-16 09:26:04 -0700', '0', '4.4.2', '2011-07-16 16:26:44 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310833564&type=1', '301293', 'Health Mate', '4040200', '2011-07-16 09:26:04 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-07-17 09:52:59 -0700', '0', '4.4.2', '2011-07-17 16:53:38 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1310921579&type=1', '301293', 'Health Mate', '4040200', '2011-07-17 09:52:59 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2011-07-19 08:56:16 -0700', '0', '4.4.2', '2011-07-19 15:57:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311090976&type=1', '301293', 'Health Mate', '4040200', '2011-07-19 08:56:16 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-07-21 08:21:20 -0700', '0', '4.4.2', '2011-07-21 15:22:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311261680&type=1', '301293', 'Health Mate', '4040200', '2011-07-21 08:21:20 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-07-23 08:49:56 -0700', '0', '4.4.2', '2011-07-23 15:50:40 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311436196&type=1', '301293', 'Health Mate', '4040200', '2011-07-23 08:49:56 -0700', 'count', '29.7222', '2019-06-27 22:55:10 -0700', '2011-07-24 09:17:35 -0700', '0', '4.4.2', '2011-07-24 16:18:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311524255&type=1', '301293', 'Health Mate', '4040200', '2011-07-24 09:17:35 -0700', 'count', '29.5833', '2019-06-27 22:55:10 -0700', '2011-07-25 07:51:55 -0700', '0', '4.4.2', '2011-07-25 14:52:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1311605515&type=1', '301293', 'Health Mate', '4040200', '2011-07-25 07:51:55 -0700', 'count', '29.5525', '2019-06-27 22:55:10 -0700', '2011-08-06 10:04:05 -0700', '0', '4.4.2', '2011-08-06 17:04:47 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312650245&type=1', '301293', 'Health Mate', '4040200', '2011-08-06 10:04:05 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-08-08 07:52:22 -0700', '0', '4.4.2', '2011-08-08 14:53:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312815142&type=1', '301293', 'Health Mate', '4040200', '2011-08-08 07:52:22 -0700', 'count', '29.6605', '2019-06-27 22:55:10 -0700', '2011-08-10 07:57:30 -0700', '0', '4.4.2', '2011-08-10 14:58:12 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1312988250&type=1', '301293', 'Health Mate', '4040200', '2011-08-10 07:57:30 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-08-12 07:51:14 -0700', '0', '4.4.2', '2011-08-12 14:51:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313160674&type=1', '301293', 'Health Mate', '4040200', '2011-08-12 07:51:14 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2011-08-13 07:45:28 -0700', '0', '4.4.2', '2011-08-13 14:46:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313246728&type=1', '301293', 'Health Mate', '4040200', '2011-08-13 07:45:28 -0700', 'count', '29.5833', '2019-06-27 22:55:10 -0700', '2011-08-17 09:06:20 -0700', '0', '4.4.2', '2011-08-17 16:07:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1313597180&type=1', '301293', 'Health Mate', '4040200', '2011-08-17 09:06:20 -0700', 'count', '29.5679', '2019-06-27 22:55:10 -0700', '2011-08-22 08:28:08 -0700', '0', '4.4.2', '2011-08-22 15:28:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314026888&type=1', '301293', 'Health Mate', '4040200', '2011-08-22 08:28:08 -0700', 'count', '29.9846', '2019-06-27 22:55:10 -0700', '2011-08-25 08:59:30 -0700', '0', '4.4.2', '2011-08-25 16:00:15 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314287970&type=1', '301293', 'Health Mate', '4040200', '2011-08-25 08:59:30 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-08-30 08:13:59 -0700', '0', '4.4.2', '2011-08-30 15:46:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1314717239&type=1', '301293', 'Health Mate', '4040200', '2011-08-30 08:13:59 -0700', 'count', '29.784', '2019-06-27 22:55:10 -0700', '2011-09-12 08:47:51 -0700', '0', '4.4.2', '2011-09-12 15:48:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1315842471&type=1', '301293', 'Health Mate', '4040200', '2011-09-12 08:47:51 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-09-13 09:17:27 -0700', '0', '4.4.2', '2011-09-13 16:48:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1315930647&type=1', '301293', 'Health Mate', '4040200', '2011-09-13 09:17:27 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-10-01 09:12:20 -0700', '0', '4.4.2', '2011-10-01 16:13:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1317485540&type=1', '301293', 'Health Mate', '4040200', '2011-10-01 09:12:20 -0700', 'count', '29.8148', '2019-06-27 22:55:10 -0700', '2011-10-11 11:14:11 -0700', '0', '4.4.2', '2011-10-11 18:15:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1318356851&type=1', '301293', 'Health Mate', '4040200', '2011-10-11 11:14:11 -0700', 'count', '29.7377', '2019-06-27 22:55:10 -0700', '2011-10-16 09:29:47 -0700', '0', '4.4.2', '2011-10-16 16:30:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1318782587&type=1', '301293', 'Health Mate', '4040200', '2011-10-16 09:29:47 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2011-10-19 09:21:44 -0700', '0', '4.4.2', '2011-10-19 16:22:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1319041304&type=1', '301293', 'Health Mate', '4040200', '2011-10-19 09:21:44 -0700', 'count', '29.7685', '2019-06-27 22:55:10 -0700', '2011-10-24 07:04:22 -0700', '0', '4.4.2', '2011-10-24 14:05:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1319465062&type=1', '301293', 'Health Mate', '4040200', '2011-10-24 07:04:22 -0700', 'count', '29.5988', '2019-06-27 22:55:10 -0700', '2011-11-07 09:33:17 -0700', '0', '4.4.2', '2011-11-07 16:33:58 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1320683597&type=1', '301293', 'Health Mate', '4040200', '2011-11-07 09:33:17 -0700', 'count', '29.8611', '2019-06-27 22:55:10 -0700', '2011-11-10 07:59:03 -0700', '0', '4.4.2', '2011-11-10 14:59:48 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1320937143&type=1', '301293', 'Health Mate', '4040200', '2011-11-10 07:59:03 -0700', 'count', '29.9383', '2019-06-27 22:55:10 -0700', '2011-11-13 09:28:31 -0700', '0', '4.4.2', '2011-11-13 16:29:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1321201711&type=1', '301293', 'Health Mate', '4040200', '2011-11-13 09:28:31 -0700', 'count', '29.7531', '2019-06-27 22:55:10 -0700', '2011-11-21 08:45:06 -0700', '0', '4.4.2', '2011-11-21 15:46:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1321890306&type=1', '301293', 'Health Mate', '4040200', '2011-11-21 08:45:06 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-11-23 09:55:44 -0700', '0', '4.4.2', '2011-11-23 16:56:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322067344&type=1', '301293', 'Health Mate', '4040200', '2011-11-23 09:55:44 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2011-11-29 09:50:44 -0700', '0', '4.4.2', '2011-11-29 16:51:31 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322585444&type=1', '301293', 'Health Mate', '4040200', '2011-11-29 09:50:44 -0700', 'count', '30.1698', '2019-06-27 22:55:10 -0700', '2011-11-30 11:13:21 -0700', '0', '4.4.2', '2011-11-30 18:14:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1322676801&type=1', '301293', 'Health Mate', '4040200', '2011-11-30 11:13:21 -0700', 'count', '30.0617', '2019-06-27 22:55:10 -0700', '2011-12-04 10:24:36 -0700', '0', '4.4.2', '2011-12-04 17:25:24 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1323019476&type=1', '301293', 'Health Mate', '4040200', '2011-12-04 10:24:36 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2011-12-10 09:22:18 -0700', '0', '4.4.2', '2011-12-10 16:23:07 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1323534138&type=1', '301293', 'Health Mate', '4040200', '2011-12-10 09:22:18 -0700', 'count', '29.9537', '2019-06-27 22:55:10 -0700', '2011-12-26 10:36:42 -0700', '0', '4.4.2', '2011-12-26 17:37:31 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1324921002&type=1', '301293', 'Health Mate', '4040200', '2011-12-26 10:36:42 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2012-01-11 11:24:13 -0700', '0', '4.4.2', '2012-01-11 18:25:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326306253&type=1', '301293', 'Health Mate', '4040200', '2012-01-11 11:24:13 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2012-01-15 10:17:09 -0700', '0', '4.4.2', '2012-01-15 17:17:51 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326647829&type=1', '301293', 'Health Mate', '4040200', '2012-01-15 10:17:09 -0700', 'count', '29.8302', '2019-06-27 22:55:10 -0700', '2012-01-19 09:24:32 -0700', '0', '4.4.2', '2012-01-19 16:25:21 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1326990272&type=1', '301293', 'Health Mate', '4040200', '2012-01-19 09:24:32 -0700', 'count', '29.7994', '2019-06-27 22:55:10 -0700', '2012-01-29 10:26:13 -0700', '0', '4.4.2', '2012-01-29 17:26:52 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1327857973&type=1', '301293', 'Health Mate', '4040200', '2012-01-29 10:26:13 -0700', 'count', '30.0154', '2019-06-27 22:55:10 -0700', '2012-02-03 10:13:28 -0700', '0', '4.4.2', '2012-02-03 17:15:01 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1328289208&type=1', '301293', 'Health Mate', '4040200', '2012-02-03 10:13:28 -0700', 'count', '29.8457', '2019-06-27 22:55:10 -0700', '2012-02-12 09:23:01 -0700', '0', '4.4.2', '2012-02-12 16:23:53 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1329063781&type=1', '301293', 'Health Mate', '4040200', '2012-02-12 09:23:01 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2012-03-03 09:26:06 -0700', '0', '4.4.2', '2012-03-03 16:26:54 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1330791966&type=1', '301293', 'Health Mate', '4040200', '2012-03-03 09:26:06 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2012-03-11 11:23:15 -0700', '0', '4.4.2', '2012-03-11 18:24:16 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1331490195&type=1', '301293', 'Health Mate', '4040200', '2012-03-11 11:23:15 -0700', 'count', '30.2161', '2019-06-27 22:55:10 -0700', '2012-03-16 09:39:36 -0700', '0', '4.4.2', '2012-03-16 16:40:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1331915976&type=1', '301293', 'Health Mate', '4040200', '2012-03-16 09:39:36 -0700', 'count', '30.2778', '2019-06-27 22:55:10 -0700', '2012-03-21 08:33:07 -0700', '0', '4.4.2', '2012-03-21 15:34:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1332343987&type=1', '301293', 'Health Mate', '4040200', '2012-03-21 08:33:07 -0700', 'count', '30.1389', '2019-06-27 22:55:10 -0700', '2012-04-11 08:49:34 -0700', '0', '4.4.2', '2012-04-11 15:50:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334159374&type=1', '301293', 'Health Mate', '4040200', '2012-04-11 08:49:34 -0700', 'count', '30.0154', '2019-06-27 22:55:10 -0700', '2012-04-13 08:32:06 -0700', '0', '4.4.2', '2012-04-13 15:32:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334331126&type=1', '301293', 'Health Mate', '4040200', '2012-04-13 08:32:06 -0700', 'count', '29.9383', '2019-06-27 22:55:10 -0700', '2012-04-20 08:21:38 -0700', '0', '4.4.2', '2012-04-20 15:52:45 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1334935298&type=1', '301293', 'Health Mate', '4040200', '2012-04-20 08:21:38 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-04-25 09:00:01 -0700', '0', '4.4.2', '2012-04-25 16:00:42 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1335369601&type=1', '301293', 'Health Mate', '4040200', '2012-04-25 09:00:01 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-05-04 11:10:18 -0700', '0', '4.4.2', '2012-05-04 18:10:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1336155018&type=1', '301293', 'Health Mate', '4040200', '2012-05-04 11:10:18 -0700', 'count', '30.4321', '2019-06-27 22:55:10 -0700', '2012-05-12 09:35:00 -0700', '0', '4.4.2', '2012-05-12 16:35:43 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1336840500&type=1', '301293', 'Health Mate', '4040200', '2012-05-12 09:35:00 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2012-05-22 09:27:53 -0700', '0', '4.4.2', '2012-05-22 16:28:37 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1337704073&type=1', '301293', 'Health Mate', '4040200', '2012-05-22 09:27:53 -0700', 'count', '30.4167', '2019-06-27 22:55:10 -0700', '2012-05-31 09:23:16 -0700', '0', '4.4.2', '2012-05-31 16:24:04 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1338481396&type=1', '301293', 'Health Mate', '4040200', '2012-05-31 09:23:16 -0700', 'count', '30.2006', '2019-06-27 22:55:10 -0700', '2012-06-08 09:29:07 -0700', '0', '4.4.2', '2012-06-08 16:29:52 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1339172947&type=1', '301293', 'Health Mate', '4040200', '2012-06-08 09:29:07 -0700', 'count', '30.5247', '2019-06-27 22:55:10 -0700', '2012-06-21 08:07:33 -0700', '0', '4.4.2', '2012-06-21 15:08:20 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1340291253&type=1', '301293', 'Health Mate', '4040200', '2012-06-21 08:07:33 -0700', 'count', '30.5864', '2019-06-27 22:55:10 -0700', '2012-08-08 10:02:22 -0700', '0', '4.4.2', '2012-08-08 17:03:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1344445342&type=1', '301293', 'Health Mate', '4040200', '2012-08-08 10:02:22 -0700', 'count', '30.6636', '2019-06-27 22:55:10 -0700', '2012-08-17 09:11:32 -0700', '0', '4.4.2', '2012-08-17 16:42:05 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1345219892&type=1', '301293', 'Health Mate', '4040200', '2012-08-17 09:11:32 -0700', 'count', '30.8796', '2019-06-27 22:55:10 -0700', '2012-09-10 08:27:21 -0700', '0', '4.4.2', '2012-09-10 15:28:07 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1347290841&type=1', '301293', 'Health Mate', '4040200', '2012-09-10 08:27:21 -0700', 'count', '31.034', '2019-06-27 22:55:10 -0700', '2012-09-17 08:35:33 -0700', '0', '4.4.2', '2012-09-17 15:35:33 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1347896133&type=1', '301293', 'Health Mate', '4040200', '2012-09-17 08:35:33 -0700', 'count', '30.7099', '2019-06-27 22:55:10 -0700', '2012-09-26 08:59:46 -0700', '0', '4.4.2', '2012-09-26 16:13:18 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1348675186&type=1', '301293', 'Health Mate', '4040200', '2012-09-26 08:59:46 -0700', 'count', '30.679', '2019-06-27 22:55:10 -0700', '2012-10-18 08:51:16 -0700', '0', '4.4.2', '2012-10-18 15:51:59 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1350575476&type=1', '301293', 'Health Mate', '4040200', '2012-10-18 08:51:16 -0700', 'count', '30.7716', '2019-06-27 22:55:10 -0700', '2012-11-15 08:54:57 -0700', '0', '4.4.2', '2012-11-15 15:55:58 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1352994897&type=1', '301293', 'Health Mate', '4040200', '2012-11-15 08:54:57 -0700', 'count', '31.0802', '2019-06-27 22:55:10 -0700', '2012-12-17 09:13:40 -0700', '0', '4.4.2', '2012-12-17 16:20:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1355760820&type=1', '301293', 'Health Mate', '4040200', '2012-12-17 09:13:40 -0700', 'count', '29.784', '2019-06-27 22:55:10 -0700', '2012-12-19 11:09:55 -0700', '0', '4.4.2', '2012-12-19 18:10:37 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1355940595&type=1', '301293', 'Health Mate', '4040200', '2012-12-19 11:09:55 -0700', 'count', '29.6914', '2019-06-27 22:55:10 -0700', '2012-12-25 10:37:41 -0700', '0', '4.4.2', '2012-12-25 17:38:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1356457061&type=1', '301293', 'Health Mate', '4040200', '2012-12-25 10:37:41 -0700', 'count', '29.8765', '2019-06-27 22:55:10 -0700', '2013-01-01 10:44:02 -0700', '0', '4.4.2', '2013-01-01 17:44:46 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1357062242&type=1', '301293', 'Health Mate', '4040200', '2013-01-01 10:44:02 -0700', 'count', '30.0772', '2019-06-27 22:55:10 -0700', '2013-01-15 09:10:46 -0700', '0', '4.4.2', '2013-01-15 16:11:28 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1358266246&type=1', '301293', 'Health Mate', '4040200', '2013-01-15 09:10:46 -0700', 'count', '29.9691', '2019-06-27 22:55:10 -0700', '2013-01-20 11:03:39 -0700', '0', '4.4.2', '2013-01-20 18:04:22 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1358705019&type=1', '301293', 'Health Mate', '4040200', '2013-01-20 11:03:39 -0700', 'count', '30.108', '2019-06-27 22:55:10 -0700', '2013-01-30 08:56:30 -0700', '0', '4.4.2', '2013-01-30 15:57:14 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1359561390&type=1', '301293', 'Health Mate', '4040200', '2013-01-30 08:56:30 -0700', 'count', '30.0926', '2019-06-27 22:55:10 -0700', '2013-02-04 11:02:35 -0700', '0', '4.4.2', '2013-02-04 18:03:25 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1360000955&type=1', '301293', 'Health Mate', '4040200', '2013-02-04 11:02:35 -0700', 'count', '29.8148', '2019-06-27 22:55:10 -0700', '2013-02-07 09:07:06 -0700', '0', '4.4.2', '2013-02-07 16:07:49 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1360253226&type=1', '301293', 'Health Mate', '4040200', '2013-02-07 09:07:06 -0700', 'count', '30.1389', '2019-06-27 22:55:10 -0700', '2013-02-19 08:49:57 -0700', '0', '4.4.2', '2013-02-19 15:50:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1361288997&type=1', '301293', 'Health Mate', '4040200', '2013-02-19 08:49:57 -0700', 'count', '30.1235', '2019-06-27 22:55:10 -0700', '2013-03-02 11:20:54 -0700', '0', '4.4.2', '2013-03-02 18:21:38 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1362248454&type=1', '301293', 'Health Mate', '4040200', '2013-03-02 11:20:54 -0700', 'count', '30', '2019-06-27 22:55:10 -0700', '2013-04-23 08:05:30 -0700', '0', '4.4.2', '2013-04-23 15:06:59 +0000', 'withings-bd2://timeline/measure?user                    """
  id=301293&date=1366729530&type=1', '301293', 'Health Mate', '4040200', '2013-04-23 08:05:30 -0700', 'count', '30.5247', '2019-06-27 22:55:10 -0700', '2013-05-09 09:49:18 -0700', '0', '4.4.2', '2013-05-09 16:50:02 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1368118158&type=1', '301293', 'Health Mate', '4040200', '2013-05-09 09:49:18 -0700', 'count', '30.4167', '2019-06-27 22:55:10 -0700', '2013-06-09 09:28:47 -0700', '0', '4.4.2', '2013-06-09 16:29:30 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1370795327&type=1', '301293', 'Health Mate', '4040200', '2013-06-09 09:28:47 -0700', 'count', '30.8333', '2019-06-27 22:55:10 -0700', '2013-07-09 08:00:17 -0700', '0', '4.4.2', '2013-07-09 15:01:00 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1373382017&type=1', '301293', 'Health Mate', '4040200', '2013-07-09 08:00:17 -0700', 'count', '30.8179', '2019-06-27 22:55:10 -0700', '2013-07-28 09:16:55 -0700', '0', '4.4.2', '2013-07-28 16:17:39 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1375028215&type=1', '301293', 'Health Mate', '4040200', '2013-07-28 09:16:55 -0700', 'count', '30.5556', '2019-06-27 22:55:10 -0700', '2013-09-13 09:22:19 -0700', '0', '4.4.2', '2013-09-13 16:23:08 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1379089339&type=1', '301293', 'Health Mate', '4040200', '2013-09-13 09:22:19 -0700', 'count', '30.9568', '2019-06-27 22:55:10 -0700', '2013-09-24 08:08:23 -0700', '0', '4.4.2', '2013-09-24 15:09:03 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1380035303&type=1', '301293', 'Health Mate', '4040200', '2013-09-24 08:08:23 -0700', 'count', '31.4352', '2019-06-27 22:55:10 -0700', '2013-10-01 08:15:13 -0700', '0', '4.4.2', '2013-10-01 15:15:57 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1380640513&type=1', '301293', 'Health Mate', '4040200', '2013-10-01 08:15:13 -0700', 'count', '31.2037', '2019-06-27 22:55:10 -0700', '2013-10-23 09:31:25 -0700', '0', '4.4.2', '2013-10-23 16:32:13 +0000', 'withings-bd2://timeline/measure?userid=301293&date=1382545885&type=1', '301293', 'Health Mate', '4040200', '2013-10-23 09:31:25 -0700', 'count', '31.8056']
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzMwODMyNjA=
title: "Too many SQL variables" on large inserts
state: closed
created_at: 2019-07-25T21:43:31Z
updated_at: 2022-11-04T14:38:36Z
closed_at: 2019-07-28T11:59:33Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Reported here: https://github.com/dogsheep/healthkit-to-sqlite/issues/9

  It looks like there's a default limit of 999 variables - we need to be smart about that, maybe dynamically lower the batch size based on the number of columns.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzAxNDgzNjEz
title: First proof-of-concept of Datasette Library
state: open
created_at: 2019-07-26T10:22:26Z
updated_at: 2023-02-07T15:14:11Z
author_association: OWNER
pull_request: simonw/datasette/pulls/564
type: pull
body:
  Refs #417. Run it like this:

       datasette -d ~/Library

  Uses a new plugin hook - available_databases()

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/564/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzMzMDc3OTQ=
title: Conflict between datasette and uvicorn click versions
state: closed
created_at: 2019-07-26T11:13:40Z
updated_at: 2020-10-02T00:09:55Z
closed_at: 2020-10-02T00:09:55Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hello Datasette is awesome thanks so much!

  I not very familiar with Python but I think there is a problem with datasette docker builds 

  I keep getting this error

  ```
  ERROR: uvicorn 0.8.4 has requirement click==7.*, but you'll have click 6.0 which is incompatible.
  ERROR: datasette 0.29.2 has requirement click~=7.0, but you'll have click 6.0 which is incompatible.
  ```

  The full log from the docker build is here - https://gist.github.com/jonheslop/e01cd322e761cfaf34f0cb83f86411b0

  Just in case it’s helpful this is my setup - https://github.com/dotwatcher/dotwatcher-data

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/565/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzAxODI0MDk3
title: Fix for too many SQL variables, closes #50
state: closed
created_at: 2019-07-28T11:30:30Z
updated_at: 2019-07-28T11:59:32Z
closed_at: 2019-07-28T11:59:32Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/51
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/51/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzY0MTMyOTM=
title: Throws error if .insert_all() / .upsert_all() called with empty list
state: closed
created_at: 2019-08-03T04:09:00Z
updated_at: 2019-11-07T04:32:39Z
closed_at: 2019-11-07T04:32:39Z
author_association: OWNER
body: See also https://github.com/simonw/db-to-sqlite/issues/18
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/52/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzAzOTkwNjgz
title: Work in progress: m2m() method for creating many-to-many records
state: closed
created_at: 2019-08-03T10:03:56Z
updated_at: 2019-08-04T03:38:10Z
closed_at: 2019-08-04T03:37:33Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/53
type: pull
body:
  - [x] `table.insert({"name": "Barry"}).m2m("tags", lookup={"tag": "Coworker"})`
  - [x] Explicit table name `.m2m("humans", ..., m2m_table="relationships")`
  - [x] Automatically use an existing m2m table if a single obvious candidate exists (a table with two foreign keys in the correct directions)
  - [x] Require the explicit `m2m_table=` argument if multiple candidates for the m2m table exist
  - [x] Documentation

  Refs #23

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/53/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzY0MzcyMTM=
title: Unexpected keyword argument 'hidden'
state: closed
created_at: 2019-08-03T10:07:57Z
updated_at: 2019-08-03T16:13:36Z
closed_at: 2019-08-03T16:13:36Z
author_association: NONE
type: issue
state_reason: completed
body:
  I couldn't get a test example running. I am running python 3.6.8 and tried both windows and windows subsystem for linux, getting the same error. My test.db was created by converting a five line csv file with csvs-to-sqlite. The csv file is:
  col1, col2, col3
  1,2,3
  4,5,6
  7,8,9
  10,11,12
  Here is the error message:
  (myvenv) davido@DESKTOP-L29G79U:~/dot/datasette-eg$ datasette test.db                                                   Traceback (most recent call last):                                                                                        File "/home/davido/dot/datasette-eg/myvenv/bin/datasette", line 7, in <module>                                            from datasette.cli import cli                                                                                         File "/home/davido/dot/datasette-eg/myvenv/lib/python3.6/site-packages/datasette/cli.py", line 2, in <module>             import uvicorn                                                                                                        File "/home/davido/dot/datasette-eg/myvenv/lib/python3.6/site-packages/uvicorn/__init__.py", line 2, in <module>          from uvicorn.main import Server, main, run                                                                            File "/home/davido/dot/datasette-eg/myvenv/lib/python3.6/site-packages/uvicorn/main.py", line 224, in <module>            headers: typing.List[str],                                                                                            File "/home/davido/dot/datasette-eg/myvenv/lib/python3.6/site-packages/click/decorators.py", line 170, in decorator       _param_memo(f, OptionClass(param_decls, **attrs))                                                                     File "/home/davido/dot/datasette-eg/myvenv/lib/python3.6/site-packages/click/core.py", line 1430, in __init__             Parameter.__init__(self, param_decls, type=type, **attrs)                                                           TypeError: __init__() got an unexpected keyword argument 'hidden'  

  Thanks.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/566/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzY1NzM4NzU=
title: Datasette Edit
state: closed
created_at: 2019-08-04T17:09:28Z
updated_at: 2020-02-25T03:40:50Z
closed_at: 2020-02-25T03:40:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Datasette started out immutable. Then it gained the ability to run against read-only databases that were being modified by other processes. It's time for the next logical progression: the option to allow Datasette (or more likely individual plugins) to write to the database!

  This is going to require some careful rethinking of how connection management works.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/567/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0NzY4NTI4NjE=
title: Add database_color as a configurable option
state: open
created_at: 2019-08-05T13:14:45Z
updated_at: 2023-08-11T05:19:42Z
author_association: NONE
body: This would be really useful as it would allow us to tie in with colour schemes.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/568/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODA5NjEzMzA=
title: Ability to list views, and to access db["view_name"].rows / rows_where / etc
state: closed
created_at: 2019-08-15T02:00:28Z
updated_at: 2019-08-23T12:41:09Z
closed_at: 2019-08-23T12:20:15Z
author_association: NONE
type: issue
state_reason: completed
body:
  The docs show me how to create a view via `db.create_view()` but I can't seem to get back to that view post-creation; if I query it as a table it returns `None`, and it doesn't appear in the table listing, even though querying the view works fine from inside the sqlite3 command-line.

  It'd be great to have the view as a pseudo-table, or if the python/sqlite3 module makes that hard to pull off (I couldn't figure it out), to have that edge-case documented next to the `db.create_view()` docs.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/54/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODE4ODUyNzk=
title: More advanced connection pooling
state: open
created_at: 2019-08-17T13:20:41Z
updated_at: 2019-10-02T22:44:37Z
author_association: OWNER
type: issue
body:
  We need a much smarter way of handling database connections.

  Today, connections are simple: Datasette runs a number of threads (defaults to 3) and each thread gets a threadlocal read-only (or immutable) connection to each attached database - opened on demand.

  For Datasette Library (#417) I want to support potentially hundreds of attached databases. Datasette Edit (#567) is going to introduce a need for writable connections too.

  I'd also like to be able to run joins across multiple databases (#283) which further complicates things.

  Supporting thousands of open SQLite connections at once feels like it won't provide good enough performance (though I should benchmark that to be sure). Some kind of connection pooling is likely to be necessary.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/569/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzA4MjkyNDQ3
title: Ability to introspect and run queries against views
state: closed
created_at: 2019-08-17T13:40:56Z
updated_at: 2019-08-23T12:19:42Z
closed_at: 2019-08-23T12:19:42Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/55
body: See #54
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/55/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODc1OTgwNDI=
title: Implement code to pull checkins from the Foursquare API
state: closed
created_at: 2019-08-30T17:40:02Z
updated_at: 2019-08-30T18:23:24Z
closed_at: 2019-08-30T18:23:24Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  The tool currently only works with a pre-prepared JSON file of checkins.

  When called without options, it should prompt the user to paste in a Foursquare OAuth token.

  The `--token=` option should work too, and should be backed up by an optional environment variable.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODc1OTg0Njg=
title: --save option to dump checkins to a JSON file on disk
state: closed
created_at: 2019-08-30T17:41:06Z
updated_at: 2019-08-31T02:40:21Z
closed_at: 2019-08-31T02:40:21Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  This is a complement to the `--load` option - mainly useful for development purposes.

  (I'll rename `--file` to `--load` as part of this issue).

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODc2MDA1OTU=
title: Option to fetch only checkins more recent than the current max checkin
state: closed
created_at: 2019-08-30T17:46:45Z
updated_at: 2019-10-16T20:41:23Z
closed_at: 2019-10-16T20:39:59Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  The Foursquare checkins API supports "return every checkin occurring after this point" - I can pass it the maximum createdAt date currently stored in the database. This will allow for quick incremental fetches via a cron.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODc2MDExMjE=
title: Online tool for getting a Foursquare OAuth token
state: closed
created_at: 2019-08-30T17:48:14Z
updated_at: 2019-08-31T18:07:26Z
closed_at: 2019-08-31T18:07:26Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I will link to this from the documentation. See also this conversation on Twitter: https://twitter.com/simonw/status/1166822603023011840

  I've decided to go with "copy and paste in a token" rather than hooking up a local web server that can have tokens passed to it.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODc3MjE4ODQ=
title: Treat Foursquare timestamps as UTC
state: closed
created_at: 2019-08-31T02:44:47Z
updated_at: 2019-08-31T02:50:41Z
closed_at: 2019-08-31T02:50:41Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Current test failure is due to timezone differences between my laptop and Circle CI:

  https://circleci.com/gh/dogsheep/swarm-to-sqlite/3

  ```
  E         Full diff:
  E         - [{'created': '2018-07-01T04:48:19',
  E         ?                           ^
  E         + [{'created': '2018-07-01T02:48:19',
  E         ?                           ^
  E         'createdAt': 1530413299,
  ```

  The timestamps I store in `created` should always be UTC.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzEzMDA3NDgz
title: Escape the table name in populate_fts and search.
state: closed
created_at: 2019-09-01T06:29:05Z
updated_at: 2019-09-02T17:23:21Z
closed_at: 2019-09-02T17:23:21Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/56
type: pull
body:
  The table names weren't escaped using double quotes in the populate_fts method. 

  Reproducible case: 
  ```
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database("abc.db")
  >>> db["http://example.com"].insert_all([
  ...     {"id": 1, "age": 4, "name": "Cleo"},
  ...     {"id": 2, "age": 2, "name": "Pancakes"}
  ... ], pk="id")
  <Table http://example.com (id, age, name)>
  >>> db["http://example.com"].enable_fts(["name"])
  Traceback (most recent call last):
    File "<input>", line 1, in <module>
      db["http://example.com"].enable_fts(["name"])
    File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py", l
  ine 705, in enable_fts
      self.populate_fts(columns)
    File "/home/amjith/.virtualenvs/itsysearch/lib/python3.7/site-packages/sqlite_utils/db.py", l
  ine 715, in populate_fts
      self.db.conn.executescript(sql)
  sqlite3.OperationalError: unrecognized token: ":"
  >>> 
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/56/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzEzMTA1NjM0
title: Add triggers while enabling FTS
state: closed
created_at: 2019-09-02T04:23:40Z
updated_at: 2019-09-03T01:03:59Z
closed_at: 2019-09-02T23:42:29Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/57
type: pull
body:
  This adds the option for a user to set up triggers in the database to keep their FTS table in sync with the parent table. 

  Ref: https://sqlite.org/fts5.html#external_content_and_contentless_tables

  I would prefer to make the creation of triggers the default behavior, but that will break existing usage where people have been calling `populate_fts` after inserting new rows.

  I am happy to make changes to the PR as you see fit. 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/57/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODgyOTM5MjY=
title: Support enabling FTS on views
state: closed
created_at: 2019-09-02T18:56:36Z
updated_at: 2020-10-16T18:39:36Z
closed_at: 2020-10-16T18:39:31Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Right now enable_fts() is only implemented for Table(). Technically sqlite supports enabling fts on views. But it requires deeper thought since views don't have `rowid` and the current implementation of enable_fts() relies on the presence of `rowid` column. 

  It is possible to provide an alternative rowid using the `content_rowid` option to the FTS5() function. 

  Ref: https://sqlite.org/fts5.html#fts5_table_creation_and_initialization

  >  The "content_rowid" option, used to set the rowid field of an external content table. 

  This will further complicate `enable_fts()` function by adding an extra argument. I'm wondering if that is outside the scope of this tool or should I work on that feature and send a PR? 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/58/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODgzMzg1MTY=
title: detect_fts should handle alternative table escaping
state: closed
created_at: 2019-09-02T23:43:29Z
updated_at: 2019-09-03T00:32:28Z
closed_at: 2019-09-03T00:32:28Z
author_association: OWNER
type: issue
state_reason: completed
body:
  sqlite-utils now uses a better way of escaping table names, which has highlighted a bug in Datasette.

  Datasette has its own version of the `detect_fts` function - at https://github.com/simonw/datasette/blob/d224ee2c98ac39c2c6e21a0ac0c62e5c3e1ccd11/datasette/utils/__init__.py#L466-L479 - which fails to pick up FTS tables created using the new escaping pattern.

  _Originally posted by @simonw in https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527258212_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/570/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODgzMzg5NjU=
title: Ability to introspect triggers
state: closed
created_at: 2019-09-02T23:47:16Z
updated_at: 2019-09-03T01:52:36Z
closed_at: 2019-09-03T00:09:42Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Now that we're creating triggers (thanks to @amjith in #57) it would be neat if we could introspect them too.

  I'm thinking:

  `db.triggers` - lists all triggers for the database
  `db["tablename"].triggers` - lists triggers for that table

  The underlying query for this is `select * from sqlite_master where type = 'trigger'`

  I'll return the trigger information in a new namedtuple, similar to how Indexes and ForeignKeys work.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/59/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzEzMzgzMzE3
title: db.triggers and table.triggers introspection
state: closed
created_at: 2019-09-03T00:04:32Z
updated_at: 2019-09-03T00:09:42Z
closed_at: 2019-09-03T00:09:42Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/60
body: Closes #59
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/60/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzEzMzg0OTI2
title: detect_fts now works with alternative table escaping
state: closed
created_at: 2019-09-03T00:23:39Z
updated_at: 2019-09-03T00:32:28Z
closed_at: 2019-09-03T00:32:28Z
author_association: OWNER
pull_request: simonw/datasette/pulls/571
body: Fixes #570
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/571/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODg4MzMxMzY=
title: Imported followers should go in "users", relationships in "following"
state: closed
created_at: 2019-09-03T21:27:37Z
updated_at: 2019-09-04T20:23:04Z
closed_at: 2019-09-04T20:23:04Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Right now `twitter-to-sqlite followers` dumps everything in a `followers` table, and doesn't actually record which account they are following!

  It should instead save them all in a global `users` table and then set up m2m relationships in a `following` table. This also means it should create a record for the specified user in order to record both sides of each relationship.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODg4MzM2OTg=
title: "twitter-to-sqlite user-timeline" command for pulling tweets by a specific user
state: closed
created_at: 2019-09-03T21:29:12Z
updated_at: 2019-09-04T20:02:11Z
closed_at: 2019-09-04T20:02:11Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Twitter only allows up to 3,200 tweets to be retrieved from https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html

  I'm going to do:

      $ twitter-to-sqlite tweets simonw

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODg4MzM5NzU=
title: Command for running a search and saving tweets for that search
state: closed
created_at: 2019-09-03T21:29:56Z
updated_at: 2019-11-04T05:31:56Z
closed_at: 2019-11-04T05:31:16Z
author_association: MEMBER
body: $ twitter-to-sqlite search dogsheep
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODg4MzU1ODY=
title: Command for importing data from a Twitter Export file
state: closed
created_at: 2019-09-03T21:34:13Z
updated_at: 2019-10-11T06:45:02Z
closed_at: 2019-10-11T06:45:02Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Twitter lets you export all of your data as an archive file: https://twitter.com/settings/your_twitter_data

  A command for importing this data into SQLite would be extremely useful.

      $ twitter-to-sqlite import twitter.db path-to-archive.zip

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODg4NzQ4MTU=
title: Write tests that simulate the Twitter API
state: open
created_at: 2019-09-03T23:55:35Z
updated_at: 2019-09-03T23:56:28Z
author_association: MEMBER
body: I can use betamax for this: https://pypi.org/project/betamax/
type: issue
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODk0MTk3ODI=
title: Extract extended_entities into a media table
state: closed
created_at: 2019-09-04T21:59:10Z
updated_at: 2019-09-04T22:08:01Z
closed_at: 2019-09-04T22:08:01Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  <img width="747" alt="cleo__select___from_tweets_order_by_id_limit_101" src="https://user-images.githubusercontent.com/9599/64294799-880f7f00-cf24-11e9-83b8-75c1ee54182c.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0ODk0MjkyODQ=
title: Error running datasette publish with just --source_url
state: closed
created_at: 2019-09-04T22:19:22Z
updated_at: 2019-11-13T04:28:44Z
closed_at: 2019-11-13T04:28:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  datasette publish now cleo.db \
      --source_url="https://twitter.com/cleopaws" \
  ```
  Gave me this error:
  <img width="338" alt="Error_500" src="https://user-images.githubusercontent.com/9599/64295924-74b1e300-cf27-11e9-9aed-c69e99c97030.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/572/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTA3OTgxMzA=
title: users-lookup command for fetching users
state: closed
created_at: 2019-09-08T19:47:59Z
updated_at: 2019-09-08T20:32:13Z
closed_at: 2019-09-08T20:32:13Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup
  ```
  https://api.twitter.com/1.1/users/lookup.json?user_id=783214,6253282
  https://api.twitter.com/1.1/users/lookup.json?screen_name=simonw,cleopaws
  ```
  CLI design:
  ```
  $ twitter-to-sqlite users-lookup simonw cleopaws
  $ twitter-to-sqlite users-lookup 783214 6253282 --ids
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTA4MDMxNzY=
title: --sql and --attach options for feeding commands from SQL queries
state: closed
created_at: 2019-09-08T20:35:49Z
updated_at: 2020-03-20T23:13:01Z
closed_at: 2020-03-20T23:13:01Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Say you want to fetch Twitter profiles for a list of accounts that are stored in another database:

      $ twitter-to-sqlite users-lookup users.db --attach attending.db \
          --sql "select Twitter from attending.attendes where Twitter is not null"

  The SQL query you feed in is expected to return a list of screen names suitable for processing further by the command.

  Should be supported by all three of:

  - [x] `twitter-to-sqlite users-lookup`
  - [x] `twitter-to-sqlite user-timeline`
  - [x] `twitter-to-sqlite followers` and `friends`

  The `--attach` option allows other SQLite databases to be attached to the connection. Without it the SQL query will have to read from the single attached database.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTEyMTk5MTA=
title: importing CSV to SQLite as library
state: closed
created_at: 2019-09-09T17:12:40Z
updated_at: 2019-11-04T16:25:01Z
closed_at: 2019-11-04T16:25:01Z
author_association: NONE
type: issue
state_reason: completed
body:
  CSV can be imported to SQLite when used CLI, but I don't see documentation for when using as library. 

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/61/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTE3OTExNTI=
title: followers-ids and friends-ids subcommands
state: closed
created_at: 2019-09-10T16:58:15Z
updated_at: 2019-09-10T17:36:55Z
closed_at: 2019-09-10T17:36:55Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  These will import follower and friendship IDs into the following tables, using these APIs:

  https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-followers-ids
  https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-ids

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTIxNTM1MzI=
title: Exposing Datasette via Jupyter-server-proxy
state: closed
created_at: 2019-09-11T10:32:36Z
updated_at: 2020-03-26T09:41:30Z
closed_at: 2020-03-26T09:41:30Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  It is possible to expose a running `datasette` service in a Jupyter environment such as a MyBinder environment using the [`jupyter-server-proxy`](https://github.com/jupyterhub/jupyter-server-proxy).

  For example, using [this demo Binder](https://mybinder.org/v2/gh/binder-examples/r/master?filepath=index.ipynb) which has the server proxy installed, we can then upload a simple test database from the notebook homepage, from a Jupyter termianl install datasette and set it running against the test db on eg port 8001 and then view it via the path `proxy/8001`.

  Clicking links results in 404s though because  the `datasette` links aren't relative to the current path?

  ![image](https://user-images.githubusercontent.com/82988/64689964-44b69280-d487-11e9-8f9f-3681422bcc9f.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/573/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTIyOTc5MzA=
title: Rethink progress bars for various commands
state: closed
created_at: 2019-09-11T15:06:47Z
updated_at: 2020-04-01T03:45:48Z
closed_at: 2020-04-01T03:45:48Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Progress bars and the `--silent` option are implemented inconsistently across commands at the moment.

  This is made more challenging by the fact that for many operations the total length is not known.

  https://click.palletsprojects.com/en/7.x/api/#click.progressbar

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTM1OTk4MTg=
title: Command for fetching starred repos
state: closed
created_at: 2019-09-14T08:36:29Z
updated_at: 2019-09-14T21:30:48Z
closed_at: 2019-09-14T21:30:48Z
author_association: MEMBER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTM2Njg4NjI=
title: Extract licenses from repos into a separate table
state: closed
created_at: 2019-09-14T21:33:41Z
updated_at: 2019-09-14T21:46:58Z
closed_at: 2019-09-14T21:46:58Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  <img width="588" alt="starred3__repos__800_rows" src="https://user-images.githubusercontent.com/9599/64913845-a2074980-d6fc-11e9-815a-f103ad0c6f56.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTM2NzA0MjY=
title: Command to fetch all repos belonging to a user or organization
state: closed
created_at: 2019-09-14T21:54:21Z
updated_at: 2019-09-17T00:17:53Z
closed_at: 2019-09-17T00:17:53Z
author_association: MEMBER
body:
  How about this:

  $ github-to-sqlite repos simonw

type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTM2NzA3MzA=
title: Command to fetch stargazers for one or more repos
state: closed
created_at: 2019-09-14T21:58:22Z
updated_at: 2020-05-02T21:30:27Z
closed_at: 2020-05-02T21:30:27Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Maybe this:

      $ github-to-sqlite stargazers github.db simonw/datasette

  It could accept more than one repos.

  Maybe have options similar to `--sql` in [twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) so you can e.g. fetch all stargazers for all of the repos you have fetched into the database already (or all of the repos belonging to owner X)

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTM2NzEwMTQ=
title: Add "incomplete" boolean to users table for incomplete profiles
state: closed
created_at: 2019-09-14T22:01:50Z
updated_at: 2020-03-23T19:23:31Z
closed_at: 2020-03-23T19:23:30Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  User profiles that are fetched from e.g. stargazers (#4) are incomplete - they have a login but they don't have name, company etc. 

  Add a `incomplete` boolean flag to the `users` table to record this. Then later I can add a `backfill-users` command which loops through and fetches missing data for those incomplete profiles.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/5/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTQ2ODU3OTE=
title: Improve usage description of --host option
state: closed
created_at: 2019-09-17T15:12:12Z
updated_at: 2019-11-01T21:58:17Z
closed_at: 2019-11-01T21:57:54Z
author_association: NONE
type: issue
state_reason: completed
body:
  It would be nice if the `--host` option had a clearer description. I tried to get datasette running on an AWS instance and it took a while to realize it was only listening on localhost. So I wanted to make it listen on an non-localhost interface and tried giving a couple of values to `--host` (a host name, then an interface name), but none of them did. In the end I read the source to see that the option is passed to `uvicorn` and looked at the uvicorn docs, which also didn't help. Then I searched the web for "example running datasette on a host" which led me to https://github.com/simonw/datasette/issues/514 where I saw someone using `-h 0.0.0.0`. I tried that and it works.  That usage could be mentioned somewhere, and might save someone else some time.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/574/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTY0MTUzMjE=
title: Figure out some interesting example SQL queries
state: open
created_at: 2019-09-20T15:28:07Z
updated_at: 2021-05-03T03:46:23Z
author_association: MEMBER
type: issue
body:
  My knowledge of genetics has left me short here. I'd love to be able to provide some interesting example SELECT queries - maybe one that spots if you are [likely to have red hair?](https://www.snpedia.com/index.php/Rs1805007)

reactions:
  {"url": "https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTcxNjIyODg=
title: Plugin documentation should cover how to bundle static/templates in setup.py
state: closed
created_at: 2019-09-23T15:15:18Z
updated_at: 2020-10-24T20:06:17Z
closed_at: 2020-10-24T20:03:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  These sections here should cover it: https://datasette.readthedocs.io/en/latest/plugins.html#static-assets

  Example: https://github.com/simonw/datasette-auth-github/blob/bf01f8f01b87a6cb09c47380ba0a86e0546ebb38/setup.py#L30
  ```
  package_data={"datasette_auth_github": ["templates/*.html"]},
  ```
  Also from https://github.com/simonw/datasette-plugin-demos/blob/0ccf9e6189e923046047acd7878d1d19a2cccbb1/setup.py#L18-L22

      package_data={
          'datasette_plugin_demos': [
              'static/plugin.js',
          ],
      },

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/575/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTcxNzAzNTU=
title: Documented internals API for use in plugins
state: closed
created_at: 2019-09-23T15:28:50Z
updated_at: 2021-01-05T23:12:51Z
closed_at: 2021-01-05T23:12:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Quite a few of the plugin hooks make a `datasette”`instance of the Datasette class available to the plugins, so that they can look up configuration settings and execute database queries.

  This means it should provide a documented, stable API so that plugin authors can rely on it.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/576/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU0OTcxNzEzOTA=
title: Utility mechanism for plugins to render templates
state: closed
created_at: 2019-09-23T15:30:36Z
updated_at: 2020-02-04T20:26:20Z
closed_at: 2020-02-04T20:26:19Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Sometimes a plugin will need to render a template for some custom UI. We need a documented API for doing this, which ensures that everything will work correctly if you extend base.html etc.

  See also #576. This could be a `.render()` method on the Datasette class, but that feels a bit weird - should that class also take responsibility for rendering?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/577/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzIyNTI5Mzgx
title: Added support for multi arch builds
state: closed
created_at: 2019-09-29T18:43:03Z
updated_at: 2019-11-13T19:13:15Z
closed_at: 2019-11-13T19:13:15Z
author_association: NONE
pull_request: simonw/datasette/pulls/578
type: pull
body:
  Minor changes in Dockerfile and new Makefile to support Docker multi architecture builds. `make`will build one image per architecture and push them as one Docker manifest to Docker Hub. Feel free to change `IMAGE_NAME ` to `datasetteproject/datasette` to update your official Docker Hub image(s).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/578/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDA3ODMzNzM=
title: [enhancement] Method to delete a row in python
state: closed
created_at: 2019-10-01T09:45:47Z
updated_at: 2019-11-04T16:30:34Z
closed_at: 2019-11-04T16:18:18Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi !
  Thanks for the lib !

  Obviously, every possible sql queries won't have a dedicated method.

  But I was thinking : a method to delete a row (I'm terrible with names, maybe `delete_where()` or something, would be useful.

  I have a Database, with primary key.

  For the moment, I use :

  ```Python3
  db.conn.execute(f"DELETE FROM table WHERE key = {key_id}")
  db.conn.commit()
  ```
  to delete a row I don't need anymore, giving his primary key.

  Works like a charm.

  Just an idea :

  ```Python3
  table.delete_where_pkey({'key': key_id})
  ```
  or something (I know, I'm terrible at naming methods...).

  Pros : well, no need to write SQL query.

  Cons : WHERE normally allows to do many more things (operators =, <>, >, <, BETWEEN), not to mention AND, OR, etc...
  Method is maybe to specific, and/or a pain to render more flexible.

  Again, just a thought. Writing his own sql works too, so...

  Thanks again.
  See yah.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/62/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzIzOTgzNzMy
title: New connection pooling
state: open
created_at: 2019-10-02T23:22:19Z
updated_at: 2019-11-15T22:57:21Z
author_association: OWNER
pull_request: simonw/datasette/pulls/579
body: See #569
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/579/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDIzNTUzODQ=
title: Testing utilities should be available to plugins
state: closed
created_at: 2019-10-03T23:58:26Z
updated_at: 2020-02-28T07:58:46Z
closed_at: 2020-02-28T07:58:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I'm trying to write a plugin at the moment ([datasette-atom](https://github.com/simonw/datasette-atom)) which needs to run unit tests against a full in-memory Datasette instance, in the same way that the Datasette test suite itself works.

  I got it working by creating copies of the [TestClient and TestResponse classes](https://github.com/simonw/datasette/blob/a314b761866d250c16f1ff6dd682010cf4181eb4/tests/fixtures.py#L22-L96) within the plugin itself:

  https://github.com/simonw/datasette-atom/commit/c0e3bd9556d7b31f253a8bf666d42205cd24f4fc#diff-33337525d2d877f7cc7f33737bfd2d7b

  I had to do this because those classes are in the `tests/` directory within Datasette, so they don't get included in the package that ships to PyPI.

  It would be better if these classes were included in the main package in a way that made it easy for plugins to reuse them to write their own tests.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/580/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDI5OTM1MDk=
title: Redesign register_output_renderer callback
state: closed
created_at: 2019-10-05T17:43:23Z
updated_at: 2020-05-28T02:24:14Z
closed_at: 2020-05-28T02:21:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  In building https://github.com/simonw/datasette-atom it became clear that the callback function (which currently accepts just args, data and view_name) would also benefit from access to a mechanism to render templates and a `datasette` instance so it can execute SQL.

  To maintain backwards compatibility with existing plugins, we can introspect the callback function to see if it wants those new arguments or not.

  At a minimum I want to make `datasette` and ASGI `scope` available.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/581/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMwNDUyMjE=
title: Commands for recording real-time tweets from the streaming API
state: closed
created_at: 2019-10-06T03:09:30Z
updated_at: 2019-10-06T04:54:17Z
closed_at: 2019-10-06T04:48:31Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://developer.twitter.com/en/docs/tweets/filter-realtime/api-reference/post-statuses-filter

  We can support tracking keywords and following specific users.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMwNTMyNDM=
title: Datasette should not completely crash if one SQLite database is malformed
state: open
created_at: 2019-10-06T05:11:43Z
updated_at: 2019-10-06T05:11:43Z
author_association: OWNER
type: issue
body:
  If you run Datasette against a number of database files and one of them is malformed, you get this 500 error on the index page:

  <img width="373" alt="Error_500" src="https://user-images.githubusercontent.com/9599/66264592-0463df00-e7bd-11e9-816f-82d0f05d625a.png">

  It would be better if Datasette still worked and listed the databases that were NOT malformed, then showed an inline error message just for the one that could not be accessed.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/582/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMwNTM4MDA=
title: Extract "source" into a separate lookup table
state: closed
created_at: 2019-10-06T05:17:23Z
updated_at: 2019-10-17T15:49:24Z
closed_at: 2019-10-17T15:49:24Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  It's pretty bulky and ugly at the moment:

  <img width="334" alt="trump__tweets__1_820_rows" src="https://user-images.githubusercontent.com/9599/66264630-df23a080-e7bd-11e9-9154-403c2e69f841.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMwODUwMTM=
title: statuses-lookup command
state: closed
created_at: 2019-10-06T11:00:20Z
updated_at: 2019-10-07T00:33:49Z
closed_at: 2019-10-07T00:31:44Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  For bulk retrieving tweets by their ID.

  https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup

  Rate limit is 900/15 minutes (1 call per second) but each call can pull up to 100 IDs, so we can pull 6,000 per minute.

  Should support `--SQL` and `--attach` #8 

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMxMjg5MTQ=
title: Enable "explain" and "explain query plan" for CTEs
state: closed
created_at: 2019-10-06T17:00:10Z
updated_at: 2019-10-06T17:24:07Z
closed_at: 2019-10-06T17:24:07Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This currently throws an error:

  https://latest.datasette.io/fixtures?sql=explain+WITH+RECURSIVE%0D%0A++xaxis%28x%29+AS+%28VALUES%28-2.0%29+UNION+ALL+SELECT+x%2B0.05+FROM+xaxis+WHERE+x%3C1.2%29%2C%0D%0A++yaxis%28y%29+AS+%28VALUES%28-1.0%29+UNION+ALL+SELECT+y%2B0.1+FROM+yaxis+WHERE+y%3C1.0%29%2C%0D%0A++m%28iter%2C+cx%2C+cy%2C+x%2C+y%29+AS+%28%0D%0A++++SELECT+0%2C+x%2C+y%2C+0.0%2C+0.0+FROM+xaxis%2C+yaxis%0D%0A++++UNION+ALL%0D%0A++++SELECT+iter%2B1%2C+cx%2C+cy%2C+x*x-y*y+%2B+cx%2C+2.0*x*y+%2B+cy+FROM+m+%0D%0A+++++WHERE+%28x*x+%2B+y*y%29+%3C+4.0+AND+iter%3C28%0D%0A++%29%2C%0D%0A++m2%28iter%2C+cx%2C+cy%29+AS+%28%0D%0A++++SELECT+max%28iter%29%2C+cx%2C+cy+FROM+m+GROUP+BY+cx%2C+cy%0D%0A++%29%2C%0D%0A++a%28t%29+AS+%28%0D%0A++++SELECT+group_concat%28+substr%28%27+.%2B*%23%27%2C+1%2Bmin%28iter%2F7%2C4%29%2C+1%29%2C+%27%27%29+%0D%0A++++FROM+m2+GROUP+BY+cy%0D%0A++%29%0D%0ASELECT+group_concat%28rtrim%28t%29%2Cx%270a%27%29+FROM+a%3B

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/583/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMxOTAyNDE=
title: Codec error in some CSV exports
state: closed
created_at: 2019-10-07T01:15:34Z
updated_at: 2021-06-17T18:13:20Z
closed_at: 2019-10-18T05:23:16Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Got this exploring my Swarm checkins:

  ![448DBFC4-71F8-4846-83C0-BEA511B2157A](https://user-images.githubusercontent.com/9599/66279259-3af53480-e865-11e9-9651-04fd2d895392.jpeg)

  `/swarm/stickers.csv?stickerType=messageOnly&_size=max`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/584/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMyMTczNzU=
title: Databases on index page should display in order they were passed to "datasette serve"?
state: closed
created_at: 2019-10-07T03:42:39Z
updated_at: 2019-10-14T03:52:34Z
closed_at: 2019-10-14T03:52:34Z
author_association: OWNER
type: issue
state_reason: completed
body:
  If you run this:

      datasette serve -h 127.0.0.1 -p 8000 -m phone-locations.db healthkit.db locations.db genome.db

  Then the index page for that Datasette instance should show the databases in the order they were specified on the command-line.

  Mind you when we add pagination to that page in #468 we may want to do something different here.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/585/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMyMTgyMDU=
title: Enable browser caching for plugin statics with datasette-auth
state: closed
created_at: 2019-10-07T03:47:14Z
updated_at: 2019-10-07T15:46:04Z
closed_at: 2019-10-07T15:46:03Z
author_association: OWNER
type: issue
state_reason: completed
body:
  An authenticated Datasette I run is seeing delays on every page load. On looking at the network inspector it turns out it's because datasette-vega is nearly 1MB and a `cache-control: private` is preventing it from being cached!

  <img width="1071" alt="github__repos__32_rows_where_where_private___1" src="https://user-images.githubusercontent.com/9599/66284064-f96f8400-e87a-11e9-875f-5a6c4a8c4f41.png">


  This may well turn out to be a bug in `datasette-auth-github` but it's still worth tracking here because caching of static assets from plugins is very important.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/586/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMyMzMwMjE=
title: Use better pagination (and implement progress bar)
state: closed
created_at: 2019-10-07T04:58:11Z
updated_at: 2020-03-27T22:13:57Z
closed_at: 2020-03-27T22:13:57Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Right now we attempt to load everything at once - which caps out at 5,000 items and is really slow.

  We can do better by implementing pagination using count and offset.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMyMzQxNjk=
title: Track and use the 'since' value
state: closed
created_at: 2019-10-07T05:02:59Z
updated_at: 2020-03-27T22:22:30Z
closed_at: 2020-03-27T22:22:30Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Pocket says:

  > Whenever possible, you should use the since parameter, or count and and offset parameters when retrieving a user's list. After retrieving the list, you should store the current time (which is provided along with the list response) and pass that in the next request for the list. This way the server only needs to return a small set (changes since that time) instead of the user's entire list every time.

  At the bottom of https://getpocket.com/developer/docs/v3/retrieve

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMyNDM3ODQ=
title: Extract images into separate tables
state: open
created_at: 2019-10-07T05:43:01Z
updated_at: 2020-09-01T06:17:45Z
author_association: MEMBER
type: issue
body:
  As already done with authors. Slightly harder because images do not have a universally unique ID. Also need to figure out what to do about there being columns for both `image` and `images`.

  <img width="1522" alt="memory__items" src="https://user-images.githubusercontent.com/9599/66287418-9ab20680-e88a-11e9-96bf-6c80d881eff0.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDMyNDQ0MTA=
title: When importing favorites, record which user favorited them
state: closed
created_at: 2019-10-07T05:45:11Z
updated_at: 2019-10-14T03:30:25Z
closed_at: 2019-10-14T03:30:25Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  This code currently just dumps them into the `tweets` table without recording who it was who had favorited them.

  https://github.com/dogsheep/twitter-to-sqlite/blob/436a170d74ec70903d1b4ca430c2c6b6435cdfcc/twitter_to_sqlite/cli.py#L152-L157

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDQyMzg0NjE=
title: sqlite3.OperationalError: table users has no column named bio
state: closed
created_at: 2019-10-08T19:39:52Z
updated_at: 2019-10-13T05:31:28Z
closed_at: 2019-10-13T05:30:19Z
author_association: NONE
type: issue
state_reason: completed
body:
  ```
  $ github-to-sqlite repos github.db
  $ github-to-sqlite starred github.db dazzag24

  Traceback (most recent call last):
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/bin/github-to-sqlite", line 10, in <module>
      sys.exit(cli())
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 106, in starred
      utils.save_stars(db, user, stars)
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 177, in save_stars
      user_id = save_user(db, user)
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 61, in save_user
      return db["users"].upsert(to_save, pk="id").last_pk
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1067, in upsert
      extracts=extracts,
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 916, in insert
      extracts=extracts,
    File "/home/darreng/.virtualenvs/dogsheep-d2PjdrD7/lib/python3.6/site-packages/sqlite_utils/db.py", line 1024, in insert_all
      result = self.db.conn.execute(sql, values)
  sqlite3.OperationalError: table users has no column named bio

  ```

  ```
  $ pipenv graph
  github-to-sqlite==0.4
    - requests [required: Any, installed: 2.22.0]
      - certifi [required: >=2017.4.17, installed: 2019.9.11]
      - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
      - idna [required: >=2.5,<2.9, installed: 2.8]
      - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6]
    - sqlite-utils [required: ~=1.11, installed: 1.11]
      - click [required: Any, installed: 7.0]
      - click-default-group [required: Any, installed: 1.2.2]
        - click [required: Any, installed: 7.0]
      - tabulate [required: Any, installed: 0.8.5]

  Python 3.6.8
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDQ3MjA3MzE=
title: Add more details on how to request data from google takeout correctly.
state: open
created_at: 2019-10-09T15:17:34Z
updated_at: 2019-10-09T15:17:34Z
author_association: NONE
type: issue
body:
  The default is to download everything.  This can result in an enormous amount of data when you only really need 2 types of data for now:

  - My Activity
  - Location History

  In addition unless you specify that "My Activity" is downloaded in JSON format the default is HTML.  This then causes the 

  `google-takeout-to-sqlite my-activity takeout.db takeout.zip`

  command to fail as it only contains html files not json files.

  Thanks

reactions:
  {"url": "https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDQ4MDU4NTc=
title: Use --platform=managed for publish cloudrun
state: closed
created_at: 2019-10-09T18:02:16Z
updated_at: 2019-10-17T21:51:57Z
closed_at: 2019-10-17T21:51:57Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Running `datasette publish cloudrun` now shows this message:

  > Please choose a target platform:
  > [1] Cloud Run (fully managed)
  > [2] Cloud Run on GKE
  > [3] a Kubernetes cluster
  > [4] cancel
  >Please enter your numeric choice:  1
  >
  > To specify the platform yourself, pass `--platform managed`. Or, to make this the default target platform, run `gcloud config set run/platform managed`.

  May as well set that as a default.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/587/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDU1MTIyNTE=
title: Queries per DB table in metadata.json
state: closed
created_at: 2019-10-10T21:08:19Z
updated_at: 2019-10-21T12:58:22Z
closed_at: 2019-10-21T01:48:42Z
author_association: NONE
type: issue
state_reason: completed
body:
  It doesn't appear possible to have separate queries defined per database table.  When I do something like below, my table descriptions show up but not the queries:

  ` "databases": {
        "MYDB": {
          "tables": {
            "MYFIRSTTABLE": {
              "source": "Test",
              "source_url": "https://www.google.com",
              "queries": {
                  "Query 1": {
                      "sql": "select * from MYFIRSTTABLE",
                      "title": "Query 1",
                      "description": "This is the first query"
                    },
              }
            },
            "MYSECONDTABLE": {
                "source":"Test2",
                "source_url":"https://www.google.com",
              "queries": {
                  "Query 2" : {
                      "sql":"select * from MYSECONDTABLE;",
                      "title": "Query 2",
                      "description":"This is the second query"
                  }
              }
            }
          }`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/588/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI3MDUxNjcz
title: twitter-to-sqlite import command, refs #4
state: closed
created_at: 2019-10-11T06:37:14Z
updated_at: 2019-10-11T06:45:01Z
closed_at: 2019-10-11T06:45:01Z
author_association: MEMBER
pull_request: dogsheep/twitter-to-sqlite/pulls/15
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDU2NzM2NDU=
title: Do a better job with archived direct message threads
state: open
created_at: 2019-10-11T06:55:21Z
updated_at: 2019-10-11T06:55:27Z
author_association: MEMBER
type: issue
body:
  https://github.com/dogsheep/twitter-to-sqlite/blob/fb2698086d766e0333a55bb73435e7283feeb438/twitter_to_sqlite/archive.py#L98-L99

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/16/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDU2NzQ5NDk=
title: import command should empty all archive-* tables first
state: closed
created_at: 2019-10-11T06:58:43Z
updated_at: 2019-10-11T15:40:08Z
closed_at: 2019-10-11T15:40:08Z
author_association: MEMBER
body: Can have a CLI option for NOT doing that.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI3MTY5NzQ4
title: Display metadata footer on custom SQL queries
state: closed
created_at: 2019-10-11T12:10:28Z
updated_at: 2019-10-14T08:58:23Z
closed_at: 2019-10-14T03:53:22Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/589
body: Closes #408
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/589/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI3MTcyNTQ1
title: Handle spaces in DB names
state: closed
created_at: 2019-10-11T12:18:22Z
updated_at: 2019-11-04T23:16:31Z
closed_at: 2019-11-04T23:16:30Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/590
body: Closes #503
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/590/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI3MTg4MDg3
title: Sort databases on homepage by argument order
state: closed
created_at: 2019-10-11T12:57:38Z
updated_at: 2019-10-14T08:57:50Z
closed_at: 2019-10-14T03:52:34Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/591
body: Closes #585
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/591/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDU5Mjg1MzA=
title: Command to import home-timeline
state: closed
created_at: 2019-10-11T15:47:54Z
updated_at: 2019-10-11T16:51:33Z
closed_at: 2019-10-11T16:51:12Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Feature request: https://twitter.com/johankj/status/1182563563136868352
  > Would it be possible to save all tweets in my timeline from the last X days? I would love to see how big a percentage some users are of my daily timeline as a metric on whether I should unfollow them/move them to a list.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI3Mjc5ODE4
title: Offer SQL formatting
state: closed
created_at: 2019-10-11T16:35:49Z
updated_at: 2019-10-14T08:57:12Z
closed_at: 2019-10-14T03:46:13Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/592
type: pull
body:
  SQL code will be formatted on page load, and can additionally be formatted by clicking the "Format SQL" button. Closes #136

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/592/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDYwODcyNjc=
title: since_id support for home-timeline
state: closed
created_at: 2019-10-11T22:48:24Z
updated_at: 2019-10-16T19:13:06Z
closed_at: 2019-10-16T19:12:46Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Currently every time you run `home-timeline` we pull all 800 available tweets. We should offer to support `since_id` (which can be provided or can be pulled directly from the database) in order to work more efficiently if this command is executed e.g. on a cron.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDYxODMyNDE=
title: make uvicorn optional dependancy (because not ok on windows python yet)
state: closed
created_at: 2019-10-12T12:51:07Z
updated_at: 2019-10-13T06:22:08Z
closed_at: 2019-10-13T06:22:07Z
author_association: NONE
type: issue
state_reason: completed
body:
  would it be possible to:
  - remove uvicorn mandatory dependancy ?
  - eventually make a fallback to hypercorn ?

  reason:
  - uvloop not yet supported on Windows/Python-3.8 and below, may happen with Python-3.9 only.
  - it seems a 6 lines effort (but I'm not expert)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/593/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDYyNjg5NDU=
title: --since support for various commands for refresh-by-cron
state: closed
created_at: 2019-10-13T03:40:46Z
updated_at: 2019-10-21T03:32:04Z
closed_at: 2019-10-16T19:26:11Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I want to run a cron that updates my Twitter database every X minutes.

  It should be able to retrieve the following without needing to paginate through everything:

  - [x] Tweets I have tweeted
  - [x] My home timeline (see #19)
  - [x] Tweets I have favourited

  It would be nice if this could be standardized across all commands as a `--since` option.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDYyNzY4OTM=
title: issue-comments command for importing issue comments
state: closed
created_at: 2019-10-13T05:23:58Z
updated_at: 2019-10-14T14:44:12Z
closed_at: 2019-10-13T05:24:30Z
author_association: MEMBER
body: Using this API: https://developer.github.com/v3/issues/comments/
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDYyOTcwNDg=
title: upgrade to uvicorn-0.9 to be Python-3.8 friendly
state: closed
created_at: 2019-10-13T09:23:43Z
updated_at: 2019-11-12T04:47:04Z
closed_at: 2019-11-12T04:47:04Z
author_association: NONE
body: uvicorn-0.8 relies on websockets-0.7 which lacks python-3.8 compatiblity
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/594/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI3NTQxMDQ2
title: bump uvicorn to 0.9.0 to be Python-3.8 friendly
state: closed
created_at: 2019-10-13T10:00:04Z
updated_at: 2019-11-12T04:46:48Z
closed_at: 2019-11-12T04:46:48Z
author_association: NONE
pull_request: simonw/datasette/pulls/595
body: as uvicorn-0.9 is needed to get websockets-8.0.2, which is needed to have Python-3.8 compatibility
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/595/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDY0MzI1NzI=
title: Fix &amp; escapes in tweet text
state: closed
created_at: 2019-10-14T03:37:28Z
updated_at: 2019-10-15T18:48:16Z
closed_at: 2019-10-15T18:48:16Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  <img width="1136" alt="twitter__tweets__21_773_rows_where_sorted_by_id_descending" src="https://user-images.githubusercontent.com/9599/66728360-38f91b80-edf9-11e9-95b5-ce6d097fe18e.png">

  Shouldn't be storing `&amp;` here.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDc0NTQ5NTg=
title: Handle really wide tables better
state: open
created_at: 2019-10-15T20:05:46Z
updated_at: 2022-09-07T00:58:41Z
author_association: OWNER
type: issue
body:
  If a table has hundreds of columns the Datasette UI starts getting unwieldy.

  Addressing this would be neat. One option would be to only select the first 30 columns by default and provide a UI for selecting more.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/596/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDgwMjQwMzI=
title: Ability to import from uncompressed archive or from specific files
state: closed
created_at: 2019-10-16T18:31:57Z
updated_at: 2019-10-16T18:53:36Z
closed_at: 2019-10-16T18:53:36Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Currently you can only import like this:

      $ twitter-to-sqlite import path-to-twitter.zip

  It would be useful if you could import from a folder that was decompressed from that zip:

      $ twitter-to-sqlite import path-to-twitter/

  AND from individual files within that folder - since that would allow you to e.g. selectively import certain files:

      $ twitter-to-sqlite import path-to-twitter/favorites.js path-to-twitter/tweets.js

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDgwNzA5Nzc=
title: If you have databases called foo.db and foo-bar.db you cannot visit /foo-bar
state: closed
created_at: 2019-10-16T20:07:41Z
updated_at: 2019-10-18T22:51:08Z
closed_at: 2019-10-18T22:51:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Weird bug I just came across.

  It appears that if you have one database called `foo.db` and another called `foo-bar.db` any attempts to visit `/foo-bar` will redirect to `/foo`.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/597/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDgxMDA4NDQ=
title: Character encoding bug with CSV export
state: closed
created_at: 2019-10-16T21:09:30Z
updated_at: 2021-06-17T18:13:20Z
closed_at: 2019-10-18T22:52:21Z
author_association: NONE
type: issue
state_reason: completed
body:
  I was just poking around, and at [this URL](https://sql-murder-mystery.datasette.io/sql-murder-mystery/crime_scene_report.csv?_stream=on&type=arson&_size=max), I encountered this error:

  ```
  'latin-1' codec can't encode character '\u2019' in position 27: ordinal not in range(256)
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/598/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDgxOTA3MzA=
title: Extremely simple migration system
state: closed
created_at: 2019-10-17T02:13:57Z
updated_at: 2019-10-17T16:57:17Z
closed_at: 2019-10-17T16:57:17Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Needed for #12. This is going to be an incredibly simple version of the Django migration system.

  * A `migrations` table, keeping track of which migrations were applied (and when)
  * A `migrate()` function which applies any pending migrations
  * A `MIGRATIONS` constant which is a list of functions to be applied

  The function names will be detected and used as the names of the migrations.

  Every time you run the CLI tool it will call the `migrate()` function before doing anything else.

  Needs to take into account that there might be no tables at all. As such, migration functions should sanity check that the tables they are going to work on actually exist.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI5MzI0MzY4
title: Tweet source extraction and new migration system
state: closed
created_at: 2019-10-17T15:24:56Z
updated_at: 2019-10-17T15:49:29Z
closed_at: 2019-10-17T15:49:24Z
author_association: MEMBER
pull_request: dogsheep/twitter-to-sqlite/pulls/24
body: Closes #12 and #23
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/24/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDg1Nzg3ODA=
title: Ensure migrations don't accidentally create foreign key twice
state: closed
created_at: 2019-10-17T16:08:50Z
updated_at: 2019-10-17T16:56:47Z
closed_at: 2019-10-17T16:56:47Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Is it possible for these lines to run against a database table that already has these foreign keys?

  https://github.com/dogsheep/twitter-to-sqlite/blob/c9295233f219c446fa2085cace987067488a31b9/twitter_to_sqlite/migrations.py#L21-L22

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI5ODkwMzIw
title: Fix for /foo v.s. /foo-bar issue in #597
state: closed
created_at: 2019-10-18T19:22:55Z
updated_at: 2019-10-18T22:51:07Z
closed_at: 2019-10-18T22:51:07Z
author_association: OWNER
pull_request: simonw/datasette/pulls/599
body: Refs #597
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/599/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDkzMzk5OTk=
title: Don't auto-format SQL on first page load
state: closed
created_at: 2019-10-18T22:36:10Z
updated_at: 2019-10-18T23:56:46Z
closed_at: 2019-10-18T23:56:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I've gone back and forth on this a bit, but I've decided I'm not keen on the way Datasette now automatically formats SQL when a query (or canned query) page first loads.

  I like having an optional "Format SQL" button, but applying formatting automatically means that if the user has carefully formatted their SQL to a specific style their formatting will be automatically over-ridden.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/600/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzI5OTQ3MTgw
title: Don't auto-format SQL on page load
state: closed
created_at: 2019-10-18T22:37:39Z
updated_at: 2019-10-20T02:29:49Z
closed_at: 2019-10-18T23:56:45Z
author_association: OWNER
pull_request: simonw/datasette/pulls/601
body: Refs #600
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/601/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzMwMDc2MjYz
title: Offer to format readonly SQL
state: closed
created_at: 2019-10-20T02:29:32Z
updated_at: 2019-11-04T07:29:33Z
closed_at: 2019-11-04T02:39:56Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/602
type: pull
body:
  Following discussion in #601, this PR adds a "Format SQL" button to
  read-only SQL (if the SQL actually differs from the formatting result).

  It also removes a console error on readonly SQL queries.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/602/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzMwMTI5MzU4
title: always pop as_format off args dict
state: closed
created_at: 2019-10-20T15:44:22Z
updated_at: 2019-10-30T19:12:22Z
closed_at: 2019-10-21T02:03:09Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/603
body: closes #563
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/603/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MDk2OTM3NzM=
title: _where= parameter is not persisted in hidden form fields
state: closed
created_at: 2019-10-21T02:14:10Z
updated_at: 2019-10-30T19:12:38Z
closed_at: 2019-10-30T18:49:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. on this page: https://v0-30.datasette.io/fixtures/roadside_attractions?_where=name%20like%20%27%museum%%27

  Click the "Apply" button and the `_where=` parameter will be dropped.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/604/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTAwNzYzNjg=
title: Support queries at the table level
state: open
created_at: 2019-10-21T15:58:30Z
updated_at: 2019-10-30T18:55:37Z
author_association: NONE
type: issue
body:
  Per the issue described in [issue #588](https://github.com/simonw/datasette/issues/588), it was determined queries are not supported at the table level.  Per my last comment in the issue, I'd like to request support for this as it would help eliminate errors in the event certain tables are not present in the database.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/605/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTIyMTg4NTg=
title: /-/plugins shows incorrect name for plugins
state: closed
created_at: 2019-10-24T22:53:25Z
updated_at: 2019-11-01T05:41:04Z
closed_at: 2019-11-01T05:40:07Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://fivethirtyeight.datasettes.com/-/plugins

  ```json
  [
      {
          "name": "datasette_jellyfish",
          "static": false,
          "templates": false,
          "version": "0.3"
      },
      {
          "name": "datasette_vega",
          "static": true,
          "templates": false,
          "version": "0.6.2"
      }
  ]
  ```

  These should be shown as `datasette-jellyfish` and `datasette-vega` since those are the names on PyPI.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/606/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTI5OTY0Njk=
title: Ways to improve fuzzy search speed on larger data sets?
state: closed
created_at: 2019-10-27T17:31:37Z
updated_at: 2019-11-07T03:38:10Z
closed_at: 2019-11-07T03:38:10Z
author_association: NONE
type: issue
state_reason: completed
body:
  I have an sqlite table with 16 million rows in it. Having read @simonw article "[Fast Autocomplete Search for Your Website](https://24ways.org/2018/fast-autocomplete-search-for-your-website/)" I was curious to try datasette to see what kind of query performance I could get out of it. In truth I don't need to do full text search since all I would like to do is give my users a way to search for the names of investors such as "Warren Buffet", or "Tim Cook" (who's names are in a single column).

  On the first search, Datasette takes over 20 seconds to return all records associated with `elon musk`:

  > ![image](https://user-images.githubusercontent.com/8431341/67638889-a86e1100-f8b7-11e9-9f7e-a9d13a42e988.png)

  > ![image](https://user-images.githubusercontent.com/8431341/67638825-ed457800-f8b6-11e9-94d1-b44f1a40ee8c.png)

  If I rerun the same search, it then takes almost 9 seconds:
  > ![image](https://user-images.githubusercontent.com/8431341/67638908-e4a17180-f8b7-11e9-9d00-748c80ef1f21.png)

  That's far to slow to implement an autocomplete feature. I could reduce the latency by making a special table of only unique investor names, thereby reducing the search space to less than a million rows (then I'd need to implement a way to add only new investor names to the table as I received new data.. about 4,000 rows a day). If I did that, I'm still concerned the new table wouldn't be lean enough to lookup investor names quickly. Plus, even if I can implement the autocomplete feature, I would still finally have to lookup records for that investors which would take between 8 - 20 seconds. 

  Are there any tricks for speeding this up?

  Here's my hardware:
  > ![image](https://user-images.githubusercontent.com/8431341/67638861-55945980-f8b7-11e9-96a8-ca76c7c68c5d.png)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/607/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTMwMDg5MzY=
title: Improve UI of "datasette publish cloudrun" to reduce chances of accidentally over-writing a service
state: closed
created_at: 2019-10-27T19:21:28Z
updated_at: 2019-11-08T02:51:36Z
closed_at: 2019-11-08T02:48:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The concept of a "service" in Cloud Run is crucial: if you deploy to the same service, you will over-write what you deployed there last!

  As such, I'd like to make service a required positional argument for `publish cloudrun`:

      datasette publish cloudrun my-service one.db two.db three.db

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/608/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTMwNzQ1MDE=
title: Command for importing mentions timeline
state: closed
created_at: 2019-10-28T03:14:27Z
updated_at: 2019-10-30T02:36:13Z
closed_at: 2019-10-30T02:20:47Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-mentions_timeline

  Almost identical to home-timeline #18 but it uses `https://api.twitter.com/1.1/statuses/mentions_timeline.json` instead.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTQ0NTkwNjI=
title: retweets-of-me command
state: closed
created_at: 2019-10-30T07:43:01Z
updated_at: 2019-11-03T01:12:58Z
closed_at: 2019-11-03T01:12:58Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets_of_me

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM0NDQ4MjU4
title: Update to latest black
state: closed
created_at: 2019-10-30T18:42:35Z
updated_at: 2019-10-30T18:49:01Z
closed_at: 2019-10-30T18:49:01Z
author_association: OWNER
pull_request: simonw/datasette/pulls/609
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/609/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTU2NTg4NjE=
title: Add indexes to followers table
state: closed
created_at: 2019-10-31T18:40:22Z
updated_at: 2019-11-09T20:15:42Z
closed_at: 2019-11-09T20:11:48Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  `select follower_id from following where followed_id = 12497` takes over a second for me at the moment.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTYzMTA2NzA=
title: Don't suggest array facet if column is only [] empty arrays
state: closed
created_at: 2019-11-01T19:42:02Z
updated_at: 2019-11-01T21:46:08Z
closed_at: 2019-11-01T21:46:08Z
author_association: OWNER
body: Follow on from #562
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/610/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTYzNzA4MjI=
title: Static assets no longer loading for installed plugins
state: closed
created_at: 2019-11-01T22:07:00Z
updated_at: 2019-11-01T22:15:55Z
closed_at: 2019-11-01T22:15:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Caused by fix I made in #606

  e.g. `/-/static-plugins/datasette_leaflet_geojson/datasette-leaflet-geojson.js` is a 404, but view-`/-/static-plugins/datasette-leaflet-geojson/datasette-leaflet-geojson.js` works correctly.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/611/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTY3NDg4NDk=
title: CSV export is broken for tables with null foreign keys
state: closed
created_at: 2019-11-02T22:52:47Z
updated_at: 2021-06-17T18:13:20Z
closed_at: 2019-11-02T23:12:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Following on from #406 - this CSV export appears to be broken:

  https://14da705.datasette.io/fixtures/foreign_key_references.csv?_labels=on&_size=max
  ```csv
  pk,foreign_key_with_label,foreign_key_with_label_label,foreign_key_with_no_label,foreign_key_with_no_label_label
  1,1,hello,1,1
  2,,
  ```
  That second row should have 5 values, but it only has 4.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/612/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM1OTgwMjQ2
title: stargazers command, refs #4
state: closed
created_at: 2019-11-03T00:37:36Z
updated_at: 2020-05-02T20:00:27Z
closed_at: 2020-05-02T20:00:26Z
author_association: MEMBER
pull_request: dogsheep/github-to-sqlite/pulls/8
body: Needs tests. Refs #4.
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/8/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTY3NjkyNzY=
title: Commands do not work without an auth.json file
state: closed
created_at: 2019-11-03T01:54:28Z
updated_at: 2019-11-11T05:30:48Z
closed_at: 2019-11-11T05:30:48Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  `auth.json` is meant to be optional. If it's not provided, the tool should make heavily rate-limited unauthenticated requests.

  ```
  $ github-to-sqlite repos .data/repos.db simonw
  Usage: github-to-sqlite repos [OPTIONS] DB_PATH [USERNAME]
  Try "github-to-sqlite repos --help" for help.

  Error: Invalid value for "-a" / "--auth": File "auth.json" does not exist.
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/9/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTY4NzQ3MzU=
title: Basic join support for table view
state: open
created_at: 2019-11-03T19:12:53Z
updated_at: 2019-11-03T19:14:01Z
author_association: OWNER
type: issue
body:
  I think it would be possible to support basic foreign key joins on the table page.

  The user could specify columns that should result in a join (from a set of suggestions similar to how facets work right now) and they could then be passed as `?_join=city_id`  arguments.

  This feature will make a lot of sense when combined with the ability to show / hide / customize columns, see #292

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/613/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTY5NTA3NDg=
title: Add "not in" filter - ?pk__notin=x,y,z
state: closed
created_at: 2019-11-04T04:07:17Z
updated_at: 2019-11-04T04:31:58Z
closed_at: 2019-11-04T04:12:00Z
author_association: OWNER
type: issue
state_reason: completed
body:
  We have a `__in` filter at the moment: https://latest.datasette.io/fixtures/facetable?pk__in=1,2,3

  Today I found myself needing the inverse, a `?pk__notin=` filter, which isn't currently supported.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/614/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTY5Njc2ODI=
title: Add this repos_starred view
state: closed
created_at: 2019-11-04T05:44:38Z
updated_at: 2020-05-02T16:37:36Z
closed_at: 2020-05-02T16:37:36Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```sql
  create view repos_starred as select
    stars.starred_at,
    users.login,
    repos.*
  from
    repos
    join stars on repos.id = stars.repo
    join users on repos.owner = users.id
  order by
    starred_at desc;
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTcyNDEwNDA=
title: ensure_index() method
state: closed
created_at: 2019-11-04T15:51:22Z
updated_at: 2019-11-04T16:20:36Z
closed_at: 2019-11-04T16:20:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```python
  db["table"].ensure_index(["col1", "col2"])
  ```

  This will do the following:
  - if the specified table or column does not exist, do nothing
  - if they exist and already have an index, do nothing
  - otherwise, create the index

  I want this for tools like [twitter-to-sqlite search](https://github.com/dogsheep/twitter-to-sqlite/blob/801c0c2daf17d8abce9dcb5d8d610410e7e25dbe/README.md#running-searches) where the `search_runs` table may or not have been created yet but, if it IS created, I want to put an index on the `hash` column.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/63/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTc0NTEyMzQ=
title: ?_col= and ?_nocol= support for toggling columns on table view
state: closed
created_at: 2019-11-04T22:55:41Z
updated_at: 2021-05-27T04:26:10Z
closed_at: 2021-05-27T04:17:44Z
author_association: OWNER
body: Split off from #292 (I guess this is a re-opening of #312).
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/615/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTg1MDYyNDI=
title: Datasette FTS detection bug
state: closed
created_at: 2019-11-06T14:25:47Z
updated_at: 2019-11-08T15:31:33Z
closed_at: 2019-11-08T02:06:56Z
author_association: NONE
type: issue
state_reason: completed
body:
  I'm having a trouble with datasette. I deployed EXACTLY the same project on two different apps on Heroku. Both have databases (not all) with FTS activated but only one detects and works fine. You can take a look here:
  With search: http://teste-templates.herokuapp.com/amazonia_protege/car
  Without search: http://bases.vortex.media/amazonia_protege/car
  ![teste](https://user-images.githubusercontent.com/49656826/68306310-11a80e00-0088-11ea-8d1c-db3bd3375518.jpg)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/616/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTg3MjUwNjQ=
title: `import` command fails on empty files
state: closed
created_at: 2019-11-06T20:34:26Z
updated_at: 2019-11-09T20:33:38Z
closed_at: 2019-11-09T19:36:36Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  If a file in the export is empty (in my case it was `account-suspensions.js`), `twitter-to-sqlite import` fails:

  ```
  $ twitter-to-sqlite import twitter.db ~/Downloads/twitter-2019-11-06-926f4f3be4b3b1fcb1aa387c40cd14f7c8aaf9bbcdb2d78ac14d9989add501bb.zip
  Traceback (most recent call last):
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/bin/twitter-to-sqlite", line 10, in <module>
      sys.exit(cli())
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py", line 627, in import_
      archive.import_from_file(db, filename, content)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/archive.py", line 224, in import_from_file
      db[table_name].upsert_all(rows, hash_id="pk")
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/sqlite_utils/db.py", line 1113, in upsert_all
      extracts=extracts,
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/sqlite_utils/db.py", line 980, in insert_all
      first_record = next(records)
  StopIteration
  ```

  This appears to be because `db.upsert_all` is called with no rows -- I think? 

  I hacked around this by modifying `import_from_file` to have an `if rows:` clause:

  ```
      for table, rows in to_insert.items():
          if rows:
              table_name = "archive_{}".format(table.replace("-", "_"))
              ...
  ```

  I'm happy to work up a real PR if that's the right approach, but I'm not sure it is.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTg3Mzk2OTc=
title: `followers` fails because `transform_user` is called twice
state: closed
created_at: 2019-11-06T20:44:52Z
updated_at: 2019-11-09T20:15:28Z
closed_at: 2019-11-09T19:55:52Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Trying to run `twitter-to-sqlite followers` errors out:

  ```
  Traceback (most recent call last):
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/bin/twitter-to-sqlite", line 10, in <module>
      sys.exit(cli())
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py", line 130, in followers
      go(bar.update)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py", line 116, in go
      utils.save_users(db, [profile])
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/utils.py", line 302, in save_users
      transform_user(user)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/utils.py", line 181, in transform_user
      user["created_at"] = parser.parse(user["created_at"])
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1374, in parse
      return DEFAULTPARSER.parse(timestr, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 646, in parse
      res, skipped_tokens = self._parse(timestr, **kwargs)
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 725, in _parse
      l = _timelex.split(timestr)         # Splits the timestr into tokens
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 207, in split
      return list(cls(s))
    File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 76, in __init__
      '{itype}'.format(itype=instream.__class__.__name__))
  TypeError: Parser must be a string or character stream, not datetime
  ```

  This appears to be because https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L111 calls `transform_user`, and then https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L116 calls `transform_user` again, which fails because the user is already transformed.

  I was able to work around this by commenting out https://github.com/dogsheep/twitter-to-sqlite/blob/master/twitter_to_sqlite/cli.py#L116. 

  Shall I work up a patch for that, or is there a better approach?

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM3ODQ3NTcz
title: test_insert_upsert_all_empty_list
state: closed
created_at: 2019-11-07T04:24:45Z
updated_at: 2019-11-07T04:32:38Z
closed_at: 2019-11-07T04:32:38Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/64
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/64/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTkwMzg5Nzk=
title: Failed to import workout points
state: closed
created_at: 2019-11-07T04:50:22Z
updated_at: 2019-11-08T01:18:37Z
closed_at: 2019-11-08T01:18:37Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I just ran the script and it failed to import any `workout_points`, though it did import `workouts`.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM3ODUzMzk0
title: Release 1.12.1
state: closed
created_at: 2019-11-07T04:51:29Z
updated_at: 2019-11-07T04:58:48Z
closed_at: 2019-11-07T04:58:47Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/65
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/65/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MTk2MTMxMTY=
title: Refactor TableView.data() method
state: closed
created_at: 2019-11-08T01:55:41Z
updated_at: 2021-12-18T01:41:47Z
closed_at: 2021-12-11T19:17:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This is by far the most complex piece of Datasette - the `TableView.data()` method is over 500 lines long and is increasingly getting in the way of cleanly implementing new features (e.g. #615 and #613).

  Need to break it up into smaller, cleaner pieces.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/617/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM4NjQ3Mzc4
title: Add parkrun-to-sqlite
state: closed
created_at: 2019-11-08T12:05:32Z
updated_at: 2020-10-12T00:35:16Z
closed_at: 2020-10-12T00:35:16Z
author_association: CONTRIBUTOR
pull_request: dogsheep/dogsheep.github.io/pulls/1
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA1MDczMDY=
title: Mechanism for seeing indexes on a specific table
state: closed
created_at: 2019-11-09T20:10:41Z
updated_at: 2019-11-10T01:40:05Z
closed_at: 2019-11-10T01:30:25Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The only way to see the indexes that apply to a specific table at the moment is to run the following SQL manually:
  ```sql
  select * from sqlite_master where type = 'index' and tbl_name=?
  ```
  For example:

  <img width="964" alt="f__select___from_sqlite_master_where_tbl_name____following__and_type__index_" src="https://user-images.githubusercontent.com/9599/68534478-d4db5180-02e9-11ea-9b4d-44dab2c314c9.png">

  It would be good if this list of indexes was displayed in a neater way on the table page.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/618/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA1MDg1MDI=
title: "friends" command (similar to "followers")
state: closed
created_at: 2019-11-09T20:20:20Z
updated_at: 2022-09-20T05:05:03Z
closed_at: 2020-02-07T07:03:28Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Current list of commands:
  ```
    followers          Save followers for specified user (defaults to...
    followers-ids      Populate followers table with IDs of account followers
    friends-ids        Populate followers table with IDs of account friends
  ```
  Obvious omission here is `friends`, which would be powered by `https://api.twitter.com/1.1/friends/list.json`: https://developer.twitter.com/en/docs/accounts-and-users/follow-search-get-users/api-reference/get-friends-list

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA1MjE4NDM=
title: Command to fetch releases
state: closed
created_at: 2019-11-09T22:23:30Z
updated_at: 2019-11-09T22:57:00Z
closed_at: 2019-11-09T22:57:00Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  https://developer.github.com/v3/repos/releases/#list-releases-for-a-repository

  `GET /repos/:owner/:repo/releases`

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/11/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA2NTU5ODM=
title: "Invalid SQL" page should let you edit the SQL
state: closed
created_at: 2019-11-10T20:54:12Z
updated_at: 2022-01-13T22:21:42Z
closed_at: 2021-06-02T04:15:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D

  <img src="https://user-images.githubusercontent.com/9599/68550604-1ab31b00-03b9-11ea-8a33-5dc339b8c5f5.jpeg" width="500">

  Would be useful if this page showed you the invalid SQL you entered so you can edit it and try again.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/619/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA2Njc3NzM=
title: Mechanism for indicating foreign key relationships in the table and query page URLs
state: open
created_at: 2019-11-10T22:26:27Z
updated_at: 2021-04-05T03:57:22Z
author_association: OWNER
type: issue
body:
  Datasette currently only inflates foreign keys (into names hyperlinks) if it detects them as foreign key constraints in the underlying database.

  It would be useful if you could specify additional "foreign keys" using both `metadata.json` and the querystring - similar time how you can pass `?_fts_table=x` https://datasette.readthedocs.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/620/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1}


node_id: MDU6SXNzdWU1MjA2ODE3MjU=
title: Syntax for ?_through= that works as a form field
state: open
created_at: 2019-11-11T00:19:03Z
updated_at: 2021-12-18T01:42:33Z
author_association: OWNER
type: issue
body:
  The current syntax for `?_through=` uses JSON to avoid any risk of confusion with table or column names that contain special characters.

  This means you can't target a form field at it.

  We should be able to support both - `?x.y.z=value` for tables and columns with "regular" names, falling back to the current JSON syntax for columns or tables that won't work with the key/value syntax.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/621/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA3MTUxODg=
title: Datasette should work with Python 3.8 (and drop compatibility with Python 3.5)
state: closed
created_at: 2019-11-11T03:12:36Z
updated_at: 2019-11-12T05:52:49Z
closed_at: 2019-11-12T05:09:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  See #595, #594, #404.

  The big thing holding me back from ditching Python 3.5 was glitch.com - but they now offer Python 3.7: https://support.glitch.com/t/can-you-upgrade-python-to-latest-version/7980/25?u=simonw

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/622/reactions", "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM5MjM2NjQ3
title: Test against Python 3.8 in Travis
state: closed
created_at: 2019-11-11T03:24:54Z
updated_at: 2019-11-11T03:45:35Z
closed_at: 2019-11-11T03:45:35Z
author_association: OWNER
pull_request: simonw/datasette/pulls/623
body: Needed for #622
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/623/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM5MjQ0ODg4
title: Bump pint to 0.9
state: closed
created_at: 2019-11-11T04:07:07Z
updated_at: 2019-11-11T04:19:02Z
closed_at: 2019-11-11T04:19:02Z
author_association: OWNER
pull_request: simonw/datasette/pulls/624
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/624/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA3NDA3NDE=
title: If you apply ?_facet_array=tags then &_facet=tags does nothing
state: closed
created_at: 2019-11-11T04:59:29Z
updated_at: 2022-01-13T22:26:58Z
closed_at: 2021-12-16T20:12:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Start here: https://v0-30-2.datasette.io/fixtures/facetable?_facet_array=tags

  <img width="969" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/68562011-c0d44480-03fc-11ea-8354-def0ba35d365.png">

  Note that `tags` is offered as a suggested facet. But if you click that you get this:

  https://v0-30-2.datasette.io/fixtures/facetable?_facet_array=tags&_facet=tags

  The `_facet=tags` is added to the URL and it's removed from the list of suggested tags... but the facet itself is not displayed:

  <img width="950" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/68562039-df3a4000-03fc-11ea-9778-81f7a931d608.png">

  The `_facet=tags` facet should look like this:

  <img width="946" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/68562067-f8db8780-03fc-11ea-8211-26281b7df9f9.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/625/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjA3NTY1NDY=
title: Add this view for seeing new releases
state: closed
created_at: 2019-11-11T06:00:12Z
updated_at: 2020-05-02T18:58:18Z
closed_at: 2020-05-02T18:58:17Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```sql
  CREATE VIEW recent_releases AS select
    json_object("label", repos.full_name, "href", repos.html_url) as repo,
    json_object(
      "href",
      releases.html_url,
      "label",
      releases.name
    ) as release,
    substr(releases.published_at, 0, 11) as date,
    releases.body as body_markdown,
    releases.published_at
  from
    releases
    join repos on repos.id = releases.repo
  order by
    releases.published_at desc
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/12/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjEyNzUyODE=
title: Set up a live demo Datasette instance
state: closed
created_at: 2019-11-12T01:27:02Z
updated_at: 2020-03-24T00:03:26Z
closed_at: 2020-03-24T00:03:25Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  I deployed https://github-to-sqlite-releases-j7hipcg4aq-uc.a.run.app/ by running this:
  ```
  #!/bin/bash
  # Fetch repos for simonw and dogsheep
  github-to-sqlite repos github.db simonw dogsheep -a auth.json

  # Fetch releases for the repos tagged 'datasette-io'
  sqlite-utils github.db "
  select full_name from repos where rowid in (
      select repos.rowid from repos, json_each(repos.topics) j
      where j.value = 'datasette-io'
  )" --csv --no-headers | while read repo;
      do github-to-sqlite releases \
              github.db $(echo $repo | tr -d '\r') \
              -a auth.json;
          sleep 2;
      done;
  ```
  And then deploying using this:
  ```
  $ datasette publish cloudrun github.db \
    --title "github-to-sqlite releases demo" \
    --about_url="https://github.com/simonw/github-to-sqlite" \
    --about='github-to-sqlite' \
    --install=datasette-render-markdown \
    --install=datasette-json-html \
    --service=github-to-sqlite-releases
  ```
  This should happen automatically for every release. I can run it once a day in Circle CI to keep the demo database up-to-date.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/13/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjEyODIwMTM=
title: Unit tests should fail under Python 3.8
state: closed
created_at: 2019-11-12T01:54:11Z
updated_at: 2019-11-12T04:31:26Z
closed_at: 2019-11-12T04:31:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The unit tests currently pass under Python 3.8.

  But... when you actually attempt to run Datasette you get an error:
  ```
  ~/Dropbox/Development/datasette $ venv-py3.8.0/bin/datasette --memory -p 8855
  Serve! files=() (immutables=()) on port 8855
  Traceback (most recent call last):
    File "venv-py3.8.0/bin/datasette", line 11, in <module>
      load_entry_point('datasette', 'console_scripts', 'datasette')()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/simonw/Dropbox/Development/datasette/datasette/cli.py", line 365, in serve
      uvicorn.run(ds.app(), host=host, port=port, log_level="info")
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/main.py", line 279, in run
      server.run()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/main.py", line 305, in run
      self.config.setup_event_loop()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/config.py", line 218, in setup_event_loop
      loop_setup()
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvicorn/loops/auto.py", line 3, in auto_loop_setup
      import uvloop
    File "/Users/simonw/Dropbox/Development/datasette/venv-py3.8.0/lib/python3.8/site-packages/uvloop/__init__.py", line 7, in <module>
      from .loop import Loop as __BaseLoop  # NOQA
    File "uvloop/includes/stdlib.pxi", line 114, in init uvloop.loop
  AttributeError: module 'sys' has no attribute 'set_coroutine_wrapper'
  ~/Dropbox/Development/datasette $ 
  ```
  If Datasette doesn't work under Python 3.8 the tests should fail.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/626/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM5NzIyNzkw
title: Support Python 3.8, stop supporting Python 3.5
state: closed
created_at: 2019-11-12T04:36:33Z
updated_at: 2020-04-05T10:23:58Z
closed_at: 2019-11-12T05:09:12Z
author_association: OWNER
pull_request: simonw/datasette/pulls/627
body: Refs #622
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/627/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjEzMjk3NzE=
title: Render jinja2 templates in async mode
state: closed
created_at: 2019-11-12T05:01:55Z
updated_at: 2019-11-14T23:28:09Z
closed_at: 2019-11-14T23:14:24Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I started playing with this in #404 and got good results but it didn't work in Python 3.5. As of #627 I don't support 3.5 any more so this can go ahead.

  Rendering templates in async mode will mean that template plugins can include async code... which opens the door to custom template functions that execute SQL queries!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/628/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjEzMzUzMzU=
title: "datasette publish" commands should deploy with Python 3.8
state: closed
created_at: 2019-11-12T05:22:31Z
updated_at: 2019-11-12T06:03:10Z
closed_at: 2019-11-12T06:03:10Z
author_association: OWNER
body: Now that we support 3.8 (#627) `datasette publish` should always deploy using Python 3.8.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/629/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzM5NzQyNDMy
title: Use python:3.8 base Docker image
state: closed
created_at: 2019-11-12T06:02:37Z
updated_at: 2019-11-12T06:03:10Z
closed_at: 2019-11-12T06:03:10Z
author_association: OWNER
pull_request: simonw/datasette/pulls/630
body: Closes #629
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/630/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjE4Njg4NjQ=
title: The ".upsert()" method is misnamed
state: closed
created_at: 2019-11-12T23:48:28Z
updated_at: 2019-12-31T01:30:21Z
closed_at: 2019-12-31T01:30:20Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This thread here is illuminating: https://stackoverflow.com/questions/3634984/insert-if-not-exists-else-update

  The term `UPSERT` in SQLite has a specific meaning as-of 3.24.0 (2018-06-04): https://www.sqlite.org/lang_UPSERT.html

  It means "behave as an UPDATE or a no-op if the INSERT would violate a uniqueness constraint". The syntax in 3.24.0+ looks like this (confusingly it does not use the term "upsert"):
  ```sql
  INSERT INTO phonebook(name,phonenumber) VALUES('Alice','704-555-1212')
    ON CONFLICT(name) DO UPDATE SET phonenumber=excluded.phonenumber
  ```
  Here's the problem: the `sqlite-utils` `.upsert()` and `.upsert_all()` methods don't do this. They use the following SQL:

  ```sql
  INSERT OR REPLACE INTO [{table}] ({columns}) VALUES {rows};
  ```

  If the record already exists, it will be entirely replaced by a new record - as opposed to updating any specified fields but leaving existing fields as they are (the behaviour of "upsert" in SQLite itself).

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/66/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzQwMjExMTQ5
title: bugfix issue 572
state: closed
created_at: 2019-11-13T02:46:50Z
updated_at: 2019-11-13T04:28:43Z
closed_at: 2019-11-13T04:28:42Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/631
body: closes bugfix issue #572
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/631/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjE5OTUwMzk=
title: Upgrade datasette publish Heroku runtime
state: closed
created_at: 2019-11-13T06:46:19Z
updated_at: 2019-11-13T16:44:07Z
closed_at: 2019-11-13T16:43:23Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  Python has released a security update! Please consider upgrading to python-3.6.9
  ```
  https://devcenter.heroku.com/articles/python-support#supported-runtimes shows 3.8.0 is now supported.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/632/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjIzMzQ3NzE=
state: closed
created_at: 2019-11-13T16:32:11Z
updated_at: 2020-04-28T20:37:50Z
closed_at: 2019-11-13T16:43:23Z
author_association: OWNER
type: issue
state_reason: completed
title:
  Publish to Heroku is broken: "WARNING:  You must pass the application as an import string to enable 'reload' or 'workers"

body:
  ```
  2019-11-13T16:27:59.821483+00:00 heroku[web.1]: Starting process with command `datasette serve --host 0.0.0.0 -i fixtures.db --cors --port 36817 --inspect-file inspect-data.json`
  2019-11-13T16:28:01.856471+00:00 heroku[web.1]: State changed from starting to crashed
  2019-11-13T16:28:01.750253+00:00 app[web.1]: Serve! files=() (immutables=('fixtures.db',)) on port 36817
  2019-11-13T16:28:01.771524+00:00 app[web.1]: WARNING:  You must pass the application as an import string to enable 'reload' or 'workers'.
  2019-11-13T16:28:01.837839+00:00 heroku[web.1]: Process exited with status 1
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/633/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjIzNTI1MjA=
title: Don't run tests twice when releasing a tag
state: closed
created_at: 2019-11-13T17:02:42Z
updated_at: 2020-09-15T20:37:58Z
closed_at: 2020-09-15T20:37:58Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Shipping a release currently runs the tests twice: https://travis-ci.org/simonw/datasette/builds/611463728

  It does a regular test run on Python 3.6/7/8 - then the "Release tagged version" step runs the tests again before publishing to PyPI! This second run is not necessary.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/634/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzQwNzQzMjIw
title: Use Jinja async mode
state: closed
created_at: 2019-11-14T01:20:57Z
updated_at: 2019-11-14T23:14:23Z
closed_at: 2019-11-14T23:14:23Z
author_association: OWNER
pull_request: simonw/datasette/pulls/635
body: Refs #628. Still needs documentation.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/635/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjUyNTQ5NzM=
title: rowid is not included in dropdown filter menus
state: closed
created_at: 2019-11-19T20:43:04Z
updated_at: 2019-11-19T23:01:17Z
closed_at: 2019-11-19T23:01:17Z
author_association: OWNER
type: issue
state_reason: completed
body:
  For `rowid` tables the `rowid` column isn't shown in the list of filter options:

  <img width="652" alt="md__md__53_805_rows_where_where_rowid___1060124_sorted_by_rowid_descending" src="https://user-images.githubusercontent.com/9599/69184590-00202680-0aca-11ea-8522-3a4690924b83.png">

  This also means if you link to e.g. `?rowid__gt=1060124` the resulting filter interface will be slightly broken: clicking the "apply" button again will lose your filter for example.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/636/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjU5OTMwMzQ=
title: Custom queries with 0 results should say "0 results"
state: closed
created_at: 2019-11-20T18:28:14Z
updated_at: 2019-11-23T06:17:23Z
closed_at: 2019-11-23T06:07:08Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Consider https://latest.datasette.io/fixtures/neighborhood_search?text=foop

  <img width="803" alt="fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_" src="https://user-images.githubusercontent.com/9599/69266652-6a939e00-0b80-11ea-8b08-960a05a8c8d0.png">

  It's currently not obvious that the query executed and returned 0 results.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/637/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MjY5MTMxMzM=
title: Don't suggest column for faceting if all values are 1
state: closed
created_at: 2019-11-22T00:14:22Z
updated_at: 2019-11-22T01:14:59Z
closed_at: 2019-11-22T00:57:49Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://www.niche-museums.com/museums/museums?_facet=wikipedia_url

  <img width="759" alt="museums__museums__42_rows" src="https://user-images.githubusercontent.com/9599/69387171-e58caf80-0c79-11ea-8b4e-cfe5861bc0ab.png">

  Challenge is how to do this efficiently, since suggested facet queries need to be lightning fast.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/638/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Mjc2NzA3OTk=
title: updating metadata.json without recreating the app
state: open
created_at: 2019-11-24T09:19:53Z
updated_at: 2019-11-30T06:08:50Z
author_association: NONE
type: issue
body:
  I've sucessfully "uploaded" an SQLite database (with a metadata.json file) to heroku using:

      $ datasette publish heroku so-sales.db -m metadata.json -n so-sales

  The question is: how can I modify the (small) metadata.json file without having to upload the (large) SQLite database.

  The directions on heroku indicate I should run:

      heroku git:clone -a so-sales

  But this just results in an empty directory with a warning:
  warning: You appear to have cloned an empty repository.

  I've been able to "clone" the heroku "app" using the command:

      $ heroku slugs:download -a so-sales

  but this is not a git repository....

  Ideally, it seems to me, there'd be an option of the `datasette` CLI to allow a file
  to be updated, or there'd be some way to create a local git "clone" of the app
  so that the heroku instructions for "Deploying with git" would apply.

  (p.s. I ran `datasette publish heroku -m metadata.json -n so-sales`
  in the hope that that would not cause the .db file to be wiped, but of course
  it was.)

  (p.p.s. Thanks for Datasette!)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/639/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Mjc3MTAwNTU=
title: Nicer error message for heroku publish name clash
state: open
created_at: 2019-11-24T14:57:07Z
updated_at: 2019-12-06T07:19:34Z
author_association: CONTRIBUTOR
type: issue
body:
  If you try to publish to Heroku using no set name (i.e. the default `datasette` name) and a project already exists under that name, you get a meaningful error report on the first line followed by Py error messages that drown it out:

  ```
  Creating datasette... !
   ▸    Name datasette is already taken
  Traceback (most recent call last):
    File "/usr/local/bin/datasette", line 10, in <module>
      sys.exit(cli())
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/NNNNN/Library/Python/3.7/lib/python/site-packages/datasette/publish/heroku.py", line 124, in heroku
      create_output = check_output(cmd).decode("utf8")
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 411, in check_output
      **kwargs).stdout
    File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 512, in run
      output=stdout, stderr=stderr)
  subprocess.CalledProcessError: Command '['heroku', 'apps:create', 'datasette', '--json']' returned non-zero exit status 1.
  ```

  It would be neater if:

  - the Py error message was caught;
  - the report suggested setting a project name using `-n` etc.

  It may also be useful to provide a command to list the current names that are being used, which I assume is available via a Heroku call?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/640/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Mjg0NDIxMjY=
title: Better documentation for --static option
state: closed
created_at: 2019-11-26T02:07:57Z
updated_at: 2019-11-26T03:30:02Z
closed_at: 2019-11-26T02:31:53Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This is misleading:

  https://github.com/simonw/datasette/blob/aca41618f8761f99c47c8ae8e81b07a6d4af4d7a/docs/datasette-serve-help.txt#L23

  The correct format is e.g. `static:static/`

  Also it's not mentioned in the regular documentation at all.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/641/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzQ2MjY0OTI2
title: Run tests against 3.5 too
state: closed
created_at: 2019-11-27T14:20:35Z
updated_at: 2019-12-31T01:29:44Z
closed_at: 2019-12-31T01:29:43Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/67
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/67/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Mjk0MjkyMTQ=
title: Provide a cookiecutter template for creating new plugins
state: closed
created_at: 2019-11-27T15:46:36Z
updated_at: 2020-06-20T03:20:33Z
closed_at: 2020-06-20T03:20:25Z
author_association: OWNER
body: See this conversation: https://twitter.com/psychemedia/status/1199707352540368896
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/642/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzA0NjgyMTI=
title: Set up some basic benchmarks as part of the unit tests
state: open
created_at: 2019-11-29T19:24:19Z
updated_at: 2019-11-29T19:24:19Z
author_association: OWNER
type: issue
body:
  https://pypi.org/project/pytest-benchmark/ looks great for this.

  Here's how to run it as a github action:  https://github.com/rhysd/github-action-benchmark/blob/master/examples/pytest/README.md

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/643/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzA0OTEwNzQ=
title: Command for importing events
state: open
created_at: 2019-11-29T21:28:58Z
updated_at: 2020-04-14T19:38:34Z
author_association: MEMBER
type: issue
body:
  Eg from https://api.github.com/users/simonw/events

  Docs here: https://developer.github.com/v3/activity/events/#list-events-performed-by-a-user

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/14/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzQ3MTc5MDgx
title: Validate metadata json on startup
state: closed
created_at: 2019-11-30T00:32:15Z
updated_at: 2021-07-28T17:58:45Z
closed_at: 2021-07-28T17:58:45Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/644
type: pull
body:
  This PR adds a sanity check which builds up a marshmallow schema on-the-fly based on the structure of the database(s) on startup and then validates the metadata json against it.

  In case of invalid data, this will raise with a descriptive error e.g:

  ```
  marshmallow.exceptions.ValidationError: {'databases': {'fixtures': {'tables': {'not_a_table': ['Unknown field.']}}}}
  ```

  Closes #260

  ---

  This was intended to be fairly self-contained, but then while I was working on it, I hit some problems getting the tests to pass in the context of the test suite as a whole. My tests passed in isolation, but then failed while doing a full test suite run. That's when the worms started coming out of the can :bug: After some sleuthing, it turned out this was essentially the result of several issues intersecting:

  * There are certain events in the application lifecycle where the metadata schema can be modified after it is loaded e.g: https://github.com/simonw/datasette/blob/a562f2965552fb2dbbbd74df245c9965ee23d886/datasette/app.py#L299-L320 This means that sometimes what goes in isn't always exactly what comes out when you call `/-/metadata`.
  * Because the test fixtures use session scope for performance reasons if one unit test performs an action which mutates the metadata, that can impact on other unit tests which run after it using the same fixture.
  * Because the `self._metadata` property was being set with a simple assignment `self._metadata = metadata`, that created an object reference to the test fixture data, so operating on `self._metadata` was actually modifying the test fixture `METADATA` meaning that depending on when it was loaded in the test suite lifecycle, `METADATA` had different content, which was somewhat unexpected.

  As such, I've added some band-aids in 3552024 and 6859fd8:
  * Switching the metadata object to a `deepcopy` of the input prevents us directly mutating the input fixture.
  * I've switched some of the tests to use a fixture with function scope instead of session scope so we're working on a clean copy that hasn't been mutated by other tests where necessary but keeping session scope in most cases for performance.
  * I haven't really addressed the fact that sometimes the metadata object gets mutated in place, so the object that is served from `/-/metadata` isn't necessarily always exactly the same as the file you fed into it on init. I'm not sure how much of a problem that is. The way the tests were written makes me think it was unexpected, but getting into it feels like too much scope creep for this PR so its probably best addressed as another issue.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/644/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzA2NTM2MzM=
title: Mechanism for register_output_renderer to suggest extension or not
state: closed
created_at: 2019-12-01T01:26:27Z
updated_at: 2020-05-28T02:22:18Z
closed_at: 2020-05-28T02:22:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  [datasette-atom](https://github.com/simonw/datasette-atom) only works if the user constructs a SQL query with specific output columns (`atom_id` ,`atom_updated` etc).

  It would be good if the `.atom` link wasn't shown on the query/table page unless those columns were present. Right now you get a link which results in a 400 error:

  <img width="622" alt="browse__museums__51_rows" src="https://user-images.githubusercontent.com/9599/69908058-81af6880-1396-11ea-99d7-19297fa1bb9b.png">

  See also #581.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/645/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzE1MDIzNjU=
title: Make database level information from metadata.json available in the index.html template
state: open
created_at: 2019-12-02T19:55:10Z
updated_at: 2022-03-15T20:50:34Z
author_association: NONE
type: issue
body:
  Did a search on the issues here and didn't find anything related to what I want.

  I want to have information that is on the database level of the JSON like title, source and source_url, and use it on the index page.

  I tried some small tweaks on the python and html files, but failed to get that result.

  Is there a way? Thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/646/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzE1ODM2NTg=
title: Add support for porter stemming in FTS
state: closed
created_at: 2019-12-02T22:35:52Z
updated_at: 2020-09-20T04:25:53Z
closed_at: 2020-09-20T04:25:47Z
author_association: OWNER
body: FTS5 can have porter stemming enabled.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/68/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzE3NTU5NTk=
title: Move hashed URL mode out to a plugin
state: closed
created_at: 2019-12-03T06:29:03Z
updated_at: 2022-03-19T11:56:05Z
closed_at: 2022-03-15T23:13:06Z
author_association: OWNER
type: issue
state_reason: completed
body:
  They used to be the default until #418. Since making them optional I haven't felt the need to use them even once.

  That suggests to me that they should be removed. I think their effect could be entirely handled by an ASGI wrapping plugin.

  https://datasette.readthedocs.io/en/0.32/performance.html#hashed-url-mode

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/647/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzQ0OTI1MDE=
title: Mechanism for adding arbitrary pages like /about
state: closed
created_at: 2019-12-08T04:55:19Z
updated_at: 2020-05-07T15:21:19Z
closed_at: 2020-04-26T18:46:45Z
author_association: OWNER
type: issue
state_reason: completed
body:
  For www.niche-museums.com I solved this by creating an empty `about.db` database file - see https://simonwillison.net/2019/Nov/25/niche-museums/

  I want a neater mechanism for this.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/648/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzQ1MDcxNDI=
title: Feature request: enable extensions loading
state: closed
created_at: 2019-12-08T08:06:25Z
updated_at: 2022-02-05T00:04:25Z
closed_at: 2020-10-16T18:42:49Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi, it would be great to add a parameter that enables the load of a sqlite extension you need.

  Something like "-ext modspatialite".

  In this way your great tool would be even more comfortable and powerful.


  Thank you very much

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/69/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzQ1MzA5NzM=
title: Reduce table counts on index page with many databases
state: closed
created_at: 2019-12-08T11:56:37Z
updated_at: 2020-02-29T01:08:29Z
closed_at: 2020-02-29T01:08:29Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Since #467 the index page has attempted to optimistically count times.

  My personal Dogsheep has enough connected databases and tables that the page can still take way too long to load - sometimes more than twenty seconds.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/649/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzQ2Mjk2MzE=
title: Add a glossary to the documentation
state: open
created_at: 2019-12-09T00:23:45Z
updated_at: 2022-01-13T22:04:56Z
author_association: OWNER
type: issue
body:
  Call it `glossary.rst` - it can use a definition list something like this:
  ```rst
  .. _glossary:

  Glossary
  ========

  Term
    A definition of the term.

  Another term
    Another definition.
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/650/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1MzkyMDQ0MzI=
title: Implement ON DELETE and ON UPDATE actions for foreign keys
state: open
created_at: 2019-12-17T17:19:10Z
updated_at: 2020-02-27T04:18:53Z
author_association: NONE
type: issue
body:
  Hi! I did not find any mention on the library about ON DELETE and ON UPDATE actions for foreign keys. Are those expected to be implemented? If not, it would be a nice thing to include!

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/70/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Mzk1OTAxNDg=
title: fts5 syntax error when using punctuation
state: closed
created_at: 2019-12-18T10:25:35Z
updated_at: 2021-07-14T19:26:06Z
closed_at: 2019-12-30T06:42:55Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon

  I get a syntax error when using punctuation or special characters in a fulltext search (using fts5). I created the virtual table using sqlite-utils' "enable-fts"-command.

  The same error appears on Niche Museums [https://www.niche-museums.com/browse/search?q=park.](https://www.niche-museums.com/browse/search?q=park.), but works fine in most of your other datasette-examples, e.g. register-of-members-interests [https://register-of-members-interests.datasettes.com/regmem-98dc8b7/items?_search=mins.](https://register-of-members-interests.datasettes.com/regmem-98dc8b7/items?_search=mins.)

  What am I doing wrong? Many thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/651/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzU0ODY5Mzkx
state: closed
created_at: 2019-12-18T23:37:16Z
updated_at: 2020-03-24T22:14:50Z
closed_at: 2020-03-24T22:14:50Z
author_association: NONE
pull_request: simonw/datasette/pulls/652
type: pull
title:
  Quick (and uninformed and perhaps misguided) attempt to add a <base> url for hosting datasette at a particular host/URI

body:
  As usual, I don't really know what I'm doing...  so this is just a suggested approach. I've not written tests, I've not run the tests, I don't know if I've missed some absolute URLs that would need to have the leading slash dropped.

  BUT, I tested it with `--config base_url:http://127.0.0.1:8001/` on the command line and from what little I know about datasette it's at least working in some obvious cases.

  My changes are based on what I saw in https://github.com/simonw/datasette/commit/8da2db4b71096b19e7a9ef1929369b8483d448bf (thanks!)

  I'm happy to be more thorough on this if you think it's worth pursuing.

  Fixes #394  (he said, optimistically).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/652/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDEyNzQ2ODE=
title: Add linkedin-to-sqlite
state: open
created_at: 2019-12-21T03:13:40Z
updated_at: 2019-12-21T03:13:40Z
author_association: NONE
type: issue
body:
  There is an API available. https://developer.linkedin.com/docs/rest-api#

  At the minimum, I would think contact list and messages would be of interest.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/2/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzU2MDA0MjQy
title: allow leading comments in SQL input field
state: closed
created_at: 2019-12-21T14:19:52Z
updated_at: 2020-02-05T02:35:41Z
closed_at: 2020-02-05T02:13:25Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/653
type: pull
body:
  this changes the SQL validation to allow for lines that are commented out

  my main use case for this is that I like to write a succession of queries when trying to solve a problem.
  In most native SQL clients there is a key binding that will run just the current highlighted query or the program is smart enough to run just the query that the cursor is in if it's properly delimited with a ';'.
  Typically my workflow will start with a single simple query and I'll copy/paste it to a new query below when I want to make big changes while debugging.  This makes it easy to go back to a working version above when the query doesn't work.
  Since datasette sends the whole query to the DB I have to comment out the older queries by prefixing each line with `--`.  This gets caught by the validators when I use my typical strategy of copy/pasting each successive query below the last one.   
  so this is just a simple fix to allow for a query to be sent to the DB with leading comments.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/653/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDE0Njc1OTA=
title: Template debug mode that outputs template context
state: closed
created_at: 2019-12-22T15:51:25Z
updated_at: 2019-12-22T16:13:11Z
closed_at: 2019-12-22T16:04:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It would make writing templates (including custom templates) easier if there was an option to dump out the full template context - maybe `?_context=1`

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/654/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDI1NTMzNTA=
title: Copy and paste doesn't work reliably on iPhone for SQL editor
state: closed
created_at: 2019-12-26T13:15:10Z
updated_at: 2020-09-30T20:36:12Z
closed_at: 2020-08-30T17:51:40Z
author_association: OWNER
body: I'm having a lot of trouble copying and pasting from the codemirror editor on my iPhone.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/655/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDI4MTQ3NTY=
title: Tests are failing due to missing FTS5
state: closed
created_at: 2019-12-27T09:41:16Z
updated_at: 2019-12-27T09:49:37Z
closed_at: 2019-12-27T09:49:37Z
author_association: OWNER
type: issue
state_reason: completed
body:
  https://travis-ci.com/simonw/sqlite-utils/jobs/268436167

  This is a recent change: 2 months ago they worked fine.

  I'm not sure what changed here. Maybe something to do with https://launchpad.net/~jonathonf/+archive/ubuntu/backports ?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/71/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzU3NjQwMTg2
title: don't break if source is missing
state: closed
created_at: 2019-12-29T10:46:47Z
updated_at: 2020-03-28T02:28:11Z
closed_at: 2020-03-28T02:28:11Z
author_association: CONTRIBUTOR
pull_request: dogsheep/swarm-to-sqlite/pulls/6
body: broke for me. very old checkins in 2010 had no source set.
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzU3OTc0MzI2
title: Add todoist-to-sqlite
state: closed
created_at: 2019-12-30T04:02:59Z
updated_at: 2020-10-12T00:35:58Z
closed_at: 2020-10-12T00:35:57Z
author_association: CONTRIBUTOR
pull_request: dogsheep/dogsheep.github.io/pulls/3
type: pull
body:
  Really enjoying getting into the dogsheep/datasette ecosystem. I made a downloader for Todoist, and I think/hope others might find this useful

reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/3/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzU3OTkyNTg4
title: Fixed implementation of upsert
state: closed
created_at: 2019-12-30T05:08:05Z
updated_at: 2019-12-30T05:29:24Z
closed_at: 2019-12-30T05:29:24Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/72
body: Refs #66
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/72/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDQ1NzEwOTI=
title: Assets table with downloads
state: closed
created_at: 2020-01-02T13:05:28Z
updated_at: 2020-03-28T12:17:01Z
closed_at: 2020-03-23T19:17:32Z
author_association: NONE
type: issue
state_reason: completed
body:
  The `releases` command extracts the releases table, but data about the individual assets are locked up in the JSON document in the `assets` field. My main interest is in individual and aggregate download counts. I was wondering if creating a new table with a record per asset may be useful?
  If so I'm happy to send a PR when I get a moment. Do you have opinions about that simply being part of the `releases` command or would you prefer a separate command as well?

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/15/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDU0MDc5MTY=
title: upsert_all() throws issue when upserting to empty table
state: closed
created_at: 2020-01-05T11:58:57Z
updated_at: 2020-01-31T14:21:09Z
closed_at: 2020-01-05T17:20:18Z
author_association: NONE
type: issue
state_reason: completed
body:
  If I try to add a list of `dict`s to an empty table using `upsert_all`, I get an error:

  ```python
  import sqlite3
  from sqlite_utils import Database
  import pandas as pd

  conx = sqlite3.connect(':memory')
  cx = conx.cursor()
  cx.executescript('CREATE TABLE "test" ("Col1" TEXT);')

  q="SELECT * FROM test;"
  pd.read_sql(q, conx) #shows empty table

  db = Database(conx)
  db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])

  ---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  <ipython-input-74-8c26d93d7587> in <module>
        1 db = Database(conx)
  ----> 2 db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])

  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)
     1157             alter=alter,
     1158             extracts=extracts,
  -> 1159             upsert=True,
     1160         )
     1161 

  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)
     1040                     sql = "INSERT OR IGNORE INTO [{table}]({pks}) VALUES({pk_placeholders});".format(
     1041                         table=self.name,
  -> 1042                         pks=", ".join(["[{}]".format(p) for p in pks]),
     1043                         pk_placeholders=", ".join(["?" for p in pks]),
     1044                     )

  TypeError: 'NoneType' object is not iterable

  ```

  A hacky workaround in use is:

  ```python
  try:
    db['test'].upsert_all([{'Col1':'a'},{'Col1':'b'}])
  except:
    db['test'].insert_all([{'Col1':'a'},{'Col1':'b'}])
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/73/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDYwNTExODE=
title: Exception running first command: IndexError: list index out of range
state: closed
created_at: 2020-01-07T03:01:58Z
updated_at: 2020-04-14T18:37:21Z
closed_at: 2020-04-14T18:37:21Z
author_association: NONE
type: issue
state_reason: completed
body:
  Exception running first command without an existing db or auth.

  ```py
  > mkdir ~/.github/coala
  > /usr/bin/github-to-sqlite repos ~/.github/coala coala
  Traceback (most recent call last):
    File "/usr/bin/github-to-sqlite", line 11, in <module>
      load_entry_point('github-to-sqlite==0.6', 'console_scripts', 'github-to-sqlite')()
    File "/usr/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/usr/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/usr/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/usr/lib/python3.7/site-packages/github_to_sqlite/cli.py", line 163, in repos
      utils.save_repo(db, repo)
    File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 120, in save_repo
      to_save["owner"] = save_user(db, to_save["owner"])
    File "/usr/lib/python3.7/site-packages/github_to_sqlite/utils.py", line 61, in save_user
      return db["users"].upsert(to_save, pk="id", alter=True).last_pk
    File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1135, in upsert
      extracts=extracts,
    File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1162, in upsert_all
      upsert=True,
    File "/usr/lib/python3.7/site-packages/sqlite_utils/db.py", line 1105, in insert_all
      row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
  IndexError: list index out of range
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/16/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDYwNzM5ODA=
title: Test failures on openSUSE 15.1: AssertionError: Explicit other_table and other_column
state: open
created_at: 2020-01-07T04:35:50Z
updated_at: 2020-01-12T07:21:17Z
author_association: CONTRIBUTOR
type: issue
body:
  openSUSE 15.1 is using python 3.6.5 and click-7.0 , however it has test failures while openSUSE Tumbleweed on py37 passes.

  Most fail on the cli exit code like
  ```py
  [   74s] =================================== FAILURES ===================================
  [   74s] _________________________________ test_tables __________________________________
  [   74s] 
  [   74s] db_path = '/tmp/pytest-of-abuild/pytest-0/test_tables0/test.db'
  [   74s] 
  [   74s]     def test_tables(db_path):
  [   74s]         result = CliRunner().invoke(cli.cli, ["tables", db_path])
  [   74s] >       assert '[{"table": "Gosh"},\n {"table": "Gosh2"}]' == result.output.strip()
  [   74s] E       assert '[{"table": "...e": "Gosh2"}]' == ''
  [   74s] E         - [{"table": "Gosh"},
  [   74s] E         -  {"table": "Gosh2"}]
  [   74s] 
  [   74s] tests/test_cli.py:28: AssertionError
  ```

  packaging project at https://build.opensuse.org/package/show/home:jayvdb:py-new/python-sqlite-utils

  I'll keep digging into this after I have github-to-sqlite working on Tumbleweed, as I'll need openSUSE Leap 15.1 working before I can submit this into the main python repo.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/74/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzU5ODIyNzcz
title: Explicitly include tests and docs in sdist
state: closed
created_at: 2020-01-07T04:53:20Z
updated_at: 2020-01-31T00:21:27Z
closed_at: 2020-01-31T00:21:27Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/75
body: Also exclude 'tests' from runtime installation.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/75/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDY5NjEzNTc=
title: Display of the column definitions
state: closed
created_at: 2020-01-08T16:16:53Z
updated_at: 2020-01-20T14:17:11Z
closed_at: 2020-01-20T14:14:33Z
author_association: CONTRIBUTOR
type: issue
state_reason: completed
body:
  Hello,
  Is the nice display of headers and definitions at the top of https://fivethirtyeight.datasettes.com/fivethirtyeight-ac35616/antiquities-act%2Factions_under_antiquities_act is configured in the metadata.json file ?
  Thank you,

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/656/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDg1OTEwODk=
title: Allow creation of virtual tables at startup
state: open
created_at: 2020-01-12T16:10:55Z
updated_at: 2021-01-15T20:24:35Z
author_association: NONE
type: issue
body:
  Hi,
   
  I've been experimenting with SQLite reading from huge datasets using this excellent Parquet extension from @cldellow.
  https://cldellow.com/2018/06/22/sqlite-parquet-vtable.html
  https://github.com/cldellow/sqlite-parquet-vtable

  This works really well, but I was keen to see if I could combine datasette with this. Having previously experimented with the spatialite extension I knew that datasette supports loading extensions in the underlying sqlite instance.  However I hit a blocker as the current design only allows SELECT statements to be executed and so I am unable to execute the crucial 

  CREATE VIRTUAL TABLE .........

  command that is required to load the data from the parquet file into the table.

  It seems like this would be a simple-ish change, but I don't know enough about the architecture of datasette to start implementing this myself?  Could this be done as a datasette plugin?  or would this require more fundamental changes at initialisation time?

  My thoughts are that something at init time could detect that the user was loading a *.parquet file and then switch to a mode were it loads that via the "CREATE VIRTUAL TABLE..." rather than loading the *.db file in the default case??

  I'm happy to contribute code and testing, I just need some pointers on the best approach.

  Thanks
  Darren

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/657/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NDkyODczMTA=
title: order_by mechanism
state: closed
created_at: 2020-01-14T02:06:03Z
updated_at: 2020-04-16T06:23:29Z
closed_at: 2020-04-16T03:13:06Z
author_association: NONE
type: issue
state_reason: completed
body:
  In some cases, I want to iterate rows in a table  with `ORDER BY` clause. It would be nice to have a `rows_order_by` function similar to `rows_where`.
  In a more general case, `rows_filter` function might be added to allow more customized filtering to iterate rows.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/76/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTAyOTM3NzA=
title: How do I use the app.css as style sheet?
state: open
created_at: 2020-01-15T16:27:57Z
updated_at: 2020-02-07T00:29:50Z
author_association: NONE
type: issue
body:
  Simon,

  I'm trying to use the app.css (in static folder) as style sheet but the datasette on Heroku simply ignore it! I read everything about customization here and on readthedocs but still can't.

  Is this possible?

  Thanks!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/658/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTE4MzQ4NDI=
title: README information is obscured by feature history
state: closed
created_at: 2020-01-18T22:34:51Z
updated_at: 2020-12-10T23:28:51Z
closed_at: 2020-12-10T23:28:51Z
author_association: NONE
type: issue
state_reason: completed
body:
  While it's sometimes valuable to know how a project has developed, there is usually little justification for including this information in the README, and certainly not immediately after other key information such as "what does this package do, and who might want to use it?"

  Might I recommend that the feature history is migrated to an Appendix in the documentation?

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/659/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzY1MjE4Mzkx
title: gcloud run is now GA, s/beta//
state: closed
created_at: 2020-01-21T10:08:38Z
updated_at: 2020-01-22T03:41:09Z
closed_at: 2020-01-21T23:28:12Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/660
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/660/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTU4MzI1ODU=
title: --port option to expose a port other than 8001 in "datasette package"
state: closed
created_at: 2020-01-27T21:05:56Z
updated_at: 2020-01-30T04:17:52Z
closed_at: 2020-01-29T22:46:45Z
author_association: NONE
type: issue
state_reason: completed
body:
  I see  how to alter the port using `datasette serve -p XXX` per the docs. However, I'm packaging up to server the container on AppEngine flexible, which [requires](https://cloud.google.com/appengine/docs/flexible/custom-runtimes/build#listening_to_port_8080)  that the container is serving traffic on port 8080.

  https://github.com/simonw/datasette/blob/7950105c278b140e6cb665c68b59df219870f9bc/Dockerfile#L41

  Is there a way to inject a non-default port into the Dockerfile, or should I just do something like `sed` to replace 8001 with 8080 after `dataset package` has done it's thing? Thanks for the advice.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/661/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTY4MTQ4NzY=
title: Escape_fts5_query-hookimplementation does not work with queries to standard tables
state: closed
created_at: 2020-01-29T11:56:03Z
updated_at: 2020-01-30T00:30:20Z
closed_at: 2020-01-30T00:30:19Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon

  Thank you for adding the escape_function, but it does not work on my datasette-installation (0.33). I've added the following file to my datasette-dir: /plugins/sql_functions.py:

  `from datasette import hookimpl

  def escape_fts_query(query):
      bits = query.split()
      return ' '.join('"{}"'.format(bit.replace('"', '')) for bit in bits)

  @hookimpl
  def prepare_connection(conn):
      conn.create_function("escape_fts_query", 1, escape_fts_query)`

  It has no effect on the standard queries to the tables though, as they still produce errors when including any characters like '-', '/', '+' or '?'

  Does the function only work when using costum queries, where I can include the escape_fts-function explicitly in the sql-query?

  PS. I'm calling datasette with --plugins=plugins, and my other plugins work just fine.
  PPS. The fts5 virtual table is created with 'sqlite3' like so:

  `CREATE VIRTUAL TABLE "cases_fts" USING FTS5(
    title,
    subtitle,
    resume,
    suggestion,
    presentation,
    detail = full,
    content_rowid = 'id',
    content = 'cases',
    tokenize='unicode61', 'remove_diacritics 2', 'tokenchars "-_"'
  );`

  Thanks!

  _Originally posted by @clausjuhl in https://github.com/simonw/datasette/issues/651#issuecomment-579675357_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/662/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzY4NzM0NTAw
title: -p argument for datasette package, plus tests - refs #661
state: closed
created_at: 2020-01-29T19:47:50Z
updated_at: 2020-01-29T22:46:43Z
closed_at: 2020-01-29T22:46:43Z
author_association: OWNER
pull_request: simonw/datasette/pulls/663
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/663/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTc4MjUwMzI=
title: Ability to insert data that is transformed by a SQL function
state: closed
created_at: 2020-01-30T23:45:55Z
updated_at: 2022-02-05T00:04:25Z
closed_at: 2020-01-31T00:24:32Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I want to be able to run the equivalent of this SQL insert:
  ```python
  # Convert to "Well Known Text" format
  wkt = shape(geojson['geometry']).wkt
  # Insert and commit the record
  conn.execute("INSERT INTO places (id, name, geom) VALUES(null, ?, GeomFromText(?, 4326))", (
     "Wales", wkt
  ))
  conn.commit()
  ```
  From the Datasette SpatiaLite docs: https://datasette.readthedocs.io/en/stable/spatialite.html

  To do this, I need a way of telling `sqlite-utils` that a specific column should be wrapped in `GeomFromText(?, 4326)`.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/77/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzY5MzQ4MDg0
title: New conversions= feature, refs #77
state: closed
created_at: 2020-01-31T00:02:33Z
updated_at: 2020-09-22T07:48:29Z
closed_at: 2020-01-31T00:24:31Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/78
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/78/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTc4NDIyNDU=
title: Helper methods for working with SpatiaLite
state: closed
created_at: 2020-01-31T00:39:19Z
updated_at: 2022-02-05T00:04:25Z
closed_at: 2022-02-04T05:55:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  As demonstrated by this piece of documentation, using SpatiaLite with sqlite-utils requires a fair bit of boilerplate:
  https://github.com/simonw/sqlite-utils/blob/f7289174e66ae4d91d57de94bbd9d09fabf7aff4/docs/python-api.rst#L880-L909

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/79/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzY5Mzk0MDQz
title: on_create mechanism for after table creation
state: closed
created_at: 2020-01-31T03:38:48Z
updated_at: 2020-01-31T05:08:04Z
closed_at: 2020-01-31T05:08:04Z
author_association: OWNER
pull_request: simonw/sqlite-utils/pulls/80
type: pull
body:
  I need this for `geojson-to-sqlite`, in particular https://github.com/simonw/geojson-to-sqlite/issues/6

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/80/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTg2MDAyNzQ=
title: Remove .detect_column_types() from table, make it a documented API
state: closed
created_at: 2020-02-01T21:25:54Z
updated_at: 2020-02-01T21:55:35Z
closed_at: 2020-02-01T21:55:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I used it in `geojson-to-sqlite` here: https://github.com/simonw/geojson-to-sqlite/blob/f10e44264712dd59ae7dfa2e6fd5a904b682fb33/geojson_to_sqlite/utils.py#L45-L50

  It would make more sense for this method to live on the Database rather than the Table - or even to exist as a separate utility method entirely.

  Then it should be documented.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/81/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzcwMDI0Njk3
title: Add beeminder-to-sqlite
state: closed
created_at: 2020-02-02T15:51:36Z
updated_at: 2020-10-12T00:36:16Z
closed_at: 2020-10-12T00:36:16Z
author_association: CONTRIBUTOR
pull_request: dogsheep/dogsheep.github.io/pulls/4
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/dogsheep.github.io/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTkxOTc3NDU=
title: Tutorial command no longer works
state: closed
created_at: 2020-02-03T16:36:11Z
updated_at: 2020-02-27T04:16:43Z
closed_at: 2020-02-27T04:16:30Z
author_association: NONE
type: issue
state_reason: completed
body:
  Issue with command on [tutorial](https://simonwillison.net/2019/Feb/25/sqlite-utils/) on Simon's site.

  The following command no longer works, and breaks with the previous too many variables error: #50

  ``` cmd
  > curl "https://data.nasa.gov/resource/y77d-th95.json" | \
      sqlite-utils insert meteorites.db meteorites - --pk=id
  ```

  Output:
  ``` cmd
  Traceback (most recent call last):
    File "continuum\miniconda3\envs\main\lib\runpy.py", line 193, in _run_module_as_main
      "__main__", mod_spec)
    File "continuum\miniconda3\envs\main\lib\runpy.py", line 85, in _run_code
      exec(code, run_globals)
    File "Continuum\miniconda3\envs\main\Scripts\sqlite-utils.exe\__main__.py", line 9, in <module>
    File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 717, in main
      rv = self.invoke(ctx)
    File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "continuum\miniconda3\envs\main\lib\site-packages\click\core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "continuum\miniconda3\envs\main\lib\site-packages\sqlite_utils\cli.py", line 434, in insert
      default=default,
    File "continuum\miniconda3\envs\main\lib\site-packages\sqlite_utils\cli.py", line 384, in insert_upsert_implementation
      docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs
    File "continuum\miniconda3\envs\main\lib\site-packages\sqlite_utils\db.py", line 1081, in insert_all
      result = self.db.conn.execute(query, params)
  sqlite3.OperationalError: too many SQL variables
  ```

  My thought is that maybe the dataset grew over the last few years and so didn't run into this issue before.

  No error when I reduce the count of entries to 83. Once the number of entries hits 84 the command fails.

  // This passes
  ``` cmd
  type meteorite_83.txt | sqlite-utils insert meteorites.db meteorites - --pk=id
  ```

  // But this fails
  ``` cmd
  type meteorite_84.txt | sqlite-utils insert meteorites.db meteorites - --pk=id
  ```

  A potential fix might be to chunk the incoming data? I can work on a PR if pointed in right direction.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/82/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTkzNzQ0MTA=
title: Make db["table"].exists a documented API
state: closed
created_at: 2020-02-03T22:31:44Z
updated_at: 2020-02-08T23:58:35Z
closed_at: 2020-02-08T23:56:23Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now it's a static thing which might get out-of-sync with the database. It should probably be a live check. Maybe call it `.exists()` instead?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/83/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzcwNjc1MDA3
title: Datasette.render_template() method
state: closed
created_at: 2020-02-04T06:53:59Z
updated_at: 2020-02-04T20:26:18Z
closed_at: 2020-02-04T20:26:18Z
author_association: OWNER
pull_request: simonw/datasette/pulls/664
body: Refs #577
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/664/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NTk5NjQxNDk=
title: Introduce a SQL statement parser in Python
state: open
created_at: 2020-02-04T20:36:05Z
updated_at: 2020-02-04T20:36:48Z
author_association: OWNER
body: #254 and #653 are both examples of problems that could be solved using a real SQL parser in Python.
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/665/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjE0NTQwNzE=
title: Documentation for " favorites" command
state: closed
created_at: 2020-02-07T06:50:11Z
updated_at: 2020-02-07T06:59:10Z
closed_at: 2020-02-07T06:59:10Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  It looks like I forgot to document this one in the README.

  https://github.com/dogsheep/twitter-to-sqlite/blob/6ebd482619bd94180e54bb7b56549c413077d329/twitter_to_sqlite/cli.py#L183-L194

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/32/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjE0NjAyNzQ=
title: .upsert() with hash_id throws error
state: closed
created_at: 2020-02-07T07:08:19Z
updated_at: 2020-02-07T07:17:11Z
closed_at: 2020-02-07T07:17:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```python
  db[table_name].upsert_all(rows, hash_id="pk")
  ```
  This throws an error: `PrimaryKeyRequired('upsert() requires a pk')`

  The problem is, if you try this:

  ```python
  db[table_name].upsert_all(rows, hash_id="pk", pk="pk")
  ```
  You get this error: `AssertionError('Use either pk= or hash_id=')`

  `hash_id=` should imply that `pk=` that column.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/84/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzcyMjczNjA4
title: Upgrade to sqlite-utils 2.2.1
state: closed
created_at: 2020-02-07T07:32:12Z
updated_at: 2020-03-20T19:21:42Z
closed_at: 2020-03-20T19:21:41Z
author_association: MEMBER
pull_request: dogsheep/twitter-to-sqlite/pulls/33
type: pull
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/33/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzcyNzYzOTA2
title: Use inspect-file, if possible, for total row count
state: closed
created_at: 2020-02-08T22:10:35Z
updated_at: 2020-03-09T02:47:15Z
closed_at: 2020-02-25T20:19:29Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/666
type: pull
body:
  For large tables, counting the number of rows in the table can take a
  signficant amount of time. Instead, where an inspect-file is provided
  for an immutable database, look up the row-count for a plain count(*).

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/666/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjI3ODc3ODU=
title: Allow injecting configuration data from plugins
state: closed
created_at: 2020-02-10T19:50:15Z
updated_at: 2020-02-12T16:18:22Z
closed_at: 2020-02-12T09:21:22Z
author_association: NONE
type: issue
state_reason: completed
body:
  I'm trying to customize datasette as explorer for [CLDF](https://cldf.clld.org) datasets. Such datasets can be converted automatically to SQLite, which then can be fed to datasette, (e.g. https://github.com/cldf/cookbook/blob/master/recipes/datasette/README.md).

  Part of this customization would be support for the "special" data types described in the [CLDF ontology](https://cldf.clld.org/v1.0/terms.rdf). But while rendering of the values can be customized via the `render_cell` hook in a plugin, e.g. custom labels for foreign keys must be specified through the config file.

  It would be nice to be able to programmatically inject config data from plugins as well.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/667/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjI5MTE4NjM=
title: Create index doesn't work for columns containing spaces
state: closed
created_at: 2020-02-11T00:34:46Z
updated_at: 2020-02-11T05:13:20Z
closed_at: 2020-02-11T05:13:20Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/85/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjMzNDc2Nzk=
title: Make it easier to load SpatiaLite
state: closed
created_at: 2020-02-11T17:03:43Z
updated_at: 2022-01-20T21:29:41Z
closed_at: 2021-01-04T20:18:39Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  $ datasette spatial.db 
  Serve! files=('spatial.db',) (immutables=()) on port 8001
  ERROR: conn=<sqlite3.Connection object at 0x11e388f10>, sql = 'PRAGMA table_info(SpatialIndex);', params = None: no such module: VirtualSpatialIndex
  Usage: datasette serve [OPTIONS] [FILES]...

  Error: It looks like you're trying to load a SpatiaLite database without first loading the SpatiaLite module.

  Read more: https://datasette.readthedocs.io/en/latest/spatialite.html
  ```
  This error message could sniff around in the common locations for the SpatiaLite module and output the CLI command you should use to enable it:
  ```
  datasette spatial.db --load-extension=/usr/local/lib/mod_spatialite.dylib
  ```
  Even better: if Datasette had a `--spatialite` option which automatically loads the extension from common locations, if it can find it.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/668/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzczNzc1Nzg4
title: fix db-to-sqlite command in ecosystem doc page
state: closed
created_at: 2020-02-11T17:05:41Z
updated_at: 2020-02-22T02:32:18Z
closed_at: 2020-02-22T02:32:17Z
author_association: CONTRIBUTOR
pull_request: simonw/datasette/pulls/669
body: the `--connection` parameter has become positional
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/669/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjQ1Nzk0MzA=
title: Problem with square bracket in CSV column name
state: closed
created_at: 2020-02-13T10:19:57Z
updated_at: 2020-02-27T04:16:08Z
closed_at: 2020-02-27T04:16:07Z
author_association: NONE
type: issue
state_reason: completed
body:
  testing some data from european power information (entsoe.eu), the title of the csv contains square brackets.
  as I am playing with glitch, sqlite-utils are used for creating the db.

  Traceback (most recent call last):

    File "/app/.local/bin/sqlite-utils", line 8, in <module>

      sys.exit(cli())

    File "/app/.local/lib/python3.7/site-packages/click/core.py", line 764, in __call__

      return self.main(*args, **kwargs)

    File "/app/.local/lib/python3.7/site-packages/click/core.py", line 717, in main

      rv = self.invoke(ctx)

    File "/app/.local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke

      return _process_result(sub_ctx.command.invoke(sub_ctx))

    File "/app/.local/lib/python3.7/site-packages/click/core.py", line 956, in invoke

      return ctx.invoke(self.callback, **ctx.params)

    File "/app/.local/lib/python3.7/site-packages/click/core.py", line 555, in invoke

      return callback(*args, **kwargs)

    File "/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 434, in insert

      default=default,

    File "/app/.local/lib/python3.7/site-packages/sqlite_utils/cli.py", line 384, in insert_upsert_implementation

      docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs

    File "/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py", line 997, in insert_all

      extracts=extracts,

    File "/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py", line 618, in create

      extracts=extracts,

    File "/app/.local/lib/python3.7/site-packages/sqlite_utils/db.py", line 310, in create_table

      self.conn.execute(sql)

  sqlite3.OperationalError: unrecognized token: "]"

  entsoe_2016.csv

  renamed to txt for uploading compatibility

  [entsoe_2016.txt](https://github.com/simonw/sqlite-utils/files/4197688/entsoe_2016.txt)

  code is remixed directly from your https://glitch.com/edit/#!/datasette-csvs repo

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/86/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjQ4MzM2OTY=
title: Prototoype for Datasette on PostgreSQL
state: open
created_at: 2020-02-13T17:17:55Z
updated_at: 2023-11-17T15:32:21Z
author_association: OWNER
type: issue
body:
  I thought this would never happen, but now that I'm deep in the weeds of running SQLite in production for Datasette Cloud I'm starting to reconsider my policy of only supporting SQLite.

  Some of the factors making me think PostgreSQL support could be worth the effort:
  - Serverless. I'm getting increasingly excited about writable-database use-cases for Datasette. If it could talk to PostgreSQL then users could easily deploy it on Heroku or other serverless providers that can talk to a managed RDS-style PostgreSQL.
  - Existing databases. Plenty of organizations have PostgreSQL databases. They can export to SQLite using [db-to-sqlite](https://github.com/simonw/db-to-sqlite) but that's a pretty big barrier to getting started - being able to run `datasette postgresql://connection-string` and start trying it out would be a massively better experience.
  - Data size. I keep running into use-cases where I want to run Datasette against many GBs of data. SQLite can do this but PostgreSQL is much more optimized for large data, especially given the existence of tools like Citus.
  - Marketing. Convincing people to trust their data to SQLite is potentially a big barrier to adoption. Even if I've convinced myself it's trustworthy I still have to convince everyone else.
  - It might not be that hard? If this required a ground-up rewrite it wouldn't be worth the effort, but I have a hunch that it may not be too hard - most of the SQL in Datasette should work on both databases since it's almost all portable SELECT statements. If Datasette did DML this would be a lot harder, but it doesn't.
  - Plugins! This feels like a natural surface for a plugin - at which point people could add MySQL support and suchlike in the future.

  The above reasons feel strong enough to justify a prototype.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/670/reactions", "total_count": 19, "+1": 14, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 5, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjUwNDE2MjQ=
title: datasette.add_database(name, db) and datasette.remove_database(name) methods
state: closed
created_at: 2020-02-14T01:05:48Z
updated_at: 2020-02-14T01:30:35Z
closed_at: 2020-02-14T01:30:30Z
author_association: OWNER
type: issue
state_reason: completed
body:
  - `datasette.add_database(name, db)` - adds a new named database to the list of connected databases. `db` will be a `Database()` object, which may prove useful in the future for things like #670 and could also allow some plugins to provide in-memory SQLite databases.
  - `datasette.remove_database(name)`

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/417#issuecomment-586047995_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/671/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mzc1MTgwODMy
title: --dirs option for scanning directories for SQLite databases
state: open
created_at: 2020-02-14T02:25:52Z
updated_at: 2020-03-27T01:03:53Z
author_association: OWNER
pull_request: simonw/datasette/pulls/672
body: Refs #417.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/672/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjU1MTg3NzI=
title: Mechanism for checking if a SQLite database file is safe to open
state: closed
created_at: 2020-02-14T19:36:04Z
updated_at: 2020-02-14T20:13:59Z
closed_at: 2020-02-14T20:13:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Opening a SpatiaLite database file without SpatiaLite will result in errors later on. Same for database files which use custom extensions, like the Apple Photos database.

  I've figured out how to tell if a database is safe to open or not:
  ```sql
  select sql from sqlite_master where sql like 'CREATE VIRTUAL TABLE%';
  ```
  This returns the SQL definitions for virtual tables. The bit after `using` tells you what they need.

  Run this against a SpatiaLite database and you get the following:
  ```sql
  CREATE VIRTUAL TABLE SpatialIndex USING VirtualSpatialIndex()
  CREATE VIRTUAL TABLE ElementaryGeometries USING VirtualElementary()
  ```
  Run it against an Apple Photos `photos.db` file (found with `find ~/Library | grep photos.db`) and you get this (partial list):
  ```sql
  CREATE VIRTUAL TABLE RidList_VirtualReader using RidList_VirtualReaderModule
  CREATE VIRTUAL TABLE Array_VirtualReader using Array_VirtualReaderModule
  CREATE VIRTUAL TABLE LiGlobals_VirtualBufferReader using VirtualBufferReaderModule
  CREATE VIRTUAL TABLE RKPlace_RTree using rtree (modelId,minLongitude,maxLongitude,minLatitude,maxLatitude)
  ```
  For a database with FTS4 you get:
  ```sql
  CREATE VIRTUAL TABLE "docs_fts" USING FTS4 (
      [title], [content], content="docs"
  )
  ```
  FTS5:
  ```sql
  CREATE VIRTUAL TABLE [FARA_All_Registrants_fts] USING FTS5 (
                  [Name], [Address_1], [Address_2],
                  content=[FARA_All_Registrants]
              )
  ```
  So I can use this to figure out all of the `using` pieces and then compare them to a list of known support ones.

  _Originally posted by @simonw in https://github.com/simonw/datasette/pull/672#issuecomment-586441484_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/673/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjU1NTIyMTc=
title: Rethink how sanity checks work
state: closed
created_at: 2020-02-14T20:57:02Z
updated_at: 2020-03-26T17:19:23Z
closed_at: 2020-02-15T17:57:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  If you specify a file to open using `files` or `-i` then Datasette should show a useful error message and fail to start.

  Files found by scanning a directory #672 should just be skipped.

  _Split off from comment by @simonw in https://github.com/simonw/datasette/issues/673#issuecomment-586455321_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/674/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjU4Mzc5NjU=
title: Should detect collections.OrderedDict as a regular dictionary
state: closed
created_at: 2020-02-16T02:06:34Z
updated_at: 2020-02-16T02:20:59Z
closed_at: 2020-02-16T02:20:59Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
    File "...python3.7/site-packages/sqlite_utils/db.py", line 292, in create_table
      column_type=COLUMN_TYPE_MAPPING[column_type],
  KeyError: <class 'collections.OrderedDict'>
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/87/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Njc5MDI3MDQ=
state: open
created_at: 2020-02-19T22:55:56Z
updated_at: 2020-12-28T18:49:21Z
author_association: NONE
type: issue
title:
  --cp option for datasette publish and datasette package for shipping additional files and directories

body:
  I’m working on integrating Datasette into a documentation-oriented publishing workflow internally in my company, and in order to deploy the Docker image created by `datasette package` I need to add an additional file to the image — in my case, it’s a sort of a deployment directive. I’ve worked out a way to do this after the image has been created, but it’s convoluted and brittle.

  So it’d be excellent if there was an additional option for this command, something like, like, `--copy`.

  I’d envision it looking something like:

  ```shell
  $ datasette package --copy /the/source/path:/the/target/path data.db
  ```

  I’d be happy to help design, specify, implement, and test this feature, if you’d be interested.

  Thanks for the fantastic tools!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/675/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjgwOTExMzM=
title: ?_searchmode=raw option for running FTS searches without escaping characters
state: closed
created_at: 2020-02-20T06:56:57Z
updated_at: 2020-02-25T05:57:24Z
closed_at: 2020-02-25T05:56:04Z
author_association: NONE
type: issue
state_reason: completed
body:
  After the version 0.34.  I am not able to use the wildchar in the _search option( or the full text search). It will not return any result unless I specify the whole word for text search.  

  If I use 'match :search || "*" ' in the sql statement then it will work as expected.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/676/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjkyMzc1Njg=
title: The first time you click sort by ID it should show you results in reverse order
state: closed
created_at: 2020-02-21T23:38:50Z
updated_at: 2020-03-21T23:57:46Z
closed_at: 2020-03-21T23:57:46Z
author_association: OWNER
type: issue
state_reason: completed
body:
  e.g. on https://latest.datasette.io/fixtures/roadside_attractions

  Clicking the "pk" column header doesn't actually do anything - it sorts by pk asc but since the page was already sorted like that nothing useful changes.

  The first click on a primary key column that the page is already implicitly sorted by should instead enable sort descending on that column.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/677/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjkyNTMwNzI=
title: prepare_connection() plugin hook should accept optional datasette argument
state: closed
created_at: 2020-02-22T00:50:26Z
updated_at: 2020-02-22T03:53:19Z
closed_at: 2020-02-22T02:28:51Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I want to build a plugin that allows users to configure certain database columns to be "masked" - so the `password` column on a users table is never revealed, for example.

  To do this, I need to use the `conn.set_authorizer()` SQLite mechanism. So the plugin needs to build off the `prepare_connection(conn)` hook. But that hook doesn't currently get passed `datasette` so it doesn't have a way of looking up its plugin configuration!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/678/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjkyNjg2MTI=
title: Release 0.36
state: closed
created_at: 2020-02-22T02:41:01Z
updated_at: 2020-02-22T03:52:13Z
closed_at: 2020-02-22T03:52:13Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I think we have enough changes to warrant a release - and I want to take advantage of the changes to the `prepare_connection()` plugin hook in #678

  Changes since 0.35 so far: https://github.com/simonw/datasette/compare/0.35...be2265b0e811d0ac2875c2f748125c17b0f9289e

  - [x] Update ecosystem page
  - [x] Write release notes
  - [x] Ship the release

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/679/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjkyNzU3NjM=
title: Release automation: automate the bit that posts the GitHub release
state: closed
created_at: 2020-02-22T03:50:40Z
updated_at: 2020-09-12T18:18:50Z
closed_at: 2020-09-12T18:18:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The most manual part of [the release process](https://datasette.readthedocs.io/en/stable/contributing.html#release-process) right now is having to post a GitHub release that matches the updated changelog.

  This is particularly annoying because the changelog is in `.rst` while the GitHub release needs markdown - so I currently manually translate between the two.

  Having the release script automatically post a GitHub release at the end would be much more convenient.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/680/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NjkzMTczNzc=
title: Cashe-header missing in http-response
state: closed
created_at: 2020-02-22T10:50:45Z
updated_at: 2020-02-24T20:53:57Z
closed_at: 2020-02-24T20:53:56Z
author_association: NONE
type: issue
state_reason: completed
body:
  Hi Simon. I need some help with both understanding and adding http-headers. If I call datasette on localhost with --config default_cache_ttl:120 and --cors, I only get the following response-headers:

  access-control-allow-origin: *
  content-type: text/html; charset=utf-8
  date: Sat, 22 Feb 2020 10:32:15 GMT
  referrer-policy: no-referrer
  server: uvicorn
  transfer-encoding: chunked

  Cors works, but no caching-header is set? Same thing happens if I use the command in a Dockerfile and run datasette with docker.

  Second, how can one add headers to uvicorn? I've tried to add uvicorn commands to the Dockerfile, before the final datasette command, but it doesn't work. Is there any way to add headers to the uvicorn.run() command i datasette? I particular, I would like to add some of the missing security-headers:

  <img width="1010" alt="Screenshot 2020-02-22 at 11 48 03" src="https://user-images.githubusercontent.com/2181410/75091037-5ab59c80-5569-11ea-8dbb-22357f1aa4c8.png">

  Thank you for a great product!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/681/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Njk2MTM1NjM=
title: Mechanism for writing to database via a queue
state: closed
created_at: 2020-02-24T03:10:07Z
updated_at: 2020-02-25T04:45:10Z
closed_at: 2020-02-25T04:45:10Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I've been mulling this over for a long time, and I have a new approach that I think is worth exploring.

  The catch with writing to SQLite is that it should only accept one write at a time. I'm now thinking that an easy way to manage that would be with a write queue for each database which is then read by a single dedicated write thread which manages its own writable connection.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/682/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mzc5MTkyMjU4
title: .execute_write() and .execute_write_fn() methods on Database
state: closed
created_at: 2020-02-24T19:51:58Z
updated_at: 2020-05-30T18:40:20Z
closed_at: 2020-02-25T04:45:08Z
author_association: OWNER
pull_request: simonw/datasette/pulls/683
type: pull
body:
  See #682

  - [x] Come up with design for `.execute_write()` and `.execute_write_fn()`
  - [x] Build some quick demo plugins to exercise the design
  - [x] Write some unit tests
  - [x] Write the documentation

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/683/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzAzMDEzMzM=
title: Add documentation on Database introspection methods to internals.rst
state: closed
created_at: 2020-02-25T04:20:24Z
updated_at: 2020-06-04T18:56:15Z
closed_at: 2020-05-30T18:40:39Z
author_association: OWNER
body: `internals.rst` will be landing as part of #683
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/684/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzAzMDk1NDY=
state: closed
created_at: 2020-02-25T04:49:44Z
updated_at: 2020-05-30T13:20:50Z
closed_at: 2020-05-08T17:42:18Z
author_association: OWNER
type: issue
state_reason: completed
title:
  Document (and reconsider design of) Database.execute() and Database.execute_against_connection_in_thread()

body:
  In #683 I started a new section of internals documentation covering the `Database` class: https://datasette.readthedocs.io/en/latest/internals.html#database-class

  I decided not to document `.execute()` and `.execute_against_connection_in_thread()` yet because I'm not 100% happy with their API design yet.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/685/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mzc5Mzc4Nzgw
title: ?_searchmode=raw option
state: closed
created_at: 2020-02-25T05:45:50Z
updated_at: 2020-02-25T05:56:09Z
closed_at: 2020-02-25T05:56:04Z
author_association: OWNER
pull_request: simonw/datasette/pulls/686
body: Closes #676
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/686/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzE4MDUzMDA=
title: table.disable_fts() method and "sqlite-utils disable-fts ..." command
state: closed
created_at: 2020-02-27T04:00:50Z
updated_at: 2020-02-27T04:40:44Z
closed_at: 2020-02-27T04:40:44Z
author_association: OWNER
type: issue
state_reason: completed
body:
  This would make it easier to iterate on the FTS configuration for a database without having to wipe and recreate the database each time.

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/88/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzI4OTYyOTM=
title: Expand plugins documentation to multiple pages
state: closed
created_at: 2020-02-28T17:26:21Z
updated_at: 2020-06-22T03:55:20Z
closed_at: 2020-06-22T03:53:54Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I think the plugins docs need to extend beyond a single page now. I want to add a whole section on writing tests for plugins, showing how `httpx` can be used as seen in https://github.com/simonw/datasette-atom/issues/3 and suchlike.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/687/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzgxNjY2Nzc3
title: Don't count rows on homepage for DBs > 100MB
state: closed
created_at: 2020-02-29T01:01:06Z
updated_at: 2020-02-29T01:08:30Z
closed_at: 2020-02-29T01:08:29Z
author_association: OWNER
pull_request: simonw/datasette/pulls/688
body: Closes #649.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/688/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzM1Nzg1NDg=
title: Ability to customize columns used by extracts= feature
state: open
created_at: 2020-03-01T16:54:48Z
updated_at: 2020-10-16T19:17:50Z
author_association: OWNER
type: issue
body:
  @simonw any thoughts on allow extracts to specify the lookup column name? If I'm understanding the documentation right, `.lookup()` allows you to define the "value" column (the documentation uses name), but when you use `extracts` keyword as part of `.insert()`, `.upsert()` etc. the lookup must be done against a column named "value". I have an existing lookup table that I've populated with columns "id" and "name" as opposed to "id" and "value", and seems I can't use `extracts=`, unless I'm missing something...

  Initial thought on how to do this would be to allow the dictionary value to be a tuple of table name column pair... so:
  ```
  table = db.table("trees", extracts={"species_id": ("Species", "name"})
  ```

  I haven't dug too much into the existing code yet, but does this make sense? Worth doing?

  _Originally posted by @chrishas35 in https://github.com/simonw/sqlite-utils/issues/46#issuecomment-592999503_

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/89/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzM1ODM5NzE=
title: "Templates considered" comment broken in >=0.35
state: closed
created_at: 2020-03-01T17:31:21Z
updated_at: 2020-04-05T19:39:44Z
closed_at: 2020-04-05T19:39:44Z
author_association: NONE
type: issue
state_reason: completed
body:
  Noticed that the "Templates Considered" comment is missing in 0.37.  Believe I traced it back to #664 as you can see it in https://v0-34.datasette.io/ but not https://v0-35.datasette.io/.  Looking at the template context debug between the two you can see what is missing from 0.35 vs. 0.34:

  ```diff
  <     "datasette_version": "0.34",
  <     "app_css_hash": "ffa51a",
  <     "select_templates": [
  <         "*index.html"
  <     ],
  <     "zip": "<class 'zip'>",
  <     "body_scripts": [],
  <     "extra_css_urls": "<generator object BaseView._asset_urls at 0x7f6529ac05f0>",
  <     "extra_js_urls": "<generator object BaseView._asset_urls at 0x7f6529ac0660>",
  <     "format_bytes": "<function format_bytes at 0x7f652a1588b0>",
  <     "database_url": "<bound method BaseView.database_url of <datasette.views.index.IndexView object at 0x7f6529b03e50>>",
  <     "database_color": "<bound method BaseView.database_color of <datasette.views.index.IndexView object at 0x7f6529b03e50>>"
  ---
  >     "datasette_version": "0.35",
  >     "database_url": "<bound method BaseView.database_url of <datasette.views.index.IndexView object at 0x7f6140dacd90>>",
  >     "database_color": "<bound method BaseView.database_color of <datasette.views.index.IndexView object at 0x7f6140dacd90>>"
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/689/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzM3NDA3MTI=
title: Cannot .enable_fts() for columns with spaces in their names
state: closed
created_at: 2020-03-02T06:06:03Z
updated_at: 2020-03-02T06:10:49Z
closed_at: 2020-03-02T06:10:49Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  import sqlite_utils
  db = sqlite_utils.Database(memory=True)                                 
  db["test"].insert({"space in name": "hello"})                           
  db["test"].enable_fts(["space in name"])                                
  ---------------------------------------------------------------------------
  OperationalError                          Traceback (most recent call last)
  <ipython-input-8-ce4b87dd1c7a> in <module>
  ----> 1 db['test'].enable_fts(["space in name"])

  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in enable_fts(self, columns, fts_version, create_triggers)
      755         )
      756         self.db.conn.executescript(sql)
  --> 757         self.populate_fts(columns)
      758 
      759         if create_triggers:

  /usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in populate_fts(self, columns)
      787             table=self.name, columns=", ".join(columns)
      788         )
  --> 789         self.db.conn.executescript(sql)
      790         return self
      791 

  OperationalError: near "in": syntax error
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/90/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzM3NTU3MjY=
title: Mechanism for plugins to add action menu items for various things
state: closed
created_at: 2020-03-02T06:48:36Z
updated_at: 2020-10-30T05:20:43Z
closed_at: 2020-10-30T05:20:42Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Now that we have support for plugins that can write I'm seeing all sorts of places where a plugin might need to add UI to the table page.

  Some examples:

  - `datasette-configure-fts` needs to add a "configure search for this table" link
  - a plugin that lets you render or delete tables needs to add a link or button somewhere
  - existing plugins like `datasette-vega` and `datasette-cluster-map` already do this with JavaScript

  The challenge here is that multiple plugins may want to do this, so simply overriding templates and populating names blocks doesn't entirely work as templates may override each other.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/690/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzQwMjExOTQ=
title: --reload sould reload server if code in --plugins-dir changes
state: open
created_at: 2020-03-02T14:42:21Z
updated_at: 2020-06-14T02:35:17Z
author_association: OWNER
type: issue
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/691/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzQwMzU0MzI=
title: is_hidden_table context variable on table.html page
state: open
created_at: 2020-03-02T15:03:25Z
updated_at: 2020-03-02T15:03:48Z
author_association: OWNER
type: issue
body:
  It's useful to know if a table is hidden when rendering that page. `datasette-configure-fts` for example may want to disallow enabling search on hidden tables.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/692/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzQwNDMyMTg=
title: Variables from extra_template_vars() not exposed in _context=1
state: closed
created_at: 2020-03-02T15:14:51Z
updated_at: 2020-04-05T19:12:48Z
closed_at: 2020-04-05T19:12:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  The `_context=1` debugging mode does not show variables that should have been added to the context by the `extra_template_vars()` plugin hook.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/693/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzY1ODI2MDQ=
title: datasette publish cloudrun --memory option
state: closed
created_at: 2020-03-05T22:59:57Z
updated_at: 2020-06-23T17:10:51Z
closed_at: 2020-03-05T23:49:41Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Got this error deploying large (603MB) database with Cloud Run
  ```
  X Deploying... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revi
  sion might contain more information.                                                                                                                               
    X Creating Revision... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for
     this revision might contain more information.                                                                                                                   
    . Routing traffic...                                                                                                                                             
    ✓ Setting IAM Policy...                                                                                                                                          
  Deployment failed                                                                                                                                                  
  ERROR: (gcloud.run.deploy) Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.
  ```

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/694/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzY3MTE1ODk=
title: Update SQLite bundled with Docker container
state: closed
created_at: 2020-03-06T05:42:12Z
updated_at: 2020-03-08T23:33:23Z
closed_at: 2020-03-06T06:15:27Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It's 3.26.0 at the moment:
  https://github.com/simonw/datasette/blob/af9cd4ca64652fae262e6f7b5d201f6e0adc989b/Dockerfile#L9-L11
  Most recent release is 3.31.1: https://www.sqlite.org/releaselog/3_31_1.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/695/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzY3MjIxMTU=
title: Single failing unit test when run inside the Docker image
state: closed
created_at: 2020-03-06T06:16:36Z
updated_at: 2021-03-29T17:04:19Z
closed_at: 2021-03-07T07:41:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  ```
  docker run -it -v `pwd`:/mnt datasetteproject/datasette:latest /bin/bash
  root@0e1928cfdf79:/# cd /mnt
  root@0e1928cfdf79:/mnt# pip install -e .[test]
  root@0e1928cfdf79:/mnt# pytest
  ```
  I get one failure!

  It was for `test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]`
  ```
      def test_searchable(app_client, path, expected_rows):
          response = app_client.get(path)
  >       assert expected_rows == response.json["rows"]
  E       AssertionError: assert [[1, 'barry c...sel', 'puma']] == []
  E         Left contains 2 more items, first extra item: [1, 'barry cat', 'terry dog', 'panther']
  E         Full diff:
  E         + []
  E         - [[1, 'barry cat', 'terry dog', 'panther'],
  E         -  [2, 'terry dog', 'sara weasel', 'puma']]
  ```

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/695#issuecomment-595614469_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/696/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1NzczMDIyMjk=
title: Enable ordering FTS results by rank
state: closed
created_at: 2020-03-07T08:43:51Z
updated_at: 2020-11-06T23:53:26Z
closed_at: 2020-11-06T23:53:25Z
author_association: NONE
type: issue
state_reason: completed
body:
  According to https://www.sqlite.org/fts5.html (not sure about FTS4) results can be sorted by relevance. At the moment results are returned by default by `rowid`. Perhaps a flag can be added to the `search` method?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/91/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Nzc1NzgzMDY=
title: index.html is not reliably loaded from a plugin
state: closed
created_at: 2020-03-08T22:37:55Z
updated_at: 2020-03-08T23:33:28Z
closed_at: 2020-03-08T23:11:27Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Lots of detail in https://github.com/simonw/datasette-search-all/issues/2 - short version is that I have a plugin with its own `index.html` template and Datasette intermittently fails to load it and uses the default `index.html` that ships with Datasette instead.

  Related:

  * #689: "Templates considered" comment broken in >=0.35
  * #693: Variables from extra_template_vars() not exposed in _context=1 (may as well fix this while I'm in there)

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/697/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1Nzg4ODM3MjU=
title: Command for importing commits
state: closed
created_at: 2020-03-10T21:55:12Z
updated_at: 2020-03-11T02:47:37Z
closed_at: 2020-03-11T02:47:37Z
author_association: MEMBER
body: Using this API: https://api.github.com/repos/dogsheep/github-to-sqlite/commits
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/17/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODEzMzk5NjE=
title: .columns_dict doesn't work for all possible column types
state: closed
created_at: 2020-03-14T19:30:35Z
updated_at: 2020-03-15T18:37:43Z
closed_at: 2020-03-14T20:04:14Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Got this error:
  ```
    File ".../python3.7/site-packages/sqlite_utils/db.py", line 462, in <dictcomp>
      for column in self.columns
  KeyError: 'REAL'
  ```
  `.columns_dict` uses `REVERSE_COLUMN_TYPE_MAPPING`:
  https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L457-L463
  `REVERSE_COLUMN_TYPE_MAPPING` defines `FLOAT` not `REAL`A
  https://github.com/simonw/sqlite-utils/blob/43f1c6ab4e3a6b76531fb6f5447adb83d26f3971/sqlite_utils/db.py#L68-L74

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/92/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODE3OTU1NzA=
title: Support more string values for types in .add_column()
state: open
created_at: 2020-03-15T19:32:49Z
updated_at: 2020-09-24T20:36:46Z
author_association: OWNER
type: issue
body:
  https://sqlite-utils.readthedocs.io/en/2.4.2/python-api.html#adding-columns says:
  > SQLite types you can specify are "TEXT", "INTEGER", "FLOAT" or "BLOB".

  As discovered in #92 this isn't the right list of values. I should expand this to match https://www.sqlite.org/datatype3.html

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/93/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODI1MTc5NjU=
title: Ability for a canned query to write to the database
state: closed
created_at: 2020-03-16T18:31:59Z
updated_at: 2020-06-06T19:43:49Z
closed_at: 2020-06-06T19:43:48Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Canned queries are currently read-only: https://datasette.readthedocs.io/en/0.38/sql_queries.html#canned-queries

  Add a `"write": true` option to their definition in `metadata.json` which turns them into queries that are submitted via POST and send their queries to the write queue.

  Then they can be used as a really quick way to define a writable interface and JSON API!

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/698/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODI1MjY5NjE=
title: Authentication (and permissions) as a core concept
state: closed
created_at: 2020-03-16T18:48:00Z
updated_at: 2020-06-06T19:42:11Z
closed_at: 2020-06-06T19:42:11Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now Datasette authentication is provided exclusively by plugins:

  * https://github.com/simonw/datasette-auth-github
  * https://github.com/simonw/datasette-auth-existing-cookies

  This is an all-or-nothing approach: either your Datasette instance requires authentication at the top level or it does not.

  But... as I build new plugins like https://github.com/simonw/datasette-configure-fts and https://github.com/simonw/datasette-edit-tables I increasingly have individual features which should be reserved for logged-in users while still wanting other parts of Datasette to be open to all.

  This is too much for plugins to own independently of Datasette core. Datasette needs to ship a single "user is authenticated" concept (independent of how users actually sign in) so that different plugins can integrate with it.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/699/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODI3MTM1NTQ=
title: Request object utility for handling POST form data
state: closed
created_at: 2020-03-17T02:44:59Z
updated_at: 2020-03-17T02:47:50Z
closed_at: 2020-03-17T02:47:50Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > This is also going to need me to handle POST form submissions which means I need to be able to parse the form body. I guess that will go in [datasette/utils/asgi.py](https://github.com/simonw/datasette/blob/master/datasette/utils/asgi.py).

  _Originally posted by @simonw in https://github.com/simonw/datasette/issues/698#issuecomment-599704264_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/700/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODM5NzAxOTY=
title: Search box CSS doesn't look great on OS X Safari
state: closed
created_at: 2020-03-18T20:00:52Z
updated_at: 2020-03-24T22:57:18Z
closed_at: 2020-03-24T22:57:18Z
author_association: OWNER
type: issue
state_reason: completed
body:
  <img width="1447" alt="twitter__tweets__68_773_rows_where_sorted_by_id_descending_and_twitter__tweets__955_rows" src="https://user-images.githubusercontent.com/9599/77002262-73bb2e80-6918-11ea-87a3-e3f537365c9f.png">

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/701/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODUyNjY3NjM=
title: IndexError running user-timeline command
state: closed
created_at: 2020-03-20T18:54:08Z
updated_at: 2020-03-20T19:20:52Z
closed_at: 2020-03-20T19:20:37Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
  $ twitter-to-sqlite user-timeline data.db --screen_name Allen_Joines
  Traceback (most recent call last):
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/bin/twitter-to-sqlite", line 11, in <module>
      load_entry_point('twitter-to-sqlite', 'console_scripts', 'twitter-to-sqlite')()
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/cli.py", line 256, in user_timeline
      utils.save_tweets(db, chunk)
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 289, in save_tweets
      db["users"].upsert(user, pk="id", alter=True)
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1128, in upsert
      conversions=conversions,
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1157, in upsert_all
      upsert=True,
    File "/Users/simonw/.local/share/virtualenvs/twitter-to-sqlite-4ech4lJi/lib/python3.7/site-packages/sqlite_utils/db.py", line 1096, in insert_all
      row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
  IndexError: list index out of range
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODUyODIyMTI=
title: twitter-to-sqlite user-timeline [screen_names] --sql / --attach
state: closed
created_at: 2020-03-20T19:26:07Z
updated_at: 2020-03-20T20:17:00Z
closed_at: 2020-03-20T20:16:35Z
author_association: MEMBER
body: Split from #8.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODUzMDY4NDc=
title: twitter-to-sqlite followers/friends --sql / --attach
state: closed
created_at: 2020-03-20T20:20:33Z
updated_at: 2020-03-20T23:12:38Z
closed_at: 2020-03-20T23:12:38Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Split from #8. The `friends` and `followers` commands don't yet support `--sql` and `--attach`.

  (`friends-ids` and `followers-ids` do though).

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/36/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODUzNTM1OTg=
title: Handle "User not found" error
state: closed
created_at: 2020-03-20T22:14:32Z
updated_at: 2020-04-17T23:43:46Z
closed_at: 2020-04-17T23:43:46Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  While running `user-timeline` I got this bug (because a screen name I asked for didn't exist):
  ```
    File "/Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py", line 185, in transform_user
      user["created_at"] = parser.parse(user["created_at"])
  KeyError: 'created_at'
  >>> import pdb
  >>> pdb.pm()
  > /Users/simonw/Dropbox/Development/twitter-to-sqlite/twitter_to_sqlite/utils.py(185)transform_user()
  -> user["created_at"] = parser.parse(user["created_at"])
  (Pdb) user
  {'errors': [{'code': 50, 'message': 'User not found.'}]}
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODUzNTkzNjM=
title: Screen name display for user-timeline is uneven
state: closed
created_at: 2020-03-20T22:30:23Z
updated_at: 2020-03-20T22:37:17Z
closed_at: 2020-03-20T22:37:17Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
  CDPHE  [####################################]  67
  CHFSKy  [####################################]  3216
  DHSWI  [####################################]  41
  DPHHSMT  [####################################]  742
  Delaware_DHSS  [####################################]  3231
  DhhsNevada  [####################################]  639
  ```
  I could format them to match the length of the longest screen name instead.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODUzOTA0ODI=
title: Option in metadata.json to set default sort order for a table
state: closed
created_at: 2020-03-21T00:19:56Z
updated_at: 2020-03-25T04:19:36Z
closed_at: 2020-03-22T02:40:35Z
author_association: OWNER
type: issue
state_reason: completed
body:
  If you access the table page without any `?_sort` or `?_sort_desc` arguments it currently defaults to order by primary key - would be neat to be able to change that.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/702/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODU0MTE1NDc=
title: Commits in GitHub API can have null author
state: closed
created_at: 2020-03-21T02:20:56Z
updated_at: 2020-03-23T20:44:49Z
closed_at: 2020-03-23T20:44:26Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  ```
  Traceback (most recent call last):
    File "/home/ubuntu/datasette-venv/bin/github-to-sqlite", line 8, in <module>
      sys.exit(cli())
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
      return self.main(*args, **kwargs)
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 717, in main
      rv = self.invoke(ctx)
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
      return callback(*args, **kwargs)
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/cli.py", line 235, in commits
      utils.save_commits(db, commits, repo_full["id"])
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 290, in save_commits
      commit_to_insert["author"] = save_user(db, commit["author"])
    File "/home/ubuntu/datasette-venv/lib/python3.6/site-packages/github_to_sqlite/utils.py", line 54, in save_user
      for key, value in user.items()
  AttributeError: 'NoneType' object has no attribute 'items'
  ```
  Got this running the `commits` command from cron.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/18/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODU1MjYyOTI=
title: Set up full text search
state: closed
created_at: 2020-03-21T15:57:35Z
updated_at: 2020-03-21T19:47:46Z
closed_at: 2020-03-21T19:45:52Z
author_association: MEMBER
body: Should run against `title` and `text` in `items`, and `about` and `id` in `users`.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/1/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzkxOTI0NTA5
title: WIP implementation of writable canned queries
state: closed
created_at: 2020-03-21T22:23:51Z
updated_at: 2020-06-03T00:08:14Z
closed_at: 2020-06-02T23:57:35Z
author_association: OWNER
pull_request: simonw/datasette/pulls/703
body: Refs #698.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/703/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODU1OTczMjk=
title: Add datasette-publish-fly to Datasette Publish documentation
state: closed
created_at: 2020-03-21T22:25:10Z
updated_at: 2020-03-24T22:39:09Z
closed_at: 2020-03-24T22:39:09Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It's a cool example of a plugin that provides a new publish provider - worth mentioning on https://datasette.readthedocs.io/en/stable/publish.html

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/704/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODU2MjYxOTk=
title: latest.datasette.io is no longer updating
state: closed
created_at: 2020-03-22T01:59:30Z
updated_at: 2020-03-25T02:30:24Z
closed_at: 2020-03-25T02:30:24Z
author_association: OWNER
body: https://latest.datasette.io/-/versions is stuck on 0.35.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/705/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODU2MzMxNDI=
title: Documentation for the "request" object
state: closed
created_at: 2020-03-22T02:55:50Z
updated_at: 2020-05-30T13:20:00Z
closed_at: 2020-05-27T22:31:22Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Since that object is passed to the `extra_template_vars` hooks AND the classes registered by `register_facet_classes` it should be part of the documented interface on https://datasette.readthedocs.io/en/stable/internals.html

  I could also start passing it to the `register_output_renderer` callback.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/706/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODU4NTA3MTU=
title: Enable full-text search for more stuff (like commits, issues and issue_comments)
state: closed
created_at: 2020-03-23T00:19:56Z
updated_at: 2020-03-23T19:06:39Z
closed_at: 2020-03-23T19:06:39Z
author_association: MEMBER
body: Currently FTS is only enabled for repos and releases.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/19/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODY0NTQ1MTM=
title: Upgrade to sqlite-utils 2.x
state: closed
created_at: 2020-03-23T19:17:58Z
updated_at: 2020-03-23T19:22:52Z
closed_at: 2020-03-23T19:22:52Z
author_association: MEMBER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/20/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODY0Nzc3NTc=
title: If column data is a mixture of integers and nulls, detected type should be INTEGER
state: closed
created_at: 2020-03-23T19:51:46Z
updated_at: 2020-03-23T19:57:10Z
closed_at: 2020-03-23T19:57:10Z
author_association: OWNER
body: It looks like detected type for that case is TEXT at the moment.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/94/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODY0ODYzNjc=
title: Columns with only null values are no longer created in the database
state: closed
created_at: 2020-03-23T20:07:42Z
updated_at: 2020-03-23T20:31:15Z
closed_at: 2020-03-23T20:31:15Z
author_association: OWNER
body: Bug introduced in #94, and released in `2.4.3`.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/95/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODY1NjE3Mjc=
title: Turn GitHub API errors into exceptions
state: closed
created_at: 2020-03-23T22:37:24Z
updated_at: 2020-03-23T23:48:23Z
closed_at: 2020-03-23T23:48:22Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  This would have really helped in debugging the mess in #13. Running with this `auth.json` is a useful demo:
  ```json
  {"github_personal_token": ""}
  ```

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/21/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODY1NjczNzk=
title: Handle empty git repositories
state: closed
created_at: 2020-03-23T22:49:48Z
updated_at: 2020-03-23T23:13:11Z
closed_at: 2020-03-23T23:13:11Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  Got this error:
  ```
  github_to_sqlite.utils.GitHubError: {'message': 'Git Repository is empty.', 'documentation_url': 'https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository'}
  ```
  From https://api.github.com/repos/dogsheep/beta/commits

reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/22/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODY1OTU4Mzk=
title: Release 1.0
state: closed
created_at: 2020-03-24T00:03:55Z
updated_at: 2020-03-24T00:15:50Z
closed_at: 2020-03-24T00:15:50Z
author_association: MEMBER
body: Need to compile release notes.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/github-to-sqlite/issues/23/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODcyMjIzNTQ=
title: Consider configuring Jinja in Datasette() constructor, not .app()
state: closed
created_at: 2020-03-24T19:19:58Z
updated_at: 2020-03-27T01:12:57Z
closed_at: 2020-03-27T01:12:57Z
author_association: OWNER
type: issue
state_reason: completed
body:
  Right now the following fails with an error:
  ```python
  ds = Datasette([], template_dir=".")
  rendered = await ds.render_template("index.html")
  ```
  The error is:
  ```
      async def render_template(
          self, templates, context=None, request=None, view_name=None
      ):
          context = context or {}
          if isinstance(templates, Template):
              template = templates
              select_templates = []
          else:
              if isinstance(templates, str):
                  templates = [templates]
  >           template = self.jinja_env.select_template(templates)
  E           AttributeError: 'Datasette' object has no attribute 'jinja_env'
  ```
  This is because `jinja_env` is configured in the `.app()` method, here:

  https://github.com/simonw/datasette/blob/a498d0fe6590f9bdbc4faf9e0dd5faeb3b06002c/datasette/app.py#L609-L633

  This is a little surprising, especially now that `.render_template()` is part of the documented internals API: https://datasette.readthedocs.io/en/stable/internals.html#render-template-template-context-none-request-none

  Maybe this should happen in the Datasette class constructor instead.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/707/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0MzkzMjc0NDMz
title: base_url configuration setting, refs #394
state: closed
created_at: 2020-03-24T21:52:00Z
updated_at: 2020-03-25T00:18:44Z
closed_at: 2020-03-25T00:18:44Z
author_association: OWNER
pull_request: simonw/datasette/pulls/708
body: Pull request implementing #394
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/708/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODczMTQwMDI=
title: Each plugin hook should link to example plugins built with it
state: closed
created_at: 2020-03-24T22:18:48Z
updated_at: 2020-03-24T22:30:10Z
closed_at: 2020-03-24T22:29:43Z
author_association: OWNER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/709/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODczMjI0NDM=
title: Remove Zeit Now v1 support
state: closed
created_at: 2020-03-24T22:39:49Z
updated_at: 2020-04-04T23:05:12Z
closed_at: 2020-04-04T23:05:12Z
author_association: OWNER
type: issue
state_reason: completed
body:
  It will remain supported as a plugin but since no-one can sign up for Docker hosting any more (for over a year now) there's no point including it in Datasette core.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/710/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODczOTg3MDM=
title: Release notes for Datasette 0.39
state: closed
created_at: 2020-03-25T02:31:13Z
updated_at: 2020-03-25T04:06:55Z
closed_at: 2020-03-25T04:06:55Z
author_association: OWNER
body: Then I can ship it.
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/711/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODgxMDg0Mjg=
title: base_url doesn't entirely work for running Datasette inside Binder
state: closed
created_at: 2020-03-26T02:25:55Z
updated_at: 2020-03-26T15:11:49Z
closed_at: 2020-03-26T14:35:43Z
author_association: OWNER
type: issue
state_reason: completed
body:
  > Thanks! I'm trying to launch Datasette from *within* a notebook using the jupyter-server-proxy and the new `base_url` parameter. While the assets load ok, and the breadcrumb navigation works, the facet links don't seem to use the `base_url`. Or have I missed something?

  _Originally posted by @wragge in https://github.com/simonw/datasette/issues/394#issuecomment-604166918_

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/712/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODk0MDI5Mzk=
title: Store authentication information as "pocket_access_token" etc
state: closed
created_at: 2020-03-27T20:43:22Z
updated_at: 2020-03-27T20:43:59Z
closed_at: 2020-03-27T20:43:59Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  The `pocket_` prefix will mean that the same `auth.json` file can be used for other Dogsheep tools without Pocket over-riding a value set by some other tool.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/4/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1ODk0OTE3MTE=
title: Upgrade to sqlite-utils 2.x
state: closed
created_at: 2020-03-28T02:24:51Z
updated_at: 2020-03-28T02:25:03Z
closed_at: 2020-03-28T02:25:03Z
author_association: MEMBER
type: issue
state_reason: completed
reactions:
  {"url": "https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/7/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mzk1MjU4Njg3
title: Add type conversion for Panda's Timestamp
state: closed
created_at: 2020-03-29T14:13:09Z
updated_at: 2020-03-31T04:40:49Z
closed_at: 2020-03-31T04:40:48Z
author_association: CONTRIBUTOR
pull_request: simonw/sqlite-utils/pulls/96
type: pull
body:
  Add type conversion for Panda's Timestamp, if Panda library is present in system
  (thanks for this project, I was about to do the same thing from scratch)

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/96/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTA2NjY3NjA=
title: --since feature can be confused by retweets
state: closed
created_at: 2020-03-30T23:25:33Z
updated_at: 2020-04-01T03:45:16Z
closed_at: 2020-04-01T03:45:16Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  If you run `twitter-to-sqlite user-timeline ... --since` it's supposed to fetch Tweets those specific users tweeted since last time the command was run.

  It does this by seeking out the max ID of their previous tweets:

  https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L305-L311

  BUT... this has a nasty flaw: if another account had retweeted one of their recent tweets the retweeted-tweet will have been loaded into the database - so we may treat that as the most recent since ID and miss a bunch of their tweets!

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTA2Njk3OTM=
title: Feature: record history of follower counts
state: closed
created_at: 2020-03-30T23:32:28Z
updated_at: 2020-04-01T04:13:05Z
closed_at: 2020-04-01T04:13:05Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  We currently over-write the follower count every time we import a tweet (when we import that user profile again):

  https://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/utils.py#L293-L294

  It would be neat if we noticed if that user's follower count (and maybe other counts?) had changed since we last saved them and recorded that change in a separate history table. This would be an inexpensive way of building up rough charts of follower count over time.

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTE2MTM1Nzk=
title: Bug: recorded a since_id for None, None
state: closed
created_at: 2020-04-01T04:29:43Z
updated_at: 2020-04-01T04:31:11Z
closed_at: 2020-04-01T04:31:11Z
author_association: MEMBER
type: issue
state_reason: completed
body:
  This shouldn't happen in the `since_ids` table (relates to #39):

  <img width="422" alt="twitter__since_ids__2_rows" src="https://user-images.githubusercontent.com/9599/78099419-b0fed200-7396-11ea-87d8-992a171d6fab.png">

reactions:
  {"url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/41/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTI4MjkxMzU=
title: Support YAML in metadata - metadata.yaml
state: closed
created_at: 2020-04-02T18:10:05Z
updated_at: 2020-04-02T19:36:17Z
closed_at: 2020-04-02T19:30:55Z
author_association: OWNER
type: issue
state_reason: completed
body:
  I was originally going to do this with a plugin - see #357 - but the more I work with `metadata.json` the more I want it to just accept YAML as an optional alternative to JSON.

  The best example why is still this one: https://github.com/simonw/russian-ira-facebook-ads-datasette/blob/master/russian-ads-metadata.yaml

  YAML is just SO much better than JSON for multi-line strings - in particular HTML and SQL, both of which are common in `metadata.json` files.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/713/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDExOlB1bGxSZXF1ZXN0Mzk3NzQ5NjUz
title: --metadata accepts YAML as well as JSON
state: closed
created_at: 2020-04-02T18:36:02Z
updated_at: 2020-04-02T19:30:54Z
closed_at: 2020-04-02T19:30:54Z
author_association: OWNER
pull_request: simonw/datasette/pulls/714
body: Refs #713. Still needs tests and documentation.
type: pull
reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/714/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTMwMDY4MTQ=
title: Refactor duplicate cell display logic
state: open
created_at: 2020-04-03T00:58:11Z
updated_at: 2020-04-03T00:58:11Z
author_association: OWNER
type: issue
body:
  The logic for rendering cells in table view and in database (or canned query) view is currently very similar:

  https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/base.py#L514-L539

  Compared with:

  https://github.com/simonw/datasette/blob/7656fd64d8b6a32ebc34d89c1b8711cc5ea240f7/datasette/views/table.py#L104-L195

  I'll be changing this a bit in #698 but I should still try to clean this up more further in the future.

reactions:
  {"url": "https://api.github.com/repos/simonw/datasette/issues/715/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


node_id: MDU6SXNzdWU1OTM3NTEyOTM=
title: Adding a "recreate" flag to the `Database` constructor
state: closed
created_at: 2020-04-04T05:41:10Z
updated_at: 2020-04-15T14:29:31Z
closed_at: 2020-04-13T03:52:29Z
author_association: NONE
type: issue
state_reason: completed
body:
  I have a [script](https://github.com/betatim/binder-datasette/blob/master/create-db.ipynb) that imports data into a sqlite DB. When I re-run that script I'd like to remove the existing sqlite DB, instead of adding to it. The pragmatic answer is to add the check and file deletion to my script.

  However I thought it would be easy and useful for others to add a `recreate=True` flag to `db = sqlite_utils.Database("binder-launches.db")`. After taking a look at the code for it I am not so sure any more. This is because the connection string could be a URL (or "connection string") like `"file:///tmp/foo.db"`. I don't know what the equivalent of `os.path.exists()` is for a connection string or how to detect that something is a connection string and raise an error "can't use recreate=True and conn_string at the same time".

  Does anyone have an idea/suggestion where to start investigating?

reactions:
  {"url": "https://api.github.com/repos/simonw/sqlite-utils/issues/97/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


