# issue_comments

html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753390262
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5MDI2Mg==
created_at: 2021-01-01T21:58:11Z
updated_at: 2021-01-01T21:58:11Z
author_association: OWNER
body:
  One possibility: plugins could write directly to that in-memory database table. But how would they know to write again should the server restart? Maybe they would write to it once when called by the `startup` plugin hook, and then update it (and their own backing store) when metadata changes for some reason. Feels a bit messy though.

  Also: if I want to support metadata optionally living in a `_metadata` table colocated with the data in a SQLite database file itself, how would that affect the `metadata` columns in `_internal`? How often would Datasette denormalize and copy data across from the on-disk `_metadata` tables to the `_internal` in-memory columns?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753390791
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5MDc5MQ==
created_at: 2021-01-01T22:00:42Z
updated_at: 2021-01-01T22:00:42Z
author_association: OWNER
body:
  Here are the requirements I'm currently trying to satisfy:

  - It should be possible to query the metadata for ALL attached tables in one place, potentially with pagination and filtering
  - Metadata should be able to exist in the current `metadata.json` file
  - It should also be possible to bundle metadata in a table in the SQLite database files themselves
  - Plugins should be able to define their own special mechanisms for metadata. This is particularly interesting for providing a UI that allows users to edit the metadata for their existing tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753391869
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5MTg2OQ==
created_at: 2021-01-01T22:04:30Z
updated_at: 2021-01-01T22:04:30Z
author_association: OWNER
body:
  The sticking point here seems to be the plugin hook. Allowing plugins to over-ride the way the question "give me the metadata for this database/table/column" is answered makes the database-backed metadata mechanisms much more complicated to think about.

  What if plugins didn't get to over-ride metadata in this way, but could instead update the metadata in a persistent Datasette-managed storage mechanism?

  Then maybe Datasette could do the following:

  - Maintain metadata in `_internal` that has been loaded from `metadata.json`
  - Know how to check a database for baked-in metadata (maybe in a `_metadata` table)
  - Know how to fall back on the `_internal` metadata if no baked-in metadata is available

  If database files were optionally allowed to store metadata about tables that live in another database file this could perhaps solve the plugin needs - since an "edit metadata" plugin would be able to edit records in a separate, dedicated `metadata.db` database to store new information about tables in other files.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753392102
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5MjEwMg==
created_at: 2021-01-01T22:06:33Z
updated_at: 2021-01-01T22:06:33Z
author_association: OWNER
body:
  Some SQLite databases include SQL comments in the schema definition which tell you what each column means:

  ```sql
  CREATE TABLE User
          -- A table comment
  (
          uid INTEGER,    -- A field comment
          flags INTEGER   -- Another field comment
  );
  ```
  The problem with these is that they're not exposed to SQLite in any mechanism other than parsing the `CREATE TABLE` statement from the `sqlite_master` table to extract those columns.

  I had an idea to build a plugin that could return these. That would be easy with a "get metadata for this column" plugin hook - in the absence of one a plugin could still run that reads the schemas on startup and uses them to populate a metadata database table somewhere.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753398542
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5ODU0Mg==
created_at: 2021-01-01T22:37:24Z
updated_at: 2021-01-01T22:37:24Z
author_association: OWNER
body:
  The direction I'm leaning in now is the following:

  - Metadata always lives in SQLite tables
  - These tables can be co-located with the database they describe (same DB file)
  - ... or they can be in a different DB file and reference the other database that they are describing
  - Metadata provided on startup in a `metadata.json` file is loaded into an in-memory metadata table using that same mechanism

  Plugins that want to provide metadata can do so by populating a table. They could even maintain their own in-memory database for this, or they could write to the `_internal` in-memory database, or they could write to a table in a database on disk.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753399366
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5OTM2Ng==
created_at: 2021-01-01T22:42:37Z
updated_at: 2021-01-01T22:42:37Z
author_association: OWNER
body:
  So what would the database schema for this look like?

  I'm leaning towards a single table called `_metadata`, because that's a neater fit for baking the metadata into the database file along with the data that it is describing. Alternatively I could have multiple tables sharing that prefix - `_metadata_database` and `_metadata_tables` and `_metadata_columns` perhaps.

  If it's just a single `_metadata` table, the schema could look like this:

  | database | table | column | metadata |
  | --- | --- | --- | --- |
  |  | mytable |  | {"title": "My Table" } |
  |  | mytable | mycolumn | {"description": "Column description" } |
  | otherdb | othertable | | {"description": "Table in another DB" } |

  If the `database` column is `null` it means "this is describing a table in the same database file as this `_metadata` table".

  The alternative to the `metadata` JSON column would be separate columns for each potential metadata value - `license`, `source`, `about`, `about_url` etc. But that makes it harder for people to create custom metadata fields.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753399428
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5OTQyOA==
created_at: 2021-01-01T22:43:14Z
updated_at: 2021-01-01T22:43:22Z
author_association: OWNER
body:
  Could this use a compound primary key on `database, table, column`? Does that work with null values?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753399635
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzM5OTYzNQ==
created_at: 2021-01-01T22:45:21Z
updated_at: 2021-01-01T22:50:21Z
author_association: OWNER
body:
  Would also need to figure out the precedence rules:

  - What happens if the database has a `_metadata` table with data that conflicts with a remote metadata record from another database? I think the other database should win, because that allows plugins to over-ride the default metadata for something.
  - Do JSON values get merged together? So if one table provides a description and another provides a title do both values get returned?
  - If a database has a `license`, does that "cascade" down to the tables? What about `source` and `about`?
  - What if there are two databases (or more) that provide conflicting metadata for a table in some other database? Also, `_internal` may have loaded data from `metadata.json` that conflicts with some other remote table metadata definition.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753400265
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzQwMDI2NQ==
created_at: 2021-01-01T22:52:09Z
updated_at: 2021-01-01T22:52:09Z
author_association: OWNER
body:
  From an implementation perspective, I think the way this works is SQL queries read the relevant metadata from ALL available metadata tables, then Python code solves the precedence rules to produce the final, combined metadata for a database/table/column.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753400306
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzQwMDMwNg==
created_at: 2021-01-01T22:52:44Z
updated_at: 2021-01-01T22:52:44Z
author_association: OWNER
body:
  Also: probably load column metadata as part of the table metadata rather than loading column metadata individually, since it's going to be rare to want the metadata for a single column rather than for an entire table full of columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753400420
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzQwMDQyMA==
created_at: 2021-01-01T22:53:58Z
updated_at: 2021-01-01T22:53:58Z
author_association: OWNER
body:
  Precedence idea:
  - First priority is non-_internal metadata from other databases - if those conflict then pick then the alphabetically-ordered-first database name wins
  - Next priority: `_internal` metadata, which should have been loaded from `metadata.json`
  - Last priority: the `_metadata` table from that database itself, i.e. the default "baked in" metadata

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753401001
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzQwMTAwMQ==
created_at: 2021-01-01T23:01:45Z
updated_at: 2021-01-01T23:01:45Z
author_association: OWNER
body:
  I need to prototype this. Could I do that as a plugin? I think so - I could try out the algorithm for loading metadata and display it on pages using some custom templates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753402423
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzQwMjQyMw==
created_at: 2021-01-01T23:16:05Z
updated_at: 2021-01-01T23:16:05Z
author_association: OWNER
body:
  One catch: solving the "show me all metadata for everything in this Datasette instance" problem.

  Ideally there would be a SQLite table that can be queried for this. But the need to resolve the potentially complex set of precedence rules means that table would be difficult if not impossible to provide at run-time.

  Ideally a denormalized table would be available that featured the results of running those precedence rule calculations. But how to handle keeping this up-to-date? It would need to be recalculated any time a `_metadata` table in any of the attached databases had an update.

  This is a much larger problem - but one potential fix would be to use triggers to maintain a "version number" for the `_metadata` table - similar to SQLite's own built-in `schema_version` mechanism. Triggers could increment a counter any time a record in that table was added, deleted or updated.

  Such a mechanism would have applications outside of just this `_metadata` system. The ability to attach a version number to any table and have it automatically incremented when that table changes (via triggers) could help with all kinds of other Datasette-at-scale problems, including things like cached table counts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/210#issuecomment-753406744
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/210
node_id: MDEyOklzc3VlQ29tbWVudDc1MzQwNjc0NA==
created_at: 2021-01-02T00:02:39Z
updated_at: 2021-01-02T00:02:39Z
author_association: OWNER
body:
  It looks like https://github.com/ofajardo/pyreadr is a good library for this.

  I won't add this to `sqlite-utils` because it's quite a bulky dependency for a relatively small feature.

  Normally I'd write a `rdata-to-sqlite` tool similar to https://pypi.org/project/dbf-to-sqlite/ - but I'm actually working on a new plugin hook for Datasette that might be an even better fit for this. The idea is to allow Datasette plugins to define input formats - such as RData - which would then result in being able to import them on the command-line with `datasette insert my.db file.rdata` or by uploading a file through the Datasette web interface.

  That work is happening over here: https://github.com/simonw/datasette/issues/1160 - I'll close this issue in favour of a sometime-in-the-future `datasette-import-rdata` plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/212#issuecomment-753422324
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/212
node_id: MDEyOklzc3VlQ29tbWVudDc1MzQyMjMyNA==
created_at: 2021-01-02T03:00:34Z
updated_at: 2021-01-02T03:00:34Z
author_association: OWNER
body:
  Here's a prototype:
  ```python
  with db.conn:
      db.conn.executescript("""
  CREATE TABLE IF NOT EXISTS [_counts] ([table] TEXT PRIMARY KEY, [count] INTEGER DEFAULT 0);
  CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ai] AFTER INSERT ON [Street_Tree_List] BEGIN
      INSERT OR REPLACE INTO _counts
          VALUES ('Street_Tree_List', COALESCE(
              (SELECT count FROM _counts
                  WHERE [table]='Street_Tree_List'),
              0) + 1);
  END;
  CREATE TRIGGER IF NOT EXISTS [Street_Tree_List_counts_ad] AFTER DELETE ON [Street_Tree_List] BEGIN
      INSERT OR REPLACE INTO _counts
          VALUES ('Street_Tree_List', COALESCE(
              (SELECT count FROM _counts
                  WHERE [table]='Street_Tree_List'),
              0) - 1);
  END;
  INSERT OR REPLACE INTO _counts VALUES ('Street_Tree_List', (select count(*) from [Street_Tree_List]));
  """)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1168#issuecomment-753524779
issue_url: https://api.github.com/repos/simonw/datasette/issues/1168
node_id: MDEyOklzc3VlQ29tbWVudDc1MzUyNDc3OQ==
created_at: 2021-01-02T20:19:26Z
updated_at: 2021-01-02T20:19:26Z
author_association: OWNER
body:
  Idea: version the metadata scheme. If the table is called `_metadata_v1` it gives me a clear path to designing a new scheme in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1012#issuecomment-753531657
issue_url: https://api.github.com/repos/simonw/datasette/issues/1012
node_id: MDEyOklzc3VlQ29tbWVudDc1MzUzMTY1Nw==
created_at: 2021-01-02T21:25:36Z
updated_at: 2021-01-02T21:25:36Z
author_association: CONTRIBUTOR
body:
  Actually, on more research, I found out this is handled by the [trove-classifiers package](https://github.com/pypa/trove-classifiers/blob/master/src/trove_classifiers/__init__.py#L2) now, so it's just a one-liner pr instead of fire-up-a-docker-container-and-do-some-migrations

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753533775
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/213
node_id: MDEyOklzc3VlQ29tbWVudDc1MzUzMzc3NQ==
created_at: 2021-01-02T21:47:10Z
updated_at: 2021-01-02T21:47:10Z
author_association: OWNER
body:
  I'm going to skip virtual tables, which I can identify using this property: https://github.com/simonw/sqlite-utils/blob/1cad7fad3e7a5b734088f5cc545b69a055e636da/sqlite_utils/db.py#L720-L726

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/213#issuecomment-753535488
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/213
node_id: MDEyOklzc3VlQ29tbWVudDc1MzUzNTQ4OA==
created_at: 2021-01-02T22:03:48Z
updated_at: 2021-01-02T22:03:48Z
author_association: OWNER
body:
  I got this error while prototyping this:

      too many levels of trigger recursion

  It looks like that's because SQLite doesn't like triggers on a table that themselves then update that table - so I'm going to exclude the `_counts` table from this mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753544914
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/217
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU0NDkxNA==
created_at: 2021-01-02T23:47:42Z
updated_at: 2021-01-02T23:47:42Z
author_association: OWNER
body:
  https://github.com/simonw/sqlite-utils/blob/9a5c92b63e7917c93cc502478493c51c781b2ecc/sqlite_utils/db.py#L231-L239

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545381
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU0NTM4MQ==
created_at: 2021-01-02T23:52:52Z
updated_at: 2021-01-02T23:52:52Z
author_association: OWNER
body:
  Idea: a `db.cached_counts()` method that returns a dictionary of data from the `_counts` table. Call it with a list of tables to get back the counts for just those tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753545757
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU0NTc1Nw==
created_at: 2021-01-02T23:58:07Z
updated_at: 2021-01-02T23:58:07Z
author_association: OWNER
body:
  Thought: maybe there should be a `.reset_counts()` method too, for if the table gets out of date with the triggers.

  One way that could happen is if a table is dropped and recreated - the counts in the `_counts` table would likely no longer match the number of rows in that table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/217#issuecomment-753566184
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/217
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU2NjE4NA==
created_at: 2021-01-03T04:27:38Z
updated_at: 2021-01-03T04:27:38Z
author_association: OWNER
body:
  Documented here: https://sqlite-utils.datasette.io/en/latest/python-api.html#quoting-strings-for-use-in-sql

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567508
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU2NzUwOA==
created_at: 2021-01-03T04:48:17Z
updated_at: 2021-01-03T04:48:17Z
author_association: OWNER
body:
  Sorry for taking so long to review this!

  This approach looks great to me - being able to optionally pass a tuple anywhere the API currently expects a column is smart, and it's consistent with how the `pk=` parameter works elsewhere.

  There's just one problem I can see with this: the way it changes the `ForeignKey(...)` interface to always return a tuple for `.column` and `.other_column`, even if that tuple only contains a single item.

  This represents a breaking change to the existing API - any code that expects `ForeignKey.column` to be a single string (which is any code that has been written against that) will break.

  As such, I'd have to bump the major version of `sqlite-utils` to `4.0` in order to ship this.

  Ideally I'd like to make this change in a way that doesn't represent an API compatibility break. I need to think a bit harder about how that might be achieved.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567744
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU2Nzc0NA==
created_at: 2021-01-03T04:51:44Z
updated_at: 2021-01-03T04:51:44Z
author_association: OWNER
body:
  One way that this could avoid a breaking change would be to have `fk.column` and `fk.other_column` remain as strings for non-compound-foreign-keys, but turn into tuples for a compound foreign key.

  This is a bit of an ugly API design, and it could still break existing code that encounters a compound foreign key for the first time - but it would leave code working for the more common case of a non-compound-foreign-key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/203#issuecomment-753567932
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU2NzkzMg==
created_at: 2021-01-03T04:54:43Z
updated_at: 2021-01-03T04:54:43Z
author_association: OWNER
body:
  Another option: expand the `ForeignKey` object to have `.columns` and `.other_columns` properties in addition to the existing `.column` and `.other_column` properties. These new plural properties would always return a tuple, which would be a one-item tuple for a non-compound-foreign-key.

  The question then is what should `.column` and `.other_column` return for compound foreign keys?

  I'd be inclined to say they should return `None` - which would trigger errors in code that encounters a compound foreign key for the first time, but those errors would at least be a strong indicator as to what had gone wrong.

  We can label `.column` and `.other_column` as deprecated and then remove them in `sqlite-utils 4.0`.

  Since this would still be a breaking change in some minor edge-cases I'm thinking maybe 4.0 needs to happen in order to land this feature. I'm not opposed to doing that, I was just hoping it might be avoidable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/202#issuecomment-753568264
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/202
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU2ODI2NA==
created_at: 2021-01-03T05:00:24Z
updated_at: 2021-01-03T05:00:24Z
author_association: OWNER
body:
  I'm not going to implement this, because it actually needs several additional options that already exist on `sqlite-utils enable-fts`:
  ```
    --fts4                 Use FTS4
    --fts5                 Use FTS5
    --tokenize TEXT        Tokenizer to use, e.g. porter
    --create-triggers      Create triggers to update the FTS tables when the
                           parent table changes.
  ```
  I'd rather not add all four of those options to `sqlite-utils insert` just to support this shortcut.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1160#issuecomment-753568428
issue_url: https://api.github.com/repos/simonw/datasette/issues/1160
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU2ODQyOA==
created_at: 2021-01-03T05:02:32Z
updated_at: 2021-01-03T05:02:32Z
author_association: OWNER
body:
  Should this command include a `--fts` option for configuring full-text search on one-or-more columns?

  I thought about doing that for `sqlite-utils insert` in https://github.com/simonw/sqlite-utils/issues/202 and decided not to because of the need to include extra options covering the FTS version, porter stemming options and whether or not to create triggers.

  But maybe I can set sensible defaults for that with `datasette insert ... -f title -f body`? Worth thinking about a bit more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753570710
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU3MDcxMA==
created_at: 2021-01-03T05:29:56Z
updated_at: 2021-01-03T05:29:56Z
author_association: OWNER
body:
  I thought about using browser events, but they don't quite match the API that I'm looking to provide. In particular, the great thing about Pluggy is that if you have multiple handlers registered for a specific plugin hook each of those handlers can return a value, and Pluggy will combine those values into a list of replies.

  This is great for things like plugin hooks that add extra menu items - each plugin can return a menu item (maybe as a label/URL/click-callback object) and the calling code can then add all of those items to the menu. See https://docs.datasette.io/en/stable/plugin_hooks.html#table-actions-datasette-actor-database-table for a Python example.

  I'm on the fence about relying on JavaScript modules. I need to think about browser compatibility for them - but I'm already commited to requiring support for `() => {}` arrow functions so maybe I'm committed to module support too already?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753587963
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzU4Nzk2Mw==
created_at: 2021-01-03T09:02:50Z
updated_at: 2021-01-03T10:00:05Z
author_association: NONE
body:
  > but I'm already commited to requiring support for () => {} arrow functions

  Don't think you are :) (e.g. gzipped, using arrow functions in my example saves 2 bytes over spelling out function). On FMS, past month, looking at popular browsers, looks like we'd have 95.41% arrow support, 94.19% module support, and 4.58% (mostly IE9/IE11/Safari 9) supporting neither.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753600999
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzYwMDk5OQ==
created_at: 2021-01-03T11:11:21Z
updated_at: 2021-01-03T11:11:21Z
author_association: NONE
body:
  With regards to JS/Browser events, given your example of menu items that plugins could add, I could imagine this code to work:

  ```js
  // as part of datasette
  datasette.events.AddMenuItem = 'DatasetteAddMenuItemEvent';
  document.addEventListener(datasette.events.AddMenuItem, (e) => {
    // do whatever is needed to add the menu item. Data comes from `e`
    alert(e.title + ' ' + e.link);
  });

  // as part of a plugin
  const event = new Event(datasette.events.AddMenuItem, {link: '/foo/bar', title: 'Go somewhere'});
  Document.dispatchEvent(event)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1169#issuecomment-753653033
issue_url: https://api.github.com/repos/simonw/datasette/issues/1169
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY1MzAzMw==
created_at: 2021-01-03T17:52:53Z
updated_at: 2021-01-03T17:52:53Z
author_association: OWNER
body:
  Oh that's so frustrating! I was worried about that - I spotted a few runs that seemed faster and hoped that it meant that the package was coming out of the `~/.npm` cache, but evidently that's not the case.

  You've convinced me that Datasette itself should have a `package.json` - the Dependabot argument is a really good one.

  But... I'd really love to figure out a general pattern for using `npx` scripts in GitHub Actions workflows in a cache-friendly way. I have plenty of other projects that I'd love to run Prettier or Uglify or `puppeteer-cli` in without adding a `package.json` to them.

  Any ideas? The best I can think of is for the workflow itself to write out a `package.json` file (using `echo '{ ... }' > package.json`) as part of the run - that way the cache should work (I think) but I don't get a misleading `package.json` file sitting in the repo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1169#issuecomment-753657180
issue_url: https://api.github.com/repos/simonw/datasette/issues/1169
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY1NzE4MA==
created_at: 2021-01-03T18:23:30Z
updated_at: 2021-01-03T18:23:30Z
author_association: OWNER
body:
  Also welcome in that PR would be a bit of documentation for contributors, see #1167 - but no problem if you leave that out, I'm happy to add it later.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/206#issuecomment-753659260
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/206
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY1OTI2MA==
created_at: 2021-01-03T18:42:01Z
updated_at: 2021-01-03T18:42:01Z
author_association: OWNER
body:
  ```
  % sqlite-utils insert blah.db blah global_power_plant_database.csv
  Error: Invalid JSON - use --csv for CSV or --tsv for TSV files
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660379
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY2MDM3OQ==
created_at: 2021-01-03T18:50:15Z
updated_at: 2021-01-03T18:50:15Z
author_association: OWNER
body:
  ```python
      def cached_counts(self, tables=None):
          sql = "select [table], count from {}".format(self._counts_table_name)
          if tables:
              sql += " where [table] in ({})".format(", ".join("?" for table in tables))
          return {r[0]: r[1] for r in self.execute(sql, tables).fetchall()}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753660814
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY2MDgxNA==
created_at: 2021-01-03T18:53:05Z
updated_at: 2021-01-03T18:53:05Z
author_association: OWNER
body:
  Here's the current `.count` property: https://github.com/simonw/sqlite-utils/blob/036ec6d32313487527c66dea613a3e7118b97459/sqlite_utils/db.py#L597-L609

  It's implemented on `Queryable` which means it's available on both `Table` and `View` - the optimization doesn't make sense for views.

  I'm a bit cautious about making that property so much more complex. In order to decide if it should try the `_counts` table first it needs to know:

  - Should it be trusting the counts? I'm thinking a `.should_trust_counts` property on `Database` which defaults to `True` would be good - then advanced users can turn that off if they know the counts should not be trusted.
  - Does the `_counts` table exist?
  - Are the triggers defined?

  Then it can do the query, and if the query fails it can fall back on the `count(*)`. That's quite a lot of extra activity though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661158
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY2MTE1OA==
created_at: 2021-01-03T18:55:16Z
updated_at: 2021-01-03T18:55:16Z
author_association: OWNER
body:
  Alternative implementation: provided `db.should_trust_counts` is `True`, try running the query:
  ```sql
  select count from _counts where [table] = ?
  ```
  If the query fails to return a result OR throws an error because the table doesn't exist, run the `count(*)` query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753661292
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY2MTI5Mg==
created_at: 2021-01-03T18:56:06Z
updated_at: 2021-01-03T18:56:23Z
author_association: OWNER
body:
  Another option: on creation of the `Database()` object, check to see if the `_counts` table exists and use that as the default for a `use_counts_table` property. Also flip that property to `True` if the user calls `.enable_counts()` at any time.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753665521
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY2NTUyMQ==
created_at: 2021-01-03T19:31:33Z
updated_at: 2021-01-03T19:31:33Z
author_association: OWNER
body:
  I'm having second thoughts about this being the default behaviour. It's pretty weird. I feel like HUGE databases that need this are rare, so having it on by default doesn't make sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/215#issuecomment-753668099
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/215
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY2ODA5OQ==
created_at: 2021-01-03T19:55:53Z
updated_at: 2021-01-03T19:55:53Z
author_association: OWNER
body:
  So if you instantiate the `Database()` constructor with `use_counts_table=True` any access to the `.count` properties will go through this table - otherwise regular `count(*)` queries will be executed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753670833
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/219
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY3MDgzMw==
created_at: 2021-01-03T20:20:54Z
updated_at: 2021-01-03T20:20:54Z
author_association: OWNER
body:
  This is a little tricky. We should assume that the existing values in the `_counts` table cannot be trusted at all when this method is called - so we should probably clear that table entirely and then re-populate it.

  But that means we need to figure out which tables in the database have the counts triggers defined.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/219#issuecomment-753671235
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/219
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY3MTIzNQ==
created_at: 2021-01-03T20:25:10Z
updated_at: 2021-01-03T20:25:10Z
author_association: OWNER
body:
  To detect tables, look at the names of the triggers - `{table}{counts_table}_insert` and `{table}{counts_table}_delete`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-753690280
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1MzY5MDI4MA==
created_at: 2021-01-03T23:13:30Z
updated_at: 2021-01-03T23:13:30Z
author_association: OWNER
body:
  Oh that's interesting, I hadn't thought about plugins firing events - just responding to events fired by the rest of the application.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1170#issuecomment-754002859
issue_url: https://api.github.com/repos/simonw/datasette/issues/1170
node_id: MDEyOklzc3VlQ29tbWVudDc1NDAwMjg1OQ==
created_at: 2021-01-04T14:22:52Z
updated_at: 2021-01-04T14:22:52Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=h1) Report
  > Merging [#1170](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=desc) (a5761cc) into [main](https://codecov.io/gh/simonw/datasette/commit/1e8fa3ac7cb2d6e516c47c306c86ed2334fc3dc0?el=desc) (1e8fa3a) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1170/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1170   +/-   ##
  =======================================
    Coverage   91.55%   91.55%           
  =======================================
    Files          32       32           
    Lines        3932     3932           
  =======================================
    Hits         3600     3600           
    Misses        332      332           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=footer). Last update [1e8fa3a...a5761cc](https://codecov.io/gh/simonw/datasette/pull/1170?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1170#issuecomment-754004715
issue_url: https://api.github.com/repos/simonw/datasette/issues/1170
node_id: MDEyOklzc3VlQ29tbWVudDc1NDAwNDcxNQ==
created_at: 2021-01-04T14:25:44Z
updated_at: 2021-01-04T14:25:44Z
author_association: CONTRIBUTOR
body:
  I was going to re-add the filter to only run Prettier when there have been changes in `datasette/static` but that would mean it wouldn't run when the package is updated. That plus the fact that [the last run of the job took only 8 seconds](https://github.com/benpickles/datasette/runs/1640121514) is why I decided not to re-add the filter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1169#issuecomment-754007242
issue_url: https://api.github.com/repos/simonw/datasette/issues/1169
node_id: MDEyOklzc3VlQ29tbWVudDc1NDAwNzI0Mg==
created_at: 2021-01-04T14:29:57Z
updated_at: 2021-01-04T14:29:57Z
author_association: CONTRIBUTOR
body:
  I somewhat share your reluctance to add a package.json to seemingly every project out there but ultimately if they're project dependencies it's important they're managed within the codebase.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-754181647
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE4MTY0Nw==
created_at: 2021-01-04T19:52:40Z
updated_at: 2021-01-04T19:52:40Z
author_association: NONE
body:
  I was thinking JavaScript plugins going with server side template extensions custom HTML. Attach my own widgets on there and listen for Datasette events to refresh when user interacts with main UI. Like a map view or table that updates according to selected column. There's certainly other ways to look at this. Perhaps you could list possible hooks or high level design doc on what would be possible with the plugin system?

  Re: modules. I would like to see modules supported at least in development. The developer experience is so much better than what JavaScript coding has been in the past. With large parts of NPM at your disposal I’d imagine even less experienced coder can whisk a custom plugin in no time. Proper production build system (like one you get with Pika or Parcel) could package everything up into bundles that older browsers can understand. Though that does come with performance and size penalties alongside the added complexity. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1111#issuecomment-754184287
issue_url: https://api.github.com/repos/simonw/datasette/issues/1111
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE4NDI4Nw==
created_at: 2021-01-04T19:57:53Z
updated_at: 2021-01-04T19:57:53Z
author_association: OWNER
body:
  Relevant new feature in sqlite-utils: the ability to use triggers to maintain fast counts. This optimization could help a lot here. https://github.com/simonw/sqlite-utils/issues/212

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/913#issuecomment-754187326
issue_url: https://api.github.com/repos/simonw/datasette/issues/913
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE4NzMyNg==
created_at: 2021-01-04T20:03:50Z
updated_at: 2021-01-04T20:03:50Z
author_association: OWNER
body:
  I renamed `--config` to `--setting` and changed it to work like this:

      datasette --setting sql_time_limit_ms 1000

  Note the lack of colons.

  This actually makes colons cleaner to use for plugins - I could support this:

      datasette --setting datasette-insert:unsafe 1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/221#issuecomment-754191699
issue_url: https://api.github.com/repos/simonw/datasette/issues/221
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE5MTY5OQ==
created_at: 2021-01-04T20:12:14Z
updated_at: 2021-01-04T20:12:14Z
author_association: OWNER
body:
  I'm going to close this. Plugins can register their own CLI tools (see https://github.com/simonw/click-app) if they need to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/102#issuecomment-754192267
issue_url: https://api.github.com/repos/simonw/datasette/issues/102
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE5MjI2Nw==
created_at: 2021-01-04T20:13:19Z
updated_at: 2021-01-04T20:13:19Z
author_association: OWNER
body:
  I'm more likely to do Lambda than Elastic Beanstalk, especially now the size limit for Lambdas has been increased as part of their support for Docker.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/371#issuecomment-754192873
issue_url: https://api.github.com/repos/simonw/datasette/issues/371
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE5Mjg3Mw==
created_at: 2021-01-04T20:14:28Z
updated_at: 2021-01-04T20:14:28Z
author_association: OWNER
body:
  Now that Digital Ocean has App Platform this is less necessary, especially since the documentation covers how to use App Platform here: https://docs.datasette.io/en/stable/deploying.html#deploying-using-buildpacks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/436#issuecomment-754193501
issue_url: https://api.github.com/repos/simonw/datasette/issues/436
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE5MzUwMQ==
created_at: 2021-01-04T20:15:41Z
updated_at: 2021-01-04T20:15:41Z
author_association: OWNER
body:
  Sadly `publish.datasettes.com` was broken by changes to Zeit, and I don't think I'll be bringing it back.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/668#issuecomment-754194996
issue_url: https://api.github.com/repos/simonw/datasette/issues/668
node_id: MDEyOklzc3VlQ29tbWVudDc1NDE5NDk5Ng==
created_at: 2021-01-04T20:18:39Z
updated_at: 2021-01-04T20:18:39Z
author_association: OWNER
body:
  I fixed this in #1115 - you can run `--load-extension=spatialite` now and it will look for the extension in common places.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-754210356
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc1NDIxMDM1Ng==
created_at: 2021-01-04T20:49:05Z
updated_at: 2021-01-04T20:49:05Z
author_association: NONE
body:
  For reasons [I've written about elsewhere](https://blog.carlmjohnson.net/post/2020/time-to-kill-ie11/), I'm in favor of modules. It has several beneficial effects. One, old browsers just ignore it all together. Two, if you include the same plain script on the page more than once, it will be executed twice, but if you include the same module script on a page twice, it will only execute once. Three, you get a module local namespace, instead of having to use the global window namespace or a function private namespace.

  OTOH, if you are going to use an old style script, the code from before isn't ideal, because you wipe out your registry if the script it included more than once. Also you may as well use object methods and splat arguments.

  The event based architecture probably makes more sense though. Just make up some event names prefixed with `datasette:` and listen for them on the root. The only concern with that approach is it can sometimes be tricky to make sure your plugins are run after datasette has run. Maybe 

  ```js
  function mycallback(){
    // whatever
  }

  if (window.datasette) {
    window.datasette.init(mycallback);
  } else {
    document.addEventListener('datasette:init', mycallback);
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-754215392
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDc1NDIxNTM5Mg==
created_at: 2021-01-04T20:59:20Z
updated_at: 2021-01-04T21:03:14Z
author_association: OWNER
body:
  Updated `pyinstaller` recipe - lots of hidden imports needed now:
  ```
  pip install wheel
  pip install datasette pyinstaller

  BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))') \
      pyinstaller -F \
          --add-data "$BASE/templates:datasette/templates" \
          --add-data "$BASE/static:datasette/static" \
          --hidden-import datasette.publish \
          --hidden-import datasette.publish.heroku \
          --hidden-import datasette.publish.cloudrun \
          --hidden-import datasette.facets \
          --hidden-import datasette.sql_functions \
          --hidden-import datasette.actor_auth_cookie \
          --hidden-import datasette.default_permissions \
          --hidden-import datasette.default_magic_parameters \
          --hidden-import datasette.blob_renderer \
          --hidden-import datasette.default_menu_links \
          --hidden-import uvicorn \
          --hidden-import uvicorn.logging \
          --hidden-import uvicorn.loops \
          --hidden-import uvicorn.loops.auto \
          --hidden-import uvicorn.protocols \
          --hidden-import uvicorn.protocols.http \
          --hidden-import uvicorn.protocols.http.auto \
          --hidden-import uvicorn.protocols.websockets \
          --hidden-import uvicorn.protocols.websockets.auto \
          --hidden-import uvicorn.lifespan \
          --hidden-import uvicorn.lifespan.on \
          $(which datasette)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-754215793
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDc1NDIxNTc5Mw==
created_at: 2021-01-04T21:00:14Z
updated_at: 2021-01-04T21:00:14Z
author_association: OWNER
body:
  ```
  (pyinstaller-datasette) pyinstaller-datasette % file dist/datasette
  dist/datasette: Mach-O 64-bit executable x86_64
  (pyinstaller-datasette) pyinstaller-datasette % ls -lah dist/datasette
  -rwxr-xr-x  1 simon  wheel   8.0M Jan  4 12:58 dist/datasette
  ```
  I'm surprised it's only 8MB!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-754219002
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDc1NDIxOTAwMg==
created_at: 2021-01-04T21:06:49Z
updated_at: 2021-01-04T21:22:27Z
author_association: OWNER
body:
  Works on Linux/Ubuntu too, except I had to do `export BASE=` on a separate line. I also did this:
  ```
  apt-get install python3 python3-venv
  python3 -m venv pyinstaller-venv
  source pyinstaller-venv/bin/activate
  pip install wheel
  pip install datasette pyinstaller

  export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))')

  pyinstaller -F \
      --add-data "$DATASETTE_BASE/templates:datasette/templates" \
      --add-data "$DATASETTE_BASE/static:datasette/static" \
      --hidden-import datasette.publish \
      --hidden-import datasette.publish.heroku \
      --hidden-import datasette.publish.cloudrun \
      --hidden-import datasette.facets \
      --hidden-import datasette.sql_functions \
      --hidden-import datasette.actor_auth_cookie \
      --hidden-import datasette.default_permissions \
      --hidden-import datasette.default_magic_parameters \
      --hidden-import datasette.blob_renderer \
      --hidden-import datasette.default_menu_links \
      --hidden-import uvicorn \
      --hidden-import uvicorn.logging \
      --hidden-import uvicorn.loops \
      --hidden-import uvicorn.loops.auto \
      --hidden-import uvicorn.protocols \
      --hidden-import uvicorn.protocols.http \
      --hidden-import uvicorn.protocols.http.auto \
      --hidden-import uvicorn.protocols.websockets \
      --hidden-import uvicorn.protocols.websockets.auto \
      --hidden-import uvicorn.lifespan \
      --hidden-import uvicorn.lifespan.on \
      $(which datasette)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-754227543
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDc1NDIyNzU0Mw==
created_at: 2021-01-04T21:23:13Z
updated_at: 2021-01-04T21:23:13Z
author_association: OWNER
body:
  ```
  (pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# dist/datasette --get /-/databases.json
  [{"name": ":memory:", "path": null, "size": 0, "is_mutable": true, "is_memory": true, "hash": null}]
  (pyinstaller-venv) root@dogsheep:/tmp/pyinstaller-venv# ls -lah dist/datasette 
  -rwxr-xr-x 1 root root 8.9M Jan  4 21:05 dist/datasette
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-754229977
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDc1NDIyOTk3Nw==
created_at: 2021-01-04T21:28:01Z
updated_at: 2021-01-04T21:28:01Z
author_association: OWNER
body:
  As an experiment, I put the macOS one in a zip file and attached it to the latest release:

  ```
  mkdir datasette-0.53-macos-binary
  cp dist/datasette datasette-0.53-macos-binary
  zip -r datasette-0.53-macos-binary.zip datasette-0.53-macos-binary
  ```

  It's available here: https://github.com/simonw/datasette/releases/tag/0.53 - download URL is https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/93#issuecomment-754233960
issue_url: https://api.github.com/repos/simonw/datasette/issues/93
node_id: MDEyOklzc3VlQ29tbWVudDc1NDIzMzk2MA==
created_at: 2021-01-04T21:35:37Z
updated_at: 2021-01-04T21:35:37Z
author_association: OWNER
body:
  I tested it by running a `tmate` session on the GitHub macOS machines, and it worked!
  ```
  Mac-1609795972770:tmp runner$ wget 'https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip'
  --2021-01-04 21:34:10--  https://github.com/simonw/datasette/releases/download/0.53/datasette-0.53-macos-binary.zip
  Resolving github.com (github.com)... 140.82.114.4
  Connecting to github.com (github.com)|140.82.114.4|:443... connected.
  HTTP request sent, awaiting response... 302 Found
  Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream [following]
  --2021-01-04 21:34:14--  https://github-production-release-asset-2e65be.s3.amazonaws.com/107914493/74658700-4e90-11eb-8f3b-ee77e6dfad90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210104T213414Z&X-Amz-Expires=300&X-Amz-Signature=6f3c54211077092553590b33a7c36cd052895c9d4619607ad1df094782f64acf&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=107914493&response-content-disposition=attachment%3B%20filename%3Ddatasette-0.53-macos-binary.zip&response-content-type=application%2Foctet-stream
  Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.43.164
  Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.43.164|:443... connected.
  HTTP request sent, awaiting response... 200 OK
  Length: 8297283 (7.9M) [application/octet-stream]
  Saving to: ‘datasette-0.53-macos-binary.zip’

  datasette-0.53-maco 100%[===================>]   7.91M  --.-KB/s    in 0.1s    

  2021-01-04 21:34:14 (73.4 MB/s) - ‘datasette-0.53-macos-binary.zip’ saved [8297283/8297283]

  Mac-1609795972770:tmp runner$ unzip datasette-0.53-macos-binary.zip 
  Archive:  datasette-0.53-macos-binary.zip
     creating: datasette-0.53-macos-binary/
    inflating: datasette-0.53-macos-binary/datasette  
  Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --help
  Usage: datasette [OPTIONS] COMMAND [ARGS]...

    Datasette!

  Options:
    --version  Show the version and exit.
    --help     Show this message and exit.

  Commands:
    serve*     Serve up specified SQLite database files with a web UI
    inspect
    install    Install Python packages - e.g.
    package    Package specified SQLite files into a new datasette Docker...
    plugins    List currently available plugins
    publish    Publish specified SQLite database files to the internet along...
    uninstall  Uninstall Python packages (e.g.
  Mac-1609795972770:tmp runner$ datasette-0.53-macos-binary/datasette --get /-/versions.json
  {"python": {"version": "3.9.1", "full": "3.9.1 (default, Dec 10 2020, 10:36:35) \n[Clang 12.0.0 (clang-1200.0.32.27)]"}, "datasette": {"version": "0.53"}, "asgi": "3.0", "uvicorn": "0.13.3", "sqlite": {"version": "3.34.0", "fts_versions": ["FTS5", "FTS4", "FTS3"], "extensions": {"json1": null}, "compile_options": ["COMPILER=clang-12.0.0", "ENABLE_COLUMN_METADATA", "ENABLE_FTS3", "ENABLE_FTS3_PARENTHESIS", "ENABLE_FTS4", "ENABLE_FTS5", "ENABLE_GEOPOLY", "ENABLE_JSON1", "ENABLE_PREUPDATE_HOOK", "ENABLE_RTREE", "ENABLE_SESSION", "MAX_VARIABLE_NUMBER=250000", "THREADSAFE=1"]}}
  Mac-1609795972770:tmp runner$ 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1152#issuecomment-754285588
issue_url: https://api.github.com/repos/simonw/datasette/issues/1152
node_id: MDEyOklzc3VlQ29tbWVudDc1NDI4NTU4OA==
created_at: 2021-01-04T23:34:30Z
updated_at: 2021-01-04T23:34:30Z
author_association: OWNER
body:
  I think the way to do this is to have a new plugin hook that returns two SQL where clauses: one returning a list of resources that the user should be able to access (the allow-list) and one returning a list of resources they are explicitly forbidden from accessing (the deny-list). Either of these can be blank.

  Datasette can then combine those into a full SQL query and use it to answer the question "show me a list of resources that the user is allowed to perform action X on". It can also answer the existing question, "is user X allowed to perform action Y on resource Z"?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1171#issuecomment-754286618
issue_url: https://api.github.com/repos/simonw/datasette/issues/1171
node_id: MDEyOklzc3VlQ29tbWVudDc1NDI4NjYxOA==
created_at: 2021-01-04T23:37:45Z
updated_at: 2021-01-04T23:37:45Z
author_association: OWNER
body:
  https://github.com/actions/virtual-environments/issues/1820#issuecomment-719549887 looks useful - not sure if those notes are for iOS or macOS though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1171#issuecomment-754286783
issue_url: https://api.github.com/repos/simonw/datasette/issues/1171
node_id: MDEyOklzc3VlQ29tbWVudDc1NDI4Njc4Mw==
created_at: 2021-01-04T23:38:18Z
updated_at: 2021-01-04T23:38:18Z
author_association: OWNER
body:
  Oh wow maybe I need to Notarize it too? https://developer.apple.com/documentation/xcode/notarizing_macos_software_before_distribution

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1171#issuecomment-754287882
issue_url: https://api.github.com/repos/simonw/datasette/issues/1171
node_id: MDEyOklzc3VlQ29tbWVudDc1NDI4Nzg4Mg==
created_at: 2021-01-04T23:40:10Z
updated_at: 2021-01-04T23:42:32Z
author_association: OWNER
body:
  This looks VERY useful: https://github.com/mitchellh/gon - " Sign, notarize, and package macOS CLI tools and applications written in any language. Available as both a CLI and a Go library."

  And it installs like this:

      brew install mitchellh/gon/gon

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1171#issuecomment-754295380
issue_url: https://api.github.com/repos/simonw/datasette/issues/1171
node_id: MDEyOklzc3VlQ29tbWVudDc1NDI5NTM4MA==
created_at: 2021-01-04T23:54:32Z
updated_at: 2021-01-04T23:54:32Z
author_association: OWNER
body:
  https://github.com/search?l=YAML&q=gon+json&type=Code reveals some examples of people using `gon` in workflows.

  These look useful:

  * https://github.com/coherence/hub-server/blob/3b7e9c7c5bce9e244b14b854f1f89d66f53a5a39/.github/workflows/release_build.yml
  * https://github.com/simoncozens/pilcrow/blob/5abc145e7fb9577086afe47b48fd730cb8195386/.github/workflows/buildapp.yaml

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1171#issuecomment-754296761
issue_url: https://api.github.com/repos/simonw/datasette/issues/1171
node_id: MDEyOklzc3VlQ29tbWVudDc1NDI5Njc2MQ==
created_at: 2021-01-04T23:55:44Z
updated_at: 2021-01-04T23:55:44Z
author_association: OWNER
body:
  Bit uncomfortable that it looks like you need to include your Apple ID username and password in the CI configuration to do this. I'll use GitHub Secrets for this but I don't like it - I'll definitely setup a dedicated code signing account that's not my access-to-everything AppleID for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1173#issuecomment-754463845
issue_url: https://api.github.com/repos/simonw/datasette/issues/1173
node_id: MDEyOklzc3VlQ29tbWVudDc1NDQ2Mzg0NQ==
created_at: 2021-01-05T07:41:43Z
updated_at: 2021-01-05T07:41:43Z
author_association: OWNER
body:
  https://github.com/oleksis/pyinstaller-manylinux looks useful, via https://twitter.com/oleksis/status/1346341987876823040

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1167#issuecomment-754619930
issue_url: https://api.github.com/repos/simonw/datasette/issues/1167
node_id: MDEyOklzc3VlQ29tbWVudDc1NDYxOTkzMA==
created_at: 2021-01-05T12:57:57Z
updated_at: 2021-01-05T12:57:57Z
author_association: CONTRIBUTOR
body:
  Not sure where exactly to put the actual docs (presumably somewhere in [docs/contributing.rst](https://github.com/simonw/datasette/blob/main/docs/contributing.rst)) but I've made a slight change to make it easier to run locally (copying [the approach in excalidraw](https://github.com/excalidraw/excalidraw/blob/ade2565f497243a5e428f4906d8ed80c872fd981/package.json#L90-L94)): https://github.com/simonw/datasette/compare/main...benpickles:prettier-docs

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754721153
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54
node_id: MDEyOklzc3VlQ29tbWVudDc1NDcyMTE1Mw==
created_at: 2021-01-05T15:51:09Z
updated_at: 2021-01-05T15:51:09Z
author_association: CONTRIBUTOR
body:
  Correction: the failure is on `lists-member.js` (I was thrown by the `block` variable name, but that's just a coincidence)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-754728696
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55
node_id: MDEyOklzc3VlQ29tbWVudDc1NDcyODY5Ng==
created_at: 2021-01-05T16:02:55Z
updated_at: 2021-01-05T16:02:55Z
author_association: CONTRIBUTOR
body:
  This now works for me, though I'm entirely ensure if it's a just-my-export thing or a wider issue. Also, this doesn't contain any tests. So I'm not sure if there's more work to be done here, or if this is good enough.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-754729035
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54
node_id: MDEyOklzc3VlQ29tbWVudDc1NDcyOTAzNQ==
created_at: 2021-01-05T16:03:29Z
updated_at: 2021-01-05T16:03:29Z
author_association: CONTRIBUTOR
body:
  I was able to fix this, at least enough to get _my_ archive to import. Not sure if there's more work to be done here or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1171#issuecomment-754911290
issue_url: https://api.github.com/repos/simonw/datasette/issues/1171
node_id: MDEyOklzc3VlQ29tbWVudDc1NDkxMTI5MA==
created_at: 2021-01-05T21:31:15Z
updated_at: 2021-01-05T21:31:15Z
author_association: NONE
body:
  We did this for [Sno](https://sno.earth) under macOS — it's a PyInstaller binary/setup which uses [Packages](http://s.sudre.free.fr/Software/Packages/about.html) for packaging.

  * [Building & Signing](https://github.com/koordinates/sno/blob/master/platforms/Makefile#L67-L95)
  * [Packaging & Notarizing](https://github.com/koordinates/sno/blob/master/platforms/Makefile#L121-L215)
  * [Github Workflow](https://github.com/koordinates/sno/blob/master/.github/workflows/build.yml#L228-L269) has the CI side of it

  FYI (if you ever get to it) for Windows you need to get a code signing certificate. And if you want automated CI, you'll want to get an "EV CodeSigning for HSM" certificate from GlobalSign, which then lets you put the certificate into Azure Key Vault. Which you can use with [azuresigntool](https://github.com/vcsjones/AzureSignTool) to sign your code & installer. (Non-EV certificates are a waste of time, the user still gets big warnings at install time).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-754951786
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: MDEyOklzc3VlQ29tbWVudDc1NDk1MTc4Ng==
created_at: 2021-01-05T22:56:27Z
updated_at: 2021-01-05T22:56:43Z
author_association: OWNER
body:
  Idea: introduce a `@documented` decorator which marks specific functions as part of the public, documented API. The unit tests can then confirm that anything with that decorator is both documented and tested.
  ```python
  @documented
  def escape_css_string(s):
      return _css_re.sub(
          lambda m: "\\" + (f"{ord(m.group()):X}".zfill(6)),
          s.replace("\r\n", "\n"),
      )
  ```
  Or maybe `@public`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-754952040
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: MDEyOklzc3VlQ29tbWVudDc1NDk1MjA0MA==
created_at: 2021-01-05T22:57:09Z
updated_at: 2021-01-05T22:57:09Z
author_association: OWNER
body:
  It might be neater to move all of the non-public functions into a separate module - `datasette.utils.internal` perhaps.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-754957378
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: MDEyOklzc3VlQ29tbWVudDc1NDk1NzM3OA==
created_at: 2021-01-05T23:12:03Z
updated_at: 2021-01-05T23:12:03Z
author_association: OWNER
body:
  This needs to be done for Datasette 1.0. At the very least I need to ensure it's clear that `datasette.utils` is not part of the public API unless explicitly marked as such.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-754958610
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc1NDk1ODYxMA==
created_at: 2021-01-05T23:15:24Z
updated_at: 2021-01-05T23:15:24Z
author_association: OWNER
body:
  https://latest-with-plugins.datasette.io/fixtures/roadside_attraction_characteristics/1.json-preview returns a 500 error at the moment - a KeyError on 'filtered_table_rows_count'.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-755128038
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: MDEyOklzc3VlQ29tbWVudDc1NTEyODAzOA==
created_at: 2021-01-06T07:10:22Z
updated_at: 2021-01-06T07:10:22Z
author_association: OWNER
body:
  Yet another use-case for this: I want to be able to stream newline-delimited JSON in order to better import into Pandas:

      pandas.read_json("https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_nl=on", lines=True)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-755133937
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: MDEyOklzc3VlQ29tbWVudDc1NTEzMzkzNw==
created_at: 2021-01-06T07:25:48Z
updated_at: 2021-01-06T07:26:43Z
author_association: OWNER
body:
  Idea: instead of returning a dictionary, `register_output_renderer` could return an object. The object could have the following properties:

  - `.extension` - the extension to use
  - `.can_render(...)` - says if it can render this
  - `.can_stream(...)` - says if streaming is supported
  - `async .stream_rows(rows_iterator, send)` - method that loops through all rows and uses `send` to send them to the response in the correct format

  I can then deprecate the existing `dict` return type for 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1101#issuecomment-755134771
issue_url: https://api.github.com/repos/simonw/datasette/issues/1101
node_id: MDEyOklzc3VlQ29tbWVudDc1NTEzNDc3MQ==
created_at: 2021-01-06T07:28:01Z
updated_at: 2021-01-06T07:28:01Z
author_association: OWNER
body:
  With this structure it will become possible to stream non-newline-delimited JSON array-of-objects too - the `stream_rows()` method could output `[` first, then each row followed by a comma, then `]` after the very last row.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1178#issuecomment-755156606
issue_url: https://api.github.com/repos/simonw/datasette/issues/1178
node_id: MDEyOklzc3VlQ29tbWVudDc1NTE1NjYwNg==
created_at: 2021-01-06T08:21:49Z
updated_at: 2021-01-06T08:21:49Z
author_association: OWNER
body:
  https://github.com/simonw/datasette-export-notebook/blob/aec398eab4f34791d240d7bc47b6eec575b357be/datasette_export_notebook/__init__.py#L18-L23

  Maybe this is a bug in `datasette.absolute_url`? Perhaps it doesn't take the scheme into account.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1178#issuecomment-755157066
issue_url: https://api.github.com/repos/simonw/datasette/issues/1178
node_id: MDEyOklzc3VlQ29tbWVudDc1NTE1NzA2Ng==
created_at: 2021-01-06T08:22:47Z
updated_at: 2021-01-06T08:22:47Z
author_association: OWNER
body:
  Weird... https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/app.py#L609-L613

  ```python
      def absolute_url(self, request, path):
          url = urllib.parse.urljoin(request.url, path)
          if url.startswith("http://") and self.setting("force_https_urls"):
              url = "https://" + url[len("http://") :]
          return url
  ```
  That looks like it should work. Needs more digging.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1178#issuecomment-755157732
issue_url: https://api.github.com/repos/simonw/datasette/issues/1178
node_id: MDEyOklzc3VlQ29tbWVudDc1NTE1NzczMg==
created_at: 2021-01-06T08:24:12Z
updated_at: 2021-01-06T08:24:12Z
author_association: OWNER
body:
  https://latest-with-plugins.datasette.io/fixtures/sortable.json has the bug too - the `next_url` is `http://` when it should be `https://`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1176#issuecomment-755159583
issue_url: https://api.github.com/repos/simonw/datasette/issues/1176
node_id: MDEyOklzc3VlQ29tbWVudDc1NTE1OTU4Mw==
created_at: 2021-01-06T08:28:20Z
updated_at: 2021-01-06T08:28:20Z
author_association: OWNER
body:
  I used `from datasette.utils import path_with_format` in https://github.com/simonw/datasette-export-notebook/blob/0.1/datasette_export_notebook/__init__.py just now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1178#issuecomment-755160187
issue_url: https://api.github.com/repos/simonw/datasette/issues/1178
node_id: MDEyOklzc3VlQ29tbWVudDc1NTE2MDE4Nw==
created_at: 2021-01-06T08:29:35Z
updated_at: 2021-01-06T08:29:35Z
author_association: OWNER
body:
  https://latest-with-plugins.datasette.io/-/asgi-scope

  ```
  {'asgi': {'spec_version': '2.1', 'version': '3.0'},
   'client': ('169.254.8.129', 54971),
   'headers': [(b'host', b'latest-with-plugins.datasette.io'),
               (b'user-agent',
                b'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko'
                b'/20100101 Firefox/84.0'),
               (b'accept',
                b'text/html,application/xhtml+xml,application/xml;q=0.9,image/'
                b'webp,*/*;q=0.8'),
               (b'accept-language', b'en-US,en;q=0.5'),
               (b'dnt', b'1'),
               (b'cookie',
                b'_ga_LL6M7BK6D4=GS1.1.1609886546.49.1.1609886923.0; _ga=GA1.1'
                b'.894633707.1607575712'),
               (b'upgrade-insecure-requests', b'1'),
               (b'x-client-data', b'CgSL6ZsV'),
               (b'x-cloud-trace-context',
                b'e776af843c657d2a3da28a73b726e6fe/14187666787557102189;o=1'),
               (b'x-forwarded-for', b'148.64.98.14'),
               (b'x-forwarded-proto', b'https'),
               (b'forwarded', b'for="148.64.98.14";proto=https'),
               (b'accept-encoding', b'gzip, deflate, br'),
               (b'content-length', b'0')],
   'http_version': '1.1',
   'method': 'GET',
   'path': '/-/asgi-scope',
   'query_string': b'',
   'raw_path': b'/-/asgi-scope',
   'root_path': '',
   'scheme': 'http',
   'server': ('169.254.8.130', 8080),
   'type': 'http'}
  ```
  Note the `'scheme': 'http'` but also the `(b'x-forwarded-proto', b'https')`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1179#issuecomment-755161574
issue_url: https://api.github.com/repos/simonw/datasette/issues/1179
node_id: MDEyOklzc3VlQ29tbWVudDc1NTE2MTU3NA==
created_at: 2021-01-06T08:32:31Z
updated_at: 2021-01-06T08:32:31Z
author_association: OWNER
body:
  An optional `path` argument to https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette which shows the path WITHOUT the `.Notebook` extension would be useful here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1178#issuecomment-755163886
issue_url: https://api.github.com/repos/simonw/datasette/issues/1178
node_id: MDEyOklzc3VlQ29tbWVudDc1NTE2Mzg4Ng==
created_at: 2021-01-06T08:37:51Z
updated_at: 2021-01-06T08:37:51Z
author_association: OWNER
body:
  Easiest fix would be for `publish cloudrun` to set `force_https_urls`:

  `datasette publish now` used to do this: https://github.com/simonw/datasette/blob/07e208cc6d9e901b87552c1be2854c220b3f9b6d/datasette/publish/now.py#L59-L63

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1178#issuecomment-755476820
issue_url: https://api.github.com/repos/simonw/datasette/issues/1178
node_id: MDEyOklzc3VlQ29tbWVudDc1NTQ3NjgyMA==
created_at: 2021-01-06T18:24:47Z
updated_at: 2021-01-06T18:24:47Z
author_association: OWNER
body:
  Issue fixed - https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on displays the correct schemes now.

  I can't think of a reason anyone on Cloud Run would ever NOT want the `force_https_urls` option, but just in case I've made it so if you pass `--extra-options --setting force_https_urls off` to `publish cloudrun` your setting will be respected.

  https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/publish/cloudrun.py#L105-L110

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-755484384
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc1NTQ4NDM4NA==
created_at: 2021-01-06T18:31:14Z
updated_at: 2021-01-06T18:31:57Z
author_association: OWNER
body:
  In building https://latest-with-plugins.datasette.io/github/issue_comments.Notebook?_labels=on I discovered the following patterns for importing data into both Pandas and Observable/d3:
  ```python
  import pandas
  df = pandas.read_json(
    "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array"
  )
  ```
  And:
  ```javascript
  d3 = require("d3@5")
  rows = d3.json(
      "https://latest-with-plugins.datasette.io/github/issue_comments.json?_shape=array"
  )
  ```
  Once again I find myself torn on the best possible default. A list of JSON objects is instantly compatible with both `pandas.read_json()` and `d3.json()` - but it leaves nowhere to put the extra information like pagination and suchlike!

  Even given this I still think the correct default is an object with `"rows"`, `"total"` and `"next_url"` keys. I should commit to that and implement it - this thought exercise has been running for far too long.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1179#issuecomment-755486103
issue_url: https://api.github.com/repos/simonw/datasette/issues/1179
node_id: MDEyOklzc3VlQ29tbWVudDc1NTQ4NjEwMw==
created_at: 2021-01-06T18:32:41Z
updated_at: 2021-01-06T18:34:11Z
author_association: OWNER
body:
  This parameter will return the URL path, with querystring arguments, to the HTML version of the page - e.g. `/github/issue_comments` or `/github/issue_comments?_sort_desc=created_at`

  Open questions:

  - What should it be called? `path` could be misleading since it also includes the querystring.
  - Should I provide a `url` or `full_url` version which includes `https://blah.com/...`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1179#issuecomment-755495387
issue_url: https://api.github.com/repos/simonw/datasette/issues/1179
node_id: MDEyOklzc3VlQ29tbWVudDc1NTQ5NTM4Nw==
created_at: 2021-01-06T18:39:23Z
updated_at: 2021-01-06T18:39:23Z
author_association: OWNER
body:
  In that case maybe there are three new arguments: `path`, `full_path` and `url`.

  I'll also add `request.full_path` for consistency with these: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/asgi.py#L77-L90

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1180#issuecomment-755500475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1180
node_id: MDEyOklzc3VlQ29tbWVudDc1NTUwMDQ3NQ==
created_at: 2021-01-06T18:43:41Z
updated_at: 2021-01-06T18:43:41Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/utils/__init__.py#L919-L940

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1180#issuecomment-756312213
issue_url: https://api.github.com/repos/simonw/datasette/issues/1180
node_id: MDEyOklzc3VlQ29tbWVudDc1NjMxMjIxMw==
created_at: 2021-01-07T18:56:24Z
updated_at: 2021-01-07T18:56:24Z
author_association: OWNER
body:
  The `async_call_with_supported_arguments` version should be able to await any of the lazy arguments that are awaitable - can use `await_me_maybe` for that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1171#issuecomment-756335394
issue_url: https://api.github.com/repos/simonw/datasette/issues/1171
node_id: MDEyOklzc3VlQ29tbWVudDc1NjMzNTM5NA==
created_at: 2021-01-07T19:35:59Z
updated_at: 2021-01-07T19:35:59Z
author_association: OWNER
body:
  I requested a D-U-N-S number as a first step in getting a developer certificate: https://developer.apple.com/support/D-U-N-S/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-756425587
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDc1NjQyNTU4Nw==
created_at: 2021-01-07T22:27:19Z
updated_at: 2021-01-07T22:27:19Z
author_association: NONE
body:
  I found this issue while troubleshooting the same behavior with an nginx reverse proxy. The solution was to make sure I set:

  `proxy_pass http://server:8001/baseurl/ 
  `
  instead of just:

  `proxy_pass http://server:8001
  `
  The custom SQL query and header links are now correct.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-756453010
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDc1NjQ1MzAxMA==
created_at: 2021-01-07T23:39:58Z
updated_at: 2021-01-07T23:40:25Z
author_association: OWNER
body:
  @tballison I think that's the solution! It looks like you need to use this in your config:

  `ProxyPass /datasette http://127.0.0.1:8001/datasette`

  Instead of this:

  `ProxyPass /datasette http://127.0.0.1:8001/`

  Give that a go and let me know if it fixes it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-756453945
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDc1NjQ1Mzk0NQ==
created_at: 2021-01-07T23:42:50Z
updated_at: 2021-01-07T23:42:50Z
author_association: OWNER
body:
  @henry501 it looks like you spotted a bug in the documentation - I just addressed that, the fix is now live here: https://docs.datasette.io/en/latest/deploying.html#running-datasette-behind-a-proxy

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1181#issuecomment-756482163
issue_url: https://api.github.com/repos/simonw/datasette/issues/1181
node_id: MDEyOklzc3VlQ29tbWVudDc1NjQ4MjE2Mw==
created_at: 2021-01-08T01:06:23Z
updated_at: 2021-01-08T01:06:54Z
author_association: OWNER
body:
  Yes, that logic is definitely at fault. It looks like it applies `urllib.parse.unquote_plus()` AFTER it's tried to do the `-` hash splitting thing, which is why it's failing here:

  https://github.com/simonw/datasette/blob/97fb10c17dd007a275ab743742e93e932335ad67/datasette/views/base.py#L184-L198

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1182#issuecomment-757373082
issue_url: https://api.github.com/repos/simonw/datasette/issues/1182
node_id: MDEyOklzc3VlQ29tbWVudDc1NzM3MzA4Mg==
created_at: 2021-01-09T21:55:33Z
updated_at: 2021-01-09T21:55:33Z
author_association: OWNER
body:
  I'll leave the page there but change it into more of a blurb about the existence of the plugins and tools directories.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1091#issuecomment-758280611
issue_url: https://api.github.com/repos/simonw/datasette/issues/1091
node_id: MDEyOklzc3VlQ29tbWVudDc1ODI4MDYxMQ==
created_at: 2021-01-11T23:06:10Z
updated_at: 2021-01-11T23:06:10Z
author_association: NONE
body:
  +1

  Yep!  Fixes it.  If I navigate to https://corpora.tika.apache.org/datasette, I get a 404 (database not found: datasette), but if I navigate to https://corpora.tika.apache.org/datasette/file_profiles/, everything WORKS!

  Thank you!

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1183#issuecomment-758356097
issue_url: https://api.github.com/repos/simonw/datasette/issues/1183
node_id: MDEyOklzc3VlQ29tbWVudDc1ODM1NjA5Nw==
created_at: 2021-01-12T02:40:30Z
updated_at: 2021-01-12T02:40:30Z
author_association: OWNER
body:
  So how would this work?

  I think I'm going to automatically use these values if the `_counts` table exists, unless a `ignore_counts_table` boolean setting has been set. I won't bother looking to see if the triggers have been created.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1183#issuecomment-758356640
issue_url: https://api.github.com/repos/simonw/datasette/issues/1183
node_id: MDEyOklzc3VlQ29tbWVudDc1ODM1NjY0MA==
created_at: 2021-01-12T02:42:08Z
updated_at: 2021-01-12T02:42:08Z
author_association: OWNER
body:
  Should Datasette have subcommands for this? `datasette enable-counts data.db` and `datasette disable-counts data.db` and `datasette reset-counts data.db` commands?

  Maybe. The `sqlite-utils` CLI tool could be used here instead, but that won't be easily available if Datasette was installed as a standalone binary or using `brew install datasette` or `pipx install datasette`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1185#issuecomment-759066777
issue_url: https://api.github.com/repos/simonw/datasette/issues/1185
node_id: MDEyOklzc3VlQ29tbWVudDc1OTA2Njc3Nw==
created_at: 2021-01-12T22:07:58Z
updated_at: 2021-01-12T22:07:58Z
author_association: OWNER
body:
  https://docs.datasette.io/en/stable/sql_queries.html?highlight=pragma#named-parameters documentation is out-of-date as well:

  > Datasette disallows custom SQL containing the string PRAGMA, as SQLite pragma statements can be used to change database settings at runtime. If you need to include the string "pragma" in a query you can do so safely using a named parameter.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1185#issuecomment-759067427
issue_url: https://api.github.com/repos/simonw/datasette/issues/1185
node_id: MDEyOklzc3VlQ29tbWVudDc1OTA2NzQyNw==
created_at: 2021-01-12T22:09:21Z
updated_at: 2021-01-12T22:09:21Z
author_association: OWNER
body:
  That allow-list was added in #761 but is not currently documented. It's here in the code:

  https://github.com/simonw/datasette/blob/8e8fc5cee5c78da8334495c6d6257d5612c40792/datasette/utils/__init__.py#L173-L186

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/220#issuecomment-759098964
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/220
node_id: MDEyOklzc3VlQ29tbWVudDc1OTA5ODk2NA==
created_at: 2021-01-12T23:19:55Z
updated_at: 2021-01-12T23:19:55Z
author_association: OWNER
body:
  I don't think it makes sense to call `.enable_fts()` on a view does it? When I'm working with views and FTS I tend to write my queries against a FTS table for one of the tables that is used by the view - https://docs.datasette.io/en/stable/full_text_search.html#configuring-full-text-search-for-a-table-or-view describes how I do that in Datasette for example.

  Can you expand on your use-case for FTS and views? I'm ready to be convinced otherwise, but I don't see how it would work right now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1186#issuecomment-759874332
issue_url: https://api.github.com/repos/simonw/datasette/issues/1186
node_id: MDEyOklzc3VlQ29tbWVudDc1OTg3NDMzMg==
created_at: 2021-01-14T01:59:35Z
updated_at: 2021-01-14T01:59:35Z
author_association: OWNER
body:
  Updated documentation: https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript and https://docs.datasette.io/en/latest/plugin_hooks.html#extra-js-urls-template-database-table-columns-view-name-request-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1187#issuecomment-759875239
issue_url: https://api.github.com/repos/simonw/datasette/issues/1187
node_id: MDEyOklzc3VlQ29tbWVudDc1OTg3NTIzOQ==
created_at: 2021-01-14T02:02:24Z
updated_at: 2021-01-14T02:02:31Z
author_association: OWNER
body:
  This plugin hook currently returns a string of JavaScript. It could optionally return this instead of a string:
  ```json
  {
    "script": "string of JavaScript goes here",
    "module": true
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/pull/55#issuecomment-760950128
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/55
node_id: MDEyOklzc3VlQ29tbWVudDc2MDk1MDEyOA==
created_at: 2021-01-15T13:44:52Z
updated_at: 2021-01-15T13:44:52Z
author_association: CONTRIBUTOR
body:
  I found and fixed another bug, this one around importing the tweets table. @simonw let me know if you'd prefer this broken out into multiple PRs, happy to do that if it makes review/merging easier.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/220#issuecomment-761015218
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/220
node_id: MDEyOklzc3VlQ29tbWVudDc2MTAxNTIxOA==
created_at: 2021-01-15T15:40:08Z
updated_at: 2021-01-15T15:40:08Z
author_association: NONE
body:
  Make sense. If you're coming from the sqlite3 side of things, rather than the datasette side, wanting the fts methods to work for views makes more sense. sqlite3 allows fts5 tables on views, so I was looking for CLI functionality to build the fts virtual tables. Ultimately, though, sharing fts virtual tables across tables and derivative views is likely more efficient. 

  Maybe an explicit error message like, "fts is not supported for views" rather than just throwing an exception that the method doesn't exist" might be helpful.  Not critical though.

  Thanks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/657#issuecomment-761101878
issue_url: https://api.github.com/repos/simonw/datasette/issues/657
node_id: MDEyOklzc3VlQ29tbWVudDc2MTEwMTg3OA==
created_at: 2021-01-15T18:16:01Z
updated_at: 2021-01-15T18:16:01Z
author_association: OWNER
body:
  I think the `startup()` plugin hook at https://docs.datasette.io/en/stable/plugin_hooks.html#startup-datasette should be able to fit this. You can write a plugin which uses that hook to execute `CREATE VIRTUAL TABLE` against  one or more databases when Datasette first starts running.

  Would that work here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-761103910
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2MTEwMzkxMA==
created_at: 2021-01-15T18:19:29Z
updated_at: 2021-01-15T18:19:29Z
author_association: OWNER
body:
  This relates to #987 (documented HTML hooks for JavaScript plugins) but is not quite the same thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-761104933
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2MTEwNDkzMw==
created_at: 2021-01-15T18:21:26Z
updated_at: 2021-12-17T07:03:02Z
author_association: OWNER
body:
  Also related: #857 (comprehensive documentation of variables available to templates) - since then the plugin hook could be fed the full template context and use that to do its thing.

  Or maybe the plugin hooks gets to return the name of a template that should be `{% include %}` into the page at that point? But the plugin may want to add extra context that is available to that template include.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/657#issuecomment-761179229
issue_url: https://api.github.com/repos/simonw/datasette/issues/657
node_id: MDEyOklzc3VlQ29tbWVudDc2MTE3OTIyOQ==
created_at: 2021-01-15T20:24:35Z
updated_at: 2021-01-15T20:24:35Z
author_association: OWNER
body:
  I'm not sure how I missed this issue but it's almost a year later and I'm finally taking a look at your Parquet work.

  This is yet more evidence that allowing plugins to provide their own custom `Database` objects would be a good idea.

  I started exploring what Datasette would like on PostgreSQL in #670 - my concern was that I would need to add a large amount of database abstraction code which would dramatically increase the complexity of the core project, but my thinking now is that it might be tractable - Datasette doesn't actually construct SQL in complex ways anywhere outside of the `TableView` class so abstracting away just that bit should be feasible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-761703022
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2MTcwMzAyMg==
created_at: 2021-01-17T00:20:00Z
updated_at: 2021-01-17T00:20:00Z
author_association: OWNER
body:
  Plugins that want to provide extra context to the template can already do so using the `extra_template_vars()` plugin hook.

  So this hook could work by returning a list of template filenames to be included. Those templates can be bundled with the plugin. Since they are included they will have access to the template context and to any `extra_template_vars()` values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-761703232
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2MTcwMzIzMg==
created_at: 2021-01-17T00:21:31Z
updated_at: 2021-01-17T00:21:54Z
author_association: OWNER
body:
  I think this ends up being a whole collection of new plugin hooks, something like:

  - `include_table_top`
  - `include_table_bottom`
  - `include_row_top`
  - `include_row_bottom`
  - `include_database_top`
  - `include_database_bottom`
  - `include_query_bottom`
  - `include_query_bottom`
  - `include_index_bottom`
  - `include_index_bottom`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1181#issuecomment-761703555
issue_url: https://api.github.com/repos/simonw/datasette/issues/1181
node_id: MDEyOklzc3VlQ29tbWVudDc2MTcwMzU1NQ==
created_at: 2021-01-17T00:24:20Z
updated_at: 2021-01-17T00:24:40Z
author_association: OWNER
body:
  Here's the incomplete sketch of a test - to go at the bottom of `test_cli.py`.

  ```python
  @pytest.mark.parametrize(
      "filename", ["test-database (1).sqlite", "database (1).sqlite"]
  )
  def test_weird_database_names(ensure_eventloop, tmpdir, filename):
      # https://github.com/simonw/datasette/issues/1181
      runner = CliRunner()
      db_path = str(tmpdir / filename)
      sqlite3.connect(db_path).execute("vacuum")
      result1 = runner.invoke(cli, [db_path, "--get", "/"])
      assert result1.exit_code == 0, result1.output
      homepage_html = result1.output
      assert False
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-761705076
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2MTcwNTA3Ng==
created_at: 2021-01-17T00:35:13Z
updated_at: 2021-01-17T00:37:51Z
author_association: OWNER
body:
  I'm going to try using Jinja macros to implement this: https://jinja.palletsprojects.com/en/2.11.x/templates/#macros

  Maybe using one of these tricks to auto-load the macro? http://codyaray.com/2015/05/auto-load-jinja2-macros

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11
node_id: MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA==
created_at: 2021-01-18T04:11:13Z
updated_at: 2021-01-18T04:11:13Z
author_association: MEMBER
body:
  I just got a similar error:

  ```
    File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py", line 79, in save_checkin
      checkins_table.m2m("users", user, m2m_table="with", pk="id")
    File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 2048, in m2m
      id = other_table.insert(record, pk=pk, replace=True).last_pk
    File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1781, in insert
      return self.insert_all(
    File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1899, in insert_all
      self.insert_chunk(
    File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1709, in insert_chunk
      result = self.db.execute(query, params)
    File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 226, in execute
      return self.conn.execute(sql, parameters)
  pysqlite3.dbapi2.OperationalError: table users has no column named countryCode

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-762385981
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDc2MjM4NTk4MQ==
created_at: 2021-01-18T17:32:13Z
updated_at: 2021-01-18T17:34:50Z
author_association: NONE
body:
  Hi Simon

  Just finding this old issue regarding downloading blobs. Nice work!

  <img width="212" alt="image" src="https://user-images.githubusercontent.com/4997607/104946741-df4bad80-59ba-11eb-96e3-727c85cc4dc6.png">

  As a feature request, maybe it would be possible to assign a blob column as a certain data type (e.g. `.jpg`) and then each blob could be downloaded as that type of file (perhaps if the file types were constrained to normal blobs that people store in sqlite databases, this could avoid the execution stuff mentioned above).

  I guess the column blob-type definition could fit into this dropdown selection:

  <img width="201" alt="image" src="https://user-images.githubusercontent.com/4997607/104947000-479a8f00-59bb-11eb-87d9-1644e5940894.png">

  Let me know if I should open a new issue with a feature request. (This could slowly go in the direction of displaying image blob-types in the browser.)

  Thanks for the great tool!


  ---

  edit: just reading the rest of the twitter thread: https://twitter.com/simonw/status/1318685933256855552

  perhaps this is already possible in some form with the plugin datasette-media: https://github.com/simonw/datasette-media

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-762387875
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDc2MjM4Nzg3NQ==
created_at: 2021-01-18T17:36:36Z
updated_at: 2021-01-18T17:36:36Z
author_association: OWNER
body:
  As you can see, I'm pretty paranoid about serving content with `Content-Type` HTTP headers because I'm so worried about execution vulnerabilities. I'm much more comfortable exploring that kind of thing in plugins, since that way people can opt-in to riskier features.

  You found `datasette-media` which is my most comprehensive exploration of that idea so far - but there's definitely lots of room for more plugins along those lines.

  Maybe even an output plugin? `.jpg` as an export format which returns the `BLOB` column for a row as a JPEG image with the correct `content-type` header (but first verifies that the binary content does indeed look like a real JPEG) could be interesting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1194#issuecomment-762390401
issue_url: https://api.github.com/repos/simonw/datasette/issues/1194
node_id: MDEyOklzc3VlQ29tbWVudDc2MjM5MDQwMQ==
created_at: 2021-01-18T17:42:38Z
updated_at: 2021-01-18T17:42:38Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/a882d679626438ba0d809944f06f239bcba8ee96/datasette/views/table.py#L815-L827

  It looks like there are other arguments that may not be persisted too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1194#issuecomment-762390568
issue_url: https://api.github.com/repos/simonw/datasette/issues/1194
node_id: MDEyOklzc3VlQ29tbWVudDc2MjM5MDU2OA==
created_at: 2021-01-18T17:43:03Z
updated_at: 2021-01-18T17:43:03Z
author_association: OWNER
body:
  Should I just blanket copy over any query string argument that starts with an underscore? Any reason _not_ to do that?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-762391426
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDc2MjM5MTQyNg==
created_at: 2021-01-18T17:45:00Z
updated_at: 2021-01-18T17:45:00Z
author_association: NONE
body:
  It might be possible with this library: https://docs.python.org/3/library/imghdr.html

  quick test of the downloaded blob:

  ```
  >>> import imghdr
  >>> imghdr.what('material_culture-1-image.blob')
  'jpeg'
  ```

  The output plugin would be cool. I'll look into making my first datasette plugin.  I'm also imagining displaying the image in the browser -- but that would be a step 2.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1175#issuecomment-762488336
issue_url: https://api.github.com/repos/simonw/datasette/issues/1175
node_id: MDEyOklzc3VlQ29tbWVudDc2MjQ4ODMzNg==
created_at: 2021-01-18T21:59:28Z
updated_at: 2021-01-18T22:00:31Z
author_association: NONE
body:
  I encountered your issue when trying to find a solution and came up with the following, maybe it can help.

  ```python
  import logging.config
  from typing import Tuple

  import structlog
  import uvicorn

  from example.config import settings

  shared_processors: Tuple[structlog.types.Processor, ...] = (
      structlog.contextvars.merge_contextvars,
      structlog.stdlib.add_logger_name,
      structlog.stdlib.add_log_level,
      structlog.processors.TimeStamper(fmt="iso"),
  )

  logging_config = {
      "version": 1,
      "disable_existing_loggers": False,
      "formatters": {
          "json": {
              "()": structlog.stdlib.ProcessorFormatter,
              "processor": structlog.processors.JSONRenderer(),
              "foreign_pre_chain": shared_processors,
          },
          "console": {
              "()": structlog.stdlib.ProcessorFormatter,
              "processor": structlog.dev.ConsoleRenderer(),
              "foreign_pre_chain": shared_processors,
          },
          **uvicorn.config.LOGGING_CONFIG["formatters"],
      },
      "handlers": {
          "default": {
              "level": "DEBUG",
              "class": "logging.StreamHandler",
              "formatter": "json" if not settings.debug else "console",
          },
          "uvicorn.access": {
              "level": "INFO",
              "class": "logging.StreamHandler",
              "formatter": "access",
          },
          "uvicorn.default": {
              "level": "INFO",
              "class": "logging.StreamHandler",
              "formatter": "default",
          },
      },
      "loggers": {
          "": {"handlers": ["default"], "level": "INFO"},
          "uvicorn.error": {
              "handlers": ["default" if not settings.debug else "uvicorn.default"],
              "level": "INFO",
              "propagate": False,
          },
          "uvicorn.access": {
              "handlers": ["default" if not settings.debug else "uvicorn.access"],
              "level": "INFO",
              "propagate": False,
          },
      },
  }


  def setup_logging() -> None:
      structlog.configure(
          processors=[
              structlog.stdlib.filter_by_level,
              *shared_processors,
              structlog.stdlib.PositionalArgumentsFormatter(),
              structlog.processors.StackInfoRenderer(),
              structlog.processors.format_exc_info,
              structlog.stdlib.ProcessorFormatter.wrap_for_formatter,
          ],
          context_class=dict,
          logger_factory=structlog.stdlib.LoggerFactory(),
          wrapper_class=structlog.stdlib.AsyncBoundLogger,
          cache_logger_on_first_use=True,
      )
      logging.config.dictConfig(logging_config)
  ```

  And then it'll be run on the startup event:
  ```python
  @app.on_event("startup")
  async def startup_event() -> None:
      setup_logging()
  ```

  It depends on a setting called `debug` which controls if it should output the regular uvicorn logging or json. 

reactions:
  {"total_count": 15, "+1": 7, "-1": 0, "laugh": 1, "hooray": 1, "confused": 0, "heart": 5, "rocket": 1, "eyes": 0}


html_url: https://github.com/simonw/sqlite-utils/issues/223#issuecomment-762540514
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/223
node_id: MDEyOklzc3VlQ29tbWVudDc2MjU0MDUxNA==
created_at: 2021-01-19T01:14:58Z
updated_at: 2021-01-19T01:15:54Z
author_association: OWNER
body:
  Example from https://docs.python.org/3/library/csv.html#csv.reader
  ```pycon
  >>> import csv
  >>> with open('eggs.csv', newline='') as csvfile:
  ...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
  ...     for row in spamreader:
  ...         print(', '.join(row))
  Spam, Spam, Spam, Spam, Spam, Baked Beans
  Spam, Lovely Spam, Wonderful Spam
  ```
  I'm going to add `--quotechar` as well as `--delimiter`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1195#issuecomment-763108730
issue_url: https://api.github.com/repos/simonw/datasette/issues/1195
node_id: MDEyOklzc3VlQ29tbWVudDc2MzEwODczMA==
created_at: 2021-01-19T20:22:37Z
updated_at: 2021-01-19T20:22:37Z
author_association: OWNER
body:
  I can use this test: https://github.com/simonw/datasette/blob/c38c42948cbfddd587729413fd6082ba352eaece/tests/test_plugins.py#L238-L294

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765495861
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDc2NTQ5NTg2MQ==
created_at: 2021-01-22T15:44:00Z
updated_at: 2021-01-22T15:44:00Z
author_association: NONE
body:
  Risk of autoimmune disorders: https://www.snpedia.com/index.php/Genotype
  ```
  select rsid, genotype, case genotype
    when 'AA' then '2x risk of rheumatoid arthritis and other autoimmune diseases'
    when 'GG' then 'Normal risk for autoimmune disorders'
  end as interpretation from genome where rsid = 'rs2476601'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765498984
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDc2NTQ5ODk4NA==
created_at: 2021-01-22T15:48:25Z
updated_at: 2021-01-22T15:49:33Z
author_association: NONE
body:
  The "Warrior Gene" https://www.snpedia.com/index.php/Rs4680

  ```
  select rsid, genotype, case genotype
    when 'AA' then '(worrier) advantage in memory and attention tasks'
    when 'AG' then 'Intermediate dopamine levels, other effects'
    when 'GG' then '(warrior) multiple associations, see details'
  end as interpretation from genome where rsid = 'rs4680'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765502845
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDc2NTUwMjg0NQ==
created_at: 2021-01-22T15:53:19Z
updated_at: 2021-01-22T15:53:19Z
author_association: NONE
body:
  rs7903146 Influences risk of Type-2 diabetes
  https://www.snpedia.com/index.php/Rs7903146
  ```
  select rsid, genotype, case genotype
    when 'CC' then 'Normal (lower) risk of Type 2 Diabetes and Gestational Diabetes.'
    when 'CT' then '1.4x increased risk for diabetes (and perhaps colon cancer).'
    when 'TT' then '2x increased risk for Type-2 diabetes'
  end as interpretation from genome where rsid = 'rs7903146'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765506901
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDc2NTUwNjkwMQ==
created_at: 2021-01-22T15:58:41Z
updated_at: 2021-01-22T15:58:58Z
author_association: NONE
body:
  Both rs10757274 and rs2383206 can both indicate higher risks of heart disease
  https://www.snpedia.com/index.php/Rs2383206

  ```
  select rsid, genotype, case genotype
    when 'AA' then 'Normal'
    when 'AG' then '~1.2x increased risk for heart disease'
    when 'GG' then '~1.3x increased risk for heart disease'
  end as interpretation from genome where rsid = 'rs10757274'
  ```

  ```
  select rsid, genotype, case genotype
    when 'AA' then 'Normal'
    when 'AG' then '1.4x increased risk for heart disease'
    when 'GG' then '1.7x increased risk for heart disease'
  end as interpretation from genome where rsid = 'rs2383206'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765523517
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDc2NTUyMzUxNw==
created_at: 2021-01-22T16:20:25Z
updated_at: 2021-01-22T16:20:25Z
author_association: NONE
body:
  rs53576: the oxytocin receptor (OXTR) gene

  ```
  select rsid, genotype, case genotype
    when 'AA' then 'Lack of empathy?'
    when 'AG' then 'Lack of empathy?'
    when 'GG' then 'Optimistic and empathetic; handle stress well'
  end as interpretation from genome where rsid = 'rs53576'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-765525338
issue_url: https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1
node_id: MDEyOklzc3VlQ29tbWVudDc2NTUyNTMzOA==
created_at: 2021-01-22T16:22:44Z
updated_at: 2021-01-22T16:22:44Z
author_association: NONE
body:
  rs1333049 associated with coronary artery disease
  https://www.snpedia.com/index.php/Rs1333049
  ```

  select rsid, genotype, case genotype
    when 'CC' then '1.9x increased risk for coronary artery disease'
    when 'CG' then '1.5x increased risk for CAD'
    when 'GG' then 'normal'
  end as interpretation from genome where rsid = 'rs1333049'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/224#issuecomment-765678057
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/224
node_id: MDEyOklzc3VlQ29tbWVudDc2NTY3ODA1Nw==
created_at: 2021-01-22T20:53:06Z
updated_at: 2021-01-23T20:13:27Z
author_association: NONE
body:
  I'm using the FTS methods in sqlite-utils for this website: [drwn.io](https://drwn.io/). I wanted to get pagination to have some kind of infinite scrolling in the landing page, and I ended up using that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-765757433
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2NTc1NzQzMw==
created_at: 2021-01-22T23:43:43Z
updated_at: 2021-01-22T23:43:43Z
author_association: OWNER
body:
  Another potential use for this: plugins that provide authentication (like `datasette-auth-passwords` and `datasette-auth-github`) could use it to add a chunk of HTML to the "permission denied" page that links to their mechanism of authenticating.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1199#issuecomment-766181628
issue_url: https://api.github.com/repos/simonw/datasette/issues/1199
node_id: MDEyOklzc3VlQ29tbWVudDc2NjE4MTYyOA==
created_at: 2021-01-23T21:25:18Z
updated_at: 2021-01-23T21:25:18Z
author_association: OWNER
body:
  Comment thread here: https://news.ycombinator.com/item?id=25881911 - cperciva says:

  >  There's an even better reason for databases to not write to memory mapped pages: Pages get synched out to disk at the kernel's leisure. This can be ok for a cache but it's definitely not what you want for a database!

  But... Datasette is often used in read-only mode, so that disadvantage often doesn't apply.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1198#issuecomment-766428183
issue_url: https://api.github.com/repos/simonw/datasette/issues/1198
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQyODE4Mw==
created_at: 2021-01-24T20:40:37Z
updated_at: 2021-01-24T20:40:37Z
author_association: OWNER
body:
  https://docs.datasette.io/en/latest/testing_plugins.html#testing-outbound-http-calls-with-pytest-httpx

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1197#issuecomment-766430111
issue_url: https://api.github.com/repos/simonw/datasette/issues/1197
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQzMDExMQ==
created_at: 2021-01-24T20:53:40Z
updated_at: 2021-01-24T20:53:40Z
author_association: OWNER
body:
  https://devcenter.heroku.com/articles/slug-compiler#slug-size says that the maximum allowed size is 500MB - my hunch is that the Datasette application itself weighs in at only a dozen or so MB but I haven't measured it. So I would imagine anything up to around 450MB should work OK on Heroku.

  Cloud Run works for up to about 2GB in my experience.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1190#issuecomment-766430644
issue_url: https://api.github.com/repos/simonw/datasette/issues/1190
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQzMDY0NA==
created_at: 2021-01-24T20:57:03Z
updated_at: 2021-01-24T20:57:03Z
author_association: OWNER
body:
  I really like this idea. It feels like an opportunity for a plugin that adds two things: an API endpoint to Datasette for accepting uploaded databases, and a `datasette publish upload` subcommand which can upload files to that endpoint (with some kind of authentication mechanism).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1190#issuecomment-766433153
issue_url: https://api.github.com/repos/simonw/datasette/issues/1190
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQzMzE1Mw==
created_at: 2021-01-24T21:13:25Z
updated_at: 2021-01-24T21:13:25Z
author_association: OWNER
body:
  This ties in to a bunch of other ideas that are in flight at the moment.

  If you're publishing databases by uploading them, how do you attach metadata? Ideally by baking it into the database file itself, using the mechanism from #1169.

  How could this interact with the `datasette insert` concept from #1163? Could you pass a CSV file to the `upload` command and have that converted and uploaded for you, or would you create the database file locally using `datasette insert` and then upload it as a separate `datasette upload` step?

  Lots to think about here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1179#issuecomment-766434629
issue_url: https://api.github.com/repos/simonw/datasette/issues/1179
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQzNDYyOQ==
created_at: 2021-01-24T21:23:47Z
updated_at: 2021-01-24T21:23:47Z
author_association: OWNER
body:
  I'm just going to do `path` and `full_path` (which includes the querystring)`. The `datasette.absolute_url()` method can be used by plugins that need the full URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1154#issuecomment-766462197
issue_url: https://api.github.com/repos/simonw/datasette/issues/1154
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ2MjE5Nw==
created_at: 2021-01-24T23:47:06Z
updated_at: 2021-01-24T23:47:06Z
author_association: OWNER
body:
  I'm going to document this but mark it as unstable, using a new documentation convention for marking unstable APIs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1202#issuecomment-766462475
issue_url: https://api.github.com/repos/simonw/datasette/issues/1202
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ2MjQ3NQ==
created_at: 2021-01-24T23:49:28Z
updated_at: 2021-01-24T23:50:33Z
author_association: OWNER
body:
  Can use an "admonition" similar to this:
  ```sphinx
  .. warning::
      Restricting access to tables and views in this way will NOT prevent users from querying them using arbitrary SQL queries, `like this <https://latest.datasette.io/fixtures?sql=select+*+from+facetable>`__ for example.
  ```
  As seen on https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-tables-and-views

  Documentation: https://docutils.sourceforge.io/docs/ref/rst/directives.html#specific-admonitions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1090#issuecomment-766463496
issue_url: https://api.github.com/repos/simonw/datasette/issues/1090
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ2MzQ5Ng==
created_at: 2021-01-24T23:57:00Z
updated_at: 2021-01-24T23:57:00Z
author_association: OWNER
body:
  Related: I built [datasette-leaflet-freedraw](https://datasette.io/plugins/datasette-leaflet-freedraw) which turns any canned query field called `freedraw` or `something_freedraw` into an interactive map that you can draw on to create a GeoJSON MultiPolygon.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1154#issuecomment-766465719
issue_url: https://api.github.com/repos/simonw/datasette/issues/1154
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ2NTcxOQ==
created_at: 2021-01-25T00:09:22Z
updated_at: 2021-01-25T00:09:22Z
author_association: OWNER
body:
  https://docs.datasette.io/en/latest/internals.html#the-internal-database

  <img width="738" alt="Internals_for_plugins_—_Datasette_documentation" src="https://user-images.githubusercontent.com/9599/105648087-76eb4900-5e5e-11eb-9c73-c00ba37b7b10.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-766466030
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ2NjAzMA==
created_at: 2021-01-25T00:11:04Z
updated_at: 2021-01-25T00:11:04Z
author_association: OWNER
body:
  I can combine this with #987 - each of these areas of the page can be wrapped in a `<div>` with a class that matches the name of the plugin hook, that way JavaScript plugins can append their content in the same place as Python plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1179#issuecomment-766484257
issue_url: https://api.github.com/repos/simonw/datasette/issues/1179
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ4NDI1Nw==
created_at: 2021-01-25T01:30:57Z
updated_at: 2021-01-25T01:30:57Z
author_association: OWNER
body:
  The challenge here is figuring out what the original path, without the `.format`, actually was - while taking into account that Datasette has a special case for tables that themselves end in a `.something`.

  The `path_with_format()` function nearly does what we need here:

  https://github.com/simonw/datasette/blob/b6a7b58fa01af0cd5a5e94bd17d686d283a46819/datasette/utils/__init__.py#L710-L729

  It can be called with `replace_format="csv"` to REMOVE the `.csv` format and replace it with something else.

  Problem is, we want to use it to get rid of the format entirely.

  We could update `path_with_format()` to accept `format=''` to mean "remove the format entirely", but it's a bit messy. It may be better to reconsider the design of `path_with_format()` and related utility functions entirely.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1179#issuecomment-766484435
issue_url: https://api.github.com/repos/simonw/datasette/issues/1179
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ4NDQzNQ==
created_at: 2021-01-25T01:31:36Z
updated_at: 2021-01-25T01:31:36Z
author_association: OWNER
body:
  Relevant existing tests: https://github.com/simonw/datasette/blob/461670a0b87efa953141b449a9a261919864ceb3/tests/test_utils.py#L365-L398

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-766484915
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ4NDkxNQ==
created_at: 2021-01-25T01:33:29Z
updated_at: 2021-01-25T01:33:29Z
author_association: OWNER
body:
  I'm going to ship a version of this in Datasette 0.54 with a warning that the interface should be considered unstable (see #1202) so that we can start trying this out.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1167#issuecomment-766487520
issue_url: https://api.github.com/repos/simonw/datasette/issues/1167
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ4NzUyMA==
created_at: 2021-01-25T01:44:43Z
updated_at: 2021-01-25T01:44:43Z
author_association: OWNER
body:
  Thanks @benpickles, I just merged that in. I'll use it in the documentation.

      # To check code is conformant
      npm run prettier -- --check
      # To fix it if it isn't
      npm run fix

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1167#issuecomment-766491566
issue_url: https://api.github.com/repos/simonw/datasette/issues/1167
node_id: MDEyOklzc3VlQ29tbWVudDc2NjQ5MTU2Ng==
created_at: 2021-01-25T02:01:19Z
updated_at: 2021-01-25T02:01:19Z
author_association: OWNER
body:
  New documentation section here (I documented Black as well): https://docs.datasette.io/en/latest/contributing.html#code-formatting

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-766523866
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: MDEyOklzc3VlQ29tbWVudDc2NjUyMzg2Ng==
created_at: 2021-01-25T03:58:34Z
updated_at: 2021-01-25T03:58:34Z
author_association: OWNER
body:
  I've got a good prototype working now, but I'm dropping this from the Datasette 0.54 milestone because it requires a bunch of additional work to make sure it is really well tested and documented.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1204#issuecomment-766525337
issue_url: https://api.github.com/repos/simonw/datasette/issues/1204
node_id: MDEyOklzc3VlQ29tbWVudDc2NjUyNTMzNw==
created_at: 2021-01-25T04:04:28Z
updated_at: 2021-01-25T04:04:28Z
author_association: OWNER
body:
  Writing the tests will be a bit tricky since we need to confirm that the `include_table_top(datasette, database, actor, table)` arguments were all passed correctly but the only thing we get back from the plugin is a list of templates. Maybe encode those values into the template names somehow?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1195#issuecomment-766534634
issue_url: https://api.github.com/repos/simonw/datasette/issues/1195
node_id: MDEyOklzc3VlQ29tbWVudDc2NjUzNDYzNA==
created_at: 2021-01-25T04:38:30Z
updated_at: 2021-01-25T04:38:30Z
author_association: OWNER
body:
  This has proved surprisingly difficult to implement, due to the weird way the QueryView is actually called. The class itself isn't used like other view classes - instead, the `.data()` methods of both `DatabaseView` and `TableView` dispatch out to `QueryView.data()` when they need to:

  https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L259-L270

  https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/table.py#L290-L294

  https://github.com/simonw/datasette/blob/07e163561592c743e4117f72102fcd350a600909/datasette/views/database.py#L39-L44

  It turns out this is a bad pattern because it makes changes like this one WAY harder than they should be.

  I think I should clean this up as part of #878.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1195#issuecomment-766534748
issue_url: https://api.github.com/repos/simonw/datasette/issues/1195
node_id: MDEyOklzc3VlQ29tbWVudDc2NjUzNDc0OA==
created_at: 2021-01-25T04:38:56Z
updated_at: 2021-01-25T04:38:56Z
author_association: OWNER
body:
  Here's a diff showing how far I got before I abandoned this particular effort:

  ```diff
  diff --git a/datasette/views/base.py b/datasette/views/base.py
  index a21b929..04e4aa9 100644
  --- a/datasette/views/base.py
  +++ b/datasette/views/base.py
  @@ -120,7 +120,7 @@ class BaseView:
           handler = getattr(self, request.method.lower(), None)
           return await handler(request, *args, **kwargs)
   
  -    async def render(self, templates, request, context=None):
  +    async def render(self, templates, request, context=None, view_name=None):
           context = context or {}
           template = self.ds.jinja_env.select_template(templates)
           template_context = {
  @@ -135,7 +135,7 @@ class BaseView:
           }
           return Response.html(
               await self.ds.render_template(
  -                template, template_context, request=request, view_name=self.name
  +                template, template_context, request=request, view_name=view_name
               )
           )
   
  @@ -155,7 +155,7 @@ class BaseView:
   
   
   class DataView(BaseView):
  -    name = ""
  +    view_name = "no-view-name"
       re_named_parameter = re.compile(":([a-zA-Z0-9_]+)")
   
       async def options(self, request, *args, **kwargs):
  @@ -414,6 +414,10 @@ class DataView(BaseView):
               args["table"] = urllib.parse.unquote_plus(args["table"])
           return _format, args
   
  +    async def get_view_name(self, request, database, hash, **kwargs):
  +        # Defaults to self.view_name, but can be over-ridden by subclasses
  +        return self.view_name
  +
       async def view_get(self, request, database, hash, correct_hash_provided, **kwargs):
           _format, kwargs = await self.get_format(request, database, kwargs)
   
  @@ -424,6 +428,8 @@ class DataView(BaseView):
               # HTML views default to expanding all foreign key labels
               kwargs["default_labels"] = True
   
  +        view_name = await self.get_view_name(request, database, hash, **kwargs)
  +
           extra_template_data = {}
           start = time.perf_counter()
           status_code = 200
  @@ -489,7 +495,7 @@ class DataView(BaseView):
                   database=database,
                   table=data.get("table"),
                   request=request,
  -                view_name=self.name,
  +                view_name=view_name,
                   # These will be deprecated in Datasette 1.0:
                   args=request.args,
                   data=data,
  @@ -533,7 +539,7 @@ class DataView(BaseView):
                       database=database,
                       table=data.get("table"),
                       request=request,
  -                    view_name=self.name,
  +                    view_name=view_name,
                   )
                   it_can_render = await await_me_maybe(it_can_render)
                   if it_can_render:
  @@ -565,7 +571,7 @@ class DataView(BaseView):
               }
               if "metadata" not in context:
                   context["metadata"] = self.ds.metadata
  -            r = await self.render(templates, request=request, context=context)
  +            r = await self.render(templates, request=request, context=context, view_name=view_name)
               r.status = status_code
   
           ttl = request.args.get("_ttl", None)
  diff --git a/datasette/views/database.py b/datasette/views/database.py
  index f6fd579..e425213 100644
  --- a/datasette/views/database.py
  +++ b/datasette/views/database.py
  @@ -23,7 +23,11 @@ from .base import DatasetteError, DataView
   
   
   class DatabaseView(DataView):
  -    name = "database"
  +    async def get_view_name(self, request, db_name, table_and_format):
  +        if request.args.get("sql"):
  +            return "query"
  +        else:
  +            return "database"
   
       async def data(self, request, database, hash, default_labels=False, _size=None):
           await self.check_permissions(
  @@ -145,7 +149,7 @@ class DatabaseView(DataView):
   
   
   class DatabaseDownload(DataView):
  -    name = "database_download"
  +    view_name = "database_download"
   
       async def view_get(self, request, database, hash, correct_hash_present, **kwargs):
           await self.check_permissions(
  diff --git a/datasette/views/index.py b/datasette/views/index.py
  index b6b8cbe..d750e3d 100644
  --- a/datasette/views/index.py
  +++ b/datasette/views/index.py
  @@ -16,7 +16,7 @@ COUNT_DB_SIZE_LIMIT = 100 * 1024 * 1024
   
   
   class IndexView(BaseView):
  -    name = "index"
  +    view_name = "index"
   
       async def get(self, request, as_format):
           await self.check_permission(request, "view-instance")
  diff --git a/datasette/views/special.py b/datasette/views/special.py
  index 9750dd0..dbd1e00 100644
  --- a/datasette/views/special.py
  +++ b/datasette/views/special.py
  @@ -6,7 +6,7 @@ import secrets
   
   
   class JsonDataView(BaseView):
  -    name = "json_data"
  +    view_name = "json_data"
   
       def __init__(self, datasette, filename, data_callback, needs_request=False):
           self.ds = datasette
  @@ -42,7 +42,7 @@ class JsonDataView(BaseView):
   
   
   class PatternPortfolioView(BaseView):
  -    name = "patterns"
  +    view_name = "patterns"
   
       async def get(self, request):
           await self.check_permission(request, "view-instance")
  @@ -50,7 +50,7 @@ class PatternPortfolioView(BaseView):
   
   
   class AuthTokenView(BaseView):
  -    name = "auth_token"
  +    view_name = "auth_token"
   
       async def get(self, request):
           token = request.args.get("token") or ""
  @@ -68,7 +68,7 @@ class AuthTokenView(BaseView):
   
   
   class LogoutView(BaseView):
  -    name = "logout"
  +    view_name = "logout"
   
       async def get(self, request):
           if not request.actor:
  @@ -87,7 +87,7 @@ class LogoutView(BaseView):
   
   
   class PermissionsDebugView(BaseView):
  -    name = "permissions_debug"
  +    view_name = "permissions_debug"
   
       async def get(self, request):
           await self.check_permission(request, "view-instance")
  @@ -102,7 +102,7 @@ class PermissionsDebugView(BaseView):
   
   
   class AllowDebugView(BaseView):
  -    name = "allow_debug"
  +    view_name = "allow_debug"
   
       async def get(self, request):
           errors = []
  @@ -136,7 +136,7 @@ class AllowDebugView(BaseView):
   
   
   class MessagesDebugView(BaseView):
  -    name = "messages_debug"
  +    view_name = "messages_debug"
   
       async def get(self, request):
           await self.check_permission(request, "view-instance")
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index 0a3504b..45d298a 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -257,7 +257,16 @@ class RowTableShared(DataView):
   
   
   class TableView(RowTableShared):
  -    name = "table"
  +    view_name = "table"
  +
  +    async def get_view_name(self, request, db_name, table_and_format):
  +        canned_query = await self.ds.get_canned_query(
  +            db_name, table_and_format, request.actor
  +        )
  +        if canned_query:
  +            return "query"
  +        else:
  +            return "table"
   
       async def post(self, request, db_name, table_and_format):
           # Handle POST to a canned query
  @@ -923,7 +932,7 @@ async def _sql_params_pks(db, table, pk_values):
   
   
   class RowView(RowTableShared):
  -    name = "row"
  +    view_name = "row"
   
       async def data(self, request, database, hash, table, pk_path, default_labels=False):
           await self.check_permissions(
  diff --git a/tests/test_plugins.py b/tests/test_plugins.py
  index 715c7c1..7ce2b1b 100644
  --- a/tests/test_plugins.py
  +++ b/tests/test_plugins.py
  @@ -252,7 +252,7 @@ def test_plugin_config_file(app_client):
               },
           ),
           (
  -            "/fixtures/",
  +            "/fixtures",
               {
                   "template": "database.html",
                   "database": "fixtures",
  @@ -285,6 +285,38 @@ def test_plugin_config_file(app_client):
                   ],
               },
           ),
  +        (
  +            "/fixtures?sql=select+1+as+one",
  +            {
  +                "template": "query.html",
  +                "database": "fixtures",
  +                "table": None,
  +                "config": {"depth": "database"},
  +                "view_name": "query",
  +                "request_path": "/fixtures",
  +                "added": 15,
  +                "columns": [
  +                    "one",
  +                ],
  +            },
  +        ),
  +        (
  +            "/fixtures/neighborhood_search",
  +            {
  +                "template": "query.html",
  +                "database": "fixtures",
  +                "table": None,
  +                "config": {"depth": "database"},
  +                "view_name": "query",
  +                "request_path": "/fixtures/neighborhood_search",
  +                "added": 15,
  +                "columns": [
  +                    "neighborhood",
  +                    "name",
  +                    "state",
  +                ],
  +            },
  +        ),
       ],
   )
   def test_hook_extra_body_script(app_client, path, expected_extra_body_script):
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1195#issuecomment-766535046
issue_url: https://api.github.com/repos/simonw/datasette/issues/1195
node_id: MDEyOklzc3VlQ29tbWVudDc2NjUzNTA0Ng==
created_at: 2021-01-25T04:40:08Z
updated_at: 2021-01-25T04:40:08Z
author_association: OWNER
body:
  Also: should the view name really be the same for both of these pages?

  - https://latest.datasette.io/fixtures?sql=select+*+from+facetable
  - https://latest.datasette.io/fixtures/neighborhood_search

  Where one of them is a canned query and the other is an arbitrary query?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-766536076
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDc2NjUzNjA3Ng==
created_at: 2021-01-25T04:43:53Z
updated_at: 2021-01-25T04:43:53Z
author_association: OWNER
body:
  ... actually not going to include this in 0.54, I need to write a couple of plugins myself using it before I even make it available in preview.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1201#issuecomment-766543387
issue_url: https://api.github.com/repos/simonw/datasette/issues/1201
node_id: MDEyOklzc3VlQ29tbWVudDc2NjU0MzM4Nw==
created_at: 2021-01-25T05:07:40Z
updated_at: 2021-01-25T05:13:29Z
author_association: OWNER
body:
  Changes: https://github.com/simonw/datasette/compare/0.53...a5ede3cdd455e2bb1a1fb2f4e1b5a9855caf5179

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1201#issuecomment-766545442
issue_url: https://api.github.com/repos/simonw/datasette/issues/1201
node_id: MDEyOklzc3VlQ29tbWVudDc2NjU0NTQ0Mg==
created_at: 2021-01-25T05:13:59Z
updated_at: 2021-01-25T05:13:59Z
author_association: OWNER
body:
  The big stuff:

  - Database(memory_name=) for shared in-memory databases, closes #1151 
  - The `_internal` database -  #1150
  -  script type=module support, closes #1186 , #1187
  - Improved design for the `.add_database()` method 8919f99c2f7f245aca7f94bd53d5ac9d04aa42b5 - which means databases with the same stem can now be opened, #509
  - Adopted Prettier #1166 

  Smaller:

  - force_https_urls on for publish cloudrun, refs #1178 
  - Fixed bug in example nginx config, refs #1091 
  -  Shrunk ecosystem docs in favour of datasette.io, closes #1182 
  -  request.full_path property, closes #1184 
  -  Better PRAGMA error message, closes #1185 
  -  publish heroku now uses python-3.8.7 
  -  Plugin testing documentation on using pytest-httpx Closes #1198
  -  Contributing docs for Black and Prettier, closes #1167 
  -  All ?_ parameters now copied to hidden form fields, closes #1194 
  - Fixed bug loading database called 'test-database (1).sqlite' - Closes #1181.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1201#issuecomment-766586151
issue_url: https://api.github.com/repos/simonw/datasette/issues/1201
node_id: MDEyOklzc3VlQ29tbWVudDc2NjU4NjE1MQ==
created_at: 2021-01-25T06:44:43Z
updated_at: 2021-01-25T06:44:43Z
author_association: OWNER
body:
  OK, release notes are ready to merge from that branch. I'll ship the release in the morning, to give me time to write the accompanying annotated release notes.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1206#issuecomment-766588020
issue_url: https://api.github.com/repos/simonw/datasette/issues/1206
node_id: MDEyOklzc3VlQ29tbWVudDc2NjU4ODAyMA==
created_at: 2021-01-25T06:48:20Z
updated_at: 2021-01-25T06:48:20Z
author_association: OWNER
body:
  Issues to reference in the commit message: #509, #1091, #1150, #1151, #1166, #1167, #1178, #1181, #1182, #1184, #1185, #1186, #1187, #1194, #1198

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1206#issuecomment-766588371
issue_url: https://api.github.com/repos/simonw/datasette/issues/1206
node_id: MDEyOklzc3VlQ29tbWVudDc2NjU4ODM3MQ==
created_at: 2021-01-25T06:49:06Z
updated_at: 2021-01-25T06:49:06Z
author_association: OWNER
body:
  Last thing to do: write up the annotated version of these release notes, assign it a URL on my blog and link to it from the release notes here so I can publish them simultaneously.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1206#issuecomment-766589070
issue_url: https://api.github.com/repos/simonw/datasette/issues/1206
node_id: MDEyOklzc3VlQ29tbWVudDc2NjU4OTA3MA==
created_at: 2021-01-25T06:50:30Z
updated_at: 2021-01-25T17:31:11Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=h1) Report
  > Merging [#1206](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=desc) (06480e1) into [main](https://codecov.io/gh/simonw/datasette/commit/a5ede3cdd455e2bb1a1fb2f4e1b5a9855caf5179?el=desc) (a5ede3c) will **not change** coverage.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1206/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1206   +/-   ##
  =======================================
    Coverage   91.53%   91.53%           
  =======================================
    Files          32       32           
    Lines        3947     3947           
  =======================================
    Hits         3613     3613           
    Misses        334      334           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/version.py](https://codecov.io/gh/simonw/datasette/pull/1206/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZlcnNpb24ucHk=) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=footer). Last update [a5ede3c...571476d](https://codecov.io/gh/simonw/datasette/pull/1206?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/991#issuecomment-767761155
issue_url: https://api.github.com/repos/simonw/datasette/issues/991
node_id: MDEyOklzc3VlQ29tbWVudDc2Nzc2MTE1NQ==
created_at: 2021-01-26T19:05:21Z
updated_at: 2021-01-26T19:06:36Z
author_association: OWNER
body:
  Idea: implement this using the existing table view, with a custom template called `table-internal-bb0ec0-tables.html` - that's the custom template listed in the HTML comments at the bottom of https://latest.datasette.io/_internal/tables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1208#issuecomment-767823684
issue_url: https://api.github.com/repos/simonw/datasette/issues/1208
node_id: MDEyOklzc3VlQ29tbWVudDc2NzgyMzY4NA==
created_at: 2021-01-26T20:58:51Z
updated_at: 2021-01-26T20:58:51Z
author_association: OWNER
body:
  This is a good catch - I've been lazy about this, but you're right that it's an issue that needs cleaning up.  Would be very happy to apply a PR, thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/54#issuecomment-767888743
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/54
node_id: MDEyOklzc3VlQ29tbWVudDc2Nzg4ODc0Mw==
created_at: 2021-01-26T23:07:41Z
updated_at: 2021-01-26T23:07:41Z
author_association: NONE
body:
  My import got much further with the applied fixes than 0.21.3, but not 100%. I do appear to have all of the tweets imported at least. 
  Not sure when I'll have a chance to look further to try to fix or see what didn't make it into the import.

  Here's my output:

  ```
  direct-messages-group: not yet implemented
  branch-links: not yet implemented
  periscope-expired-broadcasts: not yet implemented
  direct-messages: not yet implemented
  mute: not yet implemented
  periscope-comments-made-by-user: not yet implemented
  periscope-ban-information: not yet implemented
  periscope-profile-description: not yet implemented
  screen-name-change: not yet implemented
  manifest: not yet implemented
  fleet: not yet implemented
  user-link-clicks: not yet implemented
  periscope-broadcast-metadata: not yet implemented
  contact: not yet implemented
  fleet-mute: not yet implemented
  device-token: not yet implemented
  protected-history: not yet implemented
  direct-message-mute: not yet implemented
  Traceback (most recent call last):
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/bin/twitter-to-sqlite", line 33, in <module>
      sys.exit(load_entry_point('twitter-to-sqlite==0.21.3', 'console_scripts', 'twitter-to-sqlite')())
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/cli.py", line 772, in import_
      archive.import_from_file(db, filepath.name, open(filepath, "rb").read())
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 233, in import_from_file
      to_insert = transformer(data)
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 21, in callback
      return {filename: [fn(item) for item in data]}
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 21, in <listcomp>
      return {filename: [fn(item) for item in data]}
    File "/Users/henry/.local/share/virtualenvs/python-sqlite-testing-mF3G2xKl/lib/python3.9/site-packages/twitter_to_sqlite/archive.py", line 88, in ageinfo
      return item["ageMeta"]["ageInfo"]
  KeyError: 'ageInfo'

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1210#issuecomment-769274591
issue_url: https://api.github.com/repos/simonw/datasette/issues/1210
node_id: MDEyOklzc3VlQ29tbWVudDc2OTI3NDU5MQ==
created_at: 2021-01-28T18:10:02Z
updated_at: 2021-01-28T18:10:02Z
author_association: OWNER
body:
  That definitely sounds like a bug! Can you provide a copy of your `metadata.JSON` and the command-line you are using to launch Datasette?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1205#issuecomment-769452084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1205
node_id: MDEyOklzc3VlQ29tbWVudDc2OTQ1MjA4NA==
created_at: 2021-01-28T22:52:23Z
updated_at: 2021-01-28T22:52:23Z
author_association: OWNER
body:
  Here are the redirect tests: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/tests/test_api.py#L635-L648

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1205#issuecomment-769453074
issue_url: https://api.github.com/repos/simonw/datasette/issues/1205
node_id: MDEyOklzc3VlQ29tbWVudDc2OTQ1MzA3NA==
created_at: 2021-01-28T22:54:49Z
updated_at: 2021-01-28T22:55:02Z
author_association: OWNER
body:
  I also checked that the following works:

      echo '{"foo": "bar"}' | sqlite-utils insert _memory.db demo -
      datasette _memory.db --memory

  Sure enough, it results in the following Datasette homepage - thanks to #509

  <img width="274" alt="Datasette___memory___memory_2" src="https://user-images.githubusercontent.com/9599/106208790-c8564980-6178-11eb-8b8b-053a9f1d0193.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1209#issuecomment-769455370
issue_url: https://api.github.com/repos/simonw/datasette/issues/1209
node_id: MDEyOklzc3VlQ29tbWVudDc2OTQ1NTM3MA==
created_at: 2021-01-28T23:00:21Z
updated_at: 2021-01-28T23:00:21Z
author_association: OWNER
body:
  Good catch on the workaround here. The root problem is that `datasette-template-sql` looks for the first available databsae if you don't provide it with a `database=` argument, and in Datasette 0.54 the first available database changed to being the new `_internal` database.

  Is this a bug? I think it is - because the documented behaviour on https://docs.datasette.io/en/stable/internals.html#get-database-name is this:

  > `name` - string, optional
  >
  > The name to be used for this database - this will be used in the URL path, e.g. `/dbname`. If not specified Datasette will pick one based on the filename or memory name.

  Since the new behaviour differs from what was in the documentation I'm going to treat this as a bug and fix it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1207#issuecomment-769534187
issue_url: https://api.github.com/repos/simonw/datasette/issues/1207
node_id: MDEyOklzc3VlQ29tbWVudDc2OTUzNDE4Nw==
created_at: 2021-01-29T02:37:19Z
updated_at: 2021-01-29T02:37:19Z
author_association: OWNER
body:
  https://docs.datasette.io/en/latest/testing_plugins.html#using-pdb-for-errors-thrown-inside-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769957751
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56
node_id: MDEyOklzc3VlQ29tbWVudDc2OTk1Nzc1MQ==
created_at: 2021-01-29T17:59:40Z
updated_at: 2021-01-29T17:59:40Z
author_association: MEMBER
body:
  This is interesting - how did you create that initial table? Was this using the `twitter-to-sqlite import archive.db ~/Downloads/twitter-2019-06-25-b31f2.zip` command, or something else?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-769973212
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56
node_id: MDEyOklzc3VlQ29tbWVudDc2OTk3MzIxMg==
created_at: 2021-01-29T18:29:02Z
updated_at: 2021-01-29T18:31:55Z
author_association: NONE
body:
  I think it was with `twitter-to-sqlite home-timeline home.db -a  auth.json --since`
  and Im using only this command to grab tweets 

  from cron tab
  `2,7,12,17,22,27,32,37,42,47,52,57 * * * * run-one /home/gsajko/miniconda3/bin/twitter-to-sqlite home-timeline /home/gsajko/work/custom_twitter_feed/home.db -a /home/gsajko/work/custom_twitter_feed/auth/auth.json --since`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770069864
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60
node_id: MDEyOklzc3VlQ29tbWVudDc3MDA2OTg2NA==
created_at: 2021-01-29T21:52:05Z
updated_at: 2021-02-12T18:29:43Z
author_association: CONTRIBUTOR
body:
  For the purposes below I am assuming the organization I would get all the repositories and their related commits from is called `gh-organization`. The github's owner id of gh-orgnization is `123456789`.

  ```bash
  github-to-sqlite  repos github.db gh-organization
  ```

  I'm on a windows computer running git bash to be able to use the `|` command. This works for me
  ```bash
  sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n\r' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }
  ```

  On a pure linux system I think this would work because the new line character is normally `\n`
  ```bash
  sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }`
  ```

  As expected I ran into rate limit issues #51 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770071568
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60
node_id: MDEyOklzc3VlQ29tbWVudDc3MDA3MTU2OA==
created_at: 2021-01-29T21:56:15Z
updated_at: 2021-01-29T21:56:15Z
author_association: MEMBER
body:
  I really like the way you're using pipes here - really smart. It's similar to how I build the demo database in this GitHub Actions workflow:

  https://github.com/dogsheep/github-to-sqlite/blob/62dfd3bc4014b108200001ef4bc746feb6f33b45/.github/workflows/deploy-demo.yml#L52-L82

  `twitter-to-sqlite` actually has a mechanism for doing this kind of thing, documented at https://github.com/dogsheep/twitter-to-sqlite#providing-input-from-a-sql-query-with---sql-and---attach

  It lets you do things like:

  ```
  $ twitter-to-sqlite users-lookup my.db --sql="select follower_id from following" --ids
  ```
  Maybe I should add something similar to `github-to-sqlite`? Feels like it could be really useful.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770112248
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60
node_id: MDEyOklzc3VlQ29tbWVudDc3MDExMjI0OA==
created_at: 2021-01-30T00:01:03Z
updated_at: 2021-01-30T01:14:42Z
author_association: CONTRIBUTOR
body:
  Yes that would be cool! I wouldn't mind helping. Is this the meat of it? https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/utils.py#L512

  It looks like the cli option is added with this decorator : https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/cli.py#L14

  I looked a bit at utils.py in the GitHub repository. I was surprised at the amount of manual mapping of the API response you had to do to get this to work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-770150526
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51
node_id: MDEyOklzc3VlQ29tbWVudDc3MDE1MDUyNg==
created_at: 2021-01-30T03:44:19Z
updated_at: 2021-01-30T03:47:24Z
author_association: CONTRIBUTOR
body:
  I don't have much experience with github's rate limiting. In my day job we use the [tenacity library](https://github.com/jd/tenacity) to handle http errors we get.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1211#issuecomment-770343684
issue_url: https://api.github.com/repos/simonw/datasette/issues/1211
node_id: MDEyOklzc3VlQ29tbWVudDc3MDM0MzY4NA==
created_at: 2021-01-31T08:03:40Z
updated_at: 2021-01-31T08:03:40Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=h1) Report
  > Merging [#1211](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=desc) (e33ccaa) into [main](https://codecov.io/gh/simonw/datasette/commit/dde3c500c73ace33529672f7d862b76753d309cc?el=desc) (dde3c50) will **decrease** coverage by `0.00%`.
  > The diff coverage is `92.85%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1211/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1211      +/-   ##
  ==========================================
  - Coverage   91.54%   91.53%   -0.01%     
  ==========================================
    Files          32       32              
    Lines        3948     3959      +11     
  ==========================================
  + Hits         3614     3624      +10     
  - Misses        334      335       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `77.29% <66.66%> (-0.31%)` | :arrow_down: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.62% <100.00%> (+<0.01%)` | :arrow_up: |
  | [datasette/publish/cloudrun.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY2xvdWRydW4ucHk=) | `96.96% <100.00%> (+0.09%)` | :arrow_up: |
  | [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.73% <100.00%> (+0.60%)` | :arrow_up: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1211/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.13% <100.00%> (+0.02%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=footer). Last update [dde3c50...e33ccaa](https://codecov.io/gh/simonw/datasette/pull/1211?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1213#issuecomment-771965281
issue_url: https://api.github.com/repos/simonw/datasette/issues/1213
node_id: MDEyOklzc3VlQ29tbWVudDc3MTk2NTI4MQ==
created_at: 2021-02-02T20:37:08Z
updated_at: 2021-02-02T20:39:24Z
author_association: OWNER
body:
  Starlette's gzip middleware implementation is here: https://github.com/encode/starlette/blob/0.14.2/starlette/middleware/gzip.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1213#issuecomment-771968177
issue_url: https://api.github.com/repos/simonw/datasette/issues/1213
node_id: MDEyOklzc3VlQ29tbWVudDc3MTk2ODE3Nw==
created_at: 2021-02-02T20:41:13Z
updated_at: 2021-02-02T20:41:13Z
author_association: OWNER
body:
  Starlette accumulates the full response body in a `body` variable and then does this:
  ```python
          elif message_type == "http.response.body":
              # Remaining body in streaming GZip response.
              body = message.get("body", b"")
              more_body = message.get("more_body", False)

              self.gzip_file.write(body)
              if not more_body:
                  self.gzip_file.close()

              message["body"] = self.gzip_buffer.getvalue()
              self.gzip_buffer.seek(0)
              self.gzip_buffer.truncate()

              await self.send(message)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1213#issuecomment-771968675
issue_url: https://api.github.com/repos/simonw/datasette/issues/1213
node_id: MDEyOklzc3VlQ29tbWVudDc3MTk2ODY3NQ==
created_at: 2021-02-02T20:41:55Z
updated_at: 2021-02-02T20:41:55Z
author_association: OWNER
body:
  So maybe I could a special response header which ASGI middleware can pick up that means "Don't attempt to gzip this, just stream it through".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1212#issuecomment-771975941
issue_url: https://api.github.com/repos/simonw/datasette/issues/1212
node_id: MDEyOklzc3VlQ29tbWVudDc3MTk3NTk0MQ==
created_at: 2021-02-02T20:52:36Z
updated_at: 2021-02-02T20:52:36Z
author_association: OWNER
body:
  37 minutes, wow! They're a little slow for me (4-5 minutes perhaps) but not nearly that bad.

  Thanks for running that profile. I think you're right: figuring out how to use more session scopes would definitely help.

  The `:memory:` idea is interesting too. The new `memory_name=` feature added in #1151 (released in Datasette 0.54) could help a lot here, since it allows Datasette instances to share the same in-memory database across multiple HTTP requests and connections.

  Note that `memory_name=` also persists within test runs themselves, independently of any `scope=` options on the fixtures. That might actually help us here!

  I'd be delighted if you explored this issue further, especially the option of using `memory_name=` for the fixtures databases used by the tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1212#issuecomment-771976561
issue_url: https://api.github.com/repos/simonw/datasette/issues/1212
node_id: MDEyOklzc3VlQ29tbWVudDc3MTk3NjU2MQ==
created_at: 2021-02-02T20:53:27Z
updated_at: 2021-02-02T20:53:27Z
author_association: OWNER
body:
  It would be great if we could get `python-xdist` to run too - I tried it in the past and gave up when I ran into those race conditions, but I've not done any further digging to see if there's a way to fix that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1214#issuecomment-771992025
issue_url: https://api.github.com/repos/simonw/datasette/issues/1214
node_id: MDEyOklzc3VlQ29tbWVudDc3MTk5MjAyNQ==
created_at: 2021-02-02T21:14:16Z
updated_at: 2021-02-02T21:14:16Z
author_association: OWNER
body:
  As a result, navigating to https://github-to-sqlite.dogsheep.net/github/labels?_search=help and clearing out the `_search` field then submitting the form does NOT clear the search term.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1214#issuecomment-771992628
issue_url: https://api.github.com/repos/simonw/datasette/issues/1214
node_id: MDEyOklzc3VlQ29tbWVudDc3MTk5MjYyOA==
created_at: 2021-02-02T21:15:18Z
updated_at: 2021-02-02T21:15:18Z
author_association: OWNER
body:
  The cause of this bug is form fields which begin with `_` but ARE displayed as form inputs on the page - hence should not be duplicated in an `<input type="hidden">` element.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1214#issuecomment-772001787
issue_url: https://api.github.com/repos/simonw/datasette/issues/1214
node_id: MDEyOklzc3VlQ29tbWVudDc3MjAwMTc4Nw==
created_at: 2021-02-02T21:28:53Z
updated_at: 2021-02-02T21:28:53Z
author_association: OWNER
body:
  Fix is now live on https://latest.datasette.io/fixtures/searchable?_search=terry - clearing "terry" and re-submitting the form now works as expected.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/twitter-to-sqlite/issues/56#issuecomment-772408273
issue_url: https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/56
node_id: MDEyOklzc3VlQ29tbWVudDc3MjQwODI3Mw==
created_at: 2021-02-03T10:36:36Z
updated_at: 2021-02-03T10:36:36Z
author_association: NONE
body:
  I figured it out.
  Those tweets are in database, because somebody quote tweeted them, or retweeted them.
  And if you grab quoted tweet or reweeted tweet from other tweet json, It doesn't grab all of the details.

  So if someone quote tweeted a quote tweet, the second quote tweet won't have `quoted_status`. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1216#issuecomment-772796111
issue_url: https://api.github.com/repos/simonw/datasette/issues/1216
node_id: MDEyOklzc3VlQ29tbWVudDc3Mjc5NjExMQ==
created_at: 2021-02-03T20:20:48Z
updated_at: 2021-02-03T20:20:48Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/1600d2a3ec3ada1f6fb5b1eb73bdaeccb5f80530/datasette/app.py#L620-L632

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1210#issuecomment-773977128
issue_url: https://api.github.com/repos/simonw/datasette/issues/1210
node_id: MDEyOklzc3VlQ29tbWVudDc3Mzk3NzEyOA==
created_at: 2021-02-05T11:30:34Z
updated_at: 2021-02-05T11:30:34Z
author_association: NONE
body:
  Thanks for your quick reply! Having changed my `metadata.yml`, queries AND database I can't really reproduce it anymore, sorry. But at least I'm happy to say that it works now! :) Thanks again for the super nifty tool, very appreciated.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/203#issuecomment-774217792
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/203
node_id: MDEyOklzc3VlQ29tbWVudDc3NDIxNzc5Mg==
created_at: 2021-02-05T18:44:13Z
updated_at: 2021-02-05T18:44:13Z
author_association: NONE
body:
  Thanks for looking at this - home schooling kids has prevented me from replying. 

  I'd struggled with how to adapt the API for the foreign keys too - I definitely tried the String/Tuple approach. I hadn't considered the breaking changes that would introduce though. I can take a look at this and try and make the change - see which of your options works best.

  I've got a workaround for the use-case I was looking at this for, so it wouldn't be a problem for me if it was put on the back burner until a hypothetical v4.0 anyway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1217#issuecomment-774385092
issue_url: https://api.github.com/repos/simonw/datasette/issues/1217
node_id: MDEyOklzc3VlQ29tbWVudDc3NDM4NTA5Mg==
created_at: 2021-02-06T02:49:11Z
updated_at: 2021-02-06T02:49:11Z
author_association: NONE
body:
  A good reference seems to be the note to run `datasette` as a module in https://github.com/simonw/datasette/pull/556

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/pocket-to-sqlite/issues/9#issuecomment-774730656
issue_url: https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDc3NDczMDY1Ng==
created_at: 2021-02-07T18:45:04Z
updated_at: 2021-02-07T18:45:04Z
author_association: NONE
body:
  That URL uses TLS 1.3, but maybe only if the client supports it.
  It could be your Python version or your SSL library that’s not recent enough.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1219#issuecomment-775442039
issue_url: https://api.github.com/repos/simonw/datasette/issues/1219
node_id: MDEyOklzc3VlQ29tbWVudDc3NTQ0MjAzOQ==
created_at: 2021-02-08T20:39:52Z
updated_at: 2021-02-08T22:13:00Z
author_association: OWNER
body:
  This comment helped me find a pattern for running Scalene against the Datasette test suite: https://github.com/emeryberger/scalene/issues/70#issuecomment-755245858

  ```
  pip install scalene
  ```
  Then I created a file called `run_tests.py` with the following contents:
  ```python
  if __name__ == "__main__":
      import sys, pytest
      pytest.main(sys.argv)
  ```
  Then I ran this:
  ```
  scalene --profile-all run_tests.py -sv -x .
  ```
  But... it quit with a segmentation fault!
  ```
  (datasette) datasette % scalene --profile-all run_tests.py -sv -x .
  ======================================================================== test session starts ========================================================================
  platform darwin -- Python 3.8.6, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- python
  cachedir: .pytest_cache
  rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini
  plugins: asyncio-0.14.0, timeout-1.4.2
  collecting ... Fatal Python error: Segmentation fault

  Current thread 0x0000000110c1edc0 (most recent call first):
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 553 in detect_json1
    File "/Users/simon/Dropbox/Development/datasette/datasette/filters.py", line 168 in Filters
    File "/Users/simon/Dropbox/Development/datasette/datasette/filters.py", line 94 in <module>
    File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
    File "<frozen importlib._bootstrap_external>", line 783 in exec_module
    File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 27 in <module>
    File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
    File "<frozen importlib._bootstrap_external>", line 783 in exec_module
    File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 42 in <module>
    File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
    File "<frozen importlib._bootstrap_external>", line 783 in exec_module
    File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "/Users/simon/Dropbox/Development/datasette/tests/test_api.py", line 1 in <module>
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 170 in exec_module
    File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
    File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 991 in _find_and_load
    File "<frozen importlib._bootstrap>", line 1014 in _gcd_import
    File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127 in import_module
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/pathlib.py", line 520 in import_path
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 552 in _importtestmodule
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 484 in _getobj
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 288 in obj
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 500 in _inject_setup_module_fixture
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/python.py", line 487 in collect
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 324 in <lambda>
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 294 in from_call
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 324 in pytest_make_collect_report
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/runner.py", line 441 in collect_one_node
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 768 in genitems
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 771 in genitems
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 568 in _perform_collect
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 516 in perform_collect
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 306 in pytest_collection
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 295 in _main
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 240 in wrap_session
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/main.py", line 289 in pytest_cmdline_main
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/_pytest/config/__init__.py", line 157 in main
    File "run_tests.py", line 3 in <module>
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/scalene_profiler.py", line 1525 in main
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py", line 7 in main
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/scalene/__main__.py", line 14 in <module>
    File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87 in _run_code
    File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194 in _run_module_as_main
  Scalene error: received signal SIGSEGV
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1219#issuecomment-775497449
issue_url: https://api.github.com/repos/simonw/datasette/issues/1219
node_id: MDEyOklzc3VlQ29tbWVudDc3NTQ5NzQ0OQ==
created_at: 2021-02-08T22:11:34Z
updated_at: 2021-02-08T22:11:34Z
author_association: OWNER
body:
  https://github.com/emeryberger/scalene/issues/110 reports a "received signal SIGSEGV" error that was fixed by upgrading to the latest Scalene version, but I'm running that already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1200#issuecomment-777132761
issue_url: https://api.github.com/repos/simonw/datasette/issues/1200
node_id: MDEyOklzc3VlQ29tbWVudDc3NzEzMjc2MQ==
created_at: 2021-02-11T00:29:52Z
updated_at: 2021-02-11T00:29:52Z
author_association: CONTRIBUTOR
body:
  I'm probably missing something but what's the use case here - what would this offer over adding `limit 10` to the query?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1200#issuecomment-777178728
issue_url: https://api.github.com/repos/simonw/datasette/issues/1200
node_id: MDEyOklzc3VlQ29tbWVudDc3NzE3ODcyOA==
created_at: 2021-02-11T03:13:59Z
updated_at: 2021-02-11T03:13:59Z
author_association: OWNER
body:
  I came up with the need for this while playing with this tool: https://calands.datasettes.com/calands?sql=select%0D%0A++AsGeoJSON(geometry)%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+'%25mini%25'+and%0D%0A++Intersects(GeomFromGeoJSON(%3Afreedraw)%2C+geometry)+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+(%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+'CPAD_2020a_SuperUnits'%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON(%3Afreedraw)%0D%0A++)&freedraw={"type"%3A"MultiPolygon"%2C"coordinates"%3A[[[[-122.42202758789064%2C37.82280243352759]%2C[-122.39868164062501%2C37.823887203271454]%2C[-122.38220214843751%2C37.81846319511331]%2C[-122.35061645507814%2C37.77071473849611]%2C[-122.34924316406251%2C37.74465712069939]%2C[-122.37258911132814%2C37.703380457832374]%2C[-122.39044189453125%2C37.690340943717715]%2C[-122.41241455078126%2C37.680559803205135]%2C[-122.44262695312501%2C37.67295135774715]%2C[-122.47283935546876%2C37.67295135774715]%2C[-122.52502441406251%2C37.68382032669382]%2C[-122.53463745117189%2C37.6892542140253]%2C[-122.54699707031251%2C37.690340943717715]%2C[-122.55798339843751%2C37.72945260537781]%2C[-122.54287719726564%2C37.77831314799672]%2C[-122.49893188476564%2C37.81303878836991]%2C[-122.46185302734376%2C37.82822612280363]%2C[-122.42889404296876%2C37.82822612280363]%2C[-122.42202758789064%2C37.82280243352759]]]]} - before I fixed https://github.com/simonw/datasette-leaflet-geojson/issues/16 it was loading a LOT of maps, which felt bad. I wanted to be able to link people to that page with a hard limit on the number of rows displayed on that page.

  It's mainly to guard against unexpected behaviour from limit-less queries though. It's not a very high priority feature!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/11#issuecomment-777690332
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/11
node_id: MDEyOklzc3VlQ29tbWVudDc3NzY5MDMzMg==
created_at: 2021-02-11T18:16:01Z
updated_at: 2021-02-11T18:16:01Z
author_association: NONE
body:
  I solved this issue by modifying line 31 of utils.py

      content = ET.tostring(ET.fromstring(content_xml.strip())).decode("utf-8")

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/9#issuecomment-777821383
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/9
node_id: MDEyOklzc3VlQ29tbWVudDc3NzgyMTM4Mw==
created_at: 2021-02-11T22:01:28Z
updated_at: 2021-02-11T22:01:28Z
author_association: MEMBER
body:
  Aha! I think I've figured out what's going on here.

  The CData blocks containing the notes look like this:

  `<![CDATA[<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd"><en-note><div>This note includes two images.</div><div><br /></div>...`

  The DTD at http://xml.evernote.com/pub/enml2.dtd includes some entities:

  ```
  <!--=========== External character mnemonic entities ===================-->

  <!ENTITY % HTMLlat1 PUBLIC
     "-//W3C//ENTITIES Latin 1 for XHTML//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent">
  %HTMLlat1;

  <!ENTITY % HTMLsymbol PUBLIC
     "-//W3C//ENTITIES Symbols for XHTML//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent">
  %HTMLsymbol;

  <!ENTITY % HTMLspecial PUBLIC
     "-//W3C//ENTITIES Special for XHTML//EN"
     "http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent">
  %HTMLspecial;
  ```
  So I need to be able to handle all of those different entities. I think I can do that using `html.entities.entitydefs` from the Python standard library, which looks a bit like this:

  ```python
  {'Aacute': 'Á',
   'aacute': 'á',
   'Aacute;': 'Á',
   'aacute;': 'á',
   'Abreve;': 'Ă',
   'abreve;': 'ă',
   'ac;': '∾',
   'acd;': '∿',
  # ...
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/7#issuecomment-777827396
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/7
node_id: MDEyOklzc3VlQ29tbWVudDc3NzgyNzM5Ng==
created_at: 2021-02-11T22:13:14Z
updated_at: 2021-02-11T22:13:14Z
author_association: MEMBER
body:
  My best guess is that you have an older version of `sqlite-utils` installed here - the `replace=True` argument was added in version 2.0. I've bumped the dependency in `setup.py`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1221#issuecomment-777883452
issue_url: https://api.github.com/repos/simonw/datasette/issues/1221
node_id: MDEyOklzc3VlQ29tbWVudDc3Nzg4MzQ1Mg==
created_at: 2021-02-12T00:19:30Z
updated_at: 2021-02-12T00:19:40Z
author_association: OWNER
body:
  Uvicorn supports these options: https://www.uvicorn.org/#command-line-options
  ```
    --ssl-keyfile TEXT              SSL key file
    --ssl-certfile TEXT             SSL certificate file
    --ssl-keyfile-password TEXT     SSL keyfile password
    --ssl-version INTEGER           SSL version to use (see stdlib ssl module's)
                                    [default: 2]

    --ssl-cert-reqs INTEGER         Whether client certificate is required (see
                                    stdlib ssl module's)  [default: 0]

    --ssl-ca-certs TEXT             CA certificates file
    --ssl-ciphers TEXT              Ciphers to use (see stdlib ssl module's)
                                    [default: TLSv1]
  ```
  For the moment I'm going to support just `--ssl-keyfile` and `--ssl-certfile` as arguments to `datasette serve`. I'll add other options if people ask for them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1221#issuecomment-777887190
issue_url: https://api.github.com/repos/simonw/datasette/issues/1221
node_id: MDEyOklzc3VlQ29tbWVudDc3Nzg4NzE5MA==
created_at: 2021-02-12T00:29:18Z
updated_at: 2021-02-12T00:29:18Z
author_association: OWNER
body:
  I can use this recipe to start a `datasette` server in a sub-process during the pytest run and exercise it with real HTTP requests: https://til.simonwillison.net/pytest/subprocess-server

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1221#issuecomment-777901052
issue_url: https://api.github.com/repos/simonw/datasette/issues/1221
node_id: MDEyOklzc3VlQ29tbWVudDc3NzkwMTA1Mg==
created_at: 2021-02-12T01:09:54Z
updated_at: 2021-02-12T01:09:54Z
author_association: OWNER
body:
  I also tested this manually. I generated certificate files like so:

      cd /tmp
      python -m trustme

  This created `/tmp/server.pem`, `/tmp/client.pem` and `/tmp/server.key`

  Then I started Datasette like this:

      datasette --memory --ssl-keyfile=/tmp/server.key --ssl-certfile=/tmp/server.pem

  And exercise it using `curl` like so:

      /tmp % curl --cacert /tmp/client.pem 'https://localhost:8001/_memory.json'
      {"database": "_memory", "path": "/_memory", "size": 0, "tables": [], "hidden_count": 0, "views": [], "queries": [],
      "private": false, "allow_execute_sql": true, "query_ms": 0.8843200000114848}

  Note that without the `--cacert` option I get an error:

  ```
  /tmp % curl 'https://localhost:8001/_memory.json' 
  curl: (60) SSL certificate problem: Invalid certificate chain
  More details here: https://curl.haxx.se/docs/sslcerts.html

  curl failed to verify the legitimacy of the server and therefore could not
  establish a secure connection to it. To learn more about this situation and
  how to fix it, please visit the web page mentioned above.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1220#issuecomment-777927946
issue_url: https://api.github.com/repos/simonw/datasette/issues/1220
node_id: MDEyOklzc3VlQ29tbWVudDc3NzkyNzk0Ng==
created_at: 2021-02-12T02:29:54Z
updated_at: 2021-02-12T02:29:54Z
author_association: CONTRIBUTOR
body:
  According to https://github.com/simonw/datasette/blob/master/docs/installation.rst#using-docker it should be

  ```
  docker run -p 8001:8001 -v `pwd`:/mnt \
      datasetteproject/datasette \
      datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  ```

  This uses `/mnt/fixtures.db` whereas you're using `fixtures.db` - did you try using this path instead?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1223#issuecomment-777949755
issue_url: https://api.github.com/repos/simonw/datasette/issues/1223
node_id: MDEyOklzc3VlQ29tbWVudDc3Nzk0OTc1NQ==
created_at: 2021-02-12T03:45:31Z
updated_at: 2021-02-12T03:45:31Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=h1) Report
  > Merging [#1223](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=desc) (d1cd1f2) into [main](https://codecov.io/gh/simonw/datasette/commit/9603d893b9b72653895318c9104d754229fdb146?el=desc) (9603d89) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1223/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1223   +/-   ##
  =======================================
    Coverage   91.42%   91.42%           
  =======================================
    Files          32       32           
    Lines        3955     3955           
  =======================================
    Hits         3616     3616           
    Misses        339      339           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=footer). Last update [9603d89...d1cd1f2](https://codecov.io/gh/simonw/datasette/pull/1223?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778002092
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33
node_id: MDEyOklzc3VlQ29tbWVudDc3ODAwMjA5Mg==
created_at: 2021-02-12T06:19:32Z
updated_at: 2021-02-12T06:19:32Z
author_association: NONE
body:
  hi @leafgarland that results in a new error:
  ```
  (venv) (base) Robins-MacBook:datasette robin$ dogsheep-photos apple-photos photos.db
  Traceback (most recent call last):
    File "/Users/robin/datasette/venv/bin/dogsheep-photos", line 8, in <module>
      sys.exit(cli())
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/robin/datasette/venv/lib/python3.8/site-packages/dogsheep_photos/cli.py", line 206, in apple_photos
      db.conn.execute(
  sqlite3.OperationalError: no such table: attached.ZGENERICASSET
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1220#issuecomment-778008752
issue_url: https://api.github.com/repos/simonw/datasette/issues/1220
node_id: MDEyOklzc3VlQ29tbWVudDc3ODAwODc1Mg==
created_at: 2021-02-12T06:37:34Z
updated_at: 2021-02-12T06:37:34Z
author_association: NONE
body:
  I have used my path, I'm running it from the folder in wich I have the db.

  Do I must an absolute path?

  Do I must create exactly that folder?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778014990
issue_url: https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33
node_id: MDEyOklzc3VlQ29tbWVudDc3ODAxNDk5MA==
created_at: 2021-02-12T06:54:14Z
updated_at: 2021-02-12T06:54:14Z
author_association: NONE
body:
  Ahh, that might be because macOS Big Sur has changed the structure of the photos db. Might need to wait for a later release, there is a PR which adds support for Big Sur. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/227#issuecomment-778349142
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/227
node_id: MDEyOklzc3VlQ29tbWVudDc3ODM0OTE0Mg==
created_at: 2021-02-12T17:59:35Z
updated_at: 2021-02-12T17:59:35Z
author_association: OWNER
body:
  It looks like I can at least bump this size limit up to the maximum allowed by Python - I'll take a look at that. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778349672
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODM0OTY3Mg==
created_at: 2021-02-12T18:00:43Z
updated_at: 2021-02-12T18:00:43Z
author_association: OWNER
body:
  I could combine this with #131 to allow types to be specified in addition to column names.

  Probably need an option that means "ignore the existing heading row and use this one instead".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1220#issuecomment-778439617
issue_url: https://api.github.com/repos/simonw/datasette/issues/1220
node_id: MDEyOklzc3VlQ29tbWVudDc3ODQzOTYxNw==
created_at: 2021-02-12T20:33:27Z
updated_at: 2021-02-12T20:33:27Z
author_association: CONTRIBUTOR
body:
  That Docker command will mount your current directory inside the Docker container at `/mnt` - so you shouldn't need to change anything locally, just run

  ```
  docker run -p 8001:8001 -v `pwd`:/mnt \
      datasetteproject/datasette \
      datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
  ```

  and it will use the `fixtures.db` file within your current directory

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778508887
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/131
node_id: MDEyOklzc3VlQ29tbWVudDc3ODUwODg4Nw==
created_at: 2021-02-12T23:20:11Z
updated_at: 2021-02-12T23:20:11Z
author_association: OWNER
body:
  Annoyingly `-c` is currently a shortcut for `--csv` - so I'd have to do a major version bump to use that.

  https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L601-L603

  Particularly annoying because I attempted to remove the `-c` shortcut in https://github.com/simonw/sqlite-utils/commit/2c00567aac6d9c79087cfff0d054f64922b1473d#diff-76294b3d4afeb27e74e738daa01c26dd4dc9ccb6f4477451483a2ece1095902eL48 but forgot to remove it from the input options (I removed it from the output options).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/131#issuecomment-778510528
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/131
node_id: MDEyOklzc3VlQ29tbWVudDc3ODUxMDUyOA==
created_at: 2021-02-12T23:25:06Z
updated_at: 2021-02-12T23:25:06Z
author_association: OWNER
body:
  If `-c` isn't available, maybe `-t` or `--type` would work for specifying column types:
  ```
  sqlite-utils insert db.db images images.tsv \
    --tsv \
    --type id int \
    --type score float
  ```
  or
  ```
  sqlite-utils insert db.db images images.tsv \
    --tsv \
    -t id int \
    -t score float
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778511347
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODUxMTM0Nw==
created_at: 2021-02-12T23:27:50Z
updated_at: 2021-02-12T23:27:50Z
author_association: OWNER
body:
  For the moment, a workaround can be to `cat` an additional row onto the start of the file.

      echo "name,url,description" | cat - missing_headings.csv | sqlite-utils insert blah.db table - --csv

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811746
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxMTc0Ng==
created_at: 2021-02-14T17:39:30Z
updated_at: 2021-02-14T21:16:54Z
author_association: OWNER
body:
  I'm going to detach this from the #131 column types idea.

  The three things I need to handle here are:

  - The CSV file doesn't have a header row at all, so I need to specify what the column names should be
  - The CSV file DOES have a header row but I want to ignore it and use alternative column names
  - The CSV doesn't have a header row at all and I want to automatically use `unknown1,unknown2...` so I can start exploring it as quickly as possible.

  Here's a potential design that covers the first two:

  `--replace-header="foo,bar,baz"` - ignore whatever is in the first row and pretend it was this instead
  `--add-header="foo,bar,baz"` - add a first row with these details, to use as the header

  It doesn't cover the "give me unknown column names" case though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778811934
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxMTkzNA==
created_at: 2021-02-14T17:40:48Z
updated_at: 2021-02-14T17:40:48Z
author_association: OWNER
body:
  Another pattern that might be useful is to generate a header that is just "unknown1,unknown2,unknown3" for each of the columns in the rest of the file. This makes it easy to e.g. facet-explore within Datasette to figure out the correct names, then use `sqlite-utils transform --rename` to rename the columns.

  I needed to do that for the https://bl.iro.bl.uk/work/ns/3037474a-761c-456d-a00c-9ef3c6773f4c example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778812050
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxMjA1MA==
created_at: 2021-02-14T17:41:30Z
updated_at: 2021-02-14T17:41:30Z
author_association: OWNER
body:
  I just spotted that `csv.Sniffer` in the Python standard library has a `.has_header(sample)` method which detects if the first row appears to be a header or not, which is interesting. https://docs.python.org/3/library/csv.html#csv.Sniffer

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778812684
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxMjY4NA==
created_at: 2021-02-14T17:45:16Z
updated_at: 2021-02-14T17:45:16Z
author_association: OWNER
body:
  Running this could take any CSV (or TSV) file and automatically detect the delimiter. If no header row is detected it could add `unknown1,unknown2` headers:

      sqlite-utils insert db.db data file.csv --sniff

  (Using `--sniff` would imply `--csv`)

  This could be called `--sniffer` instead but I like `--sniff` better.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778815740
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxNTc0MA==
created_at: 2021-02-14T18:05:03Z
updated_at: 2021-02-14T18:05:03Z
author_association: OWNER
body:
  The challenge here is how to read the first 2048 bytes and then reset the incoming file.

  The Python docs example looks like this:

  ```python
  with open('example.csv', newline='') as csvfile:
      dialect = csv.Sniffer().sniff(csvfile.read(1024))
      csvfile.seek(0)
      reader = csv.reader(csvfile, dialect)
  ```
  Here's the relevant code in `sqlite-utils`: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L671-L679

  The challenge is going to be having the `--sniff` option work with the progress bar. Here's how `file_progress()` works: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/utils.py#L106-L113

  If `file.raw` is `stdin` can I do the equivalent of `csvfile.seek(0)` on it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778816333
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxNjMzMw==
created_at: 2021-02-14T18:08:44Z
updated_at: 2021-02-14T18:08:44Z
author_association: OWNER
body:
  No, you can't `.seek(0)` on stdin:
  ```
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py", line 678, in insert_upsert_implementation
      json_file.raw.seek(0)
  OSError: [Errno 29] Illegal seek
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778817494
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxNzQ5NA==
created_at: 2021-02-14T18:16:06Z
updated_at: 2021-02-14T18:16:06Z
author_association: OWNER
body:
  Types involved:
  ```
  (Pdb) type(json_file.raw)
  <class '_io.FileIO'>
  (Pdb) type(json_file)
  <class 'encodings.utf_8.StreamReader'>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778818639
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgxODYzOQ==
created_at: 2021-02-14T18:22:38Z
updated_at: 2021-02-14T18:22:38Z
author_association: OWNER
body:
  Maybe I shouldn't be using `StreamReader` at all - https://www.python.org/dev/peps/pep-0400/ suggests that it should be deprecated in favour of `io.TextIOWrapper`. I'm using `StreamReader` due to this line: https://github.com/simonw/sqlite-utils/blob/726219c3503e77440975cd15b74d006639feb0f8/sqlite_utils/cli.py#L667-L668

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778821403
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgyMTQwMw==
created_at: 2021-02-14T18:38:16Z
updated_at: 2021-02-14T18:38:16Z
author_association: OWNER
body:
  There are two code paths here that matter:

  - For a regular file, can read the first 2048 bytes, then `.seek(0)` before continuing. That's easy.
  - `stdin` is harder. I need to read and buffer the first 2048 bytes, then pass an object to `csv.reader()` which will replay that chunk and then play the rest of stdin.

  I'm a bit stuck on the second one. Ideally I could use something like `itertools.chain()` but I can't find an alternative for file-like objects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778824361
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgyNDM2MQ==
created_at: 2021-02-14T18:59:22Z
updated_at: 2021-02-14T18:59:22Z
author_association: OWNER
body:
  I think I've got it. I can use `io.BufferedReader()` to get an object I can run `.peek(2048)` on, then wrap THAT in `io.TextIOWrapper`:

  ```python
      encoding = encoding or "utf-8"
      buffered = io.BufferedReader(json_file, buffer_size=4096)
      decoded = io.TextIOWrapper(buffered, encoding=encoding, line_buffering=True)
      if pk and len(pk) == 1:
          pk = pk[0]
      if csv or tsv:
          if sniff:
              # Read first 2048 bytes and use that to detect
              first_bytes = buffered.peek(2048)
              print('first_bytes', first_bytes)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/230#issuecomment-778827570
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/230
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgyNzU3MA==
created_at: 2021-02-14T19:24:20Z
updated_at: 2021-02-14T19:24:20Z
author_association: OWNER
body:
  Here's the implementation in Python: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L204-L225

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778828758
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/231
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgyODc1OA==
created_at: 2021-02-14T19:33:14Z
updated_at: 2021-02-14T19:33:14Z
author_association: OWNER
body:
  The `limit=` parameter is currently only available on the `.search()` method - it would make sense to add this to other methods as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/231#issuecomment-778829456
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/231
node_id: MDEyOklzc3VlQ29tbWVudDc3ODgyOTQ1Ng==
created_at: 2021-02-14T19:37:52Z
updated_at: 2021-02-14T19:37:52Z
author_association: OWNER
body:
  I'm going to add `limit` and `offset` to the following methods:

  - `rows_where()`
  - `search_sql()`
  - `search()`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/234#issuecomment-778841278
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/234
node_id: MDEyOklzc3VlQ29tbWVudDc3ODg0MTI3OA==
created_at: 2021-02-14T21:02:11Z
updated_at: 2021-02-14T21:02:11Z
author_association: OWNER
body:
  I managed to replicate this in a test:
  ```python
  def test_insert_all_with_extra_columns_in_later_chunks(fresh_db):
      chunk = [
          {"record": "Record 1"},
          {"record": "Record 2"},
          {"record": "Record 3"},
          {"record": "Record 4", "extra": 1},
      ]
      fresh_db["t"].insert_all(chunk, batch_size=2, alter=True)
      assert list(fresh_db["t"].rows) == [
          {"record": "Record 1", "extra": None},
          {"record": "Record 2", "extra": None},
          {"record": "Record 3", "extra": None},
          {"record": "Record 4", "extra": 1},
      ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778842982
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODg0Mjk4Mg==
created_at: 2021-02-14T21:15:11Z
updated_at: 2021-02-14T21:15:11Z
author_association: OWNER
body:
  Implementation tip: I have code that reads the first row and uses it as headers here: https://github.com/simonw/sqlite-utils/blob/8f042ae1fd323995d966a94e8e6df85cc843b938/sqlite_utils/cli.py#L689-L691

  So If I want to use `unknown1,unknown2...` I can do that by reading the first row, counting the number of columns, generating headers based on that range and then continuing to build that generator (maybe with `itertools.chain()` to replay the record we already read).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778843086
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODg0MzA4Ng==
created_at: 2021-02-14T21:15:43Z
updated_at: 2021-02-14T21:15:43Z
author_association: OWNER
body:
  I'm not convinced the `.has_header()` rules are useful for the kind of CSV files I work with: https://github.com/python/cpython/blob/63298930fb531ba2bb4f23bc3b915dbf1e17e9e1/Lib/csv.py#L383

  ```python
      def has_header(self, sample):
          # Creates a dictionary of types of data in each column. If any
          # column is of a single type (say, integers), *except* for the first
          # row, then the first row is presumed to be labels. If the type
          # can't be determined, it is assumed to be a string in which case
          # the length of the string is the determining factor: if all of the
          # rows except for the first are the same length, it's a header.
          # Finally, a 'vote' is taken at the end for each column, adding or
          # subtracting from the likelihood of the first row being a header.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778843503
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/229
node_id: MDEyOklzc3VlQ29tbWVudDc3ODg0MzUwMw==
created_at: 2021-02-14T21:18:51Z
updated_at: 2021-02-14T21:18:51Z
author_association: OWNER
body:
  I want to set this to the maximum allowed limit, which seems to be surprisingly hard! That StackOverflow thread is full of ideas for that, many of them involving `ctypes`. I'm a bit loathe to add a dependency on `ctypes` though - even though it's in the Python standard library I worry that it might not be available on some architectures.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/229#issuecomment-778844016
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/229
node_id: MDEyOklzc3VlQ29tbWVudDc3ODg0NDAxNg==
created_at: 2021-02-14T21:22:45Z
updated_at: 2021-02-14T21:22:45Z
author_association: OWNER
body:
  I'm going to use this pattern from https://stackoverflow.com/a/15063941
  ```python
  import sys
  import csv
  maxInt = sys.maxsize

  while True:
      # decrease the maxInt value by factor 10 
      # as long as the OverflowError occurs.

      try:
          csv.field_size_limit(maxInt)
          break
      except OverflowError:
          maxInt = int(maxInt/10)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778849394
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODg0OTM5NA==
created_at: 2021-02-14T22:06:53Z
updated_at: 2021-02-14T22:06:53Z
author_association: OWNER
body:
  For the moment I think just adding `--no-header` - which causes column names "unknown1,unknown2,..." to be used - should be enough.

  Users can import with that option, then use `sqlite-utils transform --rename` to rename them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/228#issuecomment-778851721
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/228
node_id: MDEyOklzc3VlQ29tbWVudDc3ODg1MTcyMQ==
created_at: 2021-02-14T22:23:46Z
updated_at: 2021-02-14T22:23:46Z
author_association: OWNER
body:
  I called this `--no-headers` for consistency with the existing output option: https://github.com/simonw/sqlite-utils/blob/427dace184c7da57f4a04df07b1e84cdae3261e8/sqlite_utils/cli.py#L61-L64

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779409770
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQwOTc3MA==
created_at: 2021-02-15T19:23:11Z
updated_at: 2021-02-15T19:23:11Z
author_association: OWNER
body:
  On my Mac right now I'm seeing a limit of 500,000:
  ```
  % sqlite3 -cmd ".limits variable_number"
       variable_number 500000
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779416619
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQxNjYxOQ==
created_at: 2021-02-15T19:40:57Z
updated_at: 2021-02-15T21:27:55Z
author_association: OWNER
body:
  Tried this experiment (not proper binary search, it only searches downwards):
  ```python
  import sqlite3

  db = sqlite3.connect(":memory:")

  def tryit(n):
      sql = "select 1 where 1 in ({})".format(", ".join("?" for i in range(n)))
      db.execute(sql, [0 for i in range(n)])


  def find_limit(min=0, max=5_000_000):
      value = max
      while True:
          print('Trying', value)
          try:
              tryit(value)
              return value
          except:
              value = value // 2
  ```
  Running `find_limit()` with those default parameters takes about 1.47s on my laptop:
  ```
  In [9]: %timeit find_limit()
  Trying 5000000
  Trying 2500000...
  1.47 s ± 28 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  ```
  Interestingly the value it suggested was 156250 - suggesting that the macOS `sqlite3` binary with a 500,000 limit isn't the same as whatever my Python is using here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779417723
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQxNzcyMw==
created_at: 2021-02-15T19:44:02Z
updated_at: 2021-02-15T19:47:00Z
author_association: OWNER
body:
  `%timeit find_limit(max=1_000_000)` took 378ms on my laptop

  `%timeit find_limit(max=500_000)` took 197ms

  `%timeit find_limit(max=200_000)` reported 53ms per loop

  `%timeit find_limit(max=100_000)` reported 26.8ms per loop.

  All of these are still slow enough that I'm not comfortable running this search for every time the library is imported. Allowing users to opt-in to this as a performance enhancement might be better.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779445423
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQ0NTQyMw==
created_at: 2021-02-15T21:00:44Z
updated_at: 2021-02-15T21:01:09Z
author_association: OWNER
body:
  I tried changing the hard-coded value from 999 to 156_250 and running `sqlite-utils insert` against a 500MB CSV file, with these results:
  ```
  (sqlite-utils) sqlite-utils % time sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
    [###################################-]   99%  00:00:00sqlite-utils insert slow-ethos.db ethos ../ethos-datasette/ethos.csv
  44.74s user 7.61s system 92% cpu 56.601 total
  # Increased the setting here
  (sqlite-utils) sqlite-utils % time sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv --no-headers
    [###################################-]   99%  00:00:00sqlite-utils insert fast-ethos.db ethos ../ethos-datasette/ethos.csv
  39.40s user 5.15s system 96% cpu 46.320 total
  ```
  Not as big a difference as I was expecting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779446652
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQ0NjY1Mg==
created_at: 2021-02-15T21:04:19Z
updated_at: 2021-02-15T21:04:19Z
author_association: OWNER
body:
  ... but it looks like `batch_size` is hard-coded to 100, rather than `None` - which means it's not being calculated using that value:

  https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L704

  And

  https://github.com/simonw/sqlite-utils/blob/1f49f32814a942fa076cfe5f504d1621188097ed/sqlite_utils/db.py#L1877

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/147#issuecomment-779448912
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/147
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQ0ODkxMg==
created_at: 2021-02-15T21:09:50Z
updated_at: 2021-02-15T21:09:50Z
author_association: OWNER
body:
  I fiddled around and replaced that line with `batch_size = SQLITE_MAX_VARS // num_columns` - which evaluated to `10416` for this particular file. That got me this:

         40.71s user 1.81s system 98% cpu 43.081 total

  43s is definitely better than 56s, but it's still not as big as the  ~26.5s to ~3.5s improvement described by @simonwiles at the top of this issue. I wonder what I'm missing here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1226#issuecomment-779467160
issue_url: https://api.github.com/repos/simonw/datasette/issues/1226
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQ2NzE2MA==
created_at: 2021-02-15T22:01:53Z
updated_at: 2021-02-15T22:01:53Z
author_association: OWNER
body:
  This check needs to happen in two places:

  https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L222-L227

  https://github.com/simonw/datasette/blob/9603d893b9b72653895318c9104d754229fdb146/datasette/cli.py#L328-L333

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1226#issuecomment-779467451
issue_url: https://api.github.com/repos/simonw/datasette/issues/1226
node_id: MDEyOklzc3VlQ29tbWVudDc3OTQ2NzQ1MQ==
created_at: 2021-02-15T22:02:46Z
updated_at: 2021-02-15T22:02:46Z
author_association: OWNER
body:
  I'm OK with the current error message shown if you try to use too low a port:
  ```
  datasette fivethirtyeight.db -p 800 
  INFO:     Started server process [45511]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  ERROR:    [Errno 13] error while attempting to bind on address ('127.0.0.1', 800): permission denied
  INFO:     Waiting for application shutdown.
  INFO:     Application shutdown complete.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1229#issuecomment-780830464
issue_url: https://api.github.com/repos/simonw/datasette/issues/1229
node_id: MDEyOklzc3VlQ29tbWVudDc4MDgzMDQ2NA==
created_at: 2021-02-17T20:24:30Z
updated_at: 2021-03-29T00:17:21Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=h1) Report
  > Merging [#1229](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=desc) (a095248) into [main](https://codecov.io/gh/simonw/datasette/commit/8e18c7943181f228ce5ebcea48deb59ce50bee1f?el=desc) (8e18c79) will **not change** coverage.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1229/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1229   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1229/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=footer). Last update [8e18c79...a095248](https://codecov.io/gh/simonw/datasette/pull/1229?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-780991910
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4MDk5MTkxMA==
created_at: 2021-02-18T02:13:56Z
updated_at: 2021-02-18T02:13:56Z
author_association: NONE
body:
  I was going ask you about this issue when we talk during your office-hours schedule this Friday, but was there any support ever added for doing this cross-database joining?

  I have a use-case where could be pretty neat to do analysis using this tool on time-specific databases from snapshots

  https://ilsweb.cincinnatilibrary.org/collection-analysis/

  ![image](https://user-images.githubusercontent.com/9308268/108294883-ba3a8e00-7164-11eb-9206-fcd5a8cdd883.png)

  and thanks again for such an amazing tool!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-781077127
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4MTA3NzEyNw==
created_at: 2021-02-18T05:56:30Z
updated_at: 2021-02-18T05:57:34Z
author_association: OWNER
body:
  I'm going to to try prototyping the `--crossdb` option that causes `/_memory` to connect to all databases as a starting point and see how well that works.

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/1230#issuecomment-781330466
issue_url: https://api.github.com/repos/simonw/datasette/issues/1230
node_id: MDEyOklzc3VlQ29tbWVudDc4MTMzMDQ2Ng==
created_at: 2021-02-18T13:06:22Z
updated_at: 2021-02-18T15:22:15Z
author_association: NONE
body:
  [Edit] Oh, I just saw the "Load all" button under the cluster map as well as the [setting to alter the max number or results](https://docs.datasette.io/en/stable/settings.html#max-returned-rows). So I guess this issue only is about the Vega charts.

  <details>
  Note that datasette-cluster-map also seems to be limited to 998 displayed points: 

  ![ss-2021-02-18_140548](https://user-images.githubusercontent.com/7107523/108361225-15fb2a80-71ea-11eb-9a19-d885e8513f55.png)
  </details>

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1226#issuecomment-781530157
issue_url: https://api.github.com/repos/simonw/datasette/issues/1226
node_id: MDEyOklzc3VlQ29tbWVudDc4MTUzMDE1Nw==
created_at: 2021-02-18T18:00:15Z
updated_at: 2021-02-18T18:00:15Z
author_association: OWNER
body:
  I can use `click.IntRange(min=None, max=None)` for this. https://click.palletsprojects.com/en/7.x/options/#ranges - inclusive on both edges.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1231#issuecomment-781560865
issue_url: https://api.github.com/repos/simonw/datasette/issues/1231
node_id: MDEyOklzc3VlQ29tbWVudDc4MTU2MDg2NQ==
created_at: 2021-02-18T18:50:38Z
updated_at: 2021-02-18T18:50:38Z
author_association: OWNER
body:
  I started trying to use locks to resolve this but I've not figured out the right way to do that yet - here's my first experiment:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 9e15a16..1681c9d 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -217,6 +217,7 @@ class Datasette:
           self.inspect_data = inspect_data
           self.immutables = set(immutables or [])
           self.databases = collections.OrderedDict()
  +        self._refresh_schemas_lock = threading.Lock()
           if memory or not self.files:
               self.add_database(Database(self, is_memory=True), name="_memory")
           # memory_name is a random string so that each Datasette instance gets its own
  @@ -324,6 +325,13 @@ class Datasette:
           self.client = DatasetteClient(self)
   
       async def refresh_schemas(self):
  +        return
  +        if self._refresh_schemas_lock.locked():
  +            return
  +        with self._refresh_schemas_lock:
  +            await self._refresh_schemas()
  +
  +    async def _refresh_schemas(self):
           internal_db = self.databases["_internal"]
           if not self.internal_db_created:
               await init_internal_db(internal_db)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-781573676
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4MTU3MzY3Ng==
created_at: 2021-02-18T19:13:30Z
updated_at: 2021-02-18T19:13:30Z
author_association: OWNER
body:
  It turns out SQLite defaults to a maximum of 10 attached databases. This can be increased using a compile-time constant, but even with that it cannot be more than 62: https://stackoverflow.com/questions/9845448/attach-limit-10

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-781574786
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4MTU3NDc4Ng==
created_at: 2021-02-18T19:15:37Z
updated_at: 2021-02-18T19:15:37Z
author_association: OWNER
body:
  `select * from pragma_database_list();` is useful - shows all attached databases for the current connection.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-781591015
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4MTU5MTAxNQ==
created_at: 2021-02-18T19:44:02Z
updated_at: 2021-02-18T19:44:02Z
author_association: OWNER
body:
  For the moment I'm going to hard-code a `SQLITE_LIMIT_ATTACHED=10` constant and only attach the first 10 databases to the `_memory` connection.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1232#issuecomment-781594632
issue_url: https://api.github.com/repos/simonw/datasette/issues/1232
node_id: MDEyOklzc3VlQ29tbWVudDc4MTU5NDYzMg==
created_at: 2021-02-18T19:50:21Z
updated_at: 2021-02-18T19:50:21Z
author_association: OWNER
body:
  It would be neat if the `/_memory` page showed a list of attached databases, to indicate that the `--crossdb` option is working and give people links to click to start running queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1232#issuecomment-781598585
issue_url: https://api.github.com/repos/simonw/datasette/issues/1232
node_id: MDEyOklzc3VlQ29tbWVudDc4MTU5ODU4NQ==
created_at: 2021-02-18T19:57:30Z
updated_at: 2021-02-18T19:57:30Z
author_association: OWNER
body:
  It would also be neat if https://latest.datasette.io/ had multiple databases attached in order to demonstrate this feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1232#issuecomment-781599929
issue_url: https://api.github.com/repos/simonw/datasette/issues/1232
node_id: MDEyOklzc3VlQ29tbWVudDc4MTU5OTkyOQ==
created_at: 2021-02-18T19:59:54Z
updated_at: 2021-02-18T22:06:42Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=h1) Report
  > Merging [#1232](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=desc) (8876499) into [main](https://codecov.io/gh/simonw/datasette/commit/4df548e7668b5b21d64a267964951e67894f4712?el=desc) (4df548e) will **increase** coverage by `0.03%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1232/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1232      +/-   ##
  ==========================================
  + Coverage   91.42%   91.46%   +0.03%     
  ==========================================
    Files          32       32              
    Lines        3955     3970      +15     
  ==========================================
  + Hits         3616     3631      +15     
    Misses        339      339              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.68% <100.00%> (+0.06%)` | :arrow_up: |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `76.62% <100.00%> (+0.36%)` | :arrow_up: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1232/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.19% <100.00%> (+0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=footer). Last update [4df548e...8876499](https://codecov.io/gh/simonw/datasette/pull/1232?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1232#issuecomment-781629841
issue_url: https://api.github.com/repos/simonw/datasette/issues/1232
node_id: MDEyOklzc3VlQ29tbWVudDc4MTYyOTg0MQ==
created_at: 2021-02-18T20:57:23Z
updated_at: 2021-02-18T20:57:23Z
author_association: OWNER
body:
  The new warning looks like this:

  <img width="514" alt="datasette_—_pipenv_shell_▸_Python_—_182×66" src="https://user-images.githubusercontent.com/9599/108420562-d41db480-71e8-11eb-87e3-1cddd65627c2.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1233#issuecomment-781636590
issue_url: https://api.github.com/repos/simonw/datasette/issues/1233
node_id: MDEyOklzc3VlQ29tbWVudDc4MTYzNjU5MA==
created_at: 2021-02-18T21:10:08Z
updated_at: 2021-02-18T21:10:08Z
author_association: OWNER
body:
  I think the bug is here: https://github.com/simonw/datasette/blob/640ac7071b73111ba4423812cd683756e0e1936b/datasette/utils/__init__.py#L349-L373

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1232#issuecomment-781637292
issue_url: https://api.github.com/repos/simonw/datasette/issues/1232
node_id: MDEyOklzc3VlQ29tbWVudDc4MTYzNzI5Mg==
created_at: 2021-02-18T21:11:31Z
updated_at: 2021-02-18T21:11:31Z
author_association: OWNER
body:
  Due to bug #1233 I'm going to publish the additional database as `extra_database.db` rather than `extra database.db` as it is used in the tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1232#issuecomment-781641728
issue_url: https://api.github.com/repos/simonw/datasette/issues/1232
node_id: MDEyOklzc3VlQ29tbWVudDc4MTY0MTcyOA==
created_at: 2021-02-18T21:19:34Z
updated_at: 2021-02-18T21:19:34Z
author_association: OWNER
body:
  I tested the demo deployment like this:
  ```
  datasette publish cloudrun fixtures.db extra_database.db \                                      
              -m fixtures.json \
              --plugins-dir=plugins \
              --branch=crossdb \
              --extra-options="--setting template_debug 1 --crossdb" \
              --install=pysqlite3-binary \
              --service=datasette-latest-crossdb
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1232#issuecomment-781651283
issue_url: https://api.github.com/repos/simonw/datasette/issues/1232
node_id: MDEyOklzc3VlQ29tbWVudDc4MTY1MTI4Mw==
created_at: 2021-02-18T21:37:55Z
updated_at: 2021-02-18T21:37:55Z
author_association: OWNER
body:
  UI listing the attached tables:

  <img width="888" alt="_memory" src="https://user-images.githubusercontent.com/9599/108424809-8015ce80-71ee-11eb-8752-03c459f89320.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-781665560
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4MTY2NTU2MA==
created_at: 2021-02-18T22:06:14Z
updated_at: 2021-02-18T22:06:14Z
author_association: OWNER
body:
  The implementation in #1232 is ready to land. It's the simplest-thing-that-could-possibly-work: you can run `datasette one.db two.db three.db --crossdb` and then use the `/_memory` page to run joins across tables from multiple databases.

  It only works on the first 10 databases that were passed to the command-line. This means that if you have a Datasette instance with hundreds of attached databases (see [Datasette Library](https://github.com/simonw/datasette/issues/417)) this won't be particularly useful for you.

  So... a better, future version of this feature would be one that lets you join across databases on command - maybe by hitting `/_memory?attach=db1&attach=db2` to get a special connection.

  Also worth noting: plugins that implement the [prepare_connection()](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette) hook can attach additional databases - so if you need better, customized support for this one way to handle that would be with a custom plugin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-781670827
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4MTY3MDgyNw==
created_at: 2021-02-18T22:16:46Z
updated_at: 2021-02-18T22:16:46Z
author_association: OWNER
body:
  Demo is now live here: https://latest.datasette.io/_memory

  The documentation is at https://docs.datasette.io/en/latest/sql_queries.html#cross-database-queries - it links to this example query: https://latest.datasette.io/_memory?sql=select%0D%0A++%27fixtures%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bfixtures%5D.sqlite_master%0D%0Aunion%0D%0Aselect%0D%0A++%27extra_database%27+as+database%2C+*%0D%0Afrom%0D%0A++%5Bextra_database%5D.sqlite_master

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1235#issuecomment-781735887
issue_url: https://api.github.com/repos/simonw/datasette/issues/1235
node_id: MDEyOklzc3VlQ29tbWVudDc4MTczNTg4Nw==
created_at: 2021-02-19T00:50:21Z
updated_at: 2021-02-19T00:50:55Z
author_association: OWNER
body:
  I'll bump to `3.7.10` for the moment - the fix for 3.8 isn't out until March 1st according to https://news.ycombinator.com/item?id=26186434

  https://www.python.org/downloads/release/python-3710/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1235#issuecomment-781736855
issue_url: https://api.github.com/repos/simonw/datasette/issues/1235
node_id: MDEyOklzc3VlQ29tbWVudDc4MTczNjg1NQ==
created_at: 2021-02-19T00:52:47Z
updated_at: 2021-02-19T01:47:53Z
author_association: OWNER
body:
  I bumped the two lines in the `Dockerfile` to `FROM python:3.7.10-slim-stretch as build` and ran this to build it:

      docker build -f Dockerfile -t datasetteproject/datasette:python-3-7-10 .

  Then I ran it with:

      docker run -p 8001:8001 -v `pwd`:/mnt datasetteproject/datasette:python-3-7-10 datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

  http://0.0.0.0:8001/-/versions confirmed that it was now running Python 3.7.10

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/113#issuecomment-781825187
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/113
node_id: MDEyOklzc3VlQ29tbWVudDc4MTgyNTE4Nw==
created_at: 2021-02-19T05:09:12Z
updated_at: 2021-02-19T05:09:12Z
author_association: OWNER
body:
  Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#attaching-additional-databases

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1236#issuecomment-782458983
issue_url: https://api.github.com/repos/simonw/datasette/issues/1236
node_id: MDEyOklzc3VlQ29tbWVudDc4MjQ1ODk4Mw==
created_at: 2021-02-19T23:43:34Z
updated_at: 2021-02-19T23:43:34Z
author_association: OWNER
body:
  I only want it to resize up and down, not left to right - so I'm not keen on the default resize handle:

  <img width="722" alt="cm-resize_demo" src="https://user-images.githubusercontent.com/9599/108573363-364de680-72c9-11eb-8741-5112463ebfaa.png">

  https://rawgit.com/Sphinxxxx/cm-resize/master/demo/index.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1236#issuecomment-782459405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1236
node_id: MDEyOklzc3VlQ29tbWVudDc4MjQ1OTQwNQ==
created_at: 2021-02-19T23:45:02Z
updated_at: 2021-02-19T23:45:02Z
author_association: OWNER
body:
  I'm going to use a variant of the Datasette menu icon. Here it is in `#ccc` with an ARIA label:

  ```svg
  <svg aria-labelledby="cm-drag-to-resize" role="img" fill="#ccc" stroke="#ccc" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16">
    <title id="cm-drag-to-resize">Drag to resize</title>
    <path fill-rule="evenodd" d="M1 2.75A.75.75 0 011.75 2h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 2.75zm0 5A.75.75 0 011.75 7h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 7.75zM1.75 12a.75.75 0 100 1.5h12.5a.75.75 0 100-1.5H1.75z"></path>
  </svg>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1236#issuecomment-782459550
issue_url: https://api.github.com/repos/simonw/datasette/issues/1236
node_id: MDEyOklzc3VlQ29tbWVudDc4MjQ1OTU1MA==
created_at: 2021-02-19T23:45:30Z
updated_at: 2021-02-19T23:45:30Z
author_association: OWNER
body:
  Encoded using https://meyerweb.com/eric/tools/dencoder/

  `%3Csvg%20aria-labelledby%3D%22cm-drag-to-resize%22%20role%3D%22img%22%20fill%3D%22%23ccc%22%20stroke%3D%22%23ccc%22%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20viewBox%3D%220%200%2016%2016%22%20width%3D%2216%22%20height%3D%2216%22%3E%0A%20%20%3Ctitle%20id%3D%22cm-drag-to-resize%22%3EDrag%20to%20resize%3C%2Ftitle%3E%0A%20%20%3Cpath%20fill-rule%3D%22evenodd%22%20d%3D%22M1%202.75A.75.75%200%20011.75%202h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%202.75zm0%205A.75.75%200%20011.75%207h12.5a.75.75%200%20110%201.5H1.75A.75.75%200%20011%207.75zM1.75%2012a.75.75%200%20100%201.5h12.5a.75.75%200%20100-1.5H1.75z%22%3E%3C%2Fpath%3E%0A%3C%2Fsvg%3E`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1236#issuecomment-782462049
issue_url: https://api.github.com/repos/simonw/datasette/issues/1236
node_id: MDEyOklzc3VlQ29tbWVudDc4MjQ2MjA0OQ==
created_at: 2021-02-19T23:51:12Z
updated_at: 2021-02-19T23:51:12Z
author_association: OWNER
body:
  ![resize-demo](https://user-images.githubusercontent.com/9599/108573758-4914eb00-72ca-11eb-989c-e642eee68021.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782708938
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4MjcwODkzOA==
created_at: 2021-02-20T16:22:14Z
updated_at: 2021-02-20T16:22:14Z
author_association: OWNER
body:
  I'm leaning back in the direction of a flat JSON array of objects as the default - this:

  `/fixtures/roadside_attractions.json`

  Would return:

  ```json
  [
      {
        "pk": 1,
        "name": "The Mystery Spot",
        "address": "465 Mystery Spot Road, Santa Cruz, CA 95065",
        "latitude": 37.0167,
        "longitude": -122.0024
      },
      {
        "pk": 2,
        "name": "Winchester Mystery House",
        "address": "525 South Winchester Boulevard, San Jose, CA 95128",
        "latitude": 37.3184,
        "longitude": -121.9511
      },
      {
        "pk": 3,
        "name": "Burlingame Museum of PEZ Memorabilia",
        "address": "214 California Drive, Burlingame, CA 94010",
        "latitude": 37.5793,
        "longitude": -122.3442
      },
      {
        "pk": 4,
        "name": "Bigfoot Discovery Museum",
        "address": "5497 Highway 9, Felton, CA 95018",
        "latitude": 37.0414,
        "longitude": -122.0725
      }
  ]
  ```
  To get the version that includes pagination information you would use the `?_extra=` parameter. For example:

  `/fixtures/roadside_attractions.json?_extra=total&_extra=next_url`

  ```json
  {
    "rows": [
      {
        "pk": 1,
        "name": "The Mystery Spot",
        "address": "465 Mystery Spot Road, Santa Cruz, CA 95065",
        "latitude": 37.0167,
        "longitude": -122.0024
      },
      {
        "pk": 2,
        "name": "Winchester Mystery House",
        "address": "525 South Winchester Boulevard, San Jose, CA 95128",
        "latitude": 37.3184,
        "longitude": -121.9511
      },
      {
        "pk": 3,
        "name": "Burlingame Museum of PEZ Memorabilia",
        "address": "214 California Drive, Burlingame, CA 94010",
        "latitude": 37.5793,
        "longitude": -122.3442
      },
      {
        "pk": 4,
        "name": "Bigfoot Discovery Museum",
        "address": "5497 Highway 9, Felton, CA 95018",
        "latitude": 37.0414,
        "longitude": -122.0725
      }
    ],
    "total": 4,
    "next_url": null
  }
  ```
  ANY usage of the `?_extra=` parameter would turn the list into an object with a `"rows"` key.

  Opting in to the `total` is nice because it's actually expensive to run a count, so only doing a count if the user requests it feels good.

  But... having to add `?_extra=total&_extra=next_url` for the common case of wanting both the total count and the URL to get the next page of results is a bit verbose. So maybe support aliases, like `?_extra=paginated` which is a shortcut for `?_extra=total&_extra=next_url`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782709270
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4MjcwOTI3MA==
created_at: 2021-02-20T16:23:51Z
updated_at: 2021-02-20T16:24:11Z
author_association: OWNER
body:
  Also how would you opt out of returning the `"rows"` key? I sometimes want to do this - if I want to get back just the count or just the facets for example.

  Some options:

  * `/fixtures/roadside_attractions.json?_extra=total&_extra=-rows`
  * `/fixtures/roadside_attractions.json?_extra=total&_skip=rows`
  * `/fixtures/roadside_attractions.json?_extra=total&_size=0`

  I quite like that last one with `?_size=0`. I think it would still return `"rows": []` but that's OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782739926
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4MjczOTkyNg==
created_at: 2021-02-20T19:51:30Z
updated_at: 2021-02-20T19:52:19Z
author_association: OWNER
body:
  Demos:

  - https://latest-with-plugins.datasette.io/github/commits.json-preview
  - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url
  - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total
  - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=next_url&_extra=total
  - https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782740488
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0MDQ4OA==
created_at: 2021-02-20T19:55:23Z
updated_at: 2021-02-20T19:55:23Z
author_association: OWNER
body:
  Am I saying you won't get back a key in the response unless you explicitly request it, either by name or by specifying a bundle of extras (e.g. `all` or `paginated`)?

  The `"truncated": true` key that tells you that your arbitrary query returned more than X results but was truncated is pretty important, do I really want people to have to opt-in to that one?

  Also: having bundles like `all` or `paginated` live in the same namespace as single keys like `next_url` or `total` is a little odd - you can't tell by looking at them if they'll add a key called `all` or if they'll add a bunch of other stuff.

  Maybe bundles could be prefixed with something, perhaps an underscore? `?_extra=_all` and `?_extra=_paginated` for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782740604
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0MDYwNA==
created_at: 2021-02-20T19:56:21Z
updated_at: 2021-02-20T19:56:33Z
author_association: OWNER
body:
  I think I want to support `?_extra=next_url,total` in addition to `?_extra=next_url&_extra=total` - partly because it's less characters to type, and also because I know there exist URL handling library that don't know how to handle the same parameter multiple times (though they're going to break against Datasette already, so it's not a big deal).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782740985
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0MDk4NQ==
created_at: 2021-02-20T19:59:21Z
updated_at: 2021-02-20T19:59:21Z
author_association: OWNER
body:
  This design should be influenced by how it's implemented.

  One implementation that could be nice is that each of the keys that can be requested - `next_url`, `total` etc - maps to an `async def` function which can do the work. So that expensive `count(*)` will only be executed by the `async def total` function if it is requested.

  This raises more questions: Both `next` and `next_url` work off the same underlying data, so if they are both requested can we re-use the work that `next` does somehow? Maybe by letting these functions depend on each other (so `next_url()` knows to first call `next()`, but only if it hasn't been called already.

  I think I need to flesh out the full default collection of `?_extra=` parameters in order to design how they will work under the hood.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782741107
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0MTEwNw==
created_at: 2021-02-20T20:00:22Z
updated_at: 2021-02-20T20:00:22Z
author_association: OWNER
body:
  A really exciting opportunity this opens up is for parallel execution - the `facets()` and `suggested_facets()` and `total()` async functions could be called in parallel, which could speed things up if I'm confident the SQLite thread pool can execute on multiple CPU cores (it should be able to because the Python `sqlite3` module releases the GIL while it's executing C code).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782741719
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0MTcxOQ==
created_at: 2021-02-20T20:05:04Z
updated_at: 2021-02-20T20:05:04Z
author_association: OWNER
body:
  > The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.

  This is something I've not thought very hard about. If there's an error, I need to return a top-level object, not a top-level array, so I can provide details of the error.

  But this means that client code will have to handle this difference - it will have to know that the returned data can be array-shaped if nothing went wrong, and object-shaped if there's an error.

  The HTTP status code helps here - calling client code can know that a 200 status code means there will be an array, but an error status code means an object.

  If developers really hate that the shape could be different, they can always use `?_extra=next` to ensure that the top level item is an object whether or not an error occurred. So I think this is OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782742233
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0MjIzMw==
created_at: 2021-02-20T20:09:16Z
updated_at: 2021-02-20T20:09:16Z
author_association: OWNER
body:
  I just noticed that https://latest-with-plugins.datasette.io/github/commits.json-preview?_extra=total&_size=0&_trace=1 executes 35 SQL queries at the moment! A great reminder that a big improvement from this change will be a reduction in queries through not calculating things like suggested facets unless they are explicitly requested.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782745199
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0NTE5OQ==
created_at: 2021-02-20T20:32:03Z
updated_at: 2021-02-20T20:32:03Z
author_association: NONE
body:
  I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default. Mainly because it allows you to add extra keys in a backwards-compatible way. Also just seems more expected somehow.

  The API design guidance for the UK government also recommends this: https://www.gov.uk/guidance/gds-api-technical-and-data-standards#use-json

  I also strongly dislike having versioned APIs (eg with a `/v1/` path prefix, as it invariably means that old versions stop working at some point, even though the bit of the API you’re using might not have changed at all.

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1}


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782746633
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0NjYzMw==
created_at: 2021-02-20T20:43:07Z
updated_at: 2021-02-20T20:43:07Z
author_association: OWNER
body:
  Another option: `.json` always returns an object with a list of keys that gets increased through adding `?_extra=` parameters.

  `.jsona` always returns a JSON array of objects

  I had something similar to this in Datasette a few years ago - a `.jsono` extension, which still redirects to the `shape=array` version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782746755
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0Njc1NQ==
created_at: 2021-02-20T20:44:05Z
updated_at: 2021-02-20T20:44:05Z
author_association: NONE
body:
  Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?

  I like the idea of specifying a limit of 0 if you don’t want any rows data - and returning an empty array under the `rows` key seems fine.

  Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782747743
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzc0Mw==
created_at: 2021-02-20T20:52:10Z
updated_at: 2021-02-20T20:52:10Z
author_association: OWNER
body:
  > Minor suggestion: rename `size` query param to `limit`, to better reflect that it’s a maximum number of rows returned rather than a guarantee of getting that number, and also for consistency with the SQL keyword?

  The problem there is that `?_size=x` isn't actually doing the same thing as the SQL `limit` keyword. Consider this query:

  https://latest-with-plugins.datasette.io/github?sql=select+*+from+commits - `select * from commits`

  Datasette returns 1,000 results, and shows a "Custom SQL query returning more than 1,000 rows" message at the top. That's the `size` kicking in - I only fetch the first 1,000 results from the cursor to avoid exhausting resources. In the JSON version of that at https://latest-with-plugins.datasette.io/github.json?sql=select+*+from+commits there's a `"truncated": true` key to let you know what happened.

  I find myself using `?_size=2` against Datasette occasionally if I know the rows being returned are really big and I don't want to load 10+MB of HTML.

  This is only really a concern for arbitrary SQL queries though - for table pages such as https://latest-with-plugins.datasette.io/github/commits?_size=10 adding `?_size=10` actually puts a `limit 10` on the underlying SQL query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782747878
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0Nzg3OA==
created_at: 2021-02-20T20:53:11Z
updated_at: 2021-02-20T20:53:11Z
author_association: OWNER
body:
  ... though thinking about this further, I could re-implement the `select * from commits` (but only return a max of 10 results) feature using a nested `select * from (select * from commits) limit 10` query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782748093
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0ODA5Mw==
created_at: 2021-02-20T20:54:52Z
updated_at: 2021-02-20T20:54:52Z
author_association: OWNER
body:
  > Have you given any thought as to whether to pretty print (format with spaces) the output or not? Can be useful for debugging/exploring in a browser or other basic tools which don’t parse the JSON. Could be default (can’t be much bigger with gzip?) or opt-in.

  Adding a `?_pretty=1` option that does that is a great idea, I'm filing a ticket for it: #1237

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782748501
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc0ODUwMQ==
created_at: 2021-02-20T20:58:18Z
updated_at: 2021-02-20T20:58:18Z
author_association: OWNER
body:
  Yet another option: support a `?_path=x` option which returns a nested path from the result. So you could do this:

  `/github/commits.json?_path=rows` - to get back a top-level array pulled from the `"rows"` key.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782756398
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc1NjM5OA==
created_at: 2021-02-20T22:05:48Z
updated_at: 2021-02-20T22:05:48Z
author_association: NONE
body:
  > I think it’s a good idea if the top level item of the response JSON is always an object, rather than an array, at least as the default.

  I agree it is more predictable if the top level item is an object with a rows or data object that contains an array of data, which then allows for other top-level meta data. 

  I can see the argument for removing this and just using an array for convenience - but I think that's OK as an option (as you have now).

  Rather than have lots of top-level keys you could have a "meta" object to contain non-data stuff. You could use something like "links" for API endpoint URLs (or use a standard like HAL). Which would then leave the top level a bit cleaner - if that's what you what. 

  Have you had much feedback from users who use the Datasette API a lot?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782765665
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc2NTY2NQ==
created_at: 2021-02-20T23:34:41Z
updated_at: 2021-02-20T23:34:41Z
author_association: OWNER
body:
  OK, I'm back to the "top level object as the default" side of things now - it's pretty much unanimous at this point, and it's certainly true that it's not a decision you'll even regret.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-782789598
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4Mjc4OTU5OA==
created_at: 2021-02-21T03:30:02Z
updated_at: 2021-02-21T03:30:02Z
author_association: OWNER
body:
  Another benefit to default:object - I could include a key that shows a list of available extras. I could then use that to power an interactive API explorer.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-783265830
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDc4MzI2NTgzMA==
created_at: 2021-02-22T10:21:14Z
updated_at: 2021-02-22T10:21:14Z
author_association: NONE
body:
  @simonw:

  > The problem there is that ?_size=x isn't actually doing the same thing as the SQL limit keyword.

  Interesting! Although I don't think it matters too much what the underlying implementation is - I more meant that `limit` is familiar to developers conceptually as "up to and including this number, if they exist", whereas "size" is potentially more ambiguous. However, it's probably no big deal either way.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1166#issuecomment-783560017
issue_url: https://api.github.com/repos/simonw/datasette/issues/1166
node_id: MDEyOklzc3VlQ29tbWVudDc4MzU2MDAxNw==
created_at: 2021-02-22T18:00:57Z
updated_at: 2021-02-22T18:13:11Z
author_association: NONE
body:
  Hi! I don't think Prettier supports this syntax for globs: `datasette/static/*[!.min].js` Are you sure that works?
  Prettier uses https://github.com/mrmlnc/fast-glob, which in turn uses https://github.com/micromatch/micromatch, and the docs for these packages don't mention this syntax. As per the docs, square brackets should work as in regexes (`foo-[1-5].js`).

  Tested it. Apparently, it works as a negated character class in regexes (like `[^.min]`). I wonder where this syntax comes from. Micromatch doesn't support that:

  ```js
  micromatch(['static/table.js', 'static/n.js'], ['static/*[!.min].js']);
  // result: ["static/n.js"] -- brackets are treated like [!.min] in regexes, without negation
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/220#issuecomment-783662968
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/220
node_id: MDEyOklzc3VlQ29tbWVudDc4MzY2Mjk2OA==
created_at: 2021-02-22T20:44:51Z
updated_at: 2021-02-22T20:44:51Z
author_association: NONE
body:
  Actually, coming back to this, I have a clearer use case for enabling fts generation for views: making it easier to bring in text from lookup tables and other joins.  

  The datasette documentation describes populating an fts table like so:
  ```
  INSERT INTO "items_fts" (rowid, name, description, category_name)
      SELECT items. rowid,
      items.name,
      items.description,
      categories.name
      FROM items JOIN categories ON items.category_id=categories.id;
  ```
  Alternatively if you have fts support in sqlite_utils for views (which sqlite and fts5 support), you can do the same thing just by creating a view that captures the above joins as columns, then creating an fts table from that view. Such an fts table can be created using sqlite_utils, where one created with your method can't. 

  The resulting fts table can then be used by a whole family of related tables and views in the manner you described earlier in this issue. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1234#issuecomment-783674659
issue_url: https://api.github.com/repos/simonw/datasette/issues/1234
node_id: MDEyOklzc3VlQ29tbWVudDc4MzY3NDY1OQ==
created_at: 2021-02-22T21:06:28Z
updated_at: 2021-02-22T21:06:28Z
author_association: OWNER
body:
  I'm not going to work on this for a while, but if anyone has needs or ideas around that they can add them to this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-783688547
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDc4MzY4ODU0Nw==
created_at: 2021-02-22T21:31:28Z
updated_at: 2021-02-22T21:31:28Z
author_association: NONE
body:
  @Btibert3 I've opened a PR with my initial attempt at this.  Would you be willing to give this a try?

  https://github.com/dogsheep/google-takeout-to-sqlite/pull/5

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1239#issuecomment-783774084
issue_url: https://api.github.com/repos/simonw/datasette/issues/1239
node_id: MDEyOklzc3VlQ29tbWVudDc4Mzc3NDA4NA==
created_at: 2021-02-23T00:18:56Z
updated_at: 2021-02-23T00:19:18Z
author_association: OWNER
body:
  Bug is here: https://github.com/simonw/datasette/blob/42caabf7e9e6e4d69ef6dd7de16f2cd96bc79d5b/datasette/filters.py#L149-L165

  Those `json_each` lines should be:

      select {t}.rowid from {t}, json_each([{t}].[{c}]) j

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1218#issuecomment-784157345
issue_url: https://api.github.com/repos/simonw/datasette/issues/1218
node_id: MDEyOklzc3VlQ29tbWVudDc4NDE1NzM0NQ==
created_at: 2021-02-23T12:12:17Z
updated_at: 2021-02-23T12:12:17Z
author_association: NONE
body:
  Topline this fixed the same problem for me.
  ```
  brew install python@3.7
  ln -s /usr/local/opt/python@3.7/bin/python3.7 /usr/local/opt/python/bin/python3.7
  pip3 uninstall -y numpy
  pip3 uninstall -y setuptools
  pip3 install setuptools
  pip3 install numpy
  pip3 install datasette-publish-fly
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1240#issuecomment-784312460
issue_url: https://api.github.com/repos/simonw/datasette/issues/1240
node_id: MDEyOklzc3VlQ29tbWVudDc4NDMxMjQ2MA==
created_at: 2021-02-23T16:07:10Z
updated_at: 2021-02-23T16:08:28Z
author_association: NONE
body:
  Likewise, while answering to another issue regarding the Vega plugin, I realized that there is no such way of linking rows after a custom query, I only get this "Link" column with individual URLs for the default SQL view:

  ![ss-2021-02-23_170559](https://user-images.githubusercontent.com/7107523/108871491-1e3fd500-75f1-11eb-8f76-5d5a82cc14d7.png)

  Or is it there and I am just missing the option in my custom queries?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1241#issuecomment-784333768
issue_url: https://api.github.com/repos/simonw/datasette/issues/1241
node_id: MDEyOklzc3VlQ29tbWVudDc4NDMzMzc2OA==
created_at: 2021-02-23T16:35:51Z
updated_at: 2021-02-23T16:35:51Z
author_association: OWNER
body:
  This can definitely be done with a plugin.

  Adding to Datasette itself is an interesting idea. I think it's possible that many users these days no longer assume they can paste a URL from the browser address bar (if they ever understood that at all) because to many apps are SPAs with broken URLs.

  The shareable URLs are actually a key feature of Datasette - so maybe they should be highlighted in the default UI?

  I built a "copy to clipboard" feature for `datasette-copyable` and wrote up how that works here: https://til.simonwillison.net/javascript/copy-button

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1241#issuecomment-784334931
issue_url: https://api.github.com/repos/simonw/datasette/issues/1241
node_id: MDEyOklzc3VlQ29tbWVudDc4NDMzNDkzMQ==
created_at: 2021-02-23T16:37:26Z
updated_at: 2021-02-23T16:37:26Z
author_association: OWNER
body:
  A "Share link" button would only be needed on the table page and the arbitrary query page I think - and maybe on the row page, especially as that page starts to grow more features in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1241#issuecomment-784347646
issue_url: https://api.github.com/repos/simonw/datasette/issues/1241
node_id: MDEyOklzc3VlQ29tbWVudDc4NDM0NzY0Ng==
created_at: 2021-02-23T16:55:26Z
updated_at: 2021-02-23T16:57:39Z
author_association: NONE
body:
  > I think it's possible that many users these days no longer assume they can paste a URL from the browser address bar (if they ever understood that at all) because to many apps are SPAs with broken URLs.

  Absolutely, that's why I thought my corner case with `iframe` preventing access to the datasette URL could actually be relevant in more general situations.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1241#issuecomment-784567547
issue_url: https://api.github.com/repos/simonw/datasette/issues/1241
node_id: MDEyOklzc3VlQ29tbWVudDc4NDU2NzU0Nw==
created_at: 2021-02-23T22:45:56Z
updated_at: 2021-02-23T22:46:12Z
author_association: OWNER
body:
  I really like the way the Share feature on Stack Overflow works: https://stackoverflow.com/questions/18934149/how-can-i-use-postgresqls-text-column-type-in-django

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1243#issuecomment-785485597
issue_url: https://api.github.com/repos/simonw/datasette/issues/1243
node_id: MDEyOklzc3VlQ29tbWVudDc4NTQ4NTU5Nw==
created_at: 2021-02-25T00:28:30Z
updated_at: 2021-02-25T00:28:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=h1) Report
  > Merging [#1243](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=desc) (887bfd2) into [main](https://codecov.io/gh/simonw/datasette/commit/726f781c50e88f557437f6490b8479c3d6fabfc2?el=desc) (726f781) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1243/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1243   +/-   ##
  =======================================
    Coverage   91.56%   91.56%           
  =======================================
    Files          34       34           
    Lines        4242     4242           
  =======================================
    Hits         3884     3884           
    Misses        358      358           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=footer). Last update [726f781...32652d9](https://codecov.io/gh/simonw/datasette/pull/1243?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/238#issuecomment-785972074
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/238
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk3MjA3NA==
created_at: 2021-02-25T15:08:36Z
updated_at: 2021-02-25T15:08:36Z
author_association: OWNER
body:
  I bet the bug is in here: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L593-L602

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785978689
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk3ODY4OQ==
created_at: 2021-02-25T15:18:03Z
updated_at: 2021-02-25T15:18:03Z
author_association: OWNER
body:
  The Python `.extract()` method currently starts like this:
  ```python
  def extract(self, columns, table=None, fk_column=None, rename=None):
          rename = rename or {}
          if isinstance(columns, str):
              columns = [columns]
          if not set(columns).issubset(self.columns_dict.keys()):
              raise InvalidColumns(
                  "Invalid columns {} for table with columns {}".format(
                      columns, list(self.columns_dict.keys())
                  )
              )
          ...
  ```
  Note that it takes a list of columns (and treats a string as a single item list). That's because it can be called with a list of columns and it will use them to populate another table of unique tuples of those column values.

  So a new mechanism that can instead read JSON values from a single column needs to be compatible with that existing design.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979192
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk3OTE5Mg==
created_at: 2021-02-25T15:18:46Z
updated_at: 2021-02-25T15:18:46Z
author_association: OWNER
body:
  Likewise the `sqlite-utils extract` command takes one or more columns:
  ```
  Usage: sqlite-utils extract [OPTIONS] PATH TABLE COLUMNS...

    Extract one or more columns into a separate table

  Options:
    --table TEXT             Name of the other table to extract columns to
    --fk-column TEXT         Name of the foreign key column to add to the table
    --rename <TEXT TEXT>...  Rename this column in extracted table
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785979769
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk3OTc2OQ==
created_at: 2021-02-25T15:19:37Z
updated_at: 2021-02-25T15:19:37Z
author_association: OWNER
body:
  For the Python version I'd like to be able to provide a transformation callback function - which can be `json.loads` but could also be anything else which accepts the value of the current column and returns a Python dictionary of columns and their values to use in the new table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980083
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk4MDA4Mw==
created_at: 2021-02-25T15:20:02Z
updated_at: 2021-02-25T15:20:02Z
author_association: OWNER
body:
  It would be OK if the CLI version only allows you to specify a single column if you are using the `--json` option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785980813
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk4MDgxMw==
created_at: 2021-02-25T15:21:02Z
updated_at: 2021-02-25T15:23:47Z
author_association: OWNER
body:
  Maybe the Python version takes an optional dictionary mapping column names to transformation functions? It could then merge all of those results together - and maybe throw an error if the same key is produced by more than one column.

  ```python
      db["Reports"].extract(["Reported by"], transform={"Reported by": json.loads})
  ```
  Or it could have an option for different strategies if keys collide: first wins, last wins, throw exception, add a prefix to the new column name. That feels a bit too complex for an edge-case though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983070
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk4MzA3MA==
created_at: 2021-02-25T15:24:17Z
updated_at: 2021-02-25T15:24:17Z
author_association: OWNER
body:
  I'm going to go with last-wins - so if multiple transform functions return the same key the last one will over-write the others.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785983837
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk4MzgzNw==
created_at: 2021-02-25T15:25:21Z
updated_at: 2021-02-25T15:28:57Z
author_association: OWNER
body:
  Problem with calling this argument `transform=` is that the term "transform" already means something else in this library.

  I could use `convert=` instead.

  ... but that doesn't instantly make me think of turning a value into multiple columns.

  How about `expand=`? I've not used that term anywhere yet.

      db["Reports"].extract(["Reported by"], expand={"Reported by": json.loads})

  I think that works. You're expanding a single value into several columns of information.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-785992158
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NTk5MjE1OA==
created_at: 2021-02-25T15:37:04Z
updated_at: 2021-02-25T15:37:04Z
author_association: OWNER
body:
  Here's the current implementation of `.extract()`: https://github.com/simonw/sqlite-utils/blob/806c21044ac8d31da35f4c90600e98115aade7c6/sqlite_utils/db.py#L1049-L1074

  Tricky detail here: I create the lookup table first, based on the types of the columns that are being extracted.

  I need to do this because extraction currently uses unique tuples of values, so the table has to be created in advance.

  But if I'm using these new expand functions to figure out what's going to be extracted, I don't know the names of the columns and their types in advance. I'm only going to find those out during the transformation.

  This may turn out to be incompatible with how `.extract()` works at the moment. I may need a new method, `.extract_expand()` perhaps? It could be simpler - work only against a single column for example.

  I can still use the existing `sqlite-utils extract` CLI command though, with a `--json` flag and a rule that you can't run it against multiple columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786001768
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjAwMTc2OA==
created_at: 2021-02-25T15:50:28Z
updated_at: 2021-02-25T15:52:12Z
author_association: OWNER
body:
  One option: `.rows_where()` could grow a `ensure_pk=True` option which checks to see if the table is a `rowid` table and, if it is, includes that in the `select`.

  Or... how about you can call `.rows_where(..., pks=True)` and it will yield `(pk, rowdict)` tuple pairs instead of just returning the sequence of dictionaries?

  I'm always a little bit nervous of methods that vary their return type based on their arguments. Maybe this would be a separate method instead?
  ```python
      for pk, row in table.pk_rows_where(...):
          # ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786005078
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjAwNTA3OA==
created_at: 2021-02-25T15:54:59Z
updated_at: 2021-02-25T15:56:16Z
author_association: OWNER
body:
  Is `pk_rows_where()` a good name? It sounds like it returns "primary key rows" which isn't a thing. It actually returns rows along with their primary key.

  Other options:

  - `table.rows_with_pk_where(...)` - should this return `(row, pk)` rather than `(pk, row)`?
  - `table.rows_where_pk(...)`
  - `table.pk_and_rows_where(...)`
  - `table.pk_with_rows_where(...)`
  - `table.pks_with_rows_where(...)` - because rows is pluralized, so pks should be pluralized too?
  - `table.pks_rows_where(...)`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786006794
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjAwNjc5NA==
created_at: 2021-02-25T15:57:17Z
updated_at: 2021-02-25T15:57:28Z
author_association: OWNER
body:
  I quite like `pks_with_rows_where(...)` - but grammatically it suggests it will return the primary keys that exist where their rows match the criteria - "pks with rows" can be interpreted as "pks for the rows that..." as opposed to "pks accompanied by rows"

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786016380
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjAxNjM4MA==
created_at: 2021-02-25T16:10:01Z
updated_at: 2021-02-25T16:10:01Z
author_association: OWNER
body:
  I prototyped this and I like it:
  ```
  In [1]: import sqlite_utils
  In [2]: db = sqlite_utils.Database("/Users/simon/Dropbox/Development/datasette/fixtures.db")
  In [3]: list(db["compound_primary_key"].pks_and_rows_where())
  Out[3]: [(('a', 'b'), {'pk1': 'a', 'pk2': 'b', 'content': 'c'})]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786036355
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjAzNjM1NQ==
created_at: 2021-02-25T16:38:07Z
updated_at: 2021-02-25T16:38:07Z
author_association: OWNER
body:
  Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#listing-rows-with-their-primary-keys

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/240#issuecomment-786037219
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjAzNzIxOQ==
created_at: 2021-02-25T16:39:23Z
updated_at: 2021-02-25T16:39:23Z
author_association: OWNER
body:
  Example from the docs:
  ```pycon
  >>> db = sqlite_utils.Database(memory=True)
  >>> db["dogs"].insert({"name": "Cleo"})
  >>> for pk, row in db["dogs"].pks_and_rows_where():
  ...     print(pk, row)
  1 {'rowid': 1, 'name': 'Cleo'}

  >>> db["dogs_with_pk"].insert({"id": 5, "name": "Cleo"}, pk="id")
  >>> for pk, row in db["dogs_with_pk"].pks_and_rows_where():
  ...     print(pk, row)
  5 {'id': 5, 'name': 'Cleo'}

  >>> db["dogs_with_compound_pk"].insert(
  ...     {"species": "dog", "id": 3, "name": "Cleo"},
  ...     pk=("species", "id")
  ... )
  >>> for pk, row in db["dogs_with_compound_pk"].pks_and_rows_where():
  ...     print(pk, row)
  ('dog', 3) {'species': 'dog', 'id': 3, 'name': 'Cleo'}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786049394
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/237
node_id: MDEyOklzc3VlQ29tbWVudDc4NjA0OTM5NA==
created_at: 2021-02-25T16:56:14Z
updated_at: 2021-02-25T16:56:14Z
author_association: OWNER
body:
  Other methods (`db.create_view()` for example) have `ignore=True` to mean "don't throw an error if this causes a problem", so I'm good with adding that to `.drop_view()`.

  I don't like using it as the default partly because that would be a very minor breaking API change, but mainly because I don't want to hide mistakes people make - e.g. if you mistype the name of the table you are trying to drop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/237#issuecomment-786050562
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/237
node_id: MDEyOklzc3VlQ29tbWVudDc4NjA1MDU2Mg==
created_at: 2021-02-25T16:57:56Z
updated_at: 2021-02-25T16:57:56Z
author_association: OWNER
body:
  `sqlite-utils create-view` currently has a `--ignore` option, so adding that to `sqlite-utils drop-view` and `sqlite-utils drop-table` makes sense as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1244#issuecomment-786786645
issue_url: https://api.github.com/repos/simonw/datasette/issues/1244
node_id: MDEyOklzc3VlQ29tbWVudDc4Njc4NjY0NQ==
created_at: 2021-02-26T17:30:38Z
updated_at: 2021-02-26T17:30:38Z
author_association: OWNER
body:
  New paragraph at the top of https://docs.datasette.io/en/latest/writing_plugins.html

  > Want to start by looking at an example? The [Datasette plugins directory](https://datasette.io/plugins) lists more than 50 open source plugins with code you can explore. The [plugin hooks](https://docs.datasette.io/en/latest/plugin_hooks.html#plugin-hooks) page includes links to example plugins for each of the documented hooks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1240#issuecomment-786812716
issue_url: https://api.github.com/repos/simonw/datasette/issues/1240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjgxMjcxNg==
created_at: 2021-02-26T18:18:18Z
updated_at: 2021-02-26T18:18:18Z
author_association: OWNER
body:
  Agreed, this would be extremely useful. I'd love to be able to facet against custom queries. It's a fair bit of work to implement but it's not impossible. Closing this as a duplicate of #972.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1240#issuecomment-786813506
issue_url: https://api.github.com/repos/simonw/datasette/issues/1240
node_id: MDEyOklzc3VlQ29tbWVudDc4NjgxMzUwNg==
created_at: 2021-02-26T18:19:46Z
updated_at: 2021-02-26T18:19:46Z
author_association: OWNER
body:
  Linking to rows from custom queries is a lot harder - because given an arbitrary string of SQL it's difficult to analyze it and figure out which (if any) of the returned columns represent a primary key.

  It's possible to manually write a SQL query that returns a column that will be treated as a link to another page using this plugin, but it's not particularly straight-forward: https://datasette.io/plugins/datasette-json-html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/239#issuecomment-786830832
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/239
node_id: MDEyOklzc3VlQ29tbWVudDc4NjgzMDgzMg==
created_at: 2021-02-26T18:52:40Z
updated_at: 2021-02-26T18:52:40Z
author_association: OWNER
body:
  Could this handle lists of objects too? That would be pretty amazing - if the column has a `[{...}, {...}]` list in it could turn that into a many-to-many.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1246#issuecomment-786840425
issue_url: https://api.github.com/repos/simonw/datasette/issues/1246
node_id: MDEyOklzc3VlQ29tbWVudDc4Njg0MDQyNQ==
created_at: 2021-02-26T19:11:56Z
updated_at: 2021-02-26T19:11:56Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/facets.py#L271-L295

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1246#issuecomment-786840734
issue_url: https://api.github.com/repos/simonw/datasette/issues/1246
node_id: MDEyOklzc3VlQ29tbWVudDc4Njg0MDczNA==
created_at: 2021-02-26T19:12:39Z
updated_at: 2021-02-26T19:12:47Z
author_association: OWNER
body:
  Could I take this part:
  ```python
               suggested_facet_sql = """ 
                   select distinct json_type({column}) 
                   from ({sql}) 
               """.format( 
                   column=escape_sqlite(column), sql=self.sql 
               ) 
  ```
  And add `where {column} is not null and {column} != ''` perhaps?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1238#issuecomment-786848654
issue_url: https://api.github.com/repos/simonw/datasette/issues/1238
node_id: MDEyOklzc3VlQ29tbWVudDc4Njg0ODY1NA==
created_at: 2021-02-26T19:28:48Z
updated_at: 2021-02-26T19:28:48Z
author_association: OWNER
body:
  I added a debug line just before `for regex, wildcard_template` here:

  https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1148-L1155

  And it showed that for some reason `request.path` is `/prefix/prefix/request` here - the prefix got doubled somehow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1238#issuecomment-786849095
issue_url: https://api.github.com/repos/simonw/datasette/issues/1238
node_id: MDEyOklzc3VlQ29tbWVudDc4Njg0OTA5NQ==
created_at: 2021-02-26T19:29:38Z
updated_at: 2021-02-26T19:29:38Z
author_association: OWNER
body:
  Here's the test I wrote:
  ```diff
  git diff tests/test_custom_pages.py
  diff --git a/tests/test_custom_pages.py b/tests/test_custom_pages.py
  index 6a23192..5a71f56 100644
  --- a/tests/test_custom_pages.py
  +++ b/tests/test_custom_pages.py
  @@ -2,11 +2,19 @@ import pathlib
   import pytest
   from .fixtures import make_app_client
   
  +TEST_TEMPLATE_DIRS = str(pathlib.Path(__file__).parent / "test_templates")
  +
   
   @pytest.fixture(scope="session")
   def custom_pages_client():
  +    with make_app_client(template_dir=TEST_TEMPLATE_DIRS) as client:
  +        yield client
  +
  +
  +@pytest.fixture(scope="session")
  +def custom_pages_client_with_base_url():
       with make_app_client(
  -        template_dir=str(pathlib.Path(__file__).parent / "test_templates")
  +        template_dir=TEST_TEMPLATE_DIRS, config={"base_url": "/prefix/"}
       ) as client:
           yield client
   
  @@ -23,6 +31,12 @@ def test_request_is_available(custom_pages_client):
       assert "path:/request" == response.text
   
   
  +def test_custom_pages_with_base_url(custom_pages_client_with_base_url):
  +    response = custom_pages_client_with_base_url.get("/prefix/request")
  +    assert 200 == response.status
  +    assert "path:/prefix/request" == response.text
  +
  +
   def test_custom_pages_nested(custom_pages_client):
       response = custom_pages_client.get("/nested/nest")
       assert 200 == response.status
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-786925280
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc4NjkyNTI4MA==
created_at: 2021-02-26T22:23:10Z
updated_at: 2021-02-26T22:23:10Z
author_association: MEMBER
body:
  Thanks!

  I requested my Gmail export from takeout - once that arrives I'll test it against this and then merge the PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787118691
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzExODY5MQ==
created_at: 2021-02-27T18:53:23Z
updated_at: 2021-02-27T18:53:23Z
author_association: OWNER
body:
  Datasette has its own implementation of a write queue for exactly this purpose - and there's no reason at all that should stay in Datasette rather than being extracted out and moved over here to `sqlite-utils`.

  One small concern I have is around the API design. I'd want to keep supporting the existing synchronous API while also providing a similar API with await-based methods.

  What are some good examples of libraries that do this? I like how https://www.python-httpx.org/ handles it, maybe that's a good example to imitate?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787120136
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzEyMDEzNg==
created_at: 2021-02-27T19:04:47Z
updated_at: 2021-02-27T19:04:47Z
author_association: OWNER
body:
  Another option here would be to add https://github.com/omnilib/aiosqlite/blob/main/aiosqlite/core.py as a dependency - it's four years old now and actively marinated, and the code is pretty small so it looks like a solid, stable, reliable dependency.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787121933
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzEyMTkzMw==
created_at: 2021-02-27T19:18:57Z
updated_at: 2021-02-27T19:18:57Z
author_association: CONTRIBUTOR
body:
  I think HTTPX gets it exactly right, with a clear separation between sync and async clients, each with a basically identical API. (I'm about to switch [feed-to-sqlite](https://github.com/eyeseast/feed-to-sqlite) over to it, from Requests, to eventually make way for async support.)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787142066
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE0MjA2Ng==
created_at: 2021-02-27T21:17:10Z
updated_at: 2021-02-27T21:17:10Z
author_association: OWNER
body:
  I have a hunch this is actually going to be quite difficult, due to the internal complexity of some of the `sqlite-utils` API methods.

  Consider `db[table].extract(...)` for example. It does a whole bunch of extra queries inside the method - each of those would need to be turned into an `await` call for the async version. Here's the method body today:

  https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1060-L1152

  Writing this method twice - looking similar but with `await ...` tucked in before every internal method it calls that needs to execute SQL - is going to be pretty messy.

  One thing that would help a LOT is figuring out how to share the majority of the test code. If the exact same tests could run against both the sync and async versions with a bit of test trickery, maintaining parallel implementations would at least be a bit more feasible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787144523
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE0NDUyMw==
created_at: 2021-02-27T21:18:46Z
updated_at: 2021-02-27T21:18:46Z
author_association: OWNER
body:
  Here's a really wild idea: I wonder if it would be possible to run a source transformation against either the sync or the async versions of the code to produce the equivalent for the other paradigm?

  Could that even be as simple as a set of regular expressions against the `await ...` version that strips out or replaces the `await` and `async def` and `async for` statements?

  If so... I could maintain just the async version, generate the sync version with a script and rely on robust unit testing to guarantee that this actually works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787150276
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE1MDI3Ng==
created_at: 2021-02-27T21:27:26Z
updated_at: 2021-02-27T21:27:26Z
author_association: NONE
body:
  I had this resource by Seth Michael Larson saved https://github.com/sethmlarson/pycon-async-sync-poster I haven't had a look at it, but it may contain useful info.

  On twitter, I mentioned passing an aiosqlite connection during the `Database` creation. I'm not 100% familiar with the `sqlite-utils` codebase, so I may be wrong here, but maybe decorating internal functions could be an option? Then they are awaited or not inside the decorator depending on how they are called.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787175126
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE3NTEyNg==
created_at: 2021-02-27T21:55:05Z
updated_at: 2021-02-27T21:55:05Z
author_association: OWNER
body:
  "how to use some new tools to more easily maintain a codebase that supports both async and synchronous I/O and multiple async libraries" - yeah that's exactly what I need, thank you!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787186826
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE4NjgyNg==
created_at: 2021-02-27T22:01:54Z
updated_at: 2021-02-27T22:01:54Z
author_association: OWNER
body:
  `unasync` is an implementation of the exact pattern I was talking about above - it uses the `tokenize` module from the Python standard library to apply some clever rules to transform an async codebase into a sync one. https://unasync.readthedocs.io/en/latest/ - implementation here: https://github.com/python-trio/unasync/blob/v0.5.0/src/unasync/__init__.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787190562
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE5MDU2Mg==
created_at: 2021-02-27T22:04:00Z
updated_at: 2021-02-27T22:04:00Z
author_association: OWNER
body:
  From the poster here: https://github.com/sethmlarson/pycon-async-sync-poster/blob/master/poster.pdf

  <img width="624" alt="pycon-async-sync-poster_poster_pdf_at_master_·_sethmlarson_pycon-async-sync-poster" src="https://user-images.githubusercontent.com/9599/109401634-9f0a1400-7904-11eb-8b3a-37df0678b8dc.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787195536
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE5NTUzNg==
created_at: 2021-02-27T22:13:24Z
updated_at: 2021-02-27T22:13:24Z
author_association: OWNER
body:
  Some other interesting background reading: https://docs.sqlalchemy.org/en/14/orm/extensions/asyncio.html - in particular see how SQLALchemy has a `await conn.run_sync(meta.drop_all)` mechanism for running methods that haven't themselves been provided in an async version

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787198202
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzE5ODIwMg==
created_at: 2021-02-27T22:33:58Z
updated_at: 2021-02-27T22:33:58Z
author_association: OWNER
body:
  Hah or use this trick, which genuinely rewrites the code at runtime using a class decorator! https://github.com/python-happybase/aiohappybase/blob/0990ef45cfdb720dc987afdb4957a0fac591cb99/aiohappybase/sync/_util.py#L19-L32

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/242#issuecomment-787532279
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/242
node_id: MDEyOklzc3VlQ29tbWVudDc4NzUzMjI3OQ==
created_at: 2021-02-28T22:09:37Z
updated_at: 2021-02-28T22:09:37Z
author_association: OWNER
body:
  Microsoft's playwright Python library solves this problem by code generating both their sync AND their async libraries https://github.com/microsoft/playwright-python/tree/master/scripts

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1246#issuecomment-787611153
issue_url: https://api.github.com/repos/simonw/datasette/issues/1246
node_id: MDEyOklzc3VlQ29tbWVudDc4NzYxMTE1Mw==
created_at: 2021-03-01T03:30:57Z
updated_at: 2021-03-01T03:30:57Z
author_association: OWNER
body:
  I'm going to try a new pattern for testing this, enabled by #1151 - the test will create a new named in-memory database, write some records to it and then run some test facets against that. This will save me from having to add yet another fixtures table for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1247#issuecomment-787616158
issue_url: https://api.github.com/repos/simonw/datasette/issues/1247
node_id: MDEyOklzc3VlQ29tbWVudDc4NzYxNjE1OA==
created_at: 2021-03-01T03:49:27Z
updated_at: 2021-03-01T03:49:27Z
author_association: OWNER
body:
  A couple of options:
  ```python
  datasette.add_memory_database("test_json_array")
  # or make that first argument to add_database() optional and support:
  datasette.add_database(memory_name="test_json_array")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1247#issuecomment-787616446
issue_url: https://api.github.com/repos/simonw/datasette/issues/1247
node_id: MDEyOklzc3VlQ29tbWVudDc4NzYxNjQ0Ng==
created_at: 2021-03-01T03:50:37Z
updated_at: 2021-03-01T03:50:37Z
author_association: OWNER
body:
  I like the `.add_memory_database()` option. I also like that it makes it more obvious that this is a capability of Datasette, since I'm excited to see more plugins, features and tests that take advantage of it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1238#issuecomment-789186458
issue_url: https://api.github.com/repos/simonw/datasette/issues/1238
node_id: MDEyOklzc3VlQ29tbWVudDc4OTE4NjQ1OA==
created_at: 2021-03-02T20:19:30Z
updated_at: 2021-03-02T20:19:30Z
author_association: CONTRIBUTOR
body:
  A custom `templates/index.html` seems to work and custom `pages` as a workaround with moving them to `pages/base_url_dir`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-789409126
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDc4OTQwOTEyNg==
created_at: 2021-03-03T03:57:15Z
updated_at: 2021-03-03T03:58:40Z
author_association: NONE
body:
  In FTS5, I think doing an FTS search is actually much easier than doing a join against the main table like datasette does now. In fact, FTS5 external content tables provide a transparent interface back to the original table or view.

  Here's what I'm currently doing:
  * build a view that joins whatever tables I want and rename the columns to non-joiny names (e.g, `chapter.name AS chapter_name` in the view where needed)
  * Create an FTS5 table with `content="viewname"`
  * As described in the "external content tables" section (https://www.sqlite.org/fts5.html#external_content_tables), sql queries can be made directly to the FTS table, which behind the covers makes select calls to the content table when the content of the original columns are needed.
  * In addition, you get "rank" and "bm25()" available to you when you select on the _fts table.

  Unfortunately, datasette doesn't currently seem happy being coerced into doing a real query on an fts5 table.  This works:
  ```select col1, col2, col3 from table_fts where coll1="value" and table_fts match escape_fts("search term") order by rank```

  But this doesn't work in the datasette SQL query interface:
  ```select col1, col2, col3 from table_fts where coll1="value" and table_fts match escape_fts(:search) order by rank```   (the "search" input text field doesn't show up)

  For what datasette is doing right now, I think you could just use contentless fts5 tables (`content=""`), since all you care about is the rowid since all you're doing a subselect to get the rowid anyway. In fts5, that's just a contentless table.

  I guess if you want to follow this suggestion, you'd need a somewhat different code path for fts5.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/283#issuecomment-789680230
issue_url: https://api.github.com/repos/simonw/datasette/issues/283
node_id: MDEyOklzc3VlQ29tbWVudDc4OTY4MDIzMA==
created_at: 2021-03-03T12:28:42Z
updated_at: 2021-03-03T12:28:42Z
author_association: NONE
body:
  One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`. 

  I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790198930
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDc5MDE5ODkzMA==
created_at: 2021-03-04T00:58:40Z
updated_at: 2021-03-04T00:58:40Z
author_association: NONE
body:
  I am just seeing this sorry, yes!  I will kick the tires later on tonight.  My apologies for the delay.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/268#issuecomment-790257263
issue_url: https://api.github.com/repos/simonw/datasette/issues/268
node_id: MDEyOklzc3VlQ29tbWVudDc5MDI1NzI2Mw==
created_at: 2021-03-04T03:20:23Z
updated_at: 2021-03-04T03:20:23Z
author_association: NONE
body:
  It's kind of an ugly hack, but you can try out what using the fts5 table as an actual datasette-accessible table looks like without changing any datasette code by creating yet another view on top of the fts5 table:

  `create view proxyview as select *, rank, table_fts as fts from table_fts;`

  That's now visible from datasette, just like any other view, but you can use `fts match escape_fts(search_string) order by rank`.

  This is only good as a proof of concept because you're inefficiently going from view -> fts5 external content table -> view -> data table. However, it does show it works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790369076
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM2OTA3Ng==
created_at: 2021-03-04T06:54:46Z
updated_at: 2021-03-04T06:54:46Z
author_association: MEMBER
body:
  The Rich-powered progress bar is pretty:

  ![rich](https://user-images.githubusercontent.com/9599/109923307-71f69200-7c73-11eb-9ee2-8f0a240f3994.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790370485
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM3MDQ4NQ==
created_at: 2021-03-04T06:57:25Z
updated_at: 2021-03-04T06:57:48Z
author_association: MEMBER
body:
  The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:

  https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67

  I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790372621
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM3MjYyMQ==
created_at: 2021-03-04T07:01:18Z
updated_at: 2021-03-04T07:01:18Z
author_association: MEMBER
body:
  I'm not sure if it would work, but there is an alternative pattern for showing a progress bar against a really large file that I've used in `healthkit-to-sqlite` - you set the progress bar size to the size of the file in bytes, then update a counter as you read the file.

  https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/cli.py#L24-L57 and https://github.com/dogsheep/healthkit-to-sqlite/blob/3eb2b06bfe3b4faaf10e9cf9dfcb28e3d16c14ff/healthkit_to_sqlite/utils.py#L4-L19 (the `progress_callback()` bit) is where that happens.

  It can be a bit of a convoluted pattern, and I'm not at all sure it would work for `mbox` files since it looks like that library has other reasons it needs to do a file scan rather than streaming it through one chunk of bytes at a time. So I imagine this would not work here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790373024
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM3MzAyNA==
created_at: 2021-03-04T07:01:58Z
updated_at: 2021-03-04T07:04:06Z
author_association: MEMBER
body:
  I got 9 warnings that look like this:
  ```
  Errors: 1
  Traceback (most recent call last):
    File "/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py", line 103, in get_mbox
      message["date"] = get_message_date(email.get("Date"), email.get_from())
    File "/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py", line 167, in get_message_date
      datetime_tuple = email.utils.parsedate_tz(mail_date)
    File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py", line 50, in parsedate_tz
      res = _parsedate_tz(data)
    File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py", line 69, in _parsedate_tz
      data = data.split()
  AttributeError: 'Header' object has no attribute 'split'
  ```
  It would be useful if those warnings told me the message ID (or similar) of the affected message so I could grep for it in the `mbox` and see what was going on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790378658
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM3ODY1OA==
created_at: 2021-03-04T07:12:48Z
updated_at: 2021-03-04T07:12:48Z
author_association: MEMBER
body:
  It looks like the `body` is being loaded into a BLOB column - so in Datasette default it looks like this:

  <img width="1650" alt="mbox__mbox_emails__753_446_rows" src="https://user-images.githubusercontent.com/9599/109924808-b4b96980-7c75-11eb-8c9e-307f2ae32d5a.png">

  If I `datasette install datasette-render-binary` and then try again I get this:

  <img width="1487" alt="mbox__mbox_emails__753_446_rows" src="https://user-images.githubusercontent.com/9599/109924944-ea5e5280-7c75-11eb-9a32-404f3d68455f.png">

  It would be great if we could store the `body` as unicode text instead. May have to do something clever to decode it based on some kind of charset header?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790379629
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM3OTYyOQ==
created_at: 2021-03-04T07:14:41Z
updated_at: 2021-03-04T07:14:41Z
author_association: MEMBER
body:
  Confirmed: removing the `len()` call does not speed things up, so it's reading through the entire file for some other purpose too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790380839
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM4MDgzOQ==
created_at: 2021-03-04T07:17:05Z
updated_at: 2021-03-04T07:17:05Z
author_association: MEMBER
body:
  Looks like you're doing this:
  ```python
      elif message.get_content_type() == "text/plain":
          body = message.get_payload(decode=True)
  ```
  So presumably that decodes to a unicode string?

  I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790389335
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM4OTMzNQ==
created_at: 2021-03-04T07:32:04Z
updated_at: 2021-03-04T07:32:04Z
author_association: NONE
body:
  > The command takes quite a while to start running, presumably because this line causes it to have to scan the WHOLE file in order to generate a count:
  > 
  > https://github.com/dogsheep/google-takeout-to-sqlite/blob/a3de045eba0fae4b309da21aa3119102b0efc576/google_takeout_to_sqlite/utils.py#L66-L67
  > 
  > I'm fine with waiting though. It's not like this is a command people run every day - and without that count we can't show a progress bar, which seems pretty important for a process that takes this long.

  The wait is from python loading the mbox file. This happens regardless if you're getting the length of the mbox.  The mbox module is on the slow side. It is possible to do one's own parsing of the mbox, but I kind of wanted to avoid doing that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790391711
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDM5MTcxMQ==
created_at: 2021-03-04T07:36:24Z
updated_at: 2021-03-04T07:36:24Z
author_association: NONE
body:
  > Looks like you're doing this:
  > 
  > ```python
  >     elif message.get_content_type() == "text/plain":
  >         body = message.get_payload(decode=True)
  > ```
  > 
  > So presumably that decodes to a unicode string?
  > 
  > I imagine the reason the column is a `BLOB` for me is that `sqlite-utils` determines the column type based on the first batch of items - https://github.com/simonw/sqlite-utils/blob/09c3386f55f766b135b6a1c00295646c4ae29bec/sqlite_utils/db.py#L1927-L1928 - and I got unlucky and had something in my first batch that wasn't a unicode string.

  Ah, that's good to know. I think explicitly creating the tables will be a great improvement. I'll add that.

  Also, I noticed after I opened this PR that the `message.get_payload()` is being deprecated in favor of `message.get_content()` or something like that.  I'll see if that handles the decoding better, too.

  Thanks for the feedback. I should have time tomorrow to put together some improvements.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790668263
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDY2ODI2Mw==
created_at: 2021-03-04T14:43:58Z
updated_at: 2021-03-04T14:43:58Z
author_association: MEMBER
body:
  I added this code to output a message ID on errors:
  ```diff
               print("Errors: {}".format(num_errors))
               print(traceback.format_exc())
  +            print("Message-Id: {}".format(email.get("Message-Id", "None")))
               continue
  ```
  Having found a message ID that had an error, I ran this command to see the context:

      rg --text --context 20 '44F289B0.000001.02100@SCHWARZE-DWFXMI' ~/gmail.mbox

  This was for the following error:
  ```
    File "/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py", line 102, in get_mbox
      message["date"] = get_message_date(email.get("Date"), email.get_from())
    File "/Users/simon/Dropbox/Development/google-takeout-to-sqlite/google_takeout_to_sqlite/utils.py", line 178, in get_message_date
      datetime_tuple = email.utils.parsedate_tz(mail_date)
    File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py", line 50, in parsedate_tz
      res = _parsedate_tz(data)
    File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/email/_parseaddr.py", line 69, in _parsedate_tz
      data = data.split()
  AttributeError: 'Header' object has no attribute 'split'
  ```
  Here's what I spotted in the `ripgrep` output:
  ```
  177133570:Message-Id: <44F289B0.000001.02100@SCHWARZE-DWFXMI>
  177133571-Date: Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)
  177133572-X-Mailer: IncrediMail (5002253)
  ```
  So it could it be that `_parsedate_tz` is having trouble with that `Mon, 28 Aug 2006 08:14:08 +0200 (Westeurop�ische Sommerzeit)` string.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790669767
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDY2OTc2Nw==
created_at: 2021-03-04T14:46:06Z
updated_at: 2021-03-04T14:46:06Z
author_association: MEMBER
body:
  Solution could be to pre-process that string by splitting on `(` and dropping everything afterwards, assuming that the `(...)` bit isn't necessary for correctly parsing the date.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790693674
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDY5MzY3NA==
created_at: 2021-03-04T15:18:36Z
updated_at: 2021-03-04T15:18:36Z
author_association: MEMBER
body:
  I imported my 10GB mbox with 750,000 emails in it, ran this tool (with a hacked fix for the blob column problem) - and now a search that returns 92 results takes 25.37ms! This is fantastic.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-790695126
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MDY5NTEyNg==
created_at: 2021-03-04T15:20:42Z
updated_at: 2021-03-04T15:20:42Z
author_association: MEMBER
body:
  I'm not sure why but my most recent import, when displayed in Datasette, looks like this:

  <img width="574" alt="mbox__mbox_emails__753_446_rows" src="https://user-images.githubusercontent.com/9599/109985836-0ab00080-7cba-11eb-97d5-0631a0835b61.png">

  Sorting by `id` in the opposite order gives me the data I would expect - so it looks like a bunch of null/blank messages are being imported at some point and showing up first due to ID ordering.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/issues/4#issuecomment-790934616
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDc5MDkzNDYxNg==
created_at: 2021-03-04T20:54:44Z
updated_at: 2021-03-04T20:54:44Z
author_association: NONE
body:
  Sorry for the delay, I got sidetracked after class last night.  I am getting the following error:

  ```
  /content# google-takeout-to-sqlite mbox takeout.db Takeout/Mail/gmail.mbox 
  Usage: google-takeout-to-sqlite [OPTIONS] COMMAND [ARGS]...Try 'google-takeout-to-sqlite --help' for help.

  Error: No such command 'mbox'.
  ```

  On the box, I installed with pip after cloning: https://github.com/UtahDave/google-takeout-to-sqlite.git

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791089881
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MTA4OTg4MQ==
created_at: 2021-03-05T02:03:19Z
updated_at: 2021-03-05T02:03:19Z
author_association: NONE
body:
  I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.

  Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/766#issuecomment-791509910
issue_url: https://api.github.com/repos/simonw/datasette/issues/766
node_id: MDEyOklzc3VlQ29tbWVudDc5MTUwOTkxMA==
created_at: 2021-03-05T15:57:35Z
updated_at: 2021-03-05T16:35:21Z
author_association: CONTRIBUTOR
body:
  Hello, 
  I have the same wildcards search problems with an instance of Datasette. http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwerz&_sort=rowid is OK but http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwe* is not (FTS is activated on "Reference" "IntituleAnalyse" "NomDuProducteur" "PresentationDuContenu" "Notes"). 

  Notice that a SQL query as below launched directly from SQLite in the server's shell, retrieves results.

  `select * from fonds_auguste_dupouy_1872_1967_fts where IntituleAnalyse MATCH "gwe*";`

  Thanks,

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-791530093
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDc5MTUzMDA5Mw==
created_at: 2021-03-05T16:28:07Z
updated_at: 2021-03-05T16:28:07Z
author_association: NONE
body:
  > I just tried to run this on a small VPS instance with 2GB of memory and it crashed out of memory while processing a 12GB mbox from Takeout.
  > 
  > Is it possible to stream the emails to sqlite instead of loading it all into memory and upserting at once?

  @maxhawkins a limitation of the python mbox module is it loads the entire mbox into memory.  I did find another approach to this problem that didn't use the builtin python mbox module and created a generator so that it didn't have to load the whole mbox into memory.  I was hoping to use standard library modules, but this might be a good reason to investigate that approach a bit more. My worry is making sure a custom processor handles all the ins and outs of the mbox format correctly.

  Hm. As I'm writing this, I thought of something.  I think I can parse each message one at a time, and then use an mbox function to load each message using the python mbox module. That way the mbox module can still deal with the specifics of the mbox format, but I can use a generator.

  I'll give that a try.  Thanks for the feedback @maxhawkins and @simonw. I'll give that a try.

  @simonw can we hold off on merging this until I can test this new approach?

reactions:
  {"total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/858#issuecomment-792230560
issue_url: https://api.github.com/repos/simonw/datasette/issues/858
node_id: MDEyOklzc3VlQ29tbWVudDc5MjIzMDU2MA==
created_at: 2021-03-07T07:14:58Z
updated_at: 2021-03-07T07:14:58Z
author_association: NONE
body:
  To get it to work I had to:

  - add `shell=true` to the various commands in datasette 

  - use the name argument of the publish command. (https://docs.datasette.io/en/stable/publish.html)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/858#issuecomment-792308036
issue_url: https://api.github.com/repos/simonw/datasette/issues/858
node_id: MDEyOklzc3VlQ29tbWVudDc5MjMwODAzNg==
created_at: 2021-03-07T16:41:54Z
updated_at: 2021-03-07T16:41:54Z
author_association: NONE
body:
  Apologies if I sound dense but I don't see where you would pass
  'shell=True'. I'm using the CLI installed via pip.

  On Sun., Mar. 7, 2021, 2:15 a.m. David Smith, <notifications@github.com>
  wrote:

  > To get it to work I had to:
  >
  >    -
  >
  >    add shell=true to the various commands in datasette
  >    -
  >
  >    use the name argument of the publish command. (
  >    https://docs.datasette.io/en/stable/publish.html)
  >
  > —
  > You are receiving this because you commented.
  > Reply to this email directly, view it on GitHub
  > <https://github.com/simonw/datasette/issues/858#issuecomment-792230560>,
  > or unsubscribe
  > <https://github.com/notifications/unsubscribe-auth/AAJJTOMZMGYSCGUU4J3AVSDTCMRX5ANCNFSM4ODNEDYA>
  > .
  >

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-792383956
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDc5MjM4Mzk1Ng==
created_at: 2021-03-08T00:20:09Z
updated_at: 2021-03-08T00:20:09Z
author_association: OWNER
body:
  Worth noting that the Docker image used by `datasette publish cloudrun` doesn't actually use that Datasette docker image - it does this:

  https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L349-L353

  Where the apt extras for SpatiaLite are: https://github.com/simonw/datasette/blob/d0fd833b8cdd97e1b91d0f97a69b494895d82bee/datasette/utils/__init__.py#L344-L345

  `libsqlite3-mod-spatialite` against that official `python:3.8` image doesn't appear to install SpatiaLite 5.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-792384382
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDc5MjM4NDM4Mg==
created_at: 2021-03-08T00:22:02Z
updated_at: 2021-03-08T00:22:02Z
author_association: OWNER
body:
  I tried this patch against `Dockerfile`:
  ```diff
  diff --git a/Dockerfile b/Dockerfile
  index f4b1414..dd659e1 100644
  --- a/Dockerfile
  +++ b/Dockerfile
  @@ -1,25 +1,26 @@
  -FROM python:3.7.10-slim-stretch as build
  +FROM python:3.9.2-slim-buster as build
   
   # Setup build dependencies
   RUN apt update \
  -&& apt install -y python3-dev build-essential wget libxml2-dev libproj-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
  - && apt clean
  +  && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \
  +  libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \
  +  && apt clean
   
  -
  -RUN wget "https://www.sqlite.org/2020/sqlite-autoconf-3310100.tar.gz" && tar xzf sqlite-autoconf-3310100.tar.gz \
  -    && cd sqlite-autoconf-3310100 && ./configure --disable-static --enable-fts5 --enable-json1 CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \
  +RUN wget "https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz" && tar xzf sqlite-autoconf-3340100.tar.gz \
  +    && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \
  +    CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \
       && make && make install
   
  -RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-sources/freexl-1.0.5.tar.gz" && tar zxf freexl-1.0.5.tar.gz \
  -    && cd freexl-1.0.5 && ./configure && make && make install
  +RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz" && tar zxf freexl-1.0.6.tar.gz \
  +    && cd freexl-1.0.6 && ./configure && make && make install
   
  -RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-sources/libspatialite-4.4.0-RC0.tar.gz" && tar zxf libspatialite-4.4.0-RC0.tar.gz \
  -    && cd libspatialite-4.4.0-RC0 && ./configure && make && make install
  +RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz" && tar zxf libspatialite-5.0.1.tar.gz \
  +    && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install
   
   RUN wget "http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install
   
  -RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-sources/spatialite-tools-4.4.0-RC0.tar.gz" && tar zxf spatialite-tools-4.4.0-RC0.tar.gz \
  -    && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install
  +RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz" && tar zxf spatialite-tools-5.0.0.tar.gz \
  +    && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install
   
   
   # Add local code to the image instead of fetching from pypi.
  @@ -27,7 +28,7 @@ COPY . /datasette
   
   RUN pip install /datasette
   
  -FROM python:3.7.10-slim-stretch
  +FROM python:3.9.2-slim-buster
   
   # Copy python dependencies and spatialite libraries
   COPY --from=build /usr/local/lib/ /usr/local/lib/
  ```

  I had to use `--disable-rttopo` from the tip in https://github.com/OSGeo/gdal/pull/3443 and also needed to install `libminizip-dev`.

  This works, sort of... I'm getting a weird issue where the `/dbname` page is hanging some of the time instead of loading correctly. Other than that it seems to work, but a hanging page is bad!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-792384854
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDc5MjM4NDg1NA==
created_at: 2021-03-08T00:23:38Z
updated_at: 2021-03-08T00:23:38Z
author_association: OWNER
body:
  One reason to prioritize this issue: Homebrew upgraded to SpatiaLite 5.0 recently https://formulae.brew.sh/formula/spatialite-tools and as a result SpatiaLite database created on my laptop don't appear to be compatible with Datasette when published using `datasette publish`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1248#issuecomment-792385274
issue_url: https://api.github.com/repos/simonw/datasette/issues/1248
node_id: MDEyOklzc3VlQ29tbWVudDc5MjM4NTI3NA==
created_at: 2021-03-08T00:25:10Z
updated_at: 2021-03-08T00:25:10Z
author_association: OWNER
body:
  It's not possible yet, unfortunately. This came up on the forums recently: https://github.com/simonw/datasette/discussions/968

  I'm leaning further towards making the database connection layer itself work via a plugin hook, which would open up the possibility of supporting DuckDB and other databases as well. I've not committed to doing this yet though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1250#issuecomment-792386484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1250
node_id: MDEyOklzc3VlQ29tbWVudDc5MjM4NjQ4NA==
created_at: 2021-03-08T00:29:06Z
updated_at: 2021-03-08T00:29:06Z
author_association: OWNER
body:
  DuckDB has a read-only mechanism: https://duckdb.org/docs/api/python

  ```python
  import duckdb
  con = duckdb.connect(database="/tmp/blah.db", read_only=True)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1252#issuecomment-793308483
issue_url: https://api.github.com/repos/simonw/datasette/issues/1252
node_id: MDEyOklzc3VlQ29tbWVudDc5MzMwODQ4Mw==
created_at: 2021-03-09T03:06:10Z
updated_at: 2021-03-09T03:06:10Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=h1) Report
  > Merging [#1252](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=desc) (d22aa32) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **decrease** coverage by `0.04%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1252/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1252      +/-   ##
  ==========================================
  - Coverage   91.56%   91.51%   -0.05%     
  ==========================================
    Files          34       34              
    Lines        4244     4244              
  ==========================================
  - Hits         3886     3884       -2     
  - Misses        358      360       +2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1252/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=footer). Last update [d0fd833...d22aa32](https://codecov.io/gh/simonw/datasette/pull/1252?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1254#issuecomment-794437715
issue_url: https://api.github.com/repos/simonw/datasette/issues/1254
node_id: MDEyOklzc3VlQ29tbWVudDc5NDQzNzcxNQ==
created_at: 2021-03-09T20:51:19Z
updated_at: 2021-03-09T20:51:19Z
author_association: OWNER
body:
  Did you see my note on https://github.com/simonw/datasette/issues/1249#issuecomment-792384382 about a weird issue I was having with the `/dbname` page hanging the server? Have you seen anything like that in your work here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1254#issuecomment-794441034
issue_url: https://api.github.com/repos/simonw/datasette/issues/1254
node_id: MDEyOklzc3VlQ29tbWVudDc5NDQ0MTAzNA==
created_at: 2021-03-09T20:54:18Z
updated_at: 2021-03-09T21:12:15Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=h1) Report
  > Merging [#1254](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=desc) (b103204) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **decrease** coverage by `0.04%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1254/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1254      +/-   ##
  ==========================================
  - Coverage   91.56%   91.51%   -0.05%     
  ==========================================
    Files          34       34              
    Lines        4244     4244              
  ==========================================
  - Hits         3886     3884       -2     
  - Misses        358      360       +2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1254/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.93% <0.00%> (-0.75%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=footer). Last update [d0fd833...b103204](https://codecov.io/gh/simonw/datasette/pull/1254?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1254#issuecomment-794443710
issue_url: https://api.github.com/repos/simonw/datasette/issues/1254
node_id: MDEyOklzc3VlQ29tbWVudDc5NDQ0MzcxMA==
created_at: 2021-03-09T20:56:45Z
updated_at: 2021-03-09T20:56:45Z
author_association: NONE
body:
  Oh wow I didn't even see that you had opened an issue about this so recently. I'll check on `/dbname` and report back.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1254#issuecomment-794518438
issue_url: https://api.github.com/repos/simonw/datasette/issues/1254
node_id: MDEyOklzc3VlQ29tbWVudDc5NDUxODQzOA==
created_at: 2021-03-09T22:04:23Z
updated_at: 2021-03-09T22:04:23Z
author_association: NONE
body:
  Dang, you're absolutely right. Spatialite 5.0 had been working fine for a plugin I was developing, but it apparently is broken in several other ways.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1256#issuecomment-795085921
issue_url: https://api.github.com/repos/simonw/datasette/issues/1256
node_id: MDEyOklzc3VlQ29tbWVudDc5NTA4NTkyMQ==
created_at: 2021-03-10T08:35:17Z
updated_at: 2021-03-10T08:35:17Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=h1) Report
  > Merging [#1256](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=desc) (4eef524) into [main](https://codecov.io/gh/simonw/datasette/commit/d0fd833b8cdd97e1b91d0f97a69b494895d82bee?el=desc) (d0fd833) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1256/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1256   +/-   ##
  =======================================
    Coverage   91.56%   91.56%           
  =======================================
    Files          34       34           
    Lines        4244     4244           
  =======================================
    Hits         3886     3886           
    Misses        358      358           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=footer). Last update [d0fd833...4eef524](https://codecov.io/gh/simonw/datasette/pull/1256?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-795893813
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDc5NTg5MzgxMw==
created_at: 2021-03-10T18:43:39Z
updated_at: 2021-03-10T18:43:39Z
author_association: NONE
body:
  @simonw Unfortunately this issue as I reported it is not actually solved in version 0.55.

  Every link which is returned by the `Datasette.absolute_url` method is still wrong, because it uses the request URL as the base. This still includes the suggested facet links and pagination links.

  What I wrote originally still stands:

  > Although many of the URLs in the pages are correct (presumably because they either use absolute paths which include `base_url` or relative paths), the faceting and pagination links still use fully-qualified URLs pointing at `http://localhost:8001`.
  > 
  > I looked into this a little in the source code, and it seems to be an issue anywhere `request.url` or `request.path` is used, as these contain the values for the request between the frontend (Apache) and backend (Datasette) server. Those properties are primarily used via the `path_with_…` family of utility functions and the `Datasette.absolute_url` method.

   Would you prefer to re-open this issue or have me create a new one?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-795918377
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDc5NTkxODM3Nw==
created_at: 2021-03-10T19:01:48Z
updated_at: 2021-03-10T19:01:48Z
author_association: OWNER
body:
  The biggest challenge here I think is to replicate the exact situation here this happens in a Python unit test. The fix should be easy once we have a test in place.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-795939998
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDc5NTkzOTk5OA==
created_at: 2021-03-10T19:16:55Z
updated_at: 2021-03-10T19:16:55Z
author_association: NONE
body:
  Nod. The problem with the tests is that they're ignoring the origin (hostname, port) of links. In a reverse proxy situation, the frontend request origin is different than the backend request origin. The problem is Datasette generates links with the backend request origin.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-795950636
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDc5NTk1MDYzNg==
created_at: 2021-03-10T19:24:13Z
updated_at: 2021-03-10T19:24:13Z
author_association: NONE
body:
  I think this could be solved by one of:

  1. Stop generating absolute URLs, e.g. ones that include an origin. Relative URLs with absolute paths are fine, as long as they take `base_url` into account (as they do now, yay!).
  2. Extend `base_url` to include the expected frontend origin, and then use that information when generating absolute URLs.
  3. Document which HTTP headers the reverse proxy should set (e.g. the `X-Forwarded-*` family of conventional headers) to pass the frontend origin information to Datasette, and then use that information when generating absolute URLs.

  Option 1 seems like the easiest to me, if you can get away with never having to generate an absolute URL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1211#issuecomment-796854370
issue_url: https://api.github.com/repos/simonw/datasette/issues/1211
node_id: MDEyOklzc3VlQ29tbWVudDc5Njg1NDM3MA==
created_at: 2021-03-11T16:15:29Z
updated_at: 2021-03-11T16:15:29Z
author_association: OWNER
body:
  Thanks very much for this - it's really comprehensive. I need to bake some of these patterns into my coding habits better!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/670#issuecomment-797158641
issue_url: https://api.github.com/repos/simonw/datasette/issues/670
node_id: MDEyOklzc3VlQ29tbWVudDc5NzE1ODY0MQ==
created_at: 2021-03-12T00:59:49Z
updated_at: 2021-03-12T00:59:49Z
author_association: OWNER
body:
  > Challenge: what's the equivalent for PostgreSQL of opening a database in read only mode? Will I have to talk users through creating read only credentials?

  It looks like the answer to this is yes - I'll need users to setup read-only credentials. Here's a TIL about that: https://til.simonwillison.net/postgresql/read-only-postgresql-user

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1259#issuecomment-797790017
issue_url: https://api.github.com/repos/simonw/datasette/issues/1259
node_id: MDEyOklzc3VlQ29tbWVudDc5Nzc5MDAxNw==
created_at: 2021-03-12T22:22:12Z
updated_at: 2021-03-12T22:22:12Z
author_association: OWNER
body:
  https://sqlite.org/lang_with.html

  >  Prior to SQLite 3.35.0, all CTEs where treated as if the NOT MATERIALIZED phrase was present

  It looks like this optimization is completely unavailable on SQLite prior to 3.35.0 (released 12th March 2021). But I could still rewrite the faceting to work in this way, using the exact same SQL - it would just be significantly faster on 3.35.0+ (assuming it's actually faster in practice - would need to benchmark).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1259#issuecomment-797801075
issue_url: https://api.github.com/repos/simonw/datasette/issues/1259
node_id: MDEyOklzc3VlQ29tbWVudDc5NzgwMTA3NQ==
created_at: 2021-03-12T22:53:56Z
updated_at: 2021-03-12T22:55:16Z
author_association: OWNER
body:
  OK, a better comparison:

  https://global-power-plants.datasettes.com/global-power-plants?sql=WITH+data+as+%28%0D%0A++select%0D%0A++++*%0D%0A++from%0D%0A++++%5Bglobal-power-plants%5D%0D%0A%29%2C%0D%0Acountry_long+as+%28select+%0D%0A++%27country_long%27+as+col%2C+country_long+as+value%2C+count%28*%29+as+c+from+data+group+by+country_long%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aprimary_fuel+as+%28%0D%0Aselect%0D%0A++%27primary_fuel%27+as+col%2C+primary_fuel+as+value%2C+count%28*%29+as+c+from+data+group+by+primary_fuel%0D%0A++order+by+c+desc+limit+31%0D%0A%29%2C%0D%0Aowner+as+%28%0D%0Aselect%0D%0A++%27owner%27+as+col%2C+owner+as+value%2C+count%28*%29+as+c+from+data+group+by+owner%0D%0A++order+by+c+desc+limit+31%0D%0A%29%0D%0Aselect+*+from+primary_fuel+union+select+*+from+country_long%0D%0Aunion+select+*+from+owner+order+by+col%2C+c+desc calculates facets against three columns. It takes **78.5ms** (and 34.5ms when I refreshed it, presumably after warming some SQLite caches of some sort).

  https://global-power-plants.datasettes.com/global-power-plants/global-power-plants?_facet=country_long&_facet=primary_fuel&_trace=1&_size=0 shows those facets with size=0 on the SQL query - and shows a SQL trace at the bottom of the page.

  The country_long facet query takes 45.36ms, owner takes 38.45ms, primary_fuel takes 49.04ms - so a total of 132.85ms

  That's against https://global-power-plants.datasettes.com/-/versions says SQLite 3.27.3 - so even on a SQLite version that doesn't materialize the CTEs there's a significant performance boost to doing all three facets in a single CTE query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1259#issuecomment-797804869
issue_url: https://api.github.com/repos/simonw/datasette/issues/1259
node_id: MDEyOklzc3VlQ29tbWVudDc5NzgwNDg2OQ==
created_at: 2021-03-12T23:05:05Z
updated_at: 2021-03-12T23:05:05Z
author_association: OWNER
body:
  I wonder if I could optimize facet suggestion in the same way?

  One challenge: the query time limit will apply to the full CTE query, not to the individual columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1259#issuecomment-797827038
issue_url: https://api.github.com/repos/simonw/datasette/issues/1259
node_id: MDEyOklzc3VlQ29tbWVudDc5NzgyNzAzOA==
created_at: 2021-03-13T00:15:40Z
updated_at: 2021-03-13T00:15:40Z
author_association: OWNER
body:
  If all of the facets were being calculated in a single query, I'd be willing to bump the facet time limit up to something a lot higher, maybe even a full second. There's a chance that could work amazingly well with a materialized CTE.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798436026
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDc5ODQzNjAyNg==
created_at: 2021-03-13T14:23:16Z
updated_at: 2021-03-13T14:23:16Z
author_association: NONE
body:
  This PR allows my import to succeed.

  It looks like some events don't have an `id`, but do have `HKExternalUUID` (which gets turned into `metadata_HKExternalUUID`), so I use this as a fallback.

  If a record has neither of these, I changed it to just print the record (for debugging) and `return`.

  For some odd reason this ran fine at first, and now (after removing the generated db and trying again) I'm getting a different error (duplicate column name).

  Looks like it may have run when I had two successive runs without remembering to delete the db in between. Will try to refactor.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798468572
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14
node_id: MDEyOklzc3VlQ29tbWVudDc5ODQ2ODU3Mg==
created_at: 2021-03-13T14:47:31Z
updated_at: 2021-03-13T14:47:31Z
author_association: NONE
body:
  Ok, new PR works. I'm not `git` enough so I just force-pushed over the old one.

  I still end up with a lot of activities that are missing an `id` and therefore skipped (since this is used as the primary key). For example:

  ```
  {'workoutActivityType': 'HKWorkoutActivityTypeRunning', 'duration': '35.31666666666667', 'durationUnit': 'min', 'totalDistance': '4.010870267636999', 'totalDistanceUnit': 'mi', 'totalEnergyBurned': '660.3516235351562', 'totalEnergyBurnedUnit': 'Cal', 'sourceName': 'Strava', 'sourceVersion': '22810', 'creationDate': '2020-07-16 13:38:26 -0700', 'startDate': '2020-07-16 06:38:26 -0700', 'endDate': '2020-07-16 07:13:45 -0700'}
  ```

  I also end up with some unhappy characters (in the skipped events), such as: `'sourceName': 'Nathan’s Apple\xa0Watch',`.

  But it's successfully making it through the file, and the resulting db opens in datasette, so I'd call that progress.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1260#issuecomment-798913090
issue_url: https://api.github.com/repos/simonw/datasette/issues/1260
node_id: MDEyOklzc3VlQ29tbWVudDc5ODkxMzA5MA==
created_at: 2021-03-14T14:01:30Z
updated_at: 2021-03-14T14:01:30Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=h1) Report
  > Merging [#1260](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=desc) (90f5fb6) into [main](https://codecov.io/gh/simonw/datasette/commit/8e18c7943181f228ce5ebcea48deb59ce50bee1f?el=desc) (8e18c79) will **not change** coverage.
  > The diff coverage is `83.33%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1260/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1260   +/-   ##
  =======================================
    Coverage   91.51%   91.51%           
  =======================================
    Files          34       34           
    Lines        4255     4255           
  =======================================
    Hits         3894     3894           
    Misses        361      361           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/inspect.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2luc3BlY3QucHk=) | `36.11% <0.00%> (ø)` | |
  | [datasette/default\_magic\_parameters.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWFnaWNfcGFyYW1ldGVycy5weQ==) | `91.17% <50.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.85% <100.00%> (ø)` | |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.01% <100.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1260/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.88% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=footer). Last update [8e18c79...90f5fb6](https://codecov.io/gh/simonw/datasette/pull/1260?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-799002993
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: MDEyOklzc3VlQ29tbWVudDc5OTAwMjk5Mw==
created_at: 2021-03-14T23:41:51Z
updated_at: 2021-03-14T23:41:51Z
author_association: CONTRIBUTOR
body:
  Now that [Lambda supports Docker](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/), this probably is a bit easier and may be able to build on top of the existing package command.

  There are weirdnesses in how the command actually gets invoked; the [aws-lambda-python image](https://hub.docker.com/r/amazon/aws-lambda-python) shows a bit of that. So Datasette would probably need some sort of Lambda-specific entry point to make this work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-799003172
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: MDEyOklzc3VlQ29tbWVudDc5OTAwMzE3Mg==
created_at: 2021-03-14T23:42:57Z
updated_at: 2021-03-14T23:42:57Z
author_association: CONTRIBUTOR
body:
  Oh, and the container image can be up to 10GB, so the EFS step might not be needed except for pretty big stuff.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/236#issuecomment-799066252
issue_url: https://api.github.com/repos/simonw/datasette/issues/236
node_id: MDEyOklzc3VlQ29tbWVudDc5OTA2NjI1Mg==
created_at: 2021-03-15T03:34:52Z
updated_at: 2021-03-15T03:34:52Z
author_association: OWNER
body:
  Yeah the Lambda Docker stuff is pretty odd - you still don't get to speak HTTP, you have to speak their custom event protocol instead.

  https://github.com/glassechidna/serverlessish looks interesting here - it adds a proxy inside the container which allows your existing HTTP Docker image to run within Docker-on-Lambda. I've not tried it out yet though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/246#issuecomment-799479175
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/246
node_id: MDEyOklzc3VlQ29tbWVudDc5OTQ3OTE3NQ==
created_at: 2021-03-15T14:47:31Z
updated_at: 2021-03-15T14:47:31Z
author_association: OWNER
body:
  This is a smart feature. I have something that does this in Datasette, extracting it out to `sqlite-utils` makes a lot of sense.

  https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L818-L829

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/246#issuecomment-801816980
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/246
node_id: MDEyOklzc3VlQ29tbWVudDgwMTgxNjk4MA==
created_at: 2021-03-18T10:40:32Z
updated_at: 2021-03-18T10:43:04Z
author_association: NONE
body:
  I have found a similar problem, but I only when using that type of query (with `*` for doing a prefix search). I'm also building something on top of FTS5/sqlite-utils, and the way I decided to handle it was creating a specific function for prefixes. According to [the docs](https://www2.sqlite.org/fts5.html#fts5_prefix_queries), the query can be done in this 2 ways:

  ```sql
  ... MATCH '"one two thr" * '
  ... MATCH 'one + two + thr*'
  ```

  I thought I could build a query like the first one using this function:

  ```python
  def prefix(query: str):
      return f'"{query}" *'
  ```

  And then I use the output of that function as the query parameter for the standard `.search()` method in sqlite-utils.

  However, my use case is different because I'm the one "deciding" when to use a prefix search, not the end user. I also haven't done many tests, but maybe you found that useful. One thing I could think of is checking if the query has an `*` at the end, remove it and build the prefix query using the function above.

  This is just for prefix queries, I think having the escaping function is still useful for other use cases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/159#issuecomment-802032152
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/159
node_id: MDEyOklzc3VlQ29tbWVudDgwMjAzMjE1Mg==
created_at: 2021-03-18T15:42:52Z
updated_at: 2021-03-18T15:42:52Z
author_association: NONE
body:
  I confirm the bug. Happens for me in version 3.6. I use the call to delete all the records:
  `table.delete_where()`
  This does not delete anything. 

  I see that `delete()` method DOES use context manager `with self.db.conn:` which should help. You may want to align the code of both methods.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1262#issuecomment-802095132
issue_url: https://api.github.com/repos/simonw/datasette/issues/1262
node_id: MDEyOklzc3VlQ29tbWVudDgwMjA5NTEzMg==
created_at: 2021-03-18T16:37:45Z
updated_at: 2021-03-18T16:37:45Z
author_association: CONTRIBUTOR
body:
  This sounds like a good use case for a plugin, since this will only be useful for a subset of Datasette users. It shouldn't be too difficult to add a button to do this with the available plugin hooks - have you taken a look at https://docs.datasette.io/en/latest/writing_plugins.html?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1262#issuecomment-802099264
issue_url: https://api.github.com/repos/simonw/datasette/issues/1262
node_id: MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA==
created_at: 2021-03-18T16:43:09Z
updated_at: 2021-03-18T16:43:09Z
author_association: OWNER
body:
  I often find myself wanting this too, when I'm exploring a new dataset.

  i agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way.

  I'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables `order by random()` could enable a lot of other useful plugin features too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1262#issuecomment-802164134
issue_url: https://api.github.com/repos/simonw/datasette/issues/1262
node_id: MDEyOklzc3VlQ29tbWVudDgwMjE2NDEzNA==
created_at: 2021-03-18T17:55:00Z
updated_at: 2021-03-18T17:55:00Z
author_association: NONE
body:
  Thanks for the comments. I'll take a look at the documentation to familiarize myself, as I haven't tried to write any plugins yet. With some luck I might be ready to write it when the hook is implemented.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1265#issuecomment-802923254
issue_url: https://api.github.com/repos/simonw/datasette/issues/1265
node_id: MDEyOklzc3VlQ29tbWVudDgwMjkyMzI1NA==
created_at: 2021-03-19T15:39:15Z
updated_at: 2021-03-19T15:39:15Z
author_association: CONTRIBUTOR
body:
  It doesn't use basic auth, but you can put a whole datasette instance, or parts of this, behind a username/password prompt using https://github.com/simonw/datasette-auth-passwords

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1265#issuecomment-803130332
issue_url: https://api.github.com/repos/simonw/datasette/issues/1265
node_id: MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg==
created_at: 2021-03-19T21:03:09Z
updated_at: 2021-03-19T21:03:09Z
author_association: OWNER
body:
  This is now available in `datasette-auth-passwords` 0.4! https://github.com/simonw/datasette-auth-passwords/releases/tag/0.4

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1261#issuecomment-803466868
issue_url: https://api.github.com/repos/simonw/datasette/issues/1261
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA==
created_at: 2021-03-20T21:36:06Z
updated_at: 2021-03-20T21:36:06Z
author_association: OWNER
body:
  This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28

  I'm looking at a fix for that now, so watch that issue for updates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1261#issuecomment-803468314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1261
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA==
created_at: 2021-03-20T21:48:48Z
updated_at: 2021-03-20T21:48:48Z
author_association: OWNER
body:
  That's fixed in this release of `datasette-publish-vercel`: https://github.com/simonw/datasette-publish-vercel/releases/tag/0.9.2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-803469623
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw==
created_at: 2021-03-20T22:01:23Z
updated_at: 2021-03-20T22:01:23Z
author_association: OWNER
body:
  I'm going to keep `?_shape=array` working on the assumption that many existing uses of the Datasette API are already using that option, so it would be nice not to break them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1258#issuecomment-803471702
issue_url: https://api.github.com/repos/simonw/datasette/issues/1258
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg==
created_at: 2021-03-20T22:19:39Z
updated_at: 2021-03-20T22:19:39Z
author_association: OWNER
body:
  This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between `?sql=...&param=` and `?sql=` with no `param` specified at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-803471917
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw==
created_at: 2021-03-20T22:21:33Z
updated_at: 2021-03-20T22:21:33Z
author_association: OWNER
body:
  This has been blocking things for too long.

  If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-803472278
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA==
created_at: 2021-03-20T22:25:04Z
updated_at: 2021-03-20T22:25:04Z
author_association: OWNER
body:
  I came up with a slightly wild idea for this that would involve pytest-style dependency injection.

  Prototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7

  Copying from my private notes:

  > Use the lazy evaluated DI mechanism to break up table view into different pieces eg for faceting
  > 
  > Use that to solve JSON vs HTML views
  > 
  > Oh here's an idea: what if the various components of the table view were each defined  as async functions.... and then executed using asyncio.gather in order to run the SQL queries in parallel? Then try benchmarking with different numbers of threads?
  > 
  > The async_call_with_arguments function could do this automatically for any awaitable dependencies
  > 
  > This would give me massively parallel dependency injection
  > 
  > (I could build an entire framework around this and call it c64)
  > 
  > Idea: arguments called eg "count" are executed and the result passed to the function. If called count_fn then a reference to the not-yet-called function is passed instead 
  > 
  > I'm not going to completely combine the views mechanism and the render hooks. Instead, the core view will define a bunch of functions used to compose the page and the render hook will have conditional access to those functions - which will otherwise be asyncio.gather executed directly by the HTML page version
  > 
  > Using asyncio.gather to execute facets and suggest facets in parallel would be VERY interesting 
  > 
  > suggest facets should be VERY cachable - doesn't matter if it's wrong unlike actual facets themselves
  > 
  > What if all Datasette views were defined in terms of dependency injection - and those dependency functions could themselves depend on others just like pytest fixtures. Everything would become composable and async stuff could execute in parallel
  > 
  > FURTHER IDEA: use this for the ?_extra= mechanism as well.
  > 
  > Any view in Datasette can be defined as a collection of named keys. Each of those keys maps to a function or an async function that accepts as input other named keys, using DI to handle them.
  > 
  > The HTML view is a defined function. So are the other outputs.
  > 
  > Default original inputs include “request” and “datasette”.
  > 
  > So… maybe a view function is a class methods that use DI. One of those methods as an .html() method used for the default page.
  > 
  > Output formats are a bit more complicated because they are supposed to be defined separately in plugins. They are unified across query, row and table though.
  > 
  > I’m going to try breaking up the TableView to see what happens.

reactions:
  {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1}


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-803472595
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ==
created_at: 2021-03-20T22:28:12Z
updated_at: 2021-03-20T22:28:12Z
author_association: OWNER
body:
  Another idea I had: a view is a class that takes the `datasette` instance in its constructor, and defines a `__call__` method that accepts a request and returns a response. Except `await __call__` looks like it might be a bit messy, discussion in https://github.com/encode/starlette/issues/886

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-803473015
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ==
created_at: 2021-03-20T22:33:05Z
updated_at: 2021-03-20T22:33:05Z
author_association: OWNER
body:
  Things this mechanism needs to be able to support:

  - Returning a default JSON representation
  - Defining "extra" JSON representations blocks, which can be requested using `?_extra=`
  - Returning rendered HTML, based on the default JSON + one or more extras + a template
  - Using Datasette output renderers to return e.g. CSV data
  - Potentially also supporting streaming output renderers for streaming CSV/TSV/JSON-nl etc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng==
created_at: 2021-03-21T02:33:45Z
updated_at: 2021-03-21T02:33:45Z
author_association: OWNER
body:
  Did you run `enable-fts` before you inserted the data?

  If so you'll need to run `populate-fts` after the insert to populate the FTS index.

  A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803502424
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzUwMjQyNA==
created_at: 2021-03-21T02:43:32Z
updated_at: 2021-03-21T02:43:32Z
author_association: NONE
body:
  > Did you run `enable-fts` before you inserted the data?
  > 
  > If so you'll need to run `populate-fts` after the insert to populate the FTS index.
  > 
  > A better solution may be to add `--create-triggers` to the `enable-fts` command to add triggers that will automatically keep the index updated as you insert new records.

  Wow. Wasn't expecting a response this quick, especially during a weekend. :-) Sincerely appreciate it.
  I tried the `populate-fts` and that did the trick. My bad for not consulting the docs again. I think I forgot to add that step when I automated the workflow.
  Thanks for the suggestion. I'll close this issue. Have a great weekend and many many thanks for creating these suite of tools around sqlite.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/942#issuecomment-803631102
issue_url: https://api.github.com/repos/simonw/datasette/issues/942
node_id: MDEyOklzc3VlQ29tbWVudDgwMzYzMTEwMg==
created_at: 2021-03-21T17:48:42Z
updated_at: 2021-03-21T17:48:42Z
author_association: CONTRIBUTOR
body:
  I like this idea. Though it might be nice to have some kind of automated system from database to file, so that developers could easily track diffs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/647#issuecomment-803673225
issue_url: https://api.github.com/repos/simonw/datasette/issues/647
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ==
created_at: 2021-03-21T22:44:19Z
updated_at: 2021-03-21T22:44:19Z
author_association: OWNER
body:
  Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature.

  I'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy.

  I may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803691236
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg==
created_at: 2021-03-22T00:32:03Z
updated_at: 2021-03-22T00:32:03Z
author_association: OWNER
body:
  Here's something odd: when I run `datasette tuscany_housenumbers.sqlite --load-extension=spatialite` on macOS against SpatiaLite installed using Homebrew (which reports `"spatialite": "5.0.0"` on the `/-/versions` page) I don't get any weird errors at all, everything works fine.

  But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages.

  This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803692673
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw==
created_at: 2021-03-22T00:38:42Z
updated_at: 2021-03-22T00:38:42Z
author_association: OWNER
body:
  Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803694359
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ==
created_at: 2021-03-22T00:45:47Z
updated_at: 2021-03-22T00:45:47Z
author_association: OWNER
body:
  https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using `python:3.9-slim-buster` or `ubuntu:20.04` - but 20.04 is focal which still has SpatiaLite `4.3.0a-6build1` - It's `20.10` that has 5.0: https://packages.ubuntu.com/groovy/libspatialite-dev

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803694661
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ==
created_at: 2021-03-22T00:46:49Z
updated_at: 2021-03-22T00:46:49Z
author_association: OWNER
body:
  Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803697211
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ==
created_at: 2021-03-22T00:58:01Z
updated_at: 2021-03-22T00:58:01Z
author_association: OWNER
body:
  I'm messing around with the `Dockerfile` and after each change I'm running:

      docker build . -t datasette-spatialite

  And then:

      docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803697546
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng==
created_at: 2021-03-22T00:59:47Z
updated_at: 2021-03-22T00:59:47Z
author_association: OWNER
body:
  To debug I'm running:

      docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash

  This gets me a shell I can use.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803698168
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA==
created_at: 2021-03-22T01:02:02Z
updated_at: 2021-03-22T01:02:30Z
author_association: OWNER
body:
  This is the shortest Dockerfile that appeared to give me a working SpatiaLite module:
  ```dockerfile
  FROM ubuntu:20.10

  # Setup build dependencies
  RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

  # Add local code to the image instead of fetching from pypi.
  COPY . /datasette

  RUN pip install /datasette

  RUN rm -rf /datasette

  EXPOSE 8001
  CMD ["datasette"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803698983
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw==
created_at: 2021-03-22T01:05:36Z
updated_at: 2021-03-22T01:06:23Z
author_association: OWNER
body:
  It's pretty big though. I tried this version which avoids copying junk from my laptop in:

  ```dockerfile
  FROM ubuntu:20.10

  # Setup build dependencies
  RUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean

  RUN pip install datasette

  EXPOSE 8001
  CMD ["datasette"]
  ```
  And got this:
  ```
  datasette % docker images datasette-spatialite    
  REPOSITORY             TAG       IMAGE ID       CREATED         SIZE
  datasette-spatialite   latest    0796950653c2   2 seconds ago   528MB
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803700626
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg==
created_at: 2021-03-22T01:13:04Z
updated_at: 2021-03-22T01:13:04Z
author_association: OWNER
body:
  Building a Dockerfile containing just `FROM ubuntu:20.10` gave me `79.5MB`.

  Building this one:
  ```dockerfile
  FROM ubuntu:20.10

  # Setup build dependencies
  RUN apt update && \
      apt install -y python3-pip libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/
  ```
  Resulted in a 515MB image.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803747701
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ==
created_at: 2021-03-22T04:15:40Z
updated_at: 2021-03-22T04:15:40Z
author_association: OWNER
body:
  Here's a trick: install SpatiaLite in `ubuntu:20.10` and then copy it into the final `python:3.9.2-slim` image.

  ```dockerfile
  FROM ubuntu:20.10 as install_spatialite

  RUN apt update && \
      apt install -y libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/

  FROM python:3.9.2-slim as build

  RUN pip install datasette

  #COPY . /datasette
  #RUN pip install /datasette

  FROM python:3.9.2-slim

  # Copy python dependencies and spatialite libraries
  COPY --from=build /usr/local/lib/ /usr/local/lib/
  # Copy executables
  COPY --from=build /usr/local/bin /usr/local/bin
  # Copy spatial extensions
  COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu

  ENV LD_LIBRARY_PATH=/usr/local/lib

  EXPOSE 8001
  CMD ["datasette"]
  ```
  That produced a 265MB image.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803748469
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ==
created_at: 2021-03-22T04:17:51Z
updated_at: 2021-03-22T04:17:51Z
author_association: OWNER
body:
  ... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:

  ```
  datasette % docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db
    File "/usr/local/lib/python3.9/sqlite3/dbapi2.py", line 27, in <module>
      from _sqlite3 import *
  ImportError: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /usr/lib/x86_64-linux-gnu/libsqlite3.so.0)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803749831
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ==
created_at: 2021-03-22T04:22:35Z
updated_at: 2021-03-22T04:22:35Z
author_association: OWNER
body:
  I tried copying just the `mod_spatialite.so` file:
  ```dockerfile
  FROM ubuntu:20.10 as install_spatialite

  RUN apt update && \
      apt install -y libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/

  FROM python:3.9.2-slim as build

  RUN pip install datasette

  #COPY . /datasette
  #RUN pip install /datasette

  FROM python:3.9.2-slim

  # Copy python dependencies and spatialite libraries
  COPY --from=build /usr/local/lib/ /usr/local/lib/
  # Copy executables
  COPY --from=build /usr/local/bin /usr/local/bin
  # Copy spatial extensions
  COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

  ENV LD_LIBRARY_PATH=/usr/local/lib

  EXPOSE 8001
  CMD ["datasette"]
  ```
  But when I ran Datasette with `--load-extension=spatialite` I got this:
  ```
    File "/usr/local/lib/python3.9/site-packages/datasette/database.py", line 151, in in_thread
      self.ds._prepare_connection(conn, self.name)
    File "/usr/local/lib/python3.9/site-packages/datasette/app.py", line 502, in _prepare_connection
      conn.execute(f"SELECT load_extension('{extension}')")
  sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803750399
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ==
created_at: 2021-03-22T04:24:25Z
updated_at: 2021-03-22T04:24:25Z
author_association: OWNER
body:
  I'll try using `ubuntu:20.10` for everything:
  ```dockerfile
  FROM ubuntu:20.10 as install_spatialite

  RUN apt update && \
      apt install -y libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/

  FROM ubuntu:20.10 as build

  RUN apt update && \
      apt install -y python3-pip && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/

  RUN pip install datasette

  #COPY . /datasette
  #RUN pip install /datasette

  FROM ubuntu:20.10

  # Copy python dependencies and spatialite libraries
  COPY --from=build /usr/local/lib/ /usr/local/lib/
  # Copy executables
  COPY --from=build /usr/local/bin /usr/local/bin
  # Copy spatial extensions
  COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so

  ENV LD_LIBRARY_PATH=/usr/local/lib

  EXPOSE 8001
  CMD ["datasette"]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803750617
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw==
created_at: 2021-03-22T04:25:14Z
updated_at: 2021-03-22T04:25:14Z
author_association: OWNER
body:
  Got this error attempting to run Datasette (with or without SpatiaLite):
  ```
  standard_init_linux.go:219: exec user process caused: no such file or directory
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803751068
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA==
created_at: 2021-03-22T04:26:45Z
updated_at: 2021-03-22T04:26:45Z
author_association: OWNER
body:
  Here's why:
  ```
  datasette % docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash                                                      
  root@3430352ff378:/# datasette
  bash: /usr/local/bin/datasette: /usr/bin/python3: bad interpreter: No such file or directory
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1249#issuecomment-803753388
issue_url: https://api.github.com/repos/simonw/datasette/issues/1249
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA==
created_at: 2021-03-22T04:34:20Z
updated_at: 2021-03-22T04:35:10Z
author_association: OWNER
body:
  Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load...
  ```dockerfile
  FROM ubuntu:20.10 as install_spatialite

  RUN apt update && \
      apt install -y libsqlite3-mod-spatialite && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/

  FROM ubuntu:20.10

  RUN apt update && \
      apt install -y python3-pip && \
      apt clean && \
      rm -rf /var/lib/{apt,dpkg,cache,log}/

  RUN pip install datasette

  # Copy spatial extensions
  COPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/

  ENV LD_LIBRARY_PATH=/usr/local/lib

  EXPOSE 8001
  CMD ["datasette"]
  ```
  (Which produced a 550MB image)

  And when I ran Datasette I got that same error where the database listing page hangs!
  ```
  docker run -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest datasette -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite --load-extension=spatialite
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1267#issuecomment-803754226
issue_url: https://api.github.com/repos/simonw/datasette/issues/1267
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg==
created_at: 2021-03-22T04:37:26Z
updated_at: 2021-03-22T04:37:26Z
author_association: OWNER
body:
  Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there.

  This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins).

  It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803757746
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng==
created_at: 2021-03-22T04:50:40Z
updated_at: 2021-03-22T04:51:52Z
author_association: OWNER
body:
  Here's a fun debugging trick:

      docker run -it -p 8001:8001 -v `pwd`:/mnt datasette-spatialite:latest bash
      root@16197781a7b5:/# python3 -m trace --trace $(which datasette) \
        -p 8001 -h 0.0.0.0 /mnt/tuscany_housenumbers.sqlite \
        --load-extension=spatialite

  A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python.

  After about a minute it's finished starting and gets to this point:

  ```
  selectors.py(452):             if timeout is None:
  selectors.py(454):             elif timeout <= 0:
  selectors.py(459):                 timeout = math.ceil(timeout * 1e3) * 1e-3
  selectors.py(464):             max_ev = max(len(self._fd_to_key), 1)
  selectors.py(466):             ready = []
  selectors.py(467):             try:
  selectors.py(468):                 fd_event_list = self._selector.poll(timeout, max_ev)
  ```
  Now  I can make some HTTP requests against it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803758793
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw==
created_at: 2021-03-22T04:54:32Z
updated_at: 2021-03-22T04:54:32Z
author_association: OWNER
body:
  Hitting http://localhost:8001/tuscany_housenumbers triggers the bug. It gets stuck in a loop that looks like this:

  <img width="381" alt="datasette_—_root_16197781a7b5____—_com_docker_cli_◂_docker_run_-it_-p_8001_8001_-v___Dropbox_Development_datasette__mnt_datasette-spatialite_latest_bash_—_195×48_and_getIncidentsGit_—_-zsh_—_162×60" src="https://user-images.githubusercontent.com/9599/111942115-e0b75600-8a8f-11eb-8b6d-5c37240ce2f8.png">

  Which looks to me like this code: https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/__init__.py#L139-L158

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803759051
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ==
created_at: 2021-03-22T04:55:22Z
updated_at: 2021-03-22T04:55:22Z
author_association: OWNER
body:
  So I think there's a bug in the way the `set_progress_handler()` mechanism works when used in conjunction with SpatiaLite 5.0 on Linux.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803762609
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ==
created_at: 2021-03-22T05:05:00Z
updated_at: 2021-03-22T05:05:00Z
author_association: OWNER
body:
  Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run `gdb`. I had to use `--privileged` here because otherwise `gdb` showed a "Could not attach to process" error.
  ```
  docker exec --privileged -it 16197781a7b5 bash
  # apt-get install gdb python3-dbg
  # gdb /usr/bin/python3 -p 20
  ```
  This paused the process. I tried running this:
  ```
  (gdb) py-bt
  Traceback (most recent call first):
    File "/usr/lib/python3.8/asyncio/base_events.py", line 1845, in _run_once
      if handle._cancelled:
    File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
      self._run_once()
    File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
      self.run_forever()
    File "/usr/local/lib/python3.8/dist-packages/uvicorn/server.py", line 49, in run
      loop.run_until_complete(self.serve(sockets=sockets))
    File "/usr/local/lib/python3.8/dist-packages/uvicorn/main.py", line 386, in run
      server.run()
    File "/usr/local/lib/python3.8/dist-packages/datasette/cli.py", line 575, in serve
      uvicorn.run(ds.app(), **uvicorn_kwargs)
    File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/usr/local/bin/datasette", line 8, in <module>
      sys.exit(cli())
    <built-in method exec of module object at remote 0x7f0981a280e0>
    File "/usr/lib/python3.8/trace.py", line 450, in runctx
      exec(cmd, globals, locals)
    File "/usr/lib/python3.8/trace.py", line 6632, in main
    File "/usr/lib/python3.8/trace.py", line 756, in <module>
      main()
    <built-in method exec of module object at remote 0x7f0981a280e0>
    File "/usr/lib/python3.8/runpy.py", line 343, in _run_code
    File "/usr/lib/python3.8/runpy.py", line 450, in _run_module_as_main
  ```
  Not sure if that's useful or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803762969
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ==
created_at: 2021-03-22T05:05:51Z
updated_at: 2021-03-22T05:05:51Z
author_association: OWNER
body:
  I had to run `docker kill 16197781a7b5` to kill the broken container - Ctrl+C in the Datasette console window didn't do anything.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803764200
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA==
created_at: 2021-03-22T05:09:13Z
updated_at: 2021-03-22T05:09:13Z
author_association: OWNER
body:
  I tried building a container where the `conn.set_progress_handler(handler, n)` line was commented out... and it fixed the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803764919
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ==
created_at: 2021-03-22T05:11:11Z
updated_at: 2021-03-22T05:11:11Z
author_association: OWNER
body:
  Maybe I could implement SQLite query timeouts using the `interrupt()` method instead of the progress handler hack I'm currently using?

  https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803773484
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA==
created_at: 2021-03-22T05:32:29Z
updated_at: 2021-03-22T05:32:29Z
author_association: OWNER
body:
  To figure out which SQL query triggers the problem I added this code to write to a log file:
  ```python
              with sqlite_timelimit(conn, time_limit_ms):
                  try:
                      cursor = conn.cursor()
                      with open("/tmp/sql.log", "ab", buffering=0) as fp:
                          fp.write(("{}: {}\n".format(sql, params)).encode("utf-8"))
                      cursor.execute(sql, params if params is not None else {})
  ```
  I had to use `ab` binary mode because Python doesn't allow `buffering=0` for non-binary file operations.

  With the log enabled, I used `docker exec -it 589ae68de943 bash` to attach to the running container and `tail -f /tmp/sql.log` to see the logs. Here's where it broke:

  ```
  select count(*) from [idx_civici_geom_parent]: None
  select count(*) from [sqlite_stat1]: None
  select count(*) from [sqlite_stat3]: None
  select count(*) from [SpatialIndex]: None
  ```
  So attempting to run a `count(*)` against the `SpatialIndex` virtual table is the thing that triggers the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803774518
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA==
created_at: 2021-03-22T05:34:57Z
updated_at: 2021-03-22T05:34:57Z
author_association: OWNER
body:
  ... and sure enough, adding this code fixed the problem:
  ```diff
  diff --git a/datasette/database.py b/datasette/database.py
  index 3579cce..b466b12 100644
  --- a/datasette/database.py
  +++ b/datasette/database.py
  @@ -224,6 +226,9 @@ class Database:
           # Try to get counts for each table, $limit timeout for each count
           counts = {}
           for table in await self.table_names():
  +            if table == "SpatialIndex":
  +                counts[table] = 0
  +                continue
               try:
                   table_count = (
                       await self.execute(
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803774926
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg==
created_at: 2021-03-22T05:35:56Z
updated_at: 2021-03-22T05:35:56Z
author_association: OWNER
body:
  That's in this code here: https://github.com/simonw/datasette/blob/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b/datasette/database.py#L221-L241

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803777724
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA==
created_at: 2021-03-22T05:42:50Z
updated_at: 2021-03-22T05:43:23Z
author_association: OWNER
body:
  <img width="1129" alt="tuscany_housenumbers__select___from_sqlite_master_where_sql_like__create_virtual_table__" src="https://user-images.githubusercontent.com/9599/111945249-99809380-8a96-11eb-871b-9a1509360bcf.png">

  If I want to avoid counting virtual tables, I need to detect which tables are virtual tables.

  The safest way to do this is probably to pull the `sql` for every table and then, in Python, check for values that start with `create virtual table` after converting to lower case, using any number of spaces.

  This would catch things like ` CREATE       virtual      TABLE` which might be missed by a SQL `like` query.   

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803782705
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ==
created_at: 2021-03-22T05:54:19Z
updated_at: 2021-03-22T05:54:19Z
author_association: OWNER
body:
  Got two new TILs out of this:

  * [Tracing every executed Python statement](https://til.simonwillison.net/python/tracing-every-statement)
  * [Running gdb against a Python process in a running Docker container](https://til.simonwillison.net/docker/gdb-python-docker)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803784902
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg==
created_at: 2021-03-22T05:59:06Z
updated_at: 2021-03-22T05:59:06Z
author_association: OWNER
body:
  Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing `select count(*) from SpatialIndex`.

  That `interrupt` timeout mechanism is worth digging into further.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1268#issuecomment-803802957
issue_url: https://api.github.com/repos/simonw/datasette/issues/1268
node_id: MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw==
created_at: 2021-03-22T06:38:14Z
updated_at: 2021-03-22T06:38:14Z
author_association: OWNER
body:
  Also worth trying is to change this code:
  ```python
       n = 1000 
       if ms < 50: 
           n = 1 
  ```
  What happens with `n = 10` instead?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1270#issuecomment-803834784
issue_url: https://api.github.com/repos/simonw/datasette/issues/1270
node_id: MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA==
created_at: 2021-03-22T07:31:57Z
updated_at: 2021-03-22T16:22:19Z
author_association: OWNER
body:
  I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157

  I figured out a similar pattern in `datasette-ripgrep` here: https://github.com/simonw/datasette-ripgrep/blob/0.7/datasette_ripgrep/__init__.py#L63-L71

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1271#issuecomment-804261103
issue_url: https://api.github.com/repos/simonw/datasette/issues/1271
node_id: MDEyOklzc3VlQ29tbWVudDgwNDI2MTEwMw==
created_at: 2021-03-22T17:39:57Z
updated_at: 2021-03-22T17:39:57Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=h1) Report
  > Merging [#1271](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=desc) (fb2ad7a) into [main](https://codecov.io/gh/simonw/datasette/commit/c4f1ec7f33fd7d5b93f0f895dafb5351cc3bfc5b?el=desc) (c4f1ec7) will **decrease** coverage by `0.28%`.
  > The diff coverage is `94.28%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1271/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1271      +/-   ##
  ==========================================
  - Coverage   91.51%   91.22%   -0.29%     
  ==========================================
    Files          34       34              
    Lines        4255     4263       +8     
  ==========================================
  - Hits         3894     3889       -5     
  - Misses        361      374      +13     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/database.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `92.41% <94.28%> (-0.52%)` | :arrow_down: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1271/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `92.24% <0.00%> (-1.90%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=footer). Last update [c4f1ec7...fb2ad7a](https://codecov.io/gh/simonw/datasette/pull/1271?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


