# issue_comments

html_url: https://github.com/simonw/datasette/issues/971#issuecomment-696298614
issue_url: https://api.github.com/repos/simonw/datasette/issues/971
node_id: MDEyOklzc3VlQ29tbWVudDY5NjI5ODYxNA==
created_at: 2020-09-21T18:43:07Z
updated_at: 2020-09-21T18:43:07Z
author_association: OWNER
body:
  Or, do this:
  ```sql
  SELECT 1 FROM dbstat limit 1;
  ```
  And see if it returns a "table does not exist" error.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/971#issuecomment-696302868
issue_url: https://api.github.com/repos/simonw/datasette/issues/971
node_id: MDEyOklzc3VlQ29tbWVudDY5NjMwMjg2OA==
created_at: 2020-09-21T18:50:40Z
updated_at: 2020-09-21T18:50:40Z
author_association: OWNER
body:
  Easiest way to get this may be to run `create view dbstat_view as select * from dbstat` on databases that support it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/971#issuecomment-696307922
issue_url: https://api.github.com/repos/simonw/datasette/issues/971
node_id: MDEyOklzc3VlQ29tbWVudDY5NjMwNzkyMg==
created_at: 2020-09-21T18:59:52Z
updated_at: 2020-09-21T19:00:02Z
author_association: OWNER
body:
  Given `dbstat` isn't as widely available as I thought I'm going to let people who want to use `dbstat` run their own `select * from dbstat` queries rather than bake support directly into Datasette.

  The experience of exploring `dbstat` will improve if I land support for running facets against arbitrary custom SQL queries, which is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646

  https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696421240
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQyMTI0MA==
created_at: 2020-09-21T22:53:48Z
updated_at: 2020-09-21T22:53:48Z
author_association: OWNER
body:
  I've decided to call this `table.transform()` - I was over-thinking whether people would remember that `.transform()` actually transforms the table, but that's what documentation is for.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423066
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQyMzA2Ng==
created_at: 2020-09-21T22:59:01Z
updated_at: 2020-09-21T22:59:01Z
author_association: OWNER
body:
  I'm rethinking the API design now. Maybe it could look like this:

  To change the type of the `author_id` column from `text` to `int`:
  ```python
  books.transform({"author_id": int})
  ```
  This would leave the existing columns alone, but would change the type of this column.

  To rename `author_id` to `author_identifier`:
  ```python
  books.transform(rename={"author_id": "author_identifier"})
  ```
  To drop a column:
  ```python
  books.transform(drop=["author_id"])
  ```
  Since the parameters all operate on columns they don't need to be called `drop_column` and `rename_column`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696430843
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzMDg0Mw==
created_at: 2020-09-21T23:21:00Z
updated_at: 2020-09-21T23:21:00Z
author_association: OWNER
body:
  For FTS tables associated with the table that is being transformed, should I automatically drop the old FTS table and recreate it against the new one or will it just magically continue to work after the table is renamed?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696431058
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzMTA1OA==
created_at: 2020-09-21T23:21:37Z
updated_at: 2020-09-21T23:21:37Z
author_association: OWNER
body:
  I may need to do something special for `rowid` tables to ensure that the `rowid` values in the transformed table match those from the old table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696432690
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzMjY5MA==
created_at: 2020-09-21T23:26:32Z
updated_at: 2020-09-21T23:27:38Z
author_association: OWNER
body:
  To expand on what that first argument - the `columns` argument - does. Say you have a table like this:
  ```
  id integer
  name text
  age text
  ```
  Any columns omitted from the `columns=` argument are left alone - they have to be explicitly dropped using `drop=` if you want to drop them.

  Any new columns are added (at the end of the table):

  ```
  table.tranform({"size": float})
  ```

  Any columns that have their type changed will have their type changed:

  ```
  table.tranform({"age": int})
  ```

  Should I also re-order columns if the order doesn't match? I think so. Open question as to what happens to columns that aren't mentioned at all in the dictionary though - what order should they go in?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433542
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzMzU0Mg==
created_at: 2020-09-21T23:28:58Z
updated_at: 2020-09-21T23:28:58Z
author_association: OWNER
body:
  If you want to both change the type of a column AND rename it in the same operation, how would you do that? I think like this:
  ```python
  table.transform({"age": int}, rename={"age": "dog_age"})
  ```
  So any rename logic is applied at the end, after the type transformation or re-ordering logic.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433778
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzMzc3OA==
created_at: 2020-09-21T23:29:39Z
updated_at: 2020-09-21T23:29:39Z
author_association: OWNER
body:
  The `columns=` argument is optional - so you can do just a rename operation like so:
  ```
  table.transform(rename={"age": "dog_age"})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434097
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzNDA5Nw==
created_at: 2020-09-21T23:30:40Z
updated_at: 2020-09-21T23:30:40Z
author_association: OWNER
body:
  Since I have a `column_order=None` argument already, maybe I can ignore the order of the columns in that first argument and use that instead?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434237
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzNDIzNw==
created_at: 2020-09-21T23:31:07Z
updated_at: 2020-09-21T23:31:57Z
author_association: OWNER
body:
  Does it make sense to support the `pk=` argument for changing the primary key?

  If the user requests a primary key that doesn't make sense I think an integrity error will be raised when the SQL is being executed, which should hopefully cancel the transaction and raise an error. Need to check that this is what happens.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434638
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzNDYzOA==
created_at: 2020-09-21T23:32:26Z
updated_at: 2020-09-21T23:32:26Z
author_association: OWNER
body:
  A test that confirms that this mechanism can turn a `rowid` into a non-rowid table would be good too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696435194
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQzNTE5NA==
created_at: 2020-09-21T23:34:14Z
updated_at: 2020-09-21T23:35:00Z
author_association: OWNER
body:
  I think the fiddliest part of the implementation here is code that takes the existing `columns_dict` of the table and the incoming `columns=` and `drop=` and `rename=` parameters and produces the columns dictionary for the new table, ready to be fed to `.create_table()`.

  This logic probably also needs to return a structure that can be used to build the `INSERT INTO ... SELECT ... FROM` query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ==
created_at: 2020-09-22T00:00:23Z
updated_at: 2020-09-22T00:00:23Z
author_association: OWNER
body:
  I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg==
created_at: 2020-09-22T00:01:50Z
updated_at: 2020-09-22T00:01:50Z
author_association: OWNER
body:
  When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA==
created_at: 2020-09-22T00:02:22Z
updated_at: 2020-09-22T00:02:22Z
author_association: OWNER
body:
  How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ==
created_at: 2020-09-22T00:04:31Z
updated_at: 2020-09-22T00:04:44Z
author_association: OWNER
body:
  Good news: the `.columns` introspection does tell me those things:
  ```
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database(memory=True)
  >>> db.create_table("foo", {"id": int, "name": str, "age": int}, defaults={"age": 1}, not_null={"name", "age"})
  <Table foo (id, name, age)>
  >>> db["foo"]
  <Table foo (id, name, age)>
  >>> print(db["foo"].schema)
  CREATE TABLE [foo] (
     [id] INTEGER,
     [name] TEXT NOT NULL,
     [age] INTEGER NOT NULL DEFAULT 1
  )
  >>> db["foo"].columns
  [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0),
   Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0),
   Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw==
created_at: 2020-09-22T00:06:12Z
updated_at: 2020-09-22T00:06:12Z
author_association: OWNER
body:
  I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg==
created_at: 2020-09-22T00:07:43Z
updated_at: 2020-09-22T00:09:05Z
author_association: OWNER
body:
  Syntax challenge: I could use `.transform(defaults={"age": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it?

  I could do this: `.transform(not_not_null={"age"})` - it's a bit gross but it's also kind of funny. I actually like it!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng==
created_at: 2020-09-22T00:10:50Z
updated_at: 2020-09-22T00:11:12Z
author_association: OWNER
body:
  A less horrible interface might be the following:
  ```python
  # Ensure the 'age' column is not null:
  table.transform(not_null={"age"})
  # The 'age' column is not null but I don't want it to be:
  table.transform(not_null={"age": False})
  ```
  So if the argument is a set it means "make sure these are all not null" - if the argument is a dictionary it means "set these to be null or not null depending on if their dictionary value is true or false".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA==
created_at: 2020-09-22T00:13:55Z
updated_at: 2020-09-22T00:14:21Z
author_association: OWNER
body:
  Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.:

  ```python
  table.transform({"age": int}, conversions={"name": "upper(?)"})
  ```

  https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/162
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ==
created_at: 2020-09-22T00:22:46Z
updated_at: 2020-09-22T00:22:46Z
author_association: OWNER
body:
  Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/162
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA==
created_at: 2020-09-22T00:40:44Z
updated_at: 2020-09-22T00:40:44Z
author_association: OWNER
body:
  Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/114
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ==
created_at: 2020-09-22T00:42:35Z
updated_at: 2020-09-22T00:42:35Z
author_association: OWNER
body:
  The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696473559
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/164
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ3MzU1OQ==
created_at: 2020-09-22T02:10:37Z
updated_at: 2020-09-22T02:10:37Z
author_association: OWNER
body:
  Maybe something like this:

      sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ==
created_at: 2020-09-22T02:45:47Z
updated_at: 2020-09-22T02:45:47Z
author_association: OWNER
body:
  I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this:

      INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs;

  So passing `conversions={"name": "upper(?)"})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ==
created_at: 2020-09-22T03:10:15Z
updated_at: 2020-09-22T03:10:15Z
author_association: OWNER
body:
  Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key?

  Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this:

  ```python
  table.transform(drop_foreign_keys=[("author_id", "author", "id")])
  ```
  It's a bit crufty but it's such a rare use-case that I think this will be good enough.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/161
node_id: MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA==
created_at: 2020-09-22T03:48:58Z
updated_at: 2020-09-22T03:48:58Z
author_association: OWNER
body:
  One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be:

  >  If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. 

  And the last steps should be:

  > If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints.
  >
  > Commit the transaction started in step 2.
  >
  > If foreign keys constraints were originally enabled, reenable them now. 

  I need to implement that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696500922
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/164
node_id: MDEyOklzc3VlQ29tbWVudDY5NjUwMDkyMg==
created_at: 2020-09-22T04:22:40Z
updated_at: 2020-09-22T04:22:40Z
author_association: OWNER
body:
  Documentation for the `.transform()` method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696520928
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/164
node_id: MDEyOklzc3VlQ29tbWVudDY5NjUyMDkyOA==
created_at: 2020-09-22T05:50:17Z
updated_at: 2020-09-22T05:50:17Z
author_association: OWNER
body:
  Idea for CLI options:
  ```
  --type age integer
  --drop colname
  --rename oldname newname
  --not-null col
  --not-null-false col
  --pk new_id
  --pk-none
  --default col value
  --default-none column
  --drop-foreign-key col other_table other_column
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/167#issuecomment-696565981
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/167
node_id: MDEyOklzc3VlQ29tbWVudDY5NjU2NTk4MQ==
created_at: 2020-09-22T07:53:13Z
updated_at: 2020-09-22T07:53:13Z
author_association: OWNER
body:
  Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the `PRAGMA foreign_keys` bits before and after the transaction, not during.

  Right now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/168#issuecomment-696573944
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/168
node_id: MDEyOklzc3VlQ29tbWVudDY5NjU3Mzk0NA==
created_at: 2020-09-22T08:11:30Z
updated_at: 2020-09-22T08:11:30Z
author_association: OWNER
body:
  Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-696769501
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc2OTUwMQ==
created_at: 2020-09-22T14:45:49Z
updated_at: 2020-09-22T14:45:49Z
author_association: OWNER
body:
  I put together a minimal prototype of this and it feels pretty good:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 20aae7d..fb3bdad 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -4,6 +4,7 @@ import collections
   import datetime
   import glob
   import hashlib
  +import httpx
   import inspect
   import itertools
   from itsdangerous import BadSignature
  @@ -312,6 +313,7 @@ class Datasette:
           self._register_renderers()
           self._permission_checks = collections.deque(maxlen=200)
           self._root_token = secrets.token_hex(32)
  +        self.client = DatasetteClient(self)
   
       async def invoke_startup(self):
           for hook in pm.hook.startup(datasette=self):
  @@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath):
   
   class NotFoundExplicit(NotFound):
       pass
  +
  +
  +class DatasetteClient:
  +    def __init__(self, ds):
  +        self.app = ds.app()
  +
  +    def _fix(self, path):
  +        if path.startswith("/"):
  +            path = "http://localhost{}".format(path)
  +        return path
  +
  +    async def get(self, path, **kwargs):
  +        async with httpx.AsyncClient(app=self.app) as client:
  +            return await client.get(self._fix(path), **kwargs)
  +
  +    async def post(self, path, **kwargs):
  +        async with httpx.AsyncClient(app=self.app) as client:
  +            return await client.post(self._fix(path), **kwargs)
  +
  +    async def options(self, path, **kwargs):
  +        async with httpx.AsyncClient(app=self.app) as client:
  +            return await client.options(self._fix(path), **kwargs)
  ```
  Used like this in `ipython`:
  ```
  In [1]: from datasette.app import Datasette

  In [2]: ds = Datasette(["fixtures.db"])

  In [3]: (await ds.client.get("/-/config.json")).json()
  Out[3]: 
  {'default_page_size': 100,
   'max_returned_rows': 1000,
   'num_sql_threads': 3,
   'sql_time_limit_ms': 1000,
   'default_facet_size': 30,
   'facet_time_limit_ms': 200,
   'facet_suggest_time_limit_ms': 50,
   'hash_urls': False,
   'allow_facet': True,
   'allow_download': True,
   'suggest_facets': True,
   'default_cache_ttl': 5,
   'default_cache_ttl_hashed': 31536000,
   'cache_size_kb': 0,
   'allow_csv_stream': True,
   'max_csv_mb': 100,
   'truncate_cells_html': 2048,
   'force_https_urls': False,
   'template_debug': False,
   'base_url': '/'}

  In [4]: (await ds.client.get("/fixtures/facetable.json?_shape=array")).json()
  Out[4]: 
  [{'pk': 1,
    'created': '2019-01-14 08:00:00',
    'planet_int': 1,
    'on_earth': 1,
    'state': 'CA',
    'city_id': 1,
    'neighborhood': 'Mission',
    'tags': '["tag1", "tag2"]',
    'complex_array': '[{"foo": "bar"}]',
    'distinct_some_null': 'one'},
   {'pk': 2,
    'created': '2019-01-14 08:00:00',
    'planet_int': 1,
    'on_earth': 1,
    'state': 'CA',
    'city_id': 1,
    'neighborhood': 'Dogpatch',
    'tags': '["tag1", "tag3"]',
    'complex_array': '[]',
    'distinct_some_null': 'two'},
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-696769853
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc2OTg1Mw==
created_at: 2020-09-22T14:46:21Z
updated_at: 2020-09-22T14:46:21Z
author_association: OWNER
body:
  This adds `httpx` as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-696774711
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc3NDcxMQ==
created_at: 2020-09-22T14:53:56Z
updated_at: 2020-09-22T14:53:56Z
author_association: OWNER
body:
  How important is it to use `httpx.AsyncClient` with a context manager?

  https://www.python-httpx.org/async/#opening-and-closing-clients says:

  > Alternatively, use `await client.aclose()` if you want to close a client explicitly:
  > 
  > ```
  > client = httpx.AsyncClient()
  > ...
  > await client.aclose()
  > ```
  The `.aclose()` method has a comment saying "Close transport and proxies" - I'm not using proxies, so the relevant implementation seems to be a call to `await self._transport.aclose()` in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751

  The transport I am using is a class called `ASGITransport` in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py

  The `aclose()` method on that class does nothing. So it looks like I can instantiate a client without bothering with the `async with httpx.AsyncClient` bit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-696775516
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc3NTUxNg==
created_at: 2020-09-22T14:55:10Z
updated_at: 2020-09-22T14:55:10Z
author_association: OWNER
body:
  Even smaller `DatasetteClient` implementation:
  ```python
  class DatasetteClient:
      def __init__(self, ds):
          self._client = httpx.AsyncClient(app=ds.app())

      def _fix(self, path):
          if path.startswith("/"):
              path = "http://localhost{}".format(path)
          return path

      async def get(self, path, **kwargs):
          return await self._client.get(self._fix(path), **kwargs)

      async def post(self, path, **kwargs):
          return await self._client.post(self._fix(path), **kwargs)

      async def options(self, path, **kwargs):
          return await self._client.options(self._fix(path), **kwargs)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-696776828
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc3NjgyOA==
created_at: 2020-09-22T14:57:13Z
updated_at: 2020-09-22T14:57:13Z
author_association: OWNER
body:
  I may as well implement all of the HTTP methods supported by the `httpx` client:

  - get
  - options
  - head
  - post
  - put
  - patch
  - delete

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-696777886
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc3Nzg4Ng==
created_at: 2020-09-22T14:58:54Z
updated_at: 2020-09-22T14:58:54Z
author_association: OWNER
body:
  ```python
  class DatasetteClient:
      def __init__(self, ds):
          self._client = httpx.AsyncClient(app=ds.app())

      def _fix(self, path):
          if path.startswith("/"):
              path = "http://localhost{}".format(path)
          return path

      async def get(self, path, **kwargs):
          return await self._client.get(self._fix(path), **kwargs)

      async def options(self, path, **kwargs):
          return await self._client.options(self._fix(path), **kwargs)

      async def head(self, path, **kwargs):
          return await self._client.head(self._fix(path), **kwargs)

      async def post(self, path, **kwargs):
          return await self._client.post(self._fix(path), **kwargs)

      async def put(self, path, **kwargs):
          return await self._client.put(self._fix(path), **kwargs)

      async def patch(self, path, **kwargs):
          return await self._client.patch(self._fix(path), **kwargs)

      async def delete(self, path, **kwargs):
          return await self._client.delete(self._fix(path), **kwargs)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/943#issuecomment-696778735
issue_url: https://api.github.com/repos/simonw/datasette/issues/943
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc3ODczNQ==
created_at: 2020-09-22T15:00:13Z
updated_at: 2020-09-22T15:00:39Z
author_association: OWNER
body:
  Am I going to rewrite ALL of my tests to use this instead? It would clean up a lot of test code, at the cost of quite a bit of work.

  It would make for much neater plugin tests too, and neater testing documentation: https://docs.datasette.io/en/stable/testing_plugins.html

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/969#issuecomment-696788109
issue_url: https://api.github.com/repos/simonw/datasette/issues/969
node_id: MDEyOklzc3VlQ29tbWVudDY5Njc4ODEwOQ==
created_at: 2020-09-22T15:15:14Z
updated_at: 2020-09-22T15:15:14Z
author_association: OWNER
body:
  I don't think a standard "pass these extra arguments to the publish tool" mechanism will work because there's no guarantee that a publisher uses a CLI tool - or if it does, it might make several calls to different CLI tools. The Cloud Run one runs a couple of commands, as illustrated by this test:

  https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/tests/test_publish_cloudrun.py#L63-L73

  Adding a `--tar` option for `datasette publish heroku` is a good fix for this though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/973#issuecomment-696800410
issue_url: https://api.github.com/repos/simonw/datasette/issues/973
node_id: MDEyOklzc3VlQ29tbWVudDY5NjgwMDQxMA==
created_at: 2020-09-22T15:35:28Z
updated_at: 2020-09-22T15:35:28Z
author_association: OWNER
body:
  Confirmed in local dev:
  ```
  % datasette fixtures.db --inspect-file inspect.json
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 11, in <module>
      load_entry_point('datasette', 'console_scripts', 'datasette')()
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 829, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 782, in main
      rv = self.invoke(ctx)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py", line 610, in invoke
      return callback(*args, **kwargs)
    File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 406, in serve
      inspect_data = json.load(open(inspect_file))
  TypeError: 'bool' object is not callable
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==
created_at: 2020-09-22T18:14:33Z
updated_at: 2020-09-22T18:14:45Z
author_association: OWNER
body:
  Thinking more about this one:
  ```
  $ sqlite-utils extract my.db \
      dea_sales company_name company_address \
      --table companies
  ```
  The goal here is to pull the company name and address pair out into a separate table.

  Some questions:
  - should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this.
  - what should the foreign key column that gets added to the `companies` table be called?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==
created_at: 2020-09-22T18:15:33Z
updated_at: 2020-09-22T18:15:33Z
author_association: OWNER
body:
  I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==
created_at: 2020-09-22T20:57:57Z
updated_at: 2020-09-22T20:57:57Z
author_association: OWNER
body:
  I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID.

  That integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==
created_at: 2020-09-22T21:02:24Z
updated_at: 2020-09-22T21:02:24Z
author_association: OWNER
body:
  In Python it looks like this:
  ```python
  # Simple case - species column species_id pointing to species table
  db["trees"].extract("species")

  # Setting a custom table
  db["trees"].extract("species", table="Species")

  # Custom foreign key column on trees
  db["trees"].extract("species", fk_column="species")

  # Extracting multiple columns
  db["trees"].extract(["common_name", "latin_name"])
  # (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id)

  # Or with explicit table (fk_column here defaults to species_id because of the table name)
  db["trees"].extract(["common_name", "latin_name"], table="species")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==
created_at: 2020-09-22T21:03:11Z
updated_at: 2020-09-22T21:03:11Z
author_association: OWNER
body:
  And if you want to rename some of the columns in the new table:
  ```python
  db["trees"].extract(["common_name", "latin_name"], table="species", rename={"common_name": "name"})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==
created_at: 2020-09-22T21:04:45Z
updated_at: 2020-09-22T21:04:45Z
author_association: OWNER
body:
  `table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==
created_at: 2020-09-22T21:17:34Z
updated_at: 2020-09-22T21:17:34Z
author_association: OWNER
body:
  What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==
created_at: 2020-09-22T22:18:13Z
updated_at: 2020-09-22T22:18:13Z
author_association: OWNER
body:
  Here's how I'm generating the examples for the documentation:
  ```
  In [2]: import sqlite_utils

  In [3]: db = sqlite_utils.Database(memory=True)

  In [4]: db["Trees"].insert({"id": 1, "TreeAddress": "52 Vine St", "CommonName":
     ...: "Palm", "LatinName": "foo"}, pk="id")
  Out[4]: <Table Trees (id, TreeAddress, CommonName, LatinName)>

  In [5]: db["Trees"].extract(["CommonName", "LatinName"], table="Species", fk_col
     ...: umn="species_id")

  In [6]: print(db["Trees"].schema)
  CREATE TABLE "Trees" (
     [id] INTEGER PRIMARY KEY,
     [TreeAddress] TEXT,
     [species_id] INTEGER,
     FOREIGN KEY(species_id) REFERENCES Species(id)
  )

  In [7]: print(db["Species"].schema)
  CREATE TABLE [Species] (
     [id] INTEGER PRIMARY KEY,
     [CommonName] TEXT,
     [LatinName] TEXT
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ==
created_at: 2020-09-22T22:22:49Z
updated_at: 2020-09-22T22:22:49Z
author_association: OWNER
body:
  The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA==
created_at: 2020-09-22T22:40:00Z
updated_at: 2020-09-22T22:40:00Z
author_association: OWNER
body:
  I tried out the prototype of the CLI on the Global Power Plants data:
  ```
  wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
  sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
  sqlite-utils extract global.db power_plants country country_long
  ```
  This threw an error because `rowid` columns are not yet supported. I fixed that like so:
  ```
  sqlite-utils transform global.db power_plants --rename rowid id
  sqlite-utils extract global.db power_plants country country_long
  ```
  That worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`.

  Based on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw==
created_at: 2020-09-22T22:57:53Z
updated_at: 2020-09-22T22:57:53Z
author_association: OWNER
body:
  The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA==
created_at: 2020-09-22T23:16:00Z
updated_at: 2020-09-22T23:16:00Z
author_association: OWNER
body:
  Trying this demo again:
  ```
  wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv'
  sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv
  sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name
  ```
  It worked!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/42
node_id: MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA==
created_at: 2020-09-22T23:39:31Z
updated_at: 2020-09-22T23:39:31Z
author_association: OWNER
body:
  Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/170#issuecomment-697047591
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/170
node_id: MDEyOklzc3VlQ29tbWVudDY5NzA0NzU5MQ==
created_at: 2020-09-23T00:14:52Z
updated_at: 2020-09-23T00:14:52Z
author_association: OWNER
body:
  @simonw
  @db.register_function decorator, closes #162
  4824775
  @simonw
  table.transform() method - closes #114
  987dd12
  @simonw
  Keyword only arguments for transform()
  f8e10df

  Also renamed columns= to types=

  Closes #165

  Commits on Sep 22, 2020
  @simonw
  Implemented sqlite-utils transform command, closes #164
  752d261
  @simonw
  Applied Black
  f29f682
  @simonw
  table.extract() method, refs #42
  f855379
  @simonw
  Docstring for sqlite-utils transform
  c755f28
  @simonw
  Added table.extract(rename=) option, refs #42
  c3210f2
  @simonw
  Applied Black
  317071a
  @simonw
  New .rows_where(select=) argument
  7178231
  @simonw
  table.extract() now works with rowid tables, refs #42
  2db6c5b
  @simonw
  sqlite-utils extract, closes #42
  55cf928
  @simonw
  Progress bar for "sqlite-utils extract", closes #169
  5c4d58d
  @simonw
  Fixed PRAGMA foreign_keys handling for .transform, closes #167 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697466497
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5NzQ2NjQ5Nw==
created_at: 2020-09-23T14:41:17Z
updated_at: 2020-09-23T14:41:17Z
author_association: OWNER
body:
  Steps to produce that database:
  ```
  curl -o salaries.csv 'https://data.sfgov.org/api/views/88g8-5mnd/rows.csv?accessType=DOWNLOAD'
  sqlite-utils insert salaries.db salaries salaries.csv --csv
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697467833
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5NzQ2NzgzMw==
created_at: 2020-09-23T14:42:03Z
updated_at: 2020-09-23T14:42:03Z
author_association: OWNER
body:
  Here's the loop that's taking the time: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L892-L897

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697473247
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5NzQ3MzI0Nw==
created_at: 2020-09-23T14:45:13Z
updated_at: 2020-09-23T14:45:13Z
author_association: OWNER
body:
  `lookup_table.lookup(lookups)` is doing a SQL lookup. This could be cached in-memory, maybe with a LRU cache, to avoid looking up the primary key for records that we have recently used.

  The `.update()` method it is calling first does a `get()` and then does a SQL `UPDATE ... WHERE`:

  https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L1244-L1264

  Batching those updates may have an effect. Or finding a way to skip the `.get()` since we already know we have a valid record.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697835956
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5NzgzNTk1Ng==
created_at: 2020-09-23T18:22:49Z
updated_at: 2020-09-23T18:22:49Z
author_association: OWNER
body:
  I ran `sudo py-spy top -p 123` against the process while it was running and the most time is definitely spent in `.update()`:
  ```
  Total Samples 1000
  GIL: 0.00%, Active: 90.00%, Threads: 1

    %Own   %Total  OwnTime  TotalTime  Function (filename:line)                                                                                                                                  
   38.00%  38.00%    3.85s     3.85s   update (sqlite_utils/db.py:1283)
   27.00%  27.00%    2.12s     2.12s   execute (sqlite_utils/db.py:161)
   10.00%  10.00%   0.890s    0.890s   execute (sqlite_utils/db.py:163)
   10.00%  17.00%   0.870s     1.54s   columns (sqlite_utils/db.py:553)
    0.00%   0.00%   0.110s    0.210s   <listcomp> (sqlite_utils/db.py:554)
    0.00%   3.00%   0.100s    0.320s   table_names (sqlite_utils/db.py:191)
    0.00%   0.00%   0.100s    0.100s   __new__ (<string>:1)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697859772
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5Nzg1OTc3Mg==
created_at: 2020-09-23T18:38:43Z
updated_at: 2020-09-23T18:38:52Z
author_association: OWNER
body:
  I wonder if I could make this faster by separating it out into a few steps:
  - Create the new lookup table with all of the distinct rows
  - Add the blank foreign key column
  - run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`
  - Drop the value columns

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697863116
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5Nzg2MzExNg==
created_at: 2020-09-23T18:41:06Z
updated_at: 2020-09-23T18:41:06Z
author_association: OWNER
body:
  Problem with this approach is it's not compatible with progress bars - but if it's a multiple of times faster it's worth it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697869886
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5Nzg2OTg4Ng==
created_at: 2020-09-23T18:45:30Z
updated_at: 2020-09-23T18:45:30Z
author_association: OWNER
body:
  There's something to be said for making this operation pausable and resumable, especially if I'm going to make it available in a Datasette plugin at some point.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-697973420
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDY5Nzk3MzQyMA==
created_at: 2020-09-23T21:07:58Z
updated_at: 2020-09-23T21:07:58Z
author_association: NONE
body:
  I've just run into this after crafting a complex query and discovered that hitting back loses my query.

  Even showing me the whole bad query would be a huge improvement over the current status quo.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-697995303
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDY5Nzk5NTMwMw==
created_at: 2020-09-23T22:01:08Z
updated_at: 2020-09-23T22:01:08Z
author_association: OWNER
body:
  This is a little tricky to solve, because of the location of the form and the need to return JSON as well as HTML. It would be weird if a JSON request came in and got back the standard output from https://latest.datasette.io/fixtures.json when they were expecting to get back JSON in the shape of https://latest.datasette.io/fixtures.json?sql=select%20*%20from%20sqlite_master

  I'm going to return the HTML view that you would get for 0 results for a query - https://latest.datasette.io/fixtures?sql=select%201%20limit%200 - but with an error message.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-697995885
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDY5Nzk5NTg4NQ==
created_at: 2020-09-23T22:02:44Z
updated_at: 2020-09-23T22:08:28Z
author_association: OWNER
body:
  So the JSON (still served with a 500 code) will look something like this:
  ```json
  {
    "ok": false,
    "status": 500,
    "database": "fixtures",
    "query_name": null,
    "rows": [],
    "truncated": false,
    "error": "Error message goes here",
    "columns": [],
    "query": {
      "sql": "the query that broke goes here",
      "params": {}
    },
    "private": false,
    "allow_execute_sql": true,
    "query_ms": 0.8716583251953125,
    "source": "tests/fixtures.py",
    "source_url": "https://github.com/simonw/datasette/blob/master/tests/fixtures.py",
    "license": "Apache License 2.0",
    "license_url": "https://github.com/simonw/datasette/blob/master/LICENSE"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/619#issuecomment-698024773
issue_url: https://api.github.com/repos/simonw/datasette/issues/619
node_id: MDEyOklzc3VlQ29tbWVudDY5ODAyNDc3Mw==
created_at: 2020-09-23T23:31:46Z
updated_at: 2020-09-23T23:31:46Z
author_association: OWNER
body:
  I'm going to have to untangle Datasette's error handling a bit for this - currently the expectation is that exceptions will be handled at a higher level, but I need to rethink that to make it cleaner for views like the "execute custom SQL" view to add their own error handling (and still be able to return the correct HTTP status codes, even with custom pages).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/123#issuecomment-698110186
issue_url: https://api.github.com/repos/simonw/datasette/issues/123
node_id: MDEyOklzc3VlQ29tbWVudDY5ODExMDE4Ng==
created_at: 2020-09-24T04:49:51Z
updated_at: 2020-09-24T04:49:51Z
author_association: NONE
body:
  As a half-measure, I'd get value out of being able to upload a CSV and have datasette run csv-to-sqlite on it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/974#issuecomment-698110492
issue_url: https://api.github.com/repos/simonw/datasette/issues/974
node_id: MDEyOklzc3VlQ29tbWVudDY5ODExMDQ5Mg==
created_at: 2020-09-24T04:50:56Z
updated_at: 2020-09-24T04:51:05Z
author_association: OWNER
body:
  Come to think of it I've noticed that in the logs when it's running on my laptop, definitely worth fixing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/123#issuecomment-698174957
issue_url: https://api.github.com/repos/simonw/datasette/issues/123
node_id: MDEyOklzc3VlQ29tbWVudDY5ODE3NDk1Nw==
created_at: 2020-09-24T07:42:05Z
updated_at: 2020-09-24T07:42:05Z
author_association: NONE
body:
  Oh. Awesome. 

  On Thu, Sep 24, 2020 at 12:28:53AM -0700, Simon Willison wrote:
  > @obra there's a plugin for that! https://github.com/simonw/
  > datasette-upload-csvs
  > 
  > â
  > You are receiving this because you were mentioned.
  > Reply to this email directly, view it on GitHub, or unsubscribe.*
  > 

  -- 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/172#issuecomment-698178101
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/172
node_id: MDEyOklzc3VlQ29tbWVudDY5ODE3ODEwMQ==
created_at: 2020-09-24T07:48:57Z
updated_at: 2020-09-24T07:49:20Z
author_association: OWNER
body:
  > I wonder if I could make this faster by separating it out into a few steps:
  > 
  >     * Create the new lookup table with all of the distinct rows
  > 
  >     * Add the blank foreign key column
  > 
  >     * run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)`
  > 
  >     * Drop the value columns
  My prototype of this knocked the time down from 10 minutes to 4 seconds, so I think the change is worth it!
  ```
  % date
  sqlite-utils extract salaries.db salaries \
     'Department Code' 'Department' \
    --table 'departments' \
    --fk-column 'department_id' \
    --rename 'Department Code' code \
    --rename 'Department' name
  date
  sqlite-utils extract salaries.db salaries \
     'Union Code' 'Union' \
    --table 'unions' \
    --fk-column 'union_id' \
    --rename 'Union Code' code \
    --rename 'Union' name
  date
  sqlite-utils extract salaries.db salaries \
     'Job Family Code' 'Job Family' \
    --table 'job_families' \
    --fk-column 'job_family_id' \
    --rename 'Job Family Code' code \
    --rename 'Job Family' name
  date
  sqlite-utils extract salaries.db salaries \
     'Job Code' 'Job' \
    --table 'jobs' \
    --fk-column 'job_id' \
    --rename 'Job Code' code \
    --rename 'Job' name
  date
  Thu Sep 24 00:48:16 PDT 2020

  Thu Sep 24 00:48:20 PDT 2020

  Thu Sep 24 00:48:24 PDT 2020

  Thu Sep 24 00:48:28 PDT 2020

  Thu Sep 24 00:48:32 PDT 2020
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180113
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/174
node_id: MDEyOklzc3VlQ29tbWVudDY5ODE4MDExMw==
created_at: 2020-09-24T07:53:03Z
updated_at: 2020-09-24T07:53:03Z
author_association: OWNER
body:
  This could do with a little bit more testing - I'm worried there may be column or table name edge cases that are not covered yet. I also need to remove the progress bar code since that no longer makes sense for this implementation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180705
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/174
node_id: MDEyOklzc3VlQ29tbWVudDY5ODE4MDcwNQ==
created_at: 2020-09-24T07:54:10Z
updated_at: 2020-09-24T07:54:10Z
author_association: OWNER
body:
  After running through the steps in https://simonwillison.net/2020/Sep/23/sqlite-utils-extract/ I get a table that looks like this:

  <img width="1699" alt="salaries__salaries__683_558_rows" src="https://user-images.githubusercontent.com/9599/94116875-666b8900-fe00-11ea-9e97-2b9ccbfeae29.png">

  The foreign key columns are all at the end of the table. It would be nicer if they were arranged in the same order as the columns they replaced.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182037
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/174
node_id: MDEyOklzc3VlQ29tbWVudDY5ODE4MjAzNw==
created_at: 2020-09-24T07:56:50Z
updated_at: 2020-09-24T07:56:50Z
author_association: OWNER
body:
  I could also be a bit smarter about transaction handling. I think it may be possible to run this entire operation in a single transaction now.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182656
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/174
node_id: MDEyOklzc3VlQ29tbWVudDY5ODE4MjY1Ng==
created_at: 2020-09-24T07:58:08Z
updated_at: 2020-09-24T07:58:08Z
author_association: OWNER
body:
  The way the lookup table works here differs from the previous implementation. In the previous implementation the usage of `.lookup()` meant that an existing table would be modified to fit the new purpose. That no longer happens in this version. Need to make a design decision about how this should work.

  It should definitely be possible to use an existing lookup table - imagine a database where several tables have a "Departments" column and we want to extract all of those values out to a single shared "Departments" table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698184166
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/174
node_id: MDEyOklzc3VlQ29tbWVudDY5ODE4NDE2Ng==
created_at: 2020-09-24T08:01:07Z
updated_at: 2020-09-24T08:01:07Z
author_association: OWNER
body:
  I may revert the now unnecessary undocumented tweaks to the `.update()` method made in 66d506587eba9f0715267d6560b97c1fa44cc781 as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698400790
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/174
node_id: MDEyOklzc3VlQ29tbWVudDY5ODQwMDc5MA==
created_at: 2020-09-24T14:59:50Z
updated_at: 2020-09-24T14:59:50Z
author_association: OWNER
body:
  For reusing the lookup table: I'm going to raise an error if a lookup table exists but without the correct columns. The caller can then add those columns and try again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/970#issuecomment-698434236
issue_url: https://api.github.com/repos/simonw/datasette/issues/970
node_id: MDEyOklzc3VlQ29tbWVudDY5ODQzNDIzNg==
created_at: 2020-09-24T15:56:18Z
updated_at: 2020-09-24T15:56:50Z
author_association: OWNER
body:
  Idea: if a database only has a single table, this could open straight to `/db/table`. If it has multiple tables but a single database it could open straight to `/db`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/177#issuecomment-698444567
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/177
node_id: MDEyOklzc3VlQ29tbWVudDY5ODQ0NDU2Nw==
created_at: 2020-09-24T16:14:47Z
updated_at: 2020-09-24T16:14:47Z
author_association: OWNER
body:
  This is a backwards incompatible change, so technically I should bump the major version to 3. I'm not going to do that, because the feature is brand new and the chance that anyone has written code or shell scripts that use it is vanishingly small.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/976#issuecomment-698488971
issue_url: https://api.github.com/repos/simonw/datasette/issues/976
node_id: MDEyOklzc3VlQ29tbWVudDY5ODQ4ODk3MQ==
created_at: 2020-09-24T17:42:09Z
updated_at: 2020-09-24T17:42:35Z
author_association: OWNER
body:
  This is complex enough new logic that it will need test coverage - specifically covering tables or databases with strange names.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698577508
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/173
node_id: MDEyOklzc3VlQ29tbWVudDY5ODU3NzUwOA==
created_at: 2020-09-24T20:41:18Z
updated_at: 2020-09-24T20:41:18Z
author_association: OWNER
body:
  I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen.

  Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698578959
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/173
node_id: MDEyOklzc3VlQ29tbWVudDY5ODU3ODk1OQ==
created_at: 2020-09-24T20:44:35Z
updated_at: 2020-09-24T20:50:19Z
author_association: OWNER
body:
  I'm using a `click.File()` at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496

  I'll need to change that to be something that I can easily measure progress through. Also I should change its name - `json_file` is a bad name when it sometimes handles `csv` or `tsv` instead.

  It looks like the argument provided by `click.File` doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698579389
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/173
node_id: MDEyOklzc3VlQ29tbWVudDY5ODU3OTM4OQ==
created_at: 2020-09-24T20:45:29Z
updated_at: 2020-09-24T20:45:29Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560

  Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699524671
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/179
node_id: MDEyOklzc3VlQ29tbWVudDY5OTUyNDY3MQ==
created_at: 2020-09-26T17:31:23Z
updated_at: 2020-09-27T20:31:50Z
author_association: OWNER
body:
  SQL query for detecting integers:
  ```sql
  select
    'contains_non_integer' as result
  from
    mytable
  where
    cast(cast(mycolumn AS INTEGER) AS TEXT) != mycolumn
  limit
    1
  ```
  This will return a single row with a 1 as soon as it comes across a column that contains a non-integer - so it short circuits quickly on TEXT columns with non-integers in them.

  If everything in the column is an integer it will scan the whole thing before returning no rows.

  More extensive demo:
  ```sql
  select
    value,
    cast(cast(value AS INTEGER) AS TEXT) = value as is_valid_int
  from
    (
      select
        '1' as value
      union
      select
        '1.1' as value
      union
      select
        'dog' as value
      union
      select
        null as value
    )
  ```
  https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+INTEGER%29+AS+TEXT%29+%3D+value+as+is_valid_int%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29


  value | is_valid_int
  -- | --
    |  
  1 | 1
  1.1 | 0
  dog | 0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699684535
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/179
node_id: MDEyOklzc3VlQ29tbWVudDY5OTY4NDUzNQ==
created_at: 2020-09-27T20:30:31Z
updated_at: 2020-09-27T20:30:31Z
author_association: OWNER
body:
  This recipe looks like it might be the way to detect floats:
  ```sql
  select
    value,
    cast(cast(value AS REAL) AS TEXT) in (value, value || '.0') as is_valid_float
  from
    (
      select
        '1' as value
      union
      select
        '1.1' as value
      union
      select
        'dog' as value
      union
      select
        null as value
    )
  ```
  Demo: https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+REAL%29+AS+TEXT%29+in+%28value%2C+value+%7C%7C+%27.0%27%29+as+is_valid_float%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29


  value | is_valid_float
  -- | --
    |  
  1 | 1
  1.1 | 1
  dog | 0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/858#issuecomment-699690034
issue_url: https://api.github.com/repos/simonw/datasette/issues/858
node_id: MDEyOklzc3VlQ29tbWVudDY5OTY5MDAzNA==
created_at: 2020-09-27T21:23:04Z
updated_at: 2020-09-27T21:23:04Z
author_association: NONE
body:
  Hi Simon,

  Thanks so much for all your work on datasette, it's an excellent project and I wish you all the best with it. I particularly enjoyed your talk at the Django London Meetup a short while back. 

  I've been trying to publish to Heroku from Windows 10 and I was running into this error. I'm not sure why it can't be run without `shell=True` on Windows but this seems to help. With this change, I am able to publish if I pass in a `name` to the `publish` command. When a `name` is not passed the default of `datasette` is used and therefore this line here fails (as datasette at heroku already exists) and causes the recession error mentioned above.

  https://github.com/simonw/datasette/blob/9a6d0dce282e7fb58c5610e24c74098c923abfdc/datasette/publish/heroku.py#L126

  I tried to write a patch for this but I am really struggling with being on Windows (many of the tests seem to fail anyway?), and my lack of knowledge of Mock, so sorry for this. Hope this is of some help. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/181#issuecomment-699762881
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/181
node_id: MDEyOklzc3VlQ29tbWVudDY5OTc2Mjg4MQ==
created_at: 2020-09-28T04:29:23Z
updated_at: 2020-09-28T04:29:23Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/sqlite-utils/blob/94fc62857ee2655a21d85f6dae84b67bbfa5956d/sqlite_utils/db.py#L331-L367

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/977#issuecomment-700012161
issue_url: https://api.github.com/repos/simonw/datasette/issues/977
node_id: MDEyOklzc3VlQ29tbWVudDcwMDAxMjE2MQ==
created_at: 2020-09-28T13:37:44Z
updated_at: 2020-09-28T13:37:44Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=h1) Report
  > Merging [#977](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/9a6d0dce282e7fb58c5610e24c74098c923abfdc?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/977/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #977   +/-   ##
  =======================================
    Coverage   84.27%   84.27%           
  =======================================
    Files          28       28           
    Lines        3847     3847           
  =======================================
    Hits         3242     3242           
    Misses        605      605           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=footer). Last update [9a6d0dc...5c01344](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/978#issuecomment-700313836
issue_url: https://api.github.com/repos/simonw/datasette/issues/978
node_id: MDEyOklzc3VlQ29tbWVudDcwMDMxMzgzNg==
created_at: 2020-09-28T22:19:05Z
updated_at: 2020-09-28T22:19:05Z
author_association: OWNER
body:
  Looks like a bug in this function: https://github.com/simonw/datasette/blob/1f021c37110fc9019b0ef70062c28c335e568ae2/datasette/utils/__init__.py#L269-L274

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/978#issuecomment-700314509
issue_url: https://api.github.com/repos/simonw/datasette/issues/978
node_id: MDEyOklzc3VlQ29tbWVudDcwMDMxNDUwOQ==
created_at: 2020-09-28T22:20:51Z
updated_at: 2020-09-28T22:20:51Z
author_association: OWNER
body:
  Here's the HTML for the broken example above:
  ```html
  <style>
  @media only screen and (max-width: 576px) {

      .rows-and-columns td:nth-of-type(1):before { content: "dateutil_parse(\2210 october 2020 3pm\22)"; }

      .rows-and-columns td:nth-of-type(2):before { content: "dateutil_easter(\222020\22)"; }

      .rows-and-columns td:nth-of-type(3):before { content: "dateutil_parse_fuzzy(\22This is due 10 september\22)"; }

      .rows-and-columns td:nth-of-type(4):before { content: "dateutil_parse(\221/2/2020\22)"; }

      .rows-and-columns td:nth-of-type(5):before { content: "dateutil_parse(\222020-03-04\22)"; }

      .rows-and-columns td:nth-of-type(6):before { content: "dateutil_parse_dayfirst(\222020-03-04\22)"; }

      .rows-and-columns td:nth-of-type(7):before { content: "dateutil_easter(2020)"; }

  }
  </style>
  ```
  The glitch affects the ones where the quote is followed by digits.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/978#issuecomment-700319656
issue_url: https://api.github.com/repos/simonw/datasette/issues/978
node_id: MDEyOklzc3VlQ29tbWVudDcwMDMxOTY1Ng==
created_at: 2020-09-28T22:36:44Z
updated_at: 2020-09-28T22:36:44Z
author_association: OWNER
body:
  Weirdly even those leading 0s doesn't fix it:

  <img width="600" alt="Real-time_HTML_Editor_and_Using_character_escapes_in_markup_and_CSS" src="https://user-images.githubusercontent.com/9599/94492937-44775b00-01a0-11eb-9c5f-e991af620404.png">

  But... padding to six characters does! See https://www.w3.org/International/questions/qa-escapes

  <img width="607" alt="Real-time_HTML_Editor_and_Using_character_escapes_in_markup_and_CSS" src="https://user-images.githubusercontent.com/9599/94492988-61139300-01a0-11eb-8304-bffe448c7d2b.png">

  ```
  In [32]: print('\\' + "{:X}".format(ord('"')).zfill(6))
  \000022
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/978#issuecomment-700320480
issue_url: https://api.github.com/repos/simonw/datasette/issues/978
node_id: MDEyOklzc3VlQ29tbWVudDcwMDMyMDQ4MA==
created_at: 2020-09-28T22:39:18Z
updated_at: 2020-09-28T22:39:18Z
author_association: OWNER
body:
  ```python
  def escape_css_string(s):
      return _css_re.sub(lambda m: "\\" + ("{:X}".format(ord(m.group())).zfill(6)), s)
  ```
  That fixes it:
  <img width="363" alt="data__select_dateutil_parse__10_october_2020_3pm____dateutil_easter__2020____dateutil_parse_fuzzy__This_is_due_10_september____dateutil_parse__1_2_2020____dateutil_parse__2020-03-04____dateutil_parse_dayfirst__2020-03-04____dateutil_easter_" src="https://user-images.githubusercontent.com/9599/94493173-c23b6680-01a0-11eb-9468-e972c51b015c.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/979#issuecomment-700343229
issue_url: https://api.github.com/repos/simonw/datasette/issues/979
node_id: MDEyOklzc3VlQ29tbWVudDcwMDM0MzIyOQ==
created_at: 2020-09-28T23:55:55Z
updated_at: 2020-09-28T23:55:55Z
author_association: OWNER
body:
  Here's the code that adds it to the HTML context: https://github.com/simonw/datasette/blob/c11383e6284e000b2641569457efa16ac9e0d6ae/datasette/views/table.py#L835-L837

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/979#issuecomment-700343373
issue_url: https://api.github.com/repos/simonw/datasette/issues/979
node_id: MDEyOklzc3VlQ29tbWVudDcwMDM0MzM3Mw==
created_at: 2020-09-28T23:56:27Z
updated_at: 2020-09-28T23:56:27Z
author_association: OWNER
body:
  This would benefit https://github.com/simonw/datasette-import-table - which currently ignores the `CREATE TABLE` and derives the schema by inserting rows.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/980#issuecomment-700490225
issue_url: https://api.github.com/repos/simonw/datasette/issues/980
node_id: MDEyOklzc3VlQ29tbWVudDcwMDQ5MDIyNQ==
created_at: 2020-09-29T06:53:37Z
updated_at: 2020-09-29T06:53:37Z
author_association: OWNER
body:
  This time it's because there are newlines in the column header:
  ```html

  <style>
  @media only screen and (max-width: 576px) {

      .rows-and-columns td:nth-of-type(1):before { content: "dateutil_rrule(\000027FREQ=HOURLY;COUNT=5\000027)"; }

      .rows-and-columns td:nth-of-type(2):before { content: "dateutil_rrule_date(
  \00000A    \000027FREQ=DAILY;COUNT=3\000027,
  \00000A    \0000271st jan 2020\000027
  \00000A  )"; }

  }
  </style>
  ```
  Those need to be escaped somehow.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/980#issuecomment-700929721
issue_url: https://api.github.com/repos/simonw/datasette/issues/980
node_id: MDEyOklzc3VlQ29tbWVudDcwMDkyOTcyMQ==
created_at: 2020-09-29T19:21:50Z
updated_at: 2020-09-29T19:21:50Z
author_association: OWNER
body:
  That fixed it: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule(%27FREQ%3DHOURLY%3BCOUNT%3D5%27)%2C%0D%0A++dateutil_rrule_date(%0D%0A++++%27FREQ%3DDAILY%3BCOUNT%3D3%27%2C%0D%0A++++%271st+jan+2020%27%0D%0A++)%3B

  <img width="525" alt="fixtures__select_dateutil_rrule__FREQ_HOURLY_COUNT_5____dateutil_rrule_date___FREQ_DAILY_COUNT_3____1st_jan_2020____" src="https://user-images.githubusercontent.com/9599/94605680-4d266a80-024e-11eb-8818-bc8bd7958df4.png">

  ```html
  <style>
  @media only screen and (max-width: 576px) {

      .rows-and-columns td:nth-of-type(1):before { content: "dateutil_rrule(\000027FREQ=HOURLY;COUNT=5\000027)"; }

      .rows-and-columns td:nth-of-type(2):before { content: "dateutil_rrule_date(\00000A    \000027FREQ=DAILY;COUNT=3\000027,\00000A    \0000271st jan 2020\000027\00000A  )"; }

  }
  </style>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701153543
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTE1MzU0Mw==
created_at: 2020-09-30T04:46:05Z
updated_at: 2020-09-30T05:11:45Z
author_association: OWNER
body:
  Prototype:
  ```html
  <style>
  body {
  font-family: helvetica;
  }
  .dropdown-menu {
    display: inline-flex;
    border: 1px solid #ccc;
    border-radius: 4px;
    line-height: 1.4;
    font-size: 12px;
    box-shadow: 2px 2px 2px #aaa;
  }
  .dropdown-menu ul,
  .dropdown-menu li {
    list-style-type: none;
    margin: 0;
    padding: 0;
  }
  .dropdown-menu li {
    border-bottom: 1px solid #ccc;
  }
  .dropdown-menu li:last-child {
    border: none;
  }
  .dropdown-menu a:link,
  .dropdown-menu a:visited,
  .dropdown-menu a:hover,
  .dropdown-menu a:focus
  .dropdown-menu a:active {
    text-decoration: none;
    display: block;
    padding: 4px 8px 2px 8px;
    color: #222;
    background-color: #fff;
  }
  .dropdown-menu a:hover {
    background-color: #eee;
  }
  .hook {
    display: block;
    position: absolute;
    top: 3px;
    left: 12px;
    width: 0; 
    height: 0; 
    border-left: 5px solid transparent;
    border-right: 5px solid transparent;
    border-bottom: 5px solid #ccc;
  }
  </style>
  <div class="dropdown-menu">
  <div class="hook"></div>
  <ul>
    <li><a href="#">Sort descending</a></li>
    <li><a href="#">Sort ascending</a></li>
    <li><a href="#">Facet by this</a></li>
    <li><a href="#">Count by this</a></li>
  </ul>
  </div>
  ```
  <img width="148" alt="Real-time_HTML_Editor" src="https://user-images.githubusercontent.com/9599/94643784-7241ca00-029c-11eb-8554-863fcc255352.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701153600
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTE1MzYwMA==
created_at: 2020-09-30T04:46:18Z
updated_at: 2020-09-30T04:46:18Z
author_association: OWNER
body:
  More options:
  ```html
  <ul>
    <li><a href="#">Sort descending</a></li>
    <li><a href="#">Sort ascending</a></li>
    <li><a href="#">Facet by this</a></li>
    <li><a href="#">Unique values</a></li>
    <li><a href="#">Count unique values</a></li>
  </ul>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701153822
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTE1MzgyMg==
created_at: 2020-09-30T04:47:10Z
updated_at: 2020-09-30T04:47:10Z
author_association: OWNER
body:
  Future version could have expanding out nested side menus that let you do things like "calculate sum/avg for this column against this-other-column".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701153982
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTE1Mzk4Mg==
created_at: 2020-09-30T04:47:54Z
updated_at: 2020-09-30T04:47:54Z
author_association: OWNER
body:
  I think the accessible way to do this is with absolute positioning - have a menu icon in the `<th>` which, when clicked, causes the dropdown menu to appear as an absolutely positioned `<div>` that is not located within the DOM hierarchy of the`<th>` itself but is positioned to show up in the correct place.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701157010
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTE1NzAxMA==
created_at: 2020-09-30T05:00:42Z
updated_at: 2020-09-30T20:06:59Z
author_association: OWNER
body:
  Maybe use this as the icon:
  ```svg
  <svg width="100px" height="100px" viewBox="0 0 255 255">
    <polygon points="0,64 128,191 255,64"/>
  </svg>
  ```

  <img width="41" alt="Real-time_HTML_Editor" src="https://user-images.githubusercontent.com/9599/94734258-d22c8500-031d-11eb-982b-82cde4b1920f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/982#issuecomment-701585075
issue_url: https://api.github.com/repos/simonw/datasette/issues/982
node_id: MDEyOklzc3VlQ29tbWVudDcwMTU4NTA3NQ==
created_at: 2020-09-30T19:05:11Z
updated_at: 2020-09-30T19:05:11Z
author_association: OWNER
body:
  The form needs to switch from GET to POST if the query is a write query. JavaScript can handle this based on the checkbox - if a user does not have JavaScript submitting the form will cause the form action to be changed to POST and the form to be redisplayed.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/982#issuecomment-701585695
issue_url: https://api.github.com/repos/simonw/datasette/issues/982
node_id: MDEyOklzc3VlQ29tbWVudDcwMTU4NTY5NQ==
created_at: 2020-09-30T19:06:29Z
updated_at: 2020-09-30T19:06:29Z
author_association: OWNER
body:
  This is a little related to the error display issue #619 in that both will require some reworking of how the code is structured.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701615291
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTYxNTI5MQ==
created_at: 2020-09-30T20:04:34Z
updated_at: 2020-09-30T20:05:37Z
author_association: OWNER
body:
  Another potential action:

  * Show rows where this is not blank (equivalent to `is not blank` filter)

  This could be displayed conditionally based on if the column is detected to have any blank rows in it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701616922
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTYxNjkyMg==
created_at: 2020-09-30T20:08:02Z
updated_at: 2020-09-30T20:08:02Z
author_association: OWNER
body:
  It would be neat to provide a JavaScript plugin hook that plugins can use to add their own options to this menu. No idea what that would look like though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/182#issuecomment-701626134
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/182
node_id: MDEyOklzc3VlQ29tbWVudDcwMTYyNjEzNA==
created_at: 2020-09-30T20:27:09Z
updated_at: 2020-09-30T20:27:42Z
author_association: OWNER
body:
  It looks like http://maps.natalian.org/data.txt is encoded as `latin-1`, but `sqlite-utils` assumes `utf-8` and hence breaks.

  It would be worth improving the error message here. I could also add a `--encoding latin-1` option to `sqlite-utils insert` to help in consuming files that are stored in charsets other than `utf-8`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-701629984
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDcwMTYyOTk4NA==
created_at: 2020-09-30T20:34:43Z
updated_at: 2020-09-30T20:34:43Z
author_association: OWNER
body:
  I had a look around and there isn't an obvious pluggy equivalent in JavaScript world at the moment. Lots of frameworks like jQuery and Vue have their own custom plugin mechanisms.

  https://github.com/rekit/js-plugin is a simple standalone plugin mechanism. Not quite as full-featured as Pluggy though - in particular I like how Pluggy supports multiple plugins returning results for the same hook that get concatenated into a list of results.

  https://css-tricks.com/designing-a-javascript-plugin-system/ has some ideas.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701664306
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTY2NDMwNg==
created_at: 2020-09-30T21:47:08Z
updated_at: 2020-09-30T21:47:08Z
author_association: OWNER
body:
  The arrow icon didn't make sense because I already have a triangle icon showing sort order. I'm trying a cog icon instead:

  <img width="303" alt="data__dogs__10_rows" src="https://user-images.githubusercontent.com/9599/94743176-ce9ffa80-032b-11eb-8f4d-da3962984a4f.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701668526
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTY2ODUyNg==
created_at: 2020-09-30T21:57:22Z
updated_at: 2020-09-30T21:57:22Z
author_association: OWNER
body:
  A bunch of things to fix:

  - It clobbers existing querystring parameters - it needs to leave these alone (but replace the current sort order)
  - Facet option should not show up if you are already faceting by that column
  - There's no way to close the menu once it has opened!
  - Accessibility: SVG icon doesn't even have an alt attribute yet. Should use ARIA when the thing appears.

  It's also not visible on mobile, need to think about how that will work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/183#issuecomment-701678659
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/183
node_id: MDEyOklzc3VlQ29tbWVudDcwMTY3ODY1OQ==
created_at: 2020-09-30T22:23:36Z
updated_at: 2020-09-30T22:23:36Z
author_association: OWNER
body:
  It just ran and didn't find anything. I'll leave it enabled for a while but I may turn it off again, it could be that this kind of Python library isn't really what it's useful for.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701679729
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTY3OTcyOQ==
created_at: 2020-09-30T22:26:33Z
updated_at: 2020-09-30T22:26:33Z
author_association: OWNER
body:
  Bug: https://latest.datasette.io/fixtures/sortable?_sort=sortable

  ![sort-bug](https://user-images.githubusercontent.com/9599/94746304-4e7c9380-0331-11eb-8e81-7245529c4654.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701697918
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTY5NzkxOA==
created_at: 2020-09-30T23:24:14Z
updated_at: 2020-09-30T23:25:28Z
author_association: OWNER
body:
  I could provide the "Show non-blank values" option only for columns where there are blank values visible on the current page.

  This could be a bit confusing though, since the absence of that option could suggest that there are no blank values at all when that's actually not true.

  One option: run a separate `fetch()` call that figures out if any of the columns contain blank values, which gets a bit of extra time to execute. Only show the "Show non-blank values" option in the menu once that has returned.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701703472
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTcwMzQ3Mg==
created_at: 2020-09-30T23:43:30Z
updated_at: 2020-09-30T23:43:30Z
author_association: OWNER
body:
  I'm going to go based just on the visible values on the current page. I think that's good enough, and it avoids the complexity involved in doing a server-side check for blank values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701704716
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTcwNDcxNg==
created_at: 2020-09-30T23:48:36Z
updated_at: 2020-09-30T23:48:36Z
author_association: OWNER
body:
  Since this menu doesn't provide new functionality, I'm going to ignore the fact that it doesn't exist on portrait mobile view for the moment. Likewise, I'm going to skip making it accessible for the moment since lacking accessibility doesn't prevent functionality from being accessed - the menu-less experience currently works the same as the portrait mobile experience.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/981#issuecomment-701707116
issue_url: https://api.github.com/repos/simonw/datasette/issues/981
node_id: MDEyOklzc3VlQ29tbWVudDcwMTcwNzExNg==
created_at: 2020-09-30T23:58:17Z
updated_at: 2020-09-30T23:58:17Z
author_association: OWNER
body:
  Now live at https://latest.datasette.io/fixtures/facetable

  <img width="1264" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/94751748-1f205380-033e-11eb-940b-bb4ee7728d31.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/530#issuecomment-701720034
issue_url: https://api.github.com/repos/simonw/datasette/issues/530
node_id: MDEyOklzc3VlQ29tbWVudDcwMTcyMDAzNA==
created_at: 2020-10-01T00:42:08Z
updated_at: 2020-10-01T00:42:08Z
author_association: OWNER
body:
  I've changed my mind - especially since Datasette now ships with default JavaScript for the column menus in #981. I'm not going to extract CodeMirror into a plugin, I'll leave it where it is.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/985#issuecomment-702132019
issue_url: https://api.github.com/repos/simonw/datasette/issues/985
node_id: MDEyOklzc3VlQ29tbWVudDcwMjEzMjAxOQ==
created_at: 2020-10-01T13:23:48Z
updated_at: 2020-10-01T13:23:48Z
author_association: OWNER
body:
  The code that skips the "facet by this" menu option for primary key columns could take into account if there are multiple primary keys.

  https://github.com/simonw/datasette/blob/141544613f9e76ddb74eee38d6f8ee1e0e70f833/datasette/static/table.js#L91-L98

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/986#issuecomment-702171636
issue_url: https://api.github.com/repos/simonw/datasette/issues/986
node_id: MDEyOklzc3VlQ29tbWVudDcwMjE3MTYzNg==
created_at: 2020-10-01T14:24:11Z
updated_at: 2020-10-01T14:24:11Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=h1) Report
  > Merging [#986](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/141544613f9e76ddb74eee38d6f8ee1e0e70f833?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/986/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #986   +/-   ##
  =======================================
    Coverage   84.28%   84.28%           
  =======================================
    Files          28       28           
    Lines        3850     3850           
  =======================================
    Hits         3245     3245           
    Misses        605      605           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=footer). Last update [1415446...76f7094](https://codecov.io/gh/simonw/datasette/pull/986?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/987#issuecomment-702240916
issue_url: https://api.github.com/repos/simonw/datasette/issues/987
node_id: MDEyOklzc3VlQ29tbWVudDcwMjI0MDkxNg==
created_at: 2020-10-01T16:10:44Z
updated_at: 2020-10-01T16:10:44Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/141544613f9e76ddb74eee38d6f8ee1e0e70f833/datasette/templates/_table.html#L6

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/987#issuecomment-702241397
issue_url: https://api.github.com/repos/simonw/datasette/issues/987
node_id: MDEyOklzc3VlQ29tbWVudDcwMjI0MTM5Nw==
created_at: 2020-10-01T16:11:31Z
updated_at: 2020-10-01T16:11:31Z
author_association: OWNER
body:
  This should be a new page in the documentation, which will be expanded later on to cover things like the proposed JavaScript plugin hooks system from #983.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/986#issuecomment-702265255
issue_url: https://api.github.com/repos/simonw/datasette/issues/986
node_id: MDEyOklzc3VlQ29tbWVudDcwMjI2NTI1NQ==
created_at: 2020-10-01T16:51:45Z
updated_at: 2020-10-01T16:51:45Z
author_association: OWNER
body:
  Thanks for taking a look! The fix ended up being a little different from this because I still want to disable faceting on regular single primary keys (since faceting by those won't ever produce interesting results) - here's what I used: https://github.com/simonw/datasette/commit/5d6bc4c268f9f155e59561671f8617addd3e91bc

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/865#issuecomment-702418045
issue_url: https://api.github.com/repos/simonw/datasette/issues/865
node_id: MDEyOklzc3VlQ29tbWVudDcwMjQxODA0NQ==
created_at: 2020-10-01T21:54:00Z
updated_at: 2020-10-01T21:54:00Z
author_association: OWNER
body:
  Had a thought: this is likely to break in plugins too, such as `datasette-edit-schema` which constructs URLs for redirects e.g. here: https://github.com/simonw/datasette-edit-schema/blob/dbd0abee6dd3385b114cfe9671f7ead1c4855b60/datasette_edit_schema/__init__.py#L46-L48

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/446#issuecomment-702459432
issue_url: https://api.github.com/repos/simonw/datasette/issues/446
node_id: MDEyOklzc3VlQ29tbWVudDcwMjQ1OTQzMg==
created_at: 2020-10-02T00:08:47Z
updated_at: 2020-10-02T00:08:47Z
author_association: OWNER
body:
  I've decided not to bother with this, especially having redesigned some hooks to return Datasette standard `Response` objects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/778#issuecomment-702493047
issue_url: https://api.github.com/repos/simonw/datasette/issues/778
node_id: MDEyOklzc3VlQ29tbWVudDcwMjQ5MzA0Nw==
created_at: 2020-10-02T02:26:25Z
updated_at: 2020-10-02T02:26:25Z
author_association: OWNER
body:
  I think this could work for arbitrary SQL queries too. Those would need querystring configuration that specifies which sorted column(s) should be used for the "next" cursor.

  One example: I'd like to be able to offer a paginated list of counts of values in a table - e.g. this query:

  https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=select+replies%2C+count%28*%29+from+%5Btwitter-ratio%2Fsenators%5D+group+by+replies+order+by+count%28*%29+desc%3B

  That could even become a query that gets linked to from the column actions menu.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/994#issuecomment-703878831
issue_url: https://api.github.com/repos/simonw/datasette/issues/994
node_id: MDEyOklzc3VlQ29tbWVudDcwMzg3ODgzMQ==
created_at: 2020-10-05T20:45:39Z
updated_at: 2020-10-09T16:18:24Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/994?src=pr&el=h1) Report
  > Merging [#994](https://codecov.io/gh/simonw/datasette/pull/994?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/e807c4eac0e85ae15e013379b0dde1d797f1377d?el=desc) will **increase** coverage by `0.26%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/994/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/994?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main     #994      +/-   ##
  ==========================================
  + Coverage   84.28%   84.55%   +0.26%     
  ==========================================
    Files          28       28              
    Lines        3850     3878      +28     
  ==========================================
  + Hits         3245     3279      +34     
  + Misses        605      599       -6     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/994?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.16% <0.00%> (-4.84%)` | :arrow_down: |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `74.35% <0.00%> (ø)` | |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.92% <0.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.34% <0.00%> (+0.02%)` | :arrow_up: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.85% <0.00%> (+0.11%)` | :arrow_up: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.94% <0.00%> (+0.15%)` | :arrow_up: |
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.13% <0.00%> (+0.22%)` | :arrow_up: |
  | [datasette/publish/heroku.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3B1Ymxpc2gvaGVyb2t1LnB5) | `87.12% <0.00%> (+0.53%)` | :arrow_up: |
  | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/994/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `93.51% <0.00%> (+8.33%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/994?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/994?src=pr&el=footer). Last update [e807c4e...ecba5d2](https://codecov.io/gh/simonw/datasette/pull/994?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/993#issuecomment-703928029
issue_url: https://api.github.com/repos/simonw/datasette/issues/993
node_id: MDEyOklzc3VlQ29tbWVudDcwMzkyODAyOQ==
created_at: 2020-10-05T22:42:45Z
updated_at: 2020-10-05T22:42:59Z
author_association: OWNER
body:
  <img width="574" alt="fixtures__facetable__15_rows" src="https://user-images.githubusercontent.com/9599/95139485-65e8c180-0721-11eb-9e91-263aad6f6846.png">

  The `NOT NULL` text shows up only for columns that are not null.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/993#issuecomment-703963211
issue_url: https://api.github.com/repos/simonw/datasette/issues/993
node_id: MDEyOklzc3VlQ29tbWVudDcwMzk2MzIxMQ==
created_at: 2020-10-06T00:40:48Z
updated_at: 2020-10-06T00:40:48Z
author_association: OWNER
body:
  Now live at https://latest.datasette.io/fixtures/facetable

  New `await db.table_column_details(table)` method is documented here: https://docs.datasette.io/en/latest/internals.html#database-introspection

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/992#issuecomment-704013305
issue_url: https://api.github.com/repos/simonw/datasette/issues/992
node_id: MDEyOklzc3VlQ29tbWVudDcwNDAxMzMwNQ==
created_at: 2020-10-06T04:00:00Z
updated_at: 2020-10-06T04:00:00Z
author_association: OWNER
body:
  I can add this well in advance of Datasette 1.0 as an alias to the existing `--config` option (but taking two arguments instead of one).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/995#issuecomment-704347565
issue_url: https://api.github.com/repos/simonw/datasette/issues/995
node_id: MDEyOklzc3VlQ29tbWVudDcwNDM0NzU2NQ==
created_at: 2020-10-06T15:22:55Z
updated_at: 2020-10-06T15:22:55Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=h1) Report
  > Merging [#995](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/5a184a5d211d3226e0417ee5cf8476cd887cd35e?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/995/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main     #995   +/-   ##
  =======================================
    Coverage   84.34%   84.34%           
  =======================================
    Files          28       28           
    Lines        3865     3865           
  =======================================
    Hits         3260     3260           
    Misses        605      605           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=footer). Last update [5a184a5...aed2cf9](https://codecov.io/gh/simonw/datasette/pull/995?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/996#issuecomment-705115185
issue_url: https://api.github.com/repos/simonw/datasette/issues/996
node_id: MDEyOklzc3VlQ29tbWVudDcwNTExNTE4NQ==
created_at: 2020-10-07T18:26:11Z
updated_at: 2020-10-07T18:26:45Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1030-L1037

  https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L951-L954

  https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1197-L1207

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/996#issuecomment-705116123
issue_url: https://api.github.com/repos/simonw/datasette/issues/996
node_id: MDEyOklzc3VlQ29tbWVudDcwNTExNjEyMw==
created_at: 2020-10-07T18:27:57Z
updated_at: 2020-10-07T18:28:36Z
author_association: OWNER
body:
  This line was meant to handle this case, by only matching non-/ characters. Looks like it did not have the desired effect: https://github.com/simonw/datasette/blob/14982bd900f17a66acc3930ec1a2ff138716d198/datasette/app.py#L1204

  Maybe because the finished regex doesn't include `^` and `$` anchors?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/991#issuecomment-705244583
issue_url: https://api.github.com/repos/simonw/datasette/issues/991
node_id: MDEyOklzc3VlQ29tbWVudDcwNTI0NDU4Mw==
created_at: 2020-10-07T23:24:56Z
updated_at: 2020-10-07T23:24:56Z
author_association: OWNER
body:
  Sketched out an option:
  ```diff
  diff --git a/datasette/templates/index.html b/datasette/templates/index.html
  index c1adfc5..eab3053 100644
  --- a/datasette/templates/index.html
  +++ b/datasette/templates/index.html
  @@ -5,6 +5,14 @@
   {% block body_class %}index{% endblock %}
   
   {% block content %}
  +<style>
  +.db-table {
  +    padding-left: 2em;
  +}
  +.db-table h3 {
  +    margin: 0 0 0.15em 0;
  +}
  +</style>
   <h1>{{ metadata.title or "Datasette" }}{% if private %} 🔒{% endif %}</h1>
   
   {% block description_source_license %}{% include "_description_source_license.html" %}{% endblock %}
  @@ -21,8 +29,20 @@
               {{ "{:,}".format(database.views_count) }} view{% if database.views_count != 1 %}s{% endif %}
           {% endif %}
       </p>
  -    <p>{% for table in database.tables_and_views_truncated %}<a href="{{ database.path }}/{{ table.name|quote_plus 
  -    }}"{% if table.count %} title="{{ table.count }} rows"{% endif %}>{{ table.name }}</a>{% if table.private %} 🔒{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}{% if database.tables_and_views_more %}, <a href="{{ database.path }}">...</a>{% endif %}</p>
  +    {% for table in database.tables_and_views_truncated %}
  +        {% if show_hidden or not table.hidden %}
  +        <div class="db-table">
  +            <h3><a href="{{ database_url(database.name) }}/{{ table.name|quote_plus }}">{{ table.name }}</a>{% if table.private %} 🔒{% endif %}{% if table.hidden %}<em> (hidden)</em>{% endif %}</h3>
  +            <p><em>{% for column in table.columns[:9] %}{{ column }}{% if not loop.last %}, {% endif %}{% endfor %}{% if table.columns|length > 9 %}...{% endif %}</em></p>
  +            <p>{% if table.count is none %}Many rows{% else %}{{ "{:,}".format(table.count) }} row{% if table.count == 1 %}{% else %}s{% endif %}{% endif %}</p>
  +        </div>
  +        {% endif %}
  +    {% endfor %}
  +
  +    {% if database.tables_and_views_more %}
  +        <p><a href="{{ database.path }}">All {{ database.tables_count }} tables in {{ database.name }}</a></p>
  +    {% endif %}
  +
   {% endfor %}
   
   {% endblock %}
  ```

  ![image](https://user-images.githubusercontent.com/9599/95398019-9c594480-08b9-11eb-90c9-f322d647ee3b.png)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/997#issuecomment-705466873
issue_url: https://api.github.com/repos/simonw/datasette/issues/997
node_id: MDEyOklzc3VlQ29tbWVudDcwNTQ2Njg3Mw==
created_at: 2020-10-08T10:02:02Z
updated_at: 2020-10-08T10:02:02Z
author_association: OWNER
body:
  I asked about other hosts for these on Twitter https://twitter.com/simonw/status/1314042631949676544

  - https://buildpacks.io/
  - https://doc.scalingo.com/languages/python/start
  - https://cloud.google.com/run/docs/continuous-deployment-with-cloud-build

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/865#issuecomment-705799046
issue_url: https://api.github.com/repos/simonw/datasette/issues/865
node_id: MDEyOklzc3VlQ29tbWVudDcwNTc5OTA0Ng==
created_at: 2020-10-08T20:15:26Z
updated_at: 2020-10-08T20:15:26Z
author_association: OWNER
body:
  Can anyone give me some detailed steps to reproduce for this issue? I think I need to set up my own local nginx or Apache proxy so I can really dig into this and figure out what's going on.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/998#issuecomment-705851211
issue_url: https://api.github.com/repos/simonw/datasette/issues/998
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg1MTIxMQ==
created_at: 2020-10-08T22:14:00Z
updated_at: 2020-10-08T22:14:00Z
author_association: OWNER
body:
  I tried this out and it did cause one problem with plugins: `datasette-cluster-map` still injected itself inside the scrolling area, when instead we would want that map to stay outside of the div with the table in it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/998#issuecomment-705851405
issue_url: https://api.github.com/repos/simonw/datasette/issues/998
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg1MTQwNQ==
created_at: 2020-10-08T22:14:31Z
updated_at: 2020-10-08T22:14:31Z
author_association: OWNER
body:
  So maybe this is blocked on #987 and adding a new hook in the HTML which plugins like `datasette-cluster-map` can attach their content to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/999#issuecomment-705871513
issue_url: https://api.github.com/repos/simonw/datasette/issues/999
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg3MTUxMw==
created_at: 2020-10-08T23:17:31Z
updated_at: 2020-10-08T23:17:31Z
author_association: OWNER
body:
  This does mean that any deploys to Heroku will fail with an error until this code has been released in a non-alpha Datasette release (unless you use `--branch=main`) - so I should ship this as soon as possible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/999#issuecomment-705872767
issue_url: https://api.github.com/repos/simonw/datasette/issues/999
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg3Mjc2Nw==
created_at: 2020-10-08T23:21:36Z
updated_at: 2020-10-08T23:21:36Z
author_association: OWNER
body:
  I ran this to prove that it breaks:

      datasette publish heroku fixtures.db -n datasette-issue-999

  https://datasette-issue-999.herokuapp.com/ returned an error, as expected - here's the relevant logs:

  ```
  $ heroku logs --tail -a datasette-issue-999
  2020-10-08T23:19:01.065103+00:00 heroku[web.1]: Starting process with command `datasette serve --host 0.0.0.0 -i fixtures.db --cors --port 22329 --inspect-file inspect-data.json`
  2020-10-08T23:19:05.381374+00:00 app[web.1]: WARNING:  You must pass the application as an import string to enable 'reload' or 'workers'.
  2020-10-08T23:19:05.617519+00:00 heroku[web.1]: Process exited with status 1
  2020-10-08T23:19:05.690673+00:00 heroku[web.1]: State changed from starting to crashed
  ```
  Then I deployed again using:

      datasette publish heroku fixtures.db -n datasette-issue-999 --branch=main

  This time the deploy worked OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/969#issuecomment-705874317
issue_url: https://api.github.com/repos/simonw/datasette/issues/969
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg3NDMxNw==
created_at: 2020-10-08T23:27:00Z
updated_at: 2020-10-08T23:27:00Z
author_association: OWNER
body:
  I'm working on this now. Just ran this test and it worked (no longer showed my the warning message).

      datasette publish heroku fixtures.db -n datasette-issue-999 --branch=main --tar=/usr/local/bin/gtar

  But... I wonder if it should automatically do this if it detects that `gtar` is available?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/969#issuecomment-705874457
issue_url: https://api.github.com/repos/simonw/datasette/issues/969
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg3NDQ1Nw==
created_at: 2020-10-08T23:27:30Z
updated_at: 2020-10-08T23:27:30Z
author_association: OWNER
body:
  For the moment I'm going to ship this as the `--tar=` option. Can consider detecting `gtar` in the future.

reactions:
  {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/514#issuecomment-705878827
issue_url: https://api.github.com/repos/simonw/datasette/issues/514
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg3ODgyNw==
created_at: 2020-10-08T23:33:05Z
updated_at: 2020-10-08T23:33:05Z
author_association: OWNER
body:
  Documentation is now live at https://docs.datasette.io/en/latest/deploying.html#running-datasette-using-systemd

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/997#issuecomment-705883053
issue_url: https://api.github.com/repos/simonw/datasette/issues/997
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg4MzA1Mw==
created_at: 2020-10-08T23:36:55Z
updated_at: 2020-10-08T23:36:55Z
author_association: OWNER
body:
  Docs are now live here: https://docs.datasette.io/en/latest/deploying.html#deploying-using-buildpacks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705890365
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg5MDM2NQ==
created_at: 2020-10-09T00:03:29Z
updated_at: 2020-10-09T16:07:03Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=h1) Report
  > Merging [#1000](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/7249ac5ca04b5ddc6517750326ee7e522cc49145?el=desc) will **increase** coverage by `0.15%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1000/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1000      +/-   ##
  ==========================================
  + Coverage   84.37%   84.52%   +0.15%     
  ==========================================
    Files          28       28              
    Lines        3871     3878       +7     
  ==========================================
  + Hits         3266     3278      +12     
  + Misses        605      600       -5     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.34% <100.00%> (+0.02%)` | :arrow_up: |
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `74.35% <100.00%> (ø)` | |
  | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.16% <100.00%> (-4.84%)` | :arrow_down: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.94% <100.00%> (+0.11%)` | :arrow_up: |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `91.92% <0.00%> (ø)` | |
  | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1000/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `93.51% <0.00%> (+8.33%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=footer). Last update [7249ac5...8a80c79](https://codecov.io/gh/simonw/datasette/pull/1000?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705899629
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTg5OTYyOQ==
created_at: 2020-10-09T00:37:02Z
updated_at: 2020-10-09T00:37:02Z
author_association: OWNER
body:
  I'm going to route the existing `TestClient` through this mechanism to exercise it during the tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705902902
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkwMjkwMg==
created_at: 2020-10-09T00:50:49Z
updated_at: 2020-10-09T00:50:49Z
author_association: OWNER
body:
  Almost all of the tests are passing:
  ```
  =========================== short test summary info ============================
  FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200
  FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200
  FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...
  FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]
  FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]
  FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]
  FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]
  FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]
  FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]
  FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyE...
  FAILED tests/test_plugins.py::test_hook_register_magic_parameters - Assertion...
  ============ 11 failed, 718 passed, 6 warnings in 225.77s (0:03:45) ============
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1001#issuecomment-705904566
issue_url: https://api.github.com/repos/simonw/datasette/issues/1001
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkwNDU2Ng==
created_at: 2020-10-09T00:58:08Z
updated_at: 2020-10-09T00:58:08Z
author_association: OWNER
body:
  To get a traceback:
  ```
  datasette . -p 8009 --pdb
  ```
  And then:
  ```
  curl -XOPTIONS http://127.0.0.1:8009
  ```
  This causes the server to open a debugging prompt:
  ```
  INFO:     127.0.0.1:59514 - "OPTIONS / HTTP/1.1" 500 Internal Server Error
  > /Users/simon/Dropbox/Development/datasette/datasette/views/base.py(115)dispatch_request()
  -> return await handler(request, *args, **kwargs)
  (Pdb) list
  110  	    def database_color(self, database):
  111  	        return "ff0000"
  112  	
  113  	    async def dispatch_request(self, request, *args, **kwargs):
  114  	        handler = getattr(self, request.method.lower(), None)
  115  ->	        return await handler(request, *args, **kwargs)
  116  	
  117  	    async def render(self, templates, request, context=None):
  118  	        context = context or {}
  119  	        template = self.ds.jinja_env.select_template(templates)
  120  	        template_context = {
  (Pdb) 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1001#issuecomment-705904679
issue_url: https://api.github.com/repos/simonw/datasette/issues/1001
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkwNDY3OQ==
created_at: 2020-10-09T00:58:32Z
updated_at: 2020-10-09T00:58:32Z
author_association: OWNER
body:
  So the bug is in this code here: https://github.com/simonw/datasette/blob/703439bdc37e724b01bc6d7a1fc1d955795132f2/datasette/views/base.py#L113-L115

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1001#issuecomment-705904917
issue_url: https://api.github.com/repos/simonw/datasette/issues/1001
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkwNDkxNw==
created_at: 2020-10-09T00:59:25Z
updated_at: 2020-10-09T00:59:25Z
author_association: OWNER
body:
  ```
  ~ % curl -XOPTIONS https://www.google.com/
  <!DOCTYPE html>
  <html lang=en>
    <meta charset=utf-8>
    <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
    <title>Error 405 (Method Not Allowed)!!1</title>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1001#issuecomment-705905121
issue_url: https://api.github.com/repos/simonw/datasette/issues/1001
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkwNTEyMQ==
created_at: 2020-10-09T01:00:07Z
updated_at: 2020-10-09T01:00:07Z
author_association: OWNER
body:
  The www.djangoproject.com site returns an empty page:
  ```
  ~ % curl -vv -XOPTIONS https://www.djangoproject.com/
  *   Trying 151.101.42.217:443...
  * Connected to www.djangoproject.com (151.101.42.217) port 443 (#0)
  * ALPN, offering http/1.1
  * TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  * Server certificate: osff2.map.fastly.net
  * Server certificate: GlobalSign CloudSSL CA - SHA256 - G3
  * Server certificate: GlobalSign Root CA
  > OPTIONS / HTTP/1.1
  > Host: www.djangoproject.com
  > User-Agent: curl/7.70.0
  > Accept: */*
  > 
  * Mark bundle as not supporting multiuse
  < HTTP/1.1 200 OK
  < Connection: keep-alive
  < Content-Length: 0
  < Server: nginx
  < Content-Type: text/html; charset=utf-8
  < Allow: GET, HEAD, OPTIONS
  < Content-Language: en
  < X-Frame-Options: SAMEORIGIN
  < X-Content-Type-Options: nosniff
  < X-XSS-Protection: 1; mode=block
  < Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
  < Access-Control-Allow-Origin: https://code.djangoproject.com
  < Accept-Ranges: bytes
  < Date: Fri, 09 Oct 2020 00:59:42 GMT
  < Via: 1.1 varnish
  < X-Served-By: cache-sjc10047-SJC
  < X-Cache: MISS
  < X-Cache-Hits: 0
  < X-Timer: S1602205182.833493,VS0,VE305
  < Vary: Accept-Language, Accept-Encoding
  < 
  * Connection #0 to host www.djangoproject.com left intact
  ~ % 
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1001#issuecomment-705916614
issue_url: https://api.github.com/repos/simonw/datasette/issues/1001
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkxNjYxNA==
created_at: 2020-10-09T01:37:06Z
updated_at: 2020-10-09T01:37:06Z
author_association: OWNER
body:
  I'm tempted to imitate Django Rest Framework here: https://github.com/encode/django-rest-framework/blob/2e721cdbc85a924d0b0f093b86fe1497b58fe287/rest_framework/views.py#L514-L521
  ```python
      def options(self, request, *args, **kwargs):
          """
          Handler method for HTTP 'OPTIONS' request.
          """
          if self.metadata_class is None:
              return self.http_method_not_allowed(request, *args, **kwargs)
          data = self.metadata_class().determine_metadata(request, self)
          return Response(data, status=status.HTTP_200_OK)
  ```
  That `determine_metadata()` default method does this: https://github.com/encode/django-rest-framework/blob/335054a5d36b352a58286b303b608b6bf48152f8/rest_framework/metadata.py#L29

  Note the comment at the top:
  ```
      There are not any formalized standards for `OPTIONS` responses
      for us to base this on.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1001#issuecomment-705917015
issue_url: https://api.github.com/repos/simonw/datasette/issues/1001
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkxNzAxNQ==
created_at: 2020-10-09T01:38:49Z
updated_at: 2020-10-09T01:38:49Z
author_association: OWNER
body:
  I actually have a sensible `OPTIONS` implementation here:

  https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/datasette/views/base.py#L154-L165

  I'm going to set the default one to return a 405 (Method Not Allowed) like Google does.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705918844
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkxODg0NA==
created_at: 2020-10-09T01:46:06Z
updated_at: 2020-10-09T01:46:06Z
author_association: OWNER
body:
  For this failing test I'm suspicious that the AsyncClient may be persisting cookies in between requests:
  ```
      def test_actor_cookie(app_client):
          "A valid actor cookie sets request.scope['actor']"
          cookie = app_client.actor_cookie({"id": "test"})
          response = app_client.get("/", cookies={"ds_actor": cookie})
  >       assert {"id": "test"} == app_client.ds._last_request.scope["actor"]
  E       AssertionError: assert {'id': 'test'} == {'id': 'root'}
  E         Differing items:
  E         {'id': 'test'} != {'id': 'root'}
  E         Use -v to get the full diff
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705920055
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkyMDA1NQ==
created_at: 2020-10-09T01:51:05Z
updated_at: 2020-10-09T01:51:05Z
author_association: OWNER
body:
  The topic of disabling cookie persistence is discussed a little here: https://github.com/encode/httpx/issues/422#issuecomment-537906693

  > We have a cookiejar abstraction, I think setting it to an always-empty jar like you describe is best. :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705920228
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkyMDIyOA==
created_at: 2020-10-09T01:51:44Z
updated_at: 2020-10-09T01:51:44Z
author_association: OWNER
body:
  I'm going to switch back to having each request run through a new client. I'm worried about the impact on test performance though. I'll run a microbenchmark before and after.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705921006
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkyMTAwNg==
created_at: 2020-10-09T01:55:01Z
updated_at: 2020-10-09T01:55:01Z
author_association: OWNER
body:
  With the single client that is reused for all tests:
  ```
  % time pytest tests/test_api.py
  ...
  6.73s user 9.91s system 81% cpu 20.365 total
  ```
  After switching back to this class:
  ```python
  class DatasetteClient:
      def __init__(self, ds):
          self.app = ds.app()

      def _fix(self, path):
          if path.startswith("/"):
              path = "http://localhost{}".format(path)
          return path

      async def get(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.get(self._fix(path), **kwargs)

      async def options(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.options(self._fix(path), **kwargs)

      async def head(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.head(self._fix(path), **kwargs)

      async def post(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.post(self._fix(path), **kwargs)

      async def put(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.put(self._fix(path), **kwargs)

      async def patch(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.patch(self._fix(path), **kwargs)

      async def delete(self, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.delete(self._fix(path), **kwargs)

      async def request(self, method, path, **kwargs):
          async with httpx.AsyncClient(app=self.app) as client:
              return await client.request(method, self._fix(path), **kwargs)
  ```
  The time taken is:
  ```
  % time pytest tests/test_api.py
  ...
  7.26s user 10.02s system 82% cpu 21.014 total
  ```
  That's close enough that I don't feel I need to investigate this further.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705926035
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkyNjAzNQ==
created_at: 2020-10-09T02:14:14Z
updated_at: 2020-10-09T02:14:14Z
author_association: OWNER
body:
  Still need to handle these six failing tests:
  ```
  FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'elemen...
  FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', '...
  FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<...
  FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/] - AssertionError: {'base_url': 'https://example.com/', 'element_parent': '<p cla...
  FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'
  FAILED tests/test_plugins.py::test_hook_register_magic_parameters - AssertionError: assert [{'line': '1.0', 'rowid': 1}] == [{'line': '1.1', 'rowid': 1}]
  =========================================================== 6 failed, 731 passed, 6 warnings in 129.17s (0:02:09) ===========================================================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705926445
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkyNjQ0NQ==
created_at: 2020-10-09T02:15:38Z
updated_at: 2020-10-09T02:15:38Z
author_association: OWNER
body:
  > FAILED tests/test_messages.py::test_messages_are_displayed_and_cleared - KeyError: 'ds_messages'

  That one is caused by `response.cookies` skipping cookies that were set to the empty string. Same fix as this: https://github.com/simonw/datasette/blob/a1687351fb75b01f737fda4ad07e0781029de05c/tests/test_auth.py#L90-L95

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705937696
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTkzNzY5Ng==
created_at: 2020-10-09T02:52:53Z
updated_at: 2020-10-09T02:52:53Z
author_association: OWNER
body:
  These failures are giving me a severe "how did this ever work in the first place?" vibe:
  ```
  FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys-https://example.com/]
  FAILED tests/test_html.py::test_base_url_config[/fixtures/compound_three_primary_keys/a,a,a-https://example.com/]
  FAILED tests/test_html.py::test_base_url_config[/fixtures/paginated_view-https://example.com/]
  FAILED tests/test_html.py::test_base_url_config[/fixtures/facetable-https://example.com/]
  ```
  I have a fix for them, no idea why they weren't already failing though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705940507
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTk0MDUwNw==
created_at: 2020-10-09T03:04:15Z
updated_at: 2020-10-09T03:04:15Z
author_association: OWNER
body:
  This is really weird: new set of test failures that I wasn't seeing before, and those tests aren't failing on my laptop:
  ```
  =========================== short test summary info ============================
  FAILED tests/test_api.py::test_table_with_slashes_in_name - assert 404 == 200
  FAILED tests/test_api.py::test_row_strange_table_name - assert 404 == 200
  FAILED tests/test_html.py::test_row_strange_table_name_with_url_hash - assert...
  FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/table%2Fwith%2Fslashes.csv-expected_classes5]
  FAILED tests/test_html.py::test_templates_considered[/fixtures/table%2Fwith%2Fslashes.csv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html]
  ================== 5 failed, 738 passed in 194.73s (0:03:14) ===================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705941580
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTk0MTU4MA==
created_at: 2020-10-09T03:08:43Z
updated_at: 2020-10-09T03:08:43Z
author_association: OWNER
body:
  Most likely reason for those failures is that `path` and `raw_path` are not being simulated correctly. I used to do those here:

  https://github.com/simonw/datasette/blob/402cf870b7d65f9b5fba9e23aa99433294bd4523/datasette/utils/testing.py#L116-L125

  But now I'm delegating that to `httpx` to handle.

  WEIRD that it passes on my laptop but fails in GitHub Actions CI though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705945591
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTk0NTU5MQ==
created_at: 2020-10-09T03:24:48Z
updated_at: 2020-10-09T03:24:48Z
author_association: OWNER
body:
  I'm testing this with a `print(scope)` and `pytest -k test_table_with_slashes_in_name -s`.

  Against the `main` branch:

  `{'type': 'http', 'http_version': '1.0', 'method': 'GET', 'path': '/fixtures/table/with/slashes.csv', 'raw_path': b'/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'headers': [[b'host', b'localhost']], 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x10e2e6040>}`

  Against the broken branch:

  `tests/test_api.py {'type': 'http', 'asgi': {'version': '3.0'}, 'http_version': '1.1', 'method': 'GET', 'headers': [(b'host', b'localhost'), (b'accept', b'*/*'), (b'accept-encoding', b'gzip, deflate'), (b'connection', b'keep-alive'), (b'user-agent', b'python-httpx/0.15.0')], 'scheme': 'http', 'path': '/fixtures/table%2Fwith%2Fslashes.csv', 'query_string': b'_shape=objects&_format=json', 'server': ('localhost', None), 'client': ('127.0.0.1', 123), 'root_path': '', 'csrftoken': <function asgi_csrf_decorator.<locals>._asgi_csrf_decorator.<locals>.app_wrapped_with_csrf.<locals>.get_csrftoken at 0x109e0eca0>}`

  This is on my laptop though so both of those pass the tests.

  Key difference: the `httpx` version doesn't set a `raw_path` at all. BUT.. it does set `path` and sets it to `'/fixtures/table%2Fwith%2Fslashes.csv'`

  The non-httpx version sets `raw_path` to `b'/fixtures/table%2Fwith%2Fslashes.csv'` and `path` to `'/fixtures/table/with/slashes.csv'`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705946120
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTk0NjEyMA==
created_at: 2020-10-09T03:27:05Z
updated_at: 2020-10-09T03:27:05Z
author_association: OWNER
body:
  I may need to fuss around with how the `httpx` client sends things to the ASGI app.

  https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L26 is relevant:

      Alternatively, you can setup the transport instance explicitly.
      This allows you to include any additional configuration arguments specific
      to the ASGITransport class:
      ```
      transport = httpx.ASGITransport(
          app=app,
          root_path="/submount",
          client=("1.2.3.4", 123)
      )
      client = httpx.AsyncClient(transport=transport)
      ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-705946360
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNTk0NjM2MA==
created_at: 2020-10-09T03:28:08Z
updated_at: 2020-10-09T03:28:08Z
author_association: OWNER
body:
  Here's where `httpx` sets up the ASGI scope: https://github.com/encode/httpx/blob/92ca4d0cc654859fc2257c492e55d8752370d427/httpx/_transports/asgi.py#L82-L97

  ```python
          # ASGI scope.
          scheme, host, port, full_path = url
          path, _, query = full_path.partition(b"?")
          scope = {
              "type": "http",
              "asgi": {"version": "3.0"},
              "http_version": "1.1",
              "method": method.decode(),
              "headers": [(k.lower(), v) for (k, v) in headers],
              "scheme": scheme.decode("ascii"),
              "path": unquote(path.decode("ascii")),
              "query_string": query,
              "server": (host.decode("ascii"), port),
              "client": self.client,
              "root_path": self.root_path,
          }
  ```
  Sure enough, it doesn't set the `raw_path`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1000#issuecomment-706263157
issue_url: https://api.github.com/repos/simonw/datasette/issues/1000
node_id: MDEyOklzc3VlQ29tbWVudDcwNjI2MzE1Nw==
created_at: 2020-10-09T15:57:15Z
updated_at: 2020-10-09T15:57:15Z
author_association: OWNER
body:
  My `httpx` pull request adding `raw_path` support was just merged: https://github.com/encode/httpx/pull/1357 - but it's not in a release yet.

  I'm going to mark these tests as `xfail` so I can land this change - I'll remove that once an `httpx` release comes out that I can use to get the tests passing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1003#issuecomment-706272322
issue_url: https://api.github.com/repos/simonw/datasette/issues/1003
node_id: MDEyOklzc3VlQ29tbWVudDcwNjI3MjMyMg==
created_at: 2020-10-09T16:14:56Z
updated_at: 2020-10-09T16:14:56Z
author_association: OWNER
body:
  Yes I think that makes sense. I added `json` to the template context in Dogsheep Beta just a few days ago because I needed that: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L176

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1003#issuecomment-706273211
issue_url: https://api.github.com/repos/simonw/datasette/issues/1003
node_id: MDEyOklzc3VlQ29tbWVudDcwNjI3MzIxMQ==
created_at: 2020-10-09T16:16:38Z
updated_at: 2020-10-09T16:16:38Z
author_association: OWNER
body:
  I'm not a huge fan of `from_json` as the name for this. Some other options:

  - Expose `json` directly so templates can do `json.loads()` and `json.dumps()` - this allows for outputting JSON too, which is useful. But is there anything else on the `json` module that shouldn't be exposed in templates?
  - `json_dumps()` and `json_loads()` template functions. I quite like that.
  - Something else?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1007#issuecomment-706276831
issue_url: https://api.github.com/repos/simonw/datasette/issues/1007
node_id: MDEyOklzc3VlQ29tbWVudDcwNjI3NjgzMQ==
created_at: 2020-10-09T16:23:51Z
updated_at: 2020-10-09T16:23:51Z
author_association: OWNER
body:
  I don't appear to be using these anywhere, not sure why I spotted a warning (which I now can't find).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1003#issuecomment-706281451
issue_url: https://api.github.com/repos/simonw/datasette/issues/1003
node_id: MDEyOklzc3VlQ29tbWVudDcwNjI4MTQ1MQ==
created_at: 2020-10-09T16:33:01Z
updated_at: 2020-10-09T16:33:01Z
author_association: OWNER
body:
  I think `json_dumps()` and `json_loads()` as aliases for `json.dumps()` and `json.loads()` is the way to go here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1003#issuecomment-706302863
issue_url: https://api.github.com/repos/simonw/datasette/issues/1003
node_id: MDEyOklzc3VlQ29tbWVudDcwNjMwMjg2Mw==
created_at: 2020-10-09T17:17:06Z
updated_at: 2020-10-09T17:17:06Z
author_association: NONE
body:
  I agree on the descriptive and python-consistent naming. There is already a tojson, but frankly i find the "to" and "from" confusing in a text templating language where what's a string and what's data isn't 100% transparent.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1008#issuecomment-706383750
issue_url: https://api.github.com/repos/simonw/datasette/issues/1008
node_id: MDEyOklzc3VlQ29tbWVudDcwNjM4Mzc1MA==
created_at: 2020-10-09T20:17:29Z
updated_at: 2020-10-09T20:17:29Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1008?src=pr&el=h1) Report
  > Merging [#1008](https://codecov.io/gh/simonw/datasette/pull/1008?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/1bdbc8aa7f4fd7a768d456146e44da86cb1b36d1?el=desc) will **increase** coverage by `0.00%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1008/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1008?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1008   +/-   ##
  =======================================
    Coverage   84.55%   84.56%           
  =======================================
    Files          28       28           
    Lines        3878     3880    +2     
  =======================================
  + Hits         3279     3281    +2     
    Misses        599      599           
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1008?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1008/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.35% <100.00%> (+0.01%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1008?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1008?src=pr&el=footer). Last update [1bdbc8a...4085898](https://codecov.io/gh/simonw/datasette/pull/1008?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/983#issuecomment-706413753
issue_url: https://api.github.com/repos/simonw/datasette/issues/983
node_id: MDEyOklzc3VlQ29tbWVudDcwNjQxMzc1Mw==
created_at: 2020-10-09T21:41:12Z
updated_at: 2020-10-09T21:41:12Z
author_association: NONE
body:
  If you don't mind a somewhat bonkers idea: how about a JS client-side plugin capability that allows any user looking at a Datasette site to pull in external plugins for data manipulation, even if the Datasette owner hasn't added them? (Yes, this may be _much_ too ambitious. If you're remotely interested, maybe fork this discussion to a different issue.) 

  This is some fascinating reading about what JS sandboxing looks like these days:
  https://www.figma.com/blog/how-we-built-the-figma-plugin-system/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1010#issuecomment-706450418
issue_url: https://api.github.com/repos/simonw/datasette/issues/1010
node_id: MDEyOklzc3VlQ29tbWVudDcwNjQ1MDQxOA==
created_at: 2020-10-10T00:08:27Z
updated_at: 2020-10-10T00:09:13Z
author_association: OWNER
body:
  This commit broke it: https://github.com/simonw/datasette/commit/8f97b9b58e77f82fef1f10e9c9f6754b993544b6#diff-ed4bb3a97b7efa4d45be6a6ea6f4636cL114-R165

  <img width="1106" alt="Banners_and_Alerts_and_datasette_client_internal_requests_mechanism_·_simonw_datasette_8f97b9b" src="https://user-images.githubusercontent.com/9599/95640559-25a37f00-0a52-11eb-838e-9f771ab5a413.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1010#issuecomment-706455202
issue_url: https://api.github.com/repos/simonw/datasette/issues/1010
node_id: MDEyOklzc3VlQ29tbWVudDcwNjQ1NTIwMg==
created_at: 2020-10-10T00:33:46Z
updated_at: 2020-10-10T00:33:46Z
author_association: OWNER
body:
  Since this is an urgent fix that needs to go out I've marked the `base_url` tests as `xfail`. I need to solve those separately, see #900 and various other bug reports.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1010#issuecomment-706455505
issue_url: https://api.github.com/repos/simonw/datasette/issues/1010
node_id: MDEyOklzc3VlQ29tbWVudDcwNjQ1NTUwNQ==
created_at: 2020-10-10T00:35:41Z
updated_at: 2020-10-10T00:35:41Z
author_association: OWNER
body:
  I missed the links on the query page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1005#issuecomment-706473306
issue_url: https://api.github.com/repos/simonw/datasette/issues/1005
node_id: MDEyOklzc3VlQ29tbWVudDcwNjQ3MzMwNg==
created_at: 2020-10-10T02:45:46Z
updated_at: 2020-10-10T02:45:46Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L145-L149

  https://github.com/simonw/datasette/blob/9f6dd985bc0eff70f8a9ce65c6578bc43d2e172b/tests/test_html.py#L539-L564

  https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L1190-L1194

  https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/tests/test_api.py#L742-L746

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1011#issuecomment-706480763
issue_url: https://api.github.com/repos/simonw/datasette/issues/1011
node_id: MDEyOklzc3VlQ29tbWVudDcwNjQ4MDc2Mw==
created_at: 2020-10-10T03:42:58Z
updated_at: 2020-10-10T03:42:58Z
author_association: OWNER
body:
  Oh no! It's the same bug as #1010. Thanks for the report, I shall push out 0.50.2 as quickly as possible.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1011#issuecomment-706482542
issue_url: https://api.github.com/repos/simonw/datasette/issues/1011
node_id: MDEyOklzc3VlQ29tbWVudDcwNjQ4MjU0Mg==
created_at: 2020-10-10T03:58:58Z
updated_at: 2020-10-10T03:58:58Z
author_association: OWNER
body:
  The fix is live on the demo: https://latest.datasette.io/fixtures/compound_three_primary_keys?_sort_desc=pk2

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1014#issuecomment-706626796
issue_url: https://api.github.com/repos/simonw/datasette/issues/1014
node_id: MDEyOklzc3VlQ29tbWVudDcwNjYyNjc5Ng==
created_at: 2020-10-10T23:47:23Z
updated_at: 2020-10-10T23:48:26Z
author_association: OWNER
body:
  The GitHub API has these:

  `link: <https://api.github.com/repositories/107914493/commits?page=2>; rel="next", <https://api.github.com/repositories/107914493/commits?page=46>; rel="last"`

  WordPress API: https://css-tricks.com/wp-json/wp/v2/posts

  `link: <https://css-tricks.com/wp-json/wp/v2/posts?page=2>; rel="next"`

  It also has these:

  ```
  x-wp-total: 5162
  x-wp-totalpages: 517
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1014#issuecomment-706626971
issue_url: https://api.github.com/repos/simonw/datasette/issues/1014
node_id: MDEyOklzc3VlQ29tbWVudDcwNjYyNjk3MQ==
created_at: 2020-10-10T23:49:57Z
updated_at: 2020-10-10T23:49:57Z
author_association: OWNER
body:
  I won't offer a total-number header. That's expensive to calculate - no need to calculate it unless it's explicitly asked for.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1014#issuecomment-706627035
issue_url: https://api.github.com/repos/simonw/datasette/issues/1014
node_id: MDEyOklzc3VlQ29tbWVudDcwNjYyNzAzNQ==
created_at: 2020-10-10T23:50:40Z
updated_at: 2020-10-10T23:50:40Z
author_association: OWNER
body:
  First example page to add this to: `?_shape=array`, e.g. https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1014#issuecomment-706631006
issue_url: https://api.github.com/repos/simonw/datasette/issues/1014
node_id: MDEyOklzc3VlQ29tbWVudDcwNjYzMTAwNg==
created_at: 2020-10-11T00:36:43Z
updated_at: 2020-10-11T00:36:43Z
author_association: OWNER
body:
  Demo using [paginate-json](https://github.com/simonw/paginate-json):
  ```
  % paginate-json 'https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array' | jq '. | length' 
  https://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cd%2Cv
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ch%2Cr
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cl%2Cn
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cp%2Cj
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Ct%2Cf
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=a%2Cx%2Cb
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ca%2Cx
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ce%2Ct
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Ci%2Cp
  http://latest.datasette.io/fixtures/compound_three_primary_keys.json?_shape=array&_next=b%2Cm%2Cl
  1001
  ```
  New documentation: https://docs.datasette.io/en/latest/json_api.html#pagination

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-706735200
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcwNjczNTIwMA==
created_at: 2020-10-11T17:02:11Z
updated_at: 2020-10-11T17:14:51Z
author_association: OWNER
body:
  Since the total count can be expensive to calculate, I'm inclined to make that an opt-in extra - maybe `?_extra=count`.

  Based on that, the default JSON shape could look something like this:

  ```json
  {
    "rows": [{"id": 1}, {"id": 2}],
    "next": "2",
    "next_url": "/db/table?_next=2"
  }
  ```
  And with `?_extra=count`:
  ```json
  {
    "rows": [{"id": 1}, {"id": 2}],
    "next": "2",
    "next_url": "/db/table?_next=2",
    "count": 31
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-706735280
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcwNjczNTI4MA==
created_at: 2020-10-11T17:03:01Z
updated_at: 2020-10-11T17:03:01Z
author_association: OWNER
body:
  Should that default also include `"columns"` as a list of strings? That would be duplicate data of the keys in the `"rows"` list of objects, and I've never found myself wanting it in my own code - so I'm going to say no.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-706735341
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcwNjczNTM0MQ==
created_at: 2020-10-11T17:03:29Z
updated_at: 2020-10-11T17:15:34Z
author_association: OWNER
body:
  Maybe `.jsonfull` becomes a new renderer that returns ALL of the defined `?_extra=` blocks.

  Or... `?_extra=all` turns on ALL of the available information blocks (some of which can come from plugins).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-706736541
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcwNjczNjU0MQ==
created_at: 2020-10-11T17:12:27Z
updated_at: 2020-10-11T17:12:27Z
author_association: OWNER
body:
  The core issue that I keep reconsidering is whether the default `.json` representation should be an object or a list.

  Arguments in favour of a list:

  - It's what I always want. Almost all of the code that I've written against the API myself uses `?_shape=array`.
  - It's really easy to use. You can pipe it to e.g. `sqlite-utils insert`, you can load it into JavaScript without thinking about it.

  Arguments against:

  - Nowhere to put pagination or total counts. I added pagination to the `link:` HTTP header in #1014 (inspired by the WordPress and GitHub APIs) but I haven't solved for total count, and there's other stuff that's useful like `"truncated": true` to indicate that more than 1000 results were returned and they were truncated.
  - An array is inherently non-extensible: if the root item is an object it's easy to add new features to it in a backwards-compatible way in the future. An array is a fixed format.

  But maybe that last point is a positive? It ensures the default `.json` format remains completely predictable forever.

  If `.json` DID default to an array of objects, the `?_shape=` argument could still be used to get back alternative formats.

  Maybe `.json?_extra=total` changes the shape of that default to be this instead:

  ```json
  {
    "rows": [{"id": 1}, {"id": 2}],
    "total": 104
  }
  ```

  The thing I care about most though is `next_url`. That could be provided like so:

  `.json?_extra=total&_extra=next` - alternative syntax `.json?_extra=total,next`:

  ```json
  {
    "rows": [{"id": 1}, {"id": 2}],
    "total": 104,
    "next": "2",
    "next_url": "/db/table.json?_extra=total&_extra=next&_next=2"
  }
  ```
  This is feeling a bit verbose for a common combination though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-706738020
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcwNjczODAyMA==
created_at: 2020-10-11T17:23:18Z
updated_at: 2020-10-11T17:23:48Z
author_association: OWNER
body:
  I'm going to prototype what it would look like if the default shape was a list of objects and `?_extra=` turns that into an object with a `rows` key, in a plugin. As a separate extension (maybe `.json-preview`).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-706740250
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc0MDI1MA==
created_at: 2020-10-11T17:40:48Z
updated_at: 2020-10-11T17:43:07Z
author_association: OWNER
body:
  Building this plugin reminded me of an oddity of the `register_output_renderer()` plugin hook: one of the arguments that can be passed to it is `data`, which is the default internal data structure created by Datasette - but I deliberately avoided documenting that on https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette because it's not a stable interface.

  That's not ideal. I'd like custom renderers to be able to access this data to get at things like suggested facets, on an opt-in basis.

  So maybe that kind of stuff is re-implemented as "extras" which are awaitable callables - then renderer plugins can call the extras that they need to as part of their execution.

  To illustrate the problem (in this case the need to access `next_url`) here's my first prototype of the plugin:
  ```python
  from datasette import hookimpl
  from datasette.utils.asgi import Response


  @hookimpl
  def register_output_renderer(datasette):
      return {
          "extension": "json-preview",
          "render": json_preview,
      }


  def json_preview(data, columns, rows):
      next_url = data.get("next_url")
      headers = {}
      if next_url:
          headers["link"] = '<{}>; rel="next"'.format(next_url)
      return Response.json([dict(zip(columns, row)) for row in rows], headers=headers)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-706745236
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc0NTIzNg==
created_at: 2020-10-11T18:16:05Z
updated_at: 2020-10-11T18:16:05Z
author_association: OWNER
body:
  Here's the `datasette-json-preview` plugin I'll be using to experiment with different formats: https://github.com/simonw/datasette-json-preview

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1015#issuecomment-706756879
issue_url: https://api.github.com/repos/simonw/datasette/issues/1015
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc1Njg3OQ==
created_at: 2020-10-11T19:35:03Z
updated_at: 2020-10-11T19:35:03Z
author_association: OWNER
body:
  Since plugins are installed via pip this would require Datasette to be restarted. This StackOverflow thread looks relevant to that: https://stackoverflow.com/questions/11329917/restart-python-script-from-within-itself

  This recipe looks promising:
  ```python
  import os
  import sys
  import psutil
  import logging

  def restart_program():
      """Restarts the current program, with file objects and descriptors
         cleanup
      """

      try:
          p = psutil.Process(os.getpid())
          for handler in p.get_open_files() + p.connections():
              os.close(handler.fd)
      except Exception, e:
          logging.error(e)

      python = sys.executable
      os.execl(python, python, *sys.argv)
  ```
  https://docs.python.org/3/library/os.html#os.execl says about `os.execl`:

  > These functions all execute a new program, replacing the current process; they do not return. On Unix, the new executable is loaded into the current process, and will have the same process id as the caller

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776180
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc3NjE4MA==
created_at: 2020-10-11T22:17:55Z
updated_at: 2020-10-11T22:17:55Z
author_association: MEMBER
body:
  We could even do server-side thumbnailing for some of these images, but I'm inclined to serve up the full size ones and set a width on the image element based on the `width` attribute on `<en-media>`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776242
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc3NjI0Mg==
created_at: 2020-10-11T22:18:30Z
updated_at: 2020-10-11T22:19:48Z
author_association: MEMBER
body:
  Alternatively, rather than relying on `datasette-media` this could base64-embed the images. `evernote-to-sqlite` could register itself as a Datasette plugin that knows how to do this.

  Maybe rename the column to `evernote_content` and register a render cell hook that knows how to rewrite those note bodies so that they are visible?

  Might need to feed them through Bleach too, just in case any nasty code can get into them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776447
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc3NjQ0Nw==
created_at: 2020-10-11T22:20:32Z
updated_at: 2020-10-11T22:20:32Z
author_association: MEMBER
body:
  Or... I could do this client-side. JavaScript that looks for `<en-media>` tags and fetches the data using `fetch()` wouldn't be too hard to write.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776680
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc3NjY4MA==
created_at: 2020-10-11T22:22:16Z
updated_at: 2020-10-11T22:22:16Z
author_association: MEMBER
body:
  Maybe the best way do this is with a custom route, `/-/evernote/note-id` - that way I can clean the HTML and resolve the other things in the `<en-note>` structure without using `render_cell()` and the like. My concern about using `render_cell()` is that it could lead to weird security problems when combined with `?sql=` queries.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706776808
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc3NjgwOA==
created_at: 2020-10-11T22:23:14Z
updated_at: 2020-10-11T22:23:14Z
author_association: MEMBER
body:
  ... but it's still important to be able to get to the rendered note directly from the browse notes `/evernote/notes` page. Maybe use a simple `render_cell()` hook that just knows how to generate the link to the rendered note page?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/4#issuecomment-706784028
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/4
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc4NDAyOA==
created_at: 2020-10-11T23:20:32Z
updated_at: 2020-10-11T23:20:32Z
author_association: MEMBER
body:
  I haven't done the FTS on OCR yet. I'm going to move that to another ticket because it requires more thought.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785086
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc4NTA4Ng==
created_at: 2020-10-11T23:28:50Z
updated_at: 2020-10-11T23:28:50Z
author_association: MEMBER
body:
  The XML for the OCR stuff is a bit weird. Currently I'm doing this to it:

  https://github.com/dogsheep/evernote-to-sqlite/blob/c33d7b043a45eb3e88676e5fa3ce31755199d9f8/evernote_to_sqlite/utils.py#L70-L78

  This can produce some odd results, for example:

  > Sure 'Sure, 'Sure. Sure, Sure. sure sure. sure ? If you If Yau [you live jive In m 1n an area devoid of natural wonders, wanders, wonders ? wonders wonders. your mind will be blown, blown' blown. blown ? -e i ? ,1 IL it ? at ? KY ? fl ft bat at

  Which came from this image:

  ![image](https://user-images.githubusercontent.com/9599/95692952-5dd7c880-0bde-11eb-939a-d10b800a4105.png)

  The XML for that is:

  ```xml
  <recoIndex docType="unknown" objType="image" objID="05ffb72b307bf495f064243c7099d94f" engineVersion="6.5.17.7" recoType="service" lang="en" objWidth="1000" objHeight="1504">
    <item x="68" y="75" w="104" h="37">
      <t w="60">Sure</t>
      <t w="52">'Sure,</t>
      <t w="47">'Sure.</t>
      <t w="33">Sure,</t>
      <t w="26">Sure.</t>
    </item>
    <item x="182" y="83" w="92" h="26">
      <t w="62">sure</t>
      <t w="58">sure.</t>
      <t w="46">sure ?</t>
    </item>
    <item x="69" y="132" w="107" h="45">
      <t w="81">If you</t>
      <t w="64">If Yau</t>
      <t w="31">[you</t>
    </item>
    <item x="186" y="132" w="67" h="35">
      <t w="85">live</t>
      <t w="51">jive</t>
    </item>
    <item x="263" y="140" w="36" h="27">
      <t w="82">In</t>
      <t w="56">m</t>
      <t w="53">1n</t>
    </item>
    <item x="309" y="140" w="53" h="27">
      <t w="82">an</t>
    </item>
    <item x="372" y="141" w="90" h="26">
      <t w="94">area</t>
    </item>
    <item x="472" y="132" w="138" h="35">
      <t w="85">devoid</t>
    </item>
    <item x="620" y="132" w="43" h="35">
      <t w="82">of</t>
    </item>
    <item x="68" y="190" w="137" h="35">
      <t w="87">natural</t>
    </item>
    <item x="215" y="190" w="187" h="39">
      <t w="57">wonders,</t>
      <t w="55">wanders,</t>
      <t w="52">wonders ?</t>
      <t w="45">wonders</t>
      <t w="42">wonders.</t>
    </item>
    <item x="410" y="198" w="98" h="36">
      <t w="88">your</t>
    </item>
    <item x="518" y="190" w="102" h="35">
      <t w="86">mind</t>
    </item>
    <item x="630" y="190" w="69" h="34">
      <t w="87">will</t>
    </item>
    <item x="709" y="190" w="55" h="35">
      <t w="82">be</t>
    </item>
    <item x="774" y="190" w="137" h="34">
      <t w="56">blown,</t>
      <t w="55">blown'</t>
      <t w="48">blown.</t>
      <t w="48">blown ?</t>
    </item>
    <item x="166" y="736" w="8" h="6">
      <t w="66">-e</t>
    </item>
    <item x="273" y="966" w="29" h="21">
      <t w="11">i ?</t>
    </item>
    <item x="281" y="1004" w="28" h="11">
      <t w="11">,1</t>
    </item>
    <item x="512" y="1083" w="10" h="7">
      <t w="10">IL</t>
    </item>
    <item x="29" y="1447" w="7" h="23">
      <t w="17">it ?</t>
      <t w="15">at ?</t>
      <t w="13">KY ?</t>
    </item>
    <item x="414" y="841" w="8" h="16">
      <t w="22">fl</t>
      <t w="20">ft</t>
      <t w="20">bat</t>
      <t w="19">at</t>
    </item>
  </recoIndex>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/6#issuecomment-706785201
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/6
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc4NTIwMQ==
created_at: 2020-10-11T23:29:39Z
updated_at: 2020-10-11T23:29:39Z
author_association: MEMBER
body:
  It looks to me like each of those `<item>` blocks has a number of guesses in order of confidence:
  ```xml
    <item x="215" y="190" w="187" h="39">
      <t w="57">wonders,</t>
      <t w="55">wanders,</t>
      <t w="52">wonders ?</t>
      <t w="45">wonders</t>
      <t w="42">wonders.</t>
    </item>
  ```
  So maybe the best approach here is to just take the first `t` element within each `item`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1016#issuecomment-706788010
issue_url: https://api.github.com/repos/simonw/datasette/issues/1016
node_id: MDEyOklzc3VlQ29tbWVudDcwNjc4ODAxMA==
created_at: 2020-10-11T23:50:39Z
updated_at: 2020-10-11T23:50:39Z
author_association: OWNER
body:
  For consistency can reuse the icon used on selected facets:

  <img width="721" alt="evernote__notes__8_rows_where_where_application-data____ACD4B543-4663-40B2-BAF5-666FA54D4204__and_source-url_is_not_blank_sorted_by_id" src="https://user-images.githubusercontent.com/9599/95693564-e0ae5280-0be1-11eb-881b-fa0cec8397f5.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1016#issuecomment-706821561
issue_url: https://api.github.com/repos/simonw/datasette/issues/1016
node_id: MDEyOklzc3VlQ29tbWVudDcwNjgyMTU2MQ==
created_at: 2020-10-12T02:33:52Z
updated_at: 2020-10-12T02:33:52Z
author_association: OWNER
body:
  Should this do the equivalent of clicking Apply for you? I don't think so - that wouldn't fit with how the rest of that block of controls works.

  But if it makes the row vanish it may not be clear to the user that they still need to click Apply.

  Easiest thing would be for it to select the "remove filter" column and blank out the boxes. I will try that first.

  The code can go in `table.js`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1016#issuecomment-706828375
issue_url: https://api.github.com/repos/simonw/datasette/issues/1016
node_id: MDEyOklzc3VlQ29tbWVudDcwNjgyODM3NQ==
created_at: 2020-10-12T03:01:58Z
updated_at: 2020-10-12T03:01:58Z
author_association: OWNER
body:
  Demo: https://latest.datasette.io/fixtures/facetable?_facet=created&created=2019-01-17+08%3A00%3A00&_facet=state&state=MI

  ![x-button](https://user-images.githubusercontent.com/9599/95700744-9b4b4e80-0bfc-11eb-85a3-6da82d9e0cbf.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/evernote-to-sqlite/issues/5#issuecomment-706834800
issue_url: https://api.github.com/repos/dogsheep/evernote-to-sqlite/issues/5
node_id: MDEyOklzc3VlQ29tbWVudDcwNjgzNDgwMA==
created_at: 2020-10-12T03:24:57Z
updated_at: 2020-10-16T20:16:28Z
author_association: MEMBER
body:
  Here's my first attempt at a plugin for this:
  ```python
  from datasette import hookimpl
  import jinja2

  START = "<en-note"
  END = "</en-note>"
  TEMPLATE = """
  <div style="max-width: 500px; white-space: normal; overflow-wrap: break-word;">{}</div>
  """.strip()

  EN_MEDIA_SCRIPT = """
  Array.from(document.querySelectorAll('en-media')).forEach(el => {
      let hash = el.getAttribute('hash');
      let type = el.getAttribute('type');
      let path = `/evernote/resources_data/${hash}.json?_shape=array`;
      fetch(path).then(r => r.json()).then(rows => {
          let b64 = rows[0].data.encoded;
          let data = `data:${type};base64,${b64}`;
          el.innerHTML = `<img style="max-width: 300px" src="${data}">`;
      });
  });
  """


  @hookimpl
  def render_cell(value, table):
      if not table:
          # Don't render content from arbitrary SQL queries, could be XSS hole
          return
      if not value or not isinstance(value, str):
          return
      value = value.strip()
      if value.startswith(START) and value.endswith(END):
          trimmed = value[len(START) : -len(END)]
          trimmed = trimmed.split(">", 1)[1]
          # Replace those horrible double newlines
          trimmed = trimmed.replace("<div><br /></div>", "<br>")
          return jinja2.Markup(TEMPLATE.format(trimmed))


  @hookimpl
  def extra_body_script():
      return EN_MEDIA_SCRIPT
  ```
  It works!

  It does however demonstrate that Evernote's "clip this webpage" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-706845950
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDcwNjg0NTk1MA==
created_at: 2020-10-12T04:07:28Z
updated_at: 2020-10-12T04:07:28Z
author_association: OWNER
body:
  Next step is to design where these will go and what they will look like - in particular the navigation links that lead to plugins such as `datasette-upload-csvs` and `datasette-configure-fts` and `datasette-edit-schema`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1017#issuecomment-707123799
issue_url: https://api.github.com/repos/simonw/datasette/issues/1017
node_id: MDEyOklzc3VlQ29tbWVudDcwNzEyMzc5OQ==
created_at: 2020-10-12T13:34:51Z
updated_at: 2020-10-12T13:34:51Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1017?src=pr&el=h1) Report
  > Merging [#1017](https://codecov.io/gh/simonw/datasette/pull/1017?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/acf07a67722aa74828744726187690b59d342494?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1017/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1017?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1017   +/-   ##
  =======================================
    Coverage   84.56%   84.56%           
  =======================================
    Files          28       28           
    Lines        3882     3882           
  =======================================
    Hits         3283     3283           
    Misses        599      599           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1017?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1017?src=pr&el=footer). Last update [acf07a6...f30d9da](https://codecov.io/gh/simonw/datasette/pull/1017?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1018#issuecomment-707125737
issue_url: https://api.github.com/repos/simonw/datasette/issues/1018
node_id: MDEyOklzc3VlQ29tbWVudDcwNzEyNTczNw==
created_at: 2020-10-12T13:38:35Z
updated_at: 2020-10-12T13:38:35Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1018?src=pr&el=h1) Report
  > Merging [#1018](https://codecov.io/gh/simonw/datasette/pull/1018?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/acf07a67722aa74828744726187690b59d342494?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1018/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1018?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1018   +/-   ##
  =======================================
    Coverage   84.56%   84.56%           
  =======================================
    Files          28       28           
    Lines        3882     3882           
  =======================================
    Hits         3283     3283           
    Misses        599      599           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1018?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1018?src=pr&el=footer). Last update [acf07a6...4b021be](https://codecov.io/gh/simonw/datasette/pull/1018?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-707332912
issue_url: https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8
node_id: MDEyOklzc3VlQ29tbWVudDcwNzMzMjkxMg==
created_at: 2020-10-12T20:35:06Z
updated_at: 2020-10-12T20:35:06Z
author_association: MEMBER
body:
  Shipped a fix for this in [swarm-to-sqlite 0.3.2](https://github.com/dogsheep/swarm-to-sqlite/releases/tag/0.3.2).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1019#issuecomment-708113989
issue_url: https://api.github.com/repos/simonw/datasette/issues/1019
node_id: MDEyOklzc3VlQ29tbWVudDcwODExMzk4OQ==
created_at: 2020-10-14T02:22:40Z
updated_at: 2020-10-14T02:22:40Z
author_association: OWNER
body:
  I've found myself wanting this too.

  It should respect the allow-sql permission so it doesn't display if the user isn't able to run custom SQL.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1019#issuecomment-708114636
issue_url: https://api.github.com/repos/simonw/datasette/issues/1019
node_id: MDEyOklzc3VlQ29tbWVudDcwODExNDYzNg==
created_at: 2020-10-14T02:24:56Z
updated_at: 2020-10-14T02:24:56Z
author_association: OWNER
body:
  So it would appear on this page: https://latest.datasette.io/fixtures/neighborhood_search#fragment-goes-here

  Probably as a blue "Edit SQL" link next to those buttons.

  ![575FB160-4F75-43B2-B2AB-D75E60F10477](https://user-images.githubusercontent.com/9599/95936143-c2835680-0d89-11eb-9cf3-06f86d923103.jpeg)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1019#issuecomment-708127294
issue_url: https://api.github.com/repos/simonw/datasette/issues/1019
node_id: MDEyOklzc3VlQ29tbWVudDcwODEyNzI5NA==
created_at: 2020-10-14T03:08:13Z
updated_at: 2020-10-14T03:08:13Z
author_association: OWNER
body:
  Maybe like this:

  <img width="1385" alt="fixtures__select_neighborhood__facet_cities_name__state_from_facetable_join_facet_cities_on_facetable_city_id___facet_cities_id_where_neighborhood_like_________text________order_by_neighborhood_" src="https://user-images.githubusercontent.com/9599/95938930-ce721700-0d8f-11eb-87c8-f67ea4e57d55.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1019#issuecomment-708128286
issue_url: https://api.github.com/repos/simonw/datasette/issues/1019
node_id: MDEyOklzc3VlQ29tbWVudDcwODEyODI4Ng==
created_at: 2020-10-14T03:11:33Z
updated_at: 2020-10-14T03:14:07Z
author_association: OWNER
body:
  Another edge-case: https://latest.datasette.io/fixtures/pragma_cache_size (`PRAGMA cache_size;`) isn't an allowed query usually, so linking to "Edit SQL" for it would link to an error page: https://latest.datasette.io/fixtures?sql=PRAGMA+cache_size%3B

  Can use `datasette.utils.validate_sql_select(sql)` to check for that - it raises `datasette.utils.InvalidSql` if there's a problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1019#issuecomment-708130290
issue_url: https://api.github.com/repos/simonw/datasette/issues/1019
node_id: MDEyOklzc3VlQ29tbWVudDcwODEzMDI5MA==
created_at: 2020-10-14T03:17:57Z
updated_at: 2020-10-14T03:17:57Z
author_association: OWNER
body:
  One last edge-case (I think) - magic parameters aren't supported for non-canned-queries, so if a query includes those it shouldn't show an Edit SQL link: https://latest.datasette.io/fixtures/magic_parameters

  I don't have a clean way of detecting if a query contains defined magic parameters, but I can instead do a dumb substring match for `:_` and skip showing the Edit SQL link if that is present.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1019#issuecomment-708139822
issue_url: https://api.github.com/repos/simonw/datasette/issues/1019
node_id: MDEyOklzc3VlQ29tbWVudDcwODEzOTgyMg==
created_at: 2020-10-14T03:53:21Z
updated_at: 2020-10-14T03:53:21Z
author_association: OWNER
body:
  Demos:
  - https://latest.datasette.io/fixtures/neighborhood_search has the link
  - https://latest.datasette.io/fixtures/neighborhood_search?text=ber has the link, and the link passes through the parameter that has already been entered
  - https://latest.datasette.io/fixtures/%F0%9D%90%9C%F0%9D%90%A2%F0%9D%90%AD%F0%9D%90%A2%F0%9D%90%9E%F0%9D%90%AC has the link
  - https://latest.datasette.io/fixtures/pragma_cache_size does not have the link (`PRAGMA` is not allowed in regular editable queries)
  - https://latest.datasette.io/fixtures/magic_parameters does not have the link (magic parameters)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1020#issuecomment-708669178
issue_url: https://api.github.com/repos/simonw/datasette/issues/1020
node_id: MDEyOklzc3VlQ29tbWVudDcwODY2OTE3OA==
created_at: 2020-10-14T21:26:37Z
updated_at: 2020-10-14T21:26:37Z
author_association: OWNER
body:
  One option: add an optional request=... parameter which can be passed the current request, and will use that to populate the mock request with the exception of the bits that are passed explicitly (like the path):

  ```python
  response = await datasette.client.get("/db/table.json", request=request)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1020#issuecomment-708669778
issue_url: https://api.github.com/repos/simonw/datasette/issues/1020
node_id: MDEyOklzc3VlQ29tbWVudDcwODY2OTc3OA==
created_at: 2020-10-14T21:27:58Z
updated_at: 2020-10-14T21:27:58Z
author_association: OWNER
body:
  Maybe these internal requests should have some kind of flag that lets the underlying code tell that it's being called internally.

  One option: add a `x-internal: 1` request header - and ensure that any requests from outside Datasette have that header stripped.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1020#issuecomment-708670392
issue_url: https://api.github.com/repos/simonw/datasette/issues/1020
node_id: MDEyOklzc3VlQ29tbWVudDcwODY3MDM5Mg==
created_at: 2020-10-14T21:29:22Z
updated_at: 2020-10-14T21:29:22Z
author_association: OWNER
body:
  I should also verify (and probably unit-test) that things like the `?_trace=1` mechanism work across the internal request boundary.

  `/-/permissions` appears to work across this boundary, but again a test would be useful confirmation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708693133
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/182
node_id: MDEyOklzc3VlQ29tbWVudDcwODY5MzEzMw==
created_at: 2020-10-14T22:26:26Z
updated_at: 2020-10-14T22:26:26Z
author_association: OWNER
body:
  I could use https://github.com/chardet/chardet to help here, though I'd rather not add it as a dependency (`sqlite-utils` has very few dependencies at the moment). I could add it as an optional dependency though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1022#issuecomment-708693231
issue_url: https://api.github.com/repos/simonw/datasette/issues/1022
node_id: MDEyOklzc3VlQ29tbWVudDcwODY5MzIzMQ==
created_at: 2020-10-14T22:26:41Z
updated_at: 2020-10-14T22:26:41Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1022?src=pr&el=h1) Report
  > Merging [#1022](https://codecov.io/gh/simonw/datasette/pull/1022?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/7f2edb5dd2074dce0090659021991695a984844b?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1022/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1022?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1022   +/-   ##
  =======================================
    Coverage   84.60%   84.60%           
  =======================================
    Files          28       28           
    Lines        3892     3892           
  =======================================
    Hits         3293     3293           
    Misses        599      599           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1022?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1022?src=pr&el=footer). Last update [7f2edb5...7cef70a](https://codecov.io/gh/simonw/datasette/pull/1022?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/163#issuecomment-708693290
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/163
node_id: MDEyOklzc3VlQ29tbWVudDcwODY5MzI5MA==
created_at: 2020-10-14T22:26:50Z
updated_at: 2020-10-14T22:27:48Z
author_association: OWNER
body:
  This could even be exposed as a feature of the CLI tool - similar to how `sqlite-transform` works: https://github.com/simonw/sqlite-transform

      $ sqlite-transform lambda my.db mytable mycolumn --code='str(value).upper()'

  Maybe `sqlite-utils` should absorb all of the functionality from `sqlite-transform` - having two separate tools doesn't necessarily make sense.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/163#issuecomment-708695369
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/163
node_id: MDEyOklzc3VlQ29tbWVudDcwODY5NTM2OQ==
created_at: 2020-10-14T22:32:39Z
updated_at: 2020-10-14T22:32:39Z
author_association: OWNER
body:
  `sqlite-utils convert my.db mytable mycolumn` could become the new `sqlite-transform`. The sub-command could take `--code` but could also take `--parsedate` and `--parsedatetime` options.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/182#issuecomment-708695761
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/182
node_id: MDEyOklzc3VlQ29tbWVudDcwODY5NTc2MQ==
created_at: 2020-10-14T22:33:55Z
updated_at: 2020-10-14T22:35:17Z
author_association: OWNER
body:
  For the moment I'm going to add a `--encoding` option and some code that catches `UnicodeDecodeError` and shows an error message that suggests using `--encoding`.

  That error message could detect if the `file` command is available and, if it is, suggest running `file filename.txt` to detect the character encoding.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-709497595
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDcwOTQ5NzU5NQ==
created_at: 2020-10-15T18:04:35Z
updated_at: 2020-10-15T18:12:15Z
author_association: OWNER
body:
  For the table actions: attaching it to a cog icon next to the table name could make sense.

  <img width="503" alt="data__faa-wildlife-strikes__26_rows_where_where_AIRPORT____OIL_RIG__sorted_by_rowid_and_Fix_table_name_in_spatialite_example_command_by_jsfenfen_·_Pull_Request__1022_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/96168896-45192c80-0ed6-11eb-9d90-6172a8346468.png">

  This is the column action icon at twice the size, color `#666`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-709498425
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDcwOTQ5ODQyNQ==
created_at: 2020-10-15T18:06:08Z
updated_at: 2020-10-15T18:06:08Z
author_association: OWNER
body:
  And for instance-level actions (linking to `datasette-import-csv` for example) an actions menu anchored against a burger-bar menu icon in the navigation bar.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-709499944
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDcwOTQ5OTk0NA==
created_at: 2020-10-15T18:09:06Z
updated_at: 2020-10-15T18:09:06Z
author_association: OWNER
body:
  I'm going to dedicate this issue to thinking about action menus. #981 added column action menus, others to build could be:

  - Table action menus (for things like configure FTS, edit schema)
  - Database action menus (import CSV file into this table)
  - Instance action menus (manage users, import CSV into this instance)
  - Column action menus (already there, plugins could include extract this column)
  - Row action menus?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-709500715
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUwMDcxNQ==
created_at: 2020-10-15T18:10:32Z
updated_at: 2020-10-15T18:10:32Z
author_association: OWNER
body:
  Row action menus are a question mark for me. Adding them to the table page itself could get really noisy - though maybe they're a vertical ... menu at the end of the row? They could also go on the `row.html` template, though that page is rarely used at the moment.

  I'm going to skip row actions for the moment and concentrate on the other four, which I know I have use-cases for.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/878#issuecomment-709502889
issue_url: https://api.github.com/repos/simonw/datasette/issues/878
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUwMjg4OQ==
created_at: 2020-10-15T18:14:34Z
updated_at: 2020-10-15T18:14:34Z
author_association: OWNER
body:
  The `BaseView` class does this for Datasette internals at the moment, but I'm not convinced it works as well as it could.

  I'd like to turn this into a class that is documented and available to plugins as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709505147
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUwNTE0Nw==
created_at: 2020-10-15T18:18:45Z
updated_at: 2020-10-15T18:50:50Z
author_association: OWNER
body:
  This needs querystring parameter design. Some options:

  * `?_sort_clause=cast(mycol+as+integer)` and `?_sort_clause_desc=cast(mycol+as+integer)` - allowing any expression. This would need to be disabled if arbitrary SQL was turned off, similar to the restrictions on `?_where=`.
  * `?_sort_numeric=mycol` and `?_sort_numeric_desc=mycol` - this would cast to `real` which would work on integer values as well. It could be allowed even when arbitrary SQL was disabled.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709505608
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUwNTYwOA==
created_at: 2020-10-15T18:19:35Z
updated_at: 2020-10-15T18:19:35Z
author_association: OWNER
body:
  I could even let plugins define new sort types. Imagine a plugin that enables this:

  `?_sort_date_desc=mycol` - where it knows how to handle specific date formats, or even uses `dateutil.parser.parse`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709505953
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUwNTk1Mw==
created_at: 2020-10-15T18:20:13Z
updated_at: 2020-10-15T18:20:13Z
author_association: OWNER
body:
  Sorting by date when the column has a junk date format in it is such a column need it should maybe ship in Datasette by default - though I've been trying to avoid adding heavy dependencies like `dateutil` if I can get away with it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709509635
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUwOTYzNQ==
created_at: 2020-10-15T18:24:57Z
updated_at: 2020-10-15T18:24:57Z
author_association: OWNER
body:
  This does feel like a weird plugin hook just because there aren't really THAT many different use-cases that plugins could solve. The ones I can think of are:

  - Sort numeric
  - Sort by parsed date
  - Sort by FTS rank

  Could this work if I just allow `_sort_clause=`?

  One possible solution for the no-arbitrary-SQL case: users can define sort orders in `metadata.json/yml`. So if you want to enable sort-by-distance without enabling arbitrary SQL you could add something like this:

  ```yaml
  databases:
    mydb:
      tables:
        museums:
          sort_clause: bm25(fts)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709510422
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUxMDQyMg==
created_at: 2020-10-15T18:26:25Z
updated_at: 2020-10-15T18:26:25Z
author_association: OWNER
body:
  There's something interesting about figuring out which sort options should be offered in the column actions menu.

  Two options:

  - Try to detect if a text column is all-integers or all-floats, either by scanning the entire table (if it's small enough) or by scanning the first X rows, where X might be the size of the first page or maybe the first 1,000 or similar.
  - Could also let users define this in `metadata.yml` for the table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709511399
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUxMTM5OQ==
created_at: 2020-10-15T18:28:09Z
updated_at: 2020-10-15T18:28:09Z
author_association: OWNER
body:
  The simplest solution would be for Python code to scan all of the visible values on the current page and show the column action for "sort by this numeric" based purely on that. I already do that in the JavaScript for "are there any blank values in the first page?" here:

  https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/static/table.js#L106-L118

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709513483
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUxMzQ4Mw==
created_at: 2020-10-15T18:31:56Z
updated_at: 2020-10-15T18:31:56Z
author_association: OWNER
body:
  I think the first version of this feature involves implementing `?_sort_numeric=col` and `?_sort_numeric_desc=col` plus the JavaScript to detect if those values should be shown in the column actions menu.

  One question: how to reflect that this is happening in the current sort UI. This menu here for example:

  <img width="440" alt="fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk3_descending" src="https://user-images.githubusercontent.com/9599/96171519-e35ac180-0ed9-11eb-9945-6780054e4ad5.png">

  And this interface: how should it indicate that a text is currently sorted numerically v.s. sorted alphabetically, and allow the user to switch from one to the other?

  <img width="794" alt="fixtures__sortable__201_rows_where_sorted_by_sortable_with_nulls_descending" src="https://user-images.githubusercontent.com/9599/96171618-04bbad80-0eda-11eb-946c-ed3b17a8c749.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709524123
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUyNDEyMw==
created_at: 2020-10-15T18:50:25Z
updated_at: 2020-10-15T18:50:25Z
author_association: OWNER
body:
  For the "Sort by X" select menu case... I could automatically expand that menu to contain extra options for "Sort numerically by X" for each TEXT column in the table. That's a pretty good option.

  For the action cog menu, I can add the extra options to the cog menu - and rely on the fact that the title of the page will say "Sorted numerically by colname descending".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709531343
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUzMTM0Mw==
created_at: 2020-10-15T19:03:12Z
updated_at: 2020-10-15T19:03:12Z
author_association: OWNER
body:
  The Sort by `<select>` menu needs a rethink: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/templates/table.html#L80-L87

  If it's going to include sort by numeric options it needs a different format - since `?_sort=colname` needs to also support `?_sort=numeric_colname` - but shouldn't clash with a weird column already named `numeric_colname`.

  Can I come up with a value syntax for this that is guaranteed not to clash with a weirdly named existing column?

  I think so. I could use `{"type": "numeric", "column": "mycolumn"}` as the value - then if a column has a name that is itself valid JSON (weird but possible) the `_sort=` value would be `{"type": "default", "column": "{\"type\": \"numeric\", \"name\": \"column\"}"}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709532369
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUzMjM2OQ==
created_at: 2020-10-15T19:05:07Z
updated_at: 2020-10-15T19:07:35Z
author_association: OWNER
body:
  Simpler option: `?_sort=` column values look like this:

  - `mycolumn` - for sort by column
  - `mycolumn$numeric` - for sort by column after cast to float
  - `mycolumn$morename$default` - for the edge case where the column name itself contains a $ symbol

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709534197
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUzNDE5Nw==
created_at: 2020-10-15T19:08:53Z
updated_at: 2020-10-15T19:17:55Z
author_association: OWNER
body:
  Even better solution: use URL encoding in the parameter details. This is consistent with how `?_next=` tokens work, e.g. `?_next=0.291861560261786%2Ce%2Cj`.

  So the format can be:

  - `mycolumn`
  - `urlencoded-mycolumn$castname`

  For most columns this will look like: `?_sort=score$numeric`

  For columns with a `$` in their name it will be `?_sort=score%24hasdollar$numeric`

  Problem: both `$` and `,` are usually URL encoded anyway. I need a character which isn't encoded by default, so that I can use its encoded form to show it is part of the column name and its un-encoded form to split the cast indicator.

  `_` is a candidate here - not encoded by default, but can be encoded as `%5F`.

  The other unreserved non-alphanumeric characters are `-`, `.`, `_`, `~`.

  Of these, `~` is least likely to show up in a column name. So I'll use that.

  - `mycolumn`
  - `mycolumn~numeric`
  - `mycolumn%7Ewith%7Etildes~numeric`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709539257
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTUzOTI1Nw==
created_at: 2020-10-15T19:19:29Z
updated_at: 2020-10-15T19:34:07Z
author_association: OWNER
body:
  Urgh this isn't going to work. `%7E~%7E` gets decoded as `~~~` so I wouldn't be able to tell the difference.

  I could use double-percentage-encoding here instead. I feel like there's a simpler solution that I'm missing (and that may well be in use within Datasette already, I'm not doing great thinking this morning).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709546976
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU0Njk3Ng==
created_at: 2020-10-15T19:35:55Z
updated_at: 2020-10-15T19:36:38Z
author_association: OWNER
body:
  Much easier solution: if the suffix is `~numeric` then treat it as the column name sorted numerically. If the suffix is missing OR the suffix is `~default`, sort without casting. Only add the `~default` suffix if the column name itself contains at least one `~` symbol.

  Using `~` because it doesn't need to be URL-encoded.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709562940
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU2Mjk0MA==
created_at: 2020-10-15T20:08:16Z
updated_at: 2020-10-15T20:08:16Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/views/table.py#L485-L510

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709569951
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU2OTk1MQ==
created_at: 2020-10-15T20:23:02Z
updated_at: 2020-10-15T20:23:02Z
author_association: OWNER
body:
  Something to watch out for: `""` empty strings cast to `0.0`:

  `select cast("100" as real), "100", cast(null as real), cast("" as real)`

  cast("100" as real) | "100" | cast(null as real) | cast("" as real)
  -- | -- | -- | --
  100.0 | 100 |   | 0.0

  https://latest.datasette.io/fixtures?sql=select+cast%28%22100%22+as+real%29%2C+%22100%22%2C+cast%28null+as+real%29%2C+cast%28%22%22+as+real%29

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709572425
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU3MjQyNQ==
created_at: 2020-10-15T20:28:18Z
updated_at: 2020-10-15T20:28:18Z
author_association: OWNER
body:
  Also need to rethink this template logic that decides if to show a column as sorted or not:
  https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/templates/_table.html#L10-L14

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/894#issuecomment-709575818
issue_url: https://api.github.com/repos/simonw/datasette/issues/894
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU3NTgxOA==
created_at: 2020-10-15T20:35:03Z
updated_at: 2020-10-15T20:35:03Z
author_association: OWNER
body:
  Prototype so far:
  ```diff
  diff --git a/datasette/views/table.py b/datasette/views/table.py
  index ea11a51..d61f8bd 100644
  --- a/datasette/views/table.py
  +++ b/datasette/views/table.py
  @@ -497,17 +497,32 @@ class TableView(RowTableShared):
           if sort and sort_desc:
               raise DatasetteError("Cannot use _sort and _sort_desc at the same time")
   
  +        def parse_sort(sort):
  +            if "~" in sort:
  +                if sort.endswith("~default"):
  +                    col = sort.rsplit("~", 1)[0]
  +                    return col, escape_sqlite(col)
  +                elif sort.endswith("~numeric"):
  +                    col = sort.rsplit("~", 1)[0]
  +                    return col, "cast(nullif({}, '') as real)".format(escape_sqlite(col))
  +                else:
  +                    return sort, escape_sqlite(sort)
  +            else:
  +                return sort, escape_sqlite(sort)
  +
           if sort:
  -            if sort not in sortable_columns:
  -                raise DatasetteError("Cannot sort table by {}".format(sort))
  +            sort_column, sort_clause = parse_sort(sort)
  +            if sort_column not in sortable_columns:
  +                raise DatasetteError("Cannot sort table by {}".format(sort_column))
   
  -            order_by = escape_sqlite(sort)
  +            order_by = sort_clause
   
           if sort_desc:
  -            if sort_desc not in sortable_columns:
  -                raise DatasetteError("Cannot sort table by {}".format(sort_desc))
  +            sort_column, sort_clause = parse_sort(sort_desc)
  +            if sort_column not in sortable_columns:
  +                raise DatasetteError("Cannot sort table by {}".format(sort_column))
   
  -            order_by = "{} desc".format(escape_sqlite(sort_desc))
  +            order_by = "{} desc".format(sort_clause)
   
           from_sql = "from {table_name} {where}".format(
               table_name=escape_sqlite(table),
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709589297
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU4OTI5Nw==
created_at: 2020-10-15T21:04:31Z
updated_at: 2020-10-15T21:04:31Z
author_association: OWNER
body:
  I think nginx or Apache would be the best tools for this. I'm inclined to try with nginx first since I know it better.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709590337
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU5MDMzNw==
created_at: 2020-10-15T21:06:24Z
updated_at: 2020-10-15T21:07:19Z
author_association: OWNER
body:
  From https://stackoverflow.com/questions/32549684/nginx-proxy-and-remove-proxy-pass-prefix/32550251 it looks like the config I should use is:
  ```
  server {
      listen 80;
      server_name example.com;

      location /datasette/ {
          proxy_pass              http://127.0.0.1:8001;
          proxy_set_header        X-Real-IP $remote_addr;
          proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header        X-Forwarded-Proto $scheme;
          proxy_read_timeout      90;
      }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709590941
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU5MDk0MQ==
created_at: 2020-10-15T21:07:47Z
updated_at: 2020-10-15T21:07:47Z
author_association: OWNER
body:
  On macOS I ran `brew install nginx`. I'm going to try running it on port 8000 so I don't have to run it as root.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709595960
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU5NTk2MA==
created_at: 2020-10-15T21:18:14Z
updated_at: 2020-10-15T21:18:14Z
author_association: OWNER
body:
  Typing `nginx` starts it running as a daemon listening on port `http-alt` aka 8080. It uses the config file from ` /usr/local/etc/nginx/nginx.conf`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709597589
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU5NzU4OQ==
created_at: 2020-10-15T21:21:53Z
updated_at: 2020-10-15T21:23:25Z
author_association: OWNER
body:
  Here's a recipe for running nginx against a custom config file: https://gist.github.com/simonw/35f0ebf9c1d6df158759

  ```
  daemon off;

  events {
    worker_connections  1024;
  }

  http {
    access_log /dev/stdout;
    error_log /dev/stderr;

    types {
      text/html                             html htm shtml;
      text/css                              css;
      image/gif                             gif;
      image/jpeg                            jpeg jpg;
      application/javascript                js;
    }

    server {
      listen 8002;
      index index.html;
      root  app;
    }
  }
  ```
  ```
  nginx -p `pwd` -c `pwd`/nginx.conf
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709598324
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTU5ODMyNA==
created_at: 2020-10-15T21:23:33Z
updated_at: 2020-10-15T21:26:55Z
author_association: OWNER
body:
  Combining these two examples, here's the config file I am going to use for this. I'll save this as `nginx.conf`:
  ```
  daemon off;

  events {
    worker_connections  1024;
  }

  http {
    server {
      listen 8000;

      location /datasette {
          proxy_pass              http://127.0.0.1:8001;
          proxy_set_header        X-Real-IP $remote_addr;
          proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      }
    }
  }
  ```
  Then start the server with:
  ```
  nginx -p `pwd` -c `pwd`/nginx.conf
  ```
  And start Datasette like this:
  ```
  datasette fixtures.db --config base_url:/datasette/
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709600335
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYwMDMzNQ==
created_at: 2020-10-15T21:28:02Z
updated_at: 2020-10-15T22:25:43Z
author_association: OWNER
body:
  This is working OK so far:

  <img width="897" alt="fixtures__compound_three_primary_keys__1_001_rows_where_sorted_by_pk2" src="https://user-images.githubusercontent.com/9599/96187878-96371980-0ef2-11eb-875e-5b1ae9b445fd.png">

  I'll try crawling it with `wget -r` to see if I get any errors.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709622973
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYyMjk3Mw==
created_at: 2020-10-15T22:27:31Z
updated_at: 2020-10-15T22:27:31Z
author_association: OWNER
body:
  Here's how I tested it:
  ```
  time wget -r 'http://localhost:8000/datasette/' 2>&1 | grep -i -C 5 "failed\|error" > /tmp/errors.txt
  ```
  This wrote out any errors (plus context) to the `errors.txt` log - and reported that the full crawl took 33s.

  Here's what I got in `errors.txt`:
  ```

       0K .                                                      71.6M=0s

  2020-10-15 15:23:09 (71.6 MB/s) - ‘localhost:8000/datasette/index.html’ saved [1276]

  Loading robots.txt; please ignore errors.
  --2020-10-15 15:23:09--  http://localhost:8000/robots.txt
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 404 Not Found
  --
  --2020-10-15 15:23:09--  http://localhost:8000/robots.txt
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 404 Not Found
  2020-10-15 15:23:09 ERROR 404: Not Found.

  --2020-10-15 15:23:09--  http://localhost:8000/datasette/-/static/app.css?b576be
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 200 OK
  Length: 8563 (8.4K) [text/css]
  --
  --
  2020-10-15 15:23:13 (7.90 MB/s) - ‘localhost:8000/datasette/fixtures/primary_key_multiple_columns_explicit_label.json?_shape=object’ saved [58]

  --2020-10-15 15:23:13--  http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 404 Not Found
  2020-10-15 15:23:13 ERROR 404: Not Found.

  --2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 200 OK
  Length: unspecified [text/html]
  --
  --
  2020-10-15 15:23:13 (84.3 MB/s) - ‘localhost:8000/datasette/fixtures/roadside_attractions.json?_shape=object’ saved [619]

  --2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures/%5C%22https://www.openstreetmap.org/copyright%5C%22
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 404 Not Found
  2020-10-15 15:23:13 ERROR 404: Not Found.

  --2020-10-15 15:23:13--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+text1%2C+text2%2C+%5Bname+with+.+and+spaces%5D+from+searchable+order+by+pk+limit+101
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 200 OK
  Length: unspecified [text/html]
  --
  --
  2020-10-15 15:23:14 (28.6 MB/s) - ‘localhost:8000/datasette/fixtures/searchable_view_configured_by_metadata.json?_shape=array&_nl=on’ saved [180]

  --2020-10-15 15:23:14--  http://localhost:8000/
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 404 Not Found
  2020-10-15 15:23:14 ERROR 404: Not Found.

  --2020-10-15 15:23:14--  http://localhost:8000/datasette/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101&_hide_sql=1
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 200 OK
  Length: unspecified [text/html]
  --
  --
  2020-10-15 15:23:21 (64.1 MB/s) - ‘localhost:8000/datasette/fixtures.csv?sql=select+pk,+name,+address,+latitude,+longitude+from+roadside_attractions+order+by+pk+limit+101&_size=max’ saved [403]

  --2020-10-15 15:23:21--  http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 404 Not Found
  2020-10-15 15:23:21 ERROR 404: Not Found.

  --2020-10-15 15:23:21--  http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+desc+limit+101
  Reusing existing connection to localhost:8000.
  HTTP request sent, awaiting response... 200 OK
  Length: unspecified [text/html]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1024#issuecomment-709625063
issue_url: https://api.github.com/repos/simonw/datasette/issues/1024
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYyNTA2Mw==
created_at: 2020-10-15T22:33:22Z
updated_at: 2020-10-15T22:33:22Z
author_association: OWNER
body:
  Of those errors...

  `http://localhost:8000/robots.txt` 404 is fine.

  `http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22` looks to me like a `wget` parsing bug where it got confused by this JavaScript:

  ```
  window.DATASETTE_CLUSTER_MAP_TILE_LAYER_OPTIONS = {"maxZoom": 19, "detectRetina": true, "attribution": "&copy; <a href=\"https://www.openstreetmap.org/copyright\">OpenStreetMap</a> contributors"};</script>
  ```

  `http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js` is a real bug. It's a bug in `datasette-cluster-map` but also requires me to solve #988 - mechanism for plugins to construct URLs that obey `base_url`.

  I'm not sure why I'm getting a hit to `http://localhost:8000/` since I wouldn't expect to link to `/` anywhere.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/865#issuecomment-709626786
issue_url: https://api.github.com/repos/simonw/datasette/issues/865
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYyNjc4Ng==
created_at: 2020-10-15T22:38:38Z
updated_at: 2020-10-15T22:38:38Z
author_association: OWNER
body:
  I managed to recreate proxying using `nginx` in #1024 - but I could not replicate this bug. I did NOT use `ProxyPassReverse` though. I think that may be what caused the problem.

  I'll add a section to the documentation about this shortly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1025#issuecomment-709629920
issue_url: https://api.github.com/repos/simonw/datasette/issues/1025
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYyOTkyMA==
created_at: 2020-10-15T22:48:20Z
updated_at: 2020-10-15T22:48:20Z
author_association: OWNER
body:
  Also these:
  ```
  datasette % git grep '"/' -- '*.html' ':(exclude)*/patterns.html'
  datasette/templates/allow_debug.html:<form action="/-/allow-debug" method="get">
  datasette/templates/base.html:        <form action="/-/logout" method="post">
  datasette/templates/error.html:        <a href="/">home</a>
  datasette/templates/logout.html:<form action="/-/logout" method="post">
  datasette/templates/messages_debug.html:<form action="/-/messages" method="post">
  datasette/templates/query.html:        <a href="/">home</a> /
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1025#issuecomment-709632136
issue_url: https://api.github.com/repos/simonw/datasette/issues/1025
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzMjEzNg==
created_at: 2020-10-15T22:55:44Z
updated_at: 2020-10-15T22:55:44Z
author_association: OWNER
body:
  It looks like there are also some generated redirect responses that don't take `base_url` into account:
  ```
  datasette % git grep '"/' -- '*.py' ':(exclude)*test_*.py' ':(exclude)datasette/app.py'
  datasette/_version.py:        for i in cfg.versionfile_source.split("/"):
  datasette/utils/asgi.py:        path="/",
  datasette/views/base.py:            should_redirect = "/{}-{}".format(name, expected)
  datasette/views/base.py:                should_redirect += "/" + urllib.parse.quote_plus(kwargs["table"])
  datasette/views/base.py:                should_redirect += "/" + kwargs["pk_path"]
  datasette/views/special.py:            response = Response.redirect("/")
  datasette/views/special.py:            return Response.redirect("/")
  datasette/views/special.py:        response = Response.redirect("/")
  datasette/views/special.py:        return Response.redirect("/")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1025#issuecomment-709632314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1025
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzMjMxNA==
created_at: 2020-10-15T22:56:25Z
updated_at: 2020-10-15T22:56:34Z
author_association: OWNER
body:
  That `utils/asgi.py` line is the default path for setting cookies. That should likely take `base_url` into account too:
  https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/utils/asgi.py#L331-L342

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/900#issuecomment-709632765
issue_url: https://api.github.com/repos/simonw/datasette/issues/900
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzMjc2NQ==
created_at: 2020-10-15T22:57:55Z
updated_at: 2020-10-15T22:57:55Z
author_association: OWNER
body:
  I believe this particular bug has been fixed, based on my testing here: https://github.com/simonw/datasette/issues/1024#issuecomment-709622973

  Please re-open the ticket if you are still experiencing it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/865#issuecomment-709633080
issue_url: https://api.github.com/repos/simonw/datasette/issues/865
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzMzA4MA==
created_at: 2020-10-15T22:58:51Z
updated_at: 2020-10-15T22:58:51Z
author_association: OWNER
body:
  It looks like there are places where Datasette might return a redirect that doesn't take `base_url` into account - I'm planning on fixing those here, after which I think `ProxyPassReverse` should no longer be necessary. https://github.com/simonw/datasette/issues/1025#issuecomment-709632136

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-709634261
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzNDI2MQ==
created_at: 2020-10-15T23:02:43Z
updated_at: 2020-10-15T23:02:43Z
author_association: OWNER
body:
  Here's the current implementation of `database_url` - on the `BaseView` class, but only because it needs access to a `datasette` instance (to read `base_url`):
  https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/datasette/views/base.py#L102-L108

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-709635021
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzNTAyMQ==
created_at: 2020-10-15T23:05:11Z
updated_at: 2020-10-15T23:05:11Z
author_association: OWNER
body:
  I think this should be a family of functions:

  - `instance_url()` - the root URL of the instance (usually `/` unless `base_url` is set)
  - `database_url(database_name)` - already got this
  - `table_url(database_name, table_name)`
  - `row_url(database_name, table_name, row)` - not sure about this one. The idea would be for `row` to be correctly turned into a URL by introspecting the primary keys for that table, then pulling those values out of the SQLite `row` object. Might not be necessary though.

  I also need a way for plugins to link to e.g. `/-/configure-fts` - or even `/-/configure-fts/database-name/table-name`. What should that look like?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-709635276
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzNTI3Ng==
created_at: 2020-10-15T23:05:54Z
updated_at: 2020-10-15T23:05:54Z
author_association: OWNER
body:
  Could have `instance_url()` take an optional path argument which is then turned into the correct path.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1026#issuecomment-709636372
issue_url: https://api.github.com/repos/simonw/datasette/issues/1026
node_id: MDEyOklzc3VlQ29tbWVudDcwOTYzNjM3Mg==
created_at: 2020-10-15T23:09:34Z
updated_at: 2020-10-15T23:09:34Z
author_association: OWNER
body:
  I'm inclined to say that internal requests should ignore `base_url` - since that seems like the right thing for plugins that need to access default Datasette APIs.

  The one catch here is plugins that might want to proxy the current incoming URL for some reason - where that incoming `request.path` could include the `base_url`.

  Actually those should be fine - because it will have been stripped off earlier:

  https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/app.py#L963-L968

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1027#issuecomment-709646865
issue_url: https://api.github.com/repos/simonw/datasette/issues/1027
node_id: MDEyOklzc3VlQ29tbWVudDcwOTY0Njg2NQ==
created_at: 2020-10-15T23:47:08Z
updated_at: 2020-10-15T23:47:08Z
author_association: OWNER
body:
  It should cover both nginx and Apache. nginx config is here: https://github.com/simonw/datasette/issues/1024#issuecomment-709598324

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1027#issuecomment-709647525
issue_url: https://api.github.com/repos/simonw/datasette/issues/1027
node_id: MDEyOklzc3VlQ29tbWVudDcwOTY0NzUyNQ==
created_at: 2020-10-15T23:49:51Z
updated_at: 2020-10-15T23:51:39Z
author_association: OWNER
body:
  I'll install Apache on macOS to figure this out using https://formulae.brew.sh/formula/httpd

  `brew install httpd` output this at the end:

  ```
  ==> httpd
  DocumentRoot is /usr/local/var/www.

  The default ports have been set in /usr/local/etc/httpd/httpd.conf to 8080 and in
  /usr/local/etc/httpd/extra/httpd-ssl.conf to 8443 so that httpd can run without sudo.

  To have launchd start httpd now and restart at login:
    brew services start httpd
  Or, if you don't want/need a background service you can just run:
    apachectl start
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705624
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/186
node_id: MDEyOklzc3VlQ29tbWVudDcwOTcwNTYyNA==
created_at: 2020-10-16T03:14:39Z
updated_at: 2020-10-16T03:14:39Z
author_association: OWNER
body:
  How should this work with extractions covering multiple columns?

  If there's a single column then it makes sense that a `null` value would not be extracted into the lookup table, but would instead become stay as `null`.

  For a multiple column extraction, provided at least one of those columns is not null It should map to a record in the lookup table. Only if ALL of the extracted columns are null should the lookup value stay null.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705885
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/186
node_id: MDEyOklzc3VlQ29tbWVudDcwOTcwNTg4NQ==
created_at: 2020-10-16T03:15:39Z
updated_at: 2020-10-16T03:15:39Z
author_association: OWNER
body:
  The alternative solution here would be that a single `null` value DOES get extracted. To implement this I would need to add some logic that uses `is null` instead of `=`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706065
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/186
node_id: MDEyOklzc3VlQ29tbWVudDcwOTcwNjA2NQ==
created_at: 2020-10-16T03:16:22Z
updated_at: 2020-10-16T03:16:22Z
author_association: OWNER
body:
  Either way I think I'm going to need to add some SQL which uses `where a = b or (a is null and b is null)`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706260
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/186
node_id: MDEyOklzc3VlQ29tbWVudDcwOTcwNjI2MA==
created_at: 2020-10-16T03:17:02Z
updated_at: 2020-10-16T03:17:17Z
author_association: OWNER
body:
  Actually I think this should be an option to `.extract()` which controls if nulls are extracted or left alone. Maybe called `extract_nulls=True/False`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710178871
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/182
node_id: MDEyOklzc3VlQ29tbWVudDcxMDE3ODg3MQ==
created_at: 2020-10-16T16:27:39Z
updated_at: 2020-10-16T16:28:14Z
author_association: OWNER
body:
  The file is opened for me by `click.File()`, which also handles things like `-` for stdin. But i neee to be able to switch the encoding used to read from that based on the `--encoding` option.

  I think the way to do that is to open the file in binary mode and then wrap it in a codec reader:
  ```python
  fp = codecs.getreader(encoding)(fp)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/186#issuecomment-710198162
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/186
node_id: MDEyOklzc3VlQ29tbWVudDcxMDE5ODE2Mg==
created_at: 2020-10-16T16:41:00Z
updated_at: 2020-10-16T16:41:00Z
author_association: OWNER
body:
  Failing test:
  ```python
  def test_extract_null_values(fresh_db):
      fresh_db["species"].insert({"id": 1, "species": "Wolf"}, pk="id")
      fresh_db["individuals"].insert_all(
          [
              {"id": 10, "name": "Terriana", "species": "Fox"},
              {"id": 11, "name": "Spenidorm", "species": None},
              {"id": 12, "name": "Grantheim", "species": "Wolf"},
              {"id": 13, "name": "Turnutopia", "species": None},
              {"id": 14, "name": "Wargal", "species": "Wolf"},
          ],
          pk="id",
      )
      fresh_db["individuals"].extract("species")
      assert fresh_db["species"].schema == (
          "CREATE TABLE [species] (\n"
          "   [id] INTEGER PRIMARY KEY,\n"
          "   [species] TEXT\n"
          ")"
      )
      assert fresh_db["individuals"].schema == (
          'CREATE TABLE "individuals" (\n'
          "   [id] INTEGER PRIMARY KEY,\n"
          "   [name] TEXT,\n"
          "   [species_id] INTEGER,\n"
          "   FOREIGN KEY(species_id) REFERENCES species(id)\n"
          ")"
      )
      assert list(fresh_db["species"].rows) == [
          {"id": 1, "species": "Wolf"},
          {"id": 2, "species": "Fox"},
      ]
      assert list(fresh_db["individuals"].rows) == [
          {"id": 10, "name": "Terriana", "species_id": 2},
          {"id": 11, "name": "Spenidorm", "species_id": None},
          {"id": 12, "name": "Grantheim", "species_id": 1},
          {"id": 13, "name": "Turnutopia", "species_id": None},
          {"id": 14, "name": "Wargal", "species_id": 1},
      ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA==
created_at: 2020-10-16T18:08:52Z
updated_at: 2020-10-16T18:09:21Z
author_association: OWNER
body:
  The new `.extract()` method can handle multiple columns:

  https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA==
created_at: 2020-10-16T18:15:31Z
updated_at: 2020-10-16T18:15:31Z
author_association: OWNER
body:
  Using a tuple would work:
  ```python
  fresh_db.table("tree", extracts=[("common_name", "latin_name")])
  ```
  Or to define a custom name:
  ```python
  fresh_db.table("tree", extracts={("common_name", "latin_name"): "names"})
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ==
created_at: 2020-10-16T18:18:05Z
updated_at: 2020-10-16T18:18:05Z
author_association: OWNER
body:
  I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg==
created_at: 2020-10-16T18:18:48Z
updated_at: 2020-10-16T18:18:48Z
author_association: OWNER
body:
  I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ==
created_at: 2020-10-16T18:34:26Z
updated_at: 2020-10-16T18:34:50Z
author_association: OWNER
body:
  Here's the most complex example of `.extracts()`:
  ```python
  db["Trees"].extract(
      ["CommonName", "LatinName"],
      table="Species",
      fk_column="species_id",
      rename={"CommonName": "name", "LatinName": "latin"}
  )
  ```
  Resulting in:
  ```sql
  CREATE TABLE [Species] (
      [id] INTEGER PRIMARY KEY,
      [name] TEXT,
      [latin] TEXT
  )
  ```
  From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA==
created_at: 2020-10-16T18:35:57Z
updated_at: 2020-10-16T18:36:39Z
author_association: OWNER
body:
  If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it:
  ```python
  fresh_db.table("tree", extracts=[Extract(
      columns=("CommonName", "LatinName"),
      table="Species",
      fk_column="species_id",
      rename={"CommonName": "name", "LatinName": "latin"}
  )])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA==
created_at: 2020-10-16T18:37:10Z
updated_at: 2020-10-16T18:37:10Z
author_association: OWNER
body:
  But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA==
created_at: 2020-10-16T18:38:21Z
updated_at: 2020-10-16T18:38:21Z
author_association: OWNER
body:
  I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/58
node_id: MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==
created_at: 2020-10-16T18:39:31Z
updated_at: 2020-10-16T18:39:31Z
author_association: OWNER
body:
  I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/48
node_id: MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ==
created_at: 2020-10-16T18:41:06Z
updated_at: 2020-10-16T18:41:06Z
author_association: OWNER
body:
  I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/69
node_id: MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==
created_at: 2020-10-16T18:42:48Z
updated_at: 2020-10-16T18:42:48Z
author_association: OWNER
body:
  Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d

  I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/136#issuecomment-710428802
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/136
node_id: MDEyOklzc3VlQ29tbWVudDcxMDQyODgwMg==
created_at: 2020-10-16T18:56:47Z
updated_at: 2020-10-16T18:56:47Z
author_association: OWNER
body:
  To keep the code cleaner, I'm tempted to support this instead:

      --load-extension=spatialite

  Where `spatialite` is a special shortcut value that triggers a search for that module in known locations.

  Users could still load a module in a file called `spatialite` in the current directory using:

      --load-extension=./spatialite

  In fact, `--load-extension=spatialite` could handle that case too by always checking for a file called `spatialite` before attempting to search for it in known locations.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/89
node_id: MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==
created_at: 2020-10-16T19:17:27Z
updated_at: 2020-10-16T19:17:50Z
author_association: OWNER
body:
  I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works:
  ```python
  fresh_db.table("tree", extracts=[Extract(
      columns=("CommonName", "LatinName"),
      table="Species",
      fk_column="species_id",
      rename={"CommonName": "name", "LatinName": "latin"}
  )])
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/49
node_id: MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA==
created_at: 2020-10-16T19:18:19Z
updated_at: 2020-10-16T19:18:19Z
author_association: OWNER
body:
  Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-710487083
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcxMDQ4NzA4Mw==
created_at: 2020-10-16T19:34:10Z
updated_at: 2020-10-19T17:39:04Z
author_association: OWNER
body:
  Alternatively, I could expose a single object that knows how to construct all kinds of URLs. Something like this:
  ```
  {{ urls.instance() }}
  {{ urls.database(database_name) }}
  {{ urls.table(database_name, table_name) }}
  etc
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/991#issuecomment-710694144
issue_url: https://api.github.com/repos/simonw/datasette/issues/991
node_id: MDEyOklzc3VlQ29tbWVudDcxMDY5NDE0NA==
created_at: 2020-10-16T23:21:41Z
updated_at: 2020-10-16T23:21:41Z
author_association: OWNER
body:
  Relevant: https://github.com/simonw/datasette/discussions/1021 asks about controlling the order of databases on that page that have been loaded using configuration directory mode.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/991#issuecomment-710694607
issue_url: https://api.github.com/repos/simonw/datasette/issues/991
node_id: MDEyOklzc3VlQ29tbWVudDcxMDY5NDYwNw==
created_at: 2020-10-16T23:23:34Z
updated_at: 2020-10-16T23:23:34Z
author_association: OWNER
body:
  Thinking more about pagination and search from #461: if the job of the homepage is to showcase the data that is available in the instance - data that is mostly in tables - maybe it's the tables themselves (and the ability to paginate and search through them) that becomes key to the design of the page.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/991#issuecomment-710694711
issue_url: https://api.github.com/repos/simonw/datasette/issues/991
node_id: MDEyOklzc3VlQ29tbWVudDcxMDY5NDcxMQ==
created_at: 2020-10-16T23:24:03Z
updated_at: 2020-10-16T23:24:03Z
author_association: OWNER
body:
  I'm really interested in exploring how this page could work with hundreds of database files attached and thousands of total tables.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1027#issuecomment-710750038
issue_url: https://api.github.com/repos/simonw/datasette/issues/1027
node_id: MDEyOklzc3VlQ29tbWVudDcxMDc1MDAzOA==
created_at: 2020-10-17T05:00:19Z
updated_at: 2020-10-17T05:00:19Z
author_association: OWNER
body:
  Running `apachectl -X` appears to run Apache in the foreground so I can hit Ctrl+C to quit it again, using the config file at `/usr/local/etc/httpd/httpd.conf` and serving the `index.html` file from `/usr/local/var/www` (which confusingly says "Welcome to nginx", presumably because I installed nginx via Homebrew earlier).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/188#issuecomment-710778368
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/188
node_id: MDEyOklzc3VlQ29tbWVudDcxMDc3ODM2OA==
created_at: 2020-10-17T08:52:58Z
updated_at: 2020-10-17T08:52:58Z
author_association: NONE
body:
  I have done a stupid question.

  If I run

  ```
  sqlite-utils :memory: "select spatialite_version()" --load-extension=/usr/local/lib/mod_spatialite.so
  ```

  I have `[{"spatialite_version()": "5.0.0"}]`

  Thank you for this great tool

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711074031
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11
node_id: MDEyOklzc3VlQ29tbWVudDcxMTA3NDAzMQ==
created_at: 2020-10-17T20:14:01Z
updated_at: 2020-10-17T20:14:01Z
author_association: MEMBER
body:
  I'd be happy to teach the tool to look for `export.xml` or `eksport.xml` - and then expand that list to other languages.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711078917
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11
node_id: MDEyOklzc3VlQ29tbWVudDcxMTA3ODkxNw==
created_at: 2020-10-17T20:51:55Z
updated_at: 2020-10-17T20:52:03Z
author_association: MEMBER
body:
  I switched my phone to Spanish and ran an export - I got a file called `exportar.zip`. Unzipped I still got a `apple_
  health_export` folder but the root contained:
  ```
  electrocardiograms/
  export_cda.xml
  exportar.xml
  workout-routes/
  ```
  It looks like `export_cda.xml` does not have a translated name, so maybe I can ignore it and look for the _other_ `.xml` file in that directory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079056
issue_url: https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11
node_id: MDEyOklzc3VlQ29tbWVudDcxMTA3OTA1Ng==
created_at: 2020-10-17T20:53:00Z
updated_at: 2020-10-17T20:53:00Z
author_association: MEMBER
body:
  I think the safest thing is to sniff the first few lines of the file. Those should be the same no matter the language that was used:
  ```xml
  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE HealthData [
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/dogsheep-beta/issues/28#issuecomment-711089647
issue_url: https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28
node_id: MDEyOklzc3VlQ29tbWVudDcxMTA4OTY0Nw==
created_at: 2020-10-17T22:43:13Z
updated_at: 2020-10-17T22:43:13Z
author_association: MEMBER
body:
  Since my personal Dogsheep uses Datasette authentication, I'm going to need to pass through cookies. https://github.com/simonw/datasette/issues/1020 will solve that in the future but for now I need to solve it explicitly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1030#issuecomment-711407607
issue_url: https://api.github.com/repos/simonw/datasette/issues/1030
node_id: MDEyOklzc3VlQ29tbWVudDcxMTQwNzYwNw==
created_at: 2020-10-18T19:31:31Z
updated_at: 2020-10-19T08:01:51Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=h1) Report
  > Merging [#1030](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/568bd7bbf590861687db8c318f3d8cfcd1dfb47a?el=desc) will **decrease** coverage by `0.10%`.
  > The diff coverage is `68.75%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1030/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1030      +/-   ##
  ==========================================
  - Coverage   84.63%   84.53%   -0.11%     
  ==========================================
    Files          28       28              
    Lines        3892     3905      +13     
  ==========================================
  + Hits         3294     3301       +7     
  - Misses        598      604       +6     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1030/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.35% <68.75%> (-0.79%)` | :arrow_down: |
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1030/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=footer). Last update [568bd7b...e082533](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063
issue_url: https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50
node_id: MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw==
created_at: 2020-10-19T05:01:29Z
updated_at: 2020-10-19T05:01:29Z
author_association: MEMBER
body:
  Demo of `--accept`:

      github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html'

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1031#issuecomment-711792622
issue_url: https://api.github.com/repos/simonw/datasette/issues/1031
node_id: MDEyOklzc3VlQ29tbWVudDcxMTc5MjYyMg==
created_at: 2020-10-19T07:57:17Z
updated_at: 2020-10-19T07:57:17Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=h1) Report
  > Merging [#1031](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/568bd7bbf590861687db8c318f3d8cfcd1dfb47a?el=desc) will **decrease** coverage by `0.02%`.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1031/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1031      +/-   ##
  ==========================================
  - Coverage   84.63%   84.60%   -0.03%     
  ==========================================
    Files          28       28              
    Lines        3892     3892              
  ==========================================
  - Hits         3294     3293       -1     
  - Misses        598      599       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1031/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `74.22% <ø> (ø)` | |
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1031/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=footer). Last update [568bd7b...7e7eaa4](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1027#issuecomment-712320103
issue_url: https://api.github.com/repos/simonw/datasette/issues/1027
node_id: MDEyOklzc3VlQ29tbWVudDcxMjMyMDEwMw==
created_at: 2020-10-19T17:35:04Z
updated_at: 2020-10-19T17:35:04Z
author_association: OWNER
body:
  Still need to configure proxying though. https://www.netnea.com/cms/apache-tutorial-9_setting-up-a-reverse-proxy/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-712324077
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcxMjMyNDA3Nw==
created_at: 2020-10-19T17:39:38Z
updated_at: 2020-10-19T17:39:38Z
author_association: OWNER
body:
  If I do these methods I think this should be available on the `datasette` object too, as an internal API for plugins to use to construct redirects and suchlike.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-712354600
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM1NDYwMA==
created_at: 2020-10-19T18:15:03Z
updated_at: 2020-10-19T18:15:39Z
author_association: OWNER
body:
  Related: #1026 (How should datasette.client interact with base_url)

  Also this comment from https://github.com/simonw/datasette/issues/943#issuecomment-675752436

  > One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted.
  > 
  > If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-712355706
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM1NTcwNg==
created_at: 2020-10-19T18:17:03Z
updated_at: 2020-10-19T18:17:03Z
author_association: OWNER
body:
  Options:
  - `datasette.urls.instance()` (or `datasette.urls.root()`), `datasette.urls.table()` etc
  - `datasette.url_for.instance()`...
  - `datasette.url.instance()`...
  - `datasette.root_url()`, `datasette.table_url()`...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1032#issuecomment-712363344
issue_url: https://api.github.com/repos/simonw/datasette/issues/1032
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM2MzM0NA==
created_at: 2020-10-19T18:31:46Z
updated_at: 2020-10-19T18:31:46Z
author_association: OWNER
body:
  As always with dates, the challenge is America. `mm/dd/yy` is indistinguishable from the more sensible `dd/mm/yy`. It's crucially important to visibly tell people which format you assumed, otherwise all kinds of weird invisible bugs can manifest.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1032#issuecomment-712363825
issue_url: https://api.github.com/repos/simonw/datasette/issues/1032
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM2MzgyNQ==
created_at: 2020-10-19T18:32:43Z
updated_at: 2020-10-19T18:32:43Z
author_association: OWNER
body:
  Horrible thought: I bet there is data out there that uses more than one date format in the same table! So this needs to be exposed as a visible per-column setting.

  Column action menu can help here, although it's not yet the full solution because it isn't yet visible on mobile.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1032#issuecomment-712364317
issue_url: https://api.github.com/repos/simonw/datasette/issues/1032
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM2NDMxNw==
created_at: 2020-10-19T18:33:42Z
updated_at: 2020-10-19T18:33:42Z
author_association: OWNER
body:
  Related challenge: timezones. I think I'll punt on those for the moment and just concentrate on dates, but I should keep them in mind.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1032#issuecomment-712365236
issue_url: https://api.github.com/repos/simonw/datasette/issues/1032
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM2NTIzNg==
created_at: 2020-10-19T18:35:25Z
updated_at: 2020-10-19T18:35:25Z
author_association: OWNER
body:
  Since I'm dealing with tables of data I usually have a whole column of examples, so heuristics that check for numbers-greater-than-12 could actually work well for many cases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1032#issuecomment-712365439
issue_url: https://api.github.com/repos/simonw/datasette/issues/1032
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM2NTQzOQ==
created_at: 2020-10-19T18:35:50Z
updated_at: 2020-10-19T18:37:57Z
author_association: OWNER
body:
  Maybe I don't need to add `dateutil` as a dependency here?

  `pendulum` and `arrow` both depend on it so it's pretty deeply embedded in the Python date ecosystem. Adding it as a dependency seems reasonable.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1032#issuecomment-712367285
issue_url: https://api.github.com/repos/simonw/datasette/issues/1032
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM2NzI4NQ==
created_at: 2020-10-19T18:39:32Z
updated_at: 2020-10-19T18:39:32Z
author_association: OWNER
body:
  https://github.com/digital-land/brownfield-land-collection/blob/a09ddf9960a6af59e72dc02448f7b645e59bf227/bin/harmonise.py#L217-L247 is a beautiful example of this problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1032#issuecomment-712397537
issue_url: https://api.github.com/repos/simonw/datasette/issues/1032
node_id: MDEyOklzc3VlQ29tbWVudDcxMjM5NzUzNw==
created_at: 2020-10-19T19:37:55Z
updated_at: 2020-10-19T19:37:55Z
author_association: NONE
body:
  python-dateutil is awesome, but it can only guess at one date at a time. So if you have a column of dates that are (presumably) in the same format, it can't use the full set of dates to deduce the format.  Also, once it has parsed a date, you can't get the format it used, whether to parse or render other dates.  These limitations prevent it from being a silver bullet for date parsing, though they're not enough for me to stop using it!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1020#issuecomment-712481568
issue_url: https://api.github.com/repos/simonw/datasette/issues/1020
node_id: MDEyOklzc3VlQ29tbWVudDcxMjQ4MTU2OA==
created_at: 2020-10-19T22:41:59Z
updated_at: 2020-10-19T22:41:59Z
author_association: OWNER
body:
  It turns out this works just fine:
  ```python
  response = await datasette.client.get(path, cookies=request.cookies)
  ```
  So I don't need a mechanism for this. I'm going to add this to the documentation instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1020#issuecomment-712482015
issue_url: https://api.github.com/repos/simonw/datasette/issues/1020
node_id: MDEyOklzc3VlQ29tbWVudDcxMjQ4MjAxNQ==
created_at: 2020-10-19T22:43:24Z
updated_at: 2020-10-19T22:43:24Z
author_association: OWNER
body:
  ... unless I want to support authentication mechanisms that work based on incoming IP address instead, in which case there's an argument for copying more over from the incoming request.

  Tailscale is a good example of a system where authentication based on IP address can actually work well, so this is worth doing. Also, there might be authentication mechanisms which work by setting a custom header on the incoming request (not to mention the `Authorization` header).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1020#issuecomment-712482504
issue_url: https://api.github.com/repos/simonw/datasette/issues/1020
node_id: MDEyOklzc3VlQ29tbWVudDcxMjQ4MjUwNA==
created_at: 2020-10-19T22:45:01Z
updated_at: 2020-10-19T22:45:01Z
author_association: OWNER
body:
  I'm having trouble coming up with the syntax for this. Here's one option:
  ```python
  response = await datasette.client.get(path, request=request)
  ```
  But this feels confusing to me. We're not using the `request=` argument as a request - we're using it as a source of some default request values (the cookies and incoming headers, but not the path).

  We're essentially combining that request with the other arguments passed to `.get()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-712483066
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcxMjQ4MzA2Ng==
created_at: 2020-10-19T22:46:48Z
updated_at: 2020-10-19T22:46:48Z
author_association: OWNER
body:
  OK, I'm committing to `datasette.urls.table()` and friends, which will be available to (and documented for use in) plugins and will be exposed to templates as `{{ urls.table(...) }}`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/904#issuecomment-712524699
issue_url: https://api.github.com/repos/simonw/datasette/issues/904
node_id: MDEyOklzc3VlQ29tbWVudDcxMjUyNDY5OQ==
created_at: 2020-10-20T01:04:12Z
updated_at: 2020-10-20T01:04:12Z
author_association: OWNER
body:
  Documentation is https://docs.datasette.io/en/latest/writing_plugins.html#building-urls-within-plugins and https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1025#issuecomment-712525557
issue_url: https://api.github.com/repos/simonw/datasette/issues/1025
node_id: MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw==
created_at: 2020-10-20T01:07:02Z
updated_at: 2020-10-20T01:07:02Z
author_association: OWNER
body:
  I fixed the `queries.html` one. I'm not going to fix these two:
  ```
  datasette/templates/error.html:        <a href="/">home</a>
  datasette/templates/patterns.html:        <a href="/">home</a> /
  ```
  Because the `error.html` one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-712569695
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcxMjU2OTY5NQ==
created_at: 2020-10-20T03:45:48Z
updated_at: 2020-10-20T03:46:14Z
author_association: NONE
body:
  I vote against headers. It has a lot of strikes against it: poor discoverability, new developers often don’t know how to use them, makes CORS harder, makes it hard to use eg with JQ, needs ad hoc specification for each bit of metadata, etc. 

  The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1025#issuecomment-712579674
issue_url: https://api.github.com/repos/simonw/datasette/issues/1025
node_id: MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA==
created_at: 2020-10-20T04:24:10Z
updated_at: 2020-10-20T04:24:10Z
author_association: OWNER
body:
  Changed my mind, `error.html` needs access to `urls` in order to link to its CSS file. Passing it after all (it already got passed `ds.config("base_url")` so `ds` was available previously).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-712581994
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxMjU4MTk5NA==
created_at: 2020-10-20T04:33:28Z
updated_at: 2020-10-20T04:33:28Z
author_association: OWNER
body:
  The [datasette-render-binary](https://github.com/simonw/datasette-render-binary) plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text:

  ![image](https://user-images.githubusercontent.com/9599/96540435-9c125f00-1252-11eb-85aa-5fc8d0e63728.png)

  Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-712585687
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcxMjU4NTY4Nw==
created_at: 2020-10-20T04:47:02Z
updated_at: 2020-10-20T04:47:12Z
author_association: OWNER
body:
  Great point about CORS, I hadn't considered that.

  I think I'm going to keep the `Link:` header (added in #1014) because I quite enjoy using it with GitHub and WordPress, but I'm not going to have it be the default way of doing pagination. For the default shape I'm now leaning towards this:

  ```json
  {
    "total": 36,
    "rows": [{"id": 1, "name": "Cleo"}],
    "next_url": "https://latest-with-plugins.datasette.io/fixtures/facetable.json?_next=5"
  }
  ```

  So three keys: `total`, `rows` and `next_url`. Then extra keys can be added using `?_extra=` with various named bundles.

reactions:
  {"total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0}


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-712590398
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcxMjU5MDM5OA==
created_at: 2020-10-20T05:03:46Z
updated_at: 2020-10-20T05:04:09Z
author_association: OWNER
body:
  OK, https://latest-with-plugins.datasette.io/ is running that now - e.g. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions.json-preview or https://latest-with-plugins.datasette.io/fixtures/compound_three_primary_keys.json-preview

  ```json
  {
    "rows": [
      {
        "pk": 1,
        "name": "The Mystery Spot",
        "address": "465 Mystery Spot Road, Santa Cruz, CA 95065",
        "latitude": 37.0167,
        "longitude": -122.0024
      },
      {
        "pk": 2,
        "name": "Winchester Mystery House",
        "address": "525 South Winchester Boulevard, San Jose, CA 95128",
        "latitude": 37.3184,
        "longitude": -121.9511
      },
      {
        "pk": 3,
        "name": "Burlingame Museum of PEZ Memorabilia",
        "address": "214 California Drive, Burlingame, CA 94010",
        "latitude": 37.5793,
        "longitude": -122.3442
      },
      {
        "pk": 4,
        "name": "Bigfoot Discovery Museum",
        "address": "5497 Highway 9, Felton, CA 95018",
        "latitude": 37.0414,
        "longitude": -122.0725
      }
    ],
    "total": 4,
    "next_url": null
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-712604364
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDcxMjYwNDM2NA==
created_at: 2020-10-20T05:39:15Z
updated_at: 2020-10-20T05:39:15Z
author_association: OWNER
body:
  OK, I've made a ton of improvements to how the `base_url` setting works - see tickets linked from #1023. I've just pushed out an alpha release with those changes in it: https://github.com/simonw/datasette/releases/tag/0.51a0

  @tsibley @tballison @ChristopherWilks I'd really appreciate your help testing this alpha! 

  You can install it with:

      pip install datasette==0.51a0

  It should work with just `ProxyPass`, without needing the `ProxyPassReverse` setting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1023#issuecomment-712604541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1023
node_id: MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ==
created_at: 2020-10-20T05:39:44Z
updated_at: 2020-10-20T05:39:44Z
author_association: OWNER
body:
  Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1026#issuecomment-712607227
issue_url: https://api.github.com/repos/simonw/datasette/issues/1026
node_id: MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw==
created_at: 2020-10-20T05:46:44Z
updated_at: 2020-10-20T05:46:44Z
author_association: OWNER
body:
  We have a solution for this now: `datasette.urls` from #1033 can be used by plugins to assemble the correct URLs to pass to `.get()` and friends.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/991#issuecomment-712855389
issue_url: https://api.github.com/repos/simonw/datasette/issues/991
node_id: MDEyOklzc3VlQ29tbWVudDcxMjg1NTM4OQ==
created_at: 2020-10-20T13:36:41Z
updated_at: 2020-10-20T13:36:41Z
author_association: NONE
body:
  Here is one quick sketch (done in Figma :P) for an idea: a possible filter to switch between showing all tables from all databases, or grouping tables by database. 

  (the switch is interactive) 

  All tables: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A2&viewport=536%2C348%2C0.5&scaling=min-zoom

  Grouped: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=3%3A974&viewport=536%2C348%2C0.5&scaling=min-zoom

  When only 1 database: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A162&viewport=536%2C348%2C0.5&scaling=min-zoom

  Is this is useful, I can send some more suggestions/sketches. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1026#issuecomment-712959034
issue_url: https://api.github.com/repos/simonw/datasette/issues/1026
node_id: MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA==
created_at: 2020-10-20T16:03:33Z
updated_at: 2020-10-20T16:03:55Z
author_association: OWNER
body:
  Reconsidering this: I think the `.get()` etc methods should automatically add the `base_url` prefix for you, since these APIs are only intended to make internal calls.

  The clincher on this is when I went to add a section to the `datasette.client` documentation recommending you use `datasette.urls.path()` for every call to them that you make.

  But there's a problem: to handle table name escaping users are likely to want to use `datasette.urls.table()` anyway, like this:

      response = await datasette.client.get(datasette.urls.table("db", "table") + ".json")

  This risks adding the `base_url` prefix twice. Maybe the `.table()` method could return a string-like object that is marked as already having the `base_url` prefix added, so the `client.get()` method knows not to add it again.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1026#issuecomment-712962517
issue_url: https://api.github.com/repos/simonw/datasette/issues/1026
node_id: MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw==
created_at: 2020-10-20T16:09:12Z
updated_at: 2020-10-20T16:09:12Z
author_association: OWNER
body:
  That `datasette.urls.table("db", "table") + ".json"` example is bad because if the table name contains a `.` it should be `?_format=json` instead.

  Maybe `.table()` should have a `format="json"` option that knows how to do this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1035#issuecomment-712963959
issue_url: https://api.github.com/repos/simonw/datasette/issues/1035
node_id: MDEyOklzc3VlQ29tbWVudDcxMjk2Mzk1OQ==
created_at: 2020-10-20T16:11:25Z
updated_at: 2020-10-20T16:11:25Z
author_association: OWNER
body:
  Relevant code:

  https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/utils/__init__.py#L681-L696

  Only used here:

  https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/views/base.py#L498-L502

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1035#issuecomment-712965574
issue_url: https://api.github.com/repos/simonw/datasette/issues/1035
node_id: MDEyOklzc3VlQ29tbWVudDcxMjk2NTU3NA==
created_at: 2020-10-20T16:13:57Z
updated_at: 2020-10-20T16:13:57Z
author_association: OWNER
body:
  That `renderers[key] = path_with_format(` is in a base class which can be used for both arbitrary queries, canned queries and the table view. I think that's OK, but it means that the `format="json"` argument on `datasette.urls.table()` won't be used by Datasette internally, it will just be available for plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1035#issuecomment-712976314
issue_url: https://api.github.com/repos/simonw/datasette/issues/1035
node_id: MDEyOklzc3VlQ29tbWVudDcxMjk3NjMxNA==
created_at: 2020-10-20T16:21:42Z
updated_at: 2020-10-20T16:21:42Z
author_association: OWNER
body:
  Makes me question if `datasette.urls` should grow functionality equivalent to the other path and querystring manipulation methods in `datasette.utils`:

  https://github.com/simonw/datasette/blob/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc/datasette/utils/__init__.py#L216-L279

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/782#issuecomment-712986115
issue_url: https://api.github.com/repos/simonw/datasette/issues/782
node_id: MDEyOklzc3VlQ29tbWVudDcxMjk4NjExNQ==
created_at: 2020-10-20T16:28:46Z
updated_at: 2020-10-20T16:29:51Z
author_association: OWNER
body:
  I think this all comes down to how the `?_extras=` mechanism works (see #262), as first hinted at in a30c5b220c15360d575e94b0e67f3255e120b916 (see commit message) when I added this long-forgotten undocumented feature: https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

  Extras need to be able to execute additional SQL, since that would solve the problem we have now where the expensive "suggested facets" code runs on all `.json` output even when its results are not being shown.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-712988146
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDcxMjk4ODE0Ng==
created_at: 2020-10-20T16:32:02Z
updated_at: 2023-01-17T01:54:13Z
author_association: OWNER
body:
  Just realized I added an undocumented `?_extras=` option to the row view years ago and forgot about it - it's not even documented. Added in a30c5b220c15360d575e94b0e67f3255e120b916 - https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables

  That will need to be made consistent with the new mechanism. I think `?_extra=a&_extra=b` is more consistent with other Datasette features (like `?_facet=col1&_facet=col2`) but potentially quite verbose.

  So I could support `?_extra=a,b,c` as an alternative allowed syntax, or I could allow `?_extra=single` and `?_extras=comma,separated`.

  I think I prefer allowing commas in `?_extra=`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-713170284
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE3MDI4NA==
created_at: 2020-10-20T22:13:01Z
updated_at: 2020-10-20T22:13:01Z
author_association: OWNER
body:
  In the documentation for `?_extra=` I think I'll emphasize the comma-separated version of it. Also: there will be `?_extra=` values which act as aliases for collection combinations - e.g. `?_extra=full` will toggle everything.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-713170979
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE3MDk3OQ==
created_at: 2020-10-20T22:14:37Z
updated_at: 2020-10-20T22:14:37Z
author_association: OWNER
body:
  I think it's worth having a plugin hook for this - it can be same hook that is used internally. Maybe `register_extra` - it lets you return one or more `extra` implementations, each with a name and an async function that gets called.

  Things like suggested facets will become `register_extra` hooks. Maybe actual facets too?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-713172901
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE3MjkwMQ==
created_at: 2020-10-20T22:19:10Z
updated_at: 2020-10-20T22:20:28Z
author_association: OWNER
body:
  I could go with the same format as `datasette-render-binary` but using `0x00` as the format for the hex bytes.

      0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01

  Problem with this is that it's ambiguous: if the ASCII characters `0x15` occur in the text they will be indistinguishable from those hex bytes.

  But since representing binary data in CSV fundamentally doesn't make sense I'm not sure if that really matters.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-713174341
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE3NDM0MQ==
created_at: 2020-10-20T22:22:53Z
updated_at: 2020-10-20T22:23:14Z
author_association: OWNER
body:
  An even easier option: do what the Datasette UI does and output `<Binary data: 7 bytes>` for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-713174690
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE3NDY5MA==
created_at: 2020-10-20T22:23:50Z
updated_at: 2020-10-20T22:23:50Z
author_association: OWNER
body:
  Or... default to `<Binary data: 7 bytes>` and support a `?_base64=1` option which outputs in base64 instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-713175741
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE3NTc0MQ==
created_at: 2020-10-20T22:26:45Z
updated_at: 2020-10-20T22:26:45Z
author_association: OWNER
body:
  > New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this:
  > 
  >     id,title,data
  >     1,Some title,<Binary data: 14 bytes>
  >     2,Other title,<Binary data: 57 bytes>
  > 
  > Then allow users to add ?_base64=1 to the URL to get base64 instead
  > https://twitter.com/simonw/status/1318679950635888641

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-713176082
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE3NjA4Mg==
created_at: 2020-10-20T22:27:33Z
updated_at: 2020-10-20T22:27:33Z
author_association: OWNER
body:
  This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713184374
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE4NDM3NA==
created_at: 2020-10-20T22:51:22Z
updated_at: 2020-10-20T22:51:22Z
author_association: OWNER
body:
  From https://hackerone.com/reports/126197:

  > archive.uber.com mirrors pypi. When downloading `.tar.gz` files from archive.uber.com, the MIME type is `application/octet-stream`. Injecting `<html><script>alert(0)</script>` into the start of the `.tar.gz` causes an XSS in Internet Explorer due to MIME sniffing.

  So you do have to be careful not to open accidental XSS holes with `application/octet-stream` thanks to (presumably older) versions of IE.

  From that thread it looks like the solution is to add a `X-Content-Type-Options: nosniff` header.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713185173
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE4NTE3Mw==
created_at: 2020-10-20T22:53:41Z
updated_at: 2020-10-20T22:53:41Z
author_association: OWNER
body:
  https://security.stackexchange.com/questions/12896/does-x-content-type-options-really-prevent-content-sniffing-attacks says:

  > In Tangled Web Michal Zalewski says:
  > 
  > > Refrain from using Content-Type: application/octet-stream and use application/binary instead, especially for unknown document types. Refrain from returning Content-Type: text/plain.
  > > 
  > > For example, any code-hosting platform must exercise caution when returning executables or source archives as application/octet-stream, because there is a risk they may be misinterpreted as HTML and displayed inline.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713185871
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE4NTg3MQ==
created_at: 2020-10-20T22:55:36Z
updated_at: 2020-10-20T22:55:36Z
author_association: OWNER
body:
  I can also use a `Content-Disposition` header to force a download. I'm reasonably confident that the combination of `Content-Disposition` and `X-Content-Type-Options: nosniff` and `application/binary` will let me allow users to download the contents of arbitrary BLOB columns without any XSS risk.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713186189
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE4NjE4OQ==
created_at: 2020-10-20T22:56:33Z
updated_at: 2020-10-20T22:56:33Z
author_association: OWNER
body:
  I think this plus the binary-CSV stuff in #1034 will justify a dedicated section of the documentation to talk about how Datasette handles binary BLOB columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-713191819
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxMzE5MTgxOQ==
created_at: 2020-10-20T23:12:58Z
updated_at: 2020-10-20T23:12:58Z
author_association: OWNER
body:
  Enzo has a great solution here: https://twitter.com/enzo_mdd/status/1318685442976436226

  > Or maybe an option for a url. This keeps the CSV small but allows scripts to download binary data as needed.

  In #1036 I'm planning on adding a way for users to access BLOB data. I can include that URL in the CSV output.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-713200782
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDcxMzIwMDc4Mg==
created_at: 2020-10-20T23:41:30Z
updated_at: 2020-10-20T23:41:30Z
author_association: OWNER
body:
  This is now blocking https://github.com/simonw/datasette-graphql/issues/61 because that issue needs a way to turn off suggested facets when retrieving the results of a table query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/262#issuecomment-713208667
issue_url: https://api.github.com/repos/simonw/datasette/issues/262
node_id: MDEyOklzc3VlQ29tbWVudDcxMzIwODY2Nw==
created_at: 2020-10-21T00:03:18Z
updated_at: 2020-10-21T00:03:18Z
author_association: OWNER
body:
  I think I should prioritize the facets component of this, since that could have significant performance wins while also supporting `datasette-graphql`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713226726
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzIyNjcyNg==
created_at: 2020-10-21T01:04:25Z
updated_at: 2020-10-21T01:04:25Z
author_association: OWNER
body:
  Extra security idea: a `blob_download_host` setting which can be used to indicate a host that should be used for downloads - for example `datasettestatic.com`. If this setting is populated then binary downloads are served from paths on that host only, and no other Datasette URLs from that host will be served.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1037#issuecomment-713267989
issue_url: https://api.github.com/repos/simonw/datasette/issues/1037
node_id: MDEyOklzc3VlQ29tbWVudDcxMzI2Nzk4OQ==
created_at: 2020-10-21T03:14:34Z
updated_at: 2020-10-21T03:14:34Z
author_association: OWNER
body:
  This is particularly relevant to the `datasette-cluster-map` plugin - the map is much nicer to use if the table itself can be scrolled.

  That plugin also makes this harder to build, because the plugin inserts the map as the direct predecessor of the `<table>` element and hence breaks if you try to wrap that in a `<div>`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1037#issuecomment-713268498
issue_url: https://api.github.com/repos/simonw/datasette/issues/1037
node_id: MDEyOklzc3VlQ29tbWVudDcxMzI2ODQ5OA==
created_at: 2020-10-21T03:15:44Z
updated_at: 2020-10-21T03:15:44Z
author_association: OWNER
body:
  This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713278349
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzI3ODM0OQ==
created_at: 2020-10-21T03:42:29Z
updated_at: 2020-10-21T03:42:29Z
author_association: OWNER
body:
  Possible URL for this: `/db/table/-/blob/primary-keys` - this would use the `/db/table/-/` namespace proposed in #296.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1038#issuecomment-713320666
issue_url: https://api.github.com/repos/simonw/datasette/issues/1038
node_id: MDEyOklzc3VlQ29tbWVudDcxMzMyMDY2Ng==
created_at: 2020-10-21T05:50:38Z
updated_at: 2020-10-21T05:50:38Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=h1) Report
  > Merging [#1038](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1038/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1038   +/-   ##
  =======================================
    Coverage   84.65%   84.65%           
  =======================================
    Files          28       28           
    Lines        3924     3924           
  =======================================
    Hits         3322     3322           
    Misses        602      602           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=footer). Last update [66120a7...7fc0cce](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1039#issuecomment-713754844
issue_url: https://api.github.com/repos/simonw/datasette/issues/1039
node_id: MDEyOklzc3VlQ29tbWVudDcxMzc1NDg0NA==
created_at: 2020-10-21T17:58:27Z
updated_at: 2020-10-21T17:58:27Z
author_association: OWNER
body:
  Now live: https://latest.datasette.io/fixtures/roadside_attraction_characteristics

  ![anim](https://user-images.githubusercontent.com/9599/96759016-55288480-138c-11eb-8ba0-d8e0f6dd8b1f.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713818178
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzgxODE3OA==
created_at: 2020-10-21T19:15:38Z
updated_at: 2020-10-21T19:16:34Z
author_association: OWNER
body:
  What should the suggested filename be?

  I think something that includes the table name, primary key and the name of the column would work.

  How about a file extension? I guess `.binary`, then let the user rename it? Or `.raw`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713821656
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzgyMTY1Ng==
created_at: 2020-10-21T19:22:45Z
updated_at: 2020-10-21T19:41:48Z
author_association: OWNER
body:
  So for https://latest.datasette.io/fixtures/binary_data the BLOB download URLs would be:

  `https://latest.datasette.io/fixtures/-/blob/binary_data/1/data.blob` - that last bit after the primary key is to indicate the `data` column

  With these headers:

  - `Content-Disposition: attachment; filename="binary_data-1-data.blob"`
  - `X-Content-Type-Options: nosniff`
  - `Content-Type: application/binary`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713829629
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzgyOTYyOQ==
created_at: 2020-10-21T19:38:43Z
updated_at: 2020-10-21T19:38:43Z
author_association: OWNER
body:
  Should this work just for BLOB columns, or should it work for other columns too?

  For the moment I'm going to restrict it to BLOBs, since data from other columns is available through the UI whereas BLOB columns are not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713830842
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzgzMDg0Mg==
created_at: 2020-10-21T19:41:20Z
updated_at: 2020-10-21T19:41:20Z
author_association: OWNER
body:
  Another useful demo database: https://datasette-render-images-demo.datasette.io/favicons/favicons - see https://datasette-render-images-demo.datasette.io/favicons/favicons.csv

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1036#issuecomment-713899530
issue_url: https://api.github.com/repos/simonw/datasette/issues/1036
node_id: MDEyOklzc3VlQ29tbWVudDcxMzg5OTUzMA==
created_at: 2020-10-21T21:55:00Z
updated_at: 2020-10-21T21:55:00Z
author_association: OWNER
body:
  This code needs these permission checks:
  https://github.com/simonw/datasette/blob/bf82b3d6a605c9ddadd5fb739249dfe6defaf635/datasette/views/table.py#L911-L913

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1040#issuecomment-713920562
issue_url: https://api.github.com/repos/simonw/datasette/issues/1040
node_id: MDEyOklzc3VlQ29tbWVudDcxMzkyMDU2Mg==
created_at: 2020-10-21T22:44:12Z
updated_at: 2020-10-24T23:08:14Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=h1) Report
  > Merging [#1040](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/bf82b3d6a605c9ddadd5fb739249dfe6defaf635?el=desc) will **increase** coverage by `0.10%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1040/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1040      +/-   ##
  ==========================================
  + Coverage   84.65%   84.76%   +0.10%     
  ==========================================
    Files          28       28              
    Lines        3924     3938      +14     
  ==========================================
  + Hits         3322     3338      +16     
  + Misses        602      600       -2     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `98.18% <ø> (+1.69%)` | :arrow_up: |
  | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `92.70% <ø> (-0.82%)` | :arrow_down: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.37% <100.00%> (+0.17%)` | :arrow_up: |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.77% <100.00%> (ø)` | |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.07% <100.00%> (+0.22%)` | :arrow_up: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=footer). Last update [bf82b3d...4f3165f](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/998#issuecomment-714090965
issue_url: https://api.github.com/repos/simonw/datasette/issues/998
node_id: MDEyOklzc3VlQ29tbWVudDcxNDA5MDk2NQ==
created_at: 2020-10-22T00:54:30Z
updated_at: 2020-10-22T00:54:30Z
author_association: OWNER
body:
  Easiest fix for the column action menu positioning - hide them when the user scrolls the containing div:
  ```javascript
  document.querySelector('.table-wrapper').addEventListener(
    'scroll', () => document.querySelector('.dropdown-menu').style.display = 'none'
  );                                            
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/998#issuecomment-714092002
issue_url: https://api.github.com/repos/simonw/datasette/issues/998
node_id: MDEyOklzc3VlQ29tbWVudDcxNDA5MjAwMg==
created_at: 2020-10-22T00:55:10Z
updated_at: 2020-10-22T00:55:10Z
author_association: OWNER
body:
  This isn't blocked on #987 - it just means that `datasette-cluster-map` will need to learn to look for `.table-wrapper` first and fall back on the table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/998#issuecomment-714117534
issue_url: https://api.github.com/repos/simonw/datasette/issues/998
node_id: MDEyOklzc3VlQ29tbWVudDcxNDExNzUzNA==
created_at: 2020-10-22T01:12:06Z
updated_at: 2020-10-22T01:12:06Z
author_association: OWNER
body:
  Demo:
  ![table-scroll](https://user-images.githubusercontent.com/9599/96806421-e74e7e00-13c8-11eb-95fe-44d01e4c2eb3.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/998#issuecomment-714205783
issue_url: https://api.github.com/repos/simonw/datasette/issues/998
node_id: MDEyOklzc3VlQ29tbWVudDcxNDIwNTc4Mw==
created_at: 2020-10-22T03:58:13Z
updated_at: 2020-10-22T03:58:13Z
author_association: OWNER
body:
  This is now live here: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1040#issuecomment-714206533
issue_url: https://api.github.com/repos/simonw/datasette/issues/1040
node_id: MDEyOklzc3VlQ29tbWVudDcxNDIwNjUzMw==
created_at: 2020-10-22T04:00:25Z
updated_at: 2020-10-22T04:00:25Z
author_association: OWNER
body:
  I've decided not to offer a configuration option to turn this off. I'll reconsider if someone asks for it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/171#issuecomment-714208848
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/171
node_id: MDEyOklzc3VlQ29tbWVudDcxNDIwODg0OA==
created_at: 2020-10-22T04:07:14Z
updated_at: 2020-10-22T04:07:14Z
author_association: OWNER
body:
  I made the `--load-extension` command much more widely supported in #137 - which should be useful for anyone who wants to use this extension.

  It's a bit too obscure for me to want to add direct Python library support relating to that extension though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/171#issuecomment-714219725
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/171
node_id: MDEyOklzc3VlQ29tbWVudDcxNDIxOTcyNQ==
created_at: 2020-10-22T04:38:35Z
updated_at: 2020-10-22T04:38:35Z
author_association: NONE
body:
  Thanks. As I said, I think the result (being able to query tree structures like ancestors and descendants) is more important than the implementation, and I agree that this particular sqlite extension is too obscure. Just providing an sqlite utility to build or rebuild a transitive closure table might be more generically useful. I find that hierarchical data shows up pretty frequently in some data science problems.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1031#issuecomment-714289680
issue_url: https://api.github.com/repos/simonw/datasette/issues/1031
node_id: MDEyOklzc3VlQ29tbWVudDcxNDI4OTY4MA==
created_at: 2020-10-22T07:23:52Z
updated_at: 2020-10-22T07:23:52Z
author_association: NONE
body:
  The bug is that currently when there are databases passed in, but no -i flag, e.g. in configuration directory mode, inclusion in inspect-data.json does not automatically cause databases to be considered immutable, as described in the documentation.

  The reason is that the -i flag is specified multiple=True, which means when it is not passed in we will get an empty list [], rather than None. So the current code decides that no databases are immutable rather than falling back to inspect-data.json -- as is presumably intended.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1033#issuecomment-714657366
issue_url: https://api.github.com/repos/simonw/datasette/issues/1033
node_id: MDEyOklzc3VlQ29tbWVudDcxNDY1NzM2Ng==
created_at: 2020-10-22T17:51:29Z
updated_at: 2020-10-22T17:51:29Z
author_association: CONTRIBUTOR
body:
  How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `"extra_js_urls": [ "/static/app.js"]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?) 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1033#issuecomment-714681365
issue_url: https://api.github.com/repos/simonw/datasette/issues/1033
node_id: MDEyOklzc3VlQ29tbWVudDcxNDY4MTM2NQ==
created_at: 2020-10-22T18:33:48Z
updated_at: 2020-10-22T18:33:48Z
author_association: OWNER
body:
  That's a good question - I hadn't considered that. I'm going to open a new issue to have `extra_js_urls` respect the `base_url` setting, since the static files will be served from a different location.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1041#issuecomment-714682288
issue_url: https://api.github.com/repos/simonw/datasette/issues/1041
node_id: MDEyOklzc3VlQ29tbWVudDcxNDY4MjI4OA==
created_at: 2020-10-22T18:35:15Z
updated_at: 2020-10-22T18:35:15Z
author_association: OWNER
body:
  @psychemedia said: https://github.com/simonw/datasette/issues/1033#issuecomment-714657366

  > How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `"extra_js_urls": [ "/static/app.js"]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1041#issuecomment-714682825
issue_url: https://api.github.com/repos/simonw/datasette/issues/1041
node_id: MDEyOklzc3VlQ29tbWVudDcxNDY4MjgyNQ==
created_at: 2020-10-22T18:36:10Z
updated_at: 2020-10-22T18:36:10Z
author_association: OWNER
body:
  I'll need to update these docs once there's a solution for this in place: https://docs.datasette.io/en/latest/custom_templates.html#serving-static-files

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1041#issuecomment-714683801
issue_url: https://api.github.com/repos/simonw/datasette/issues/1041
node_id: MDEyOklzc3VlQ29tbWVudDcxNDY4MzgwMQ==
created_at: 2020-10-22T18:37:47Z
updated_at: 2020-10-22T18:37:47Z
author_association: OWNER
body:
  I think I'll do this by looking for URLs that start with `/` - since it's also possible to have full `https://...` URLs in that setting.
  ```json
  {
      "extra_css_urls": [
          "/static/styles.css"
      ],
      "extra_js_urls": [
          "/static/app.js"
      ]
  }
  ```
  I need to think about the `extra_css_urls` and `extra_js_urls` plugin hooks too: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-css-urls-template-database-table-columns-view-name-request-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/173#issuecomment-714758139
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/173
node_id: MDEyOklzc3VlQ29tbWVudDcxNDc1ODEzOQ==
created_at: 2020-10-22T20:57:56Z
updated_at: 2020-10-22T20:57:56Z
author_association: OWNER
body:
  I could use `ijson` to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since `sqlite-utils` is a library dependency for many other projects and it would be using `ijson` purely for the CLI component.

  Here's how to iterate through a list of objects being read from a file:

  ```python
  import json
  parser = ijson.items(open(
      "/tmp/list.json"
  ), "item")
  for object in parser:
     # ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-714868207
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNDg2ODIwNw==
created_at: 2020-10-23T02:29:12Z
updated_at: 2020-10-23T02:29:12Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/d0cc6f4c32e1f89238ddec782086b3122f445bd4/datasette/app.py#L288-L311

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-714868624
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNDg2ODYyNA==
created_at: 2020-10-23T02:30:27Z
updated_at: 2020-10-23T02:30:37Z
author_association: OWNER
body:
  Maybe `register_template_loader(datasette)` which returns an object which is added in at the beginning of the list passed to `ChoiceLoader` here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1012#issuecomment-714908859
issue_url: https://api.github.com/repos/simonw/datasette/issues/1012
node_id: MDEyOklzc3VlQ29tbWVudDcxNDkwODg1OQ==
created_at: 2020-10-23T04:49:20Z
updated_at: 2020-10-23T04:49:20Z
author_association: CONTRIBUTOR
body:
  Good luck on 1.0! It may also be worth lobbying for a `Framework::Datasette::1.0` classifier. This would be a nice way to allow the ecosystem to self-document a bit more [discoverably](https://pypi.org/search/?q=&o=&c=Framework+%3A%3A+Datasette%3A%3A+1.0).  

  I was surprised to see the [PR for `Framework::Jupyter`](https://github.com/pypa/warehouse/pull/1905/files) is a... database migration! Of course, there may be more workflow to it!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1043#issuecomment-714915025
issue_url: https://api.github.com/repos/simonw/datasette/issues/1043
node_id: MDEyOklzc3VlQ29tbWVudDcxNDkxNTAyNQ==
created_at: 2020-10-23T05:09:09Z
updated_at: 2020-10-23T05:09:09Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=h1) Report
  > Merging [#1043](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/d0cc6f4c32e1f89238ddec782086b3122f445bd4?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1043/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1043   +/-   ##
  =======================================
    Coverage   84.65%   84.65%           
  =======================================
    Files          28       28           
    Lines        3924     3924           
  =======================================
    Hits         3322     3322           
    Misses        602      602           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=footer). Last update [d0cc6f4...dc4129c](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1044#issuecomment-714916127
issue_url: https://api.github.com/repos/simonw/datasette/issues/1044
node_id: MDEyOklzc3VlQ29tbWVudDcxNDkxNjEyNw==
created_at: 2020-10-23T05:12:52Z
updated_at: 2020-10-23T05:12:52Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=h1) Report
  > Merging [#1044](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/d0cc6f4c32e1f89238ddec782086b3122f445bd4?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1044/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1044   +/-   ##
  =======================================
    Coverage   84.65%   84.65%           
  =======================================
    Files          28       28           
    Lines        3924     3924           
  =======================================
    Hits         3322     3322           
    Misses        602      602           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=footer). Last update [d0cc6f4...6453ab1](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715490532
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTQ5MDUzMg==
created_at: 2020-10-23T17:57:34Z
updated_at: 2020-10-23T17:57:34Z
author_association: OWNER
body:
  A better version of this hook would be passed the database, table and query name depending on what was being rendered.

  This would require some re-thinking of how core templates are loaded, especially since I would want the templates considered comment to continue working.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715496859
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTQ5Njg1OQ==
created_at: 2020-10-23T18:11:27Z
updated_at: 2020-10-23T18:11:27Z
author_association: OWNER
body:
  When loading a template the filename is required, but you can optionally also send a set of extra arguments which the template loader can take into consideration.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715497419
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTQ5NzQxOQ==
created_at: 2020-10-23T18:12:40Z
updated_at: 2020-10-23T18:12:40Z
author_association: OWNER
body:
  Maybe the template loader can optionally return some extra context to pass to the template. That could be used to solve the templates considered comment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1040#issuecomment-715587715
issue_url: https://api.github.com/repos/simonw/datasette/issues/1040
node_id: MDEyOklzc3VlQ29tbWVudDcxNTU4NzcxNQ==
created_at: 2020-10-23T21:01:07Z
updated_at: 2020-10-23T21:03:10Z
author_association: OWNER
body:
  A download icon would be nice for the links in the table display.

  I like this one https://primer.style/octicons/download-24

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715614971
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTYxNDk3MQ==
created_at: 2020-10-23T22:20:14Z
updated_at: 2020-10-23T22:23:51Z
author_association: OWNER
body:
  Alternative plugin hook idea:
  ```python
  @hookspec
  def load_template(template, database, table, columns, view_name, request, datasette):
      "Load the specified template, returning the template code as a string"
  ```
  Imitating the existing `extra_template_vars` family of hooks: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715616757
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTYxNjc1Nw==
created_at: 2020-10-23T22:27:28Z
updated_at: 2020-10-23T22:27:28Z
author_association: OWNER
body:
  Almost all of the core template loading happens in the `BaseView.render` method: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/views/base.py#L114-L133

  The one exception is the 404 handling code here: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L1034-L1042

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715617405
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTYxNzQwNQ==
created_at: 2020-10-23T22:29:53Z
updated_at: 2020-10-23T22:29:53Z
author_association: OWNER
body:
  Also consider that `DatasetteRouter` uses `.list_templates()` to gather together `{slug}.html` style templates for the custom page templates mechanism: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L949-L967

  For that to work with the new plugin hook, custom template providing plugins will need a way to provide a list of templates that they know about.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715617830
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTYxNzgzMA==
created_at: 2020-10-23T22:31:26Z
updated_at: 2020-10-23T22:31:26Z
author_association: OWNER
body:
  So maybe this should be a `register_template_loader` mechanism that returns a Jinja loader after all? That would mean that only the template filename could be used as the input to the plugin, which doesn't seem as useful as emulating the `extra_template_vars()` interface.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715618077
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTYxODA3Nw==
created_at: 2020-10-23T22:32:24Z
updated_at: 2020-10-23T22:32:24Z
author_association: OWNER
body:
  Another option: the first version of the plugin hook could accept only the template filename. Subsequent releases could add more arguments, since Pluggy allows new arguments to be added without breaking backwards compatibility.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715618333
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTYxODMzMw==
created_at: 2020-10-23T22:33:24Z
updated_at: 2020-10-23T22:33:24Z
author_association: OWNER
body:
  It wouldn't be a disaster if template-loading plugins were unable to hook into the `/{slug1}/{slug2}.html` custom page mechanism, since plugins can define their own pages already using `register_routes()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1045#issuecomment-715641183
issue_url: https://api.github.com/repos/simonw/datasette/issues/1045
node_id: MDEyOklzc3VlQ29tbWVudDcxNTY0MTE4Mw==
created_at: 2020-10-24T00:19:29Z
updated_at: 2020-10-24T00:19:29Z
author_association: OWNER
body:
  It turns out it already does that:

  https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L710-L720

  But the documentation doesn't reflect that:

  > `template` - string
  > 
  > > The template file to be rendered, e.g. `my_plugin.html`. Datasette will search for this file first in the `--template-dir=` location, if it was specified - then in the plugin's bundled templates and finally in Datasette's set of default templates.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715643646
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTY0MzY0Ng==
created_at: 2020-10-24T00:33:46Z
updated_at: 2020-10-24T00:33:46Z
author_association: OWNER
body:
  I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class.

  This code is the problem:
  https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133

  I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1042#issuecomment-715643763
issue_url: https://api.github.com/repos/simonw/datasette/issues/1042
node_id: MDEyOklzc3VlQ29tbWVudDcxNTY0Mzc2Mw==
created_at: 2020-10-24T00:34:31Z
updated_at: 2020-10-24T00:34:52Z
author_association: OWNER
body:
  I'm going to rename that to template variable from `select_templates` to `templates_considered` too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1033#issuecomment-716066000
issue_url: https://api.github.com/repos/simonw/datasette/issues/1033
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA2NjAwMA==
created_at: 2020-10-24T22:58:33Z
updated_at: 2020-10-24T22:58:33Z
author_association: CONTRIBUTOR
body:
  From [the docs](https://docs.datasette.io/en/latest/internals.html#datasette-urls), I note:

  ```
  datasette.urls.instance()
  Returns the URL to the Datasette instance root page. This is usually "/"
  ```

  What about the proxy case? Eg if I am using jupyter-server-proxy on a MyBinder or local Jupyter notebook server site, `https://example.com:PORT/weirdpath/datasette`, what does `datasette.urls.instance()` refer to?

  - [ ] `https://example.com:PORT/weirdpath/datasette`
  - [ ] `https://example.com:PORT/weirdpath/`
  - [ ] `https://example.com:PORT/`
  - [ ] `https://example.com`
  - [ ] something else?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1046#issuecomment-716066342
issue_url: https://api.github.com/repos/simonw/datasette/issues/1046
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA2NjM0Mg==
created_at: 2020-10-24T23:02:07Z
updated_at: 2020-10-24T23:02:25Z
author_association: OWNER
body:
  A download icon would be nice for the links in the table display. I like this one https://primer.style/octicons/download-24

  ```svg
  <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="24" height="24">
    <path d="M4.97 11.03a.75.75 0 111.06-1.06L11 14.94V2.75a.75.75 0 011.5 0v12.19l4.97-4.97a.75.75 0 111.06 1.06l-6.25 6.25a.75.75 0 01-1.06 0l-6.25-6.25zm-.22 9.47a.75.75 0 000 1.5h14.5a.75.75 0 000-1.5H4.75z"></path>
  </svg>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1046#issuecomment-716071507
issue_url: https://api.github.com/repos/simonw/datasette/issues/1046
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA3MTUwNw==
created_at: 2020-10-25T00:06:47Z
updated_at: 2020-10-25T00:06:47Z
author_association: OWNER
body:
  I used https://primer.style/octicons/download-16 instead.

  <img width="734" alt="favicons__favicons__71_rows" src="https://user-images.githubusercontent.com/9599/97096021-44417280-161b-11eb-9bc4-88def6afaca4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-716077436
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA3NzQzNg==
created_at: 2020-10-25T01:08:35Z
updated_at: 2020-10-25T01:08:42Z
author_association: OWNER
body:
  This is actually a bit tricky to implement, for a few reasons:

  - Need to generate a full URL, including the `https://host/` bit. I've done this for `next_url` in the JSON output before, thankfully.
  - This only makes sense for CSV output for tables. If it's the CSV output of an arbitrary query there's no `/db/table/-/blob/pk/column.blob` page for me to link to.
  - Need to generate those `/.../-/blob/...` URLs for the data that is being output as CSV.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-716077508
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA3NzUwOA==
created_at: 2020-10-25T01:09:17Z
updated_at: 2020-10-25T01:09:17Z
author_association: OWNER
body:
  Here's how those absolute `next_url` values are generated: https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/views/table.py#L774-L776

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-716077541
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA3NzU0MQ==
created_at: 2020-10-25T01:09:38Z
updated_at: 2020-10-25T01:10:04Z
author_association: OWNER
body:
  I should turn `datasette.absolute_url(...)` into a documented internal API on https://docs.datasette.io/en/stable/internals.html#datasette-class

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-716078512
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA3ODUxMg==
created_at: 2020-10-25T01:21:11Z
updated_at: 2020-10-25T01:21:11Z
author_association: OWNER
body:
  What should happen for CSV export of arbitrary SQL queries, where there's no obvious BLOB to link to?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-716078777
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxNjA3ODc3Nw==
created_at: 2020-10-25T01:25:11Z
updated_at: 2020-10-25T01:25:11Z
author_association: OWNER
body:
  SQLite actually has APIs that could help here: https://www.sqlite.org/c3ref/column_database_name.html - for any given SQL query they identify the origin/table/column that is the source of each resulting column.

  Those aren't exposed in the Python `sqlite3` module though, so using them could be extremely tricky.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/838#issuecomment-716123598
issue_url: https://api.github.com/repos/simonw/datasette/issues/838
node_id: MDEyOklzc3VlQ29tbWVudDcxNjEyMzU5OA==
created_at: 2020-10-25T10:20:12Z
updated_at: 2020-10-25T10:53:24Z
author_association: CONTRIBUTOR
body:
  I'm trying to [run something behind a MyBinder proxy](https://github.com/ouseful-testing/nbsearch), but seem to have something set up incorrectly and not sure what the fix is?

  I'm starting datasette with jupyter-server-proxy setup:

  ```
  # __init__.py
  def setup_nbsearch():

      return {
          "command": [
              "datasette",
              "serve",
              f"{_NBSEARCH_DB_PATH}",
              "-p",
              "{port}",
              "--config",
              "base_url:{base_url}nbsearch/"
          ],
          "absolute_url": True,
          # The following needs a the labextension installing.
          # eg in postBuild: jupyter labextension install jupyterlab-server-proxy
          "launcher_entry": {
              "enabled": True,
              "title": "nbsearch",
          },
      }
  ```

  where the `base_url` gets automatically populated by the server-proxy. I define the loaders as:

  ```
  # __init__.py
  from datasette import hookimpl

  @hookimpl
  def extra_css_urls(database, table, columns, view_name, datasette):
      return [
          "/-/static-plugins/nbsearch/prism.css",
          "/-/static-plugins/nbsearch/nbsearch.css",
      ]
  ```
  but these seem to also need a base_url prefix set somehow?

  Currently, the generated HTML loads properly but internal links are incorrect; eg they take the form `<link rel="stylesheet" href="/-/static-plugins/nbsearch/prism.css">` which resolves to eg `https://notebooks.gesis.org/hub/-/static-plugins/nbsearch/prism.css` rather than required URL of form `https://notebooks.gesis.org/binder/jupyter/user/ouseful-testing-nbsearch-0fx1mx67/nbsearch/-/static-plugins/nbsearch/prism.css`.

  The main css is loaded correctly: `<link rel="stylesheet" href="/binder/jupyter/user/ouseful-testing-nbsearch-0fx1mx67/nbsearch/-/static/app.css?404439">`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1049#issuecomment-716146238
issue_url: https://api.github.com/repos/simonw/datasette/issues/1049
node_id: MDEyOklzc3VlQ29tbWVudDcxNjE0NjIzOA==
created_at: 2020-10-25T13:13:32Z
updated_at: 2020-10-25T13:13:32Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=h1) Report
  > Merging [#1049](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/42f4851e3e7885f1092f104d6c883cea40b12f02?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1049/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1049   +/-   ##
  =======================================
    Coverage   84.72%   84.72%           
  =======================================
    Files          28       28           
    Lines        3942     3942           
  =======================================
    Hits         3340     3340           
    Misses        602      602           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=footer). Last update [42f4851...50a743a](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1050#issuecomment-716175236
issue_url: https://api.github.com/repos/simonw/datasette/issues/1050
node_id: MDEyOklzc3VlQ29tbWVudDcxNjE3NTIzNg==
created_at: 2020-10-25T16:35:20Z
updated_at: 2020-10-25T16:35:20Z
author_association: OWNER
body:
  This is clearly a better solution than the one I  implemented in #1040 - I don't have to add a new route, I don't have to implement permission checks, it reuses mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1051#issuecomment-716204090
issue_url: https://api.github.com/repos/simonw/datasette/issues/1051
node_id: MDEyOklzc3VlQ29tbWVudDcxNjIwNDA5MA==
created_at: 2020-10-25T20:06:42Z
updated_at: 2020-10-25T20:06:42Z
author_association: OWNER
body:
  Providing a binary download link here is actually extremely difficult.

  The problem is that the SQL query itself represents data that can change from one moment to the next. It's no good showing a "Binary: 55 bytes" message that links to that same SQL query but with a `.blob` extension and arguments to select the particular result, because the data may change in a way that causes that query to return a different row - at which point the download link will give you the wrong data, not the 55 bytes you asked for.

  So providing a download link risks being misleading.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1051#issuecomment-716204271
issue_url: https://api.github.com/repos/simonw/datasette/issues/1051
node_id: MDEyOklzc3VlQ29tbWVudDcxNjIwNDI3MQ==
created_at: 2020-10-25T20:08:04Z
updated_at: 2020-10-25T20:08:04Z
author_association: OWNER
body:
  This is bad though, because if I want to provide binary data in CSV as requested in #1034 I need some way of providing that data.

  Which suggests to me that the base64 option is the only one that can make sense for arbitrary SQL queries represented as CSV. Download links won't work.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1043#issuecomment-716237524
issue_url: https://api.github.com/repos/simonw/datasette/issues/1043
node_id: MDEyOklzc3VlQ29tbWVudDcxNjIzNzUyNA==
created_at: 2020-10-26T00:14:57Z
updated_at: 2020-10-26T00:14:57Z
author_association: CONTRIBUTOR
body:
  Sorry, I was out of the loop this weekend. The missing sdists were in some the `datasette-*` plugins... i'll capture my findings more concretely in one spot when i have a chance...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1052#issuecomment-716265360
issue_url: https://api.github.com/repos/simonw/datasette/issues/1052
node_id: MDEyOklzc3VlQ29tbWVudDcxNjI2NTM2MA==
created_at: 2020-10-26T02:17:58Z
updated_at: 2020-10-26T02:17:58Z
author_association: OWNER
body:
  The default z-index values for Leaflet are defined here: https://github.com/Leaflet/Leaflet/blob/b346bb8bf7bb80899baa1f4fc1536bae58e7e3e6/dist/leaflet.css#L81-L91

  ```css
  .leaflet-pane         { z-index: 400; }

  .leaflet-tile-pane    { z-index: 200; }
  .leaflet-overlay-pane { z-index: 400; }
  .leaflet-shadow-pane  { z-index: 500; }
  .leaflet-marker-pane  { z-index: 600; }
  .leaflet-tooltip-pane   { z-index: 650; }
  .leaflet-popup-pane   { z-index: 700; }

  .leaflet-map-pane canvas { z-index: 100; }
  .leaflet-map-pane svg    { z-index: 200; }
  ```
  So a `z-index` of 1000 on the menu should fix this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/976#issuecomment-716305890
issue_url: https://api.github.com/repos/simonw/datasette/issues/976
node_id: MDEyOklzc3VlQ29tbWVudDcxNjMwNTg5MA==
created_at: 2020-10-26T05:07:10Z
updated_at: 2020-10-26T05:07:10Z
author_association: OWNER
body:
  I used the new `datasette.urls` methods to handle escaping table names.

  https://github.com/simonw/datasette/blob/f5dbe61a4568c0915ec6be820095c2960cf0857c/datasette/utils/__init__.py#L996-L1008

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1051#issuecomment-716681167
issue_url: https://api.github.com/repos/simonw/datasette/issues/1051
node_id: MDEyOklzc3VlQ29tbWVudDcxNjY4MTE2Nw==
created_at: 2020-10-26T16:51:15Z
updated_at: 2020-10-26T16:51:15Z
author_association: OWNER
body:
  Crazy idea: generate a signed URL containing a base64 of the gzip of the binary content (to try and reduce size).

  No: this will blow through URL limits in various hosting providers and possibly even browsers. It could be made to work a little bit more reliably with some extra JavaScript that turns it into a download on the browser-side, but that would be hideously complicated.

  Also the signed bit doesn't prevent people from generating SQL queries that generate nasty binary blobs for download.

  I'm beginning to think that restricting this feature to just table view, not query view, is a better idea. Query view can still get at the binary using JSON and base64.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1051#issuecomment-716681602
issue_url: https://api.github.com/repos/simonw/datasette/issues/1051
node_id: MDEyOklzc3VlQ29tbWVudDcxNjY4MTYwMg==
created_at: 2020-10-26T16:51:58Z
updated_at: 2020-10-26T16:51:58Z
author_association: OWNER
body:
  I still need to improve the current binary display on the query page though, where it outputs a Python `b'...'` literal.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/189#issuecomment-716756103
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDcxNjc1NjEwMw==
created_at: 2020-10-26T18:56:19Z
updated_at: 2020-10-26T18:56:19Z
author_association: OWNER
body:
  This is a great fix, thanks! If you add a unit test somewhere in here I'll merge the PR:  https://github.com/simonw/sqlite-utils/blob/main/tests/test_m2m.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1054#issuecomment-717050585
issue_url: https://api.github.com/repos/simonw/datasette/issues/1054
node_id: MDEyOklzc3VlQ29tbWVudDcxNzA1MDU4NQ==
created_at: 2020-10-27T07:38:50Z
updated_at: 2020-10-27T07:38:50Z
author_association: OWNER
body:
  Maybe imitate how Django does this, e.g. https://github.com/django/django/commit/6b9b2af7352908d40ca4d31bdb1b80c013cab29a

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717359145
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/189
node_id: MDEyOklzc3VlQ29tbWVudDcxNzM1OTE0NQ==
created_at: 2020-10-27T16:20:32Z
updated_at: 2020-10-27T16:20:32Z
author_association: CONTRIBUTOR
body:
  No problem. I added a test. Let me know if it looks sufficient or if you want me to to tweak something!

  If you don't mind, would you tag this PR as "hacktoberfest-accepted"? If you do mind, no problem and I'm sorry for asking :) My kiddos like the shirts.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1056#issuecomment-717489501
issue_url: https://api.github.com/repos/simonw/datasette/issues/1056
node_id: MDEyOklzc3VlQ29tbWVudDcxNzQ4OTUwMQ==
created_at: 2020-10-27T19:39:41Z
updated_at: 2020-10-27T19:39:41Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=h1) Report
  > Merging [#1056](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/26bb4a268127da2c38f4241abe45444b2a6f7874?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1056/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1056   +/-   ##
  =======================================
    Coverage   84.70%   84.70%           
  =======================================
    Files          28       28           
    Lines        3955     3955           
  =======================================
    Hits         3350     3350           
    Misses        605      605           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=footer). Last update [26bb4a2...a7b2aab](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1058#issuecomment-717527606
issue_url: https://api.github.com/repos/simonw/datasette/issues/1058
node_id: MDEyOklzc3VlQ29tbWVudDcxNzUyNzYwNg==
created_at: 2020-10-27T20:44:06Z
updated_at: 2020-10-27T20:44:06Z
author_association: OWNER
body:
  Example: https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L26-L32

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1057#issuecomment-717531272
issue_url: https://api.github.com/repos/simonw/datasette/issues/1057
node_id: MDEyOklzc3VlQ29tbWVudDcxNzUzMTI3Mg==
created_at: 2020-10-27T20:51:09Z
updated_at: 2020-10-27T20:51:09Z
author_association: OWNER
body:
  That works!

  <img width="1370" alt="Banners_and_Alerts_and_SQLite___Mike_Bostock___Observable" src="https://user-images.githubusercontent.com/9599/97360439-717c6380-185b-11eb-9996-e0064e4b22d4.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1059#issuecomment-717938992
issue_url: https://api.github.com/repos/simonw/datasette/issues/1059
node_id: MDEyOklzc3VlQ29tbWVudDcxNzkzODk5Mg==
created_at: 2020-10-28T13:38:46Z
updated_at: 2020-10-28T13:38:46Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=h1) Report
  > Merging [#1059](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/7d9fedc176717a7e3d22a96575ae0aada5a65440?el=desc) will **not change** coverage.
  > The diff coverage is `n/a`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1059/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=tree)

  ```diff
  @@           Coverage Diff           @@
  ##             main    #1059   +/-   ##
  =======================================
    Coverage   84.71%   84.71%           
  =======================================
    Files          28       28           
    Lines        3957     3957           
  =======================================
    Hits         3352     3352           
    Misses        605      605           
  ```



  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=footer). Last update [7d9fedc...e46327a](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1059#issuecomment-718078447
issue_url: https://api.github.com/repos/simonw/datasette/issues/1059
node_id: MDEyOklzc3VlQ29tbWVudDcxODA3ODQ0Nw==
created_at: 2020-10-28T17:07:59Z
updated_at: 2020-10-28T17:08:14Z
author_association: OWNER
body:
  > #### 0.6.0 (2020-10-27)
  > 
  > -   aiofiles is now tested on ppc64le.
  > -   Added name and mode properties to async file objects. [#82](https://github.com/Tinche/aiofiles/pull/82)
  > -   Fixed a DeprecationWarning internally. [#75](https://github.com/Tinche/aiofiles/pull/75)
  > -   Python 3.9 support and tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718168730
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/191
node_id: MDEyOklzc3VlQ29tbWVudDcxODE2ODczMA==
created_at: 2020-10-28T19:47:20Z
updated_at: 2020-10-28T19:47:20Z
author_association: OWNER
body:
  https://stackoverflow.com/a/3931903 looks useful:
  ```python
  def trace(*args):
      def _trace(func):
          def wrapper(*args, **kwargs):
              print enter_string
              func(*args, **kwargs)
              print exit_string
          return wrapper
      if len(args) == 1 and callable(args[0]):
          # No arguments, this is the decorator
          # Set default values for the arguments
          enter_string = 'entering'
          exit_string = 'exiting'
          return _trace(args[0])
      else:
          # This is just returning the decorator
          enter_string, exit_string = args
          return _trace
  ```
  Can improve that code with `functools.wraps`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718170295
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/191
node_id: MDEyOklzc3VlQ29tbWVudDcxODE3MDI5NQ==
created_at: 2020-10-28T19:50:16Z
updated_at: 2020-10-28T19:50:16Z
author_association: OWNER
body:
  I think I made a mistake when I designed the initial decorator. I should have had it work like this:

  ```python
  @db.register_function()
  def reverse_string(s):
      return "".join(reversed(list(s)))
  ```
  As this leaves open the option to add new parameters in the future.

  To avoid breaking backwards compatibility I'll use the hack that detects the argument this time, but in the future I'll try to remember to always design decorators to be called like `@decorator()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1060#issuecomment-718243062
issue_url: https://api.github.com/repos/simonw/datasette/issues/1060
node_id: MDEyOklzc3VlQ29tbWVudDcxODI0MzA2Mg==
created_at: 2020-10-28T22:23:33Z
updated_at: 2020-10-28T22:23:33Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=h1) Report
  > Merging [#1060](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/abcf0222496d8148b2e585ffa0ff192270a04b06?el=desc) will **increase** coverage by `6.42%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1060/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1060      +/-   ##
  ==========================================
  + Coverage   84.71%   91.13%   +6.42%     
  ==========================================
    Files          28       27       -1     
    Lines        3957     3677     -280     
  ==========================================
  - Hits         3352     3351       -1     
  + Misses        605      326     -279     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1060/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `73.63% <100.00%> (+0.13%)` | :arrow_up: |
  | [datasette/version.py](https://codecov.io/gh/simonw/datasette/pull/1060/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZlcnNpb24ucHk=) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=footer). Last update [abcf022...4725d46](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1050#issuecomment-718317997
issue_url: https://api.github.com/repos/simonw/datasette/issues/1050
node_id: MDEyOklzc3VlQ29tbWVudDcxODMxNzk5Nw==
created_at: 2020-10-29T02:24:50Z
updated_at: 2020-10-29T02:29:24Z
author_association: NONE
body:
  Unsolicited feedback for an unreleased feature of the [current](https://github.com/simonw/datasette/commit/5e0b72247ecab4ce0fcec599b77a83d73a480872) unreleased GitHub version (I casually wanted to access a blob row) – the existing #1036 route doesn't support special characters in database or table names (e.g. `@()` ). Maybe this is motivation for your new idea here.

  Also I got this error/crash with my blob and wasn't able to get the file: https://gist.github.com/thadk/28ac32af0e88747ce9056c90b0b19d34

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1049#issuecomment-718340847
issue_url: https://api.github.com/repos/simonw/datasette/issues/1049
node_id: MDEyOklzc3VlQ29tbWVudDcxODM0MDg0Nw==
created_at: 2020-10-29T03:45:47Z
updated_at: 2020-10-29T03:48:26Z
author_association: OWNER
body:
  [thebe](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html) is the first time I've seen a library that requires you to set up some global JavaScript configuration before loading the script itself.

  I'm hesitant to add an extra template block just to cover that one case since it's such a rare pattern. But it's important that `thebelab` can be used with Datasette.

  Would this pattern work for you instead?

  ```html+jinja
  {% block extra_head %}
  <script type="text/x-thebe-config">
    {
      requestKernel: true,
      binderOptions: {
        repo: "binder-examples/requirements",
      },
    }
  </script>
  <script src="https://unpkg.com/thebelab@latest/lib/index.js"></script>
  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1049#issuecomment-718341542
issue_url: https://api.github.com/repos/simonw/datasette/issues/1049
node_id: MDEyOklzc3VlQ29tbWVudDcxODM0MTU0Mg==
created_at: 2020-10-29T03:48:12Z
updated_at: 2020-10-29T03:48:12Z
author_association: OWNER
body:
  You could use Datasette's new `{{ urls.static_plugins(...) }}` template option - see https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls - to generate a link to code that was bundled with the plugin:

  ```html+jinja
  {% block extra_head %}
  <script type="text/x-thebe-config">
    {
      requestKernel: true,
      binderOptions: {
        repo: "binder-examples/requirements",
      },
    }
  </script>
  <script src="{{ urls.static_plugins("datasette-thebelab-plugin", "thebelab-index.js")"></script>
  {% endblock %}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1050#issuecomment-718342036
issue_url: https://api.github.com/repos/simonw/datasette/issues/1050
node_id: MDEyOklzc3VlQ29tbWVudDcxODM0MjAzNg==
created_at: 2020-10-29T03:49:57Z
updated_at: 2020-10-29T03:49:57Z
author_association: OWNER
body:
  @thadk from that error it looks like the problem may have been that you had a BLOB column containing a `null` value? If so that's definitely a bug, I'll fix that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1050#issuecomment-718346019
issue_url: https://api.github.com/repos/simonw/datasette/issues/1050
node_id: MDEyOklzc3VlQ29tbWVudDcxODM0NjAxOQ==
created_at: 2020-10-29T04:05:07Z
updated_at: 2020-10-29T04:05:07Z
author_association: OWNER
body:
  Yes, confirmed - this is a bug where if the `BLOB` column contains a `null` you get a nasty exception if you try to download it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1049#issuecomment-718528252
issue_url: https://api.github.com/repos/simonw/datasette/issues/1049
node_id: MDEyOklzc3VlQ29tbWVudDcxODUyODI1Mg==
created_at: 2020-10-29T09:20:34Z
updated_at: 2020-10-29T09:20:34Z
author_association: CONTRIBUTOR
body:
  That workaround is probably fine. I was trying to work out whether there might be other situations where a pre-external package load might be useful but couldn't offhand bring any other examples to mind. The static plugins option also looks interesting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1051#issuecomment-718980659
issue_url: https://api.github.com/repos/simonw/datasette/issues/1051
node_id: MDEyOklzc3VlQ29tbWVudDcxODk4MDY1OQ==
created_at: 2020-10-29T19:45:42Z
updated_at: 2020-10-29T19:45:42Z
author_association: OWNER
body:
  OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once. 

  For an arbitrary SQL query, how about if I link to this?

  `/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`

  Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?

  If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).

  I think this might be the best solution to the problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1050#issuecomment-718980944
issue_url: https://api.github.com/repos/simonw/datasette/issues/1050
node_id: MDEyOklzc3VlQ29tbWVudDcxODk4MDk0NA==
created_at: 2020-10-29T19:46:19Z
updated_at: 2020-10-29T19:46:19Z
author_association: OWNER
body:
  Had an idea in https://github.com/simonw/datasette/issues/1051#issuecomment-718980659

  > OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once.
  > 
  > For an arbitrary SQL query, how about if I link to this?
  > 
  > `/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666`
  > 
  > Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data?
  > 
  > If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated).
  > 
  > I think this might be the best solution to the problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1050#issuecomment-718987852
issue_url: https://api.github.com/repos/simonw/datasette/issues/1050
node_id: MDEyOklzc3VlQ29tbWVudDcxODk4Nzg1Mg==
created_at: 2020-10-29T20:00:32Z
updated_at: 2020-10-29T20:00:32Z
author_association: OWNER
body:
  The reason I like the `?_blob_hash=` solution is that it feels really misleading to provide a link to "download this binary" which could conceivably download some other data.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1050#issuecomment-719021514
issue_url: https://api.github.com/repos/simonw/datasette/issues/1050
node_id: MDEyOklzc3VlQ29tbWVudDcxOTAyMTUxNA==
created_at: 2020-10-29T21:05:08Z
updated_at: 2020-10-29T21:05:08Z
author_association: OWNER
body:
  Idea: what if Datasette had a custom SQLite function that could be used to generate URLs to the row-level BLOB download for a value? Then custom SQL query authors could use that function to link to the relevant content.

  This could be expanded to exposing other `datasette.urls` functionality as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1062#issuecomment-719031901
issue_url: https://api.github.com/repos/simonw/datasette/issues/1062
node_id: MDEyOklzc3VlQ29tbWVudDcxOTAzMTkwMQ==
created_at: 2020-10-29T21:25:54Z
updated_at: 2020-10-29T21:25:54Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/base.py#L258-L345

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1061#issuecomment-719033013
issue_url: https://api.github.com/repos/simonw/datasette/issues/1061
node_id: MDEyOklzc3VlQ29tbWVudDcxOTAzMzAxMw==
created_at: 2020-10-29T21:27:14Z
updated_at: 2020-10-29T21:27:14Z
author_association: OWNER
body:
  Next challenge: link to `.blob` downloads from https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data

  This will be a bit tricky. Here's how that template works at the moment: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/templates/query.html#L69-L77

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1061#issuecomment-719035336
issue_url: https://api.github.com/repos/simonw/datasette/issues/1061
node_id: MDEyOklzc3VlQ29tbWVudDcxOTAzNTMzNg==
created_at: 2020-10-29T21:29:29Z
updated_at: 2020-10-29T21:29:29Z
author_association: OWNER
body:
  Those display_rows have already been processed by the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/database.py#L320-L346

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1063#issuecomment-719043108
issue_url: https://api.github.com/repos/simonw/datasette/issues/1063
node_id: MDEyOklzc3VlQ29tbWVudDcxOTA0MzEwOA==
created_at: 2020-10-29T21:46:48Z
updated_at: 2020-10-29T21:46:48Z
author_association: OWNER
body:
  Remove this `xfail` and `import pytest`: https://github.com/simonw/datasette/blob/503a5b7b4080a26ef9ceb1ecd1a4a6f4ef4ffc59/tests/test_csv.py#L83-L96

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1061#issuecomment-719049115
issue_url: https://api.github.com/repos/simonw/datasette/issues/1061
node_id: MDEyOklzc3VlQ29tbWVudDcxOTA0OTExNQ==
created_at: 2020-10-29T22:00:57Z
updated_at: 2020-10-29T22:00:57Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=h1) Report
  > Merging [#1061](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/d6f9ff71378c4eab34dad181c23cfc143a4aef2d?el=desc) will **increase** coverage by `0.07%`.
  > The diff coverage is `96.87%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1061/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1061      +/-   ##
  ==========================================
  + Coverage   91.13%   91.20%   +0.07%     
  ==========================================
    Files          27       28       +1     
    Lines        3677     3697      +20     
  ==========================================
  + Hits         3351     3372      +21     
  + Misses        326      325       -1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.77% <0.00%> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.38% <100.00%> (+0.15%)` | :arrow_up: |
  | [datasette/blob\_renderer.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2Jsb2JfcmVuZGVyZXIucHk=) | `100.00% <100.00%> (ø)` | |
  | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `92.13% <100.00%> (+0.17%)` | :arrow_up: |
  | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.04% <100.00%> (+0.07%)` | :arrow_up: |
  | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.86% <100.00%> (-0.22%)` | :arrow_down: |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=footer). Last update [d6f9ff7...1196d08](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-719050754
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxOTA1MDc1NA==
created_at: 2020-10-29T22:04:52Z
updated_at: 2020-10-29T22:04:52Z
author_association: OWNER
body:
  I'm going to link to. the new `.blob` representation using the new `?_blob_hash=xxx` argument to ensure that the content served is the expected binary blob.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1051#issuecomment-719053669
issue_url: https://api.github.com/repos/simonw/datasette/issues/1051
node_id: MDEyOklzc3VlQ29tbWVudDcxOTA1MzY2OQ==
created_at: 2020-10-29T22:12:16Z
updated_at: 2020-10-29T22:12:16Z
author_association: OWNER
body:
  https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101 now looks like this:

  <img width="816" alt="fixtures__select_rowid__data_from_binary_data_order_by_rowid_limit_101_and_understanding_Google_auths_-_for_your_information" src="https://user-images.githubusercontent.com/9599/97638086-15e7dc80-19f9-11eb-90a9-686e012c4db9.png">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1034#issuecomment-719094027
issue_url: https://api.github.com/repos/simonw/datasette/issues/1034
node_id: MDEyOklzc3VlQ29tbWVudDcxOTA5NDAyNw==
created_at: 2020-10-30T00:11:17Z
updated_at: 2020-10-30T00:11:17Z
author_association: OWNER
body:
  Demos:

  https://latest.datasette.io/fixtures/binary_data.csv?_size=max

  ```csv
  rowid,data
  1,http://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data
  2,http://latest.datasette.io/fixtures/binary_data/2.blob?_blob_column=data
  3,
  ```

  https://latest.datasette.io/fixtures.csv?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max

  ```csv
  rowid,data
  1,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d
  2,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=b835b0483cedb86130b9a2c280880bf5fadc5318ddf8c18d0df5204d40df1724
  3,
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719104883
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTEwNDg4Mw==
created_at: 2020-10-30T00:50:01Z
updated_at: 2020-10-30T00:52:29Z
author_association: OWNER
body:
  Here's what the prototype looks like so far:

  ![menu](https://user-images.githubusercontent.com/9599/97647443-7eda4f00-1a0f-11eb-8d78-b703b7a13616.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719105197
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTEwNTE5Nw==
created_at: 2020-10-30T00:51:16Z
updated_at: 2020-10-30T00:51:16Z
author_association: OWNER
body:
  I used a `<details><summary>` for this: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L16-L36

  I added a bit of JavaScript so that clicking outside the menu would close it: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L59-L74

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719106174
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTEwNjE3NA==
created_at: 2020-10-30T00:55:12Z
updated_at: 2020-10-30T00:55:12Z
author_association: OWNER
body:
  So what should go in this menu?

  If the user is logged in as root, I'll link to the various debug pages.

  If they're not logged in at all I don't think the menu should appear.

  If they are logged in as anyone, it should display to give them access to the "log out" button.

  Plugins can add links to it. If those plugins add links, the menu will display.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719109770
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTEwOTc3MA==
created_at: 2020-10-30T01:08:14Z
updated_at: 2020-10-30T01:08:14Z
author_association: OWNER
body:
  How should the plugin hook work?

  Here's the first version of the HTML:
  ```html
          <div class="nav-menu-inner">
              <ul>
                  <li><a href="{{ urls.instance() }}">Home</a></li>
                  <li><a href="{{ urls.path('/-/plugins') }}">Installed plugins</a></li>
                  <li><a href="{{ urls.path('/-/versions') }}">Software versions</a></li>
                  <li><a href="{{ urls.path('/-/metadata') }}">Metadata</a></li>
                  {% if show_logout %}
                  <form action="{{ urls.logout() }}" method="post">
                      <input type="hidden" name="csrftoken" value="{{ csrftoken() }}">
                      <button class="button-as-link">Log out</button>
                  </form>{% endif %}
              </ul>
          </div>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719110582
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTExMDU4Mg==
created_at: 2020-10-30T01:11:13Z
updated_at: 2020-10-30T01:11:13Z
author_association: OWNER
body:
  Should plugins be able to add forms like the logout form here, or should they be restricted to adding navigation links?

  I can't think of a reason a plugin would need to add a form. The logout form is a special case to protect against logout-csrf attacks.

  So I think plugins get to return a list of dictionaries, each with a `label` and an `href`:

  ```python
  return [{
    "label": "Upload CSVs",
    "href": datasette.urls.path("/-/upload-csvs")
  }]
  ```
  But... is there an argument for returning headings, to divide up the menu?

  I think so. I also like the idea that a default plugin checks for the `root` user and outputs links to the different debugging tools - maybe those should be wrapped in a section heading.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719110808
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTExMDgwOA==
created_at: 2020-10-30T01:12:09Z
updated_at: 2020-10-30T01:12:19Z
author_association: OWNER
body:
  Or... plugins could return HTML - maybe optionally using helper functions to generate common HTML such that plugins which use the helpers can have their HTML modified in the future.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719111373
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTExMTM3Mw==
created_at: 2020-10-30T01:14:13Z
updated_at: 2020-10-30T01:14:13Z
author_association: OWNER
body:
  Plugins returning HTML makes more sense for some of the other areas that plugins will be able to inject content - e.g. injecting content on the table or row page above the table.

  If I'm going to have that as a pattern though it may make sense to use HTML here, since that will be consistent with other places that plugins can inject additional content.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719111597
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTExMTU5Nw==
created_at: 2020-10-30T01:15:05Z
updated_at: 2020-10-30T01:15:05Z
author_association: OWNER
body:
  I'm torn on this one. I think I have a very slight preference for plugins returning structured objects as opposed to HTML. Less likely to regret that choice in the future?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1064#issuecomment-719117185
issue_url: https://api.github.com/repos/simonw/datasette/issues/1064
node_id: MDEyOklzc3VlQ29tbWVudDcxOTExNzE4NQ==
created_at: 2020-10-30T01:35:17Z
updated_at: 2020-10-30T01:35:17Z
author_association: OWNER
body:
  I'm going to go with a list of `{"label": ..., "href": ...}` as the first iteration of this. The logout link will not be returned as part of the plugin output. A default plugin will provide the debug tools if the user is logged in as root.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1065#issuecomment-719153773
issue_url: https://api.github.com/repos/simonw/datasette/issues/1065
node_id: MDEyOklzc3VlQ29tbWVudDcxOTE1Mzc3Mw==
created_at: 2020-10-30T03:44:57Z
updated_at: 2020-10-30T03:44:57Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=h1) Report
  > Merging [#1065](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/1a861be19e326e0c88230a711a1b6536366697d7?el=desc) will **increase** coverage by `0.03%`.
  > The diff coverage is `100.00%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1065/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1065      +/-   ##
  ==========================================
  + Coverage   91.23%   91.27%   +0.03%     
  ==========================================
    Files          28       29       +1     
    Lines        3710     3724      +14     
  ==========================================
  + Hits         3385     3399      +14     
    Misses        325      325              
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.42% <100.00%> (+0.03%)` | :arrow_up: |
  | [datasette/default\_menu\_links.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWVudV9saW5rcy5weQ==) | `100.00% <100.00%> (ø)` | |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=footer). Last update [1a861be...5f118b5](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1066#issuecomment-719154646
issue_url: https://api.github.com/repos/simonw/datasette/issues/1066
node_id: MDEyOklzc3VlQ29tbWVudDcxOTE1NDY0Ng==
created_at: 2020-10-30T03:48:15Z
updated_at: 2020-10-30T03:48:15Z
author_association: OWNER
body:
  This will use a very similar implementation to the navigation menu in #1064 - similar plugin hook and I'll use a `<details><summary>` to implement it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1066#issuecomment-719194619
issue_url: https://api.github.com/repos/simonw/datasette/issues/1066
node_id: MDEyOklzc3VlQ29tbWVudDcxOTE5NDYxOQ==
created_at: 2020-10-30T05:18:04Z
updated_at: 2020-10-30T05:18:04Z
author_association: OWNER
body:
  The cog only appears if at least one table action has been registered by a plugin. It looks like this:

  ![table-actions](https://user-images.githubusercontent.com/9599/97662535-9bd54900-1a34-11eb-8e0f-56d159c8834e.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1066#issuecomment-719194756
issue_url: https://api.github.com/repos/simonw/datasette/issues/1066
node_id: MDEyOklzc3VlQ29tbWVudDcxOTE5NDc1Ng==
created_at: 2020-10-30T05:18:35Z
updated_at: 2020-10-30T05:18:35Z
author_association: OWNER
body:
  Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/690#issuecomment-719195346
issue_url: https://api.github.com/repos/simonw/datasette/issues/690
node_id: MDEyOklzc3VlQ29tbWVudDcxOTE5NTM0Ng==
created_at: 2020-10-30T05:20:42Z
updated_at: 2020-10-30T05:20:42Z
author_association: OWNER
body:
  I've now added two new plugin hooks: [menu_links()](https://docs.datasette.io/en/latest/plugin_hooks.html#menu-links-datasette-actor) and [table_actions()](https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table).

  I'm going to close this issue. Further work (on column actions and and database actions) can happen in separate tickets, but I won't include them in Datasette 0.51 since they're much less interesting than table and instance actions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1067#issuecomment-719320948
issue_url: https://api.github.com/repos/simonw/datasette/issues/1067
node_id: MDEyOklzc3VlQ29tbWVudDcxOTMyMDk0OA==
created_at: 2020-10-30T07:02:37Z
updated_at: 2020-10-30T07:02:37Z
author_association: OWNER
body:
  Yes, this should be possible - no point restricting what plugin authors can do with the feature. Will need to add some extra arguments to the plugin hook for this.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1067#issuecomment-719322666
issue_url: https://api.github.com/repos/simonw/datasette/issues/1067
node_id: MDEyOklzc3VlQ29tbWVudDcxOTMyMjY2Ng==
created_at: 2020-10-30T07:04:02Z
updated_at: 2020-10-30T07:04:02Z
author_association: OWNER
body:
  Maybe rename it to `actions_menu` and have it work for database, view, table and query pages using different arguments on each.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1068#issuecomment-719328661
issue_url: https://api.github.com/repos/simonw/datasette/issues/1068
node_id: MDEyOklzc3VlQ29tbWVudDcxOTMyODY2MQ==
created_at: 2020-10-30T07:09:30Z
updated_at: 2020-10-30T07:09:30Z
author_association: OWNER
body:
  Then this can make it available to root: https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_permissions.py

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1068#issuecomment-719331236
issue_url: https://api.github.com/repos/simonw/datasette/issues/1068
node_id: MDEyOklzc3VlQ29tbWVudDcxOTMzMTIzNg==
created_at: 2020-10-30T07:11:58Z
updated_at: 2020-10-30T07:11:58Z
author_association: OWNER
body:
  Document the new permission here: https://docs.datasette.io/en/stable/authentication.html#built-in-permissions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1069#issuecomment-719657478
issue_url: https://api.github.com/repos/simonw/datasette/issues/1069
node_id: MDEyOklzc3VlQ29tbWVudDcxOTY1NzQ3OA==
created_at: 2020-10-30T16:31:21Z
updated_at: 2020-10-30T17:46:36Z
author_association: NONE
body:
  # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=h1) Report
  > Merging [#1069](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/222f79bb4c6e2aa5426cc5ff25f1b2461e18a300?el=desc) will **increase** coverage by `0.01%`.
  > The diff coverage is `95.83%`.

  [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1069/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=tree)

  ```diff
  @@            Coverage Diff             @@
  ##             main    #1069      +/-   ##
  ==========================================
  + Coverage   91.30%   91.32%   +0.01%     
  ==========================================
    Files          29       29              
    Lines        3736     3756      +20     
  ==========================================
  + Hits         3411     3430      +19     
  - Misses        325      326       +1     
  ```


  | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=tree) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1069/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.94% <ø> (-0.04%)` | :arrow_down: |
  | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1069/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.38% <95.45%> (-0.05%)` | :arrow_down: |
  | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1069/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |

  ------

  [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=continue).
  > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
  > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
  > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=footer). Last update [222f79b...92f3840](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1069#issuecomment-719664530
issue_url: https://api.github.com/repos/simonw/datasette/issues/1069
node_id: MDEyOklzc3VlQ29tbWVudDcxOTY2NDUzMA==
created_at: 2020-10-30T16:43:40Z
updated_at: 2020-10-30T16:43:40Z
author_association: OWNER
body:
  I should include an example in the documentation that shows loading templates from a database table.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1069#issuecomment-719666912
issue_url: https://api.github.com/repos/simonw/datasette/issues/1069
node_id: MDEyOklzc3VlQ29tbWVudDcxOTY2NjkxMg==
created_at: 2020-10-30T16:47:44Z
updated_at: 2020-10-30T16:47:44Z
author_association: OWNER
body:
  Bringing over a comment from #1042:

  > I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class.
  > 
  > This code is the problem:
  > 
  > https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133
  > 
  > I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1069#issuecomment-719670714
issue_url: https://api.github.com/repos/simonw/datasette/issues/1069
node_id: MDEyOklzc3VlQ29tbWVudDcxOTY3MDcxNA==
created_at: 2020-10-30T16:53:56Z
updated_at: 2020-10-30T16:53:56Z
author_association: OWNER
body:
  I'm having second thoughts about the design of the plugin hook.

  Consider the following:
  ```python
              plugin_template_source = pm.hook.load_template(
                  template=template_name,
                  database=context.get("database"),
                  table=context.get("table"),
                  columns=context.get("columns"),
                  view_name=self.name,
                  request=request,
                  datasette=self.ds,
              )
  ```
  It's a bit gross that `database`, `table` and `columns` are pulled out of the context like that. This doesn't make sense for pages that are rendered by plugins, for example.

  So maybe for the first release of this plugin hook I should cut it down to just seeing `template`, `request` and `datasette`. I can add the table/view/etc stuff back in later if it turns out to be necessary.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1069#issuecomment-719672967
issue_url: https://api.github.com/repos/simonw/datasette/issues/1069
node_id: MDEyOklzc3VlQ29tbWVudDcxOTY3Mjk2Nw==
created_at: 2020-10-30T16:58:01Z
updated_at: 2020-10-30T16:58:01Z
author_association: OWNER
body:
  OK, new hook specification is:
  ```python
  @hookspec(firstresult=True)
  def load_template(template, request, datasette):
      "Load the specified template, returning the template code as a string"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1071#issuecomment-719777499
issue_url: https://api.github.com/repos/simonw/datasette/issues/1071
node_id: MDEyOklzc3VlQ29tbWVudDcxOTc3NzQ5OQ==
created_at: 2020-10-30T20:20:01Z
updated_at: 2020-10-30T20:20:01Z
author_association: OWNER
body:
  Fixed: https://latest.datasette.io/-/messages

  ![demo-message](https://user-images.githubusercontent.com/9599/97753199-9c142980-1ab2-11eb-9be9-c0be41acc68e.gif)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719784606
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTc4NDYwNg==
created_at: 2020-10-30T20:35:33Z
updated_at: 2020-10-30T20:35:33Z
author_association: OWNER
body:
  To fix this I think I need to move the `load_template` implementation into a Jinja template loader.

  I'm not sure I'll be able to keep the `Templates considered` comment working though:

  https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L745-L750

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719785005
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTc4NTAwNQ==
created_at: 2020-10-30T20:36:22Z
updated_at: 2020-10-30T20:36:22Z
author_association: OWNER
body:
  It should be easy enough to show a comment that says which original template names were considered, but I may not be able to show which one was actually used (or which ones came from plugins).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719803880
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgwMzg4MA==
created_at: 2020-10-30T21:17:11Z
updated_at: 2020-10-30T21:17:11Z
author_association: OWNER
body:
  Example from the Jinja docs: https://jinja.palletsprojects.com/en/2.11.x/api/#jinja2.BaseLoader

  ```python
  from jinja2 import BaseLoader, TemplateNotFound
  from os.path import join, exists, getmtime

  class MyLoader(BaseLoader):

      def __init__(self, path):
          self.path = path

      def get_source(self, environment, template):
          path = join(self.path, template)
          if not exists(path):
              raise TemplateNotFound(template)
          mtime = getmtime(path)
          with file(path) as f:
              source = f.read().decode('utf-8')
          return source, path, lambda: mtime == getmtime(path)
  ```
  Also available: `jinja2.FunctionLoader(load_func)` which lets me pass it a function like this one:
  ```
  >>> def load_template(name):
  ...     if name == 'index.html':
  ...         return '...'
  ...
  >>> loader = FunctionLoader(load_template)
  ```

  Just one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719807502
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgwNzUwMg==
created_at: 2020-10-30T21:26:49Z
updated_at: 2020-10-30T21:26:49Z
author_association: OWNER
body:
  It looks like Jinja does not have a mechanism for asynchronous template loading - the loader API is synchronous.

  One option may be to figure out which templates are needed (including inherited templates and includes) before rendering the template. Then async load those templates from the database into a `DictLoader`, then pass that `DictLoader` to Jinja.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719809259
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgwOTI1OQ==
created_at: 2020-10-30T21:31:10Z
updated_at: 2020-10-30T21:31:10Z
author_association: OWNER
body:
  How can we tell what template Jinja will need to render?

  One approach that could work:

  1. Set up a dummy template loader which records the name of the template that was requested
  2. Load the template
  3. Now we know the list of templates that were requested. Async load those
  4. The dummy template loader can now return the ones we have loaded. Load the template again.
  5. Did it request any more templates? If so, load those, and repeat.
  6. Keep on with this loop until a template load (which might even have to be a render) fails to request any templates that we have not yet loaded.
  7. Render the template.

  This is GROSS. It feels like a huge waste of CPU, and it could lead to very weird behaviour if any template variables have side effects.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719809780
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgwOTc4MA==
created_at: 2020-10-30T21:32:28Z
updated_at: 2020-10-30T21:32:28Z
author_association: OWNER
body:
  Here's an alternative that would definitely work and would be a lot simpler, at the cost of a fair amount of RAM:

  1. Before rendering the template, load ALL of the most-recent-versions of the templates that are stored in the DB. Use those to populate a `DictLoader`.
  2. Render the template.

  This does mean loading template bodies that we won't use. Provided an instance has less than 100 templates I imagine this will work just fine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719810023
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgxMDAyMw==
created_at: 2020-10-30T21:33:06Z
updated_at: 2020-10-30T21:33:06Z
author_association: OWNER
body:
  The ideal solution is for Jinja to offer `async` template loading. I'll file a feature request, then I'll implement the second option above (async load all templates from the DB before each render).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719810533
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgxMDUzMw==
created_at: 2020-10-30T21:34:38Z
updated_at: 2020-10-30T21:34:38Z
author_association: OWNER
body:
  ... no wait, my comments above assume that I'm just building the `datasette-edit-templates` plugin. Does this work as a general solution for all of Datasette? I don't think it does.

  This may mean I need to delay the whole feature.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719811312
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgxMTMxMg==
created_at: 2020-10-30T21:36:49Z
updated_at: 2020-10-30T21:36:49Z
author_association: OWNER
body:
  There's one other option: in `datasette-edit-templates` I could maybe use `asyncio.get_event_loop().run_in_executor(...)` to load the templates asynchronously within the Jinja template loader mechanism.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719813970
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgxMzk3MA==
created_at: 2020-10-30T21:44:40Z
updated_at: 2020-10-30T21:44:40Z
author_association: OWNER
body:
  I'm pretty sure that `run_in_executor()` workaround won't work. https://github.com/django/asgiref/blob/7becc9daca2628c46af1cb7e46b4c47c1ea27adf/asgiref/sync.py#L83 for example says "You cannot use AsyncToSync in the same thread as an async event loop".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719814279
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgxNDI3OQ==
created_at: 2020-10-30T21:45:33Z
updated_at: 2020-10-30T21:45:33Z
author_association: OWNER
body:
  Sadly I'm going to bump `load_template` from Datasette 0.51 - I don't think I should block the release on resolving this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719819234
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgxOTIzNA==
created_at: 2020-10-30T22:00:21Z
updated_at: 2020-10-30T22:00:21Z
author_association: OWNER
body:
  There might be a way to save this. Async template loading can't be supported, but what if you could define a `load_template()` hook which returned a sync function that returned templates...

  Then the `datasette-edit-templates` plugin could reply to `load_template` by loading all DB templates into memory and returning a `load_template` sync function that looked up the values in those already-loaded templates.

  It could even maintain an in-memory cache that gets updated when a template is edited.

  If I do this, I could remove the ability to return an `async` function from `load_template()` but add that in the future should Jinja implement a mechanism for async template loading.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719832651
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgzMjY1MQ==
created_at: 2020-10-30T22:46:25Z
updated_at: 2020-10-30T22:46:25Z
author_association: OWNER
body:
  I tried using a `FunctionLoader` and got this error on startup:
  ```python
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 989, in __init__
      for filepath in self.ds.jinja_env.list_templates()
    File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/environment.py", line 810, in list_templates
      names = self.loader.list_templates()
    File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py", line 434, in list_templates
      found.update(loader.list_templates())
    File "/Users/simon/.local/share/virtualenvs/datasette-edit-templates-agoZyE3x/lib/python3.8/site-packages/jinja2/loaders.py", line 99, in list_templates
      raise TypeError("this loader cannot iterate over all templates")
  TypeError: this loader cannot iterate over all templates
  ```
  So if I'm going to define a custom Jinja loader I'll need to teach plugins to answer the "list templates" query.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719832853
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgzMjg1Mw==
created_at: 2020-10-30T22:47:12Z
updated_at: 2020-10-30T22:47:12Z
author_association: OWNER
body:
  Maybe I should ditch this hook entirely in favour of the existing `prepare_jinja2_environment` hook. Could that add new template loaders?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719833070
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgzMzA3MA==
created_at: 2020-10-30T22:48:04Z
updated_at: 2020-10-30T22:48:04Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L310-L318

  So yeah that plugin hook can probably modify the list of loaders available to the `Environment`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719833744
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgzMzc0NA==
created_at: 2020-10-30T22:50:57Z
updated_at: 2020-10-30T22:50:57Z
author_association: OWNER
body:
  Yeah I'm going to remove the `load_template` plugin hook and see if it's possible to build the edit templates extension against `prepare_jinja2_environment` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1073#issuecomment-719834200
issue_url: https://api.github.com/repos/simonw/datasette/issues/1073
node_id: MDEyOklzc3VlQ29tbWVudDcxOTgzNDIwMA==
created_at: 2020-10-30T22:52:48Z
updated_at: 2020-10-30T22:52:48Z
author_association: OWNER
body:
  Should mostly be a case of backing out the changes from this commit: 81dea4b07ab2b6f4eaaf248307d2b588472054a1

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1072#issuecomment-719955491
issue_url: https://api.github.com/repos/simonw/datasette/issues/1072
node_id: MDEyOklzc3VlQ29tbWVudDcxOTk1NTQ5MQ==
created_at: 2020-10-31T16:20:58Z
updated_at: 2020-10-31T16:20:58Z
author_association: OWNER
body:
  Here's the proof of concept `FunctionLoader` that showed me that this wasn't going to work:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index 4b28e71..b076be7 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -21,7 +21,7 @@ from pathlib import Path
   from markupsafe import Markup
   from itsdangerous import URLSafeSerializer
   import jinja2
  -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
  +from jinja2 import ChoiceLoader, Environment, FileSystemLoader, FunctionLoader, PrefixLoader
   from jinja2.environment import Template
   from jinja2.exceptions import TemplateNotFound
   import uvicorn
  @@ -300,6 +300,7 @@ class Datasette:
           template_paths.append(default_templates)
           template_loader = ChoiceLoader(
               [
  +                FunctionLoader(self._load_template_from_plugins),
                   FileSystemLoader(template_paths),
                   # Support {% extends "default:table.html" %}:
                   PrefixLoader(
  @@ -322,6 +323,17 @@ class Datasette:
           self._root_token = secrets.token_hex(32)
           self.client = DatasetteClient(self)
   
  +    def _load_template_from_plugins(self, template):
  +        # "If auto reloading is enabled it’s called to check if the template changed"
  +        uptodatefunc = lambda: True
  +        source = pm.hook.load_template(
  +            template=template,
  +            datasette=self,
  +        )
  +        if source is None:
  +            return None
  +        return source, template, uptodatefunc
  +
       @property
       def urls(self):
           return Urls(self)
  @@ -719,35 +731,7 @@ class Datasette:
           else:
               if isinstance(templates, str):
                   templates = [templates]
  -
  -            # Give plugins first chance at loading the template
  -            break_outer = False
  -            plugin_template_source = None
  -            plugin_template_name = None
  -            template_name = None
  -            for template_name in templates:
  -                if break_outer:
  -                    break
  -                plugin_template_source = pm.hook.load_template(
  -                    template=template_name,
  -                    request=request,
  -                    datasette=self,
  -                )
  -                plugin_template_source = await await_me_maybe(plugin_template_source)
  -                if plugin_template_source:
  -                    break_outer = True
  -                    plugin_template_name = template_name
  -                    break
  -            if plugin_template_source is not None:
  -                template = self.jinja_env.from_string(plugin_template_source)
  -            else:
  -                template = self.jinja_env.select_template(templates)
  -            for template_name in templates:
  -                from_plugin = template_name == plugin_template_name
  -                used = from_plugin or template_name == template.name
  -                templates_considered.append(
  -                    {"name": template_name, "used": used, "from_plugin": from_plugin}
  -                )
  +            template = self.jinja_env.select_template(templates)
           body_scripts = []
           # pylint: disable=no-member
           for extra_script in pm.hook.extra_body_script(
  diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py
  index ca84b35..7804def 100644
  --- a/datasette/hookspecs.py
  +++ b/datasette/hookspecs.py
  @@ -50,7 +50,7 @@ def extra_template_vars(
   
   
   @hookspec(firstresult=True)
  -def load_template(template, request, datasette):
  +def load_template(template, datasette):
       "Load the specified template, returning the template code as a string"
   
   
  diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst
  index 3c57b6a..8f2704e 100644
  --- a/docs/plugin_hooks.rst
  +++ b/docs/plugin_hooks.rst
  @@ -273,15 +273,12 @@ Example: `datasette-cluster-map <https://github.com/simonw/datasette-cluster-map
   
   .. _plugin_hook_load_template:
   
  -load_template(template, request, datasette)
  --------------------------------------------
  +load_template(template, datasette)
  +----------------------------------
   
   ``template`` - string
       The template that is being rendered, e.g. ``database.html``
   
  -``request`` - object or None
  -    The current HTTP :ref:`internals_request`. This can be ``None`` if the request object is not available.
  -
   ``datasette`` - :ref:`internals_datasette`
       You can use this to access plugin configuration options via ``datasette.plugin_config(your_plugin_name)``
   ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


