# issue_comments

html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUa8p
created_at: 2023-08-17T22:46:40Z
updated_at: 2023-08-17T22:46:40Z
author_association: OWNER
body:
  As a reminder:
  https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUbTl
created_at: 2023-08-17T22:48:36Z
updated_at: 2023-08-17T22:48:36Z
author_association: OWNER
body:
  I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism.

  It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kUgnu
created_at: 2023-08-17T23:15:36Z
updated_at: 2023-08-17T23:15:36Z
author_association: OWNER
body:
  An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133

  Needs updating like so:
  ```diff
  diff --git a/tests/test_extract.py b/tests/test_extract.py
  index 70ad0cf..fd52534 100644
  --- a/tests/test_extract.py
  +++ b/tests/test_extract.py
  @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db):
       assert fresh_db["tree"].schema == (
           'CREATE TABLE "tree" (\n'
           "   [name] TEXT,\n"
  -        "   [common_name_latin_name_id] INTEGER,\n"
  -        "   FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n"
  +        "   [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n"
           ")"
       )
       assert (
  ```
  Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in.

  I don't think this should count as a breaking change release though, but it's still worth noting.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUkFq
created_at: 2023-08-17T23:33:14Z
updated_at: 2023-08-17T23:33:14Z
author_association: OWNER
body:
  Just one failing test left:

  ```
          # Soundness check foreign_keys point to existing tables
          for fk in foreign_keys:
              if fk.other_table == name and columns.get(fk.other_column):
                  continue
              if not any(
                  c for c in self[fk.other_table].columns if c.name == fk.other_column
              ):
  >               raise AlterError(
                      "No such column: {}.{}".format(fk.other_table, fk.other_column)
                  )
  E               sqlite_utils.db.AlterError: No such column: breeds.rowid

  sqlite_utils/db.py:882: AlterError
  ==== short test summary info ====
  FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid
  ==== 1 failed, 378 deselected in 0.49s ====
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUkOZ
created_at: 2023-08-17T23:33:58Z
updated_at: 2023-08-17T23:33:58Z
author_association: OWNER
body:
  Full test:

  https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUkrf
created_at: 2023-08-17T23:36:02Z
updated_at: 2023-08-17T23:36:02Z
author_association: OWNER
body:
  Just these three lines recreate the problem:
  ```python
  from sqlite_utils import Database

  fresh_db = Database(memory=True)

  fresh_db.create_table("dogs", {"name": str})
  fresh_db.create_table("breeds", {"name": str})
  fresh_db["dogs"].add_column("breed_id", fk="breeds")
  ```
  Traceback:
  ```
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2170, in add_column
      self.add_foreign_key(col_name, fk, fk_col)
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2273, in add_foreign_key
      self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1169, in add_foreign_keys
      self[table].transform(add_foreign_keys=fks)
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1728, in transform
      sqls = self.transform_sql(
             ^^^^^^^^^^^^^^^^^^^
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1896, in transform_sql
      self.db.create_table_sql(
    File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 882, in create_table_sql
      raise AlterError(
  sqlite_utils.db.AlterError: No such column: breeds.rowid
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUlko
created_at: 2023-08-17T23:41:10Z
updated_at: 2023-08-17T23:41:10Z
author_association: OWNER
body:
  The problem here is that the table created by this line:
  ```python
  fresh_db.create_table("breeds", {"name": str})
  ```
  Has this schema:
  ```sql
  CREATE TABLE [breeds] (
     [name] TEXT
  );
  ```
  SQLite creates an invisible `rowid` column for it automatically.

  On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added:
  ```sql
  (Pdb) print(fresh_db.schema)
  CREATE TABLE [dogs] (
     [name] TEXT
  , [breed_id] INTEGER,
     FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid])
  );
  CREATE TABLE [breeds] (
     [name] TEXT
  );
  ```
  But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884

  Here's what the debugger reveals about this code:
  ```python
          for fk in foreign_keys:
              if fk.other_table == name and columns.get(fk.other_column):
                  continue
              if not any(
                  c for c in self[fk.other_table].columns if c.name == fk.other_column
              ):
                  raise AlterError(
                      "No such column: {}.{}".format(fk.other_table, fk.other_column)
                  )
  ```
  ```
  (Pdb) fk
  ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid')
  (Pdb) self[fk.other_table].columns
  [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122490
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUmk6
created_at: 2023-08-17T23:45:44Z
updated_at: 2023-08-18T00:46:07Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`92.85%`** and project coverage change: **`-0.07%`** :warning:
  > Comparison is base [(`1dc6b5a`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/1dc6b5aa644a92d3654f7068110ed7930989ce71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.82% compared to head [(`2915050`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.76%.
  > Report is 1 commits behind head on main.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #584      +/-   ##
  ==========================================
  - Coverage   95.82%   95.76%   -0.07%     
  ==========================================
    Files           8        8              
    Lines        2829     2834       +5     
  ==========================================
  + Hits         2711     2714       +3     
  - Misses        118      120       +2     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <92.85%> (-0.15%)` | :arrow_down: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUmpP
created_at: 2023-08-17T23:46:09Z
updated_at: 2023-08-17T23:46:09Z
author_association: OWNER
body:
  Oops, `mypy` failures:

  ```
  sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]")  [assignment]
  sqlite_utils/db.py:1164: error: Need type annotation for "by_table" (hint: "by_table: Dict[<type>, <type>] = ...")  [var-annotated]
  sqlite_utils/db.py:1169: error: Item "View" of "Union[Table, View]" has no attribute "transform"  [union-attr]
  sqlite_utils/db.py:1813: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing>  [arg-type]
  sqlite_utils/db.py:1824: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing>  [arg-type]
  Found 5 errors in 1 file (checked 56 source files)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUqLr
created_at: 2023-08-18T00:06:59Z
updated_at: 2023-08-18T00:06:59Z
author_association: OWNER
body:
  The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUqmJ
created_at: 2023-08-18T00:09:20Z
updated_at: 2023-08-18T00:09:20Z
author_association: OWNER
body:
  Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop:
  ```
  ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
  ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUqro
created_at: 2023-08-18T00:09:56Z
updated_at: 2023-08-18T00:09:56Z
author_association: OWNER
body:
  Upgrading `flake8` locally replicated the error:
  ```
  pip install -U flake8
  flake8
  ```
  ```
  ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
  ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUrwr
created_at: 2023-08-18T00:14:52Z
updated_at: 2023-08-18T00:14:52Z
author_association: OWNER
body:
  Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times:

  > The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries.
  > 
  > `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up.
  > 
  > Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`.

  That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUsGW
created_at: 2023-08-18T00:16:28Z
updated_at: 2023-08-18T00:16:48Z
author_association: OWNER
body:
  One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`:

  https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708

  I should write tests for them too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUsRb
created_at: 2023-08-18T00:17:26Z
updated_at: 2023-08-18T00:17:26Z
author_association: OWNER
body:
  Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/584
node_id: IC_kwDOCGYnMM5kUw31
created_at: 2023-08-18T00:45:53Z
updated_at: 2023-08-18T00:45:53Z
author_association: OWNER
body:
  More updated documentation:
  - https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform
  - https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683195669
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/585
node_id: IC_kwDOCGYnMM5kU4cV
created_at: 2023-08-18T01:24:57Z
updated_at: 2023-08-18T01:24:57Z
author_association: OWNER
body:
  Currently:
  ```bash
  sqlite-utils transform --help
  ```
  ```
  Usage: sqlite-utils transform [OPTIONS] PATH TABLE

    Transform a table beyond the capabilities of ALTER TABLE

    Example:

        sqlite-utils transform mydb.db mytable \
            --drop column1 \
            --rename column2 column_renamed

  Options:
    --type <TEXT CHOICE>...   Change column type to INTEGER, TEXT, FLOAT or BLOB
    --drop TEXT               Drop this column
    --rename <TEXT TEXT>...   Rename this column to X
    -o, --column-order TEXT   Reorder columns
    --not-null TEXT           Set this column to NOT NULL
    --not-null-false TEXT     Remove NOT NULL from this column
    --pk TEXT                 Make this column the primary key
    --pk-none                 Remove primary key (convert to rowid table)
    --default <TEXT TEXT>...  Set default value for this column
    --default-none TEXT       Remove default from this column
    --drop-foreign-key TEXT   Drop foreign key constraint for this column
    --sql                     Output SQL without executing it
    --load-extension TEXT     Path to SQLite extension, with optional
                              :entrypoint
    -h, --help                Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683197882
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/585
node_id: IC_kwDOCGYnMM5kU4-6
created_at: 2023-08-18T01:25:53Z
updated_at: 2023-08-18T01:25:53Z
author_association: OWNER
body:
  Probably most relevant here is this snippet from:
  ```bash
  sqlite-utils create-table --help
  ```
  ```
    --default <TEXT TEXT>...  Default value that should be set for a column
    --fk <TEXT TEXT TEXT>...  Column, other table, other column to set as a
                              foreign key
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/585
node_id: IC_kwDOCGYnMM5kU5MU
created_at: 2023-08-18T01:26:47Z
updated_at: 2023-08-18T01:26:47Z
author_association: OWNER
body:
  The only CLI feature that supports providing just the column name appears to be this:
  ```bash
  sqlite-utils add-foreign-key --help
  ```
  ```
  Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE]
                                      [OTHER_COLUMN]

    Add a new foreign key constraint to an existing table

    Example:

        sqlite-utils add-foreign-key my.db books author_id authors id

    WARNING: Could corrupt your database! Back up your database file first.
  ```
  I can drop that WARNING now since I'm not writing to `sqlite_master` any more.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683200128
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/585
node_id: IC_kwDOCGYnMM5kU5iA
created_at: 2023-08-18T01:29:00Z
updated_at: 2023-08-18T01:29:00Z
author_association: OWNER
body:
  I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683201239
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/585
node_id: IC_kwDOCGYnMM5kU5zX
created_at: 2023-08-18T01:30:46Z
updated_at: 2023-08-18T01:30:46Z
author_association: OWNER
body:
  Help can now look like this:
  ```
    --drop-foreign-key TEXT         Drop foreign key constraint for this column
    --add-foreign-key <TEXT TEXT TEXT>...
                                    Add a foreign key constraint from a column
                                    to another table with another column
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/585
node_id: IC_kwDOCGYnMM5kU8cq
created_at: 2023-08-18T01:43:54Z
updated_at: 2023-08-18T01:43:54Z
author_association: OWNER
body:
  Some manual testing:
  ```bash
  sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
  sqlite-utils schema /tmp/t.db
  ```
  ```sql
  CREATE TABLE [places] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [country] INTEGER,
     [city] INTEGER,
     [continent] INTEGER
  );
  ```
  ```bash
  sqlite-utils create-table /tmp/t.db country id integer name text
  sqlite-utils create-table /tmp/t.db city id integer name text
  sqlite-utils create-table /tmp/t.db continent id integer name text
  sqlite-utils schema /tmp/t.db
  ```
  ```sql
  CREATE TABLE [places] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [country] INTEGER,
     [city] INTEGER,
     [continent] INTEGER
  );
  CREATE TABLE [country] (
     [id] INTEGER,
     [name] TEXT
  );
  CREATE TABLE [city] (
     [id] INTEGER,
     [name] TEXT
  );
  CREATE TABLE [continent] (
     [id] INTEGER,
     [name] TEXT
  );
  ```
  ```bash
  sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id
  sqlite-utils schema /tmp/t.db
  ```
  ```sql
  CREATE TABLE [country] (
     [id] INTEGER,
     [name] TEXT
  );
  CREATE TABLE [city] (
     [id] INTEGER,
     [name] TEXT
  );
  CREATE TABLE [continent] (
     [id] INTEGER,
     [name] TEXT
  );
  CREATE TABLE "places" (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [country] INTEGER REFERENCES [country]([id]),
     [city] INTEGER,
     [continent] INTEGER REFERENCES [continent]([id])
  );
  ```
  ```bash
  sqlite-utils transform /tmp/t.db places --drop-foreign-key country
  sqlite-utils schema /tmp/t.db places
  ```
  ```sql
  CREATE TABLE "places" (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [country] INTEGER,
     [city] INTEGER,
     [continent] INTEGER REFERENCES [continent]([id])
  )
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683217284
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/585
node_id: IC_kwDOCGYnMM5kU9uE
created_at: 2023-08-18T01:50:21Z
updated_at: 2023-08-18T01:50:21Z
author_association: OWNER
body:
  And a test of the `--sql` option:
  ```bash
  sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
  sqlite-utils create-table /tmp/t.db country id integer name text
  sqlite-utils create-table /tmp/t.db city id integer name text
  sqlite-utils create-table /tmp/t.db continent id integer name text
  sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql
  ```
  Outputs:
  ```sql
  CREATE TABLE [places_new_6a705d2f5a13] (
     [id] INTEGER PRIMARY KEY,
     [name] TEXT,
     [country] INTEGER REFERENCES [country]([id]),
     [city] INTEGER,
     [continent] INTEGER REFERENCES [continent]([id])
  );
  INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent])
     SELECT [id], [name], [country], [city], [continent] FROM [places];
  DROP TABLE [places];
  ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places];
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683396150
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/586
node_id: IC_kwDOCGYnMM5kVpY2
created_at: 2023-08-18T06:02:18Z
updated_at: 2023-08-18T06:06:31Z
author_association: OWNER
body:
  More notes in here:
  - https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873

  Not all Python/SQLite installations exhibit this problem by default!

  It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table

  If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view.

  Here's a one-liner to test if it is on or not:

  ```bash
  python -c 'import sqlite3; print(sqlite3.connect(":memory:").execute("PRAGMA legacy_alter_table").fetchall())'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683398866
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/586
node_id: IC_kwDOCGYnMM5kVqDS
created_at: 2023-08-18T06:05:50Z
updated_at: 2023-08-18T06:06:42Z
author_association: OWNER
body:
  Options:
  - Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default?
  - Read that `PRAGMA` and automatically do that view workaround if it's turned on
  - Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection?
  - Try the `transform()` in a transaction, detect the `"error in view"`, `"no such table"`error, if spotted then do the VIEW workaround and try again

  I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683404978
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/586
node_id: IC_kwDOCGYnMM5kVriy
created_at: 2023-08-18T06:13:46Z
updated_at: 2023-08-18T06:13:46Z
author_association: OWNER
body:
  I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1683420879
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kVvbP
created_at: 2023-08-18T06:33:24Z
updated_at: 2023-08-18T15:15:34Z
author_association: OWNER
body:
  I completely agree: metadata is a mess, and it deserves our attention.

  > 1. Metadata cannot be updated without re-starting the entire Datasette instance.

  That's not completely true - there are hacks around that. I have a plugin that applies one set of gnarly hacks for that here: https://github.com/simonw/datasette-remote-metadata - it's pretty grim though!

  > 2. The `metadata.json`/`metadata.yaml` has become a kitchen sink of unrelated (imo) features like plugin config, authentication config, canned queries

  100% this: it's a complete mess.

  Datasette used to have a `datasette --config foo:bar` mechanism, which I deprecated in favour of `datasette --setting foo bar` partly because I wanted to free up `--config` for pointing at a real config file, so we could stop dropping everything in `--metadata metadata.yml`.

  > 3. The Python APIs for defining extra metadata are a bit awkward (the `datasette.metadata()` class, `get_metadata()` hook, etc.)

  Yes, they're not pretty at all.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1683429959
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kVxpH
created_at: 2023-08-18T06:43:33Z
updated_at: 2023-08-18T15:19:07Z
author_association: OWNER
body:
  The single biggest design challenge I've had with metadata relates to how it should or should not be inherited.

  If you apply a license to a Datasette instance, it feels like that should flow down to cover all of the databases and all of the tables within those databases.

  If the license is at the database level, it should cover all tables.

  But... should source do the same thing? I made it behave the same way as license, but it's presumably common for one database to have a single license but multiple different sources of data.

  Then there's title - should that inherit? It feels like title should apply to only one level - you may want a title that applies to the instance, then a different custom title for databases and tables.

  Here's the current state of play for metadata: https://docs.datasette.io/en/1.0a3/metadata.html

  So there's `title` and `description` - and I'll be honest, I'm not 100% sure even I understand how those should be inherited down by tables/etc.

  There's `description_html` which over-rides the `description` if it is set. It's a useful customization hack, but a bit surprising.

  Then there are these six:

  - `license`
  - `license_url`
  - `source`
  - `source_url`
  - `about`
  - `about_url`

  I added `about` later than the others, because I realized that plenty of my own projects needed a link to an article explaining them somewhere - e.g. https://scotrail.datasette.io/

  Tables can also have column descriptions - just a string for each column. There's a demo of those here: https://latest.datasette.io/fixtures/roadside_attractions

  And then there's all of the other stuff, most of which feels much more like "settings" than "metadata":

  - `sort: created` - the custom sort order
  - `size: 10` for a custom page size for a specific table
  - `sortable_columns` to set which columns can be used to sort
  - `hidden: true` to hide a table
  - `label_column: title` is an interesting one - it lets you hint to Datasette which column should be displayed when there is a foreign key relationship. It's sort-of-metadata and sort-of-a-setting.
  - `facets` sets default facets, see https://docs.datasette.io/en/1.0a3/facets.html#facets-in-metadata
  - `facet_size` sets the number of facets to display
  - `fts_table` and `fts_pk` can be used to configure FTS, especially for views: https://docs.datasette.io/en/1.0a3/full_text_search.html

  And the authentication stuff! `allow` and `allow_sql` blocks: https://docs.datasette.io/en/1.0a3/authentication.html#defining-permissions-with-allow-blocks

  And the new `permissions` key in the 1.0 alphas: https://docs.datasette.io/en/1.0a3/authentication.html#other-permissions-in-metadata

  I think that might be everything (excluding the `plugins` settings stuff, which is also a bad fit for metadata.)

  And to make things even more confusing... I believe you can add arbitrary key/value pairs to your metadata and then use them in your templates! I think I've heard from at least one person who uses that ability.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1683435579
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kVzA7
created_at: 2023-08-18T06:49:39Z
updated_at: 2023-08-18T06:49:39Z
author_association: OWNER
body:
  My ideal situation then would be something like this:

  - Metadata itself is VERY clearly described, including sensible rules for metadata inheritance where it makes sense. There is a `datasette.X` method for accessing it which is much more intuitive than `datasette.metadata()`.
  - It's possible that method should be an `async` method, because that would support things like plugins that lookup metadata in database tables better.
  - All templates etc switch to the new, clean, intuitive metadata mechanism before 1.0.
  - I'm interested in the option of metadata being able to live in a `_datasette_metadata` table in the databases themselves - either as a plugin or as a core feature. I think it makes a lot of sense for metadata to optionally live with the data that it describes.
  - Configuration gets split from metadata. The stuff that configures Datasette no longer lives in the `metadata.yml` file - it lives in `config.yml` (or even `datasette.yml`).

  Currently we have three types of things:

  - Metadata - information about the data
  - Configuration - stuff like "these columns should be sortable" and "this is configured as `fts_table`" and suchlike
  - Settings - the stuff that you pass to `datasette --setting x y` on server start.

  Should settings and configuration be separate? I'm not 100% sure that they should - maybe those two concepts should be combined somehow.

  Configuration directory mode needs to be considered too: https://docs.datasette.io/en/stable/settings.html#configuration-directory-mode - interestingly it already has a thing where it can pick up settings from a `settings.json` file - where settings are things like `datasette --setting sql_time_limit_ms 4000`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1683440597
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kV0PV
created_at: 2023-08-18T06:54:49Z
updated_at: 2023-08-18T06:54:49Z
author_association: OWNER
body:
  A related point that I've been considering a lot recently: it turns out that sometimes I really want to define settings on the CLI instead of in a file, purely for convenience.

  It's pretty annoying when I want to try out a new plugin but I have to create a dedicated `metadata.yml` file for it just to setup a single option - I'd love to have the option to be able to run this instead:

  ```bash
  datasette data.db --plugin-setting datasette-upload-csvs default-database data
  ```

  So maybe there's a world in which all of the settings can be applied in a `datasette.yml` file OR with command-line options.

  That gets trickier when you need to pass a nested structure or similar, but we could always support those as JSON:

  ```bash
  datasette data.db --plugin-setting datasette-emoji-reactions emoji '["😼", "🐺"]'
  ```
  Note that we kind of have precedent for this in `datasette publish`: https://docs.datasette.io/en/stable/publish.html#custom-metadata-and-plugins

  ```bash
  datasette publish heroku my_database.db \
      --name my-heroku-app-demo \
      --install=datasette-auth-github \
      --plugin-secret datasette-auth-github client_id your_client_id \
      --plugin-secret datasette-auth-github client_secret your_client_secret
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1683443891
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kV1Cz
created_at: 2023-08-18T06:58:15Z
updated_at: 2023-08-18T06:58:15Z
author_association: OWNER
body:
  Hah, that `--plugin-secret` thing was a messy solution I came up with to the problem that all metadata is visible at `/-/metadata` - so if you need to stash a secret you need a way to keep it not-visible in there!

  Hence the whole `$env` mess: https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values

  ```json
  {
      "plugins": {
          "datasette-auth-github": {
              "client_secret": {
                  "$env": "GITHUB_CLIENT_SECRET"
              }
          }
      }
  }
  ```

  If configuration and metadata were separate we could ditch that whole messy situation - configuration can stay hidden, metadata can stay public.

  Though I have been thinking that Datasette might benefit from a "secrets" mechanism that's separate from configuration and metadata... kind of like what LLM has: https://llm.datasette.io/en/stable/help.html#llm-keys-help

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2144#issuecomment-1683963463
issue_url: https://api.github.com/repos/simonw/datasette/issues/2144
node_id: IC_kwDOBm6k_c5kXz5H
created_at: 2023-08-18T13:58:39Z
updated_at: 2023-08-18T13:58:39Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`3a97755`)](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2144   +/-   ##
  =======================================
    Coverage   92.06%   92.06%           
  =======================================
    Files          40       40           
    Lines        5937     5937           
  =======================================
    Hits         5466     5466           
    Misses        471      471           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1684202932
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kYuW0
created_at: 2023-08-18T17:10:21Z
updated_at: 2023-08-18T17:10:21Z
author_association: CONTRIBUTOR
body:
  I agree with all your points!

  I think the best solution would be having a `datasette.json` config file, where you "configure" your datasette instances, with settings, permissions/auth, plugin configuration, and table settings (sortable column, label columns, etc.). Which #2093 would do.

  Then optionally, you have a `metadata.json`, or use `datasette_metadata`, or some other plugin to define metadata (ex the future [sqlite-docs](https://github.com/asg017/sqlite-docs) plugin).

  Everything in `datasette.json` could also be overwritten by CLI flags, like `--setting key value`, `--plugin xxxx key value`.

  We could even completely remove `settings.json` in favor or just `datasette.json`. Mostly because I think the less files the better, especially if they have generic names like `settings.json` or `config.json`. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1684205563
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kYu_7
created_at: 2023-08-18T17:12:54Z
updated_at: 2023-08-18T17:12:54Z
author_association: CONTRIBUTOR
body:
  Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

  Though I imagine Python imports might make this complex to do, and json/yaml is already supported and pretty easy to write

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1684235760
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/577
node_id: IC_kwDOCGYnMM5kY2Xw
created_at: 2023-08-18T17:43:11Z
updated_at: 2023-08-18T17:43:11Z
author_association: OWNER
body:
  Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it:

  https://github.com/simonw/sqlite-utils-fast-fks

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684384750
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZavu
created_at: 2023-08-18T20:07:18Z
updated_at: 2023-08-18T20:07:18Z
author_association: OWNER
body:
  The big challenge here is what the URL to that row page should look like. How can I encode a `None` in a form that can be encoded and decoded without clashing with primary keys that are the string `None` or `null`?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kZzFK
created_at: 2023-08-18T22:12:52Z
updated_at: 2023-08-18T22:12:52Z
author_association: OWNER
body:
  Yeah, I'm convinced by that. There's not point in having both `settings.json` and `datasette.json`.

  I like `datasette.json` ( / `datasette.yml`) as a name. That can be the file that lives in your config directory too, so if you run `datasette .` in a folder containing `datasette.yml` all of those settings get picked up.

  Here's a thought for how it could look - I'll go with the YAML format because I expect that to be the default most people use, just because it supports multi-line strings better.

  I based this on the big example at https://docs.datasette.io/en/1.0a3/metadata.html#using-yaml-for-metadata - and combined some bits from https://docs.datasette.io/en/1.0a3/authentication.html as well.

  ```yaml
  title: Demonstrating Metadata from YAML
  description_html: |-
    <p>This description includes a long HTML string</p>
    <ul>
      <li>YAML is better for embedding HTML strings than JSON!</li>
    </ul>

  settings:
    default_page_size: 10
    max_returned_rows: 3000
    sql_time_limit_ms": 8000

  databases:
    docs:
      permissions:
        create-table:
          id: editor
    fixtures:
      tables:
        no_primary_key:
          hidden: true
      queries:
        neighborhood_search:
          sql: |-
            select neighborhood, facet_cities.name, state
            from facetable join facet_cities on facetable.city_id = facet_cities.id
            where neighborhood like '%' || :text || '%' order by neighborhood;
          title: Search neighborhoods
          description_html: |-
            <p>This demonstrates <em>basic</em> LIKE search

  permissions:
    debug-menu:
      id: '*'

  plugins:
    datasette-ripgrep:
      path: /usr/local/lib/python3.11/site-packages
  ```
  I'm inclined to say we try to be a super-set of the existing `metadata.yml` format, at least where it makes sense to do so. That way the upgrade path is smooth for people. Also, I don't think the format itself is terrible - it's the name that's the big problem.

  In this example I've mixed in one extra concept: that `settings:` block with a bunch of settings in it.

  There are some things in there that look a little bit like metadata - the `title` and `description_html` fields.

  But _are they_ metadata? The title and description of the overall instance feels like it could be described as general configuration. The stuff for the `query` should live where the query itself is defined.

  Note that queries can be defined by a plugin hook too: https://docs.datasette.io/en/1.0a3/plugin_hooks.html#canned-queries-datasette-database-actor

  What do you think? Is this the right direction, or are you thinking there's a more radical redesign that would make sense here?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1684485591
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kZzXX
created_at: 2023-08-18T22:14:35Z
updated_at: 2023-08-18T22:14:35Z
author_association: OWNER
body:
  Actually there is one thing that I'm not comfortable about with respect to the existing design: the way the database / tables stuff is nested.

  They assume that the user will attach the database to Datasette using a fixed name - `docs.db` or whatever.

  But what if we want to support users downloading databases from each other and attaching them to Datasette where those DBs might carry some of their own configuration?

  Moving metadata into the databases makes sense there, but what about database-specific settings like the default sort order for a table, or configured canned queries?

  Having those tied to the filename of the database itself feels unpleasant to me. But how else could we handle this?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1684488526
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kZ0FO
created_at: 2023-08-18T22:18:39Z
updated_at: 2023-08-18T22:18:39Z
author_association: OWNER
body:
  > Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

  > Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

  I'm not a fan of that. I feel like software history is full of examples of projects that implemented configuration-as-code and then later regretted it - the most recent example is `setup.py` in Python turning into `pyproject.yaml`, but I feel like I've seen that pattern play out elsewhere too.

  I don't think having people dynamically generate JSON/YAML for their configuration is a big burden. I'd have to see some very compelling use-cases to convince me otherwise.

  That said, I do really like a bias towards settings that can be changed at runtime. Datasette has suffered a bit from some settings that can't be easily changed at runtime already - hence my gnarly https://github.com/simonw/datasette-remote-metadata plugin.

  For things like Datasette Cloud for example the more people can configure without rebooting their container the better!

  I don't think live reconfiguration at runtime is incompatible with JSON/YAML configuration though. Caddy is one of my favourite examples of software that can be entirely re-configured at runtime by POSTING a big blob of JSON to it: https://caddyserver.com/docs/quick-starts/api

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684494464
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ1iA
created_at: 2023-08-18T22:27:51Z
updated_at: 2023-08-18T22:28:40Z
author_association: OWNER
body:
  Oh wow, null primary keys are bad news... SQLite lets you insert multiple rows with the same `null` value!
  ```pycon
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database(memory=True)
  >>> db["foo"].insert({"id": None, "name": "No ID"}, pk="id")
  <Table foo (id, name)>
  >>> db.schema
  'CREATE TABLE [foo] (\n   [id] TEXT PRIMARY KEY,\n   [name] TEXT\n);'
  >>> db["foo"].insert({"id": None, "name": "No ID"}, pk="id")
  <Table foo (id, name)>
  >>> db.schema
  'CREATE TABLE [foo] (\n   [id] TEXT PRIMARY KEY,\n   [name] TEXT\n);'
  >>> list(db["foo"].rows)
  [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}]
  >>> list(db.query('select * from foo where id = null'))
  []
  >>> list(db.query('select * from foo where id is null'))
  [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684495674
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ106
created_at: 2023-08-18T22:29:47Z
updated_at: 2023-08-18T22:29:47Z
author_association: OWNER
body:
  https://www.sqlite.org/lang_createtable.html#the_primary_key says:

  >According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid) or the table is a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table or a [STRICT](https://www.sqlite.org/stricttables.html) table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1684496274
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kZ1-S
created_at: 2023-08-18T22:30:45Z
updated_at: 2023-08-18T22:30:45Z
author_association: CONTRIBUTOR
body:
  > That said, I do really like a bias towards settings that can be changed at runtime

  Does this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff. 

  Well it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684497000
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ2Jo
created_at: 2023-08-18T22:31:53Z
updated_at: 2023-08-18T22:31:53Z
author_association: OWNER
body:
  So it sounds like SQLite does ensure that a `rowid` before it allows a primary key to be null.

  So one solution here would be to detect a null primary key and switch that table over to using `rowid` URLs instead. The key problem we're trying to solve here after all is how to link to a row:

  https://latest.datasette.io/fixtures/infinity/1

  But when would we run that check? And does every row in the table get a new `/rowid/` URL just because someone messed up and inserted a `null` by mistake?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684497642
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ2Tq
created_at: 2023-08-18T22:32:53Z
updated_at: 2023-08-18T22:32:53Z
author_association: OWNER
body:
  Here's a potential solution: make it so ALL `rowid` tables in SQLite can be optionally addressed by their `rowid` instead of by their primary key.

  Then teach the code that outputs the URL to a row page to spot if there are `null` primary keys and switch to that alternative addressing mechanism instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684498947
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ2oD
created_at: 2023-08-18T22:35:04Z
updated_at: 2023-08-18T22:35:04Z
author_association: OWNER
body:
  The most interesting row URL in the fixtures database right now is this one:

  https://latest.datasette.io/fixtures/compound_primary_key/a~2Fb,~2Ec-d

  <img width="163" alt="CleanShot 2023-08-18 at 15 34 52@2x" src="https://github.com/simonw/datasette/assets/9599/dd79b0b6-e1c3-4e30-b64b-973b062bfe70">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684500172
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ27M
created_at: 2023-08-18T22:37:04Z
updated_at: 2023-08-18T22:37:04Z
author_association: OWNER
body:
  Looking at the way these URLs work: because the components themselves in `a~2Fb,~2Ec-d` are tilde-encoded, any character that's "safe" in tilde-encoding could be used to indicate "this is actually a rowid".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684500540
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ3A8
created_at: 2023-08-18T22:37:37Z
updated_at: 2023-08-18T22:37:37Z
author_association: OWNER
body:
  I just found this and panicked, thinking maybe tilde encoding is a bad idea after all! https://jkorpela.fi/tilde.html

  But... "Date of last update: 1999-08-27" - I think I'm OK.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684502278
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ3cG
created_at: 2023-08-18T22:40:20Z
updated_at: 2023-08-18T22:40:20Z
author_association: OWNER
body:
  From reviewing https://simonwillison.net/2022/Mar/19/weeknotes/

      unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"

  That's how I chose the tilde character - but it also suggests that I could use `-` or `.` or `_` for my new `rowid` encoding.

  So maybe `/database/table/_4` could indicate "the row with `rowid` of 4".

  No, that doesn't work:
  ```pycon
  >>> from datasette.utils import tilde_encode
  >>> tilde_encode("_")
  '_'
  ```
  I need a character which tilde-encoding does indeed encode.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684503189
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ3qV
created_at: 2023-08-18T22:41:51Z
updated_at: 2023-08-18T22:41:51Z
author_association: OWNER
body:
  ```pycon
  >>> tilde_encode("~")
  '~7E'
  >>> tilde_encode(".")
  '~2E'
  >>> tilde_encode("-")
  '-'
  ```
  I think `.` might be the way to do this:

       /database/table/.4

  But... I worry about that colliding with my URL routing code that spots the difference between these:

       /database/table/.4
       /database/table/.4.json
       /database/table/.4.csv

  etc.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684503587
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ3wj
created_at: 2023-08-18T22:42:28Z
updated_at: 2023-08-18T22:42:39Z
author_association: OWNER
body:
  I could set a rule that extensions (including custom render extensions set by plugins) must not be valid integers, and teach Datasette that `/\.\d+` is the indication of a `rowid`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684504051
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ33z
created_at: 2023-08-18T22:43:06Z
updated_at: 2023-08-18T22:43:06Z
author_association: OWNER
body:
  Here's the regex in question at the moment: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/app.py#L1387-L1390

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684504398
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ39O
created_at: 2023-08-18T22:43:31Z
updated_at: 2023-08-18T22:43:46Z
author_association: OWNER
body:
  `(?P<pks>[^/]+?)` could instead be a regex that is restricted to the tilde-encoded set of characters, or `\.\d+`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684505071
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ4Hv
created_at: 2023-08-18T22:44:35Z
updated_at: 2023-08-18T22:44:35Z
author_association: OWNER
body:
  Also relevant:
  https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L1147-L1153

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684522567
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ8ZH
created_at: 2023-08-18T22:58:07Z
updated_at: 2023-08-18T22:58:07Z
author_association: OWNER
body:
  Here's a prototype of that:
  ```diff
  diff --git a/datasette/app.py b/datasette/app.py
  index b2644ace..acc55249 100644
  --- a/datasette/app.py
  +++ b/datasette/app.py
  @@ -1386,7 +1386,7 @@ class Datasette:
           )
           add_route(
               RowView.as_view(self),
  -            r"/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[^/]+?)(\.(?P<format>\w+))?$",
  +            r"/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[A-Za-z0-9\_\-\~]+|\.\d+)(\.(?P<format>\w+))?$",
           )
           add_route(
               TableInsertView.as_view(self),
  @@ -1440,7 +1440,15 @@ class Datasette:
       async def resolve_row(self, request):
           db, table_name, _ = await self.resolve_table(request)
           pk_values = urlsafe_components(request.url_vars["pks"])
  -        sql, params, pks = await row_sql_params_pks(db, table_name, pk_values)
  +
  +        if len(pk_values) == 1 and pk_values[0].startswith("."):
  +            # It's a special .rowid value
  +            pk_values = (pk_values[0][1:],)
  +            sql, params, pks = await row_sql_params_pks(
  +                db, table_name, pk_values, rowid=True
  +            )
  +        else:
  +            sql, params, pks = await row_sql_params_pks(db, table_name, pk_values)
           results = await db.execute(sql, params, truncate=True)
           row = results.first()
           if row is None:
  diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py
  index c388673d..96669281 100644
  --- a/datasette/utils/__init__.py
  +++ b/datasette/utils/__init__.py
  @@ -1206,9 +1206,12 @@ def truncate_url(url, length):
       return url[: length - 1] + "…"
   
   
  -async def row_sql_params_pks(db, table, pk_values):
  +async def row_sql_params_pks(db, table, pk_values, rowid=False):
       pks = await db.primary_keys(table)
  -    use_rowid = not pks
  +    if rowid:
  +        use_rowid = True
  +    else:
  +        use_rowid = not pks
       select = "*"
       if use_rowid:
           select = "rowid, *"
  ```
  It works:

  <img width="649" alt="CleanShot 2023-08-18 at 15 57 36@2x" src="https://github.com/simonw/datasette/assets/9599/93803f41-d194-4193-8320-7c47d0373372">

  <img width="649" alt="CleanShot 2023-08-18 at 15 57 53@2x" src="https://github.com/simonw/datasette/assets/9599/28d28efa-5ee0-469b-a31d-ce00f2d1f77a">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684523322
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ8k6
created_at: 2023-08-18T22:59:14Z
updated_at: 2023-08-18T22:59:14Z
author_association: OWNER
body:
  Except it looks like the Links from other tables section is broken:

  <img width="649" alt="CleanShot 2023-08-18 at 15 58 54@2x" src="https://github.com/simonw/datasette/assets/9599/ddebf0a7-9855-4cf0-b283-f06f8c25414d">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684525054
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ8_-
created_at: 2023-08-18T23:02:26Z
updated_at: 2023-08-18T23:02:26Z
author_association: OWNER
body:
  Creating a quick test database:
  ```bash
  sqlite-utils create-table nulls.db nasty id text --pk id
  sqlite-utils nulls.db 'insert into nasty (id) values (null)'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684525943
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ9N3
created_at: 2023-08-18T23:04:14Z
updated_at: 2023-08-18T23:04:14Z
author_association: OWNER
body:
  This is hard. I tried this:
  ```python
  def path_from_row_pks(row, pks, use_rowid, quote=True):
      """Generate an optionally tilde-encoded unique identifier
      for a row from its primary keys."""
      if use_rowid or any(row[pk] is None for pk in pks):
          bits = [row["rowid"]]
      else:
          bits = [
              row[pk]["value"] if isinstance(row[pk], dict) else row[pk] for pk in pks
          ]
      if quote:
          bits = [tilde_encode(str(bit)) for bit in bits]
      else:
          bits = [str(bit) for bit in bits]

      return ",".join(bits)
  ```
  The `    if use_rowid or any(row[pk] is None for pk in pks)` bit is new.

  But I got this error on http://127.0.0.1:8003/nulls/nasty :

  ```
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 1364, in run_display_columns_and_rows
      display_columns, display_rows = await display_columns_and_rows(
    File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 186, in display_columns_and_rows
      pk_path = path_from_row_pks(row, pks, not pks, False)
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 124, in path_from_row_pks
      bits = [row["rowid"]]
  IndexError: No item with that key
  ```
  Because the SQL query I ran to populate the page didn't know that it would need to select `rowid` as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1684530060
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kZ-OM
created_at: 2023-08-18T23:09:03Z
updated_at: 2023-08-18T23:09:14Z
author_association: OWNER
body:
  Ran a quick benchmark on ChatGPT Code Interpreter: https://chat.openai.com/share/8357dc01-a97e-48ae-b35a-f06249935124

  Conclusion from there is that this query returns fast no matter how much the table grows:

  ```sql
  SELECT EXISTS(SELECT 1 FROM "nasty" WHERE "id" IS NULL)
  ```
  So detecting if a table contains any null primary keys is definitely feasible without a performance hit.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096129
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/587
node_id: IC_kwDOCGYnMM5kcIbB
created_at: 2023-08-19T20:03:00Z
updated_at: 2023-08-19T20:03:00Z
author_association: OWNER
body:
  Simplest possible recreation of the bug:
  ```bash
  python -c '
  import sqlite_utils
  db = sqlite_utils.Database(memory=True)
  db.execute("""
  CREATE TABLE "logs" (
     [id] INTEGER PRIMARY KEY,
     [chat_id] INTEGER REFERENCES [log]([id]),
     [reply_to_id] INTEGER
  );
  """)
  db["logs"].add_foreign_key("reply_to_id", "logs", "id")
  '
  ```
  That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096284
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/587
node_id: IC_kwDOCGYnMM5kcIdc
created_at: 2023-08-19T20:03:59Z
updated_at: 2023-08-19T20:03:59Z
author_association: OWNER
body:
  Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in:
  - #577

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096381
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/587
node_id: IC_kwDOCGYnMM5kcIe9
created_at: 2023-08-19T20:04:32Z
updated_at: 2023-08-19T20:04:32Z
author_association: OWNER
body:
  I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1685259985
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kcwbR
created_at: 2023-08-20T11:27:21Z
updated_at: 2023-08-20T11:27:21Z
author_association: NONE
body:
  To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate "settings" from "configuration" is, I believe, an uphill fight. Even separating db/table-scope settings from pure descriptive metadata is not always easy. Like, do canned queries belong to database metadata or to settings? Do I need two separate files for this?

  One pragmatic solution I used in a project is stacking yaml configuration files. Basically, have an arbitrary number of yaml or json settings files that you load in a specified order. Every file adds to the corresponding settings in the earlier-loaded file (if it already existed). I implemented this myself but found later that there is an existing Python "cascading dict" type of thing, I forget what it's called. There is a bit of a challenge deciding whether there is "replacement" or "addition" (I think I pragmatically ran `update` on the second level of the dict but better solutions are certainly possible). 

  This way, one allows separation of settings into different blocks, while not imposing a specific idea of what belongs where that might not apply equally to all cases.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1685263948
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kcxZM
created_at: 2023-08-20T11:50:10Z
updated_at: 2023-08-20T11:50:10Z
author_association: NONE
body:
  This also makes it simple to separate out secrets.

  `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml`

  settings.yaml
  ```
  settings:
    default_page_size: 10
    max_returned_rows: 3000
    sql_time_limit_ms": 8000
  plugins:
    datasette-ripgrep:
      path: /usr/local/lib/python3.11/site-packages
  ```

  secrets.yaml
  ```
  plugins:
    datasette-auth-github:
              client_secret: SUCH_SECRET 
  ```


  db-docs.yaml
  ```
  databases:
    docs:
      permissions:
        create-table:
          id: editor
  ```

  db-fixtures.yaml
  ```
  databases:
    fixtures:
      tables:
        no_primary_key:
          hidden: true
      queries:
        neighborhood_search:
          sql: |-
            select neighborhood, facet_cities.name, state
            from facetable join facet_cities on facetable.city_id = facet_cities.id
            where neighborhood like '%' || :text || '%' order by neighborhood;
          title: Search neighborhoods
          description_html: |-
            <p>This demonstrates <em>basic</em> LIKE search
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1685471752
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kdkII
created_at: 2023-08-21T01:07:23Z
updated_at: 2023-08-21T01:07:23Z
author_association: NONE
body:
  @simonw, since you're referencing "rowid" column by name, I just want to note that there may be an existing rowid column with completely different semantics (https://www.sqlite.org/lang_createtable.html#rowid), which is likely to break this logic. I don't see a good way to detect a proper "rowid" name short of checking if there is a field with that name and using the alternative (`_rowid_` or `oid`), which is not ideal, but may work.

  In terms of the original issue, maybe a way to deal with it is to use rowid by default and then use primary key for WITHOUT ROWID tables (as they are guaranteed to be not null), but I suspect it may require significant changes to the API (and doesn't fully address the issue of what value to pass to indicate NULL when editing records). Would it make sense to generate a random string to indicate NULL values when editing?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1686683596
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kiL_M
created_at: 2023-08-21T16:49:12Z
updated_at: 2023-08-21T16:49:12Z
author_association: OWNER
body:
  Suggestion from @asg017 is that we say that if your row has a null primary key you don't get a link to a row page for that row.

  Which has some precedent, because our SQL view display doesn't link to row pages at all (since they don't make sense for views): https://latest.datasette.io/fixtures/simple_view

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2145#issuecomment-1686745094
issue_url: https://api.github.com/repos/simonw/datasette/issues/2145
node_id: IC_kwDOBm6k_c5kibAG
created_at: 2023-08-21T17:30:01Z
updated_at: 2023-08-21T17:30:01Z
author_association: CONTRIBUTOR
body:
  Another point: The new Datasette write API should refuse to insert a row with a NULL primary key. That will likely decrease the likelihood someone find themselves with NULLs in their primary keys, at least with Datasette users. Especially buggy code that uses the write API, like our `datasette-write-ui` bug that led to this issue.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2147#issuecomment-1686747420
issue_url: https://api.github.com/repos/simonw/datasette/issues/2147
node_id: IC_kwDOBm6k_c5kibkc
created_at: 2023-08-21T17:31:42Z
updated_at: 2023-08-21T17:34:19Z
author_association: OWNER
body:
  Are you talking just about queries submitted to `/database?sql=` using the interface on https://latest.datasette.io/fixtures?sql=select+*+from+facetable or are you interested in queries that are run to power other pages like https://latest.datasette.io/fixtures/facetable as well? I'll assume the former.

  There are a few ways you could solve this at the moment.

  The easiest would be with a piece of ASGI middleware that looks for URLs matching `/dbname?sql=...` and logs those. I played with a version of that a few years ago: https://simonwillison.net/2019/Dec/16/logging-sqlite-asgi-middleware/ - see also https://github.com/simonw/asgi-log-to-sqlite

  Then you can load that middleware from a plugin using https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette

  That feels a bit delicate because it's relying on the URL design not changing, but I'm happy to confirm that URL is going to stay the same for Datasette 1.0 and I have no plans to change it ever.

  There's also a tracing mechanism built into Datasette itself that you could hook into. The internals of that are documented here: https://docs.datasette.io/en/stable/internals.html#datasette-tracer - but I don't yet consider it a 100% stable API. I don't plan to change it but I won't promise not to either.

  I used that mechanism in this plugin: https://datasette.io/plugins/datasette-pretty-traces - demonstrated here: https://latest-with-plugins.datasette.io/github/commits?_trace=1

  The hackiest way to do this would be to patch Datasette itself and try to replace the `query_view`. This definitely isn't a documented, stable API though and would be very likely to break at arbitrary points in the future.

  So my recommendation for the moment is the ASGI middleware option.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2147#issuecomment-1686749342
issue_url: https://api.github.com/repos/simonw/datasette/issues/2147
node_id: IC_kwDOBm6k_c5kicCe
created_at: 2023-08-21T17:33:11Z
updated_at: 2023-08-21T17:33:11Z
author_association: OWNER
body:
  I'm definitely open to suggestions for plugin hooks that might make this kind of thing easier.

  One idea I've been mulling is whether there should be a plugin hook that files on arbitrary views - similar to Django's `process_view` mechanism: https://docs.djangoproject.com/en/4.2/topics/http/middleware/#process-view

  That would allow people to setup code that runs before or after any of the default views in Datasette.

  I'm not yet 100% sold on the idea, because I worry about implementing it in a way that guarantees plugins won't break on future releases. But I'm open to considering it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2147#issuecomment-1687433388
issue_url: https://api.github.com/repos/simonw/datasette/issues/2147
node_id: IC_kwDOBm6k_c5klDCs
created_at: 2023-08-22T05:05:33Z
updated_at: 2023-08-22T05:05:33Z
author_association: NONE
body:
  Thanks for all this! You're totally right that the ASGI option is doable, if a bit low level and coupled to the current URI design. I'm totally fine with that being the final answer.

  process_view is interesting and in the general direction of what I had in mind.

  A somewhat less powerful idea: Is there value in giving a hook for just the query that's about to be run? Maybe I'm thinking a little narrowly about this problem I decided I wanted to solve, but I could see other uses for a hook of the sketch below:

  ```
  def prepare_query(database, table, query):
     """Modify query that is about to be run in some way. Return the (possibly rewritten) query to run, or None to disallow running the query"""
  ```
  (Maybe you actually want to return a tuple so there can be an error message when you disallow, or something.)

  Maybe it's too narrowly useful and some of the other pieces of datasette obviate some of these ideas, but off the cuff I could imagine using it to:
  * Require a LIMIT. Either fail the query or add the limit if it's not there.
  * Do logging, like my usecase.
  * Do other analysis on whether you want to allow the query to run; a linter? query complexity? 

  Definitely feel free to say no, or not now. This is all me just playing around with what datasette and its plugin architecture can do with toy ideas, so don't let me push you to commit to a hook you don't feel confident fits well in the design.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2093#issuecomment-1688532012
issue_url: https://api.github.com/repos/simonw/datasette/issues/2093
node_id: IC_kwDOBm6k_c5kpPQs
created_at: 2023-08-22T16:21:40Z
updated_at: 2023-08-22T16:21:40Z
author_association: CONTRIBUTOR
body:
  OK Here's the gameplan for this, which is closely tied to #2143 :

  - We will add a new `datasette.json`/`datasette.yaml` configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format
  - Metadata will NOT be a part of this file
  - TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML
  - The `settings.json` file will be deprecated, and the `--config` arg will be brought back.
  - Command line arguments can still be used to overwrite values (ex `--setting` will overwrite settings in `datasette.yaml`

  The format of `datasette.json` will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426

  Here's the current implementation plan:

  1. Add a new `--config` flag and port over `"settings"` into a new datasette.json config file, remove settings.json
  2. Add top-level plugin config support to `datasette.json`
  3. Figure out database/table structure of config `datasette.json`
  4. Port over database/table level plugin config support `datasette.json`
  5. Port over permissions/auth settings to `datasette.json`
  6. Deprecate non-metadata values in `metadata.json`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2149#issuecomment-1688547401
issue_url: https://api.github.com/repos/simonw/datasette/issues/2149
node_id: IC_kwDOBm6k_c5kpTBJ
created_at: 2023-08-22T16:30:41Z
updated_at: 2023-08-22T16:30:41Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`76.47%`** and project coverage change: **`+0.10%`** :tada:
  > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`db720cd`)](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.17%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2149      +/-   ##
  ==========================================
  + Coverage   92.06%   92.17%   +0.10%     
  ==========================================
    Files          40       40              
    Lines        5937     5916      -21     
  ==========================================
  - Hits         5466     5453      -13     
  + Misses        471      463       -8     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `81.02% <25.00%> (+1.20%)` | :arrow_up: |
  | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.29% <92.30%> (-0.09%)` | :arrow_down: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2149#issuecomment-1689125244
issue_url: https://api.github.com/repos/simonw/datasette/issues/2149
node_id: IC_kwDOBm6k_c5krgF8
created_at: 2023-08-23T01:23:27Z
updated_at: 2023-08-23T01:23:27Z
author_association: OWNER
body:
  This is a really great start - tests pass, implementation looks clean, the new stubbed documentation page makes sense. Let's land it in `main` and iterate on it further in future PRs.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/493#issuecomment-1689128911
issue_url: https://api.github.com/repos/simonw/datasette/issues/493
node_id: IC_kwDOBm6k_c5krg_P
created_at: 2023-08-23T01:29:20Z
updated_at: 2023-08-23T01:29:20Z
author_association: OWNER
body:
  It's going to be called `datasette.json` and the concept of metadata will be split out separately. See:

  - #2149 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2148#issuecomment-1689133247
issue_url: https://api.github.com/repos/simonw/datasette/issues/2148
node_id: IC_kwDOBm6k_c5kriC_
created_at: 2023-08-23T01:36:42Z
updated_at: 2023-08-23T03:09:43Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage has no change and project coverage change: **`+0.01%`** :tada:
  > Comparison is base [(`17ec309`)](https://app.codecov.io/gh/simonw/datasette/commit/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.71% compared to head [(`a651d3c`)](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%.
  > Report is 2 commits behind head on main.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2148      +/-   ##
  ==========================================
  + Coverage   92.71%   92.73%   +0.01%     
  ==========================================
    Files          40       40              
    Lines        5919     5919              
  ==========================================
  + Hits         5488     5489       +1     
  + Misses        431      430       -1     
  ```


  [see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2148/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/516#issuecomment-1689153446
issue_url: https://api.github.com/repos/simonw/datasette/issues/516
node_id: IC_kwDOBm6k_c5krm-m
created_at: 2023-08-23T02:06:35Z
updated_at: 2023-08-23T02:06:35Z
author_association: OWNER
body:
  I just tried this again today by dropping this into `.isort.cfg`:
  ```ini
  [settings]
  multi_line_output=3
  include_trailing_comma=True
  force_grid_wrap=0
  use_parentheses=True
  line_length=88
  known_first_party=datasette
  ```
  And running this in the root of the project:
  ```
  isort .
  ```
  It produced a huge diff, but when I ran the tests:
  ```bash
  pytest -n auto
  ```
  The tests all passed. I'll push a PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/516#issuecomment-1689154837
issue_url: https://api.github.com/repos/simonw/datasette/issues/516
node_id: IC_kwDOBm6k_c5krnUV
created_at: 2023-08-23T02:08:33Z
updated_at: 2023-08-23T02:08:50Z
author_association: OWNER
body:
  Browse this commit to see the result: https://github.com/simonw/datasette/tree/59a5d336bd4336bc53103922ada4bf726f4336c9

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/516#issuecomment-1689158712
issue_url: https://api.github.com/repos/simonw/datasette/issues/516
node_id: IC_kwDOBm6k_c5kroQ4
created_at: 2023-08-23T02:14:45Z
updated_at: 2023-08-23T02:14:45Z
author_association: OWNER
body:
  Thinking about this again today. Posted about it on Discord: https://discord.com/channels/823971286308356157/823971286941302908/1143729300349132933

  I won't enforce it in a `pytest` test, I'll enforce it with a CI lint check that's also in the `Justfile` here instead:

  https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/Justfile#L19-L23

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/516#issuecomment-1689159200
issue_url: https://api.github.com/repos/simonw/datasette/issues/516
node_id: IC_kwDOBm6k_c5kroYg
created_at: 2023-08-23T02:15:36Z
updated_at: 2023-08-23T02:15:36Z
author_association: OWNER
body:
  This could play havoc with unmerged PRs. I should merge any big ones (like the JavaScript plugins work) first.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2148#issuecomment-1689173748
issue_url: https://api.github.com/repos/simonw/datasette/issues/2148
node_id: IC_kwDOBm6k_c5krr70
created_at: 2023-08-23T02:38:31Z
updated_at: 2023-08-23T02:38:31Z
author_association: OWNER
body:
  Sphinx dropped support for Python 3.8 in their version 7.2.0:

  - https://www.sphinx-doc.org/en/master/changes.html#release-7-2-0-released-aug-17-2023
  - https://www.sphinx-doc.org/en/master/internals/release-process.html#python-version-support-policy
  - https://github.com/sphinx-doc/sphinx/pull/11511
  - 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2148#issuecomment-1689175062
issue_url: https://api.github.com/repos/simonw/datasette/issues/2148
node_id: IC_kwDOBm6k_c5krsQW
created_at: 2023-08-23T02:40:46Z
updated_at: 2023-08-23T02:40:46Z
author_association: OWNER
body:
  Here's why the tests are failing:

  https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/.github/workflows/test.yml#L30-L46

  It looks like those tests don't actually need Sphinx installed - they install `pip install -e '.[test,docs]'` to get the other docs dependencies:

  https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/setup.py#L70-L80

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2148#issuecomment-1689177556
issue_url: https://api.github.com/repos/simonw/datasette/issues/2148
node_id: IC_kwDOBm6k_c5krs3U
created_at: 2023-08-23T02:44:29Z
updated_at: 2023-08-23T02:44:29Z
author_association: OWNER
body:
  Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2148#issuecomment-1689198413
issue_url: https://api.github.com/repos/simonw/datasette/issues/2148
node_id: IC_kwDOBm6k_c5krx9N
created_at: 2023-08-23T02:57:55Z
updated_at: 2023-08-23T02:57:55Z
author_association: CONTRIBUTOR
body:
  Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry!

  If you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2147#issuecomment-1689206170
issue_url: https://api.github.com/repos/simonw/datasette/issues/2147
node_id: IC_kwDOBm6k_c5krz2a
created_at: 2023-08-23T03:05:32Z
updated_at: 2023-08-23T03:05:32Z
author_association: OWNER
body:
  Interestingly enough there's actually a mechanism that looks like that a bit already: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/views/database.py#L496-L508

  That `validate_sql_select()` function is defined here: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L256-L265

  Against these constants:

  https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L223-L253

  Which isn't a million miles away from your suggestion to have a hook that can say if the query should be executed or not.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2147#issuecomment-1689206768
issue_url: https://api.github.com/repos/simonw/datasette/issues/2147
node_id: IC_kwDOBm6k_c5krz_w
created_at: 2023-08-23T03:06:32Z
updated_at: 2023-08-23T03:06:32Z
author_association: OWNER
body:
  I'm less convinced by the "rewrite the query in some way" optional idea. What kind of use-cases can you imagine for that?

  My hunch is that it's much more likely to cause weird breakages than it is to allow for useful plugin extensions, but I'm willing to be convinced otherwise.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2150#issuecomment-1690431509
issue_url: https://api.github.com/repos/simonw/datasette/issues/2150
node_id: IC_kwDOBm6k_c5kwfAV
created_at: 2023-08-23T18:22:47Z
updated_at: 2023-08-23T18:22:47Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/static/app.css#L485-L489

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2150#issuecomment-1690432928
issue_url: https://api.github.com/repos/simonw/datasette/issues/2150
node_id: IC_kwDOBm6k_c5kwfWg
created_at: 2023-08-23T18:23:26Z
updated_at: 2023-08-23T18:23:26Z
author_association: OWNER
body:
  That should be scoped just to the query filters form on the table page.

  I'll fix it in `main` but I'm still going to ship a fix for those plugins separately so they work well before people upgrade to the next Datasette release with this change in it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2150#issuecomment-1690435866
issue_url: https://api.github.com/repos/simonw/datasette/issues/2150
node_id: IC_kwDOBm6k_c5kwgEa
created_at: 2023-08-23T18:25:51Z
updated_at: 2023-08-23T18:25:51Z
author_association: OWNER
body:
  Looks like that affects a few forms:

  <img width="701" alt="CleanShot 2023-08-23 at 11 24 46@2x" src="https://github.com/simonw/datasette/assets/9599/7145aec1-832e-4eea-84a6-a735df4dfd6e">

  The search form on the table page over-rides it already:

  <img width="1243" alt="CleanShot 2023-08-23 at 11 25 26@2x" src="https://github.com/simonw/datasette/assets/9599/e002089e-e992-4cc5-bb6b-c54cf614ec95">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2150#issuecomment-1690438270
issue_url: https://api.github.com/repos/simonw/datasette/issues/2150
node_id: IC_kwDOBm6k_c5kwgp-
created_at: 2023-08-23T18:27:47Z
updated_at: 2023-08-23T18:27:47Z
author_association: OWNER
body:
  I added `outline: 3px dotted pink` to that `form label` block to help spot where else it's being used.

  Oh interesting, looks like it's over-ridden here too:

  <img width="950" alt="CleanShot 2023-08-23 at 11 27 15@2x" src="https://github.com/simonw/datasette/assets/9599/f3f3e995-2d6c-4afe-8370-01e0b5141a64">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1690693830
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5kxfDG
created_at: 2023-08-23T21:51:52Z
updated_at: 2023-08-23T21:52:58Z
author_association: OWNER
body:
  This is the hook in question: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/hookspecs.py#L108-L110

  - `True` means they are allowed to access it. You only need a single`True` from a plugin to allow it.
  - `False` means they are not, and just one `False` from a plugin will deny it (even if another one returned `True` I think)
  - `None` means that the plugin has no opinion on this question.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1690703764
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5kxheU
created_at: 2023-08-23T22:02:14Z
updated_at: 2023-08-23T22:02:14Z
author_association: OWNER
body:
  Built this new test:
  ```python
  @pytest.mark.asyncio
  async def test_view_table_token_can_access_table(perms_ds):
      actor = {
          "id": "restricted-token",
          "token": "dstok",
          # Restricted to just view-table on perms_ds_two/t1
          "_r": {"r": {"perms_ds_two": {"t1": ["vt"]}}},
      }
      cookies = {"ds_actor": perms_ds.client.actor_cookie(actor)}
      response = await perms_ds.client.get("/perms_ds_two/t1.json", cookies=cookies)
      assert response.status_code == 200
  ```
  The test fails. Running it with `pytest --pdb` let me do this:
  ```
  (Pdb) from pprint import pprint
  (Pdb) pprint(perms_ds._permission_checks)
  deque([{'action': 'view-table',
          'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                    'id': 'restricted-token',
                    'token': 'dstok'},
          'resource': ('perms_ds_two', 't1'),
          'result': None,
          'used_default': True,
          'when': '2023-08-23T21:59:45.117155'},
         {'action': 'view-database',
          'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                    'id': 'restricted-token',
                    'token': 'dstok'},
          'resource': 'perms_ds_two',
          'result': False,
          'used_default': False,
          'when': '2023-08-23T21:59:45.117189'},
         {'action': 'view-instance',
          'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                    'id': 'restricted-token',
                    'token': 'dstok'},
          'resource': None,
          'result': False,
          'used_default': False,
          'when': '2023-08-23T21:59:45.126751'},
         {'action': 'debug-menu',
          'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                    'id': 'restricted-token',
                    'token': 'dstok'},
          'resource': None,
          'result': False,
          'used_default': False,
          'when': '2023-08-23T21:59:45.126777'}],
        maxlen=200)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1690705243
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5kxh1b
created_at: 2023-08-23T22:03:54Z
updated_at: 2023-08-23T22:03:54Z
author_association: OWNER
body:
  Idea: `datasette-permissions-debug` plugin which simply prints out a stacktrace for every permission check so you can see where in the code they are.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2151#issuecomment-1690746371
issue_url: https://api.github.com/repos/simonw/datasette/issues/2151
node_id: IC_kwDOBm6k_c5kxr4D
created_at: 2023-08-23T22:51:57Z
updated_at: 2023-08-23T22:58:13Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`bdf59eb`)](https://app.codecov.io/gh/simonw/datasette/commit/bdf59eb7db42559e538a637bacfe86d39e5d17ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73% compared to head [(`b895cd2`)](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2151   +/-   ##
  =======================================
    Coverage   92.73%   92.73%           
  =======================================
    Files          40       40           
    Lines        5919     5919           
  =======================================
    Hits         5489     5489           
    Misses        430      430           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1690787394
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kx15C
created_at: 2023-08-23T23:52:02Z
updated_at: 2023-08-23T23:52:02Z
author_association: OWNER
body:
  > This also makes it simple to separate out secrets.
  > 
  > `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml`

  Having multiple configs that combine in that way is a really interesting direction.

  > To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate "settings" from "configuration" is, I believe, an uphill fight.

  I'm very keen on separating out the "metadata" - where metadata is the slimmest possible set of things, effectively the data license and the source and the column and table descriptions - from everything else, mainly because I want metadata to be able to travel with the data.

  One idea that's been discussed before is having an optional mechanism for storing metadata in the SQLite database file itself - potentially in a `_datasette_metadata` table. That way you could distribute a DB file and anyone who opened it in Datasette would also see the correct metadata about it.

  That's why I'm so keen on splitting out metadata from all of the other stuff - settings and plugin configuration and authentication rules.

  So really it becomes "true metadata" v.s. "all of the other junk that's accumulated in metadata and `settings.json`".

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kx3JC
created_at: 2023-08-24T00:00:16Z
updated_at: 2023-08-24T00:02:55Z
author_association: OWNER
body:
  I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax.

  Here's what I've come up with:
  ```
  datasette \
    -s settings.sql_time_limit_ms 1000 \
    -s plugins.datasette-auth-tokens.manage_tokens true \
    -s plugins.datasette-auth-tokens.manage_tokens_database tokens \
    -s plugins.datasette-ripgrep.path "/home/simon/code-to-search" \
    -s databases.mydatabase.tables.example_table.sort created \
    mydatabase.db tokens.db
  ```
  Which would be equivalent to `datasette.yml` containing this:
  ```yaml
  plugins:
    datasette-auth-tokens:
      manage_tokens: true
      manage_tokens_database: tokens
    datasette-ripgrep:
      path: /home/simon/code-to-search
  databases:
    mydatabase:
      tables:
        example_table:
          sort: created
  settings:
    sql_time_limit_ms: 1000
  ```
  Here's a prototype implementation of this:
  ```python
  import json
  from typing import Any, List, Tuple

  def _handle_pair(key: str, value: str) -> dict:
      """
      Turn a key-value pair into a nested dictionary.
      foo, bar => {'foo': 'bar'}
      foo.bar, baz => {'foo': {'bar': 'baz'}}
      foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}}
      foo.bar, "baz" => {'foo': {'bar': 'baz'}}
      foo.bar, '{"baz": "qux"}' => {'foo': {'bar': "{'baz': 'qux'}"}}
      """
      try:
          value = json.loads(value)
      except json.JSONDecodeError:
          # If it doesn't parse as JSON, treat it as a string
          pass

      keys = key.split('.')
      result = current_dict = {}
      
      for k in keys[:-1]:
          current_dict[k] = {}
          current_dict = current_dict[k]
      
      current_dict[keys[-1]] = value
      return result


  def _combine(base: dict, update: dict) -> dict:
      """
      Recursively merge two dictionaries.
      """
      for key, value in update.items():
          if isinstance(value, dict) and key in base and isinstance(base[key], dict):
              base[key] = _combine(base[key], value)
          else:
              base[key] = value
      return base

  def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict:
      """
      Parse a list of key-value pairs into a nested dictionary.
      """
      result = {}
      for key, value in pairs:
          parsed_pair = _handle_pair(key, value)
          result = _combine(result, parsed_pair)
      return result
  ```
  Exercised like this:
  ```python
  print(json.dumps(handle_pairs([
      ("settings.sql_time_limit_ms", "1000"),
      ("plugins.datasette-auth-tokens.manage_tokens", "true"),
      ("plugins.datasette-auth-tokens.manage_tokens_database", "tokens"),
      ("plugins.datasette-ripgrep.path", "/home/simon/code-to-search"),
      ("databases.mydatabase.tables.example_table.sort", "created"),
  ]), indent=4))
  ```
  Output:
  ```json
  {
      "settings": {
          "sql_time_limit_ms": 1000
      },
      "plugins": {
          "datasette-auth-tokens": {
              "manage_tokens": true,
              "manage_tokens_database": "tokens"
          },
          "datasette-ripgrep": {
              "path": "/home/simon/code-to-search"
          }
      },
      "databases": {
          "mydatabase": {
              "tables": {
                  "example_table": {
                      "sort": "created"
                  }
              }
          }
      }
  }
  ```
  Note that `-s` isn't currently an option for `datasette serve`.

  `--setting key value` IS an existing option, but it isn't completely compatible with this because it maps directly just to settings.

  Although... we could keep compatibility by saying that if you call `--setting known_setting value` and that `known_setting` is in this list then we treat it as if you said `-s settings.known_setting value` instead:

  https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1690799608
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kx434
created_at: 2023-08-24T00:09:47Z
updated_at: 2023-08-24T00:10:41Z
author_association: NONE
body:
  @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kx4_3
created_at: 2023-08-24T00:10:32Z
updated_at: 2023-08-24T00:39:00Z
author_association: OWNER
body:
  Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin):

  ```bash
  datasette data.db \
    -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}'
  ```
  Which would be equivalent to:
  ```yaml
  plugins:
    datasette-complex-plugin:
      configs:
        foo:
          - 1
          - 2
          - 3
        bar: baz
  ```
  This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex.

  That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again.

  For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kx5IB
created_at: 2023-08-24T00:11:16Z
updated_at: 2023-08-24T00:11:16Z
author_association: OWNER
body:
  > @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

  That's a neat example thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2147#issuecomment-1690955706
issue_url: https://api.github.com/repos/simonw/datasette/issues/2147
node_id: IC_kwDOBm6k_c5kye-6
created_at: 2023-08-24T03:54:35Z
updated_at: 2023-08-24T03:54:35Z
author_association: NONE
body:
  That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.

  But without a real motivating example, I see why you don't want to add that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1691036559
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5kyyuP
created_at: 2023-08-24T05:40:53Z
updated_at: 2023-08-24T05:40:53Z
author_association: OWNER
body:
  There might be an easier way to solve this. Here's some permission checks that run when hitting `/fixtures/facetable.json`:

  ```
  permission_allowed: action=view-table, resource=('fixtures', 'facetable'), actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

    File "/datasette/views/table.py", line 727, in table_view_traced
      view_data = await table_view_data(

    File "/datasette/views/table.py", line 875, in table_view_data
      visible, private = await datasette.check_visibility(

    File "/datasette/app.py", line 890, in check_visibility
      await self.ensure_permissions(actor, permissions)

  permission_allowed: action=view-database, resource=fixtures, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

    File "/datasette/views/table.py", line 727, in table_view_traced
      view_data = await table_view_data(

    File "/datasette/views/table.py", line 875, in table_view_data
      visible, private = await datasette.check_visibility(

    File "/datasette/app.py", line 890, in check_visibility
      await self.ensure_permissions(actor, permissions)

  permission_allowed: action=view-instance, resource=<None>, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

    File "/datasette/views/table.py", line 727, in table_view_traced
      view_data = await table_view_data(

    File "/datasette/views/table.py", line 875, in table_view_data
      visible, private = await datasette.check_visibility(

    File "/datasette/app.py", line 890, in check_visibility
      await self.ensure_permissions(actor, permissions)
  ```
  That's with a token that has the view instance, view database and view table permissions required.

  But... what if the restrictions logic said that if you have view-table you automatically also get view-database and view-instance?

  Would that actually let people do anything they shouldn't be able to do? I don't think it would even let them see a list of tables that they weren't allowed to visit, so it might be OK.

  I'll try that and see how it works.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5kyzET
created_at: 2023-08-24T05:42:47Z
updated_at: 2023-08-24T05:42:47Z
author_association: OWNER
body:
  I applied a fun trick to help test this out:
  ```diff
  diff --git a/datasette/cli.py b/datasette/cli.py
  index 58f89c1c..830f47ef 100644
  --- a/datasette/cli.py
  +++ b/datasette/cli.py
  @@ -445,6 +445,10 @@ def uninstall(packages, yes):
       "--token",
       help="API token to send with --get requests",
   )
  +@click.option(
  +    "--actor",
  +    help="Actor to use for --get requests",
  +)
   @click.option("--version-note", help="Additional note to show on /-/versions")
   @click.option("--help-settings", is_flag=True, help="Show available settings")
   @click.option("--pdb", is_flag=True, help="Launch debugger on any errors")
  @@ -499,6 +503,7 @@ def serve(
       root,
       get,
       token,
  +    actor,
       version_note,
       help_settings,
       pdb,
  @@ -611,7 +616,10 @@ def serve(
           headers = {}
           if token:
               headers["Authorization"] = "Bearer {}".format(token)
  -        response = client.get(get, headers=headers)
  +        cookies = {}
  +        if actor:
  +            cookies["ds_actor"] = client.actor_cookie(json.loads(actor))
  +        response = client.get(get, headers=headers, cookies=cookies)
           click.echo(response.text)
           exit_code = 0 if response.status == 200 else 1
           sys.exit(exit_code)
  ```
  This adds a `--actor` option to `datasette ... --get /path` which makes it easy to test an API endpoint using a fake actor with a set of `_r` restrictions.

  With that in place I can try this, with a token that has view-instance and view-database and view-table:
  ```bash
  datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
      "_r": {
          "a": [
              "vi"
          ],
          "d": {
              "fixtures": [
                  "vd"
              ]
          },
          "r": {
              "fixtures": {
                  "facetable": [
                      "vt"
                  ]
              }
          }
      },
      "a": "user"
  }'
  ```
  Or this, with a token that just has view-table but is missing the view-database and view-instance:
  ```bash
  datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
      "_r": {
          "r": {
              "fixtures": {
                  "facetable": [
                      "vt"
                  ]
              }
          }
      },
      "a": "user"
  }'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1691043475
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5ky0aT
created_at: 2023-08-24T05:50:04Z
updated_at: 2023-08-24T05:50:04Z
author_association: OWNER
body:
  On first test this seems to work!

  ```diff
  diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py
  index 63a66c3c..9303dac8 100644
  --- a/datasette/default_permissions.py
  +++ b/datasette/default_permissions.py
  @@ -187,6 +187,30 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource):
           return None
       _r = actor.get("_r")
   
  +    # Special case for view-instance: it's allowed if there are any view-database
  +    # or view-table permissions defined
  +    if action == "view-instance":
  +        database_rules = _r.get("d") or {}
  +        for rules in database_rules.values():
  +            if "vd" in rules or "view-database" in rules:
  +                return None
  +        # Now check resources
  +        resource_rules = _r.get("r") or {}
  +        for _database, resources in resource_rules.items():
  +            for rules in resources.values():
  +                if "vt" in rules or "view-table" in rules:
  +                    return None
  +
  +    # Special case for view-database: it's allowed if there are any view-table permissions
  +    # defined within that database
  +    if action == "view-database":
  +        database_name = resource
  +        resource_rules = _r.get("r") or {}
  +        resources_in_database = resource_rules.get(database_name) or {}
  +        for rules in resources_in_database.values():
  +            if "vt" in rules or "view-table" in rules:
  +                return None
  +
       # Does this action have an abbreviation?
       to_check = {action}
       permission = datasette.permissions.get(action)
  ```
  Needs a LOT of testing to make sure what it's doing is sensible though.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1691044283
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5ky0m7
created_at: 2023-08-24T05:51:02Z
updated_at: 2023-08-24T05:51:02Z
author_association: OWNER
body:
  Also need to confirm that permissions like `insert-row`, `delete-row`, `create-table` etc don't also need special cases to ensure they get through the `view-instance` etc checks, if those exist for those actions.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1691045051
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5ky0y7
created_at: 2023-08-24T05:51:59Z
updated_at: 2023-08-24T05:51:59Z
author_association: OWNER
body:
  With that fix in place, this works:
  ```bash
  datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
      "_r": {
          "r": {
              "fixtures": {
                  "facetable": [
                      "vt"
                  ]
              }
          }
      },
      "a": "user"
  }'
  ```
  But this fails, because it's for a table not explicitly listed:
  ```bash
  datasette fixtures.db --get '/fixtures/searchable.json' --actor '{
      "_r": {
          "r": {
              "fixtures": {
                  "facetable": [
                      "vt"
                  ]
              }
          }
      },
      "a": "user"
  }'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1691094870
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5kzA9W
created_at: 2023-08-24T06:43:40Z
updated_at: 2023-08-24T06:43:40Z
author_association: NONE
body:
  If I may, the "path-like" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.

  For instance:

  ```
  datasette -s plugins.datasette-complex-plugin.configs '{"foo": [1,2,3], "bar": "baz"}'
  ```

  could also be provided using:

  ```
  export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{"foo": [1,2,3], "bar": "baz"}'
  datasette
  ```

  (I do not like mixing `-` and `_` in env vars but I do not have a best easily reversible example at the moment)

  FYI, you could take some inspiration from another great open source data project, Metabase:
  https://www.metabase.com/docs/latest/configuring-metabase/config-file
  https://www.metabase.com/docs/latest/configuring-metabase/environment-variables

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2152#issuecomment-1691710474
issue_url: https://api.github.com/repos/simonw/datasette/issues/2152
node_id: IC_kwDOBm6k_c5k1XQK
created_at: 2023-08-24T13:45:21Z
updated_at: 2023-08-25T13:18:34Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`2e45686`)](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76%.

  > :exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2152   +/-   ##
  =======================================
    Coverage   92.76%   92.76%           
  =======================================
    Files          40       40           
    Lines        5943     5919   -24     
  =======================================
  - Hits         5513     5491   -22     
  + Misses        430      428    -2     
  ```


  [see 3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2152/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2153#issuecomment-1691753489
issue_url: https://api.github.com/repos/simonw/datasette/issues/2153
node_id: IC_kwDOBm6k_c5k1hwR
created_at: 2023-08-24T14:07:25Z
updated_at: 2023-08-24T14:09:16Z
author_association: OWNER
body:
  Building that `"_r"` array is the main reason this would be useful, but it's also fiddly to get right.

  `datasette create-token` has a design for that already: https://docs.datasette.io/en/1.0a4/authentication.html#datasette-create-token
  ```
  datasette create-token root \
      --secret mysecret \
      --all view-instance \
      --all view-table \
      --database docs view-query \
      --resource docs documents insert-row \
      --resource docs documents update-row
  ```
  Adding imitations of those options (excluding `--secret`, not needed here) to `datasette serve` would add a LOT of extra options, but it would also make it really convenient to attempt a request with a specific set of restrictions. Not sure if that would be worth the extra `--help` output or not.

  I feel like the names would have to have a common prefix though. Maybe something like this:

  ```bash
  datasette serve data.db --get `/data/mytable.json' \
    --actor-id root \
    --r-all view-instance \
    --r-database data view-query \
    --r-resource data documents update-row
  ```
  Other options could be the longer `--restrict-all/--restrict-database/--restrict-resource`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2153#issuecomment-1691761685
issue_url: https://api.github.com/repos/simonw/datasette/issues/2153
node_id: IC_kwDOBm6k_c5k1jwV
created_at: 2023-08-24T14:11:41Z
updated_at: 2023-08-24T14:11:41Z
author_association: OWNER
body:
  Another option: implement this as a plugin, providing a new command like `datasette get ...`

  Or implement `datasette client get ...` as core commands or a plugin - except that clashes with the `datasette client` command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2153#issuecomment-1691763427
issue_url: https://api.github.com/repos/simonw/datasette/issues/2153
node_id: IC_kwDOBm6k_c5k1kLj
created_at: 2023-08-24T14:12:43Z
updated_at: 2023-08-24T14:12:43Z
author_association: OWNER
body:
  Annoying that `datasette client ...` makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing `datasette.client` Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2153#issuecomment-1691779180
issue_url: https://api.github.com/repos/simonw/datasette/issues/2153
node_id: IC_kwDOBm6k_c5k1oBs
created_at: 2023-08-24T14:21:03Z
updated_at: 2023-08-24T14:21:03Z
author_association: OWNER
body:
  `datasette serve` currently only has a `--get` - for this to be really useful it needs to grow `--post` and maybe other verbs too.

  Which is a good argument for moving this functionality to `datasette client get ...` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1691788400
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5k1qRw
created_at: 2023-08-24T14:25:56Z
updated_at: 2023-08-24T14:25:56Z
author_association: OWNER
body:
  Can be tested with:
  ```bash
  pip install https://github.com/simonw/datasette/archive/6d57a8c23043e99b27f7a2afbe58f4d58815fd51.zip
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1691798722
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5k1szC
created_at: 2023-08-24T14:31:42Z
updated_at: 2023-08-29T16:15:12Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00%`** and project coverage change: **`+0.03%`** :tada:
  > Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`3e49fd3`)](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.85%.
  > Report is 3 commits behind head on main.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2154      +/-   ##
  ==========================================
  + Coverage   92.82%   92.85%   +0.03%     
  ==========================================
    Files          40       40              
    Lines        5948     6008      +60     
  ==========================================
  + Hits         5521     5579      +58     
  - Misses        427      429       +2     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <ø> (-0.85%)` | :arrow_down: |
  | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.37% <100.00%> (+0.07%)` | :arrow_up: |
  | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.48% <100.00%> (+0.57%)` | :arrow_up: |
  | [datasette/permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3Blcm1pc3Npb25zLnB5) | `100.00% <100.00%> (ø)` | |

  ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2154/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5k1zJJ
created_at: 2023-08-24T14:45:49Z
updated_at: 2023-08-24T14:45:49Z
author_association: OWNER
body:
  I tested this out against a Datasette Cloud instance. I created a restricted token and tested it like this:
  ```bash
  curl -H "Authorization: Bearer $TOKEN" \
    'https://$INSTANCE/-/actor.json' | jq
  ```
  ```json
  {
    "actor": {
      "id": "245",
      "token": "dsatok",
      "token_id": 2,
      "_r": {
        "r": {
          "data": {
            "all_stocks": [
              "vt"
            ]
          }
        }
      }
    }
  }
  ```
  It can access the `all_stocks` demo table:
  ```bash
  curl -H "Authorization: Bearer $TOKEN" \
    'https://$INSTANCE/data/all_stocks.json?_size=1' | jq
  ```
  ```json
  {
    "ok": true,
    "next": "1",
    "rows": [
      {
        "rowid": 1,
        "Date": "2013-01-02",
        "Open": 79.12,
        "High": 79.29,
        "Low": 77.38,
        "Close": 78.43,
        "Volume": 140124866,
        "Name": "AAPL"
      }
    ],
    "truncated": false
  }
  ```
  Accessing the database returns just information about that table, even though other tables exist:
  ```bash
  curl -H "Authorization: Bearer $TOKEN" \
    'https://$INSTANCE/data.json?_size=1'
  ```
  ```json
  {
    "database": "data",
    "private": true,
    "path": "/data",
    "size": 3796992,
    "tables": [
      {
        "name": "all_stocks",
        "columns": [
          "Date",
          "Open",
          "High",
          "Low",
          "Close",
          "Volume",
          "Name"
        ],
        "primary_keys": [],
        "count": 8813,
        "hidden": false,
        "fts_table": null,
        "foreign_keys": {
          "incoming": [],
          "outgoing": []
        },
        "private": true
      }
    ],
    "hidden_count": 0,
    "views": [],
    "queries": [],
    "allow_execute_sql": false,
    "table_columns": {}
  }
  ```
  And hitting the top-level `/.json` thing does the same - it reveals that table but not any of the other tables or databases:
  ```bash
  curl -H "Authorization: Bearer $TOKEN" \
    'https://$INSTANCE/.json?_size=1'
  ```
  ```json
  {
    "data": {
      "name": "data",
      "hash": null,
      "color": "8d777f",
      "path": "/data",
      "tables_and_views_truncated": [
        {
          "name": "all_stocks",
          "columns": [
            "Date",
            "Open",
            "High",
            "Low",
            "Close",
            "Volume",
            "Name"
          ],
          "primary_keys": [],
          "count": null,
          "hidden": false,
          "fts_table": null,
          "num_relationships_for_sorting": 0,
          "private": false
        }
      ],
      "tables_and_views_more": false,
      "tables_count": 1,
      "table_rows_sum": 0,
      "show_table_row_counts": false,
      "hidden_table_rows_sum": 0,
      "hidden_tables_count": 0,
      "views_count": 0,
      "private": false
    }
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1691842259
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5k13bT
created_at: 2023-08-24T14:55:54Z
updated_at: 2023-08-24T14:55:54Z
author_association: OWNER
body:
  So what's needed to finish this is:
  - Tests that demonstrate that nothing is revealed that shouldn't be by tokens restricted in this way
  - Similar tests for other permissions like `create-table` that check that they work (and don't also need `view-instance` etc).
  - Documentation

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1691845306
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5k14K6
created_at: 2023-08-24T14:57:39Z
updated_at: 2023-08-24T14:57:39Z
author_association: OWNER
body:
  Notes on manual testing so far - it looks like this might be working!
  - https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1692180683
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5k3KDL
created_at: 2023-08-24T18:05:17Z
updated_at: 2023-08-24T18:05:17Z
author_association: OWNER
body:
  That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.

  Need to figure out the right syntax for that. Something like this perhaps:

  ```bash
  DATASETTE_CONFIG_PLUGINS='{"datasette-ripgrep": ...}'
  ```
  Hard to know how to make this nestable though. I considered this:
  ```bash
  DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/'
  ```
  But that doesn't work, because how does the processing code know that it should split on `_` for most of the tokens but NOT split `DATASETTE_RIPGREP`, instead treating that as `datasette-ripgrep`?

  I checked and `-` is not a valid character in an environment variable, at least in zsh on macOS:
  ```
  % export FOO_BAR-BAZ=1
  export: not valid in this context: FOO_BAR-BAZ
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1692182910
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5k3Kl-
created_at: 2023-08-24T18:06:57Z
updated_at: 2023-08-24T18:08:17Z
author_association: OWNER
body:
  The other thing that could work is something like this:
  ```bash
  export AUTH_TOKENS_DB="tokens"
  datasette \
    -s settings.sql_time_limit_ms 1000 \
    -s plugins.datasette-auth-tokens.manage_tokens true \
    -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB
  ```
  So `-e` is an alternative version of `-s` which reads from the named environment variable instead of having the value provided directly as the second value in the pair.

  I quite like this, because it could replace the really ugly `$ENV` pattern we have in plugin configuration at the moment:  https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values
  ```yaml
  plugins:
    datasette-auth-github:
      client_secret:
        $env: GITHUB_CLIENT_SECRET
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2156#issuecomment-1692186522
issue_url: https://api.github.com/repos/simonw/datasette/issues/2156
node_id: IC_kwDOBm6k_c5k3Lea
created_at: 2023-08-24T18:10:04Z
updated_at: 2023-08-24T18:10:04Z
author_association: OWNER
body:
  I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2156#issuecomment-1692201647
issue_url: https://api.github.com/repos/simonw/datasette/issues/2156
node_id: IC_kwDOBm6k_c5k3PKv
created_at: 2023-08-24T18:21:53Z
updated_at: 2023-08-24T18:21:53Z
author_association: OWNER
body:
  Oops, that was meant to be a PR. It's just a utility function though so it's safe to land already. I'll do a PR for the actual integration of it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2143#issuecomment-1692210044
issue_url: https://api.github.com/repos/simonw/datasette/issues/2143
node_id: IC_kwDOBm6k_c5k3RN8
created_at: 2023-08-24T18:28:27Z
updated_at: 2023-08-24T18:28:27Z
author_association: OWNER
body:
  Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332

  https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360

  Looks to me like that second bit of code doesn't yet handle `datasette.yml`

  This code does though:

  https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335

  `parse_metadata()` is clearly a bad name for this function:

  https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L980-L990

  That ` @documented` decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content

  So we should rename it to something better like `parse_json_or_yaml()` but keep `parse_metadata` as an undocumented alias for that to avoid any unnecessary plugin breaks.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1241#issuecomment-1692322342
issue_url: https://api.github.com/repos/simonw/datasette/issues/1241
node_id: IC_kwDOBm6k_c5k3som
created_at: 2023-08-24T19:56:15Z
updated_at: 2023-08-24T20:09:52Z
author_association: NONE
body:
  Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.

  It's not uncommon to have things like this currently:

  ```https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101```

  I'm thinking a plugin like [https://datasette.io/plugins/datasette-query-files](https://datasette.io/plugins/datasette-query-files), but could be created and managed from the UI (with the right permissions).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2157#issuecomment-1692465334
issue_url: https://api.github.com/repos/simonw/datasette/issues/2157
node_id: IC_kwDOBm6k_c5k4Pi2
created_at: 2023-08-24T21:54:09Z
updated_at: 2023-08-24T21:54:09Z
author_association: OWNER
body:
  We discussed this in-person this morning and these notes reflect what we talked about perfectly.

  I've had so many bugs with plugins that I've written myself that have forgotten to special-case the `_internal` database when looping through `datasette.databases.keys()` - removing it from there entirely would help a lot.

  Just one tiny disagreement: for `datasette-comments` I think having it store things in `_internal` could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful.

  Show me all rows in `foia_requests` with at least one associated comment in `datasette_comments.comments` kind of tihng.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2160#issuecomment-1695754277
issue_url: https://api.github.com/repos/simonw/datasette/issues/2160
node_id: IC_kwDOBm6k_c5lEygl
created_at: 2023-08-28T14:00:17Z
updated_at: 2023-08-29T00:32:52Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`c81b024`)](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82%.

  > :exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2160   +/-   ##
  =======================================
    Coverage   92.82%   92.82%           
  =======================================
    Files          40       40           
    Lines        5948     5948           
  =======================================
    Hits         5521     5521           
    Misses        427      427           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2161#issuecomment-1696267473
issue_url: https://api.github.com/repos/simonw/datasette/issues/2161
node_id: IC_kwDOBm6k_c5lGvzR
created_at: 2023-08-28T19:38:28Z
updated_at: 2023-08-28T19:39:33Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`75.00%`** and project coverage change: **`+0.05%`** :tada:
  > Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`a5cbf80`)](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.81%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2161      +/-   ##
  ==========================================
  + Coverage   92.76%   92.81%   +0.05%     
  ==========================================
    Files          40       40              
    Lines        5943     5944       +1     
  ==========================================
  + Hits         5513     5517       +4     
  + Misses        430      427       -3     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.37% <75.00%> (+1.02%)` | :arrow_up: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2102#issuecomment-1696361304
issue_url: https://api.github.com/repos/simonw/datasette/issues/2102
node_id: IC_kwDOBm6k_c5lHGtY
created_at: 2023-08-28T20:23:47Z
updated_at: 2023-08-28T20:24:35Z
author_association: OWNER
body:
  Here's an existing relevant test:

  https://github.com/simonw/datasette/blob/2e2825869fc2655b5fcadc743f6f9dec7a49bc65/tests/test_permissions.py#L616-L666

  It's not quite right for this new set of tests though, since they need to be exercising actual endpoints (`/.json` etc) in order to check that this works correctly.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2160#issuecomment-1696586213
issue_url: https://api.github.com/repos/simonw/datasette/issues/2160
node_id: IC_kwDOBm6k_c5lH9nl
created_at: 2023-08-29T00:07:46Z
updated_at: 2023-08-29T00:07:46Z
author_association: OWNER
body:
  I figured out why this was failing in:
  - https://github.com/simonw/datasette/pull/2148

  It's because Sphinx dropped support for Python 3.8.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2160#issuecomment-1696586767
issue_url: https://api.github.com/repos/simonw/datasette/issues/2160
node_id: IC_kwDOBm6k_c5lH9wP
created_at: 2023-08-29T00:08:26Z
updated_at: 2023-08-29T00:08:35Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/issues/2148#issuecomment-1689177556

  > Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2148#issuecomment-1696591957
issue_url: https://api.github.com/repos/simonw/datasette/issues/2148
node_id: IC_kwDOBm6k_c5lH_BV
created_at: 2023-08-29T00:15:29Z
updated_at: 2023-08-29T00:15:29Z
author_association: CONTRIBUTOR
body:
  This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2160#issuecomment-1696592763
issue_url: https://api.github.com/repos/simonw/datasette/issues/2160
node_id: IC_kwDOBm6k_c5lH_N7
created_at: 2023-08-29T00:16:38Z
updated_at: 2023-08-29T00:16:38Z
author_association: OWNER
body:
  Since this bumps Sphinx I'm manually reviewing the ReadTheDocs preview a bit, looks good to me: https://datasette--2160.org.readthedocs.build/en/2160/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2162#issuecomment-1696594855
issue_url: https://api.github.com/repos/simonw/datasette/issues/2162
node_id: IC_kwDOBm6k_c5lH_un
created_at: 2023-08-29T00:19:29Z
updated_at: 2023-08-29T03:22:21Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning:
  > Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`73489ca`)](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.79%.
  > Report is 1 commits behind head on main.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2162      +/-   ##
  ==========================================
  - Coverage   92.82%   92.79%   -0.03%     
  ==========================================
    Files          40       40              
    Lines        5948     5953       +5     
  ==========================================
  + Hits         5521     5524       +3     
  - Misses        427      429       +2     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `96.87% <ø> (-0.04%)` | :arrow_down: |
  | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.32% <100.00%> (+0.03%)` | :arrow_up: |
  | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (-0.06%)` | :arrow_down: |
  | [datasette/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.72% <100.00%> (+0.04%)` | :arrow_up: |
  | [datasette/utils/internal\_db.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2ludGVybmFsX2RiLnB5) | `100.00% <100.00%> (ø)` | |
  | [datasette/views/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.45% <100.00%> (ø)` | |
  | [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <100.00%> (-0.85%)` | :arrow_down: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1696618784
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5lIFkg
created_at: 2023-08-29T00:58:34Z
updated_at: 2023-08-29T00:58:34Z
author_association: OWNER
body:
  Should this have `implies_can_view=True` too? Probably: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L20-L22

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1696642671
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5lILZv
created_at: 2023-08-29T01:35:37Z
updated_at: 2023-08-29T01:35:37Z
author_association: OWNER
body:
  Reminder that I also need to confirm that `insert-row` works if you have it at the instance level, the database level or the resource level in `_r`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1696644066
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5lILvi
created_at: 2023-08-29T01:37:47Z
updated_at: 2023-08-29T03:00:25Z
author_association: OWNER
body:
  Code for this might be cleaner with a `Restrictions()` class that takes a `"_r"` dictionary to the constructor and can then answer questions like `.any_resource_has_permission("view-table")` - where it can resolve aliases etc as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2162#issuecomment-1696707458
issue_url: https://api.github.com/repos/simonw/datasette/issues/2162
node_id: IC_kwDOBm6k_c5lIbOC
created_at: 2023-08-29T03:17:54Z
updated_at: 2023-08-29T03:17:54Z
author_association: OWNER
body:
  Documentation preview: https://datasette--2162.org.readthedocs.build/en/2162/internals.html#datasette-s-internal-database

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2162#issuecomment-1696709110
issue_url: https://api.github.com/repos/simonw/datasette/issues/2162
node_id: IC_kwDOBm6k_c5lIbn2
created_at: 2023-08-29T03:20:40Z
updated_at: 2023-08-29T03:22:47Z
author_association: OWNER
body:
  > However, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.

  I'm completely happy for the `core_*` tables (or `datasette_*` or some other name) to live in the persisted-to-disk `internal.db` database, even though they're effectively meant to be an in-memory cache.

  I don't think it causes any harm, and it could even be quite useful to have them visible on disk - other applications could read the `internal.db` database while Datasette itself is running, should they have some weird reason to want to do that!

  Having those tables stick around in `internal.db` after Datasette shuts down could be useful for other debugging activities as well.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1697612168
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5lL4GI
created_at: 2023-08-29T14:57:54Z
updated_at: 2023-08-29T14:57:54Z
author_association: OWNER
body:
  The code to refactor is this: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L181-L280

  I'm going to turn that into a more general `restrictions_allow_action` function.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2154#issuecomment-1697725150
issue_url: https://api.github.com/repos/simonw/datasette/issues/2154
node_id: IC_kwDOBm6k_c5lMTre
created_at: 2023-08-29T15:54:12Z
updated_at: 2023-08-29T15:54:12Z
author_association: OWNER
body:
  In that last commit I also upgraded `Permission` from a named tuple to a dataclass, and added a `implies_can_view=True` private (not documented) option to it.

  Last step is to refactor the code to use that new property.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2163#issuecomment-1697818917
issue_url: https://api.github.com/repos/simonw/datasette/issues/2163
node_id: IC_kwDOBm6k_c5lMqkl
created_at: 2023-08-29T16:56:20Z
updated_at: 2023-08-29T16:56:20Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/blob/50da908213a0fc405ecd7a40090dfea7a2e7395c/datasette/utils/internal_db.py#L8-L62

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2164#issuecomment-1699728102
issue_url: https://api.github.com/repos/simonw/datasette/issues/2164
node_id: IC_kwDOBm6k_c5lT8rm
created_at: 2023-08-30T19:35:32Z
updated_at: 2023-08-30T19:35:47Z
author_association: OWNER
body:
  Figured this out so far with the help of GPT-4: https://chat.openai.com/share/0e785865-621b-4fb3-ba05-7449e57c8496

  Now this works:
  ```bash
  DATASETTE_LOAD_PLUGINS=datasette-write-ui datasette plugins
  ```
  ```json
  [
      {
          "name": "datasette_write_ui",
          "static": true,
          "templates": true,
          "version": null,
          "hooks": [
              "extra_template_vars",
              "register_routes"
          ]
      }
  ]
  ```
  Or multiple plugins:
  ```bash
  DATASETTE_LOAD_PLUGINS=datasette-write-ui,datasette-pretty-json datasette plugins
  ```
  Outputs:
  ```json
  [
      {
          "name": "datasette_pretty_json",
          "static": false,
          "templates": false,
          "version": null,
          "hooks": [
              "render_cell"
          ]
      },
      {
          "name": "datasette_write_ui",
          "static": true,
          "templates": true,
          "version": null,
          "hooks": [
              "extra_template_vars",
              "register_routes"
          ]
      }
  ]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2165#issuecomment-1699802028
issue_url: https://api.github.com/repos/simonw/datasette/issues/2165
node_id: IC_kwDOBm6k_c5lUOus
created_at: 2023-08-30T20:35:02Z
updated_at: 2023-08-30T20:35:02Z
author_association: OWNER
body:
  Testing this is going to be a bit of a pain.

  I think I'll add a whole separate test block to CI which installs a couple of plugins and then exercises this feature using `datasette plugins`.

  I'll use `datasette-init` and `datasette-json-html` just because they are small and simple.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2165#issuecomment-1699809688
issue_url: https://api.github.com/repos/simonw/datasette/issues/2165
node_id: IC_kwDOBm6k_c5lUQmY
created_at: 2023-08-30T20:41:06Z
updated_at: 2023-08-30T20:41:06Z
author_association: OWNER
body:
  Slight weirdness: I noticed that the output from the `datasette plugins` command looks like this for plugins loaded with the new environment variable:
  ```json
      {
          "name": "datasette_pretty_json",
          "static": false,
          "templates": false,
          "version": null,
          "hooks": [
              "render_cell"
          ]
      },
  ```
  That should ideally be `datasette-pretty-json`, not `datasette_pretty_json`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2165#issuecomment-1699811810
issue_url: https://api.github.com/repos/simonw/datasette/issues/2165
node_id: IC_kwDOBm6k_c5lURHi
created_at: 2023-08-30T20:42:41Z
updated_at: 2023-08-30T20:42:41Z
author_association: OWNER
body:
  The `load_setuptools_entrypoints()` function in Pluggy [does this](https://github.com/pytest-dev/pluggy/blob/0b41c9766508a46ae666cf281684df3164b3e2a9/src/pluggy/_manager.py#L376):

  ```python
              for ep in dist.entry_points:
                  if (
                      ep.group != group
                      or (name is not None and ep.name != name)
                      # already registered
                      or self.get_plugin(ep.name)
                      or self.is_blocked(ep.name)
                  ):
                      continue
                  plugin = ep.load()
                  self.register(plugin, name=ep.name)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2165#issuecomment-1699812599
issue_url: https://api.github.com/repos/simonw/datasette/issues/2165
node_id: IC_kwDOBm6k_c5lURT3
created_at: 2023-08-30T20:43:19Z
updated_at: 2023-08-30T22:05:05Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`26.66%`** and project coverage change: **`-0.17%`** :warning:
  > Comparison is base [(`30b28c8`)](https://app.codecov.io/gh/simonw/datasette/commit/30b28c8367a9c6870386ea10a202705b40862457?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.84% compared to head [(`5eddf34`)](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.

  > :exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2165      +/-   ##
  ==========================================
  - Coverage   92.84%   92.68%   -0.17%     
  ==========================================
    Files          40       40              
    Lines        5998     6012      +14     
  ==========================================
  + Hits         5569     5572       +3     
  - Misses        429      440      +11     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/plugins.py](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `64.58% <26.66%> (-17.77%)` | :arrow_down: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2165#issuecomment-1699884314
issue_url: https://api.github.com/repos/simonw/datasette/issues/2165
node_id: IC_kwDOBm6k_c5lUi0a
created_at: 2023-08-30T21:38:29Z
updated_at: 2023-08-30T21:39:51Z
author_association: OWNER
body:
  Here's the reason for that name disparity:

  https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L54-L65

  Note how the `distinfo.project_name` name is used when available. That seems to work for regularly installed plugins but not for plugins loaded via `DATASETTE_LOAD_PLUGINS`.

  And that's looking things up in `plugin_to_distinfo` which is populated here:

  https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L37

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2165#issuecomment-1699910555
issue_url: https://api.github.com/repos/simonw/datasette/issues/2165
node_id: IC_kwDOBm6k_c5lUpOb
created_at: 2023-08-30T22:05:14Z
updated_at: 2023-08-30T22:05:14Z
author_association: OWNER
body:
  Documentation preview: https://github.com/simonw/datasette/blob/6321c9c055a640ed6ea98e231dc5813dcde1f773/docs/plugins.rst#controlling-which-plugins-are-loaded

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1092#issuecomment-1699926384
issue_url: https://api.github.com/repos/simonw/datasette/issues/1092
node_id: IC_kwDOBm6k_c5lUtFw
created_at: 2023-08-30T22:17:42Z
updated_at: 2023-08-30T22:17:42Z
author_association: OWNER
body:
  This is implemented now:
  - https://docs.datasette.io/en/stable/internals.html#await-ensure-permissions-actor-permissions

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967
issue_url: https://api.github.com/repos/simonw/datasette/issues/2157
node_id: IC_kwDOBm6k_c5lWGV_
created_at: 2023-08-31T02:45:56Z
updated_at: 2023-08-31T02:45:56Z
author_association: CONTRIBUTOR
body:
  @simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)

  My rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH="/path/to/internal.db"` to their bashrc/zshrc/whereever to not have to worry about `--internal`

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2166#issuecomment-1701045404
issue_url: https://api.github.com/repos/simonw/datasette/issues/2166
node_id: IC_kwDOBm6k_c5lY-Sc
created_at: 2023-08-31T13:31:15Z
updated_at: 2023-09-06T13:25:17Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`7c9df6e`)](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2166   +/-   ##
  =======================================
    Coverage   92.68%   92.68%           
  =======================================
    Files          40       40           
    Lines        6012     6012           
  =======================================
    Hits         5572     5572           
    Misses        440      440           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2168#issuecomment-1701823609
issue_url: https://api.github.com/repos/simonw/datasette/issues/2168
node_id: IC_kwDOBm6k_c5lb8R5
created_at: 2023-08-31T21:43:06Z
updated_at: 2023-08-31T21:44:13Z
author_association: OWNER
body:
  Not sure what to call this. Maybe `app_wrapper()`?

  Or perhaps it's simpler than that, something like this:

  ```python
  @hookspec
  def process_response(datasette, request, response):
      """Last chance to modify the response before it is returned to the client"""
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2168#issuecomment-1701826521
issue_url: https://api.github.com/repos/simonw/datasette/issues/2168
node_id: IC_kwDOBm6k_c5lb8_Z
created_at: 2023-08-31T21:46:13Z
updated_at: 2023-08-31T21:46:13Z
author_association: OWNER
body:
  This could even be a pair of hooks - `process_request()` and `process_response()`.

  Or could take a leaf from Django, which redesigned middleware to use this pattern instead:

  ```python
  def simple_middleware(get_response):
      # One-time configuration and initialization.
      def middleware(request):
          # Code to be executed for each request before
          # the view (and later middleware) are called.
          response = get_response(request)
          # Code to be executed for each request/response after
          # the view is called.
          return response
      return middleware
  ```
  Or even borrow an idea from `pytest` where fixtures can `yield` in the middle, like this:
  ```python
  @pytest.fixture
  def sending_user(mail_admin):
      user = mail_admin.create_user()
      yield user
      mail_admin.delete_user(user)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2168#issuecomment-1701828197
issue_url: https://api.github.com/repos/simonw/datasette/issues/2168
node_id: IC_kwDOBm6k_c5lb9Zl
created_at: 2023-08-31T21:48:00Z
updated_at: 2023-08-31T21:48:57Z
author_association: OWNER
body:
  A pattern like this could be interesting:
  ```python
  async def my_middleware(datasette, request, get_response):
      # Mess with request here if neccessary
      response = await get_response(request)
      # mess with response
      return response
  ```
  The Django pattern is more complicated but does have that mechanism for running one-time configuration prior to defining the `middleware()` function, which is neat.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2168#issuecomment-1701830241
issue_url: https://api.github.com/repos/simonw/datasette/issues/2168
node_id: IC_kwDOBm6k_c5lb95h
created_at: 2023-08-31T21:50:18Z
updated_at: 2023-08-31T21:50:18Z
author_association: OWNER
body:
  The hook could be called `register_middleware()` and could work like `register_routes()` and `register_commands()`:

  ```python
  @hookspec
  def register_middleware(datasette):
      """Register middleware: returns a list of async def middleware functions"""

  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2168#issuecomment-1701831013
issue_url: https://api.github.com/repos/simonw/datasette/issues/2168
node_id: IC_kwDOBm6k_c5lb-Fl
created_at: 2023-08-31T21:51:12Z
updated_at: 2023-08-31T21:52:15Z
author_association: OWNER
body:
  Need to make sure the design of this takes streaming responses into account. Those could be pretty tricky here.

  I nice thing about `asgi_wrapper()` is that it handles those already.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704383901
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/589
node_id: IC_kwDOCGYnMM5lltWd
created_at: 2023-09-03T19:34:05Z
updated_at: 2023-09-03T19:34:05Z
author_association: OWNER
body:
  For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager.

  I played with this idea a bit:

  ```python
  with db.register_functions(md5, md5_random):
      db.query(...)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384111
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/589
node_id: IC_kwDOCGYnMM5lltZv
created_at: 2023-09-03T19:35:03Z
updated_at: 2023-09-03T19:35:03Z
author_association: OWNER
body:
  Normally in Python/`sqlite3` you de-register a function by passing `None` to it.

  You can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else:

  https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384393
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/589
node_id: IC_kwDOCGYnMM5llteJ
created_at: 2023-09-03T19:36:34Z
updated_at: 2023-09-03T19:36:34Z
author_association: OWNER
body:
  Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b

  Still needs tests and documentation (and some more thought to make sure it's doing the right thing).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/590#issuecomment-1704387161
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/590
node_id: IC_kwDOCGYnMM5lluJZ
created_at: 2023-09-03T19:50:36Z
updated_at: 2023-09-03T19:50:36Z
author_association: OWNER
body:
  Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/328#issuecomment-1706701195
issue_url: https://api.github.com/repos/simonw/datasette/issues/328
node_id: IC_kwDOBm6k_c5lujGL
created_at: 2023-09-05T14:10:39Z
updated_at: 2023-09-05T14:10:39Z
author_association: NONE
body:
  Hey @simonw I hit the same issue as mentionned by @chmaynard on a fresh install, "/mnt/fixtures.db" doesn't seem to exist in the docker image

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2173#issuecomment-1707565495
issue_url: https://api.github.com/repos/simonw/datasette/issues/2173
node_id: IC_kwDOBm6k_c5lx2G3
created_at: 2023-09-06T02:33:56Z
updated_at: 2023-09-06T02:33:56Z
author_association: OWNER
body:
  Running tests to see if Pyodide works - that was the reason I switched to my `click-default-group-wheel` package originally.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2173#issuecomment-1707570044
issue_url: https://api.github.com/repos/simonw/datasette/issues/2173
node_id: IC_kwDOBm6k_c5lx3N8
created_at: 2023-09-06T02:39:59Z
updated_at: 2023-09-06T02:49:01Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`fd083e3`)](https://app.codecov.io/gh/simonw/datasette/commit/fd083e37ec53e7e625111168d324a572344a3b19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`a481ebd`)](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
  > Report is 1 commits behind head on main.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2173   +/-   ##
  =======================================
    Coverage   92.68%   92.68%           
  =======================================
    Files          40       40           
    Lines        6012     6012           
  =======================================
    Hits         5572     5572           
    Misses        440      440           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2173#issuecomment-1707570378
issue_url: https://api.github.com/repos/simonw/datasette/issues/2173
node_id: IC_kwDOBm6k_c5lx3TK
created_at: 2023-09-06T02:40:31Z
updated_at: 2023-09-06T02:40:31Z
author_association: OWNER
body:
  Surprising error:
  ```
    blacken-docs -l 60 docs/*.rst
    shell: /usr/bin/bash -e {0}
    env:
      pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64
      PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig
      Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
      Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
      Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
      LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib
  docs/writing_plugins.rst:365: code block parse error Cannot parse: 8:0: <line number missing in source>
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708693020
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/591
node_id: IC_kwDOCGYnMM5l2JYc
created_at: 2023-09-06T16:14:03Z
updated_at: 2023-11-04T00:54:25Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified and coverable lines are covered by tests :white_check_mark:
  > Comparison is base [(`347fdc8`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/347fdc865e91b8d3410f49a5c9d5b499fbb594c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74% compared to head [(`1f14df1`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #591   +/-   ##
  =======================================
    Coverage   95.74%   95.74%           
  =======================================
    Files           8        8           
    Lines        2842     2842           
  =======================================
    Hits         2721     2721           
    Misses        121      121           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708695907
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/591
node_id: IC_kwDOCGYnMM5l2KFj
created_at: 2023-09-06T16:15:59Z
updated_at: 2023-09-06T16:19:14Z
author_association: OWNER
body:
  The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant:
  - https://github.com/pypa/setuptools/issues/3661

  ```
  25h  Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
  ERROR: Exception:
  Traceback (most recent call last):
  ...
    File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py", line 697, in get_requires_for_build_wheel
      return super().get_requires_for_build_wheel(config_settings=cs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
      return self._call_hook('get_requires_for_build_wheel', {
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
      raise BackendUnavailable(data.get('traceback', ''))
  pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
      obj = import_module(mod_path)
            ^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
    File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 994, in exec_module
    File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
    File "/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
      import distutils.core
  ModuleNotFoundError: No module named 'distutils'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2174#issuecomment-1708699926
issue_url: https://api.github.com/repos/simonw/datasette/issues/2174
node_id: IC_kwDOBm6k_c5l2LEW
created_at: 2023-09-06T16:18:13Z
updated_at: 2023-09-08T00:46:13Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch and project coverage have no change.
  > Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`d75b519`)](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
  > Report is 7 commits behind head on main.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2174   +/-   ##
  =======================================
    Coverage   92.68%   92.68%           
  =======================================
    Files          40       40           
    Lines        6012     6017    +5     
  =======================================
  + Hits         5572     5577    +5     
    Misses        440      440           
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <ø> (ø)` | |

  ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2174/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   

  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2174#issuecomment-1708727204
issue_url: https://api.github.com/repos/simonw/datasette/issues/2174
node_id: IC_kwDOBm6k_c5l2Ruk
created_at: 2023-09-06T16:35:39Z
updated_at: 2023-09-07T23:45:42Z
author_association: OWNER
body:
  We can use this here: https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables
  ```python
  @click.option(..., envvar="DATASETTE_INTERNAL")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2177#issuecomment-1708777964
issue_url: https://api.github.com/repos/simonw/datasette/issues/2177
node_id: IC_kwDOBm6k_c5l2eHs
created_at: 2023-09-06T17:04:30Z
updated_at: 2023-09-06T17:04:30Z
author_association: OWNER
body:
  Here's the thinking: setting the`DATASETTE_INTERNAL` environment variable for your entire machine will cause any time you run `datasette x.db` to result in a process that shares the same `internal.db` database as other instances.

  You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:

  https://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62

  The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.

  This convinced us that actually we should move those `catalog_` tables OUT of `internal.db`. The `_internal` database will be reserved for plugins that want to use it for caching, storing progress, etc.

  I think we move them to an in-memory `_catalog` database which is excluded from `ds.databases` (like `_internal` is ) but can be accessed using `datasette.get_catalog_database()` - similar to `datasette.get_internal_database()`.

  So each instance of Datasette gets its own truly private `_catalog`, which is in-memory and so gets cleared at the end of each process.

  An interesting thing that came up about a shared `_internal` database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1479#issuecomment-1709373304
issue_url: https://api.github.com/repos/simonw/datasette/issues/1479
node_id: IC_kwDOBm6k_c5l4vd4
created_at: 2023-09-07T02:14:15Z
updated_at: 2023-09-07T02:14:15Z
author_association: NONE
body:
  I ran into the same issue on Windows using 

  `datasette publish cloudrun mydatabase.db --service=my-database`
  do do a [google cloud publish](https://docs.datasette.io/en/stable/publish.html).

  @Rik-de-Kort your fix worked perfectly! Thanks!

  I can always go back and delete the temp directories myself :)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1710380941
issue_url: https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8
node_id: IC_kwDODFE5qs5l8leN
created_at: 2023-09-07T15:39:59Z
updated_at: 2023-09-07T15:39:59Z
author_association: NONE
body:
  > @maxhawkins curious why you didn't use the stdlib `mailbox` to parse the `mbox` files?

  Mailbox parses the entire mbox into memory. Using the lower level library lets us stream the emails in one at a time to support larger archives. Both libraries are in the stdlib.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2178#issuecomment-1710565268
issue_url: https://api.github.com/repos/simonw/datasette/issues/2178
node_id: IC_kwDOBm6k_c5l9SeU
created_at: 2023-09-07T17:58:04Z
updated_at: 2023-09-07T17:59:06Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/views/table.py#L1132-L1149

  Which calls this undocumented method:

  https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/app.py#L938-L973

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2178#issuecomment-1710567329
issue_url: https://api.github.com/repos/simonw/datasette/issues/2178
node_id: IC_kwDOBm6k_c5l9S-h
created_at: 2023-09-07T17:59:59Z
updated_at: 2023-09-07T17:59:59Z
author_association: OWNER
body:
  Should I put the permission check in that undocumented `datasette.expand_foreign_keys()` method? I think so - it should accept `request.actor` as one of its arguments.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2178#issuecomment-1710871095
issue_url: https://api.github.com/repos/simonw/datasette/issues/2178
node_id: IC_kwDOBm6k_c5l-dI3
created_at: 2023-09-07T23:07:16Z
updated_at: 2023-09-07T23:07:16Z
author_association: OWNER
body:
  I ran this:

      datasette content.db -p 8043 -m fk-auth.yml --root

  Against this YAML:
  ```yaml
  databases:
    content:
      tables:
        users:
          allow:
            id: root
  ```
  And it worked as it should - here's a screenshot of an anonymous user and a root user viewing the same page:

  ![CleanShot 2023-09-07 at 16 05 34@2x](https://github.com/simonw/datasette/assets/9599/3e91da08-107c-421c-8a00-aa650b960c58)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2178#issuecomment-1710878391
issue_url: https://api.github.com/repos/simonw/datasette/issues/2178
node_id: IC_kwDOBm6k_c5l-e63
created_at: 2023-09-07T23:19:05Z
updated_at: 2023-09-07T23:19:05Z
author_association: OWNER
body:
  This fix didn't work on Datasette Cloud. I used `/-/permissions` to debug it and saw this:

  ![image](https://github.com/simonw/datasette/assets/9599/61d2bc5f-1f96-41ea-8658-91dfbcb6610c)

  Only checking `view-table` is not enough: for my instance on Datasette Cloud the view permission check that should have failed was for the database or instance.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2178#issuecomment-1710879239
issue_url: https://api.github.com/repos/simonw/datasette/issues/2178
node_id: IC_kwDOBm6k_c5l-fIH
created_at: 2023-09-07T23:20:32Z
updated_at: 2023-09-07T23:20:32Z
author_association: OWNER
body:
  To test that locally, use this YAML instead:
  ```yaml
  databases:
    content:
      allow:
        id: root
      tables:
        releases:
          allow: true
  ```
  And:
  ```yaml
  allow:
    id: root
  databases:
    content:
      tables:
        releases:
          allow: true

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/592
node_id: IC_kwDOCGYnMM5l-rv2
created_at: 2023-09-08T00:47:57Z
updated_at: 2023-09-08T00:47:57Z
author_association: OWNER
body:
  That's odd, I wrote a test for this just now and it passes already:
  ```python
  def test_transform_preserves_rowids(fresh_db):
      # Create a rowid table
      fresh_db["places"].insert_all(
          (
              {"name": "Paris", "country": "France"},
              {"name": "London", "country": "UK"},
              {"name": "New York", "country": "USA"},
          ),
      )
      assert fresh_db["places"].use_rowid
      previous_rows = list(
          tuple(row) for row in fresh_db.execute("select rowid, name from places")
      )
      # Transform it
      fresh_db["places"].transform(column_order=("country", "name"))
      # Should be the same
      next_rows = list(
          tuple(row) for row in fresh_db.execute("select rowid, name from places")
      )
      assert previous_rows == next_rows
  ```
  So maybe I'm wrong about the cause of that bug?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/592
node_id: IC_kwDOCGYnMM5l-sbU
created_at: 2023-09-08T00:52:42Z
updated_at: 2023-09-08T00:52:42Z
author_association: OWNER
body:
  I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key.

  The reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows.

  But I tried this test and it passed, too:
  ```python
  def test_transform_preserves_rowids(fresh_db):
      fresh_db["places"].insert_all(
          [
              {"id": "1", "name": "Paris", "country": "France"},
              {"id": "2", "name": "London", "country": "UK"},
              {"id": "3", "name": "New York", "country": "USA"},
          ],
          pk="id",
      )
      previous_rows = list(
          tuple(row) for row in fresh_db.execute("select rowid, id, name from places")
      )
      # Transform it
      fresh_db["places"].transform(column_order=("country", "name"))
      # Should be the same
      next_rows = list(
          tuple(row) for row in fresh_db.execute("select rowid, id, name from places")
      )
      assert previous_rows == next_rows
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/592
node_id: IC_kwDOCGYnMM5l-smw
created_at: 2023-09-08T00:54:03Z
updated_at: 2023-09-08T00:54:03Z
author_association: OWNER
body:
  Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3.

  If I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/592
node_id: IC_kwDOCGYnMM5l-szm
created_at: 2023-09-08T00:55:30Z
updated_at: 2023-09-08T00:55:30Z
author_association: OWNER
body:
  Yes! That recreated the bug:
  ```
  >       assert previous_rows == next_rows
  E       AssertionError: assert equals failed
  E         [                                                                [                                                               
  E           (1, '1', 'Paris'),                                               (1, '1', 'Paris'),                                            
  E           (3, '3', 'New York'),                                            (2, '3', 'New York'),                                         
  E           (4, '4', 'London'),                                              (3, '4', 'London'),                                           
  E         ]                                                       ...
  E         
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/593#issuecomment-1710939868
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/593
node_id: IC_kwDOCGYnMM5l-t7c
created_at: 2023-09-08T01:03:40Z
updated_at: 2023-09-09T00:44:52Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00%`** and no project coverage change.
  > Comparison is base [(`5d123f0`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/5d123f031fc4fadc98f508e0ef6b7b6671e86155?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b86374f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #593   +/-   ##
  =======================================
    Coverage   95.77%   95.77%           
  =======================================
    Files           8        8           
    Lines        2837     2840    +3     
  =======================================
  + Hits         2717     2720    +3     
    Misses        120      120           
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <100.00%> (+<0.01%)` | :arrow_up: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2180#issuecomment-1710947637
issue_url: https://api.github.com/repos/simonw/datasette/issues/2180
node_id: IC_kwDOBm6k_c5l-v01
created_at: 2023-09-08T01:17:28Z
updated_at: 2023-09-08T01:17:28Z
author_association: OWNER
body:
  I think this is both a plugin hook and a `await datasette.actors_from_ids(actor_ids)` internal API function that calls it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2180#issuecomment-1710969339
issue_url: https://api.github.com/repos/simonw/datasette/issues/2180
node_id: IC_kwDOBm6k_c5l-1H7
created_at: 2023-09-08T01:53:16Z
updated_at: 2023-09-08T01:53:16Z
author_association: OWNER
body:
  Documentation preview:
  - https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids
  - https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2181#issuecomment-1710972324
issue_url: https://api.github.com/repos/simonw/datasette/issues/2181
node_id: IC_kwDOBm6k_c5l-12k
created_at: 2023-09-08T01:58:44Z
updated_at: 2023-09-08T03:43:43Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00%`** and no project coverage change.
  > Comparison is base [(`c263704`)](https://app.codecov.io/gh/simonw/datasette/commit/c26370485a4fd4bf130da051be9163d92c57f24f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`e1c5a9d`)](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2181   +/-   ##
  =======================================
    Coverage   92.68%   92.69%           
  =======================================
    Files          40       40           
    Lines        6017     6025    +8     
  =======================================
  + Hits         5577     5585    +8     
    Misses        440      440           
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.43% <100.00%> (+0.03%)` | :arrow_up: |
  | [datasette/hookspecs.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   

  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2180#issuecomment-1711028355
issue_url: https://api.github.com/repos/simonw/datasette/issues/2180
node_id: IC_kwDOBm6k_c5l_DiD
created_at: 2023-09-08T03:36:02Z
updated_at: 2023-09-08T03:36:02Z
author_association: OWNER
body:
  I shipped the first version of 

  - https://github.com/datasette/datasette-remote-actors/issues/1

  When I land this plugin in a Datasette release I need to update that repo to depend on the new alpha.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2180#issuecomment-1711054624
issue_url: https://api.github.com/repos/simonw/datasette/issues/2180
node_id: IC_kwDOBm6k_c5l_J8g
created_at: 2023-09-08T04:23:23Z
updated_at: 2023-09-08T04:23:23Z
author_association: OWNER
body:
  I've implemented this hook once now in:
  - https://github.com/datasette/datasette-remote-actors

  And built and tested a debug tool for it in:
  - https://github.com/datasette/datasette-debug-actors-from-ids

  I'm now confident in the design of this plugin hook, I'm going to land it.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2180#issuecomment-1711057080
issue_url: https://api.github.com/repos/simonw/datasette/issues/2180
node_id: IC_kwDOBm6k_c5l_Ki4
created_at: 2023-09-08T04:28:03Z
updated_at: 2023-09-08T04:28:03Z
author_association: OWNER
body:
  Landed:

  - https://docs.datasette.io/en/latest/plugin_hooks.html#actors-from-ids-datasette-actor-ids
  - https://docs.datasette.io/en/latest/internals.html#await-actors-from-ids-actor-ids

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2180#issuecomment-1712895084
issue_url: https://api.github.com/repos/simonw/datasette/issues/2180
node_id: IC_kwDOBm6k_c5mGLRs
created_at: 2023-09-10T17:44:14Z
updated_at: 2023-09-10T17:44:14Z
author_association: OWNER
body:
  Used by `datasette-short-links` here: https://github.com/datasette/datasette-short-links/blob/468c3e25dbe06a8dcba8edda59bc16a18e126a51/datasette_short_links/__init__.py#L108-L115

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/592
node_id: IC_kwDOCGYnMM5mGLZc
created_at: 2023-09-10T17:46:41Z
updated_at: 2023-09-10T17:46:41Z
author_association: OWNER
body:
  In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`:

  https://www.sqlite.org/lang_vacuum.html#how_vacuum_works

  > The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid).

  So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2168#issuecomment-1712897194
issue_url: https://api.github.com/repos/simonw/datasette/issues/2168
node_id: IC_kwDOBm6k_c5mGLyq
created_at: 2023-09-10T17:54:07Z
updated_at: 2023-09-10T17:54:07Z
author_association: OWNER
body:
  This looks relevant:

  https://pluggy.readthedocs.io/en/stable/#wrappers

  > A *hookimpl* can be marked with the `"wrapper"` option, which indicates that the function will be called to *wrap* (or surround) all other normal *hookimpl* calls. A *hook wrapper* can thus execute some code ahead and after the execution of all corresponding non-wrappper *hookimpls*.

  This could be the perfect mechanism for implementing this hook, although I still need to figure out how it interacts with streaming.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2183#issuecomment-1714544153
issue_url: https://api.github.com/repos/simonw/datasette/issues/2183
node_id: IC_kwDOBm6k_c5mMd4Z
created_at: 2023-09-11T20:37:52Z
updated_at: 2023-09-13T20:58:51Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`95.00%`** and project coverage change: **`-0.04%`** :warning:
  > Comparison is base [(`a4c96d0`)](https://app.codecov.io/gh/simonw/datasette/commit/a4c96d01b27ce7cd06662a024da3547132a7c412?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`659dcbd`)](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.

  > :exclamation: Current head 659dcbd differs from pull request most recent head acca338. Consider uploading reports for the commit acca338 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2183      +/-   ##
  ==========================================
  - Coverage   92.69%   92.66%   -0.04%     
  ==========================================
    Files          40       40              
    Lines        6025     6039      +14     
  ==========================================
  + Hits         5585     5596      +11     
  - Misses        440      443       +3     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <95.00%> (-0.24%)` | :arrow_down: |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2183#issuecomment-1714699724
issue_url: https://api.github.com/repos/simonw/datasette/issues/2183
node_id: IC_kwDOBm6k_c5mND3M
created_at: 2023-09-11T23:01:36Z
updated_at: 2023-09-11T23:02:30Z
author_association: OWNER
body:
  On thinking about this further, I'm fine releasing it as another alpha provided it causes Datasette to error loudly with an explanatory message if you attempt to load `-m metadata.json` at a metadata file that includes `"plugins"` configuration.

  Something like this:
  ```bash
  datasette -m metadata.json
  ```
  Outputs:

  > Datasette no longer accepts plugin configuration in --metadata. Move your `"plugins"` configuration blocks to a separate file - we suggest calling that `datasette.yaml` - and start Datasette with `datasette -c datasette.yaml`.

  For added usability points, let's have it suggest `datasette.json` if they used `metadata.json` and `datasette.yaml` if they tried to use `metadata.yaml`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714919806
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/594
node_id: IC_kwDOCGYnMM5mN5l-
created_at: 2023-09-12T03:49:41Z
updated_at: 2023-09-12T03:49:41Z
author_association: OWNER
body:
  Digging in a bit more:
  ```pycon
  >>> pprint(list(db.query('PRAGMA foreign_key_list(courses)')))
  [{'from': 'campus_name',
    'id': 0,
    'match': 'NONE',
    'on_delete': 'NO ACTION',
    'on_update': 'NO ACTION',
    'seq': 0,
    'table': 'departments',
    'to': 'campus_name'},
   {'from': 'dept_code',
    'id': 0,
    'match': 'NONE',
    'on_delete': 'NO ACTION',
    'on_update': 'NO ACTION',
    'seq': 1,
    'table': 'departments',
    'to': 'dept_code'}]
  ```
  I think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`.

  Right now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714920708
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/594
node_id: IC_kwDOCGYnMM5mN50E
created_at: 2023-09-12T03:51:13Z
updated_at: 2023-09-12T03:51:13Z
author_association: OWNER
body:
  Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`:

  https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150

  I could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works:
  ```python
  for table, column, other_table, other_column in table.foreign_keys:
     # ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2183#issuecomment-1716801971
issue_url: https://api.github.com/repos/simonw/datasette/issues/2183
node_id: IC_kwDOBm6k_c5mVFGz
created_at: 2023-09-13T01:34:01Z
updated_at: 2023-09-13T01:34:01Z
author_association: CONTRIBUTOR
body:
  @simonw docs are finished, this is ready for review!

  One thing: I added "Configuration" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named "datasette.yaml Configuration" or something similar?

  Mostly because "Configuration" by itself can mean many things, but adding "datasette.yaml" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using "datasette.yaml" instead of "datasette.json", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported)

  Also, maybe this is a chance to consolidate the docs a bit? I think "Settings", "Configuration", "Metadata", and "Authentication and permissions" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2185#issuecomment-1719468727
issue_url: https://api.github.com/repos/simonw/datasette/issues/2185
node_id: IC_kwDOBm6k_c5mfQK3
created_at: 2023-09-14T13:36:07Z
updated_at: 2023-09-19T13:42:35Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage has no change and project coverage change: **`-0.04%`** :warning:
  > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fe5f881`)](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2185      +/-   ##
  ==========================================
  - Coverage   92.69%   92.66%   -0.04%     
  ==========================================
    Files          40       40              
    Lines        6039     6039              
  ==========================================
  - Hits         5598     5596       -2     
  - Misses        441      443       +2     
  ```


  [see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2185/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2186#issuecomment-1721740872
issue_url: https://api.github.com/repos/simonw/datasette/issues/2186
node_id: IC_kwDOBm6k_c5mn65I
created_at: 2023-09-15T19:26:51Z
updated_at: 2023-09-15T19:27:19Z
author_association: OWNER
body:
  Here's where it's called at the moment: https://github.com/simonw/datasette/blob/16f0b6d8222d06682a31b904d0a402c391ae1c1c/datasette/views/base.py#L297-L313

  And the docs: https://github.com/simonw/datasette/blob/1.0a6/docs/plugin_hooks.rst#register-output-renderer-datasette

  I'm tempted to add a `get_count` argument which, when awaited, returns the full count. Then plugins could do this:

  ```python
  async def render_notebook(datasette, request, get_count, rows):
      count = await get_count()
      # ...
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2186#issuecomment-1721742055
issue_url: https://api.github.com/repos/simonw/datasette/issues/2186
node_id: IC_kwDOBm6k_c5mn7Ln
created_at: 2023-09-15T19:27:59Z
updated_at: 2023-09-15T19:27:59Z
author_association: OWNER
body:
  This feels like it might be quite a nice pattern generally - providing optional arguments to plugins and views that can be `await get_x()` called to run an extra calculation.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1722257328
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5mp4-w
created_at: 2023-09-16T15:47:32Z
updated_at: 2023-09-16T15:47:32Z
author_association: OWNER
body:
  Frustrating that this warning doesn't show up in the Datasette test suite itself. It shows up in plugin test suites that run this test:
  ```python
  @pytest.mark.asyncio
  async def test_plugin_is_installed():
      datasette = Datasette(memory=True)
      response = await datasette.client.get("/-/plugins.json")
      assert response.status_code == 200
      installed_plugins = {p["name"] for p in response.json()}
      assert "datasette-chronicle" in installed_plugins
  ```
  If you run that test inside Datasette core `installed_plugins` is an empty set, which presumably is why the warning doesn't get triggered there.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1722258980
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5mp5Yk
created_at: 2023-09-16T15:56:45Z
updated_at: 2023-09-16T15:56:45Z
author_association: OWNER
body:
  Weird, I still can't get the warning to show even with this:
  ```python
  @pytest.mark.asyncio
  async def test_plugin_is_installed():
      datasette = Datasette(memory=True)

      class DummyPlugin:
          __name__ = "DummyPlugin"

          @hookimpl
          def actors_from_ids(self, datasette, actor_ids):
              return {}

      try:
          pm.register(DummyPlugin(), name="DummyPlugin")
          response = await datasette.client.get("/-/plugins.json")
          assert response.status_code == 200
          installed_plugins = {p["name"] for p in response.json()}
          assert "DummyPlugin" in installed_plugins

      finally:
          pm.unregister(name="ReturnNothingPlugin")
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1722265848
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5mp7D4
created_at: 2023-09-16T16:32:42Z
updated_at: 2023-09-16T16:32:42Z
author_association: OWNER
body:
  Here's the exception it uses:
  ```pycon
  >>> importlib.metadata.version("datasette")
  '1.0a6'
  >>> importlib.metadata.version("datasette2")
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py", line 996, in version
      return distribution(distribution_name).version
    File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py", line 969, in distribution
      return Distribution.from_name(distribution_name)
    File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py", line 548, in from_name
      raise PackageNotFoundError(name)
  importlib.metadata.PackageNotFoundError: No package metadata was found for datasette2
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1722266513
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5mp7OR
created_at: 2023-09-16T16:36:09Z
updated_at: 2023-09-16T16:36:09Z
author_association: OWNER
body:
  Now I need to switch out `pkg_resources` in `plugins.py`:

  https://github.com/simonw/datasette/blob/852f5014853943fa27f43ddaa2d442545b3259fb/datasette/plugins.py#L33-L74

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1722266942
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5mp7U-
created_at: 2023-09-16T16:38:27Z
updated_at: 2023-09-16T16:38:27Z
author_association: OWNER
body:
  The `importlib.metadata.entry_points()` function is pretty interesting:
  ```pycon
  >>> import importlib.metadata
  >>> from pprint import pprint
  >>> pprint(importlib.metadata.entry_points())
  {'babel.checkers': [EntryPoint(name='num_plurals', value='babel.messages.checkers:num_plurals', group='babel.checkers'),
                      EntryPoint(name='python_format', value='babel.messages.checkers:python_format', group='babel.checkers')],
   'babel.extractors': [EntryPoint(name='jinja2', value='jinja2.ext:babel_extract[i18n]', group='babel.extractors'),
                        EntryPoint(name='ignore', value='babel.messages.extract:extract_nothing', group='babel.extractors'),
                        EntryPoint(name='javascript', value='babel.messages.extract:extract_javascript', group='babel.extractors'),
                        EntryPoint(name='python', value='babel.messages.extract:extract_python', group='babel.extractors')],
   'console_scripts': [EntryPoint(name='datasette', value='datasette.cli:cli', group='console_scripts'),
                       EntryPoint(name='normalizer', value='charset_normalizer.cli.normalizer:cli_detect', group='console_scripts'),
                       EntryPoint(name='pypprint', value='pprintpp:console', group='console_scripts'),
                       EntryPoint(name='cog', value='cogapp:main', group='console_scripts'),
                       EntryPoint(name='icdiff', value='icdiff:start', group='console_scripts'),
                       EntryPoint(name='pycodestyle', value='pycodestyle:_main', group='console_scripts'),
                       EntryPoint(name='sphinx-autobuild', value='sphinx_autobuild.__main__:main', group='console_scripts'),
                       EntryPoint(name='sphinx-apidoc', value='sphinx.ext.apidoc:main', group='console_scripts'),
                       EntryPoint(name='sphinx-autogen', value='sphinx.ext.autosummary.generate:main', group='console_scripts'),
                       EntryPoint(name='sphinx-build', value='sphinx.cmd.build:main', group='console_scripts'),
                       EntryPoint(name='sphinx-quickstart', value='sphinx.cmd.quickstart:main', group='console_scripts'),
                       EntryPoint(name='sphinx-to-sqlite', value='sphinx_to_sqlite.cli:cli', group='console_scripts'),
                       EntryPoint(name='pybabel', value='babel.messages.frontend:main', group='console_scripts'),
                       EntryPoint(name='docutils', value='docutils.__main__:main', group='console_scripts'),
                       EntryPoint(name='isort', value='isort.main:main', group='console_scripts'),
                       EntryPoint(name='isort-identify-imports', value='isort.main:identify_imports_main', group='console_scripts'),
                       EntryPoint(name='hupper', value='hupper.cli:main', group='console_scripts'),
                       EntryPoint(name='sqlite-utils', value='sqlite_utils.cli:cli', group='console_scripts'),
                       EntryPoint(name='py.test', value='pytest:console_main', group='console_scripts'),
                       EntryPoint(name='pytest', value='pytest:console_main', group='console_scripts'),
                       EntryPoint(name='pyflakes', value='pyflakes.api:main', group='console_scripts'),
                       EntryPoint(name='livereload', value='livereload.cli:main', group='console_scripts'),
                       EntryPoint(name='uvicorn', value='uvicorn.main:main', group='console_scripts'),
                       EntryPoint(name='httpx', value='httpx:main', group='console_scripts'),
                       EntryPoint(name='flake8', value='flake8.main.cli:main', group='console_scripts'),
                       EntryPoint(name='blacken-docs', value='blacken_docs:main', group='console_scripts'),
                       EntryPoint(name='pip', value='pip._internal.cli.main:main', group='console_scripts'),
                       EntryPoint(name='pip3', value='pip._internal.cli.main:main', group='console_scripts'),
                       EntryPoint(name='pip3.10', value='pip._internal.cli.main:main', group='console_scripts'),
                       EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts'),
                       EntryPoint(name='pygmentize', value='pygments.cmdline:main', group='console_scripts'),
                       EntryPoint(name='black', value='black:patched_main', group='console_scripts'),
                       EntryPoint(name='blackd', value='blackd:patched_main [d]', group='console_scripts'),
                       EntryPoint(name='codespell', value='codespell_lib:_script_main', group='console_scripts'),
                       EntryPoint(name='tabulate', value='tabulate:_main', group='console_scripts')],
   'datasette': [EntryPoint(name='debug_permissions', value='datasette_debug_permissions', group='datasette'),
                 EntryPoint(name='codespaces', value='datasette_codespaces', group='datasette'),
                 EntryPoint(name='vega', value='datasette_vega', group='datasette'),
                 EntryPoint(name='x_forwarded_host', value='datasette_x_forwarded_host', group='datasette'),
                 EntryPoint(name='json_html', value='datasette_json_html', group='datasette'),
                 EntryPoint(name='datasette_write_ui', value='datasette_write_ui', group='datasette'),
                 EntryPoint(name='pretty_json', value='datasette_pretty_json', group='datasette'),
                 EntryPoint(name='graphql', value='datasette_graphql', group='datasette')],
   'distutils.commands': [EntryPoint(name='compile_catalog', value='babel.messages.frontend:compile_catalog', group='distutils.commands'),
                          EntryPoint(name='extract_messages', value='babel.messages.frontend:extract_messages', group='distutils.commands'),
                          EntryPoint(name='init_catalog', value='babel.messages.frontend:init_catalog', group='distutils.commands'),
                          EntryPoint(name='update_catalog', value='babel.messages.frontend:update_catalog', group='distutils.commands'),
                          EntryPoint(name='isort', value='isort.setuptools_commands:ISortCommand', group='distutils.commands'),
                          EntryPoint(name='alias', value='setuptools.command.alias:alias', group='distutils.commands'),
                          EntryPoint(name='bdist_egg', value='setuptools.command.bdist_egg:bdist_egg', group='distutils.commands'),
                          EntryPoint(name='bdist_rpm', value='setuptools.command.bdist_rpm:bdist_rpm', group='distutils.commands'),
                          EntryPoint(name='build', value='setuptools.command.build:build', group='distutils.commands'),
                          EntryPoint(name='build_clib', value='setuptools.command.build_clib:build_clib', group='distutils.commands'),
                          EntryPoint(name='build_ext', value='setuptools.command.build_ext:build_ext', group='distutils.commands'),
                          EntryPoint(name='build_py', value='setuptools.command.build_py:build_py', group='distutils.commands'),
                          EntryPoint(name='develop', value='setuptools.command.develop:develop', group='distutils.commands'),
                          EntryPoint(name='dist_info', value='setuptools.command.dist_info:dist_info', group='distutils.commands'),
                          EntryPoint(name='easy_install', value='setuptools.command.easy_install:easy_install', group='distutils.commands'),
                          EntryPoint(name='editable_wheel', value='setuptools.command.editable_wheel:editable_wheel', group='distutils.commands'),
                          EntryPoint(name='egg_info', value='setuptools.command.egg_info:egg_info', group='distutils.commands'),
                          EntryPoint(name='install', value='setuptools.command.install:install', group='distutils.commands'),
                          EntryPoint(name='install_egg_info', value='setuptools.command.install_egg_info:install_egg_info', group='distutils.commands'),
                          EntryPoint(name='install_lib', value='setuptools.command.install_lib:install_lib', group='distutils.commands'),
                          EntryPoint(name='install_scripts', value='setuptools.command.install_scripts:install_scripts', group='distutils.commands'),
                          EntryPoint(name='rotate', value='setuptools.command.rotate:rotate', group='distutils.commands'),
                          EntryPoint(name='saveopts', value='setuptools.command.saveopts:saveopts', group='distutils.commands'),
                          EntryPoint(name='sdist', value='setuptools.command.sdist:sdist', group='distutils.commands'),
                          EntryPoint(name='setopt', value='setuptools.command.setopt:setopt', group='distutils.commands'),
                          EntryPoint(name='test', value='setuptools.command.test:test', group='distutils.commands'),
                          EntryPoint(name='upload_docs', value='setuptools.command.upload_docs:upload_docs', group='distutils.commands'),
                          EntryPoint(name='bdist_wheel', value='wheel.bdist_wheel:bdist_wheel', group='distutils.commands')],
   'distutils.setup_keywords': [EntryPoint(name='message_extractors', value='babel.messages.frontend:check_message_extractors', group='distutils.setup_keywords'),
                                EntryPoint(name='cffi_modules', value='cffi.setuptools_ext:cffi_modules', group='distutils.setup_keywords'),
                                EntryPoint(name='dependency_links', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
                                EntryPoint(name='eager_resources', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
                                EntryPoint(name='entry_points', value='setuptools.dist:check_entry_points', group='distutils.setup_keywords'),
                                EntryPoint(name='exclude_package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
                                EntryPoint(name='extras_require', value='setuptools.dist:check_extras', group='distutils.setup_keywords'),
                                EntryPoint(name='include_package_data', value='setuptools.dist:assert_bool', group='distutils.setup_keywords'),
                                EntryPoint(name='install_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                                EntryPoint(name='namespace_packages', value='setuptools.dist:check_nsp', group='distutils.setup_keywords'),
                                EntryPoint(name='package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
                                EntryPoint(name='packages', value='setuptools.dist:check_packages', group='distutils.setup_keywords'),
                                EntryPoint(name='python_requires', value='setuptools.dist:check_specifier', group='distutils.setup_keywords'),
                                EntryPoint(name='setup_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                                EntryPoint(name='test_loader', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
                                EntryPoint(name='test_runner', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
                                EntryPoint(name='test_suite', value='setuptools.dist:check_test_suite', group='distutils.setup_keywords'),
                                EntryPoint(name='tests_require', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                                EntryPoint(name='use_2to3', value='setuptools.dist:invalid_unless_false', group='distutils.setup_keywords'),
                                EntryPoint(name='zip_safe', value='setuptools.dist:assert_bool', group='distutils.setup_keywords')],
   'egg_info.writers': [EntryPoint(name='PKG-INFO', value='setuptools.command.egg_info:write_pkg_info', group='egg_info.writers'),
                        EntryPoint(name='dependency_links.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                        EntryPoint(name='depends.txt', value='setuptools.command.egg_info:warn_depends_obsolete', group='egg_info.writers'),
                        EntryPoint(name='eager_resources.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                        EntryPoint(name='entry_points.txt', value='setuptools.command.egg_info:write_entries', group='egg_info.writers'),
                        EntryPoint(name='namespace_packages.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                        EntryPoint(name='requires.txt', value='setuptools.command.egg_info:write_requirements', group='egg_info.writers'),
                        EntryPoint(name='top_level.txt', value='setuptools.command.egg_info:write_toplevel_names', group='egg_info.writers')],
   'flake8.extension': [EntryPoint(name='C90', value='mccabe:McCabeChecker', group='flake8.extension'),
                        EntryPoint(name='E', value='flake8.plugins.pycodestyle:pycodestyle_logical', group='flake8.extension'),
                        EntryPoint(name='F', value='flake8.plugins.pyflakes:FlakesChecker', group='flake8.extension'),
                        EntryPoint(name='W', value='flake8.plugins.pycodestyle:pycodestyle_physical', group='flake8.extension')],
   'flake8.report': [EntryPoint(name='default', value='flake8.formatting.default:Default', group='flake8.report'),
                     EntryPoint(name='pylint', value='flake8.formatting.default:Pylint', group='flake8.report'),
                     EntryPoint(name='quiet-filename', value='flake8.formatting.default:FilenameOnly', group='flake8.report'),
                     EntryPoint(name='quiet-nothing', value='flake8.formatting.default:Nothing', group='flake8.report')],
   'pylama.linter': [EntryPoint(name='isort', value='isort.pylama_isort:Linter', group='pylama.linter')],
   'pytest11': [EntryPoint(name='icdiff', value='pytest_icdiff', group='pytest11'),
                EntryPoint(name='asyncio', value='pytest_asyncio.plugin', group='pytest11'),
                EntryPoint(name='xdist', value='xdist.plugin', group='pytest11'),
                EntryPoint(name='xdist.looponfail', value='xdist.looponfail', group='pytest11'),
                EntryPoint(name='timeout', value='pytest_timeout', group='pytest11'),
                EntryPoint(name='anyio', value='anyio.pytest_plugin', group='pytest11')],
   'setuptools.finalize_distribution_options': [EntryPoint(name='keywords', value='setuptools.dist:Distribution._finalize_setup_keywords', group='setuptools.finalize_distribution_options'),
                                                EntryPoint(name='parent_finalize', value='setuptools.dist:_Distribution.finalize_options', group='setuptools.finalize_distribution_options')],
   'sphinx.html_themes': [EntryPoint(name='alabaster', value='alabaster', group='sphinx.html_themes'),
                          EntryPoint(name='basic-ng', value='sphinx_basic_ng', group='sphinx.html_themes'),
                          EntryPoint(name='furo', value='furo', group='sphinx.html_themes')],
   'sqlite_utils': [EntryPoint(name='hello_world', value='sqlite_utils_hello_world', group='sqlite_utils')]}
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2188#issuecomment-1722662413
issue_url: https://api.github.com/repos/simonw/datasette/issues/2188
node_id: IC_kwDOBm6k_c5mrb4N
created_at: 2023-09-18T02:01:34Z
updated_at: 2023-09-18T02:01:34Z
author_association: OWNER
body:
  I'm not interested in these in Datasette core itself, but I think they have a ton of potential for plugins.

  I wonder what the best way to handle that would be?

  Right now it's possible to write a plugin that [adds extra routes](https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes-datasette), so someone could build a `/dbname/-/prql?query=xxx` endpoint.

  If this could return JSON, they could add JavaScript to the `/dbname` page that provided a UI for kicking off one of those queries.

  Something that could make that more ergonomic might be the plugin hook that allows plugins to add extra HTML to different core database pages - e.g. adding a "Query this database using PRQL" button or link or even a full form at the top of that database page. That's this issue here:

  - #1191

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1191#issuecomment-1722845490
issue_url: https://api.github.com/repos/simonw/datasette/issues/1191
node_id: IC_kwDOBm6k_c5msIky
created_at: 2023-09-18T06:55:52Z
updated_at: 2023-09-18T06:55:52Z
author_association: CONTRIBUTOR
body:
  One note here: this feature could be called "slots", similar to [Layout Slots](https://vitepress.dev/guide/extending-default-theme#layout-slots) in Vitepress.

  In Vitepress, you can add custom components/widget/gadgets into determined named "slots", like so:

  ```
  doc-top
  doc-bottom
  doc-footer-before
  doc-before
  doc-after
  ...
  ```

  Would be great to do in both Python and Javascript, with the upcoming JavaScript API #2052. In `datasette-write-ui`, all we do is add a few "Insert row" and "edit this row" buttons and that required completely capturing the `table.html` template, which isn't great for other plugins. But having "slots" like `table-footer-before` or `table-row-id` or something would be great to work with.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2188#issuecomment-1722848454
issue_url: https://api.github.com/repos/simonw/datasette/issues/2188
node_id: IC_kwDOBm6k_c5msJTG
created_at: 2023-09-18T06:58:53Z
updated_at: 2023-09-18T06:58:53Z
author_association: CONTRIBUTOR
body:
  Thinking about this more, here a list of things I imagine a "compile-to-sql" plugin would want to do:

  1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting)
  2. Add "Query using PRQL" buttons in various parts of Datasette's UI, like `/dbname` page
  3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages
  4. Have their own dedicated code editor page


  1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of "slots" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks. 

  So it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here. 

  If I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1722943484
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5msgf8
created_at: 2023-09-18T08:14:47Z
updated_at: 2023-09-18T08:14:47Z
author_association: NONE
body:
  This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2123#issuecomment-1723362847
issue_url: https://api.github.com/repos/simonw/datasette/issues/2123
node_id: IC_kwDOBm6k_c5muG4f
created_at: 2023-09-18T13:02:46Z
updated_at: 2023-09-18T13:02:46Z
author_association: NONE
body:
  Can confirm that this bug can be reproduced as follows:

  ```
  docker run datasetteproject/datasette datasette serve --reload 
  ```

  which produces the following output:

  > Starting monitor for PID 10.
  > Error: Invalid value for '[FILES]...': Path 'serve' does not exist.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724045748
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mwtm0
created_at: 2023-09-18T17:24:07Z
updated_at: 2023-09-18T17:24:07Z
author_association: OWNER
body:
  I need reliable steps to reproduce, then I can bisect and figure out which exact version of Datasette introduced the problem.

  I have a hunch that it relates to changes made to the `datasette/database.py` module, maybe one of these changes here: https://github.com/simonw/datasette/compare/0.61...0.63.1#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724048314
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mwuO6
created_at: 2023-09-18T17:25:55Z
updated_at: 2023-09-18T17:25:55Z
author_association: OWNER
body:
  The good news is that this bug is currently unlikely to affect most users since named in-memory databases (created using `datasette.add_memory_database("airtable_refs")` ([docs](https://docs.datasette.io/en/stable/internals.html#add-memory-database-name)) are a pretty obscure feature, only available to plugins.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724049538
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mwuiC
created_at: 2023-09-18T17:26:44Z
updated_at: 2023-09-18T17:26:44Z
author_association: OWNER
body:
  Just managed to get this exception trace:
  ```
      return await self.route_path(scope, receive, send, path)
    File "/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/app.py", line 1354, in route_path
      response = await view(request, send)
    File "/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py", line 134, in view
      return await self.dispatch_request(request)
    File "/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py", line 91, in dispatch_request
      return await handler(request)
    File "/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py", line 361, in get
      response_or_template_contexts = await self.data(request, **data_kwargs)
    File "/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py", line 158, in data
      return await self._data_traced(request, default_labels, _next, _size)
    File "/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py", line 568, in _data_traced
      await gather(execute_facets(), execute_suggested_facets())
    File "/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py", line 177, in _gather_parallel
      return await asyncio.gather(*args)
  asyncio.exceptions.CancelledError
  INFO:     127.0.0.1:64109 - "GET /airtable_refs/airtable_refs?_facet=table_name&table_name=Sessions HTTP/1.1" 500 Internal Server Error
  ^CError in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/concurrent/futures/thread.py", line 40, in _python_exit
      t.join()
    File "/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py", line 1011, in join
      self._wait_for_tstate_lock()
    File "/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
      elif lock.acquire(block, timeout):
  KeyboardInterrupt
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724051886
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mwvGu
created_at: 2023-09-18T17:28:20Z
updated_at: 2023-09-18T17:30:30Z
author_association: OWNER
body:
  The bug exhibits when I try to add a facet. I think it's caused by the parallel query execution I added to facets at some point.

  http://127.0.0.1:8045/airtable_refs/airtable_refs - no error
  http://127.0.0.1:8045/airtable_refs/airtable_refs?_facet=table_name#facet-table_name - hangs the server

  Crucial line in the traceback:
  ```
      await gather(execute_facets(), execute_suggested_facets())
  ```
  From here: https://github.com/simonw/datasette/blob/917272c864ad7b8a00c48c77f5c2944093babb4e/datasette/views/table.py#L568

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724055823
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mwwEP
created_at: 2023-09-18T17:31:10Z
updated_at: 2023-09-18T17:31:10Z
author_association: OWNER
body:
  That line was added in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 which first shipped in 0.62a0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724064440
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mwyK4
created_at: 2023-09-18T17:36:00Z
updated_at: 2023-09-18T17:36:00Z
author_association: OWNER
body:
  I wrote this test, but it passes:
  ```python
  @pytest.mark.asyncio
  async def test_facet_against_in_memory_database():
      ds = Datasette()
      db = ds.add_memory_database("mem")
      await db.execute_write("create table t (id integer primary key, name text)")
      await db.execute_write_many(
          "insert into t (name) values (?)", [["one"], ["one"], ["two"]]
      )
      response1 = await ds.client.get("/mem/t.json")
      assert response1.status_code == 200
      response2 = await ds.client.get("/mem/t.json?_facet=name")
      assert response2.status_code == 200
      assert response2.json() == {
          "ok": True,
          "next": None,
          "facet_results": {
              "results": {
                  "name": {
                      "name": "name",
                      "type": "column",
                      "hideable": True,
                      "toggle_url": "/mem/t.json",
                      "results": [
                          {
                              "value": "one",
                              "label": "one",
                              "count": 2,
                              "toggle_url": "http://localhost/mem/t.json?_facet=name&name=one",
                              "selected": False,
                          },
                          {
                              "value": "two",
                              "label": "two",
                              "count": 1,
                              "toggle_url": "http://localhost/mem/t.json?_facet=name&name=two",
                              "selected": False,
                          },
                      ],
                      "truncated": False,
                  }
              },
              "timed_out": [],
          },
          "rows": [
              {"id": 1, "name": "one"},
              {"id": 2, "name": "one"},
              {"id": 3, "name": "two"},
          ],
          "truncated": False,
      }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724081909
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mw2b1
created_at: 2023-09-18T17:45:27Z
updated_at: 2023-09-18T17:45:27Z
author_association: OWNER
body:
  Maybe it's not related to faceting - I just got it on a hit to `http://127.0.0.1:8045/airtable_refs/airtable_refs` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724083324
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mw2x8
created_at: 2023-09-18T17:46:21Z
updated_at: 2023-09-18T17:46:21Z
author_association: OWNER
body:
  Sometimes it takes a few clicks for the bug to occur, but it does seem to always be within the in-memory database.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724084199
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mw2_n
created_at: 2023-09-18T17:47:01Z
updated_at: 2023-09-18T17:47:01Z
author_association: OWNER
body:
  I managed to trigger it by loading `http://127.0.0.1:8045/airtable_refs/airtable_refs` - which worked - and then hitting refresh on that page a bunch of times until it hung.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724089666
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mw4VC
created_at: 2023-09-18T17:49:24Z
updated_at: 2023-09-18T17:49:24Z
author_association: OWNER
body:
  I switched that particular implementation to using an on-disk database instead of an in-memory database and could no longer recreate the bug.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724157182
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxIz-
created_at: 2023-09-18T18:30:30Z
updated_at: 2023-09-18T18:30:30Z
author_association: OWNER
body:
  OK, I can trigger the bug like this:

  ```bash
  datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs
  ```
  Can I write a bash script that fails (and terminates the process) if it takes longer than X seconds?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724159882
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxJeK
created_at: 2023-09-18T18:32:29Z
updated_at: 2023-09-18T18:32:29Z
author_association: OWNER
body:
  This worked, including on macOS even though GPT-4 thought `timeout` would not work there: https://chat.openai.com/share/cc4628e9-5240-4f35-b640-16a9c178b315
  ```bash
  #!/bin/bash

  # Run the command with a timeout of 5 seconds
  timeout 5s datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs

  # Check the exit code from timeout
  if [ $? -eq 124 ]; then
      echo "Error: Command timed out after 5 seconds."
      exit 1
  fi
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2190#issuecomment-1724169693
issue_url: https://api.github.com/repos/simonw/datasette/issues/2190
node_id: IC_kwDOBm6k_c5mxL3d
created_at: 2023-09-18T18:39:19Z
updated_at: 2023-09-18T18:39:19Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning:
  > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fc7dbe0`)](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.67%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2190      +/-   ##
  ==========================================
  - Coverage   92.69%   92.67%   -0.03%     
  ==========================================
    Files          40       40              
    Lines        6039     6044       +5     
  ==========================================
  + Hits         5598     5601       +3     
  - Misses        441      443       +2     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <100.00%> (ø)` | |
  | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (ø)` | |
  | [datasette/utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.83% <100.00%> (+0.03%)` | :arrow_up: |

  ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2190/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724257290
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxhQK
created_at: 2023-09-18T19:39:27Z
updated_at: 2023-09-18T19:44:26Z
author_association: OWNER
body:
  I'm now trying this test script:
  ```bash
  #!/bin/bash

  port=8064
  # Start datasette server in the background and get its PID
  datasette pottery2.db -p $port &
  server_pid=$!

  # Wait for a moment to ensure the server has time to start up
  sleep 2

  # Initialize counters and parameters
  retry_count=0
  max_retries=3
  success_count=0
  path="/airtable_refs/airtable_refs"

  # Function to run curl with a timeout
  function test_curl {
      # Run the curl command with a timeout of 3 seconds
      timeout 3s curl -s "http://localhost:${port}${path}" > /dev/null
      if [ $? -eq 0 ]; then
          # Curl was successful
          ((success_count++))
      fi
  }

  # Try three parallel curl requests
  while [[ $retry_count -lt $max_retries ]]; do
      # Reset the success counter
      success_count=0

      # Run the curls in parallel
      echo "  Running curls"
      test_curl
      test_curl
      test_curl #  & test_curl & test_curl &

      # Wait for all curls to finish
      #wait

      # Check the success count
      if [[ $success_count -eq 3 ]]; then
          # All curls succeeded, break out of the loop
          echo "  All curl succeeded"
          break
      fi

      ((retry_count++))
  done

  # Kill the datasette server
  echo "Killing datasette server with PID $server_pid"
  kill -9 $server_pid
  sleep 2

  # Print result
  if [[ $success_count -eq 3 ]]; then
      echo "All three curls succeeded."
      exit 0
  else
      echo "Error: Not all curls succeeded after $retry_count attempts."
      exit 1
  fi
  ```
  I run it like this:
  ```bash
  git bisect reset
  git bisect start
  git bisect good 0.59.4
  git bisect bad 1.0a6
  git bisect run ../airtable-export/testit.sh
  ```
  But... it's not having the desired result, I think because the bug is intermittent so each time I run it the bisect spits out a different commit as the one that is to blame.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724258279
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxhfn
created_at: 2023-09-18T19:40:13Z
updated_at: 2023-09-18T19:40:13Z
author_association: OWNER
body:
  Output while it is running looks like this:
  ```
  running  '../airtable-export/testit.sh'
  INFO:     Started server process [75649]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
    Running curls
    Running curls
    Running curls
  Killing datasette server with PID 75649
  ../airtable-export/testit.sh: line 54: 75649 Killed: 9               datasette pottery2.db -p $port
  Error: Not all curls succeeded after 3 attempts.
  Bisecting: 155 revisions left to test after this (roughly 7 steps)
  [247e460e08bf823142f7b84058fe44e43626787f] Update beautifulsoup4 requirement (#1703)
  running  '../airtable-export/testit.sh'
  INFO:     Started server process [75722]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
    Running curls
    Running curls
    Running curls
  Killing datasette server with PID 75722
  ../airtable-export/testit.sh: line 54: 75722 Killed: 9               datasette pottery2.db -p $port
  Error: Not all curls succeeded after 3 attempts.
  Bisecting: 77 revisions left to test after this (roughly 6 steps)
  [3ef47a0896c7e63404a34e465b7160c80eaa571d] Link rel=alternate header for tables and rows
  running  '../airtable-export/testit.sh'
  INFO:     Started server process [75818]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
    Running curls
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724259229
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxhud
created_at: 2023-09-18T19:40:56Z
updated_at: 2023-09-18T19:40:56Z
author_association: OWNER
body:
  I tried it with a path of `/` and everything passed - so it's definitely the path of `/airtable_refs/airtable_refs` (an in-memory database created by an experimental branch of https://github.com/simonw/airtable-export) that triggers the problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724276917
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxmC1
created_at: 2023-09-18T19:54:23Z
updated_at: 2023-09-18T19:54:23Z
author_association: OWNER
body:
  Turned out I wasn't running the `datasette` from the current directory, so it was not testing what I intended.

  FIxed that with `pip install -e .` in the `datasette/` directory.

  Now I'm seeing some passes, which look like this:
  ```
  running  '../airtable-export/testit.sh'
  INFO:     Started server process [77810]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
    Running curls
  INFO:     127.0.0.1:59439 - "GET /airtable_refs/airtable_refs HTTP/1.1" 200 OK
  INFO:     127.0.0.1:59440 - "GET /airtable_refs/airtable_refs HTTP/1.1" 200 OK
  INFO:     127.0.0.1:59441 - "GET /airtable_refs/airtable_refs HTTP/1.1" 200 OK
    All curl succeeded
  Killing datasette server with PID 77810
  ../airtable-export/testit.sh: line 54: 77810 Killed: 9               datasette pottery2.db -p $port
  All three curls succeeded.
  Bisecting: 4 revisions left to test after this (roughly 2 steps)
  [7463b051cf8d7f856df5eba9f7aa944183ebabe5] Cosmetic tweaks after blacken-docs, refs #1718
  running  '../airtable-export/testit.sh'
  INFO:     Started server process [77826]
  INFO:     Waiting for application startup.
  INFO:     Application startup complete.
  INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
    Running curls
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724278386
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxmZy
created_at: 2023-09-18T19:55:32Z
updated_at: 2023-09-18T19:55:32Z
author_association: OWNER
body:
  OK it looks like it found it!

  ```
  942411ef946e9a34a2094944d3423cddad27efd3 is the first bad commit
  commit 

  Author: Simon Willison <swillison@gmail.com>
  Date:   Tue Apr 26 15:48:56 2022 -0700

      Execute some TableView queries in parallel
      
      Use ?_noparallel=1 to opt out (undocumented, useful for benchmark comparisons)
      
      Refs #1723, #1715

   datasette/views/table.py | 93 ++++++++++++++++++++++++++++++++++--------------
   1 file changed, 67 insertions(+), 26 deletions(-)
  bisect found first bad commit
  ```
  https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 does look like the cause of this problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724281824
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxnPg
created_at: 2023-09-18T19:58:06Z
updated_at: 2023-09-18T19:58:06Z
author_association: OWNER
body:
  I also confirmed that `http://127.0.0.1:8064/airtable_refs/airtable_refs?_noparallel=1` does not trigger the bug but `http://127.0.0.1:8064/airtable_refs/airtable_refs` does.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724298817
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxrZB
created_at: 2023-09-18T20:11:26Z
updated_at: 2023-09-18T20:11:26Z
author_association: OWNER
body:
  Now that I've confirmed that parallel query execution of the kind introduced in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 can cause hangs (presumably some kind of locking issue) against in-memory databases, some options:

  1. Disable parallel execution entirely and rip out related code.
  2. Disable parallel execution entirely by leaving that code but having it always behave as if `_noparallel=1`
  3. Continue digging and try and find some way to avoid this problem

  The parallel execution work is something I was playing with last year in the hope of speeding up Datasette pages like the table page which need to execute a bunch of queries - one for each facet, plus one for each column to see if it should be suggested as a facet.

  I wrote about this at the time here: https://simonwillison.net/2022/May/6/weeknotes/

  My hope was that despite Python's GIL this optimization would still help, because the SQLite C module releases the GIL once it gets to SQLite.

  But... that didn't hold up. It looked like enough work was happening in Python land with the GIL that the optimization didn't improve things.

  Running the `nogil` fork of Python DID improve things though! I left the code in partly on the hope that the `nogil` fork would be accepted into Python core.

  ... which it now has! But it will still be a year or two before it fully lands: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

  So I'm not particularly concerned about dropping the parallel execution. If I do drop it though do I leave the potentially complex code in that relates to it?

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724305169
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxs8R
created_at: 2023-09-18T20:16:22Z
updated_at: 2023-09-18T20:16:36Z
author_association: OWNER
body:
  Looking again at this code:

  https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/datasette/database.py#L87-L117

  `check_same_thread=False` really stands out here.

  Python docs at https://docs.python.org/3/library/sqlite3.html

  > **check_same_thread** ([*bool*](https://docs.python.org/3/library/functions.html#bool "bool")) -- If `True` (default), [`ProgrammingError`](https://docs.python.org/3/library/sqlite3.html#sqlite3.ProgrammingError "sqlite3.ProgrammingError") will be raised if the database connection is used by a thread other than the one that created it. If `False`, the connection may be accessed in multiple threads; write operations may need to be serialized by the user to avoid data corruption. See [`threadsafety`](https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety "sqlite3.threadsafety") for more information.

  I think I'm playing with fire by allowing multiple threads to access the same connection without doing my own serialization of those requests.

  I _do_ do that using the write connection - and in this particular case the bug isn't coming from write queries, it's coming from read queries - but perhaps SQLite has issues with threading for reads, too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724315591
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxvfH
created_at: 2023-09-18T20:24:30Z
updated_at: 2023-09-18T20:24:30Z
author_association: OWNER
body:
  [Using SQLite In Multi-Threaded Applications](https://www.sqlite.org/threadsafe.html)

  That indicates that there's a SQLite option for "Serialized" mode where it's safe to access anything SQLite provides from multiple threads, but as far as I can tell Python doesn't give you an option to turn that mode on or off for a connection - you can read `sqlite3.threadsafet`y to see if that mode was compiled in or not, but not actually change it.

  On my Mac `sqlite3.threadsafety` returns 1 which means https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety "Multi-thread: In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads." - it would need to return 3 for that serialized mode.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724317367
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxv63
created_at: 2023-09-18T20:25:44Z
updated_at: 2023-09-18T20:25:44Z
author_association: OWNER
body:
  My current hunch is that SQLite gets unhappy if multiple threads access the same underlying C object - which sometimes happens with in-memory connections and Datasette presumably because they are faster than file-backed databases.

  I'm going to remove the `asyncio.gather()` code from the table view. I'll ship a 0.x release with that fix too.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1724325068
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5mxxzM
created_at: 2023-09-18T20:29:41Z
updated_at: 2023-09-18T20:29:41Z
author_association: OWNER
body:
  The one other thing affected by this change is this documentation, which suggests a not-actually-safe pattern: https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/docs/internals.rst#L1292-L1321

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2191#issuecomment-1724480716
issue_url: https://api.github.com/repos/simonw/datasette/issues/2191
node_id: IC_kwDOBm6k_c5myXzM
created_at: 2023-09-18T21:28:36Z
updated_at: 2023-10-12T16:15:40Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified lines are covered by tests :white_check_mark:
  > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`0135e7c`)](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
  > Report is 14 commits behind head on main.

  > :exclamation: Current head 0135e7c differs from pull request most recent head 18b48f8. Consider uploading reports for the commit 18b48f8 to get more accurate results

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2191      +/-   ##
  ==========================================
  - Coverage   92.69%   92.68%   -0.02%     
  ==========================================
    Files          40       40              
    Lines        6039     6042       +3     
  ==========================================
  + Hits         5598     5600       +2     
  - Misses        441      442       +1     
  ```


  | [Files](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.09% <100.00%> (-0.11%)` | :arrow_down: |
  | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.36% <100.00%> (+0.01%)` | :arrow_up: |

  ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2191/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1726749355
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5m7Bqr
created_at: 2023-09-20T01:28:16Z
updated_at: 2023-09-20T01:28:16Z
author_association: OWNER
body:
  Added a note to that example in the documentation: https://github.com/simonw/datasette/blob/4e6a34179eaedec44c1263275d7592fd83d7e2ac/docs/internals.rst?plain=1#L1320

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2192#issuecomment-1726754119
issue_url: https://api.github.com/repos/simonw/datasette/issues/2192
node_id: IC_kwDOBm6k_c5m7C1H
created_at: 2023-09-20T01:35:45Z
updated_at: 2023-09-20T01:35:45Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  Patch coverage: **`100.00%`** and project coverage change: **`+0.02%`** :tada:
  > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`4e6a341`)](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.72%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main    #2192      +/-   ##
  ==========================================
  + Coverage   92.69%   92.72%   +0.02%     
  ==========================================
    Files          40       40              
    Lines        6039     6036       -3     
  ==========================================
  - Hits         5598     5597       -1     
  + Misses        441      439       -2     
  ```


  | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [datasette/views/table.py](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.31% <100.00%> (+0.51%)` | :arrow_up: |

  ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2192/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2191#issuecomment-1728498221
issue_url: https://api.github.com/repos/simonw/datasette/issues/2191
node_id: IC_kwDOBm6k_c5nBsot
created_at: 2023-09-20T22:17:26Z
updated_at: 2023-09-20T22:17:26Z
author_association: OWNER
body:
  I tested this locally for permissions like this. `datasette.yml`:

  ```yaml
  databases:
    content:
      allow:
        id: root
  ```
  Started Datasette like this:
  ```bash
  datasette --root content.db pottery2.db -c datasette.yml
  ```
  As root I could see this (note the padlock):

  <img width="630" alt="CleanShot 2023-09-20 at 15 17 02@2x" src="https://github.com/simonw/datasette/assets/9599/3e0c3abd-9a90-43f4-ad68-bb490d19eef8">

  http://127.0.0.1:8001/-/metadata returned `{}` showing that the permissions must have come from the config file instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2191#issuecomment-1728503623
issue_url: https://api.github.com/repos/simonw/datasette/issues/2191
node_id: IC_kwDOBm6k_c5nBt9H
created_at: 2023-09-20T22:23:33Z
updated_at: 2023-09-20T22:24:10Z
author_association: OWNER
body:
  This is one of the most interesting illustrative examples in the new code:

  https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L301-L349

  Interesting to note that it now has canned queries in it, which include this bit:

  https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L341-L342

  It looks like metadata, but in this case it's configuration. That blur between metadata and configuration at the canned query level still feels a little bit odd to me, but I still think we're going in the right direction with it.

  Also interesting, from that same file:

  https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L351-L399

  There are a few things in that metadata block that are arguably configuration, not metadata - for example:

  https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L360

  I think `extra_css_urls` is definitely configuration, not metadata.

  https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L369-L395

  Most of that stuff is arguably configuration too, with the exception of the `roadside_attractions.columns` bit which is metadata about those columns.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2191#issuecomment-1728504633
issue_url: https://api.github.com/repos/simonw/datasette/issues/2191
node_id: IC_kwDOBm6k_c5nBuM5
created_at: 2023-09-20T22:24:51Z
updated_at: 2023-09-20T22:25:16Z
author_association: OWNER
body:
  The `{"units": {"distance": "m", "frequency": "Hz"}}` bit is for the [units feature](https://docs.datasette.io/en/1.0a6/metadata.html#specifying-units-for-a-column) which I've half-disabled already and would like to remove before 1.0, though ideally turning that functionality into a plugin instead (if I can figure out how to do that).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730171241
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIFFp
created_at: 2023-09-21T19:27:25Z
updated_at: 2023-09-21T19:27:25Z
author_association: OWNER
body:
  This broke in Python 3.8:

  ```
  if plugin.__name__ not in DEFAULT_PLUGINS:
      try:
          if (importlib.resources.files(plugin.__name__) / "static").is_dir():
  E                   AttributeError: module 'importlib.resources' has no attribute 'files'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730183405
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIIDt
created_at: 2023-09-21T19:34:09Z
updated_at: 2023-09-21T19:34:09Z
author_association: OWNER
body:
  Confirmed: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

  > `importlib.resources.files(package)`
  > [...]
  > New in version 3.9.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730185322
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIIhq
created_at: 2023-09-21T19:35:49Z
updated_at: 2023-09-21T19:35:49Z
author_association: OWNER
body:
  I think I can fix this using https://importlib-resources.readthedocs.io/en/latest/using.html - maybe as a dependency only installed if the Python version is less than 3.9.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730188367
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIJRP
created_at: 2023-09-21T19:38:28Z
updated_at: 2023-09-21T19:40:38Z
author_association: OWNER
body:
  I'll imitate `certbot`:

  https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot-ci/setup.py#L9

  ```python
      'importlib_resources>=1.3.1; python_version < "3.9"',
  ```
  Looks like `1.3` is the minimum version needed for compatibility with the 3.9 standard library, according to https://github.com/python/importlib_resources/blob/main/README.rst#compatibility

  https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot/certbot/_internal/constants.py#L13C29-L16

  ```python
  if sys.version_info >= (3, 9):  # pragma: no cover
      import importlib.resources as importlib_resources
  else:  # pragma: no cover
      import importlib_resources
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730201226
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIMaK
created_at: 2023-09-21T19:49:20Z
updated_at: 2023-09-21T19:49:20Z
author_association: OWNER
body:
  That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the "Test DATASETTE_LOAD_PLUGINS" test shows errors but did not fail the CI run.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2193#issuecomment-1730202533
issue_url: https://api.github.com/repos/simonw/datasette/issues/2193
node_id: IC_kwDOBm6k_c5nIMul
created_at: 2023-09-21T19:50:22Z
updated_at: 2023-09-21T19:50:22Z
author_association: OWNER
body:
  Here's the failure in CI, which did not cause the workflow to fail even though it should have:

  <img width="733" alt="CleanShot 2023-09-21 at 12 50 01@2x" src="https://github.com/simonw/datasette/assets/9599/4c1d61be-b6f6-4699-a885-ed2d4edd85ab">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2193#issuecomment-1730203356
issue_url: https://api.github.com/repos/simonw/datasette/issues/2193
node_id: IC_kwDOBm6k_c5nIM7c
created_at: 2023-09-21T19:51:04Z
updated_at: 2023-09-21T19:51:04Z
author_association: OWNER
body:
  The script:

  https://github.com/simonw/datasette/blob/b0d0a0e5de8bb5b9b6c253e8af451a532266bcf1/tests/test-datasette-load-plugins.sh#L1-L29

  I'm not sure why those `exit 1` lines did not cause a CI failure.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2193#issuecomment-1730206629
issue_url: https://api.github.com/repos/simonw/datasette/issues/2193
node_id: IC_kwDOBm6k_c5nINul
created_at: 2023-09-21T19:53:39Z
updated_at: 2023-09-21T19:53:39Z
author_association: OWNER
body:
  [GPT-4 says](https://chat.openai.com/share/65a95561-8f3c-4f9b-a4ab-c2508e87d2a9):

  > In the script, you're using a subshell `( ... )` to group commands. If you `exit 1` within the subshell, it will only exit the subshell and not the main script. This is why GitHub Actions does not see it as a failure.

  It suggested doing this instead:

  ```bash
  #!/bin/bash

  PLUGINS=$(datasette plugins)
  if ! echo "$PLUGINS" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q true; then
    echo "Test failed: datasette-json-html not found"
    exit 1
  fi

  PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins)
  if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-json-html")' | grep -q false; then
    echo "Test failed: datasette-json-html should not have been loaded"
    exit 1
  fi

  if ! echo "$PLUGINS2" | jq 'any(.[]; .name == "datasette-init")' | grep -q true; then
    echo "Test failed: datasette-init should have been loaded"
    exit 1
  fi

  PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins)
  if ! echo "$PLUGINS3" | grep -q '\[\]'; then
    echo "Test failed: datasette plugins should have returned []"
    exit 1
  fi
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2193#issuecomment-1730208566
issue_url: https://api.github.com/repos/simonw/datasette/issues/2193
node_id: IC_kwDOBm6k_c5nIOM2
created_at: 2023-09-21T19:55:19Z
updated_at: 2023-09-21T19:55:19Z
author_association: OWNER
body:
  Yes, the new script seems to work. On Python 3.11:

  ```
  tests/test-datasette-load-plugins.sh 
  echo $?
  0
  ```
  On Python 3.8:
  ```
  tests/test-datasette-load-plugins.sh
  Test failed: datasette-json-html not found
  echo $?
  1
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730210728
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIOuo
created_at: 2023-09-21T19:57:08Z
updated_at: 2023-09-21T19:57:08Z
author_association: OWNER
body:
  In my Python 3.8 environment I ran:
  ```bash
  datasette install datasette-init datasette-json-html
  ```
  And now `datasette plugins` produces this error:
  ```
    File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 192, in plugins
      click.echo(json.dumps(app._plugins(all=all), indent=4))
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1136, in _plugins
      ps.sort(key=lambda p: p["name"])
  TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730211445
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIO51
created_at: 2023-09-21T19:57:44Z
updated_at: 2023-09-21T19:57:44Z
author_association: OWNER
body:
  In the debugger:
  ```
  >>> import pdb
  >>> pdb.pm()
  > /Users/simon/Dropbox/Development/datasette/datasette/app.py(1136)_plugins()
  -> ps.sort(key=lambda p: p["name"])
  (Pdb) ps
  [{'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['prepare_connection', 'render_cell'], 'version': '1.0.1'}, {'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['startup'], 'version': '0.2'}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730212597
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIPL1
created_at: 2023-09-21T19:58:38Z
updated_at: 2023-09-21T19:58:38Z
author_association: OWNER
body:
  Relevant code: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/app.py#L1127-L1146

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730214654
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIPr-
created_at: 2023-09-21T19:59:51Z
updated_at: 2023-09-21T19:59:51Z
author_association: OWNER
body:
  So the problem is the `get_plugins()` function returning plugins with `None` for their name:

  https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/plugins.py#L61-L91

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730219703
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIQ63
created_at: 2023-09-21T20:01:54Z
updated_at: 2023-09-21T20:01:54Z
author_association: OWNER
body:
  The problem is here:
  ```
   86  	        distinfo = plugin_to_distinfo.get(plugin)
   87  	        if distinfo is None:
   88  	            breakpoint()
   89  ->	            assert False
   90  	        if distinfo.name is None:
   91  	            breakpoint()
   92  	            assert False
   93  	        if distinfo:
   94  	            plugin_info["version"] = distinfo.version
  (Pdb) distinfo
  (Pdb) plugin
  <module 'datasette.sql_functions' from '/Users/simon/Dropbox/Development/datasette/datasette/sql_functions.py'>
  ```
  That `plugin_to_distinfo` is missing some stuff.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730226107
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nISe7
created_at: 2023-09-21T20:06:19Z
updated_at: 2023-09-21T20:06:19Z
author_association: OWNER
body:
  No that's not it actually, it's something else.

  Got to this point:
  ```bash
  DATASETTE_LOAD_PLUGINS=datasette-init python -i $(which datasette) plugins
  ```
  That fails and drops me into a debugger:

  ```
    File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 186, in plugins
      app = Datasette([], plugins_dir=plugins_dir)
    File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 405, in __init__
      for plugin in get_plugins()
    File "/Users/simon/Dropbox/Development/datasette/datasette/plugins.py", line 89, in get_plugins
      plugin_info["name"] = distinfo.name or distinfo.project_name
  AttributeError: 'PathDistribution' object has no attribute 'name'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1730232308
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5nIT_0
created_at: 2023-09-21T20:11:16Z
updated_at: 2023-09-21T20:11:16Z
author_association: OWNER
body:
  We're planning a breaking change in `1.0a7`:
  - #2191 

  Since that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as `1.0a8` instead.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730241813
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIWUV
created_at: 2023-09-21T20:18:40Z
updated_at: 2023-09-21T20:18:40Z
author_association: OWNER
body:
  This looks to be the step that is failing:

  https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L50-L60

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730242734
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIWiu
created_at: 2023-09-21T20:19:29Z
updated_at: 2023-09-21T20:19:29Z
author_association: OWNER
body:
  Maybe `plugins/` does not exist? It should have been created by this line:

  https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L41-L42

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730245204
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIXJU
created_at: 2023-09-21T20:21:42Z
updated_at: 2023-09-21T20:21:42Z
author_association: OWNER
body:
  I think I see the problem - it's from here: https://github.com/simonw/datasette/commit/b2ec8717c3619260a1b535eea20e618bf95aa30b#diff-5dbc88d6e5c3615caf10e32a9d6fc6ff683f5b5814948928cb84c3ab91c038b6L770

  The `config` and `metadata` Click options are the wrong way round:

  https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L785-L786

  https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L801

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730256435
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIZ4z
created_at: 2023-09-21T20:31:22Z
updated_at: 2023-09-21T20:31:31Z
author_association: OWNER
body:
  New error: "Error: Metadata should end with .json"

  https://github.com/simonw/datasette/actions/runs/6266720924/job/17018265851

  <img width="678" alt="CleanShot 2023-09-21 at 13 31 08@2x" src="https://github.com/simonw/datasette/assets/9599/720726be-fcfc-49a4-8316-ee49d96ac86b">

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730258302
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIaV-
created_at: 2023-09-21T20:32:53Z
updated_at: 2023-09-21T20:33:02Z
author_association: OWNER
body:
  Correct usage is now:
  ```bash
  python tests/fixtures.py fixtures.db fixtures-config.json fixtures-metadata.json \
    plugins --extra-db-filename extra_database.db
  ```
  ```
  Test tables written to fixtures.db
  - metadata written to fixtures-metadata.json
  - config written to fixtures-config.json
    Wrote plugin: plugins/register_output_renderer.py
    Wrote plugin: plugins/view_name.py
    Wrote plugin: plugins/my_plugin.py
    Wrote plugin: plugins/messages_output_renderer.py
    Wrote plugin: plugins/sleep_sql_function.py
    Wrote plugin: plugins/my_plugin_2.py
  Test tables written to extra_database.db
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730259871
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIauf
created_at: 2023-09-21T20:34:09Z
updated_at: 2023-09-21T20:34:09Z
author_association: OWNER
body:
  ... which raises the challenge that `datasette publish` doesn't yet know what to do with a config file!

  https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L114-L122

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730305920
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIl-A
created_at: 2023-09-21T21:09:21Z
updated_at: 2023-09-21T21:09:21Z
author_association: OWNER
body:
  I'm going to disable this bit of the deploy for the moment, which will break the demo linked to from https://simonwillison.net/2022/Dec/2/datasette-write-api/

  https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730312128
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nInfA
created_at: 2023-09-21T21:15:11Z
updated_at: 2023-09-21T21:15:11Z
author_association: OWNER
body:
  As soon as `datasette publish cloudrun` has this I can re-enable this bit of the demo deploy:

  https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

  Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/

  https://todomvc.datasette.io/

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730313565
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nIn1d
created_at: 2023-09-21T21:16:31Z
updated_at: 2023-09-21T21:16:31Z
author_association: OWNER
body:
  The `@add_common_publish_arguments_and_options` decorator described here is bad. If I update it to support a new `config` option all plugins that use it will break.

  https://github.com/simonw/datasette/blob/f130c7c0a88e50cea4121ea18d1f6db2431b6fab/docs/plugin_hooks.rst#L347-L355

  I want to deprecate it and switch to a different, better design to address the same problem.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2193#issuecomment-1730352111
issue_url: https://api.github.com/repos/simonw/datasette/issues/2193
node_id: IC_kwDOBm6k_c5nIxPv
created_at: 2023-09-21T21:55:41Z
updated_at: 2023-09-21T21:55:41Z
author_association: OWNER
body:
  https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time.

  ```
      plugin_info["name"] = distinfo.name or distinfo.project_name
  AttributeError: 'PathDistribution' object has no attribute 'name'
  Test failed: datasette-json-html should not have been loaded
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2057#issuecomment-1730353462
issue_url: https://api.github.com/repos/simonw/datasette/issues/2057
node_id: IC_kwDOBm6k_c5nIxk2
created_at: 2023-09-21T21:57:17Z
updated_at: 2023-09-21T21:57:17Z
author_association: OWNER
body:
  Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302
  ```
      plugin_info["name"] = distinfo.name or distinfo.project_name
  AttributeError: 'PathDistribution' object has no attribute 'name'
  Test failed: datasette-json-html should not have been loaded
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2194#issuecomment-1730362441
issue_url: https://api.github.com/repos/simonw/datasette/issues/2194
node_id: IC_kwDOBm6k_c5nIzxJ
created_at: 2023-09-21T22:08:19Z
updated_at: 2023-09-21T22:08:19Z
author_association: OWNER
body:
  That worked

  https://github.com/simonw/datasette/commit/e4f868801a6633400045f59584cfe650961c3fa6 is the latest commit right now and https://latest.datasette.io/-/versions shows that as the deployed version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2189#issuecomment-1730388418
issue_url: https://api.github.com/repos/simonw/datasette/issues/2189
node_id: IC_kwDOBm6k_c5nI6HC
created_at: 2023-09-21T22:26:19Z
updated_at: 2023-09-21T22:26:19Z
author_association: OWNER
body:
  1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730437237
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nJGB1
created_at: 2023-09-21T22:36:22Z
updated_at: 2023-09-21T22:36:22Z
author_association: OWNER
body:
  I think the actual design of this is pretty simple. Current help starts like this:

  ```
  Usage: datasette publish cloudrun [OPTIONS] [FILES]...

    Publish databases to Datasette running on Cloud Run

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata
                                    to publish
    --extra-options TEXT            Extra options to pass to datasette serve
  ```
  The `-s` and `-c` short options are not being used.

  So I think `-c/--config` can point to a JSON or YAML `datasette.yaml` file, and `-s/--setting key value` can mirror the new `-s/--setting` option in `datasette serve` itself (a shortcut for populating the config file directly from the CLI).

  Here's the relevant help section from `datasette serve`:
  ```
    -m, --metadata FILENAME         Path to JSON/YAML file containing
                                    license/source metadata
    -c, --config FILENAME           Path to JSON/YAML Datasette configuration
                                    file
    -s, --setting SETTING...        nested.key, value setting to use in
                                    Datasette configuration
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730437934
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nJGMu
created_at: 2023-09-21T22:37:22Z
updated_at: 2023-09-21T22:37:22Z
author_association: OWNER
body:
  Here's the full help for Cloud Run at the moment:
  ```bash
  datasette publish cloudrun --help
  ```
  ```
  Usage: datasette publish cloudrun [OPTIONS] [FILES]...

    Publish databases to Datasette running on Cloud Run

  Options:
    -m, --metadata FILENAME         Path to JSON/YAML file containing metadata
                                    to publish
    --extra-options TEXT            Extra options to pass to datasette serve
    --branch TEXT                   Install datasette from a GitHub branch e.g.
                                    main
    --template-dir DIRECTORY        Path to directory containing custom
                                    templates
    --plugins-dir DIRECTORY         Path to directory containing custom plugins
    --static MOUNT:DIRECTORY        Serve static files from this directory at
                                    /MOUNT/...
    --install TEXT                  Additional packages (e.g. plugins) to
                                    install
    --plugin-secret <TEXT TEXT TEXT>...
                                    Secrets to pass to plugins, e.g. --plugin-
                                    secret datasette-auth-github client_id xxx
    --version-note TEXT             Additional note to show on /-/versions
    --secret TEXT                   Secret used for signing secure values, such
                                    as signed cookies
    --title TEXT                    Title for metadata
    --license TEXT                  License label for metadata
    --license_url TEXT              License URL for metadata
    --source TEXT                   Source label for metadata
    --source_url TEXT               Source URL for metadata
    --about TEXT                    About label for metadata
    --about_url TEXT                About URL for metadata
    -n, --name TEXT                 Application name to use when building
    --service TEXT                  Cloud Run service to deploy (or over-write)
    --spatialite                    Enable SpatialLite extension
    --show-files                    Output the generated Dockerfile and
                                    metadata.json
    --memory TEXT                   Memory to allocate in Cloud Run, e.g. 1Gi
    --cpu [1|2|4]                   Number of vCPUs to allocate in Cloud Run
    --timeout INTEGER               Build timeout in seconds
    --apt-get-install TEXT          Additional packages to apt-get install
    --max-instances INTEGER         Maximum Cloud Run instances
    --min-instances INTEGER         Minimum Cloud Run instances
    --help                          Show this message and exit.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730438503
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nJGVn
created_at: 2023-09-21T22:38:10Z
updated_at: 2023-09-21T22:38:10Z
author_association: OWNER
body:
  I'd really like to remove `--extra-options`. I think the new design makes that completely obsolete?

  Maybe it doesn't. You still need `--extra-options` for the `--crossdb` option for example.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730441613
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nJHGN
created_at: 2023-09-21T22:42:12Z
updated_at: 2023-09-21T22:42:12Z
author_association: OWNER
body:
  https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here:

  ```
  --extra-options="--setting sql_time_limit_ms 5000"
  --extra-options="--config default_cache_ttl:3600 --config hash_urls:1"
  --extra-options "--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1"
  --extra-options="--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000"
  --extra-options="--setting sql_time_limit_ms 5000"
  --extra-options "--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000"
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730446937
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nJIZZ
created_at: 2023-09-21T22:46:42Z
updated_at: 2023-09-21T22:46:52Z
author_association: OWNER
body:
  Found more when I [searched for YAML](https://github.com/search?q=datasette+publish+extra-options++language%3AYAML&type=code).

  Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59

  `--extra-options="--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1"`

  Uses both `--cors` and `--crossdb`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2195#issuecomment-1730458954
issue_url: https://api.github.com/repos/simonw/datasette/issues/2195
node_id: IC_kwDOBm6k_c5nJLVK
created_at: 2023-09-21T22:57:39Z
updated_at: 2023-09-21T22:57:48Z
author_association: OWNER
body:
  Worth noting that it already sets `--cors` automatically without you needing to specify it:

  https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L374-L374

  I wonder if that's actually surprising behaviour that we should change before 1.0.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1732018273
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/297
node_id: IC_kwDOCGYnMM5nPIBh
created_at: 2023-09-22T20:49:51Z
updated_at: 2023-09-22T20:49:51Z
author_association: NONE
body:
  This would be awesome to have for multi-gig tsv and csv files! I'm currently looking at a 10 hour countdown for one such important. Not a problem because I'm lazy and happy to let it run and check on it tomorrow..

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/595#issuecomment-1733312349
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/595
node_id: IC_kwDOCGYnMM5nUD9d
created_at: 2023-09-25T09:38:13Z
updated_at: 2023-09-25T09:38:57Z
author_association: NONE
body:
  Never mind

  When I created the connection using 
  `sqlite_utils.Database(path)`

  I just needed to add the following statement right after and it did the trick

  `self.db.conn.execute("PRAGMA foreign_keys = ON")`

  Hope this helps people in the future 👍 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2155#issuecomment-1737363182
issue_url: https://api.github.com/repos/simonw/datasette/issues/2155
node_id: IC_kwDOBm6k_c5njg7u
created_at: 2023-09-27T13:05:41Z
updated_at: 2023-09-27T13:05:41Z
author_association: CONTRIBUTOR
body:
  I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/1870#issuecomment-1745568725
issue_url: https://api.github.com/repos/simonw/datasette/issues/1870
node_id: IC_kwDOBm6k_c5oC0PV
created_at: 2023-10-03T19:12:37Z
updated_at: 2023-10-03T19:12:37Z
author_association: NONE
body:
  Hello! Resurrecting this issue since we're running into something similar with data.catalyst.coop as our database files have ballooned up to several GB. Our Cloud Run revisions now require huge amounts of RAM to start up without receiving a SIGBUS.

  I'd love to see this fix merged in. It sounds like we want to make the immutable/read-only mode decision more flexible before doing so, so that we can use `ro` in Docker and `immutable` outside. If that sounds right, I'm happy to take a crack at adding that as a command-line flag or something that gets set automatically based on the expected execution environment.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1747231893
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/433
node_id: IC_kwDOCGYnMM5oJKSV
created_at: 2023-10-04T16:15:09Z
updated_at: 2023-10-04T16:28:21Z
author_association: CONTRIBUTOR
body:
  I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:

  ```
    [#######-----------------------------]   20%
    [####################################]  100%%  
  ```
  The bug is reproducible for me in any terminal, including Gnome Terminal and Guake, and VSCode. With VSCode I can use this launch.json to reproduce it:

  ```json

  {
      "version": "0.2.0",
      "configurations": [
          {
              "name": "Python: Module",
              "type": "python",
              "request": "launch",
              "module": "sqlite_utils",
              "justMyCode": false,
              "args": ["insert", "test.db", "test", "--csv", "tests/sniff/example1.csv"]
          }
      ]
  }
  ```

  [edit - deleted my analysis of why the current code is wrong, which was confused and confusing]

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2197#issuecomment-1752093039
issue_url: https://api.github.com/repos/simonw/datasette/issues/2197
node_id: IC_kwDOBm6k_c5obtFv
created_at: 2023-10-08T15:59:53Z
updated_at: 2023-10-08T15:59:53Z
author_association: OWNER
body:
  Replicated this myself:
  ```bash
  cd /tmp
  mkdir dddd
  cd dddd
  pipenv shell
  pip install datasette
  pip freeze | grep click
  ```
  ```
  click==8.1.7
  click-default-group==1.2.4
  click-default-group-wheel==1.2.3
  ```
  Yeah this is bad, I'll ship a `0.64.5` release.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2197#issuecomment-1752095961
issue_url: https://api.github.com/repos/simonw/datasette/issues/2197
node_id: IC_kwDOBm6k_c5obtzZ
created_at: 2023-10-08T16:13:42Z
updated_at: 2023-10-08T16:14:39Z
author_association: OWNER
body:
  Confirmed - I ran this in a fresh virtual environment:
  ```bash
  pip install --no-cache datasette
  ```
  And now:
  ```bash
  pip freeze | grep click
  ```
  ```
  click==8.1.7
  click-default-group==1.2.4
  ```
  ```bash
  datasette --version
  ```
  ```
  datasette, version 0.64.5
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2197#issuecomment-1752096661
issue_url: https://api.github.com/repos/simonw/datasette/issues/2197
node_id: IC_kwDOBm6k_c5obt-V
created_at: 2023-10-08T16:17:04Z
updated_at: 2023-10-08T16:17:29Z
author_association: OWNER
body:
  https://lite.datasette.io/?install=datasette-packages#/-/packages confirms that Datasette Lite still works (`click-default-group-wheel` was originally built to allow it to run) and that it's now using these packages:
  ```
      "click-default-group": "1.2.4",
      "datasette": "0.64.5",
  ```
  Full list:
  ```json
  {
      "aiofiles": "23.2.1",
      "anyio": "3.7.1",
      "asgi-csrf": "0.9",
      "asgiref": "3.7.2",
      "certifi": "2022.12.7",
      "click": "8.1.3",
      "click-default-group": "1.2.4",
      "datasette": "0.64.5",
      "datasette-packages": "0.2",
      "h11": "0.12.0",
      "httpcore": "0.15.0",
      "httpx": "0.23.0",
      "hupper": "1.12",
      "idna": "3.4",
      "itsdangerous": "2.1.2",
      "janus": "1.0.0",
      "Jinja2": "3.1.2",
      "MarkupSafe": "2.1.2",
      "mergedeep": "1.3.4",
      "micropip": "0.3.0",
      "packaging": "23.0",
      "Pint": "0.22",
      "pip": "23.2.1",
      "pluggy": "1.0.0",
      "pyparsing": "3.0.9",
      "python-multipart": "0.0.6",
      "PyYAML": "6.0",
      "rfc3986": "1.5.0",
      "setuptools": "67.6.1",
      "sniffio": "1.3.0",
      "typing-extensions": "4.5.0",
      "uvicorn": "0.23.2"
  }
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2190#issuecomment-1759947021
issue_url: https://api.github.com/repos/simonw/datasette/issues/2190
node_id: IC_kwDOBm6k_c5o5qkN
created_at: 2023-10-12T16:19:38Z
updated_at: 2023-10-12T16:19:38Z
author_association: OWNER
body:
  This looks good and works well. The error from this currently looks like:

  ```
  datasette -m metadata.json -p 8844
  Traceback (most recent call last):
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette", line 33, in <module>
      sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')())
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
      return self.main(*args, **kwargs)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1055, in main
      rv = self.invoke(ctx)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py", line 760, in invoke
      return __callback(*args, **kwargs)
    File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 98, in wrapped
      return fn(*args, **kwargs)
    File "/Users/simon/Dropbox/Development/datasette/datasette/cli.py", line 546, in serve
      metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read()))
    File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 1282, in fail_if_plugins_in_metadata
      raise Exception(
  Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.
  ```
  With wrapping:

  `Exception: Datasette no longer accepts plugin configuration in --metadata. Move your "plugins" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.`

  I think we should link directly to documentation that tells people how to perform this upgrade.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2199#issuecomment-1759952247
issue_url: https://api.github.com/repos/simonw/datasette/issues/2199
node_id: IC_kwDOBm6k_c5o5r13
created_at: 2023-10-12T16:23:10Z
updated_at: 2023-10-12T16:23:10Z
author_association: OWNER
body:
  Some options for where this could go:
  - Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides.
  - On a new page? We could have a "upgrade instructions" page in the documentation.
  - At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page

  I'm leaning towards the third option at the moment.

  But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2199#issuecomment-1760396195
issue_url: https://api.github.com/repos/simonw/datasette/issues/2199
node_id: IC_kwDOBm6k_c5o7YOj
created_at: 2023-10-12T21:36:25Z
updated_at: 2023-10-12T21:36:25Z
author_association: OWNER
body:
  Related idea: how about a `datasette-upgrade` plugin which adds a `datasette upgrade` command that can be used to automate this process?

  Maybe something like this:
  ```bash
  datasette install datasette-upgrade
  datasette upgrade metadata-to-config metadata.json
  ```
  This would output two new files: `metadata.yaml` and `datasette.yaml`. If files with those names existed already in the current directory they would be called `metadata-new.yaml` and `datasette-new.yaml`.

  The command would tell you what it did:
  ```
  Your metadata.json file has been rewritten as two files:

      metadata-new.yaml
      datasette.yaml

  Start Datasette like this to try them out:

      datasette -m metadata-new.yaml -c datasette.yaml
  ```
  The command is `datasette upgrade metadata-to-config` because `metadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731
issue_url: https://api.github.com/repos/simonw/datasette/issues/2199
node_id: IC_kwDOBm6k_c5o7ZlD
created_at: 2023-10-12T21:41:42Z
updated_at: 2023-10-12T21:41:42Z
author_association: CONTRIBUTOR
body:
  I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins.

  One thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting 

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2199#issuecomment-1760411937
issue_url: https://api.github.com/repos/simonw/datasette/issues/2199
node_id: IC_kwDOBm6k_c5o7cEh
created_at: 2023-10-12T21:51:16Z
updated_at: 2023-10-12T21:51:16Z
author_association: OWNER
body:
  I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.

  If people want to keep their comments I'm happy to leave them to copy those over by hand.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2199#issuecomment-1760412424
issue_url: https://api.github.com/repos/simonw/datasette/issues/2199
node_id: IC_kwDOBm6k_c5o7cMI
created_at: 2023-10-12T21:51:44Z
updated_at: 2023-10-12T21:51:44Z
author_association: OWNER
body:
  Started playing with this plugin idea, now tearing myself away to work on something more important:
  ```python
  from datasette import hookimpl
  import click
  import pathlib


  @hookimpl
  def register_commands(cli):
      @cli.group()
      def upgrade():
          """
          Apply configuration upgrades to an existing Datasette instance
          """
          pass


      @upgrade.command()
      @click.argument(
          "metadata", type=click.Path(exists=True)
      )
      @click.option(
          "new_metadata", "-m", "--new-metadata", help="Path to new metadata.yaml file", type=click.Path(exists=False)
      )
      @click.option(
          "new_datasette", "-c", "--new-datasette", help="Path to new datasette.yaml file", type=click.Path(exists=False)
      )
      @click.option(
          "output_dir", "-e", "--output-dir", help="Directory to write new files to", type=click.Path(), default="."
      )
      def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
          """
          Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
          datasette.yaml split introduced prior to Datasette 1.0.
          """
          print("Upgrading {} to new metadata.yaml format".format(metadata))
          output_dir = pathlib.Path(output_dir)
          if not new_metadata:
              # Pick a filename for the new metadata.yaml file that does not yet exist
              new_metadata = pick_filename("metadata", output_dir)
          if not new_datasette:
              new_datasette = pick_filename("datasette", output_dir)
          print("New metadata.yaml file will be written to {}".format(new_metadata))
          print("New datasette.yaml file will be written to {}".format(new_datasette))


  def pick_filename(base, output_dir):
      options = ["{}.yaml".format(base), "{}-new.yaml".format(base)]
      i = 0
      while True:
          option = options.pop(0)
          option_path = output_dir / option
          if not option_path.exists():
              return option_path
          # If we ran out
          if not options:
              i += 1
              options = ["{}-new-{}.yaml".format(base, i)]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2199#issuecomment-1760413191
issue_url: https://api.github.com/repos/simonw/datasette/issues/2199
node_id: IC_kwDOBm6k_c5o7cYH
created_at: 2023-10-12T21:52:25Z
updated_at: 2023-10-12T21:52:25Z
author_association: OWNER
body:
  Demo of that logic:
  ```
  $ datasette upgrade metadata-to-config ../datasette/metadata.json
  Upgrading ../datasette/metadata.json to new metadata.yaml format
  New metadata.yaml file will be written to metadata-new-1.yaml
  New datasette.yaml file will be written to datasette.yaml
  $ touch metadata-new-1.yaml
  $ datasette upgrade metadata-to-config ../datasette/metadata.json
  Upgrading ../datasette/metadata.json to new metadata.yaml format
  New metadata.yaml file will be written to metadata-new-2.yaml
  New datasette.yaml file will be written to datasette.yaml
  $ touch datasette.yaml
  $ datasette upgrade metadata-to-config ../datasette/metadata.json
  Upgrading ../datasette/metadata.json to new metadata.yaml format
  New metadata.yaml file will be written to metadata-new-2.yaml
  New datasette.yaml file will be written to datasette-new.yaml
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2196#issuecomment-1760417555
issue_url: https://api.github.com/repos/simonw/datasette/issues/2196
node_id: IC_kwDOBm6k_c5o7dcT
created_at: 2023-10-12T21:54:52Z
updated_at: 2023-10-12T21:54:52Z
author_association: OWNER
body:
  I can't replicate this bug. Closing, but please re-open if it's still happening.

  As an aside, the link I promote is https://datasette.io/discord which redirects:

  ```
  curl -i 'https://datasette.io/discord'
  HTTP/2 301 
  location: https://discord.gg/ktd74dm5mw
  content-type: text/plain
  x-cloud-trace-context: 8dcfd08d3d1fa44f7ee78568e0f5305e;o=1
  date: Thu, 12 Oct 2023 21:54:17 GMT
  server: Google Frontend
  content-length: 0
  ```
  ```
  curl -i 'https://discord.gg/ktd74dm5mw'
  HTTP/2 301 
  date: Thu, 12 Oct 2023 21:54:28 GMT
  content-type: text/plain;charset=UTF-8
  content-length: 0
  location: https://discord.com/invite/ktd74dm5mw
  strict-transport-security: max-age=31536000; includeSubDomains; preload
  permissions-policy: interest-cohort=()
  x-content-type-options: nosniff
  x-frame-options: DENY
  x-robots-tag: noindex, nofollow, noarchive, nocache, noimageindex, noodp
  x-xss-protection: 1; mode=block
  report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Dzzrf%2FgGkfFxtzSAQ46slMVDLcFjsH9fsvVkzHtgUUiZ891rXAa6LvTRpHK%2BdSMSQ54F57hS9z1mZXXklIbONZW1bfBuFjSK9J4XmjjLjsFUulMXvpjfCLkB6PI%3D"}],"group":"cf-nel","max_age":604800}
  nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
  server: cloudflare
  cf-ray: 815294ddff282511-SJC
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1760542865
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5o78CR
created_at: 2023-10-12T23:44:53Z
updated_at: 2023-10-12T23:45:15Z
author_association: OWNER
body:
  Weird, the `cog` check is failing in CI.

  ```
  Run cog --check docs/*.rst
    cog --check docs/*.rst
    shell: /usr/bin/bash -e {0}
    env:
      pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64
      PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig
      Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
      Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
      Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
      LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib
  Check failed
  Checking docs/authentication.rst
  Checking docs/binary_data.rst
  Checking docs/changelog.rst
  Checking docs/cli-reference.rst
  Checking docs/configuration.rst  (changed)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2052#issuecomment-1760545012
issue_url: https://api.github.com/repos/simonw/datasette/issues/2052
node_id: IC_kwDOBm6k_c5o78j0
created_at: 2023-10-12T23:48:16Z
updated_at: 2023-10-12T23:48:16Z
author_association: OWNER
body:
  Oh! I think I broke Cog on `main` and these tests are running against this branch rebased against main.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1655#issuecomment-1761630595
issue_url: https://api.github.com/repos/simonw/datasette/issues/1655
node_id: IC_kwDOBm6k_c5pAFmD
created_at: 2023-10-13T14:37:48Z
updated_at: 2023-10-13T14:37:48Z
author_association: NONE
body:
  Hi @fgregg, I came across this issue and found your setup at labordata.bunkum.us can help me with a research project at https://database.zeropm.eu/. 
  I really like the approach [here](https://labordata.bunkum.us/f7-06c761c?sql=select+*+from+f7) when dealing with a custom SQL query returning more than 1000 rows:
  1) At the table in HTML page, only first 1000 rows displayed;
  2) When click the "Download this data as a CSV Spreadsheet(All Rows)" button, a csv with ALL ROWS (could be > 100 Mb) get downloaded.

  I am trying to repeat the setup but have yet to be successful so far. 
  What I tried:
  1) copy the query.html & table.html templates from this [github repo](https://github.com/labordata/warehouse/tree/main/templates) and use it my project
  2) use the same datasette version 1.0a2.

  Do you know what else I should try to set it correctly? I appreciate your help.

  @simonw I would like to use this opportunity to thank you for developing & maintaining such an amazing project. 
  I introduce your datasette to several projects in my institute. I am also interested in your cloud version.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1655#issuecomment-1766994810
issue_url: https://api.github.com/repos/simonw/datasette/issues/1655
node_id: IC_kwDOBm6k_c5pUjN6
created_at: 2023-10-17T19:01:59Z
updated_at: 2023-10-17T19:01:59Z
author_association: CONTRIBUTOR
body:
  hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1655#issuecomment-1767133832
issue_url: https://api.github.com/repos/simonw/datasette/issues/1655
node_id: IC_kwDOBm6k_c5pVFKI
created_at: 2023-10-17T20:37:18Z
updated_at: 2023-10-17T21:12:48Z
author_association: NONE
body:
  @fgregg Thanks for your reply. I tried to use your fork branch `datasette = {url = "https://github.com/fgregg/datasette/archive/refs/heads/no_limit_csv_publish.zip"}` and got error - TypeError: 'str' object is not callable. I used the same templates as in your branch [here ](https://github.com/labordata/warehouse/tree/main/templates).

  ```
  INFO:     127.0.0.1:47232 - "GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1" 200 OK
  Traceback (most recent call last):
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1632, in route_path
      response = await view(request, send)
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1814, in async_view_fn
      response = await async_call_with_supported_arguments(
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/utils/__init__.py", line 1016, in async_call_with_supported_arguments
      return await fn(*call_with)
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 673, in table_view
      response = await table_view_traced(datasette, request)
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py", line 822, in table_view_traced
      await datasette.render_template(
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py", line 1307, in render_template
      return await template.render_async(template_context)
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1324, in render_async
      return self.environment.handle_exception()
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
      raise rewrite_traceback_stack(source=source)
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py", line 1321, in <listcomp>
      [n async for n in self.root_render_func(ctx)]  # type: ignore
    File "templates/table.html", line 1, in top-level template code
      {% extends "base.html" %}
    File "/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/templates/base.html", line 62, in top-level template code
      {% block content %}
    File "templates/table.html", line 24, in block 'content'
      <div class="page-header" style="border-color: #{{ database_color(database) }}">
  TypeError: 'str' object is not callable
  INFO:     127.0.0.1:47228 - "GET /zeropm-v0-0-3/api_ready_query HTTP/1.1" 500 Internal Server Error
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2202#issuecomment-1777247375
issue_url: https://api.github.com/repos/simonw/datasette/issues/2202
node_id: IC_kwDOBm6k_c5p7qSP
created_at: 2023-10-24T13:49:27Z
updated_at: 2023-10-24T13:49:27Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified and coverable lines are covered by tests :white_check_mark:
  > Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`be4d0f0`)](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2202   +/-   ##
  =======================================
    Coverage   92.69%   92.69%           
  =======================================
    Files          40       40           
    Lines        6047     6047           
  =======================================
    Hits         5605     5605           
    Misses        442      442           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/411#issuecomment-1779267468
issue_url: https://api.github.com/repos/simonw/datasette/issues/411
node_id: IC_kwDOBm6k_c5qDXeM
created_at: 2023-10-25T13:23:04Z
updated_at: 2023-10-25T13:23:04Z
author_association: NONE
body:
  Using the [Counties example](https://us-counties.datasette.io/counties/county_for_latitude_longitude?longitude=-122&latitude=37), I was able to pull out the MakePoint method as 

  MakePoint(cast(rm_rnb_history_pres.rx_lng as float), cast(rm_rnb_history_pres.rx_lat as float)) as geometry

  which worked, giving me a geometry column.

  ![image](https://github.com/simonw/datasette/assets/363004/6393b712-9e3d-416d-ba37-202934d5f604)


  gave

  ![image](https://github.com/simonw/datasette/assets/363004/219db7b2-8107-41b3-a049-ef4d6bd7ac7a)

  I believe it's the cast to float that does the trick. Prior to using the cast, I also received a 'wrong number of arguments' eror.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/949#issuecomment-1791571572
issue_url: https://api.github.com/repos/simonw/datasette/issues/949
node_id: IC_kwDOBm6k_c5qyTZ0
created_at: 2023-11-02T21:36:24Z
updated_at: 2023-11-02T21:36:24Z
author_association: NONE
body:
  FWIW, code mirror 6 now has this standard although if you want table-specific suggestions, you'd have to handle parsing out which table the user is querying yourself.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793264654
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/600
node_id: IC_kwDOCGYnMM5q4wwO
created_at: 2023-11-04T00:22:07Z
updated_at: 2023-11-04T00:27:29Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified and coverable lines are covered by tests :white_check_mark:
  > Comparison is base [(`622c3a5`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/622c3a5a7dd53a09c029e2af40c2643fe7579340?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b1a6076`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main     #600   +/-   ##
  =======================================
    Coverage   95.77%   95.77%           
  =======================================
    Files           8        8           
    Lines        2840     2840           
  =======================================
    Hits         2720     2720           
    Misses        120      120           
  ```


  | [Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
  |---|---|---|
  | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <ø> (ø)` | |
  | [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <ø> (ø)` | |


  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]

performed_via_github_app:
  {"id": 254, "slug": "codecov", "node_id": "MDM6QXBwMjU0", "owner": {"login": "codecov", "id": 8226205, "node_id": "MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU=", "avatar_url": "https://avatars.githubusercontent.com/u/8226205?v=4", "gravatar_id": "", "url": "https://api.github.com/users/codecov", "html_url": "https://github.com/codecov", "followers_url": "https://api.github.com/users/codecov/followers", "following_url": "https://api.github.com/users/codecov/following{/other_user}", "gists_url": "https://api.github.com/users/codecov/gists{/gist_id}", "starred_url": "https://api.github.com/users/codecov/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/codecov/subscriptions", "organizations_url": "https://api.github.com/users/codecov/orgs", "repos_url": "https://api.github.com/users/codecov/repos", "events_url": "https://api.github.com/users/codecov/events{/privacy}", "received_events_url": "https://api.github.com/users/codecov/received_events", "type": "Organization", "site_admin": false}, "name": "Codecov", "description": "Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. ", "external_url": "https://codecov.io", "html_url": "https://github.com/apps/codecov", "created_at": "2016-09-25T14:18:27Z", "updated_at": "2023-09-08T15:29:16Z", "permissions": {"administration": "read", "checks": "write", "contents": "read", "emails": "read", "issues": "read", "members": "read", "metadata": "read", "pull_requests": "write", "statuses": "write"}, "events": ["check_run", "check_suite", "create", "delete", "fork", "member", "membership", "organization", "public", "pull_request", "push", "release", "repository", "status", "team_add"]}


html_url: https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793265952
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/600
node_id: IC_kwDOCGYnMM5q4xEg
created_at: 2023-11-04T00:25:34Z
updated_at: 2023-11-04T00:25:34Z
author_association: OWNER
body:
  The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/600
node_id: IC_kwDOCGYnMM5q4xme
created_at: 2023-11-04T00:31:34Z
updated_at: 2023-11-04T00:31:34Z
author_association: OWNER
body:
  Testing this manually on macOS using Docker Desk top like this:
  ```bash
  docker run -it --rm arm64v8/ubuntu /bin/bash
  ```
  Then inside the container:
  ```bash
  uname -m
  ```
  Outputs: `aarch64`

  Then:
  ```bash
  apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y
  cd /tmp
  git clone https://github.com/simonw/sqlite-utils
  cd sqlite-utils
  python3 -m venv venv
  source venv/bin/activate
  pip install -e '.[test]'
  sqlite-utils memory "select spatialite_version()" --load-extension=spatialite
  ```
  Which output:
  ```
  Traceback (most recent call last):
    File "/tmp/sqlite-utils/venv/bin/sqlite-utils", line 33, in <module>
      sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
    File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
      return self.main(*args, **kwargs)
    File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
      rv = self.invoke(ctx)
    File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
      return _process_result(sub_ctx.command.invoke(sub_ctx))
    File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
      return ctx.invoke(self.callback, **ctx.params)
    File "/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
      return __callback(*args, **kwargs)
    File "/tmp/sqlite-utils/sqlite_utils/cli.py", line 1959, in memory
      _load_extensions(db, load_extension)
    File "/tmp/sqlite-utils/sqlite_utils/cli.py", line 3232, in _load_extensions
      if ":" in ext:
  TypeError: argument of type 'NoneType' is not iterable
  ```
  Then I ran this:
  ```bash
  git checkout -b MikeCoats-spatialite-paths-linux-arm main
  git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm
  ```
  And now:
  ```bash
  sqlite-utils memory "select spatialite_version()" --load-extension=spatialite
  ```
  Outputs:
  ```json
  [{"spatialite_version()": "5.0.1"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/599#issuecomment-1793268750
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/599
node_id: IC_kwDOCGYnMM5q4xwO
created_at: 2023-11-04T00:33:25Z
updated_at: 2023-11-04T00:33:25Z
author_association: OWNER
body:
  See details of how I tested this here:
  - https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126

  Short version: having applied this fix, the following command (on simulated `aarch64`):
  ```bash
  sqlite-utils memory "select spatialite_version()" --load-extension=spatialite
  ```
  Outputs:
  ```json
  [{"spatialite_version()": "5.0.1"}]
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793269219
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/600
node_id: IC_kwDOCGYnMM5q4x3j
created_at: 2023-11-04T00:34:33Z
updated_at: 2023-11-04T00:34:33Z
author_association: OWNER
body:
  The GIS tests now pass in that container too:
  ```bash
  pytest tests/test_gis.py
  ```
  ```
  ======================== test session starts =========================
  platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
  rootdir: /tmp/sqlite-utils
  plugins: hypothesis-6.88.1
  collected 12 items                                                   

  tests/test_gis.py ............                                 [100%]

  ========================= 12 passed in 0.48s =========================
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1793273968
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/433
node_id: IC_kwDOCGYnMM5q4zBw
created_at: 2023-11-04T00:45:19Z
updated_at: 2023-11-04T00:45:19Z
author_association: OWNER
body:
  Here's an animated GIF that demonstrates the bug:

  ![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/18e54b38-a466-4384-9162-114eb01526a2)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1793274350
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/433
node_id: IC_kwDOCGYnMM5q4zHu
created_at: 2023-11-04T00:46:30Z
updated_at: 2023-11-04T00:46:30Z
author_association: OWNER
body:
  And a GIF of the fix after applying:
  - #598

  ![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/35829aec-c9ac-4925-a8e6-ffe7c2ab0d96)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793274485
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/598
node_id: IC_kwDOCGYnMM5q4zJ1
created_at: 2023-11-04T00:46:55Z
updated_at: 2023-11-04T00:46:55Z
author_association: OWNER
body:
  Manually tested. Before:

  ![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/3bdd30ea-1a54-4fec-b42d-793130a17bc1)

  After:

  ![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/015d4e4e-b40c-4a93-81f5-1a4adef69b11)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/1415#issuecomment-1793787454
issue_url: https://api.github.com/repos/simonw/datasette/issues/1415
node_id: IC_kwDOBm6k_c5q6wY-
created_at: 2023-11-05T16:44:49Z
updated_at: 2023-11-05T16:46:59Z
author_association: NONE
body:
  thanks for documenting this @bendnorman! got stuck at exactly the same point `gcloud builds submit ... returned non-zero exit status 1`, without a clue why this was happening. i now managed to get the github action to deploy datasette by assigning the following roles to the service account: `roles/run.admin`, `roles/storage.admin`, `roles/cloudbuild.builds.builder`, `roles/viewer`, `roles/iam.serviceAccountUser`.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2205#issuecomment-1793880738
issue_url: https://api.github.com/repos/simonw/datasette/issues/2205
node_id: IC_kwDOBm6k_c5q7HKi
created_at: 2023-11-05T23:26:14Z
updated_at: 2023-11-05T23:26:14Z
author_association: OWNER
body:
  I found this problem while trying to use WTForms with this pattern:

  ```python
  choices = [(col, col) for col in await db.table_columns(table)]

  class ConfigForm(Form):
      template = TextAreaField("Template")
      api_token = PasswordField("OpenAI API token")
      columns = MultiCheckboxField('Columns', choices=choices)
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2205#issuecomment-1794052079
issue_url: https://api.github.com/repos/simonw/datasette/issues/2205
node_id: IC_kwDOBm6k_c5q7w_v
created_at: 2023-11-06T04:06:05Z
updated_at: 2023-11-06T04:08:50Z
author_association: OWNER
body:
  It should return a `MultiParams`:

  https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/__init__.py#L900-L917

  Change needs to be made before 1.0.

  ```python
  return MultiParams(urllib.parse.parse_qs(body.decode("utf-8")))
  ```
  Need to remember why I was using `keep_blank_values=True` there and check that using `MultiParams` doesn't conflict with that reason.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2205#issuecomment-1794054390
issue_url: https://api.github.com/repos/simonw/datasette/issues/2205
node_id: IC_kwDOBm6k_c5q7xj2
created_at: 2023-11-06T04:09:43Z
updated_at: 2023-11-06T04:10:34Z
author_association: OWNER
body:
  That `keep_blank_values=True` is from https://github.com/simonw/datasette/commit/0934844c0b6d124163d0185fb6a41ba5a71433da

  Commit message:
  >  request.post_vars() no longer discards empty values 

  Relevant test:

  https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/tests/test_internals_request.py#L19-L27

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2202#issuecomment-1801876943
issue_url: https://api.github.com/repos/simonw/datasette/issues/2202
node_id: IC_kwDOBm6k_c5rZnXP
created_at: 2023-11-08T13:19:00Z
updated_at: 2023-11-08T13:19:00Z
author_association: CONTRIBUTOR
body: Superseded by #2206.
reactions:
[cell repeats an earlier one in this file: 112 characters]

performed_via_github_app:
  {"id": 29110, "slug": "dependabot", "node_id": "MDM6QXBwMjkxMTA=", "owner": {"login": "github", "id": 9919, "node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4", "gravatar_id": "", "url": "https://api.github.com/users/github", "html_url": "https://github.com/github", "followers_url": "https://api.github.com/users/github/followers", "following_url": "https://api.github.com/users/github/following{/other_user}", "gists_url": "https://api.github.com/users/github/gists{/gist_id}", "starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/github/subscriptions", "organizations_url": "https://api.github.com/users/github/orgs", "repos_url": "https://api.github.com/users/github/repos", "events_url": "https://api.github.com/users/github/events{/privacy}", "received_events_url": "https://api.github.com/users/github/received_events", "type": "Organization", "site_admin": false}, "name": "Dependabot", "description": "", "external_url": "https://dependabot-api.githubapp.com", "html_url": "https://github.com/apps/dependabot", "created_at": "2019-04-16T22:34:25Z", "updated_at": "2023-10-12T13:35:09Z", "permissions": {"checks": "write", "contents": "write", "issues": "write", "members": "read", "metadata": "read", "pull_requests": "write", "statuses": "read", "vulnerability_alerts": "read", "workflows": "write"}, "events": ["check_suite", "issues", "issue_comment", "label", "pull_request", "pull_request_review", "pull_request_review_comment", "repository"]}


html_url: https://github.com/simonw/datasette/pull/2206#issuecomment-1801888957
issue_url: https://api.github.com/repos/simonw/datasette/issues/2206
node_id: IC_kwDOBm6k_c5rZqS9
created_at: 2023-11-08T13:26:13Z
updated_at: 2023-11-08T13:26:13Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified and coverable lines are covered by tests :white_check_mark:
  > Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`eec10df`)](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2206   +/-   ##
  =======================================
    Coverage   92.69%   92.69%           
  =======================================
    Files          40       40           
    Lines        6047     6047           
  =======================================
    Hits         5605     5605           
    Misses        442      442           
  ```




  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]

performed_via_github_app:
  {"id": 254, "slug": "codecov", "node_id": "MDM6QXBwMjU0", "owner": {"login": "codecov", "id": 8226205, "node_id": "MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU=", "avatar_url": "https://avatars.githubusercontent.com/u/8226205?v=4", "gravatar_id": "", "url": "https://api.github.com/users/codecov", "html_url": "https://github.com/codecov", "followers_url": "https://api.github.com/users/codecov/followers", "following_url": "https://api.github.com/users/codecov/following{/other_user}", "gists_url": "https://api.github.com/users/codecov/gists{/gist_id}", "starred_url": "https://api.github.com/users/codecov/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/codecov/subscriptions", "organizations_url": "https://api.github.com/users/codecov/orgs", "repos_url": "https://api.github.com/users/codecov/repos", "events_url": "https://api.github.com/users/codecov/events{/privacy}", "received_events_url": "https://api.github.com/users/codecov/received_events", "type": "Organization", "site_admin": false}, "name": "Codecov", "description": "Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. ", "external_url": "https://codecov.io", "html_url": "https://github.com/apps/codecov", "created_at": "2016-09-25T14:18:27Z", "updated_at": "2023-09-08T15:29:16Z", "permissions": {"administration": "read", "checks": "write", "contents": "read", "emails": "read", "issues": "read", "members": "read", "metadata": "read", "pull_requests": "write", "statuses": "write"}, "events": ["check_run", "check_suite", "create", "delete", "fork", "member", "membership", "organization", "public", "pull_request", "push", "release", "repository", "status", "team_add"]}


html_url: https://github.com/simonw/datasette/issues/2208#issuecomment-1812617851
issue_url: https://api.github.com/repos/simonw/datasette/issues/2208
node_id: IC_kwDOBm6k_c5sClp7
created_at: 2023-11-15T14:18:58Z
updated_at: 2023-11-15T14:18:58Z
author_association: CONTRIBUTOR
body:
  Without aliases:

  ![image](https://github.com/simonw/datasette/assets/198537/d9703d3b-9733-4e87-9954-4fc60a07784a)

  The proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column).

  ![image](https://github.com/simonw/datasette/assets/198537/a37a0a1a-c36a-4c78-bdce-01b582637cc6)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2209#issuecomment-1812623778
issue_url: https://api.github.com/repos/simonw/datasette/issues/2209
node_id: IC_kwDOBm6k_c5sCnGi
created_at: 2023-11-15T14:22:42Z
updated_at: 2023-11-15T15:24:09Z
author_association: CONTRIBUTOR
body:
  Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2209#issuecomment-1812750369
issue_url: https://api.github.com/repos/simonw/datasette/issues/2209
node_id: IC_kwDOBm6k_c5sDGAh
created_at: 2023-11-15T15:29:37Z
updated_at: 2023-11-15T15:29:37Z
author_association: CONTRIBUTOR
body:
  Looks like tests are passing now but there is an issue with yaml loading and/or cog.

  https://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/pull/2209#issuecomment-1812753347
issue_url: https://api.github.com/repos/simonw/datasette/issues/2209
node_id: IC_kwDOBm6k_c5sDGvD
created_at: 2023-11-15T15:31:12Z
updated_at: 2023-11-15T15:31:12Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified and coverable lines are covered by tests :white_check_mark:
  > Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`c88414b`)](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.


  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@           Coverage Diff           @@
  ##             main    #2209   +/-   ##
  =======================================
    Coverage   92.69%   92.69%           
  =======================================
    Files          40       40           
    Lines        6047     6047           
  =======================================
    Hits         5605     5605           
    Misses        442      442           
  ```



  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]

performed_via_github_app:
  {"id": 254, "slug": "codecov", "node_id": "MDM6QXBwMjU0", "owner": {"login": "codecov", "id": 8226205, "node_id": "MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU=", "avatar_url": "https://avatars.githubusercontent.com/u/8226205?v=4", "gravatar_id": "", "url": "https://api.github.com/users/codecov", "html_url": "https://github.com/codecov", "followers_url": "https://api.github.com/users/codecov/followers", "following_url": "https://api.github.com/users/codecov/following{/other_user}", "gists_url": "https://api.github.com/users/codecov/gists{/gist_id}", "starred_url": "https://api.github.com/users/codecov/starred{/owner}{/repo}", "subscriptions_url": "https://api.github.com/users/codecov/subscriptions", "organizations_url": "https://api.github.com/users/codecov/orgs", "repos_url": "https://api.github.com/users/codecov/repos", "events_url": "https://api.github.com/users/codecov/events{/privacy}", "received_events_url": "https://api.github.com/users/codecov/received_events", "type": "Organization", "site_admin": false}, "name": "Codecov", "description": "Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. ", "external_url": "https://codecov.io", "html_url": "https://github.com/apps/codecov", "created_at": "2016-09-25T14:18:27Z", "updated_at": "2023-09-08T15:29:16Z", "permissions": {"administration": "read", "checks": "write", "contents": "read", "emails": "read", "issues": "read", "members": "read", "metadata": "read", "pull_requests": "write", "statuses": "write"}, "events": ["check_run", "check_suite", "create", "delete", "fork", "member", "membership", "organization", "public", "pull_request", "push", "release", "repository", "status", "team_add"]}


html_url: https://github.com/simonw/sqlite-utils/issues/344#issuecomment-1815825863
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/344
node_id: IC_kwDOCGYnMM5sO03H
created_at: 2023-11-17T06:44:49Z
updated_at: 2023-11-17T06:44:49Z
author_association: CONTRIBUTOR
body:
  hello Simon,

  I've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4.
  It also fixes table.transform() to preserve STRICT mode.
  Please pull if you deem appropriate. Thanks!

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2213#issuecomment-1843072926
issue_url: https://api.github.com/repos/simonw/datasette/issues/2213
node_id: IC_kwDOBm6k_c5t2w-e
created_at: 2023-12-06T15:05:44Z
updated_at: 2023-12-06T15:05:44Z
author_association: CONTRIBUTOR
body:
  it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.

  i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843465748
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/606
node_id: IC_kwDOCGYnMM5t4Q4U
created_at: 2023-12-06T18:36:51Z
updated_at: 2023-12-06T18:36:51Z
author_association: OWNER
body:
  I'll add `bytes` too - `float` already works. This makes sense because when you are working with the Python API you use `str` and `float` and `bytes` and `int` to specify column types.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843579184
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/606
node_id: IC_kwDOCGYnMM5t4skw
created_at: 2023-12-06T19:43:55Z
updated_at: 2023-12-06T19:43:55Z
author_association: OWNER
body:
  Updated documentation:
  - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column
  - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843585454
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/604
node_id: IC_kwDOCGYnMM5t4uGu
created_at: 2023-12-06T19:48:26Z
updated_at: 2023-12-08T05:05:03Z
author_association: NONE
body:
  ## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
  All modified and coverable lines are covered by tests :white_check_mark:
  > Comparison is base [(`9286c1b`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/9286c1ba432e890b1bb4b2a1f847b15364c1fa18?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`1698a9d`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.72%.
  > Report is 1 commits behind head on main.

  > :exclamation: Current head 1698a9d differs from pull request most recent head 61c6e26. Consider uploading reports for the commit 61c6e26 to get more accurate results


  <details><summary>Additional details and impacted files</summary>


  ```diff
  @@            Coverage Diff             @@
  ##             main     #604      +/-   ##
  ==========================================
  - Coverage   95.77%   95.72%   -0.06%     
  ==========================================
    Files           8        8              
    Lines        2842     2852      +10     
  ==========================================
  + Hits         2722     2730       +8     
  - Misses        120      122       +2     
  ```



  </details>

  [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
  :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).

reactions:
[cell repeats an earlier one in this file: 112 characters]

performed_via_github_app:
[cell repeats an earlier one in this file: 3255 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843586503
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/604
node_id: IC_kwDOCGYnMM5t4uXH
created_at: 2023-12-06T19:49:10Z
updated_at: 2023-12-06T19:49:29Z
author_association: OWNER
body:
  This looks really great on first glance - design is good, implementation is solid, tests and documentation look great.

  Looks like a couple of `mypy` failures in the tests at the moment:

  ```
    mypy sqlite_utils tests

  sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type "type[Table]", variable has type "type[View]")  [assignment]
  tests/test_lookup.py:156: error: Name "test_lookup_new_table" already defined on line 5  [no-redef]
  Found 2 errors in 2 files (checked 54 source files)
  Error: Process completed with exit code 1.
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/datasette/issues/2214#issuecomment-1844819002
issue_url: https://api.github.com/repos/simonw/datasette/issues/2214
node_id: IC_kwDOBm6k_c5t9bQ6
created_at: 2023-12-07T07:36:33Z
updated_at: 2023-12-07T07:36:33Z
author_association: NONE
body:
  If I uncheck `expand labels` in the Advanced CSV export dialog, the error does not occur. Re-checking that box and re-running the export does cause the error to occur.

  ![CleanShot 2023-12-06 at 23 34 58@2x](https://github.com/simonw/datasette/assets/2874/12c6c241-35ce-4ded-8dc7-fc250d809ed9)

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/605#issuecomment-1846554637
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/605
node_id: IC_kwDOCGYnMM5uEDAN
created_at: 2023-12-08T05:07:54Z
updated_at: 2023-12-08T05:07:54Z
author_association: OWNER
body:
  Thanks for opening an issue - this should help future Google searchers figure out what's going on here.

  Another approach here could be to store large integers as `TEXT` in SQLite (or even as `BLOB`).

  Both storing as `REAL` and storing as `TEXT/BLOB` feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/issues/603#issuecomment-1846555822
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/603
node_id: IC_kwDOCGYnMM5uEDSu
created_at: 2023-12-08T05:09:55Z
updated_at: 2023-12-08T05:10:31Z
author_association: OWNER
body:
  I'm unable to replicate this issue. This is with a fresh install of `sqlite-utils==3.35.2`:
  ```
  (base) ~ python3.12
  Python 3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import sqlite_utils
  >>> db = sqlite_utils.Database(memory=True)
  >>> db["foo"].insert({"bar": 1})
  <Table foo (bar)>
  >>> import sys
  >>> sys.version
  '3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]'
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


html_url: https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1846560096
issue_url: https://api.github.com/repos/simonw/sqlite-utils/issues/604
node_id: IC_kwDOCGYnMM5uEEVg
created_at: 2023-12-08T05:16:44Z
updated_at: 2023-12-08T05:17:20Z
author_association: OWNER
body:
  Also tested this manually like so:

  ```bash
  sqlite-utils create-table strict.db strictint id integer size integer --strict
  sqlite-utils create-table strict.db notstrictint id integer size integer         
  sqlite-utils install sqlite-utils-shell
  sqlite-utils shell strict.db                                             
  ```
  ```
  Attached to strict.db
  Type 'exit' to exit.
  sqlite-utils> insert into strictint (size) values (4);
  1 row affected
  sqlite-utils> insert into strictint (size) values ('four');
  An error occurred: cannot store TEXT value in INTEGER column strictint.size
  sqlite-utils> insert into notstrictint (size) values ('four');
  1 row affected
  sqlite-utils> commit;
  Done
  ```

reactions:
[cell repeats an earlier one in this file: 112 characters]


