table,id,node_id,title,user,created_at,body,repo
issue_comments,1847317568,IC_kwDODFdgUs5uG9RA,,23789,2023-12-08T14:50:13Z,Adding `&per_page=100` would reduce the number of API requests by 3x.,207052882
issue_comments,1846560096,IC_kwDOCGYnMM5uEEVg,,9599,2023-12-08T05:16:44Z,"Also tested this manually like so:

```bash
sqlite-utils create-table strict.db strictint id integer size integer --strict
sqlite-utils create-table strict.db notstrictint id integer size integer         
sqlite-utils install sqlite-utils-shell
sqlite-utils shell strict.db                                             
```
```
Attached to strict.db
Type 'exit' to exit.
sqlite-utils> insert into strictint (size) values (4);
1 row affected
sqlite-utils> insert into strictint (size) values ('four');
An error occurred: cannot store TEXT value in INTEGER column strictint.size
sqlite-utils> insert into notstrictint (size) values ('four');
1 row affected
sqlite-utils> commit;
Done
```",140912432
issue_comments,1846555822,IC_kwDOCGYnMM5uEDSu,,9599,2023-12-08T05:09:55Z,"I'm unable to replicate this issue. This is with a fresh install of `sqlite-utils==3.35.2`:
```
(base) ~ python3.12
Python 3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> db[""foo""].insert({""bar"": 1})
<Table foo (bar)>
>>> import sys
>>> sys.version
'3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]'
```",140912432
issue_comments,1846554637,IC_kwDOCGYnMM5uEDAN,,9599,2023-12-08T05:07:54Z,"Thanks for opening an issue - this should help future Google searchers figure out what's going on here.

Another approach here could be to store large integers as `TEXT` in SQLite (or even as `BLOB`).

Both storing as `REAL` and storing as `TEXT/BLOB` feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.",140912432
issue_comments,1844819002,IC_kwDOBm6k_c5t9bQ6,,2874,2023-12-07T07:36:33Z,"If I uncheck `expand labels` in the Advanced CSV export dialog, the error does not occur. Re-checking that box and re-running the export does cause the error to occur.

![CleanShot 2023-12-06 at 23 34 58@2x](https://github.com/simonw/datasette/assets/2874/12c6c241-35ce-4ded-8dc7-fc250d809ed9)
",107914493
issues,2029908157,I_kwDOBm6k_c54_fC9,CSV export fails for some `text` foreign key references,2874,2023-12-07T05:04:34Z,"I'm starting this issue without a clear reproduction in case someone else has seen this behavior, and to use the issue as a notebook for research. 

I'm using Datasette with the [SWITRS](https://iswitrs.chp.ca.gov/) data set, which is a California Highway Patrol collection of traffic incident data from the past decade or so. I receive data from them in CSV and want to work with it in Datasette, then export it to CSV for mapping in Felt.com.

Their data makes extensive use of codes for incident column data (`1` for `Monday` and so on), some of it integer codes and some of it letter/text codes. The text codes are sometimes blank or `-`. During import, I'm creating lookup tables for foreign key references to make the Datasette UI presentation of the data easier to read.

If I import the data and set up the integer foreign keys, everything works fine, but if I set up the text foreign keys, CSV export starts to fail. 

The foreign key configuration is as follows:

```
# Some tables use integer ids, like sensible tables do. Let's import them first
# since we favor them.

for TABLE in DAY_OF_WEEK CHP_SHIFT POPULATION SPECIAL_COND BEAT_TYPE COLLISION_SEVERITY
do
	sqlite-utils create-table records.db $TABLE id integer name text --pk=id
	sqlite-utils insert records.db $TABLE lookup-tables/$TABLE.csv --csv
	sqlite-utils add-foreign-key records.db collisions $TABLE $TABLE id
	sqlite-utils create-index records.db collisions $TABLE
done

# *Other* tables use letter keys, like they were raised by WOLVES. Let's put them
# at the end of the import queue.

for TABLE in WEATHER_1 WEATHER_2 LOCATION_TYPE RAMP_INTERSECTION SIDE_OF_HWY \
PRIMARY_COLL_FACTOR PCF_CODE_OF_VIOL PCF_VIOL_CATEGORY TYPE_OF_COLLISION MVIW \
PED_ACTION ROAD_SURFACE ROAD_COND_1 ROAD_COND_2 LIGHTING CONTROL_DEVICE \
STWD_VEHTYPE_AT_FAULT CHP_VEHTYPE_AT_FAULT PRIMARY_RAMP SECONDARY_RAMP
do
	sqlite-utils create-table records.db $TABLE key text name text --pk=key
	sqlite-utils insert records.db $TABLE lookup-tables/$TABLE.csv --csv
	sqlite-utils add-foreign-key records.db collisions $TABLE $TABLE key
	sqlite-utils create-index records.db collisions $TABLE
done
```

You can see the full code and import script here: https://github.com/radical-bike-lobby/switrs-db

If I run this code and then hit the CSV export link in the Datasette interface (the simple link or the ""advanced"" dialog), export fails after a small number of CSV rows are written. I am not seeing any detailed error messages but this appears in the logging output:

```
INFO:     127.0.0.1:57885 - ""GET /records/collisions.csv?_facet=PRIMARY_RD&PRIMARY_RD=ASHBY+AV&_labels=on&_size=max HTTP/1.1"" 200 OK
Caught this error: 

```

(No other output follows `error:` other than a blank line.)

I've stared at the rows directly after the error occurs and can't yet see what is causing the problem. I'm going to set up a development environment and see if I get any more detailed error output, and then stare more at some problematic lines to see if I can get a simple reproduction.",107914493
issue_comments,1843975536,IC_kwDOCGYnMM5t6NVw,,16437338,2023-12-07T01:17:05Z,Apologies - I pushed a fix that addresses the mypy failures.,140912432
issue_comments,1843586503,IC_kwDOCGYnMM5t4uXH,,9599,2023-12-06T19:49:10Z,"This looks really great on first glance - design is good, implementation is solid, tests and documentation look great.

Looks like a couple of `mypy` failures in the tests at the moment:

```
  mypy sqlite_utils tests

sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type ""type[Table]"", variable has type ""type[View]"")  [assignment]
tests/test_lookup.py:156: error: Name ""test_lookup_new_table"" already defined on line 5  [no-redef]
Found 2 errors in 2 files (checked 54 source files)
Error: Process completed with exit code 1.
```

",140912432
issue_comments,1843585454,IC_kwDOCGYnMM5t4uGu,,22429695,2023-12-06T19:48:26Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`9286c1b`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/9286c1ba432e890b1bb4b2a1f847b15364c1fa18?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`1698a9d`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.72%.
> Report is 1 commits behind head on main.

> :exclamation: Current head 1698a9d differs from pull request most recent head 61c6e26. Consider uploading reports for the commit 61c6e26 to get more accurate results


<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #604      +/-   ##
==========================================
- Coverage   95.77%   95.72%   -0.06%     
==========================================
  Files           8        8              
  Lines        2842     2852      +10     
==========================================
+ Hits         2722     2730       +8     
- Misses        120      122       +2     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issue_comments,1843579184,IC_kwDOCGYnMM5t4skw,,9599,2023-12-06T19:43:55Z,"Updated documentation:
- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column",140912432
issue_comments,1843465748,IC_kwDOCGYnMM5t4Q4U,,9599,2023-12-06T18:36:51Z,I'll add `bytes` too - `float` already works. This makes sense because when you are working with the Python API you use `str` and `float` and `bytes` and `int` to specify column types.,140912432
issues,2029161033,I_kwDOCGYnMM548opJ,str and int as aliases for text and integer,9599,2023-12-06T18:35:49Z,"I keep making this mistake:
```bash
sqlite-utils add-column content.db assets _since int
```
```
Usage: sqlite-utils add-column [OPTIONS] PATH TABLE COL_NAME [[integer|float|b
                               lob|text|INTEGER|FLOAT|BLOB|TEXT]]
Try 'sqlite-utils add-column -h' for help.

Error: Invalid value for '[[integer|float|blob|text|INTEGER|FLOAT|BLOB|TEXT]]':
  'int' is not one of 'integer', 'float', 'blob', 'text', 'INTEGER', 'FLOAT', 'BLOB', 'TEXT'.
```",140912432
issue_comments,1843072926,IC_kwDOBm6k_c5t2w-e,,536941,2023-12-06T15:05:44Z,"it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.

i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).",107914493
issues,2028698018,I_kwDOBm6k_c5463mi,feature request: gzip compression of database downloads,536941,2023-12-06T14:35:03Z,"At the bottom of database pages, datasette gives users the opportunity to download the underlying sqlite database. It would be great if that could be served gzip compressed. 

this is similar to #1213, but for me, i don't need datasette to compress html and json because my CDN layer does it for me, however, cloudflare at least, will not compress a mimetype of ""application""

(see list of mimetype: https://developers.cloudflare.com/speed/optimization/content/brotli/content-compression/)",107914493
issues,2023057255,I_kwDOBm6k_c54lWdn,Can't filter with numbers,605070,2023-12-04T05:26:29Z,"I have a schema that uses numbers for a column (actually it's a boolean 1 or 0 but SQLite doesn't have Boolean).
I can't seem to get the facet to work or even filtering on this column.

My guess is that Datasette is ""stringifying"" the number and it's not matching?
Example: https://debian-sqlelf.fly.dev/debian/elf_symbols?_sort_desc=name&_facet=exported&exported=0",107914493
issues,2019811176,I_kwDOBm6k_c54Y99o,Unreachable exception handlers for `sqlite3.OperationalError`,1214074,2023-12-01T00:50:22Z,"There are several places where `sqlite3.OperationalError` is caught as part of an exception handler which catches multiple exceptions, but is then caught again immediately afterwards by a dedicated exception handler.

Because the exception will be caught by the first handler, the logic in the second handler is unreachable and will never be executed.  If this is intended behavior, the second handler can be removed.  If this is not intended, and the second handler should be the one that catches this exception, then `sqlite3.OperationalError` should be removed from the tuple of exceptions in the first handler.

This issue was found via a CodeQL query on the repository, and I've listed the occurrences found by the query below.  There may be other instances of this issue in the code that were not surfaced by the query.  I'd be happy to share the query if others would like to view or run it.

One example:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/views/database.py#L534-L537

Other instances:

https://github.com/simonw/datasette/blob/main/datasette/views/base.py#L266-L270
https://github.com/simonw/datasette/blob/main/datasette/views/base.py#L452-L456",107914493
issues,2007893839,I_kwDOCGYnMM53rgdP,Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?,12229877,2023-11-23T10:19:46Z,"I'm currently working on a new feature for Hypothesis, where we can dump a tidy jsonlines table of all the test cases we tried - including arguments, outcomes, timings, coverage, etc.  Exploring this seems like a perfect cases for `sqlite-utils` and `datasette`, but I pretty quickly ran into an integer overflow problem and don't want to recommend that experience to my users.

I originally went to report this as a bug... and then found https://github.com/simonw/sqlite-utils/issues/309#issuecomment-895581038 almost exactly matched my repro 😅 

https://github.com/simonw/sqlite-utils/issues/110#issuecomment-626391063 suggests that using `NUMERIC` would avoid this overflow error, although ""If the TEXT value is a well-formed integer literal that is too large to fit in a 64-bit signed integer, it is converted to REAL."" suggests that this would come at the cost of rounding to the nearest float value.  Maybe I should just convert large integers to float before writing out my json?

After a bit more hacking, ""manually cast large integers to float"" seems like a decent solution for my particular case, but having written it up I thought I might as well post this issue anyway - I hope it's useful feedback, and won't mind at all if you close as wontfix if it's not.",140912432
issues,2001006157,PR_kwDOCGYnMM5f2OZC,Add more STRICT table support,16437338,2023-11-19T19:38:53Z,"- https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776

Make `table.transform()` preserve STRICT mode.

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--604.org.readthedocs.build/en/604/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1816642044,IC_kwDOBm6k_c5sR8H8,,16142258,2023-11-17T15:32:20Z,Any progress on this? It would be very helpful on my end as well. Thanks!,107914493
issue_comments,1815825863,IC_kwDOCGYnMM5sO03H,,16437338,2023-11-17T06:44:49Z,"hello Simon,

I've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4.
It also fixes table.transform() to preserve STRICT mode.
Please pull if you deem appropriate. Thanks!",140912432
issue_comments,1812753347,IC_kwDOBm6k_c5sDGvD,,22429695,2023-11-15T15:31:12Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`c88414b`)](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.


<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2209   +/-   ##
=======================================
  Coverage   92.69%   92.69%           
=======================================
  Files          40       40           
  Lines        6047     6047           
=======================================
  Hits         5605     5605           
  Misses        442      442           
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1812750369,IC_kwDOBm6k_c5sDGAh,,198537,2023-11-15T15:29:37Z,"Looks like tests are passing now but there is an issue with yaml loading and/or cog.

https://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209",107914493
issue_comments,1812623778,IC_kwDOBm6k_c5sCnGi,,198537,2023-11-15T14:22:42Z,"Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.",107914493
issue_comments,1812617851,IC_kwDOBm6k_c5sClp7,,198537,2023-11-15T14:18:58Z,"Without aliases:

![image](https://github.com/simonw/datasette/assets/198537/d9703d3b-9733-4e87-9954-4fc60a07784a)

The proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column).

![image](https://github.com/simonw/datasette/assets/198537/a37a0a1a-c36a-4c78-bdce-01b582637cc6)
",107914493
issues,1994861266,PR_kwDOBm6k_c5fhgOS,Fix query for suggested facets with column named value,198537,2023-11-15T14:13:30Z,"See discussion in https://github.com/simonw/datasette/issues/2208

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2209.org.readthedocs.build/en/2209/

<!-- readthedocs-preview datasette end -->",107914493
issues,1994857251,I_kwDOBm6k_c525xsj,No suggested facets when a column named 'value' is included,198537,2023-11-15T14:11:17Z,"When a column named 'value' is included there are no suggested facets is shown as the query uses an alias of 'value'.

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/facets.py#L168-L174

Currently the following is shown (from https://latest.datasette.io/fixtures/facetable)

![image](https://github.com/simonw/datasette/assets/198537/a919509a-ea88-461b-b25b-8b776720c7c5)

When I add a column named 'value' only the JSON facets are processed.

![image](https://github.com/simonw/datasette/assets/198537/092bd0b3-4c20-434e-88f8-47e2b8994a1d)

I think that not using aliases could be a solution (except if someone wants to use a column named `count(*)` though this seems to be unlikely). I'll open a PR with that.

There is also a TODO with a similar question in the same file. I have not looked into that yet.

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/facets.py#L512",107914493
issues,1994845152,I_kwDOBm6k_c525uvg,ModuleNotFoundError: No module named 'click_default_group,283441,2023-11-15T14:04:32Z,"No matter what I do, I'm getting this error:

```
$ datasette
Traceback (most recent call last):
  File ""/Users/honza/Library/Caches/pypoetry/virtualenvs/juniorguru-Lgaxwd2n-py3.11/bin/datasette"", line 5, in <module>
    from datasette.cli import cli
  File ""/Users/honza/Library/Caches/pypoetry/virtualenvs/juniorguru-Lgaxwd2n-py3.11/lib/python3.11/site-packages/datasette/cli.py"", line 6, in <module>
    from click_default_group import DefaultGroup
ModuleNotFoundError: No module named 'click_default_group'
```

I have datasette in my dependencies like this:

```toml
[tool.poetry.group.dev.dependencies]
datasette = {version = ""1.0a7"", allow-prereleases = true}
```

I had the latest regular version (not pre-release) there originally, but the result was the same:

```toml
[tool.poetry.group.dev.dependencies]
datasette = ""0.64.5""
```

Full pyproject.toml is at https://github.com/honzajavorek/junior.guru/ Previously datasette worked for me, but I guess something had to upgrade and now I can't even launch it.",107914493
issues,1988525411,I_kwDOCGYnMM52hn1j,Pyhton 3.12 Bug report,1324252,2023-11-10T22:57:48Z,"I start with new python3 verison 3.12.0
Also have the error where connect DataBase

```
Traceback (most recent call last):
  File ""/home/t/Development/python/FKPJ/ClinicSYS/run.py"", line 1, in <module>
    import re, os, io, json, sqlite_utils, requests, pytz, logging
  File ""/home/t/.local/lib/python3.12/site-packages/sqlite_utils/__init__.py"", line 1, in <module>
    from .db import Database
  File ""/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py"", line 277, in <module>
    class Database:
  File ""/home/t/.local/lib/python3.12/site-packages/sqlite_utils/db.py"", line 306, in Database
    filename_or_conn: Optional[Union[str, pathlib.Path, sqlite3.Connection]] = None,
                                                        ^^^^^^^^^^^^^^^^^^
```
This bug come from `sqlite-utils` since's v3.33.
Anyone get the same ?

As well now of the resolved plan just keep the sqlite-utils version in python3.12 with v3.32.1 [tested]
but where are the sqlite3.Connection problem.... 

This won't happen on python version down to 3.11[tested]
Just the python3.12.0, I have test this error are come from the sqlite3 connection
The error say from `sqlite_utils` and with the sqlite3 Connection, what can I do.

Let fix together.",140912432
issue_comments,1801888957,IC_kwDOBm6k_c5rZqS9,,22429695,2023-11-08T13:26:13Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`eec10df`)](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2206   +/-   ##
=======================================
  Coverage   92.69%   92.69%           
=======================================
  Files          40       40           
  Lines        6047     6047           
=======================================
  Hits         5605     5605           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1801876943,IC_kwDOBm6k_c5rZnXP,,49699333,2023-11-08T13:19:00Z,Superseded by #2206.,107914493
issues,1983600865,PR_kwDOBm6k_c5e7WH7,Bump the python-packages group with 1 update,49699333,2023-11-08T13:18:56Z,"Bumps the python-packages group with 1 update: [black](https://github.com/psf/black).

<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>23.11.0</h2>
<h3>Highlights</h3>
<ul>
<li>Support formatting ranges of lines with the new <code>--line-ranges</code> command-line option
(<a href=""https://redirect.github.com/psf/black/issues/4020"">#4020</a>)</li>
</ul>
<h3>Stable style</h3>
<ul>
<li>Fix crash on formatting bytes strings that look like docstrings (<a href=""https://redirect.github.com/psf/black/issues/4003"">#4003</a>)</li>
<li>Fix crash when whitespace followed a backslash before newline in a docstring (<a href=""https://redirect.github.com/psf/black/issues/4008"">#4008</a>)</li>
<li>Fix standalone comments inside complex blocks crashing Black (<a href=""https://redirect.github.com/psf/black/issues/4016"">#4016</a>)</li>
<li>Fix crash on formatting code like <code>await (a ** b)</code> (<a href=""https://redirect.github.com/psf/black/issues/3994"">#3994</a>)</li>
<li>No longer treat leading f-strings as docstrings. This matches Python's behaviour and
fixes a crash (<a href=""https://redirect.github.com/psf/black/issues/4019"">#4019</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Multiline dicts and lists that are the sole argument to a function are now
indented less (<a href=""https://redirect.github.com/psf/black/issues/3964"">#3964</a>)</li>
<li>Multiline unpacked dicts and lists as the sole argument to a function are now also
indented less (<a href=""https://redirect.github.com/psf/black/issues/3992"">#3992</a>)</li>
<li>In f-string debug expressions, quote types that are visible in the final string
are now preserved (<a href=""https://redirect.github.com/psf/black/issues/4005"">#4005</a>)</li>
<li>Fix a bug where long <code>case</code> blocks were not split into multiple lines. Also enable
general trailing comma rules on <code>case</code> blocks (<a href=""https://redirect.github.com/psf/black/issues/4024"">#4024</a>)</li>
<li>Keep requiring two empty lines between module-level docstring and first function or
class definition (<a href=""https://redirect.github.com/psf/black/issues/4028"">#4028</a>)</li>
<li>Add support for single-line format skip with other comments on the same line (<a href=""https://redirect.github.com/psf/black/issues/3959"">#3959</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Consistently apply force exclusion logic before resolving symlinks (<a href=""https://redirect.github.com/psf/black/issues/4015"">#4015</a>)</li>
<li>Fix a bug in the matching of absolute path names in <code>--include</code> (<a href=""https://redirect.github.com/psf/black/issues/3976"">#3976</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Fix mypyc builds on arm64 on macOS (<a href=""https://redirect.github.com/psf/black/issues/4017"">#4017</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Black's pre-commit integration will now run only on git hooks appropriate for a code
formatter (<a href=""https://redirect.github.com/psf/black/issues/3940"">#3940</a>)</li>
</ul>
<h2>23.10.1</h2>
<h3>Highlights</h3>
<ul>
<li>Maintanence release to get a fix out for GitHub Action edge case (<a href=""https://redirect.github.com/psf/black/issues/3957"">#3957</a>)</li>
</ul>
<h3>Preview style</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>23.11.0</h2>
<h3>Highlights</h3>
<ul>
<li>Support formatting ranges of lines with the new <code>--line-ranges</code> command-line option
(<a href=""https://redirect.github.com/psf/black/issues/4020"">#4020</a>)</li>
</ul>
<h3>Stable style</h3>
<ul>
<li>Fix crash on formatting bytes strings that look like docstrings (<a href=""https://redirect.github.com/psf/black/issues/4003"">#4003</a>)</li>
<li>Fix crash when whitespace followed a backslash before newline in a docstring (<a href=""https://redirect.github.com/psf/black/issues/4008"">#4008</a>)</li>
<li>Fix standalone comments inside complex blocks crashing Black (<a href=""https://redirect.github.com/psf/black/issues/4016"">#4016</a>)</li>
<li>Fix crash on formatting code like <code>await (a ** b)</code> (<a href=""https://redirect.github.com/psf/black/issues/3994"">#3994</a>)</li>
<li>No longer treat leading f-strings as docstrings. This matches Python's behaviour and
fixes a crash (<a href=""https://redirect.github.com/psf/black/issues/4019"">#4019</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Multiline dicts and lists that are the sole argument to a function are now indented
less (<a href=""https://redirect.github.com/psf/black/issues/3964"">#3964</a>)</li>
<li>Multiline unpacked dicts and lists as the sole argument to a function are now also
indented less (<a href=""https://redirect.github.com/psf/black/issues/3992"">#3992</a>)</li>
<li>In f-string debug expressions, quote types that are visible in the final string are
now preserved (<a href=""https://redirect.github.com/psf/black/issues/4005"">#4005</a>)</li>
<li>Fix a bug where long <code>case</code> blocks were not split into multiple lines. Also enable
general trailing comma rules on <code>case</code> blocks (<a href=""https://redirect.github.com/psf/black/issues/4024"">#4024</a>)</li>
<li>Keep requiring two empty lines between module-level docstring and first function or
class definition (<a href=""https://redirect.github.com/psf/black/issues/4028"">#4028</a>)</li>
<li>Add support for single-line format skip with other comments on the same line (<a href=""https://redirect.github.com/psf/black/issues/3959"">#3959</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Consistently apply force exclusion logic before resolving symlinks (<a href=""https://redirect.github.com/psf/black/issues/4015"">#4015</a>)</li>
<li>Fix a bug in the matching of absolute path names in <code>--include</code> (<a href=""https://redirect.github.com/psf/black/issues/3976"">#3976</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Fix mypyc builds on arm64 on macOS (<a href=""https://redirect.github.com/psf/black/issues/4017"">#4017</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Black's pre-commit integration will now run only on git hooks appropriate for a code
formatter (<a href=""https://redirect.github.com/psf/black/issues/3940"">#3940</a>)</li>
</ul>
<h2>23.10.1</h2>
<h3>Highlights</h3>
<ul>
<li>Maintenance release to get a fix out for GitHub Action edge case (<a href=""https://redirect.github.com/psf/black/issues/3957"">#3957</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/psf/black/commit/2a1c67e0b2f81df602ec1f6e7aeb030b9709dc7c""><code>2a1c67e</code></a> Prepare release 23.11.0 (<a href=""https://redirect.github.com/psf/black/issues/4032"">#4032</a>)</li>
<li><a href=""https://github.com/psf/black/commit/72e7a2e43eef2aa0c83652bb6725eb004a2a69f3""><code>72e7a2e</code></a> Remove redundant condition from <code>has_magic_trailing_comma</code> (<a href=""https://redirect.github.com/psf/black/issues/4023"">#4023</a>)</li>
<li><a href=""https://github.com/psf/black/commit/1a7d9c2f58de1ffcbbe6d133f60f283601ba3f54""><code>1a7d9c2</code></a> Preserve visible quote types for f-string debug expressions (<a href=""https://redirect.github.com/psf/black/issues/4005"">#4005</a>)</li>
<li><a href=""https://github.com/psf/black/commit/f4c7be5445c87d9af5eba3d12faea62d2635e3d8""><code>f4c7be5</code></a> docs: fix minor typo (<a href=""https://redirect.github.com/psf/black/issues/4030"">#4030</a>)</li>
<li><a href=""https://github.com/psf/black/commit/2e4fac9d87615e904a49e46a9cab2293e0b13126""><code>2e4fac9</code></a> Apply force exclude logic before symlink resolution (<a href=""https://redirect.github.com/psf/black/issues/4015"">#4015</a>)</li>
<li><a href=""https://github.com/psf/black/commit/66008fda5dc07f5626e5f5d0dcefc476a9c12ab8""><code>66008fd</code></a> [563] Fix standalone comments inside complex blocks crashing Black (<a href=""https://redirect.github.com/psf/black/issues/4016"">#4016</a>)</li>
<li><a href=""https://github.com/psf/black/commit/50ed6221d97b265025abaa66116a7b185f2df5e2""><code>50ed622</code></a> Fix long case blocks not split into multiple lines (<a href=""https://redirect.github.com/psf/black/issues/4024"">#4024</a>)</li>
<li><a href=""https://github.com/psf/black/commit/46be1f8e54ac9a7d67723c0fa28c7bec13a0a2bf""><code>46be1f8</code></a> Support formatting specified lines (<a href=""https://redirect.github.com/psf/black/issues/4020"">#4020</a>)</li>
<li><a href=""https://github.com/psf/black/commit/ecbd9e8cf71f13068c7e6803a534e00363114c91""><code>ecbd9e8</code></a> Fix crash with f-string docstrings (<a href=""https://redirect.github.com/psf/black/issues/4019"">#4019</a>)</li>
<li><a href=""https://github.com/psf/black/commit/e808e61db8c7a8f9c7fd4b2fff2281141f6b2517""><code>e808e61</code></a> Preview: Keep requiring two empty lines between module-level docstring and fi...</li>
<li>Additional commits viewable in <a href=""https://github.com/psf/black/compare/23.9.1...23.11.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=23.9.1&new-version=23.11.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2206.org.readthedocs.build/en/2206/

<!-- readthedocs-preview datasette end -->",107914493
issues,1978603203,I_kwDOCGYnMM517xbD,`sqlite-utils transform` removes the `AUTOINCREMENT` keyword,4472046,2023-11-06T08:48:43Z,"### Context

We ran into this bug randomly, noticing that deleted `ROWID` would get reused after migrating the DB. Using `transform` to change any column in the table will also unexpectedly strip away the `AUTOINCREMENT` keyword from the primary key definition, even if it was not the transformation target.

### Reproducible example

**Original database**

```sql
$ sqlite3 test.db << EOF
CREATE TABLE mytable (
    col1 INTEGER PRIMARY KEY AUTOINCREMENT,
    col2 TEXT NOT NULL
)
EOF

$ sqlite3 test.db "".schema mytable""
CREATE TABLE mytable (
    col1 INTEGER PRIMARY KEY AUTOINCREMENT,
    col2 TEXT NOT NULL
);
```

**Modified database after sqlite-utils**

```sql
$ sqlite-utils transform test.db mytable --rename col2 renamedcol2

$ sqlite3 test.db ""SELECT sql FROM sqlite_master WHERE name = 'mytable';""
CREATE TABLE IF NOT EXISTS ""mytable"" (
   [col1] INTEGER PRIMARY KEY,
   [renamedcol2] TEXT NOT NULL
);
```",140912432
issue_comments,1794054390,IC_kwDOBm6k_c5q7xj2,,9599,2023-11-06T04:09:43Z,"That `keep_blank_values=True` is from https://github.com/simonw/datasette/commit/0934844c0b6d124163d0185fb6a41ba5a71433da

Commit message:
>  request.post_vars() no longer discards empty values 

Relevant test:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/tests/test_internals_request.py#L19-L27",107914493
issue_comments,1794052079,IC_kwDOBm6k_c5q7w_v,,9599,2023-11-06T04:06:05Z,"It should return a `MultiParams`:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/__init__.py#L900-L917

Change needs to be made before 1.0.

```python
return MultiParams(urllib.parse.parse_qs(body.decode(""utf-8"")))
```
Need to remember why I was using `keep_blank_values=True` there and check that using `MultiParams` doesn't conflict with that reason.",107914493
issue_comments,1793880738,IC_kwDOBm6k_c5q7HKi,,9599,2023-11-05T23:26:14Z,"I found this problem while trying to use WTForms with this pattern:

```python
choices = [(col, col) for col in await db.table_columns(table)]

class ConfigForm(Form):
    template = TextAreaField(""Template"")
    api_token = PasswordField(""OpenAI API token"")
    columns = MultiCheckboxField('Columns', choices=choices)
```",107914493
issues,1978023780,I_kwDOBm6k_c515j9k,request.post_vars() method obliterates form keys with multiple values,9599,2023-11-05T23:25:08Z,"https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/asgi.py#L137-L139

In GET requests you can do `?foo=1&foo=2` - you can do the same in POST requests, but the `dict()` call here eliminates those duplicates.

You can't even try calling `post_body()` and implement your own custom parsing because of:
- #2204",107914493
issues,1978022687,I_kwDOBm6k_c515jsf,request.post_body() can only be called once,9599,2023-11-05T23:22:03Z,"This code here:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/asgi.py#L127-L135

It consumes the messages, which means if you try to call it a second time you won't be able to get at the body.

This is efficient - we don't end up with a `request` object property with potentially megabytes of content that we never look at again - but it's inconvenient for cases like middleware or functions where we don't know if the body has been consumed yet or not.

Potential solution: set `request._body` the first time it is called, and return that on subsequent calls.

Potential optimization: only do this for bodies that are shorter than a certain threshold - maybe 1MB - and raise an exception if you attempt to call `post_body()` multiple times against one of those larger bodies.

I'm a bit nervous about that option though, since it could result in errors that don't show up in testing but do show up in production.",107914493
issue_comments,1793787454,IC_kwDOBm6k_c5q6wY-,,45269373,2023-11-05T16:44:49Z,"thanks for documenting this @bendnorman! got stuck at exactly the same point `gcloud builds submit ... returned non-zero exit status 1`, without a clue why this was happening. i now managed to get the github action to deploy datasette by assigning the following roles to the service account: `roles/run.admin`, `roles/storage.admin`, `roles/cloudbuild.builds.builder`, `roles/viewer`, `roles/iam.serviceAccountUser`.",107914493
issues,1977726056,I_kwDOBm6k_c514bRo,custom plugin not seen as sql function,7113541,2023-11-05T10:30:19Z,"Hi, I'm not sure if this is the right repo for this issue.

I'm using datasette with the parquet (to read a duckdb), and jellyfish plugins. Both work perfectly.

Now I need to create a simple plugin that uses the python rouge package and returns a similarity score (similarly to how the jellyfish plugin works).
If I create a custom plugin, even the example hello_world one, copied directly from the tutorial, I get the following error:
```duckdb.duckdb.CatalogException: Catalog Error: Scalar Function with name hello_world does not exist!```

Since the jellyfish plugin doesn't do anything more complex, I'm wondering if there is some other kind of issue with my setup.",107914493
issues,1977155641,I_kwDOCGYnMM512QA5,Move plugin directory into documentation,9599,2023-11-04T04:07:52Z,"https://github.com/simonw/sqlite-utils-plugins should be in the official documentation.

I can use the same pattern as https://llm.datasette.io/en/stable/plugins/directory.html

https://til.simonwillison.net/readthedocs/stable-docs",140912432
issue_comments,1793278279,IC_kwDOCGYnMM5q40FH,,9599,2023-11-04T00:58:03Z,I'm going to abandon this PR and ship the 3.12 testing change directly to `main`.,140912432
issue_comments,1793274869,IC_kwDOCGYnMM5q4zP1,,9599,2023-11-04T00:47:55Z,Thanks!,140912432
issue_comments,1793274485,IC_kwDOCGYnMM5q4zJ1,,9599,2023-11-04T00:46:55Z,"Manually tested. Before:

![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/3bdd30ea-1a54-4fec-b42d-793130a17bc1)

After:

![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/015d4e4e-b40c-4a93-81f5-1a4adef69b11)
",140912432
issue_comments,1793274350,IC_kwDOCGYnMM5q4zHu,,9599,2023-11-04T00:46:30Z,"And a GIF of the fix after applying:
- #598

![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/35829aec-c9ac-4925-a8e6-ffe7c2ab0d96)
",140912432
issue_comments,1793273968,IC_kwDOCGYnMM5q4zBw,,9599,2023-11-04T00:45:19Z,"Here's an animated GIF that demonstrates the bug:

![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/18e54b38-a466-4384-9162-114eb01526a2)
",140912432
issue_comments,1793272429,IC_kwDOCGYnMM5q4ypt,,9599,2023-11-04T00:40:34Z,Thanks!,140912432
issue_comments,1793269219,IC_kwDOCGYnMM5q4x3j,,9599,2023-11-04T00:34:33Z,"The GIS tests now pass in that container too:
```bash
pytest tests/test_gis.py
```
```
======================== test session starts =========================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/sqlite-utils
plugins: hypothesis-6.88.1
collected 12 items                                                   

tests/test_gis.py ............                                 [100%]

========================= 12 passed in 0.48s =========================
```",140912432
issue_comments,1793268750,IC_kwDOCGYnMM5q4xwO,,9599,2023-11-04T00:33:25Z,"See details of how I tested this here:
- https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126

Short version: having applied this fix, the following command (on simulated `aarch64`):
```bash
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Outputs:
```json
[{""spatialite_version()"": ""5.0.1""}]
```",140912432
issue_comments,1793268126,IC_kwDOCGYnMM5q4xme,,9599,2023-11-04T00:31:34Z,"Testing this manually on macOS using Docker Desk top like this:
```bash
docker run -it --rm arm64v8/ubuntu /bin/bash
```
Then inside the container:
```bash
uname -m
```
Outputs: `aarch64`

Then:
```bash
apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y
cd /tmp
git clone https://github.com/simonw/sqlite-utils
cd sqlite-utils
python3 -m venv venv
source venv/bin/activate
pip install -e '.[test]'
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Which output:
```
Traceback (most recent call last):
  File ""/tmp/sqlite-utils/venv/bin/sqlite-utils"", line 33, in <module>
    sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1157, in __call__
    return self.main(*args, **kwargs)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1078, in main
    rv = self.invoke(ctx)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 783, in invoke
    return __callback(*args, **kwargs)
  File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 1959, in memory
    _load_extensions(db, load_extension)
  File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 3232, in _load_extensions
    if "":"" in ext:
TypeError: argument of type 'NoneType' is not iterable
```
Then I ran this:
```bash
git checkout -b MikeCoats-spatialite-paths-linux-arm main
git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm
```
And now:
```bash
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Outputs:
```json
[{""spatialite_version()"": ""5.0.1""}]
```",140912432
issue_comments,1793265952,IC_kwDOCGYnMM5q4xEg,,9599,2023-11-04T00:25:34Z,The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.,140912432
issue_comments,1793264654,IC_kwDOCGYnMM5q4wwO,,22429695,2023-11-04T00:22:07Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`622c3a5`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/622c3a5a7dd53a09c029e2af40c2643fe7579340?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b1a6076`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #600   +/-   ##
=======================================
  Coverage   95.77%   95.77%           
=======================================
  Files           8        8           
  Lines        2840     2840           
=======================================
  Hits         2720     2720           
  Misses        120      120           
```


| [Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <ø> (ø)` | |
| [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <ø> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issue_comments,1793263638,IC_kwDOCGYnMM5q4wgW,,9599,2023-11-04T00:19:58Z,Thanks for this!,140912432
issues,1977004379,PR_kwDOCGYnMM5elFZf,Add spatialite arm64 linux path,37802088,2023-11-03T22:23:26Z,"According to both [Debian](https://packages.debian.org/bookworm/arm64/libsqlite3-mod-spatialite/filelist) and [Ubuntu](https://packages.ubuntu.com/mantic/arm64/libsqlite3-mod-spatialite/filelist), the correct “target triple” for arm64 is `aarch64-linux-gnu`, so we should be looking in `/usr/lib/aarch64-linux-gnu` for `mod_spatialite.so`.

I can confirm that on both of my Debian arm64 SBCs, `libsqlite3-mod-spatialite` installs to that path.

```
$ ls -l /usr/lib/*/*spatial*
lrwxrwxrwx 1 root root      23 Dec  1  2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0
lrwxrwxrwx 1 root root      23 Dec  1  2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0
-rw-r--r-- 1 root root 7348584 Dec  1  2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0
```

This is a set of before and after snippets of pytest’s output for this PR.

### Before

```
$ pytest
tests/test_get.py ......                                                 [ 73%]
tests/test_gis.py ssssssssssss                                           [ 75%]
tests/test_hypothesis.py ....                                            [ 75%]
```

### After

```
$ pytest
tests/test_get.py ......                                                 [ 73%]
tests/test_gis.py ............                                           [ 75%]
tests/test_hypothesis.py ....                                            [ 75%]
```


Issue: #599

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--600.org.readthedocs.build/en/600/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issues,1976986318,I_kwDOCGYnMM511mrO,Cannot find spatialite on arm64 linux,37802088,2023-11-03T22:05:51Z,"Initially, I found an issue in `datasette` where it wouldn’t find `spatialite` when running on my Radxa Rock 5B - an RK3588 powered SBC, running the arm64 build of Debian Bullseye. I confirmed the same behaviour on my Raspberry Pi 4 - a BCM2711 powered SBC, running the arm64 build of Debian Bookworm.

```
$ datasette --load-extension=spatialite example.db
Error: Could not find SpatiaLite extension
```

I did some digging and realised the issue originates in this project. Even with the `libsqlite3-mod-spatialite` package installed, `pytest` skips all of the GIS tests in the project.

```
$ apt list --installed | grep spatial
[…]
libsqlite3-mod-spatialite/stable,now 5.0.1-3 arm64 [installed]

$ ls -l /usr/lib/*/*spatial*
lrwxrwxrwx 1 root root      23 Dec  1  2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so -> mod_spatialite.so.7.1.0
lrwxrwxrwx 1 root root      23 Dec  1  2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7 -> mod_spatialite.so.7.1.0
-rw-r--r-- 1 root root 7348584 Dec  1  2022 /usr/lib/aarch64-linux-gnu/mod_spatialite.so.7.1.0
```

```
$ pytest
tests/test_get.py ......                                                 [ 73%]
tests/test_gis.py ssssssssssss                                           [ 75%]
tests/test_hypothesis.py ....                                            [ 75%]
```

I tracked the issue down to the [`find_sqlite()` function in the `utils.py`](https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/utils.py#L60) file. The [`SPATIALITE_PATHS`](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L34-L39) array doesn’t have an entry for the location of this module on arm64 linux.
",140912432
issue_comments,1791911093,IC_kwDOBm6k_c5qzmS1,,9599,2023-11-03T05:28:09Z,"Datasette is using that now, see:
- #1893",107914493
issue_comments,1791571572,IC_kwDOBm6k_c5qyTZ0,,498744,2023-11-02T21:36:24Z,"FWIW, code mirror 6 now has this standard although if you want table-specific suggestions, you'd have to handle parsing out which table the user is querying yourself.",107914493
issue_comments,1779267468,IC_kwDOBm6k_c5qDXeM,,363004,2023-10-25T13:23:04Z,"Using the [Counties example](https://us-counties.datasette.io/counties/county_for_latitude_longitude?longitude=-122&latitude=37), I was able to pull out the MakePoint method as 

MakePoint(cast(rm_rnb_history_pres.rx_lng as float), cast(rm_rnb_history_pres.rx_lat as float)) as geometry

which worked, giving me a geometry column.

![image](https://github.com/simonw/datasette/assets/363004/6393b712-9e3d-416d-ba37-202934d5f604)


gave

![image](https://github.com/simonw/datasette/assets/363004/219db7b2-8107-41b3-a049-ef4d6bd7ac7a)

I believe it's the cast to float that does the trick. Prior to using the cast, I also received a 'wrong number of arguments' eror.
",107914493
issue_comments,1777247375,IC_kwDOBm6k_c5p7qSP,,22429695,2023-10-24T13:49:27Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`be4d0f0`)](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2202   +/-   ##
=======================================
  Coverage   92.69%   92.69%           
=======================================
  Files          40       40           
  Lines        6047     6047           
=======================================
  Hits         5605     5605           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1777228352,IC_kwDOBm6k_c5p7lpA,,49699333,2023-10-24T13:40:25Z,Superseded by #2202.,107914493
issues,1959278971,PR_kwDOBm6k_c5dpF-F,Bump the python-packages group with 1 update,49699333,2023-10-24T13:40:21Z,"Bumps the python-packages group with 1 update: [black](https://github.com/psf/black).

<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>23.10.1</h2>
<h3>Highlights</h3>
<ul>
<li>Maintanence release to get a fix out for GitHub Action edge case (<a href=""https://redirect.github.com/psf/black/issues/3957"">#3957</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Fix merging implicit multiline strings that have inline comments (<a href=""https://redirect.github.com/psf/black/issues/3956"">#3956</a>)</li>
<li>Allow empty first line after block open before a comment or compound statement (<a href=""https://redirect.github.com/psf/black/issues/3967"">#3967</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Change Dockerfile to hatch + compile black (<a href=""https://redirect.github.com/psf/black/issues/3965"">#3965</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The summary output for GitHub workflows is now suppressible using the <code>summary</code>
parameter. (<a href=""https://redirect.github.com/psf/black/issues/3958"">#3958</a>)</li>
<li>Fix the action failing when Black check doesn't pass (<a href=""https://redirect.github.com/psf/black/issues/3957"">#3957</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>It is known Windows documentation CI is broken
<a href=""https://redirect.github.com/psf/black/issues/3968"">psf/black#3968</a></li>
</ul>
<h2>23.10.0</h2>
<h3>Stable style</h3>
<ul>
<li>Fix comments getting removed from inside parenthesized strings (<a href=""https://redirect.github.com/psf/black/issues/3909"">#3909</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Fix long lines with power operators getting split before the line length (<a href=""https://redirect.github.com/psf/black/issues/3942"">#3942</a>)</li>
<li>Long type hints are now wrapped in parentheses and properly indented when split across
multiple lines (<a href=""https://redirect.github.com/psf/black/issues/3899"">#3899</a>)</li>
<li>Magic trailing commas are now respected in return types. (<a href=""https://redirect.github.com/psf/black/issues/3916"">#3916</a>)</li>
<li>Require one empty line after module-level docstrings. (<a href=""https://redirect.github.com/psf/black/issues/3932"">#3932</a>)</li>
<li>Treat raw triple-quoted strings as docstrings (<a href=""https://redirect.github.com/psf/black/issues/3947"">#3947</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Fix cache versioning logic when <code>BLACK_CACHE_DIR</code> is set (<a href=""https://redirect.github.com/psf/black/issues/3937"">#3937</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Fix bug where attributes named <code>type</code> were not acccepted inside <code>match</code> statements
(<a href=""https://redirect.github.com/psf/black/issues/3950"">#3950</a>)</li>
<li>Add support for PEP 695 type aliases containing lambdas and other unusual expressions
(<a href=""https://redirect.github.com/psf/black/issues/3949"">#3949</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>23.10.1</h2>
<h3>Highlights</h3>
<ul>
<li>Maintanence release to get a fix out for GitHub Action edge case (<a href=""https://redirect.github.com/psf/black/issues/3957"">#3957</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Fix merging implicit multiline strings that have inline comments (<a href=""https://redirect.github.com/psf/black/issues/3956"">#3956</a>)</li>
<li>Allow empty first line after block open before a comment or compound statement (<a href=""https://redirect.github.com/psf/black/issues/3967"">#3967</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Change Dockerfile to hatch + compile black (<a href=""https://redirect.github.com/psf/black/issues/3965"">#3965</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The summary output for GitHub workflows is now suppressible using the <code>summary</code>
parameter. (<a href=""https://redirect.github.com/psf/black/issues/3958"">#3958</a>)</li>
<li>Fix the action failing when Black check doesn't pass (<a href=""https://redirect.github.com/psf/black/issues/3957"">#3957</a>)</li>
</ul>
<h3>Documentation</h3>
<ul>
<li>It is known Windows documentation CI is broken
<a href=""https://redirect.github.com/psf/black/issues/3968"">psf/black#3968</a></li>
</ul>
<h2>23.10.0</h2>
<h3>Stable style</h3>
<ul>
<li>Fix comments getting removed from inside parenthesized strings (<a href=""https://redirect.github.com/psf/black/issues/3909"">#3909</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Fix long lines with power operators getting split before the line length (<a href=""https://redirect.github.com/psf/black/issues/3942"">#3942</a>)</li>
<li>Long type hints are now wrapped in parentheses and properly indented when split across
multiple lines (<a href=""https://redirect.github.com/psf/black/issues/3899"">#3899</a>)</li>
<li>Magic trailing commas are now respected in return types. (<a href=""https://redirect.github.com/psf/black/issues/3916"">#3916</a>)</li>
<li>Require one empty line after module-level docstrings. (<a href=""https://redirect.github.com/psf/black/issues/3932"">#3932</a>)</li>
<li>Treat raw triple-quoted strings as docstrings (<a href=""https://redirect.github.com/psf/black/issues/3947"">#3947</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Fix cache versioning logic when <code>BLACK_CACHE_DIR</code> is set (<a href=""https://redirect.github.com/psf/black/issues/3937"">#3937</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Fix bug where attributes named <code>type</code> were not accepted inside <code>match</code> statements
(<a href=""https://redirect.github.com/psf/black/issues/3950"">#3950</a>)</li>
<li>Add support for PEP 695 type aliases containing lambdas and other unusual expressions</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/psf/black/commit/744d23b34800c06e10272149b70752396e90eeb8""><code>744d23b</code></a> Prepare release 23.10.1 (<a href=""https://redirect.github.com/psf/black/issues/3969"">#3969</a>)</li>
<li><a href=""https://github.com/psf/black/commit/8de4be516879302afce542ac80a6a43ced807759""><code>8de4be5</code></a> Fix CI failing (<a href=""https://redirect.github.com/psf/black/issues/3957"">#3957</a>)</li>
<li><a href=""https://github.com/psf/black/commit/c0adca321dc0d97a704de8ed0353e5b894a6a912""><code>c0adca3</code></a> docs: specifies the use of the .git-blame-ignore-revs file (<a href=""https://redirect.github.com/psf/black/issues/3961"">#3961</a>)</li>
<li><a href=""https://github.com/psf/black/commit/a7643fac8d97c15640a2b1a79f68b3dc771aebfb""><code>a7643fa</code></a> Add summary parameter to action (<a href=""https://redirect.github.com/psf/black/issues/3958"">#3958</a>)</li>
<li><a href=""https://github.com/psf/black/commit/d291c2338c3c1feee4f3f26985c0964ec1b7eb9f""><code>d291c23</code></a> Move Docker image to hatch + compile (<a href=""https://redirect.github.com/psf/black/issues/3965"">#3965</a>)</li>
<li><a href=""https://github.com/psf/black/commit/7f1c578b89b2c256a6ce3b70fc1b970b3ffa3349""><code>7f1c578</code></a> Bump peter-evans/create-or-update-comment from 3.0.2 to 3.1.0 (<a href=""https://redirect.github.com/psf/black/issues/3966"">#3966</a>)</li>
<li><a href=""https://github.com/psf/black/commit/2db5ab0a7b3b321e4cec70964239ee88087cd810""><code>2db5ab0</code></a> Allow empty line after block open before a comment or compound statement (<a href=""https://redirect.github.com/psf/black/issues/3967"">#3967</a>)</li>
<li><a href=""https://github.com/psf/black/commit/0a37888e79059018eef9293a724b14da59d3377a""><code>0a37888</code></a> Fix typos in CHANGES.md (<a href=""https://redirect.github.com/psf/black/issues/3963"">#3963</a>)</li>
<li><a href=""https://github.com/psf/black/commit/882d8795c6ff65c02f2657e596391748d1b6b7f5""><code>882d879</code></a> Fix merging implicit multiline strings that have inline comments (<a href=""https://redirect.github.com/psf/black/issues/3956"">#3956</a>)</li>
<li><a href=""https://github.com/psf/black/commit/9edba85f71d50d12996ef7bda576426362016171""><code>9edba85</code></a> Prepare release 23.10.0 (<a href=""https://redirect.github.com/psf/black/issues/3951"">#3951</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/psf/black/compare/23.9.1...23.10.1"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=23.9.1&new-version=23.10.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2202.org.readthedocs.build/en/2202/

<!-- readthedocs-preview datasette end -->",107914493
issues,1955676270,I_kwDOBm6k_c50kUBu,Discord invite link is invalid,11708906,2023-10-21T21:50:05Z,"https://datasette.io/discord leads to https://discord.com/invite/ktd74dm5mw and returns the following:

<img width=""448"" alt=""CleanShot 2023-10-21 at 14 49 45@2x"" src=""https://github.com/simonw/datasette/assets/11708906/a126cbf6-0a4d-41b8-9986-7567d40d453f"">
",107914493
issues,1949756141,PR_kwDOBm6k_c5dJF8z,Bump the python-packages group with 1 update,49699333,2023-10-18T13:25:55Z,"Bumps the python-packages group with 1 update: [black](https://github.com/psf/black).

<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>23.10.0</h2>
<h3>Stable style</h3>
<ul>
<li>Fix comments getting removed from inside parenthesized strings (<a href=""https://redirect.github.com/psf/black/issues/3909"">#3909</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Fix long lines with power operators getting split before the line length (<a href=""https://redirect.github.com/psf/black/issues/3942"">#3942</a>)</li>
<li>Long type hints are now wrapped in parentheses and properly indented when split across
multiple lines (<a href=""https://redirect.github.com/psf/black/issues/3899"">#3899</a>)</li>
<li>Magic trailing commas are now respected in return types. (<a href=""https://redirect.github.com/psf/black/issues/3916"">#3916</a>)</li>
<li>Require one empty line after module-level docstrings. (<a href=""https://redirect.github.com/psf/black/issues/3932"">#3932</a>)</li>
<li>Treat raw triple-quoted strings as docstrings (<a href=""https://redirect.github.com/psf/black/issues/3947"">#3947</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Fix cache versioning logic when <code>BLACK_CACHE_DIR</code> is set (<a href=""https://redirect.github.com/psf/black/issues/3937"">#3937</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Fix bug where attributes named <code>type</code> were not acccepted inside <code>match</code> statements
(<a href=""https://redirect.github.com/psf/black/issues/3950"">#3950</a>)</li>
<li>Add support for PEP 695 type aliases containing lambdas and other unusual expressions
(<a href=""https://redirect.github.com/psf/black/issues/3949"">#3949</a>)</li>
</ul>
<h3>Output</h3>
<ul>
<li>Black no longer attempts to provide special errors for attempting to format Python 2
code (<a href=""https://redirect.github.com/psf/black/issues/3933"">#3933</a>)</li>
<li>Black will more consistently print stacktraces on internal errors in verbose mode
(<a href=""https://redirect.github.com/psf/black/issues/3938"">#3938</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The action output displayed in the job summary is now wrapped in Markdown (<a href=""https://redirect.github.com/psf/black/issues/3914"">#3914</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>23.10.0</h2>
<h3>Stable style</h3>
<ul>
<li>Fix comments getting removed from inside parenthesized strings (<a href=""https://redirect.github.com/psf/black/issues/3909"">#3909</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Fix long lines with power operators getting split before the line length (<a href=""https://redirect.github.com/psf/black/issues/3942"">#3942</a>)</li>
<li>Long type hints are now wrapped in parentheses and properly indented when split across
multiple lines (<a href=""https://redirect.github.com/psf/black/issues/3899"">#3899</a>)</li>
<li>Magic trailing commas are now respected in return types. (<a href=""https://redirect.github.com/psf/black/issues/3916"">#3916</a>)</li>
<li>Require one empty line after module-level docstrings. (<a href=""https://redirect.github.com/psf/black/issues/3932"">#3932</a>)</li>
<li>Treat raw triple-quoted strings as docstrings (<a href=""https://redirect.github.com/psf/black/issues/3947"">#3947</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Fix cache versioning logic when <code>BLACK_CACHE_DIR</code> is set (<a href=""https://redirect.github.com/psf/black/issues/3937"">#3937</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Fix bug where attributes named <code>type</code> were not acccepted inside <code>match</code> statements
(<a href=""https://redirect.github.com/psf/black/issues/3950"">#3950</a>)</li>
<li>Add support for PEP 695 type aliases containing lambdas and other unusual expressions
(<a href=""https://redirect.github.com/psf/black/issues/3949"">#3949</a>)</li>
</ul>
<h3>Output</h3>
<ul>
<li>Black no longer attempts to provide special errors for attempting to format Python 2
code (<a href=""https://redirect.github.com/psf/black/issues/3933"">#3933</a>)</li>
<li>Black will more consistently print stacktraces on internal errors in verbose mode
(<a href=""https://redirect.github.com/psf/black/issues/3938"">#3938</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>The action output displayed in the job summary is now wrapped in Markdown (<a href=""https://redirect.github.com/psf/black/issues/3914"">#3914</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/psf/black/commit/9edba85f71d50d12996ef7bda576426362016171""><code>9edba85</code></a> Prepare release 23.10.0 (<a href=""https://redirect.github.com/psf/black/issues/3951"">#3951</a>)</li>
<li><a href=""https://github.com/psf/black/commit/bb588073ab286a9f1f8d839ab2cebe13011dd22c""><code>bb58807</code></a> Fix parser bug where &quot;type&quot; was misinterpreted as a keyword inside a match (#...</li>
<li><a href=""https://github.com/psf/black/commit/722735d20ebdc66c0da0e0df7658293455694500""><code>722735d</code></a> Fix grammar for type alias support (<a href=""https://redirect.github.com/psf/black/issues/3949"">#3949</a>)</li>
<li><a href=""https://github.com/psf/black/commit/abe57e3d92727f1b26c717fad3978159b987fe79""><code>abe57e3</code></a> Treat raw strings like other docstrings (<a href=""https://redirect.github.com/psf/black/issues/3947"">#3947</a>)</li>
<li><a href=""https://github.com/psf/black/commit/1648ac51806d092c95cb9bb2e4a5bffda6095bc1""><code>1648ac5</code></a> Fix long lines with power operator(s) getting splitted before line length (<a href=""https://redirect.github.com/psf/black/issues/3"">#3</a>...</li>
<li><a href=""https://github.com/psf/black/commit/6f84f652850dca8a1b578581e2fbb2cb95e791cc""><code>6f84f65</code></a> Migrate mypy config to pyproject.toml (<a href=""https://redirect.github.com/psf/black/issues/3936"">#3936</a>)</li>
<li><a href=""https://github.com/psf/black/commit/3bb92146f59804a6ead47d5f2d0fdc47daa6b698""><code>3bb9214</code></a> CI Test: Deprecating 'Healthcheck.all()' from Hypothesis in fuzz.py (<a href=""https://redirect.github.com/psf/black/issues/3945"">#3945</a>)</li>
<li><a href=""https://github.com/psf/black/commit/935f303a0a7b794e722c7df00c906be285884874""><code>935f303</code></a> Fix test that was not being run (<a href=""https://redirect.github.com/psf/black/issues/3939"">#3939</a>)</li>
<li><a href=""https://github.com/psf/black/commit/b7717c3f1e73d6b847e2534a2cebbb657b96caf8""><code>b7717c3</code></a> Standardise newlines after module-level docstrings (<a href=""https://redirect.github.com/psf/black/issues/3932"">#3932</a>)</li>
<li><a href=""https://github.com/psf/black/commit/7aa37ea0adf864baf3ef3dfbcfaf5ff1ff780250""><code>7aa37ea</code></a> Report all stacktraces in verbose mode (<a href=""https://redirect.github.com/psf/black/issues/3938"">#3938</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/psf/black/compare/23.9.1...23.10.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=23.9.1&new-version=23.10.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2200.org.readthedocs.build/en/2200/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1767248394,IC_kwDOBm6k_c5pVhIK,,6262071,2023-10-17T21:53:17Z,"@fgregg, I am happy to do that and just could not find a way to create issues at your fork repo. ",107914493
issue_comments,1767219901,IC_kwDOBm6k_c5pVaK9,,536941,2023-10-17T21:29:03Z,@yejiyang why don’t you move this discussion to my fork to spare simon’s notifications ,107914493
issue_comments,1767133832,IC_kwDOBm6k_c5pVFKI,,6262071,2023-10-17T20:37:18Z,"@fgregg Thanks for your reply. I tried to use your fork branch `datasette = {url = ""https://github.com/fgregg/datasette/archive/refs/heads/no_limit_csv_publish.zip""}` and got error - TypeError: 'str' object is not callable. I used the same templates as in your branch [here ](https://github.com/labordata/warehouse/tree/main/templates).

```
INFO:     127.0.0.1:47232 - ""GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1"" 200 OK
Traceback (most recent call last):
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py"", line 1632, in route_path
    response = await view(request, send)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py"", line 1814, in async_view_fn
    response = await async_call_with_supported_arguments(
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/utils/__init__.py"", line 1016, in async_call_with_supported_arguments
    return await fn(*call_with)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py"", line 673, in table_view
    response = await table_view_traced(datasette, request)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py"", line 822, in table_view_traced
    await datasette.render_template(
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py"", line 1307, in render_template
    return await template.render_async(template_context)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py"", line 1324, in render_async
    return self.environment.handle_exception()
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py"", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py"", line 1321, in <listcomp>
    [n async for n in self.root_render_func(ctx)]  # type: ignore
  File ""templates/table.html"", line 1, in top-level template code
    {% extends ""base.html"" %}
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/templates/base.html"", line 62, in top-level template code
    {% block content %}
  File ""templates/table.html"", line 24, in block 'content'
    <div class=""page-header"" style=""border-color: #{{ database_color(database) }}"">
TypeError: 'str' object is not callable
INFO:     127.0.0.1:47228 - ""GET /zeropm-v0-0-3/api_ready_query HTTP/1.1"" 500 Internal Server Error
```
",107914493
issue_comments,1766994810,IC_kwDOBm6k_c5pUjN6,,536941,2023-10-17T19:01:59Z,"hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish
",107914493
issues,1943259395,I_kwDOEhK-wc5z08kD, time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ',3746270,2023-10-14T13:24:39Z,"
```
evernote-to-sqlite enex evernote.db ./我的笔记.enex
Importing from ENEX  [#####-------------------------------]   14%
Traceback (most recent call last):
  File ""/usr/local/bin/evernote-to-sqlite"", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/click/core.py"", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/cli.py"", line 31, in enex
    save_note(db, note)
  File ""/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py"", line 46, in save_note
    ""created"": convert_datetime(created),
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/lib/python3.11/site-packages/evernote_to_sqlite/utils.py"", line 111, in convert_datetime
    return datetime.datetime.strptime(s, ""%Y%m%dT%H%M%SZ"").isoformat()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py"", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/usr/local/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/_strptime.py"", line 349, in _strptime
    raise ValueError(""time data %r does not match format %r"" %
ValueError: time data '2014-11-21T11:44:12.000Z' does not match format '%Y%m%dT%H%M%SZ'
```

enex is exported by evernote mac client ",303218369
issue_comments,1761630595,IC_kwDOBm6k_c5pAFmD,,6262071,2023-10-13T14:37:48Z,"Hi @fgregg, I came across this issue and found your setup at labordata.bunkum.us can help me with a research project at https://database.zeropm.eu/. 
I really like the approach [here](https://labordata.bunkum.us/f7-06c761c?sql=select+*+from+f7) when dealing with a custom SQL query returning more than 1000 rows:
1) At the table in HTML page, only first 1000 rows displayed;
2) When click the ""Download this data as a CSV Spreadsheet(All Rows)"" button, a csv with ALL ROWS (could be > 100 Mb) get downloaded.

I am trying to repeat the setup but have yet to be successful so far. 
What I tried:
1) copy the query.html & table.html templates from this [github repo](https://github.com/labordata/warehouse/tree/main/templates) and use it my project
2) use the same datasette version 1.0a2.

Do you know what else I should try to set it correctly? I appreciate your help.

@simonw I would like to use this opportunity to thank you for developing & maintaining such an amazing project. 
I introduce your datasette to several projects in my institute. I am also interested in your cloud version.

",107914493
issue_comments,1760560526,IC_kwDOBm6k_c5o8AWO,,1892194,2023-10-13T00:07:07Z,That worked!,107914493
issue_comments,1760552652,IC_kwDOBm6k_c5o7-bM,,9599,2023-10-12T23:59:21Z,I'm landing this despite the cog failures. I'll fix them on main if I have to.,107914493
issue_comments,1760545012,IC_kwDOBm6k_c5o78j0,,9599,2023-10-12T23:48:16Z,Oh! I think I broke Cog on `main` and these tests are running against this branch rebased against main.,107914493
issue_comments,1760542865,IC_kwDOBm6k_c5o78CR,,9599,2023-10-12T23:44:53Z,"Weird, the `cog` check is failing in CI.

```
Run cog --check docs/*.rst
  cog --check docs/*.rst
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib
Check failed
Checking docs/authentication.rst
Checking docs/binary_data.rst
Checking docs/changelog.rst
Checking docs/cli-reference.rst
Checking docs/configuration.rst  (changed)
```",107914493
issue_comments,1760441535,IC_kwDOBm6k_c5o7jS_,,9599,2023-10-12T22:08:42Z,Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade,107914493
issue_comments,1760417555,IC_kwDOBm6k_c5o7dcT,,9599,2023-10-12T21:54:52Z,"I can't replicate this bug. Closing, but please re-open if it's still happening.

As an aside, the link I promote is https://datasette.io/discord which redirects:

```
curl -i 'https://datasette.io/discord'
HTTP/2 301 
location: https://discord.gg/ktd74dm5mw
content-type: text/plain
x-cloud-trace-context: 8dcfd08d3d1fa44f7ee78568e0f5305e;o=1
date: Thu, 12 Oct 2023 21:54:17 GMT
server: Google Frontend
content-length: 0
```
```
curl -i 'https://discord.gg/ktd74dm5mw'
HTTP/2 301 
date: Thu, 12 Oct 2023 21:54:28 GMT
content-type: text/plain;charset=UTF-8
content-length: 0
location: https://discord.com/invite/ktd74dm5mw
strict-transport-security: max-age=31536000; includeSubDomains; preload
permissions-policy: interest-cohort=()
x-content-type-options: nosniff
x-frame-options: DENY
x-robots-tag: noindex, nofollow, noarchive, nocache, noimageindex, noodp
x-xss-protection: 1; mode=block
report-to: {""endpoints"":[{""url"":""https:\/\/a.nel.cloudflare.com\/report\/v3?s=Dzzrf%2FgGkfFxtzSAQ46slMVDLcFjsH9fsvVkzHtgUUiZ891rXAa6LvTRpHK%2BdSMSQ54F57hS9z1mZXXklIbONZW1bfBuFjSK9J4XmjjLjsFUulMXvpjfCLkB6PI%3D""}],""group"":""cf-nel"",""max_age"":604800}
nel: {""success_fraction"":0,""report_to"":""cf-nel"",""max_age"":604800}
server: cloudflare
cf-ray: 815294ddff282511-SJC
```",107914493
issue_comments,1760413191,IC_kwDOBm6k_c5o7cYH,,9599,2023-10-12T21:52:25Z,"Demo of that logic:
```
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-1.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch metadata-new-1.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch datasette.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette-new.yaml
```",107914493
issue_comments,1760412424,IC_kwDOBm6k_c5o7cMI,,9599,2023-10-12T21:51:44Z,"Started playing with this plugin idea, now tearing myself away to work on something more important:
```python
from datasette import hookimpl
import click
import pathlib


@hookimpl
def register_commands(cli):
    @cli.group()
    def upgrade():
        """"""
        Apply configuration upgrades to an existing Datasette instance
        """"""
        pass


    @upgrade.command()
    @click.argument(
        ""metadata"", type=click.Path(exists=True)
    )
    @click.option(
        ""new_metadata"", ""-m"", ""--new-metadata"", help=""Path to new metadata.yaml file"", type=click.Path(exists=False)
    )
    @click.option(
        ""new_datasette"", ""-c"", ""--new-datasette"", help=""Path to new datasette.yaml file"", type=click.Path(exists=False)
    )
    @click.option(
        ""output_dir"", ""-e"", ""--output-dir"", help=""Directory to write new files to"", type=click.Path(), default="".""
    )
    def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
        """"""
        Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
        datasette.yaml split introduced prior to Datasette 1.0.
        """"""
        print(""Upgrading {} to new metadata.yaml format"".format(metadata))
        output_dir = pathlib.Path(output_dir)
        if not new_metadata:
            # Pick a filename for the new metadata.yaml file that does not yet exist
            new_metadata = pick_filename(""metadata"", output_dir)
        if not new_datasette:
            new_datasette = pick_filename(""datasette"", output_dir)
        print(""New metadata.yaml file will be written to {}"".format(new_metadata))
        print(""New datasette.yaml file will be written to {}"".format(new_datasette))


def pick_filename(base, output_dir):
    options = [""{}.yaml"".format(base), ""{}-new.yaml"".format(base)]
    i = 0
    while True:
        option = options.pop(0)
        option_path = output_dir / option
        if not option_path.exists():
            return option_path
        # If we ran out
        if not options:
            i += 1
            options = [""{}-new-{}.yaml"".format(base, i)]
```",107914493
issue_comments,1760411937,IC_kwDOBm6k_c5o7cEh,,9599,2023-10-12T21:51:16Z,"I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.

If people want to keep their comments I'm happy to leave them to copy those over by hand.",107914493
issue_comments,1760401731,IC_kwDOBm6k_c5o7ZlD,,15178711,2023-10-12T21:41:42Z,"I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins.

One thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting 

",107914493
issue_comments,1760396195,IC_kwDOBm6k_c5o7YOj,,9599,2023-10-12T21:36:25Z,"Related idea: how about a `datasette-upgrade` plugin which adds a `datasette upgrade` command that can be used to automate this process?

Maybe something like this:
```bash
datasette install datasette-upgrade
datasette upgrade metadata-to-config metadata.json
```
This would output two new files: `metadata.yaml` and `datasette.yaml`. If files with those names existed already in the current directory they would be called `metadata-new.yaml` and `datasette-new.yaml`.

The command would tell you what it did:
```
Your metadata.json file has been rewritten as two files:

    metadata-new.yaml
    datasette.yaml

Start Datasette like this to try them out:

    datasette -m metadata-new.yaml -c datasette.yaml
```
The command is `datasette upgrade metadata-to-config` because `metadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.",107914493
issue_comments,1759952247,IC_kwDOBm6k_c5o5r13,,9599,2023-10-12T16:23:10Z,"Some options for where this could go:
- Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides.
- On a new page? We could have a ""upgrade instructions"" page in the documentation.
- At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page

I'm leaning towards the third option at the moment.

But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.",107914493
issues,1940346034,I_kwDOBm6k_c5zp1Sy,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,9599,2023-10-12T16:21:25Z,"> `Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.`
>
> I think we should link directly to documentation that tells people how to perform this upgrade.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/2190#issuecomment-1759947021_
            ",107914493
issue_comments,1759948683,IC_kwDOBm6k_c5o5q-L,,9599,2023-10-12T16:20:41Z,I'm going to land this and open a new issue for the upgrade instructions.,107914493
issue_comments,1759947534,IC_kwDOBm6k_c5o5qsO,,9599,2023-10-12T16:19:59Z,It would be nice if we could catch that and turn that into a less intimidating Click exception too.,107914493
issue_comments,1759947021,IC_kwDOBm6k_c5o5qkN,,9599,2023-10-12T16:19:38Z,"

This looks good and works well. The error from this currently looks like:

```
datasette -m metadata.json -p 8844
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 33, in <module>
    sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')())
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 98, in wrapped
    return fn(*args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 546, in serve
    metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read()))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 1282, in fail_if_plugins_in_metadata
    raise Exception(
Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.
```
With wrapping:

`Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.`

I think we should link directly to documentation that tells people how to perform this upgrade.",107914493
issue_comments,1752096661,IC_kwDOBm6k_c5obt-V,,9599,2023-10-08T16:17:04Z,"https://lite.datasette.io/?install=datasette-packages#/-/packages confirms that Datasette Lite still works (`click-default-group-wheel` was originally built to allow it to run) and that it's now using these packages:
```
    ""click-default-group"": ""1.2.4"",
    ""datasette"": ""0.64.5"",
```
Full list:
```json
{
    ""aiofiles"": ""23.2.1"",
    ""anyio"": ""3.7.1"",
    ""asgi-csrf"": ""0.9"",
    ""asgiref"": ""3.7.2"",
    ""certifi"": ""2022.12.7"",
    ""click"": ""8.1.3"",
    ""click-default-group"": ""1.2.4"",
    ""datasette"": ""0.64.5"",
    ""datasette-packages"": ""0.2"",
    ""h11"": ""0.12.0"",
    ""httpcore"": ""0.15.0"",
    ""httpx"": ""0.23.0"",
    ""hupper"": ""1.12"",
    ""idna"": ""3.4"",
    ""itsdangerous"": ""2.1.2"",
    ""janus"": ""1.0.0"",
    ""Jinja2"": ""3.1.2"",
    ""MarkupSafe"": ""2.1.2"",
    ""mergedeep"": ""1.3.4"",
    ""micropip"": ""0.3.0"",
    ""packaging"": ""23.0"",
    ""Pint"": ""0.22"",
    ""pip"": ""23.2.1"",
    ""pluggy"": ""1.0.0"",
    ""pyparsing"": ""3.0.9"",
    ""python-multipart"": ""0.0.6"",
    ""PyYAML"": ""6.0"",
    ""rfc3986"": ""1.5.0"",
    ""setuptools"": ""67.6.1"",
    ""sniffio"": ""1.3.0"",
    ""typing-extensions"": ""4.5.0"",
    ""uvicorn"": ""0.23.2""
}
```",107914493
issue_comments,1752095961,IC_kwDOBm6k_c5obtzZ,,9599,2023-10-08T16:13:42Z,"Confirmed - I ran this in a fresh virtual environment:
```bash
pip install --no-cache datasette
```
And now:
```bash
pip freeze | grep click
```
```
click==8.1.7
click-default-group==1.2.4
```
```bash
datasette --version
```
```
datasette, version 0.64.5
```",107914493
issue_comments,1752093039,IC_kwDOBm6k_c5obtFv,,9599,2023-10-08T15:59:53Z,"Replicated this myself:
```bash
cd /tmp
mkdir dddd
cd dddd
pipenv shell
pip install datasette
pip freeze | grep click
```
```
click==8.1.7
click-default-group==1.2.4
click-default-group-wheel==1.2.3
```
Yeah this is bad, I'll ship a `0.64.5` release.",107914493
issues,1931794126,I_kwDOBm6k_c5zJNbO,--load-extension=spatialite not working with Windows,363004,2023-10-08T12:50:22Z,"Using each of
`python -m datasette counties.db -m metadata.yml --load-extension=SpatiaLite`

and 

`python -m datasette counties.db --load-extension=""C:\Windows\System32\mod_spatialite.dll""`

and

`python -m datasette counties.db --load-extension=C:\Windows\System32\mod_spatialite.dll`

I got the error:

```
  File ""C:\Users\m3n7es\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\datasette\database.py"", line 209, in in_thread
    self.ds._prepare_connection(conn, self.name)
  File ""C:\Users\m3n7es\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\datasette\app.py"", line 596, in _prepare_connection
    conn.execute(""SELECT load_extension(?, ?)"", [path, entrypoint])
sqlite3.OperationalError: The specified module could not be found.

```

I finally tried modifying the code in app.py to read:

```
    def _prepare_connection(self, conn, database):
        conn.row_factory = sqlite3.Row
        conn.text_factory = lambda x: str(x, ""utf-8"", ""replace"")
        if self.sqlite_extensions:
            conn.enable_load_extension(True)
            for extension in self.sqlite_extensions:
                # ""extension"" is either a string path to the extension
                # or a 2-item tuple that specifies which entrypoint to load.
                #if isinstance(extension, tuple):
                #    path, entrypoint = extension
                #    conn.execute(""SELECT load_extension(?, ?)"", [path, entrypoint])
                #else:
                conn.execute(""SELECT load_extension('C:\Windows\System32\mod_spatialite.dll')"")

```
At which point the counties example worked. 

Is there a correct way to install/use the extension on Windows? My method will cause issues if there's a second extension to be used.

On an unrelated note, my next step is to figure out how to write a query across the two loaded databases supplied from the command line:
`python -m datasette rm_toucans_23_10_07.db counties.db -m metadata.yml --load-extension=SpatiaLite`



",107914493
issues,1930008379,I_kwDOBm6k_c5zCZc7,click-default-group-wheel dependency conflict,1176293,2023-10-06T11:49:20Z,"I upgraded my dependencies, then ran into this problem running `datasette inspect`:

> env/lib/python3.9/site-packages/datasette/cli.py"", line 6, in <module>
>     from click_default_group import DefaultGroup
> ModuleNotFoundError: No module named 'click_default_group'

Turns out the released version of datasette still depends on `click-default-group-wheel`, so `click-default-group` doesn't get installed/recognized:

```
$ virtualenv venv
$ source venv/bin/activate
$ pip install datasette
$ pip list | grep click-default-group
click-default-group       1.2.4
click-default-group-wheel 1.2.3
$ python -c ""from click_default_group import DefaultGroup""
Traceback (most recent call last):
  File ""<string>"", line 1, in <module>
ModuleNotFoundError: No module named 'click_default_group'
$ pip install --force-reinstall click-default-group
...
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed.
This behaviour is the source of the following dependency conflicts.
datasette 0.64.4 requires click-default-group-wheel>=1.2.2, which is not installed.
Successfully installed click-8.1.7 click-default-group-1.2.4
```",107914493
issues,1926729132,PR_kwDOCGYnMM5b7Z_y,Fixed issue #433 - CLI eats cursor,62745,2023-10-04T18:06:58Z,"The issue is that underlying iterator is not fully consumed within the body of the `with file_progress()` block. Instead, that block creates generator expressions like `docs = (dict(zip(headers, row)) for row in reader)`

These iterables are consumed later, outside the `with file_progress()` block, which consumes the underlying iterator, and in turn updates the progress bar.

This means that the `ProgressBar.__exit__` method gets called before the last time the `ProgressBar.update` method gets called. The result is that the code to make the cursor invisible (inside the `update()` method) is called after the cleanup code to make it visible (in the `__exit__` method).

The fix is to move consumption of the `docs` iterators within the progress bar block. (

(An additional fix, to make ProgressBar more robust against this kind of misuse, would to make it refusing to update after its `__exit__` method had been called, just like files cannot be `read()` after they are closed. That requires a in the click library).

Note that Github diff obscures the simplicity of this diff, it's just indenting a block of code.


<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--598.org.readthedocs.build/en/598/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1747231893,IC_kwDOCGYnMM5oJKSV,,62745,2023-10-04T16:15:09Z,"I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:

```
  [#######-----------------------------]   20%
  [####################################]  100%%  
```
The bug is reproducible for me in any terminal, including Gnome Terminal and Guake, and VSCode. With VSCode I can use this launch.json to reproduce it:

```json

{
    ""version"": ""0.2.0"",
    ""configurations"": [
        {
            ""name"": ""Python: Module"",
            ""type"": ""python"",
            ""request"": ""launch"",
            ""module"": ""sqlite_utils"",
            ""justMyCode"": false,
            ""args"": [""insert"", ""test.db"", ""test"", ""--csv"", ""tests/sniff/example1.csv""]
        }
    ]
}
```

[edit - deleted my analysis of why the current code is wrong, which was confused and confusing]",140912432
issue_comments,1745568725,IC_kwDOBm6k_c5oC0PV,,2495794,2023-10-03T19:12:37Z,"Hello! Resurrecting this issue since we're running into something similar with data.catalyst.coop as our database files have ballooned up to several GB. Our Cloud Run revisions now require huge amounts of RAM to start up without receiving a SIGBUS.

I'd love to see this fix merged in. It sounds like we want to make the immutable/read-only mode decision more flexible before doing so, so that we can use `ro` in Docker and `immutable` outside. If that sounds right, I'm happy to take a crack at adding that as a command-line flag or something that gets set automatically based on the expected execution environment.",107914493
issues,1920416843,I_kwDOCGYnMM5ydzxL,sqlite-utils insert-files should be able to convert fields,1737541,2023-09-30T22:20:47Z,"Currently using both `insert-files` and `convert` is needed in order to create sqlar files, it would be more convenient if it could be done with just one command.

```shell
~
❯ cat test.py
import os

class Example:
    def __init__(self, arg1, arg2):
        self.arg1 = arg1

~
❯ sqlite-utils insert-files test.sqlar sqlar test.py -c name:name -c data:content -c mode:mode -c mtime:mtime -c sz:size --pk=name
  [####################################]  100%

~
❯ sqlite-utils convert test.sqlar sqlar data ""zlib.compress(value)"" --import=zlib --where ""name = 'test.py'""
[####################################]  100%

~
❯ cat test.py | sqlite-utils convert test.sqlar sqlar data ""zlib.compress(sys.stdin.buffer.read())"" --import=zlib --import=sys --where ""name = 'test.py'"" # Alternative way
  [####################################]  100%

~
❯ sqlite3 test.sqlar ""SELECT hex(data) FROM sqlar WHERE name = 'test.py';"" | python3 -c ""import sys, zlib; sys.stdout.buffer.write(zlib.decompress(bytes.fromhex(sys.stdin.read())))""
import os

class Example:
    def __init__(self, arg1, arg2):
        self.arg1 = arg1

~
❯ rm test.py

~
❯ sqlar -l test.sqlar
test.py

~
❯ sqlar -x test.sqlar

~
❯ cat test.py
import os

class Example:
    def __init__(self, arg1, arg2):
        self.arg1 = arg1

```",140912432
issues,1919296686,PR_kwDOCGYnMM5bifPC,"Fixes mapping for time fields related to mysql, closes #522",4420927,2023-09-29T13:41:48Z,"Adds `COLUMN_TYPE_MAPPING` for `TIME` fields that are mapped as `datetime.timedelta` for MySQL and json represantation for `datetime.timedelta` in order to fix #522

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--596.org.readthedocs.build/en/596/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1739816358,IC_kwDOBm6k_c5ns32m,,9599,2023-09-28T18:29:05Z,Datasette Cloud really wants this.,107914493
issue_comments,1737906995,IC_kwDOBm6k_c5nllsz,,79087,2023-09-27T18:44:02Z,@simonw Any chance we can get this tiny patch merged for an upcoming release?,107914493
issue_comments,1737363182,IC_kwDOBm6k_c5njg7u,,418191,2023-09-27T13:05:41Z,I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.,107914493
issue_comments,1733312349,IC_kwDOCGYnMM5nUD9d,,123451970,2023-09-25T09:38:13Z,"Never mind

When I created the connection using 
`sqlite_utils.Database(path)`

I just needed to add the following statement right after and it did the trick

`self.db.conn.execute(""PRAGMA foreign_keys = ON"")`

Hope this helps people in the future 👍 ",140912432
issues,1910269679,I_kwDOBm6k_c5x3Gbv,Discord invite link returns 401,1892194,2023-09-24T15:16:54Z,"I found the link to the datasette discord channel via [this query](https://github.com/search?q=repo%3Asimonw%2Fdatasette%20discord&type=code).

The following video should be self explanatory:

https://github.com/simonw/datasette/assets/1892194/8cd33e88-bcaa-41f3-9818-ab4d589c3f02


Link for reference: https://discord.com/invite/ktd74dm5mw",107914493
issue_comments,1732018273,IC_kwDOCGYnMM5nPIBh,,1108600,2023-09-22T20:49:51Z,This would be awesome to have for multi-gig tsv and csv files! I'm currently looking at a 10 hour countdown for one such important. Not a problem because I'm lazy and happy to let it run and check on it tomorrow..,140912432
issue_comments,1730458954,IC_kwDOBm6k_c5nJLVK,,9599,2023-09-21T22:57:39Z,"Worth noting that it already sets `--cors` automatically without you needing to specify it:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L374-L374

I wonder if that's actually surprising behaviour that we should change before 1.0.",107914493
issue_comments,1730457374,IC_kwDOBm6k_c5nJK8e,,9599,2023-09-21T22:56:18Z,Maybe I should add `--cors` and `--crossdb` to `datasette publish cloudrun` as well?,107914493
issue_comments,1730446937,IC_kwDOBm6k_c5nJIZZ,,9599,2023-09-21T22:46:42Z,"Found more when I [searched for YAML](https://github.com/search?q=datasette+publish+extra-options++language%3AYAML&type=code).

Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59

`--extra-options=""--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1""`

Uses both `--cors` and `--crossdb`.",107914493
issue_comments,1730441613,IC_kwDOBm6k_c5nJHGN,,9599,2023-09-21T22:42:12Z,"https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here:

```
--extra-options=""--setting sql_time_limit_ms 5000""
--extra-options=""--config default_cache_ttl:3600 --config hash_urls:1""
--extra-options ""--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1""
--extra-options=""--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000""
--extra-options=""--setting sql_time_limit_ms 5000""
--extra-options ""--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000""
```",107914493
issue_comments,1730438503,IC_kwDOBm6k_c5nJGVn,,9599,2023-09-21T22:38:10Z,"I'd really like to remove `--extra-options`. I think the new design makes that completely obsolete?

Maybe it doesn't. You still need `--extra-options` for the `--crossdb` option for example.",107914493
issue_comments,1730437934,IC_kwDOBm6k_c5nJGMu,,9599,2023-09-21T22:37:22Z,"Here's the full help for Cloud Run at the moment:
```bash
datasette publish cloudrun --help
```
```
Usage: datasette publish cloudrun [OPTIONS] [FILES]...

  Publish databases to Datasette running on Cloud Run

Options:
  -m, --metadata FILENAME         Path to JSON/YAML file containing metadata
                                  to publish
  --extra-options TEXT            Extra options to pass to datasette serve
  --branch TEXT                   Install datasette from a GitHub branch e.g.
                                  main
  --template-dir DIRECTORY        Path to directory containing custom
                                  templates
  --plugins-dir DIRECTORY         Path to directory containing custom plugins
  --static MOUNT:DIRECTORY        Serve static files from this directory at
                                  /MOUNT/...
  --install TEXT                  Additional packages (e.g. plugins) to
                                  install
  --plugin-secret <TEXT TEXT TEXT>...
                                  Secrets to pass to plugins, e.g. --plugin-
                                  secret datasette-auth-github client_id xxx
  --version-note TEXT             Additional note to show on /-/versions
  --secret TEXT                   Secret used for signing secure values, such
                                  as signed cookies
  --title TEXT                    Title for metadata
  --license TEXT                  License label for metadata
  --license_url TEXT              License URL for metadata
  --source TEXT                   Source label for metadata
  --source_url TEXT               Source URL for metadata
  --about TEXT                    About label for metadata
  --about_url TEXT                About URL for metadata
  -n, --name TEXT                 Application name to use when building
  --service TEXT                  Cloud Run service to deploy (or over-write)
  --spatialite                    Enable SpatialLite extension
  --show-files                    Output the generated Dockerfile and
                                  metadata.json
  --memory TEXT                   Memory to allocate in Cloud Run, e.g. 1Gi
  --cpu [1|2|4]                   Number of vCPUs to allocate in Cloud Run
  --timeout INTEGER               Build timeout in seconds
  --apt-get-install TEXT          Additional packages to apt-get install
  --max-instances INTEGER         Maximum Cloud Run instances
  --min-instances INTEGER         Minimum Cloud Run instances
  --help                          Show this message and exit.
```",107914493
issue_comments,1730437237,IC_kwDOBm6k_c5nJGB1,,9599,2023-09-21T22:36:22Z,"I think the actual design of this is pretty simple. Current help starts like this:

```
Usage: datasette publish cloudrun [OPTIONS] [FILES]...

  Publish databases to Datasette running on Cloud Run

Options:
  -m, --metadata FILENAME         Path to JSON/YAML file containing metadata
                                  to publish
  --extra-options TEXT            Extra options to pass to datasette serve
```
The `-s` and `-c` short options are not being used.

So I think `-c/--config` can point to a JSON or YAML `datasette.yaml` file, and `-s/--setting key value` can mirror the new `-s/--setting` option in `datasette serve` itself (a shortcut for populating the config file directly from the CLI).

Here's the relevant help section from `datasette serve`:
```
  -m, --metadata FILENAME         Path to JSON/YAML file containing
                                  license/source metadata
  -c, --config FILENAME           Path to JSON/YAML Datasette configuration
                                  file
  -s, --setting SETTING...        nested.key, value setting to use in
                                  Datasette configuration
```",107914493
issue_comments,1730388418,IC_kwDOBm6k_c5nI6HC,,9599,2023-09-21T22:26:19Z,1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21,107914493
issue_comments,1730363182,IC_kwDOBm6k_c5nIz8u,,9599,2023-09-21T22:09:10Z,Tests all pass now.,107914493
issue_comments,1730362441,IC_kwDOBm6k_c5nIzxJ,,9599,2023-09-21T22:08:19Z,"That worked

https://github.com/simonw/datasette/commit/e4f868801a6633400045f59584cfe650961c3fa6 is the latest commit right now and https://latest.datasette.io/-/versions shows that as the deployed version.",107914493
issue_comments,1730356422,IC_kwDOBm6k_c5nIyTG,,9599,2023-09-21T22:01:00Z,Tested that locally with Python 3.9 from `pyenv` and it worked.,107914493
issue_comments,1730353462,IC_kwDOBm6k_c5nIxk2,,9599,2023-09-21T21:57:17Z,"Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302
```
    plugin_info[""name""] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
Test failed: datasette-json-html should not have been loaded
```",107914493
issue_comments,1730353006,IC_kwDOBm6k_c5nIxdu,,9599,2023-09-21T21:56:43Z,"The test fails as expected now. Closing this issue, will solve the remaining problems in:
- #2057",107914493
issue_comments,1730352111,IC_kwDOBm6k_c5nIxPv,,9599,2023-09-21T21:55:41Z,"https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time.

```
    plugin_info[""name""] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
Test failed: datasette-json-html should not have been loaded
```",107914493
issue_comments,1730313565,IC_kwDOBm6k_c5nIn1d,,9599,2023-09-21T21:16:31Z,"The `@add_common_publish_arguments_and_options` decorator described here is bad. If I update it to support a new `config` option all plugins that use it will break.

https://github.com/simonw/datasette/blob/f130c7c0a88e50cea4121ea18d1f6db2431b6fab/docs/plugin_hooks.rst#L347-L355

I want to deprecate it and switch to a different, better design to address the same problem.",107914493
issue_comments,1730312128,IC_kwDOBm6k_c5nInfA,,9599,2023-09-21T21:15:11Z,"As soon as `datasette publish cloudrun` has this I can re-enable this bit of the demo deploy:

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://todomvc.datasette.io/",107914493
issue_comments,1730305920,IC_kwDOBm6k_c5nIl-A,,9599,2023-09-21T21:09:21Z,"I'm going to disable this bit of the deploy for the moment, which will break the demo linked to from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97
",107914493
issues,1907765514,I_kwDOBm6k_c5xtjEK,`datasette publish` needs support for the new config/metadata split,9599,2023-09-21T21:08:12Z,"> ... which raises the challenge that `datasette publish` doesn't yet know what to do with a config file!

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/2194#issuecomment-1730259871_
            ",107914493
issue_comments,1730259871,IC_kwDOBm6k_c5nIauf,,9599,2023-09-21T20:34:09Z,"... which raises the challenge that `datasette publish` doesn't yet know what to do with a config file!

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L114-L122",107914493
issue_comments,1730258302,IC_kwDOBm6k_c5nIaV-,,9599,2023-09-21T20:32:53Z,"Correct usage is now:
```bash
python tests/fixtures.py fixtures.db fixtures-config.json fixtures-metadata.json \
  plugins --extra-db-filename extra_database.db
```
```
Test tables written to fixtures.db
- metadata written to fixtures-metadata.json
- config written to fixtures-config.json
  Wrote plugin: plugins/register_output_renderer.py
  Wrote plugin: plugins/view_name.py
  Wrote plugin: plugins/my_plugin.py
  Wrote plugin: plugins/messages_output_renderer.py
  Wrote plugin: plugins/sleep_sql_function.py
  Wrote plugin: plugins/my_plugin_2.py
Test tables written to extra_database.db
```",107914493
issue_comments,1730256435,IC_kwDOBm6k_c5nIZ4z,,9599,2023-09-21T20:31:22Z,"New error: ""Error: Metadata should end with .json""

https://github.com/simonw/datasette/actions/runs/6266720924/job/17018265851

<img width=""678"" alt=""CleanShot 2023-09-21 at 13 31 08@2x"" src=""https://github.com/simonw/datasette/assets/9599/720726be-fcfc-49a4-8316-ee49d96ac86b"">",107914493
issue_comments,1730250337,IC_kwDOBm6k_c5nIYZh,,9599,2023-09-21T20:26:12Z,That does seem to fix the problem! ,107914493
issue_comments,1730247545,IC_kwDOBm6k_c5nIXt5,,9599,2023-09-21T20:23:47Z,Hunch: https://pypi.org/project/importlib-metadata/ may help here.,107914493
issue_comments,1730245204,IC_kwDOBm6k_c5nIXJU,,9599,2023-09-21T20:21:42Z,"I think I see the problem - it's from here: https://github.com/simonw/datasette/commit/b2ec8717c3619260a1b535eea20e618bf95aa30b#diff-5dbc88d6e5c3615caf10e32a9d6fc6ff683f5b5814948928cb84c3ab91c038b6L770

The `config` and `metadata` Click options are the wrong way round:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L785-L786

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L801",107914493
issue_comments,1730242734,IC_kwDOBm6k_c5nIWiu,,9599,2023-09-21T20:19:29Z,"Maybe `plugins/` does not exist? It should have been created by this line:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L41-L42",107914493
issue_comments,1730241813,IC_kwDOBm6k_c5nIWUV,,9599,2023-09-21T20:18:40Z,"This looks to be the step that is failing:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L50-L60",107914493
issues,1907695234,I_kwDOBm6k_c5xtR6C,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",9599,2023-09-21T20:17:49Z,"https://github.com/simonw/datasette/actions/runs/6266449018/job/17017460074

<img width=""826"" alt=""CleanShot 2023-09-21 at 13 17 12@2x"" src=""https://github.com/simonw/datasette/assets/9599/9f58f39b-4cca-455f-bc6d-ecab14e841a5"">

This is a bit of a mystery, I don't think I've changed anything recently that could have broken this.",107914493
issue_comments,1730232308,IC_kwDOBm6k_c5nIT_0,,9599,2023-09-21T20:11:16Z,"We're planning a breaking change in `1.0a7`:
- #2191 

Since that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as `1.0a8` instead.",107914493
issue_comments,1730231404,IC_kwDOBm6k_c5nITxs,,9599,2023-09-21T20:10:28Z,Release 0.64.4: https://docs.datasette.io/en/stable/changelog.html#v0-64-4,107914493
issue_comments,1730226107,IC_kwDOBm6k_c5nISe7,,9599,2023-09-21T20:06:19Z,"No that's not it actually, it's something else.

Got to this point:
```bash
DATASETTE_LOAD_PLUGINS=datasette-init python -i $(which datasette) plugins
```
That fails and drops me into a debugger:

```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 186, in plugins
    app = Datasette([], plugins_dir=plugins_dir)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 405, in __init__
    for plugin in get_plugins()
  File ""/Users/simon/Dropbox/Development/datasette/datasette/plugins.py"", line 89, in get_plugins
    plugin_info[""name""] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
```",107914493
issue_comments,1730219703,IC_kwDOBm6k_c5nIQ63,,9599,2023-09-21T20:01:54Z,"The problem is here:
```
 86  	        distinfo = plugin_to_distinfo.get(plugin)
 87  	        if distinfo is None:
 88  	            breakpoint()
 89  ->	            assert False
 90  	        if distinfo.name is None:
 91  	            breakpoint()
 92  	            assert False
 93  	        if distinfo:
 94  	            plugin_info[""version""] = distinfo.version
(Pdb) distinfo
(Pdb) plugin
<module 'datasette.sql_functions' from '/Users/simon/Dropbox/Development/datasette/datasette/sql_functions.py'>
```
That `plugin_to_distinfo` is missing some stuff.",107914493
issue_comments,1730214654,IC_kwDOBm6k_c5nIPr-,,9599,2023-09-21T19:59:51Z,"So the problem is the `get_plugins()` function returning plugins with `None` for their name:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/plugins.py#L61-L91",107914493
issue_comments,1730212597,IC_kwDOBm6k_c5nIPL1,,9599,2023-09-21T19:58:38Z,Relevant code: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/app.py#L1127-L1146,107914493
issue_comments,1730211445,IC_kwDOBm6k_c5nIO51,,9599,2023-09-21T19:57:44Z,"In the debugger:
```
>>> import pdb
>>> pdb.pm()
> /Users/simon/Dropbox/Development/datasette/datasette/app.py(1136)_plugins()
-> ps.sort(key=lambda p: p[""name""])
(Pdb) ps
[{'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['prepare_connection', 'render_cell'], 'version': '1.0.1'}, {'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['startup'], 'version': '0.2'}]
```",107914493
issue_comments,1730210728,IC_kwDOBm6k_c5nIOuo,,9599,2023-09-21T19:57:08Z,"In my Python 3.8 environment I ran:
```bash
datasette install datasette-init datasette-json-html
```
And now `datasette plugins` produces this error:
```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 192, in plugins
    click.echo(json.dumps(app._plugins(all=all), indent=4))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 1136, in _plugins
    ps.sort(key=lambda p: p[""name""])
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
```",107914493
issue_comments,1730208566,IC_kwDOBm6k_c5nIOM2,,9599,2023-09-21T19:55:19Z,"Yes, the new script seems to work. On Python 3.11:

```
tests/test-datasette-load-plugins.sh 
echo $?
0
```
On Python 3.8:
```
tests/test-datasette-load-plugins.sh
Test failed: datasette-json-html not found
echo $?
1
```",107914493
issue_comments,1730206629,IC_kwDOBm6k_c5nINul,,9599,2023-09-21T19:53:39Z,"[GPT-4 says](https://chat.openai.com/share/65a95561-8f3c-4f9b-a4ab-c2508e87d2a9):

> In the script, you're using a subshell `( ... )` to group commands. If you `exit 1` within the subshell, it will only exit the subshell and not the main script. This is why GitHub Actions does not see it as a failure.

It suggested doing this instead:

```bash
#!/bin/bash

PLUGINS=$(datasette plugins)
if ! echo ""$PLUGINS"" | jq 'any(.[]; .name == ""datasette-json-html"")' | grep -q true; then
  echo ""Test failed: datasette-json-html not found""
  exit 1
fi

PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins)
if ! echo ""$PLUGINS2"" | jq 'any(.[]; .name == ""datasette-json-html"")' | grep -q false; then
  echo ""Test failed: datasette-json-html should not have been loaded""
  exit 1
fi

if ! echo ""$PLUGINS2"" | jq 'any(.[]; .name == ""datasette-init"")' | grep -q true; then
  echo ""Test failed: datasette-init should have been loaded""
  exit 1
fi

PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins)
if ! echo ""$PLUGINS3"" | grep -q '\[\]'; then
  echo ""Test failed: datasette plugins should have returned []""
  exit 1
fi
```
",107914493
issue_comments,1730203356,IC_kwDOBm6k_c5nIM7c,,9599,2023-09-21T19:51:04Z,"The script:

https://github.com/simonw/datasette/blob/b0d0a0e5de8bb5b9b6c253e8af451a532266bcf1/tests/test-datasette-load-plugins.sh#L1-L29

I'm not sure why those `exit 1` lines did not cause a CI failure.",107914493
issue_comments,1730202533,IC_kwDOBm6k_c5nIMul,,9599,2023-09-21T19:50:22Z,"Here's the failure in CI, which did not cause the workflow to fail even though it should have:

<img width=""733"" alt=""CleanShot 2023-09-21 at 12 50 01@2x"" src=""https://github.com/simonw/datasette/assets/9599/4c1d61be-b6f6-4699-a885-ed2d4edd85ab"">
",107914493
issues,1907655261,I_kwDOBm6k_c5xtIJd,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run",9599,2023-09-21T19:49:34Z,"> That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the ""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/2057#issuecomment-1730201226_
            ",107914493
issue_comments,1730201226,IC_kwDOBm6k_c5nIMaK,,9599,2023-09-21T19:49:20Z,"That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the ""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run.",107914493
issue_comments,1730188367,IC_kwDOBm6k_c5nIJRP,,9599,2023-09-21T19:38:28Z,"I'll imitate `certbot`:

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot-ci/setup.py#L9

```python
    'importlib_resources>=1.3.1; python_version < ""3.9""',
```
Looks like `1.3` is the minimum version needed for compatibility with the 3.9 standard library, according to https://github.com/python/importlib_resources/blob/main/README.rst#compatibility

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot/certbot/_internal/constants.py#L13C29-L16

```python
if sys.version_info >= (3, 9):  # pragma: no cover
    import importlib.resources as importlib_resources
else:  # pragma: no cover
    import importlib_resources
```",107914493
issue_comments,1730185322,IC_kwDOBm6k_c5nIIhq,,9599,2023-09-21T19:35:49Z,I think I can fix this using https://importlib-resources.readthedocs.io/en/latest/using.html - maybe as a dependency only installed if the Python version is less than 3.9.,107914493
issue_comments,1730183405,IC_kwDOBm6k_c5nIIDt,,9599,2023-09-21T19:34:09Z,"Confirmed: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

> `importlib.resources.files(package)`
> [...]
> New in version 3.9.",107914493
issue_comments,1730171241,IC_kwDOBm6k_c5nIFFp,,9599,2023-09-21T19:27:25Z,"This broke in Python 3.8:

```
if plugin.__name__ not in DEFAULT_PLUGINS:
    try:
        if (importlib.resources.files(plugin.__name__) / ""static"").is_dir():
E                   AttributeError: module 'importlib.resources' has no attribute 'files'
```",107914493
issue_comments,1730162283,IC_kwDOBm6k_c5nIC5r,,9599,2023-09-21T19:19:47Z,"I'm going to release this in `1.0a7`, and I'll backport it to a `0.64.4` release too.",107914493
issue_comments,1729961503,IC_kwDOBm6k_c5nHR4f,,15178711,2023-09-21T16:56:57Z,TODO: add similar checks for permissions/allow/canned queries,107914493
issues,1907281675,I_kwDOCGYnMM5xrs8L,Cascading DELETE not working with Table.delete(pk),123451970,2023-09-21T15:46:41Z,"Hi !
I noticed that when I am trying to use the delete method of the Table object,
the record get properly deleted from the table, but the cascading delete triggers on foreign keys do not activate.

`self.db[""contact""].delete(contact_id)`

I tried querying the database directly via DB Browser and the triggers work without any issue.
Looked up the source code and behind the scene this method is just querying the database normally so I'm not exactly sure where this behavior comes from.

Thank you in advance for your time ! ",140912432
issue_comments,1728504633,IC_kwDOBm6k_c5nBuM5,,9599,2023-09-20T22:24:51Z,"The `{""units"": {""distance"": ""m"", ""frequency"": ""Hz""}}` bit is for the [units feature](https://docs.datasette.io/en/1.0a6/metadata.html#specifying-units-for-a-column) which I've half-disabled already and would like to remove before 1.0, though ideally turning that functionality into a plugin instead (if I can figure out how to do that).",107914493
issue_comments,1728503623,IC_kwDOBm6k_c5nBt9H,,9599,2023-09-20T22:23:33Z,"This is one of the most interesting illustrative examples in the new code:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L301-L349

Interesting to note that it now has canned queries in it, which include this bit:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L341-L342

It looks like metadata, but in this case it's configuration. That blur between metadata and configuration at the canned query level still feels a little bit odd to me, but I still think we're going in the right direction with it.

Also interesting, from that same file:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L351-L399

There are a few things in that metadata block that are arguably configuration, not metadata - for example:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L360

I think `extra_css_urls` is definitely configuration, not metadata.

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L369-L395

Most of that stuff is arguably configuration too, with the exception of the `roadside_attractions.columns` bit which is metadata about those columns.",107914493
issue_comments,1728498221,IC_kwDOBm6k_c5nBsot,,9599,2023-09-20T22:17:26Z,"I tested this locally for permissions like this. `datasette.yml`:

```yaml
databases:
  content:
    allow:
      id: root
```
Started Datasette like this:
```bash
datasette --root content.db pottery2.db -c datasette.yml
```
As root I could see this (note the padlock):

<img width=""630"" alt=""CleanShot 2023-09-20 at 15 17 02@2x"" src=""https://github.com/simonw/datasette/assets/9599/3e0c3abd-9a90-43f4-ad68-bb490d19eef8"">

http://127.0.0.1:8001/-/metadata returned `{}` showing that the permissions must have come from the config file instead.",107914493
issue_comments,1728192688,IC_kwDOBm6k_c5nAiCw,,173848,2023-09-20T17:53:31Z,"`/me munches popcorn at a furious rate, utterly entralled`",107914493
issue_comments,1726754119,IC_kwDOBm6k_c5m7C1H,,22429695,2023-09-20T01:35:45Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`+0.02%`** :tada:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`4e6a341`)](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.72%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2192      +/-   ##
==========================================
+ Coverage   92.69%   92.72%   +0.02%     
==========================================
  Files          40       40              
  Lines        6039     6036       -3     
==========================================
- Hits         5598     5597       -1     
+ Misses        441      439       -2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/table.py](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.31% <100.00%> (+0.51%)` | :arrow_up: |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2192/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1903932086,PR_kwDOBm6k_c5aumyn,Stop using parallel SQL queries for tables,9599,2023-09-20T01:28:43Z,"Refs:
- #2189

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2192.org.readthedocs.build/en/2192/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1726749355,IC_kwDOBm6k_c5m7Bqr,,9599,2023-09-20T01:28:16Z,Added a note to that example in the documentation: https://github.com/simonw/datasette/blob/4e6a34179eaedec44c1263275d7592fd83d7e2ac/docs/internals.rst?plain=1#L1320,107914493
issue_comments,1724480716,IC_kwDOBm6k_c5myXzM,,22429695,2023-09-18T21:28:36Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified lines are covered by tests :white_check_mark:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`0135e7c`)](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
> Report is 14 commits behind head on main.

> :exclamation: Current head 0135e7c differs from pull request most recent head 18b48f8. Consider uploading reports for the commit 18b48f8 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2191      +/-   ##
==========================================
- Coverage   92.69%   92.68%   -0.02%     
==========================================
  Files          40       40              
  Lines        6039     6042       +3     
==========================================
+ Hits         5598     5600       +2     
- Misses        441      442       +1     
```


| [Files](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.09% <100.00%> (-0.11%)` | :arrow_down: |
| [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.36% <100.00%> (+0.01%)` | :arrow_up: |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2191/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1901768721,PR_kwDOBm6k_c5anSg5,"Move `permissions`,  `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`",15178711,2023-09-18T21:21:16Z,"The PR moves the following fields from `metadata.yaml` to `datasette.yaml`:

```
permissions
allow
allow_sql
queries
extra_css_urls
extra_js_urls
```

This is a significant breaking change that users will need to upgrade their `metadata.yaml` files for. But the format/locations are similar to the previous version, so it shouldn't be too difficult to upgrade.

One note: I'm still working on the Configuration docs, specifically the ""reference"" section. Though it's pretty small, the rest of read to review",107914493
issue_comments,1724325068,IC_kwDOBm6k_c5mxxzM,,9599,2023-09-18T20:29:41Z,"The one other thing affected by this change is this documentation, which suggests a not-actually-safe pattern: https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/docs/internals.rst#L1292-L1321",107914493
issue_comments,1724317367,IC_kwDOBm6k_c5mxv63,,9599,2023-09-18T20:25:44Z,"My current hunch is that SQLite gets unhappy if multiple threads access the same underlying C object - which sometimes happens with in-memory connections and Datasette presumably because they are faster than file-backed databases.

I'm going to remove the `asyncio.gather()` code from the table view. I'll ship a 0.x release with that fix too.",107914493
issue_comments,1724315591,IC_kwDOBm6k_c5mxvfH,,9599,2023-09-18T20:24:30Z,"[Using SQLite In Multi-Threaded Applications](https://www.sqlite.org/threadsafe.html)

That indicates that there's a SQLite option for ""Serialized"" mode where it's safe to access anything SQLite provides from multiple threads, but as far as I can tell Python doesn't give you an option to turn that mode on or off for a connection - you can read `sqlite3.threadsafet`y to see if that mode was compiled in or not, but not actually change it.

On my Mac `sqlite3.threadsafety` returns 1 which means https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety ""Multi-thread: In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads."" - it would need to return 3 for that serialized mode.",107914493
issue_comments,1724305169,IC_kwDOBm6k_c5mxs8R,,9599,2023-09-18T20:16:22Z,"Looking again at this code:

https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/datasette/database.py#L87-L117

`check_same_thread=False` really stands out here.

Python docs at https://docs.python.org/3/library/sqlite3.html

> **check_same_thread** ([*bool*](https://docs.python.org/3/library/functions.html#bool ""bool"")) -- If `True` (default), [`ProgrammingError`](https://docs.python.org/3/library/sqlite3.html#sqlite3.ProgrammingError ""sqlite3.ProgrammingError"") will be raised if the database connection is used by a thread other than the one that created it. If `False`, the connection may be accessed in multiple threads; write operations may need to be serialized by the user to avoid data corruption. See [`threadsafety`](https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety ""sqlite3.threadsafety"") for more information.

I think I'm playing with fire by allowing multiple threads to access the same connection without doing my own serialization of those requests.

I _do_ do that using the write connection - and in this particular case the bug isn't coming from write queries, it's coming from read queries - but perhaps SQLite has issues with threading for reads, too.",107914493
issue_comments,1724298817,IC_kwDOBm6k_c5mxrZB,,9599,2023-09-18T20:11:26Z,"Now that I've confirmed that parallel query execution of the kind introduced in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 can cause hangs (presumably some kind of locking issue) against in-memory databases, some options:

1. Disable parallel execution entirely and rip out related code.
2. Disable parallel execution entirely by leaving that code but having it always behave as if `_noparallel=1`
3. Continue digging and try and find some way to avoid this problem

The parallel execution work is something I was playing with last year in the hope of speeding up Datasette pages like the table page which need to execute a bunch of queries - one for each facet, plus one for each column to see if it should be suggested as a facet.

I wrote about this at the time here: https://simonwillison.net/2022/May/6/weeknotes/

My hope was that despite Python's GIL this optimization would still help, because the SQLite C module releases the GIL once it gets to SQLite.

But... that didn't hold up. It looked like enough work was happening in Python land with the GIL that the optimization didn't improve things.

Running the `nogil` fork of Python DID improve things though! I left the code in partly on the hope that the `nogil` fork would be accepted into Python core.

... which it now has! But it will still be a year or two before it fully lands: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

So I'm not particularly concerned about dropping the parallel execution. If I do drop it though do I leave the potentially complex code in that relates to it?",107914493
issue_comments,1724281824,IC_kwDOBm6k_c5mxnPg,,9599,2023-09-18T19:58:06Z,I also confirmed that `http://127.0.0.1:8064/airtable_refs/airtable_refs?_noparallel=1` does not trigger the bug but `http://127.0.0.1:8064/airtable_refs/airtable_refs` does.,107914493
issue_comments,1724278386,IC_kwDOBm6k_c5mxmZy,,9599,2023-09-18T19:55:32Z,"OK it looks like it found it!

```
942411ef946e9a34a2094944d3423cddad27efd3 is the first bad commit
commit 

Author: Simon Willison <swillison@gmail.com>
Date:   Tue Apr 26 15:48:56 2022 -0700

    Execute some TableView queries in parallel
    
    Use ?_noparallel=1 to opt out (undocumented, useful for benchmark comparisons)
    
    Refs #1723, #1715

 datasette/views/table.py | 93 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 67 insertions(+), 26 deletions(-)
bisect found first bad commit
```
https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 does look like the cause of this problem.",107914493
issue_comments,1724276917,IC_kwDOBm6k_c5mxmC1,,9599,2023-09-18T19:54:23Z,"Turned out I wasn't running the `datasette` from the current directory, so it was not testing what I intended.

FIxed that with `pip install -e .` in the `datasette/` directory.

Now I'm seeing some passes, which look like this:
```
running  '../airtable-export/testit.sh'
INFO:     Started server process [77810]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
INFO:     127.0.0.1:59439 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK
INFO:     127.0.0.1:59440 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK
INFO:     127.0.0.1:59441 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK
  All curl succeeded
Killing datasette server with PID 77810
../airtable-export/testit.sh: line 54: 77810 Killed: 9               datasette pottery2.db -p $port
All three curls succeeded.
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[7463b051cf8d7f856df5eba9f7aa944183ebabe5] Cosmetic tweaks after blacken-docs, refs #1718
running  '../airtable-export/testit.sh'
INFO:     Started server process [77826]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
```",107914493
issue_comments,1724263390,IC_kwDOBm6k_c5mxive,,9599,2023-09-18T19:44:03Z,I knocked it down to 1 retry just to see what happened.,107914493
issue_comments,1724259229,IC_kwDOBm6k_c5mxhud,,9599,2023-09-18T19:40:56Z,I tried it with a path of `/` and everything passed - so it's definitely the path of `/airtable_refs/airtable_refs` (an in-memory database created by an experimental branch of https://github.com/simonw/airtable-export) that triggers the problem.,107914493
issue_comments,1724258279,IC_kwDOBm6k_c5mxhfn,,9599,2023-09-18T19:40:13Z,"Output while it is running looks like this:
```
running  '../airtable-export/testit.sh'
INFO:     Started server process [75649]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
  Running curls
  Running curls
Killing datasette server with PID 75649
../airtable-export/testit.sh: line 54: 75649 Killed: 9               datasette pottery2.db -p $port
Error: Not all curls succeeded after 3 attempts.
Bisecting: 155 revisions left to test after this (roughly 7 steps)
[247e460e08bf823142f7b84058fe44e43626787f] Update beautifulsoup4 requirement (#1703)
running  '../airtable-export/testit.sh'
INFO:     Started server process [75722]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
  Running curls
  Running curls
Killing datasette server with PID 75722
../airtable-export/testit.sh: line 54: 75722 Killed: 9               datasette pottery2.db -p $port
Error: Not all curls succeeded after 3 attempts.
Bisecting: 77 revisions left to test after this (roughly 6 steps)
[3ef47a0896c7e63404a34e465b7160c80eaa571d] Link rel=alternate header for tables and rows
running  '../airtable-export/testit.sh'
INFO:     Started server process [75818]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
```",107914493
issue_comments,1724257290,IC_kwDOBm6k_c5mxhQK,,9599,2023-09-18T19:39:27Z,"I'm now trying this test script:
```bash
#!/bin/bash

port=8064
# Start datasette server in the background and get its PID
datasette pottery2.db -p $port &
server_pid=$!

# Wait for a moment to ensure the server has time to start up
sleep 2

# Initialize counters and parameters
retry_count=0
max_retries=3
success_count=0
path=""/airtable_refs/airtable_refs""

# Function to run curl with a timeout
function test_curl {
    # Run the curl command with a timeout of 3 seconds
    timeout 3s curl -s ""http://localhost:${port}${path}"" > /dev/null
    if [ $? -eq 0 ]; then
        # Curl was successful
        ((success_count++))
    fi
}

# Try three parallel curl requests
while [[ $retry_count -lt $max_retries ]]; do
    # Reset the success counter
    success_count=0

    # Run the curls in parallel
    echo ""  Running curls""
    test_curl
    test_curl
    test_curl #  & test_curl & test_curl &

    # Wait for all curls to finish
    #wait

    # Check the success count
    if [[ $success_count -eq 3 ]]; then
        # All curls succeeded, break out of the loop
        echo ""  All curl succeeded""
        break
    fi

    ((retry_count++))
done

# Kill the datasette server
echo ""Killing datasette server with PID $server_pid""
kill -9 $server_pid
sleep 2

# Print result
if [[ $success_count -eq 3 ]]; then
    echo ""All three curls succeeded.""
    exit 0
else
    echo ""Error: Not all curls succeeded after $retry_count attempts.""
    exit 1
fi
```
I run it like this:
```bash
git bisect reset
git bisect start
git bisect good 0.59.4
git bisect bad 1.0a6
git bisect run ../airtable-export/testit.sh
```
But... it's not having the desired result, I think because the bug is intermittent so each time I run it the bisect spits out a different commit as the one that is to blame.",107914493
issue_comments,1724169693,IC_kwDOBm6k_c5mxL3d,,22429695,2023-09-18T18:39:19Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fc7dbe0`)](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.67%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2190      +/-   ##
==========================================
- Coverage   92.69%   92.67%   -0.03%     
==========================================
  Files          40       40              
  Lines        6039     6044       +5     
==========================================
+ Hits         5598     5601       +3     
- Misses        441      443       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <100.00%> (ø)` | |
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.83% <100.00%> (+0.03%)` | :arrow_up: |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2190/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1724159882,IC_kwDOBm6k_c5mxJeK,,9599,2023-09-18T18:32:29Z,"This worked, including on macOS even though GPT-4 thought `timeout` would not work there: https://chat.openai.com/share/cc4628e9-5240-4f35-b640-16a9c178b315
```bash
#!/bin/bash

# Run the command with a timeout of 5 seconds
timeout 5s datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs

# Check the exit code from timeout
if [ $? -eq 124 ]; then
    echo ""Error: Command timed out after 5 seconds.""
    exit 1
fi
```",107914493
issue_comments,1724157182,IC_kwDOBm6k_c5mxIz-,,9599,2023-09-18T18:30:30Z,"OK, I can trigger the bug like this:

```bash
datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs
```
Can I write a bash script that fails (and terminates the process) if it takes longer than X seconds?",107914493
issues,1901483874,PR_kwDOBm6k_c5amULw,"Raise an exception if a ""plugins"" block exists in metadata.json",15178711,2023-09-18T18:08:56Z,"refs #2183 #2093

From [this comment](https://github.com/simonw/datasette/pull/2183#issuecomment-1714699724) in #2183: If a `""plugins""` block appears in `metadata.json`, it means that a user hasn't migrated over their plugin configuration from `metadata.json` to `datasette.yaml`, which is a breaking change in Datasette 1.0. 

This PR will ensure that an error is raised whenever that happens.

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2190.org.readthedocs.build/en/2190/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1724089666,IC_kwDOBm6k_c5mw4VC,,9599,2023-09-18T17:49:24Z,I switched that particular implementation to using an on-disk database instead of an in-memory database and could no longer recreate the bug.,107914493
issue_comments,1724084199,IC_kwDOBm6k_c5mw2_n,,9599,2023-09-18T17:47:01Z,I managed to trigger it by loading `http://127.0.0.1:8045/airtable_refs/airtable_refs` - which worked - and then hitting refresh on that page a bunch of times until it hung.,107914493
issue_comments,1724083324,IC_kwDOBm6k_c5mw2x8,,9599,2023-09-18T17:46:21Z,"Sometimes it takes a few clicks for the bug to occur, but it does seem to always be within the in-memory database.",107914493
issue_comments,1724081909,IC_kwDOBm6k_c5mw2b1,,9599,2023-09-18T17:45:27Z,Maybe it's not related to faceting - I just got it on a hit to `http://127.0.0.1:8045/airtable_refs/airtable_refs` instead.,107914493
issue_comments,1724072390,IC_kwDOBm6k_c5mw0HG,,9599,2023-09-18T17:39:06Z,Landing a version of that test anyway.,107914493
issue_comments,1724064440,IC_kwDOBm6k_c5mwyK4,,9599,2023-09-18T17:36:00Z,"I wrote this test, but it passes:
```python
@pytest.mark.asyncio
async def test_facet_against_in_memory_database():
    ds = Datasette()
    db = ds.add_memory_database(""mem"")
    await db.execute_write(""create table t (id integer primary key, name text)"")
    await db.execute_write_many(
        ""insert into t (name) values (?)"", [[""one""], [""one""], [""two""]]
    )
    response1 = await ds.client.get(""/mem/t.json"")
    assert response1.status_code == 200
    response2 = await ds.client.get(""/mem/t.json?_facet=name"")
    assert response2.status_code == 200
    assert response2.json() == {
        ""ok"": True,
        ""next"": None,
        ""facet_results"": {
            ""results"": {
                ""name"": {
                    ""name"": ""name"",
                    ""type"": ""column"",
                    ""hideable"": True,
                    ""toggle_url"": ""/mem/t.json"",
                    ""results"": [
                        {
                            ""value"": ""one"",
                            ""label"": ""one"",
                            ""count"": 2,
                            ""toggle_url"": ""http://localhost/mem/t.json?_facet=name&name=one"",
                            ""selected"": False,
                        },
                        {
                            ""value"": ""two"",
                            ""label"": ""two"",
                            ""count"": 1,
                            ""toggle_url"": ""http://localhost/mem/t.json?_facet=name&name=two"",
                            ""selected"": False,
                        },
                    ],
                    ""truncated"": False,
                }
            },
            ""timed_out"": [],
        },
        ""rows"": [
            {""id"": 1, ""name"": ""one""},
            {""id"": 2, ""name"": ""one""},
            {""id"": 3, ""name"": ""two""},
        ],
        ""truncated"": False,
    }
```",107914493
issue_comments,1724055823,IC_kwDOBm6k_c5mwwEP,,9599,2023-09-18T17:31:10Z,That line was added in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 which first shipped in 0.62a0.,107914493
issue_comments,1724051886,IC_kwDOBm6k_c5mwvGu,,9599,2023-09-18T17:28:20Z,"The bug exhibits when I try to add a facet. I think it's caused by the parallel query execution I added to facets at some point.

http://127.0.0.1:8045/airtable_refs/airtable_refs - no error
http://127.0.0.1:8045/airtable_refs/airtable_refs?_facet=table_name#facet-table_name - hangs the server

Crucial line in the traceback:
```
    await gather(execute_facets(), execute_suggested_facets())
```
From here: https://github.com/simonw/datasette/blob/917272c864ad7b8a00c48c77f5c2944093babb4e/datasette/views/table.py#L568",107914493
issue_comments,1724049538,IC_kwDOBm6k_c5mwuiC,,9599,2023-09-18T17:26:44Z,"Just managed to get this exception trace:
```
    return await self.route_path(scope, receive, send, path)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/app.py"", line 1354, in route_path
    response = await view(request, send)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 134, in view
    return await self.dispatch_request(request)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 91, in dispatch_request
    return await handler(request)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 361, in get
    response_or_template_contexts = await self.data(request, **data_kwargs)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 158, in data
    return await self._data_traced(request, default_labels, _next, _size)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 568, in _data_traced
    await gather(execute_facets(), execute_suggested_facets())
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 177, in _gather_parallel
    return await asyncio.gather(*args)
asyncio.exceptions.CancelledError
INFO:     127.0.0.1:64109 - ""GET /airtable_refs/airtable_refs?_facet=table_name&table_name=Sessions HTTP/1.1"" 500 Internal Server Error
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/concurrent/futures/thread.py"", line 40, in _python_exit
    t.join()
  File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py"", line 1011, in join
    self._wait_for_tstate_lock()
  File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py"", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
```",107914493
issue_comments,1724048314,IC_kwDOBm6k_c5mwuO6,,9599,2023-09-18T17:25:55Z,"The good news is that this bug is currently unlikely to affect most users since named in-memory databases (created using `datasette.add_memory_database(""airtable_refs"")` ([docs](https://docs.datasette.io/en/stable/internals.html#add-memory-database-name)) are a pretty obscure feature, only available to plugins.",107914493
issue_comments,1724045748,IC_kwDOBm6k_c5mwtm0,,9599,2023-09-18T17:24:07Z,"I need reliable steps to reproduce, then I can bisect and figure out which exact version of Datasette introduced the problem.

I have a hunch that it relates to changes made to the `datasette/database.py` module, maybe one of these changes here: https://github.com/simonw/datasette/compare/0.61...0.63.1#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9",107914493
issues,1901416155,I_kwDOBm6k_c5xVU7b,Server hang on parallel execution of queries to named in-memory databases,9599,2023-09-18T17:23:18Z,"I've started to encounter a bug where queries to tables inside named in-memory databases sometimes trigger server hangs.

I'm still trying to figure out what's going on here - on one occasion I managed to Ctrl+C the server and saw an exception that mentioned a thread lock, but usually hitting Ctrl+C does nothing and I have to `kill -9` the PID instead.

This is all running on my M2 Mac.

I've seen the bug in the Datasette 1.0 alphas and in Datasette 0.64.3 - but reverting to 0.61 appeared to fix it.",107914493
issue_comments,1723362847,IC_kwDOBm6k_c5muG4f,,6523121,2023-09-18T13:02:46Z,"Can confirm that this bug can be reproduced as follows:

```
docker run datasetteproject/datasette datasette serve --reload 
```

which produces the following output:

> Starting monitor for PID 10.
> Error: Invalid value for '[FILES]...': Path 'serve' does not exist.",107914493
issue_comments,1722943484,IC_kwDOBm6k_c5msgf8,,30934,2023-09-18T08:14:47Z,This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.,107914493
issue_comments,1722848454,IC_kwDOBm6k_c5msJTG,,15178711,2023-09-18T06:58:53Z,"Thinking about this more, here a list of things I imagine a ""compile-to-sql"" plugin would want to do:

1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting)
2. Add ""Query using PRQL"" buttons in various parts of Datasette's UI, like `/dbname` page
3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages
4. Have their own dedicated code editor page


1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of ""slots"" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks. 

So it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here. 

If I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...",107914493
issue_comments,1722845490,IC_kwDOBm6k_c5msIky,,15178711,2023-09-18T06:55:52Z,"One note here: this feature could be called ""slots"", similar to [Layout Slots](https://vitepress.dev/guide/extending-default-theme#layout-slots) in Vitepress.

In Vitepress, you can add custom components/widget/gadgets into determined named ""slots"", like so:

```
doc-top
doc-bottom
doc-footer-before
doc-before
doc-after
...
```

Would be great to do in both Python and Javascript, with the upcoming JavaScript API #2052. In `datasette-write-ui`, all we do is add a few ""Insert row"" and ""edit this row"" buttons and that required completely capturing the `table.html` template, which isn't great for other plugins. But having ""slots"" like `table-footer-before` or `table-row-id` or something would be great to work with.
",107914493
issue_comments,1722662413,IC_kwDOBm6k_c5mrb4N,,9599,2023-09-18T02:01:34Z,"I'm not interested in these in Datasette core itself, but I think they have a ton of potential for plugins.

I wonder what the best way to handle that would be?

Right now it's possible to write a plugin that [adds extra routes](https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes-datasette), so someone could build a `/dbname/-/prql?query=xxx` endpoint.

If this could return JSON, they could add JavaScript to the `/dbname` page that provided a UI for kicking off one of those queries.

Something that could make that more ergonomic might be the plugin hook that allows plugins to add extra HTML to different core database pages - e.g. adding a ""Query this database using PRQL"" button or link or even a full form at the top of that database page. That's this issue here:

- #1191",107914493
issues,1900026059,I_kwDOBm6k_c5xQBjL,"Plugin Hooks for ""compile to SQL"" languages",15178711,2023-09-18T01:37:15Z,"There's a ton of tools/languages that compile to SQL, which may be nice in Datasette. Some examples:

- Logica https://logica.dev
- PRQL https://prql-lang.org
- Malloy, but not sure if it works with SQLite? https://github.com/malloydata/malloy

It would be cool if plugins could extend Datasette to use these languages, in both the code editor and API usage.

A few things I'd imagine a `datasette-prql` or `datasette-logica` plugin would do:

- `prql=` instead of `sql=`
- Code editor support (syntax highlighting, autocomplete)
- Hide/show SQL",107914493
issue_comments,1722323967,IC_kwDOBm6k_c5mqJP_,,9599,2023-09-16T21:54:33Z,Just found this migration guide: https://importlib-metadata.readthedocs.io/en/latest/migration.html,107914493
issue_comments,1722266942,IC_kwDOBm6k_c5mp7U-,,9599,2023-09-16T16:38:27Z,"The `importlib.metadata.entry_points()` function is pretty interesting:
```pycon
>>> import importlib.metadata
>>> from pprint import pprint
>>> pprint(importlib.metadata.entry_points())
{'babel.checkers': [EntryPoint(name='num_plurals', value='babel.messages.checkers:num_plurals', group='babel.checkers'),
                    EntryPoint(name='python_format', value='babel.messages.checkers:python_format', group='babel.checkers')],
 'babel.extractors': [EntryPoint(name='jinja2', value='jinja2.ext:babel_extract[i18n]', group='babel.extractors'),
                      EntryPoint(name='ignore', value='babel.messages.extract:extract_nothing', group='babel.extractors'),
                      EntryPoint(name='javascript', value='babel.messages.extract:extract_javascript', group='babel.extractors'),
                      EntryPoint(name='python', value='babel.messages.extract:extract_python', group='babel.extractors')],
 'console_scripts': [EntryPoint(name='datasette', value='datasette.cli:cli', group='console_scripts'),
                     EntryPoint(name='normalizer', value='charset_normalizer.cli.normalizer:cli_detect', group='console_scripts'),
                     EntryPoint(name='pypprint', value='pprintpp:console', group='console_scripts'),
                     EntryPoint(name='cog', value='cogapp:main', group='console_scripts'),
                     EntryPoint(name='icdiff', value='icdiff:start', group='console_scripts'),
                     EntryPoint(name='pycodestyle', value='pycodestyle:_main', group='console_scripts'),
                     EntryPoint(name='sphinx-autobuild', value='sphinx_autobuild.__main__:main', group='console_scripts'),
                     EntryPoint(name='sphinx-apidoc', value='sphinx.ext.apidoc:main', group='console_scripts'),
                     EntryPoint(name='sphinx-autogen', value='sphinx.ext.autosummary.generate:main', group='console_scripts'),
                     EntryPoint(name='sphinx-build', value='sphinx.cmd.build:main', group='console_scripts'),
                     EntryPoint(name='sphinx-quickstart', value='sphinx.cmd.quickstart:main', group='console_scripts'),
                     EntryPoint(name='sphinx-to-sqlite', value='sphinx_to_sqlite.cli:cli', group='console_scripts'),
                     EntryPoint(name='pybabel', value='babel.messages.frontend:main', group='console_scripts'),
                     EntryPoint(name='docutils', value='docutils.__main__:main', group='console_scripts'),
                     EntryPoint(name='isort', value='isort.main:main', group='console_scripts'),
                     EntryPoint(name='isort-identify-imports', value='isort.main:identify_imports_main', group='console_scripts'),
                     EntryPoint(name='hupper', value='hupper.cli:main', group='console_scripts'),
                     EntryPoint(name='sqlite-utils', value='sqlite_utils.cli:cli', group='console_scripts'),
                     EntryPoint(name='py.test', value='pytest:console_main', group='console_scripts'),
                     EntryPoint(name='pytest', value='pytest:console_main', group='console_scripts'),
                     EntryPoint(name='pyflakes', value='pyflakes.api:main', group='console_scripts'),
                     EntryPoint(name='livereload', value='livereload.cli:main', group='console_scripts'),
                     EntryPoint(name='uvicorn', value='uvicorn.main:main', group='console_scripts'),
                     EntryPoint(name='httpx', value='httpx:main', group='console_scripts'),
                     EntryPoint(name='flake8', value='flake8.main.cli:main', group='console_scripts'),
                     EntryPoint(name='blacken-docs', value='blacken_docs:main', group='console_scripts'),
                     EntryPoint(name='pip', value='pip._internal.cli.main:main', group='console_scripts'),
                     EntryPoint(name='pip3', value='pip._internal.cli.main:main', group='console_scripts'),
                     EntryPoint(name='pip3.10', value='pip._internal.cli.main:main', group='console_scripts'),
                     EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts'),
                     EntryPoint(name='pygmentize', value='pygments.cmdline:main', group='console_scripts'),
                     EntryPoint(name='black', value='black:patched_main', group='console_scripts'),
                     EntryPoint(name='blackd', value='blackd:patched_main [d]', group='console_scripts'),
                     EntryPoint(name='codespell', value='codespell_lib:_script_main', group='console_scripts'),
                     EntryPoint(name='tabulate', value='tabulate:_main', group='console_scripts')],
 'datasette': [EntryPoint(name='debug_permissions', value='datasette_debug_permissions', group='datasette'),
               EntryPoint(name='codespaces', value='datasette_codespaces', group='datasette'),
               EntryPoint(name='vega', value='datasette_vega', group='datasette'),
               EntryPoint(name='x_forwarded_host', value='datasette_x_forwarded_host', group='datasette'),
               EntryPoint(name='json_html', value='datasette_json_html', group='datasette'),
               EntryPoint(name='datasette_write_ui', value='datasette_write_ui', group='datasette'),
               EntryPoint(name='pretty_json', value='datasette_pretty_json', group='datasette'),
               EntryPoint(name='graphql', value='datasette_graphql', group='datasette')],
 'distutils.commands': [EntryPoint(name='compile_catalog', value='babel.messages.frontend:compile_catalog', group='distutils.commands'),
                        EntryPoint(name='extract_messages', value='babel.messages.frontend:extract_messages', group='distutils.commands'),
                        EntryPoint(name='init_catalog', value='babel.messages.frontend:init_catalog', group='distutils.commands'),
                        EntryPoint(name='update_catalog', value='babel.messages.frontend:update_catalog', group='distutils.commands'),
                        EntryPoint(name='isort', value='isort.setuptools_commands:ISortCommand', group='distutils.commands'),
                        EntryPoint(name='alias', value='setuptools.command.alias:alias', group='distutils.commands'),
                        EntryPoint(name='bdist_egg', value='setuptools.command.bdist_egg:bdist_egg', group='distutils.commands'),
                        EntryPoint(name='bdist_rpm', value='setuptools.command.bdist_rpm:bdist_rpm', group='distutils.commands'),
                        EntryPoint(name='build', value='setuptools.command.build:build', group='distutils.commands'),
                        EntryPoint(name='build_clib', value='setuptools.command.build_clib:build_clib', group='distutils.commands'),
                        EntryPoint(name='build_ext', value='setuptools.command.build_ext:build_ext', group='distutils.commands'),
                        EntryPoint(name='build_py', value='setuptools.command.build_py:build_py', group='distutils.commands'),
                        EntryPoint(name='develop', value='setuptools.command.develop:develop', group='distutils.commands'),
                        EntryPoint(name='dist_info', value='setuptools.command.dist_info:dist_info', group='distutils.commands'),
                        EntryPoint(name='easy_install', value='setuptools.command.easy_install:easy_install', group='distutils.commands'),
                        EntryPoint(name='editable_wheel', value='setuptools.command.editable_wheel:editable_wheel', group='distutils.commands'),
                        EntryPoint(name='egg_info', value='setuptools.command.egg_info:egg_info', group='distutils.commands'),
                        EntryPoint(name='install', value='setuptools.command.install:install', group='distutils.commands'),
                        EntryPoint(name='install_egg_info', value='setuptools.command.install_egg_info:install_egg_info', group='distutils.commands'),
                        EntryPoint(name='install_lib', value='setuptools.command.install_lib:install_lib', group='distutils.commands'),
                        EntryPoint(name='install_scripts', value='setuptools.command.install_scripts:install_scripts', group='distutils.commands'),
                        EntryPoint(name='rotate', value='setuptools.command.rotate:rotate', group='distutils.commands'),
                        EntryPoint(name='saveopts', value='setuptools.command.saveopts:saveopts', group='distutils.commands'),
                        EntryPoint(name='sdist', value='setuptools.command.sdist:sdist', group='distutils.commands'),
                        EntryPoint(name='setopt', value='setuptools.command.setopt:setopt', group='distutils.commands'),
                        EntryPoint(name='test', value='setuptools.command.test:test', group='distutils.commands'),
                        EntryPoint(name='upload_docs', value='setuptools.command.upload_docs:upload_docs', group='distutils.commands'),
                        EntryPoint(name='bdist_wheel', value='wheel.bdist_wheel:bdist_wheel', group='distutils.commands')],
 'distutils.setup_keywords': [EntryPoint(name='message_extractors', value='babel.messages.frontend:check_message_extractors', group='distutils.setup_keywords'),
                              EntryPoint(name='cffi_modules', value='cffi.setuptools_ext:cffi_modules', group='distutils.setup_keywords'),
                              EntryPoint(name='dependency_links', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
                              EntryPoint(name='eager_resources', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
                              EntryPoint(name='entry_points', value='setuptools.dist:check_entry_points', group='distutils.setup_keywords'),
                              EntryPoint(name='exclude_package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
                              EntryPoint(name='extras_require', value='setuptools.dist:check_extras', group='distutils.setup_keywords'),
                              EntryPoint(name='include_package_data', value='setuptools.dist:assert_bool', group='distutils.setup_keywords'),
                              EntryPoint(name='install_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                              EntryPoint(name='namespace_packages', value='setuptools.dist:check_nsp', group='distutils.setup_keywords'),
                              EntryPoint(name='package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
                              EntryPoint(name='packages', value='setuptools.dist:check_packages', group='distutils.setup_keywords'),
                              EntryPoint(name='python_requires', value='setuptools.dist:check_specifier', group='distutils.setup_keywords'),
                              EntryPoint(name='setup_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                              EntryPoint(name='test_loader', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
                              EntryPoint(name='test_runner', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
                              EntryPoint(name='test_suite', value='setuptools.dist:check_test_suite', group='distutils.setup_keywords'),
                              EntryPoint(name='tests_require', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                              EntryPoint(name='use_2to3', value='setuptools.dist:invalid_unless_false', group='distutils.setup_keywords'),
                              EntryPoint(name='zip_safe', value='setuptools.dist:assert_bool', group='distutils.setup_keywords')],
 'egg_info.writers': [EntryPoint(name='PKG-INFO', value='setuptools.command.egg_info:write_pkg_info', group='egg_info.writers'),
                      EntryPoint(name='dependency_links.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                      EntryPoint(name='depends.txt', value='setuptools.command.egg_info:warn_depends_obsolete', group='egg_info.writers'),
                      EntryPoint(name='eager_resources.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                      EntryPoint(name='entry_points.txt', value='setuptools.command.egg_info:write_entries', group='egg_info.writers'),
                      EntryPoint(name='namespace_packages.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                      EntryPoint(name='requires.txt', value='setuptools.command.egg_info:write_requirements', group='egg_info.writers'),
                      EntryPoint(name='top_level.txt', value='setuptools.command.egg_info:write_toplevel_names', group='egg_info.writers')],
 'flake8.extension': [EntryPoint(name='C90', value='mccabe:McCabeChecker', group='flake8.extension'),
                      EntryPoint(name='E', value='flake8.plugins.pycodestyle:pycodestyle_logical', group='flake8.extension'),
                      EntryPoint(name='F', value='flake8.plugins.pyflakes:FlakesChecker', group='flake8.extension'),
                      EntryPoint(name='W', value='flake8.plugins.pycodestyle:pycodestyle_physical', group='flake8.extension')],
 'flake8.report': [EntryPoint(name='default', value='flake8.formatting.default:Default', group='flake8.report'),
                   EntryPoint(name='pylint', value='flake8.formatting.default:Pylint', group='flake8.report'),
                   EntryPoint(name='quiet-filename', value='flake8.formatting.default:FilenameOnly', group='flake8.report'),
                   EntryPoint(name='quiet-nothing', value='flake8.formatting.default:Nothing', group='flake8.report')],
 'pylama.linter': [EntryPoint(name='isort', value='isort.pylama_isort:Linter', group='pylama.linter')],
 'pytest11': [EntryPoint(name='icdiff', value='pytest_icdiff', group='pytest11'),
              EntryPoint(name='asyncio', value='pytest_asyncio.plugin', group='pytest11'),
              EntryPoint(name='xdist', value='xdist.plugin', group='pytest11'),
              EntryPoint(name='xdist.looponfail', value='xdist.looponfail', group='pytest11'),
              EntryPoint(name='timeout', value='pytest_timeout', group='pytest11'),
              EntryPoint(name='anyio', value='anyio.pytest_plugin', group='pytest11')],
 'setuptools.finalize_distribution_options': [EntryPoint(name='keywords', value='setuptools.dist:Distribution._finalize_setup_keywords', group='setuptools.finalize_distribution_options'),
                                              EntryPoint(name='parent_finalize', value='setuptools.dist:_Distribution.finalize_options', group='setuptools.finalize_distribution_options')],
 'sphinx.html_themes': [EntryPoint(name='alabaster', value='alabaster', group='sphinx.html_themes'),
                        EntryPoint(name='basic-ng', value='sphinx_basic_ng', group='sphinx.html_themes'),
                        EntryPoint(name='furo', value='furo', group='sphinx.html_themes')],
 'sqlite_utils': [EntryPoint(name='hello_world', value='sqlite_utils_hello_world', group='sqlite_utils')]}
```",107914493
issue_comments,1722266513,IC_kwDOBm6k_c5mp7OR,,9599,2023-09-16T16:36:09Z,"Now I need to switch out `pkg_resources` in `plugins.py`:

https://github.com/simonw/datasette/blob/852f5014853943fa27f43ddaa2d442545b3259fb/datasette/plugins.py#L33-L74",107914493
issue_comments,1722265848,IC_kwDOBm6k_c5mp7D4,,9599,2023-09-16T16:32:42Z,"Here's the exception it uses:
```pycon
>>> importlib.metadata.version(""datasette"")
'1.0a6'
>>> importlib.metadata.version(""datasette2"")
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 996, in version
    return distribution(distribution_name).version
  File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 969, in distribution
    return Distribution.from_name(distribution_name)
  File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 548, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for datasette2
```",107914493
issue_comments,1722258980,IC_kwDOBm6k_c5mp5Yk,,9599,2023-09-16T15:56:45Z,"Weird, I still can't get the warning to show even with this:
```python
@pytest.mark.asyncio
async def test_plugin_is_installed():
    datasette = Datasette(memory=True)

    class DummyPlugin:
        __name__ = ""DummyPlugin""

        @hookimpl
        def actors_from_ids(self, datasette, actor_ids):
            return {}

    try:
        pm.register(DummyPlugin(), name=""DummyPlugin"")
        response = await datasette.client.get(""/-/plugins.json"")
        assert response.status_code == 200
        installed_plugins = {p[""name""] for p in response.json()}
        assert ""DummyPlugin"" in installed_plugins

    finally:
        pm.unregister(name=""ReturnNothingPlugin"")
```",107914493
issue_comments,1722257328,IC_kwDOBm6k_c5mp4-w,,9599,2023-09-16T15:47:32Z,"Frustrating that this warning doesn't show up in the Datasette test suite itself. It shows up in plugin test suites that run this test:
```python
@pytest.mark.asyncio
async def test_plugin_is_installed():
    datasette = Datasette(memory=True)
    response = await datasette.client.get(""/-/plugins.json"")
    assert response.status_code == 200
    installed_plugins = {p[""name""] for p in response.json()}
    assert ""datasette-chronicle"" in installed_plugins
```
If you run that test inside Datasette core `installed_plugins` is an empty set, which presumably is why the warning doesn't get triggered there.",107914493
issues,1899310542,I_kwDOBm6k_c5xNS3O,Datasette for serving JSON only,19705106,2023-09-16T05:48:29Z,"Hi, is there any way to use datasette for serving json only without displaying webpage? I've tried to search about this in documentation but didn't get any information",107914493
issue_comments,1721742055,IC_kwDOBm6k_c5mn7Ln,,9599,2023-09-15T19:27:59Z,This feels like it might be quite a nice pattern generally - providing optional arguments to plugins and views that can be `await get_x()` called to run an extra calculation.,107914493
issue_comments,1721740872,IC_kwDOBm6k_c5mn65I,,9599,2023-09-15T19:26:51Z,"Here's where it's called at the moment: https://github.com/simonw/datasette/blob/16f0b6d8222d06682a31b904d0a402c391ae1c1c/datasette/views/base.py#L297-L313

And the docs: https://github.com/simonw/datasette/blob/1.0a6/docs/plugin_hooks.rst#register-output-renderer-datasette

I'm tempted to add a `get_count` argument which, when awaited, returns the full count. Then plugins could do this:

```python
async def render_notebook(datasette, request, get_count, rows):
    count = await get_count()
    # ...
```",107914493
issues,1898927976,I_kwDOBm6k_c5xL1do,Mechanism for register_output_renderer hooks to access full count,9599,2023-09-15T18:57:54Z,"The cause of this bug:
- https://github.com/simonw/datasette-export-notebook/issues/17

Is that `datasette-export-notebook` was consulting `data[""filtered_table_rows_count""]` in the render output plugin function in order to show the total number of rows that would be exported.

That field is no longer available by default - the `""count""` field is only available if `?_extra=count` was passed.

It would be useful if plugins like this could access the total count on demand, should they need to.",107914493
issue_comments,1719468727,IC_kwDOBm6k_c5mfQK3,,22429695,2023-09-14T13:36:07Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage has no change and project coverage change: **`-0.04%`** :warning:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fe5f881`)](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2185      +/-   ##
==========================================
- Coverage   92.69%   92.66%   -0.04%     
==========================================
  Files          40       40              
  Lines        6039     6039              
==========================================
- Hits         5598     5596       -2     
- Misses        441      443       +2     
```


[see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2185/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1896578249,PR_kwDOBm6k_c5aWACP,Bump the python-packages group with 3 updates,49699333,2023-09-14T13:27:40Z,"Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [black](https://github.com/psf/black).

Updates `sphinx` from 7.2.5 to 7.2.6
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.2.6</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.2.6 (released Sep 13, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11679"">#11679</a>: Add the :envvar:<code>!SPHINX_AUTODOC_RELOAD_MODULES</code> environment variable,
which if set reloads modules when using autodoc with <code>TYPE_CHECKING = True</code>.
Patch by Matt Wozniski and Adam Turner.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11679"">#11679</a>: Use :py:func:<code>importlib.reload</code> to reload modules in autodoc.
Patch by Matt Wozniski and Adam Turner.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/cf7d2759af0852d67288e58d823d51fe860749ca""><code>cf7d275</code></a> Bump to 7.2.6 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/43d69755df0088fd0ec099c0893da059f50175e4""><code>43d6975</code></a> Leverage <code>importlib.reload</code> for reloading modules (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11679"">#11679</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/13da5d7b2fda0da58137534e8fcdb0da9c88e55f""><code>13da5d7</code></a> Inline makecmd in make mode</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/3d0110a95a5260cc5056322eb8c2ae7f031a6667""><code>3d0110a</code></a> Enable test_cython on Python 3.12</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/22759fb9457bd9f1b01003daa47d7a2e78b6303c""><code>22759fb</code></a> Bump version</li>
<li>See full diff in <a href=""https://github.com/sphinx-doc/sphinx/compare/v7.2.5...v7.2.6"">compare view</a></li>
</ul>
</details>
<br />

Updates `furo` from 2023.8.19 to 2023.9.10
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.09.10 -- Zesty Zaffre</h2>
<ul>
<li>Make asset hash injection idempotent, fixing Sphinx 6 compatibility.</li>
<li>Fix the check for HTML builders, fixing non-HTML Read the Docs builds.</li>
</ul>
<h2>2023.08.19 -- Xenolithic Xanadu</h2>
<ul>
<li>Fix missing search context with Sphinx 7.2, for dirhtml builds.</li>
<li>Drop support for Python 3.7.</li>
<li>Present configuration errors in a better format -- thanks <a href=""https://github.com/AA-Turner""><code>@​AA-Turner</code></a>!</li>
<li>Bump <code>require_sphinx()</code> to Sphinx 6.0, in line with dependency changes in Unassuming Ultramarine.</li>
</ul>
<h2>2023.08.17 -- Wonderous White</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.2.0 and 7.2.1.</li>
</ul>
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/2718ca42930f7c8c11bc96341e0d3db6f532b08d""><code>2718ca4</code></a> Prepare release: 2023.09.10</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c22c99d931decfa4641e400e428606bc6184af56""><code>c22c99d</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c37e84961d9ffbc77ab7be268081d1b3497f0311""><code>c37e849</code></a> Quote a not-runtime-generic type annotation</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/9cfdf44784dc90085549490ff1eddd2afc37c1d6""><code>9cfdf44</code></a> Rework infrastructure for linting</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/5abeb9fc962aee61eda02e0c75e872004635e9d5""><code>5abeb9f</code></a> Fix the check for HTML builders</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/ee2ab5459ddb0a32c3467b88c9c628611eb55073""><code>ee2ab54</code></a> Tweak how tests are run with nox</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/cdae2366c340695ba724ca8438a4cd1e605c8848""><code>cdae236</code></a> Test against Sphinx minor versions in CI</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/9e40071eb8c4c3e38845b57c4f5242bef2a44af2""><code>9e40071</code></a> Make asset hash injection idempotent</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/aab86f4624b6ef38a151440be5624746d41882b1""><code>aab86f4</code></a> Revert &quot;Exclude incompatible Sphinx releases (<a href=""https://redirect.github.com/pradyunsg/furo/issues/711"">#711</a>)&quot;</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/4dd6eec9e306b5fd0624ec8d1d90c6ad416c5182""><code>4dd6eec</code></a> Exclude incompatible Sphinx releases (<a href=""https://redirect.github.com/pradyunsg/furo/issues/711"">#711</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.08.19...2023.09.10"">compare view</a></li>
</ul>
</details>
<br />

Updates `black` from 23.7.0 to 23.9.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>23.9.1</h2>
<p>Due to various issues, the previous release (23.9.0) did not include compiled mypyc
wheels, which make Black significantly faster. These issues have now been fixed, and
this release should come with compiled wheels once again.</p>
<p>There will be no wheels for Python 3.12 due to a bug in mypyc. We will provide 3.12
wheels in a future release as soon as the mypyc bug is fixed.</p>
<h3>Packaging</h3>
<ul>
<li>Upgrade to mypy 1.5.1 (<a href=""https://redirect.github.com/psf/black/issues/3864"">#3864</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Store raw tuples instead of NamedTuples in Black's cache, improving performance and
decreasing the size of the cache (<a href=""https://redirect.github.com/psf/black/issues/3877"">#3877</a>)</li>
</ul>
<h2>23.9.0</h2>
<h3>Preview style</h3>
<ul>
<li>More concise formatting for dummy implementations (<a href=""https://redirect.github.com/psf/black/issues/3796"">#3796</a>)</li>
<li>In stub files, add a blank line between a statement with a body (e.g an
<code>if sys.version_info &gt; (3, x):</code>) and a function definition on the same level (<a href=""https://redirect.github.com/psf/black/issues/3862"">#3862</a>)</li>
<li>Fix a bug whereby spaces were removed from walrus operators within subscript(<a href=""https://redirect.github.com/psf/black/issues/3823"">#3823</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Black now applies exclusion and ignore logic before resolving symlinks (<a href=""https://redirect.github.com/psf/black/issues/3846"">#3846</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Avoid importing <code>IPython</code> if notebook cells do not contain magics (<a href=""https://redirect.github.com/psf/black/issues/3782"">#3782</a>)</li>
<li>Improve caching by comparing file hashes as fallback for mtime and size (<a href=""https://redirect.github.com/psf/black/issues/3821"">#3821</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Fix an issue in <code>blackd</code> with single character input (<a href=""https://redirect.github.com/psf/black/issues/3558"">#3558</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Black now has an
<a href=""https://github.com/psf/black-pre-commit-mirror"">official pre-commit mirror</a>. Swapping
<code>https://github.com/psf/black</code> to <code>https://github.com/psf/black-pre-commit-mirror</code> in
your <code>.pre-commit-config.yaml</code> will make Black about 2x faster (<a href=""https://redirect.github.com/psf/black/issues/3828"">#3828</a>)</li>
<li>The <code>.black.env</code> folder specified by <code>ENV_PATH</code> will now be removed on the completion
of the GitHub Action (<a href=""https://redirect.github.com/psf/black/issues/3759"">#3759</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>23.9.1</h2>
<p>Due to various issues, the previous release (23.9.0) did not include compiled mypyc
wheels, which make Black significantly faster. These issues have now been fixed, and
this release should come with compiled wheels once again.</p>
<p>There will be no wheels for Python 3.12 due to a bug in mypyc. We will provide 3.12
wheels in a future release as soon as the mypyc bug is fixed.</p>
<h3>Packaging</h3>
<ul>
<li>Upgrade to mypy 1.5.1 (<a href=""https://redirect.github.com/psf/black/issues/3864"">#3864</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Store raw tuples instead of NamedTuples in Black's cache, improving performance and
decreasing the size of the cache (<a href=""https://redirect.github.com/psf/black/issues/3877"">#3877</a>)</li>
</ul>
<h2>23.9.0</h2>
<h3>Preview style</h3>
<ul>
<li>More concise formatting for dummy implementations (<a href=""https://redirect.github.com/psf/black/issues/3796"">#3796</a>)</li>
<li>In stub files, add a blank line between a statement with a body (e.g an
<code>if sys.version_info &gt; (3, x):</code>) and a function definition on the same level (<a href=""https://redirect.github.com/psf/black/issues/3862"">#3862</a>)</li>
<li>Fix a bug whereby spaces were removed from walrus operators within subscript(<a href=""https://redirect.github.com/psf/black/issues/3823"">#3823</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Black now applies exclusion and ignore logic before resolving symlinks (<a href=""https://redirect.github.com/psf/black/issues/3846"">#3846</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Avoid importing <code>IPython</code> if notebook cells do not contain magics (<a href=""https://redirect.github.com/psf/black/issues/3782"">#3782</a>)</li>
<li>Improve caching by comparing file hashes as fallback for mtime and size (<a href=""https://redirect.github.com/psf/black/issues/3821"">#3821</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Fix an issue in <code>blackd</code> with single character input (<a href=""https://redirect.github.com/psf/black/issues/3558"">#3558</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Black now has an
<a href=""https://github.com/psf/black-pre-commit-mirror"">official pre-commit mirror</a>. Swapping
<code>https://github.com/psf/black</code> to <code>https://github.com/psf/black-pre-commit-mirror</code> in
your <code>.pre-commit-config.yaml</code> will make Black about 2x faster (<a href=""https://redirect.github.com/psf/black/issues/3828"">#3828</a>)</li>
<li>The <code>.black.env</code> folder specified by <code>ENV_PATH</code> will now be removed on the completion
of the GitHub Action (<a href=""https://redirect.github.com/psf/black/issues/3759"">#3759</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/psf/black/commit/e87737140f32d3cd7c44ede75f02dcd58e55820e""><code>e877371</code></a> Prepare release 23.9.1 (<a href=""https://redirect.github.com/psf/black/issues/3878"">#3878</a>)</li>
<li><a href=""https://github.com/psf/black/commit/62dca32dc55a850f39d78ba8b9c21cc4261a2ddf""><code>62dca32</code></a> mypyc builds on PRs, skip mypyc wheels for 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3870"">#3870</a>)</li>
<li><a href=""https://github.com/psf/black/commit/751583a1dfc691423d96d7711a4c8d9cfe3ee9c8""><code>751583a</code></a> Pickle raw tuples in FileData cache (<a href=""https://redirect.github.com/psf/black/issues/3877"">#3877</a>)</li>
<li><a href=""https://github.com/psf/black/commit/f7917453c99f8183ffd0397affcccb2c37594771""><code>f791745</code></a> Re-export black.Mode (<a href=""https://redirect.github.com/psf/black/issues/3875"">#3875</a>)</li>
<li><a href=""https://github.com/psf/black/commit/0b62b9c9a44a995e44d64ecf7cc08d8d7037642d""><code>0b62b9c</code></a> Ignore aiohttp DeprecationWarning for 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3876"">#3876</a>)</li>
<li><a href=""https://github.com/psf/black/commit/c83ad6c077e7bb281cfd3fbdd89bbeb4c980e563""><code>c83ad6c</code></a> Upgrade to Furo 2023.9.10 to fix docs build (<a href=""https://redirect.github.com/psf/black/issues/3873"">#3873</a>)</li>
<li><a href=""https://github.com/psf/black/commit/4eebfd1a7a4aa2652cfc674cf301d1f2f48098aa""><code>4eebfd1</code></a> Add mypyc test marks to new tests that patch (<a href=""https://redirect.github.com/psf/black/issues/3871"">#3871</a>)</li>
<li><a href=""https://github.com/psf/black/commit/add161b367a0d5a3cc395ec8e045f7b965edaef8""><code>add161b</code></a> Bump RTD Python version from 3.8 to 3.11 (<a href=""https://redirect.github.com/psf/black/issues/3868"">#3868</a>)</li>
<li><a href=""https://github.com/psf/black/commit/4e93f2aa01606154dc6af6e494b9f2b7e4c8c7fa""><code>4e93f2a</code></a> Add classifier for 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3866"">#3866</a>)</li>
<li><a href=""https://github.com/psf/black/commit/716fa08090b6a51e4c72dd0a14b6c45f7da4a9d0""><code>716fa08</code></a> Upgrade mypy (<a href=""https://redirect.github.com/psf/black/issues/3864"">#3864</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/psf/black/compare/23.7.0...23.9.1"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2185.org.readthedocs.build/en/2185/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1719451803,IC_kwDOBm6k_c5mfMCb,,49699333,2023-09-14T13:27:26Z,"Looks like these dependencies are updatable in another way, so this is no longer needed.",107914493
issues,1895266807,I_kwDOBm6k_c5w93n3,Design decision - should configuration be exposed at /-/config ?,9599,2023-09-13T21:07:08Z,"> This made me think. That `{""$env"": ""ENV_VAR""}` hack was introduced back here:
>
> - https://github.com/simonw/datasette/issues/538
>
> The problem it was solving was that metadata was visible to everyone with access to the instance at `/-/metadata` but plugins clearly needed a way to set secret settings.
>
> Now that this stuff is moving to config, we have some decisions to make:
>
> 1. Add `/-/config` to let people see the configuration of their instance, and keep the `$env` trick for secret settings.
> 2. Say all configuration aside from metadata is secret and make `$env` optional or ditch it entirely.
> 3. Allow plugins to announce which of their configuration options are secret so we can automatically redact them from `/-/config`
>
> I've found `/-/metadata` extraordinarily useful as a user of Datasette - it really helps me understand exactly what's going on if I run into any problems with a plugin, if I can quickly check what the settings look like.
>
> So I'm leaning towards option 1 or 3.

_Originally posted by @simonw in https://github.com/simonw/datasette/pull/2183#discussion_r1325076924_

Also refs:
- #2093",107914493
issue_comments,1718316733,IC_kwDOBm6k_c5ma269,,9599,2023-09-13T21:05:36Z,I'm going to land this and make any further documentation tweaks on `main`.,107914493
issue_comments,1716801971,IC_kwDOBm6k_c5mVFGz,,15178711,2023-09-13T01:34:01Z,"@simonw docs are finished, this is ready for review!

One thing: I added ""Configuration"" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named ""datasette.yaml Configuration"" or something similar?

Mostly because ""Configuration"" by itself can mean many things, but adding ""datasette.yaml"" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using ""datasette.yaml"" instead of ""datasette.json"", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported)

Also, maybe this is a chance to consolidate the docs a bit? I think ""Settings"", ""Configuration"", ""Metadata"", and ""Authentication and permissions"" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist?
",107914493
issue_comments,1714920708,IC_kwDOCGYnMM5mN50E,,9599,2023-09-12T03:51:13Z,"Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`:

https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150

I could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works:
```python
for table, column, other_table, other_column in table.foreign_keys:
   # ...
```",140912432
issue_comments,1714919806,IC_kwDOCGYnMM5mN5l-,,9599,2023-09-12T03:49:41Z,"Digging in a bit more:
```pycon
>>> pprint(list(db.query('PRAGMA foreign_key_list(courses)')))
[{'from': 'campus_name',
  'id': 0,
  'match': 'NONE',
  'on_delete': 'NO ACTION',
  'on_update': 'NO ACTION',
  'seq': 0,
  'table': 'departments',
  'to': 'campus_name'},
 {'from': 'dept_code',
  'id': 0,
  'match': 'NONE',
  'on_delete': 'NO ACTION',
  'on_update': 'NO ACTION',
  'seq': 1,
  'table': 'departments',
  'to': 'dept_code'}]
```
I think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`.

Right now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540",140912432
issues,1891614971,I_kwDOCGYnMM5wv8D7,Represent compound foreign keys in table.foreign_keys output,9599,2023-09-12T03:48:24Z,"Given this schema:
```sql
CREATE TABLE departments (
    campus_name TEXT NOT NULL,
    dept_code TEXT NOT NULL,
    dept_name TEXT,
    PRIMARY KEY (campus_name, dept_code)
);
CREATE TABLE courses (
    course_code TEXT PRIMARY KEY,
    course_name TEXT,
    campus_name TEXT NOT NULL,
    dept_code TEXT NOT NULL,
    FOREIGN KEY (campus_name, dept_code) REFERENCES departments(campus_name, dept_code)
);
```
The output of `db[""courses""].foreign_keys` right now is:
```
[ForeignKey(table='courses', column='campus_name', other_table='departments', other_column='campus_name'),
 ForeignKey(table='courses', column='dept_code', other_table='departments', other_column='dept_code')]
```
Which suggests two normal foreign keys, not one compound foreign key.",140912432
issue_comments,1714699724,IC_kwDOBm6k_c5mND3M,,9599,2023-09-11T23:01:36Z,"On thinking about this further, I'm fine releasing it as another alpha provided it causes Datasette to error loudly with an explanatory message if you attempt to load `-m metadata.json` at a metadata file that includes `""plugins""` configuration.

Something like this:
```bash
datasette -m metadata.json
```
Outputs:

> Datasette no longer accepts plugin configuration in --metadata. Move your `""plugins""` configuration blocks to a separate file - we suggest calling that `datasette.yaml` - and start Datasette with `datasette -c datasette.yaml`.

For added usability points, let's have it suggest `datasette.json` if they used `metadata.json` and `datasette.yaml` if they tried to use `metadata.yaml`.",107914493
issue_comments,1714544153,IC_kwDOBm6k_c5mMd4Z,,22429695,2023-09-11T20:37:52Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`95.00%`** and project coverage change: **`-0.04%`** :warning:
> Comparison is base [(`a4c96d0`)](https://app.codecov.io/gh/simonw/datasette/commit/a4c96d01b27ce7cd06662a024da3547132a7c412?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`659dcbd`)](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.

> :exclamation: Current head 659dcbd differs from pull request most recent head acca338. Consider uploading reports for the commit acca338 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2183      +/-   ##
==========================================
- Coverage   92.69%   92.66%   -0.04%     
==========================================
  Files          40       40              
  Lines        6025     6039      +14     
==========================================
+ Hits         5585     5596      +11     
- Misses        440      443       +3     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <95.00%> (-0.24%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1891212159,PR_kwDOBm6k_c5aD33C,`datasette.yaml` plugin support,15178711,2023-09-11T20:26:04Z,"Part of #2093

In #2149 , we ported over `""settings.json""` into the new `datasette.yaml` config file, with a top-level `""settings""` key. This PR ports over plugin configuration into top-level `""plugins""` key, as well as nested database/table plugin config.

From now on, no plugin-related configuration is allowed in `metadata.yaml`, and must be in `datasette.yaml` in this new format. This is a pretty significant breaking change. Thankfully, you should be able to copy-paste your legacy plugin key/values into the new `datasette.yaml` format.

An example of what `datasette.yaml` would look like with this new plugin config:

```yaml

plugins:
  datasette-my-plugin:
    config_key: value

databases:
  fixtures:
      plugins: 
        datasette-my-plugin:
          config_key: fixtures-db-value
    tables:
      students:
        plugins:
          datasette-my-plugin:
            config_key: fixtures-students-table-value

```

As an additional benefit, this now works with the new `-s` flag:

```bash
datasette --memory -s 'plugins.datasette-my-plugin.config_key' new_value
```


Marked as a ""Draft"" right now until I add better documentation. We also should have a plan for the next alpha release to document and publicize this change, especially for plugin authors (since their docs will have to change to say `datasette.yaml` instead of `metadata.yaml`

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2183.org.readthedocs.build/en/2183/

<!-- readthedocs-preview datasette end -->",107914493
issues,1890593563,PR_kwDOBm6k_c5aBx3g,Bump the python-packages group with 2 updates,49699333,2023-09-11T14:01:25Z,"Bumps the python-packages group with 2 updates: [furo](https://github.com/pradyunsg/furo) and [black](https://github.com/psf/black).

Updates `furo` from 2023.8.19 to 2023.9.10
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.09.10 -- Zesty Zaffre</h2>
<ul>
<li>Make asset hash injection idempotent, fixing Sphinx 6 compatibility.</li>
<li>Fix the check for HTML builders, fixing non-HTML Read the Docs builds.</li>
</ul>
<h2>2023.08.19 -- Xenolithic Xanadu</h2>
<ul>
<li>Fix missing search context with Sphinx 7.2, for dirhtml builds.</li>
<li>Drop support for Python 3.7.</li>
<li>Present configuration errors in a better format -- thanks <a href=""https://github.com/AA-Turner""><code>@​AA-Turner</code></a>!</li>
<li>Bump <code>require_sphinx()</code> to Sphinx 6.0, in line with dependency changes in Unassuming Ultramarine.</li>
</ul>
<h2>2023.08.17 -- Wonderous White</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.2.0 and 7.2.1.</li>
</ul>
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/2718ca42930f7c8c11bc96341e0d3db6f532b08d""><code>2718ca4</code></a> Prepare release: 2023.09.10</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c22c99d931decfa4641e400e428606bc6184af56""><code>c22c99d</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c37e84961d9ffbc77ab7be268081d1b3497f0311""><code>c37e849</code></a> Quote a not-runtime-generic type annotation</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/9cfdf44784dc90085549490ff1eddd2afc37c1d6""><code>9cfdf44</code></a> Rework infrastructure for linting</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/5abeb9fc962aee61eda02e0c75e872004635e9d5""><code>5abeb9f</code></a> Fix the check for HTML builders</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/ee2ab5459ddb0a32c3467b88c9c628611eb55073""><code>ee2ab54</code></a> Tweak how tests are run with nox</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/cdae2366c340695ba724ca8438a4cd1e605c8848""><code>cdae236</code></a> Test against Sphinx minor versions in CI</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/9e40071eb8c4c3e38845b57c4f5242bef2a44af2""><code>9e40071</code></a> Make asset hash injection idempotent</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/aab86f4624b6ef38a151440be5624746d41882b1""><code>aab86f4</code></a> Revert &quot;Exclude incompatible Sphinx releases (<a href=""https://redirect.github.com/pradyunsg/furo/issues/711"">#711</a>)&quot;</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/4dd6eec9e306b5fd0624ec8d1d90c6ad416c5182""><code>4dd6eec</code></a> Exclude incompatible Sphinx releases (<a href=""https://redirect.github.com/pradyunsg/furo/issues/711"">#711</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.08.19...2023.09.10"">compare view</a></li>
</ul>
</details>
<br />

Updates `black` from 23.7.0 to 23.9.1
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>23.9.1</h2>
<p>Due to various issues, the previous release (23.9.0) did not include compiled mypyc
wheels, which make Black significantly faster. These issues have now been fixed, and
this release should come with compiled wheels once again.</p>
<p>There will be no wheels for Python 3.12 due to a bug in mypyc. We will provide 3.12
wheels in a future release as soon as the mypyc bug is fixed.</p>
<h3>Packaging</h3>
<ul>
<li>Upgrade to mypy 1.5.1 (<a href=""https://redirect.github.com/psf/black/issues/3864"">#3864</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Store raw tuples instead of NamedTuples in Black's cache, improving performance and
decreasing the size of the cache (<a href=""https://redirect.github.com/psf/black/issues/3877"">#3877</a>)</li>
</ul>
<h2>23.9.0</h2>
<h3>Preview style</h3>
<ul>
<li>More concise formatting for dummy implementations (<a href=""https://redirect.github.com/psf/black/issues/3796"">#3796</a>)</li>
<li>In stub files, add a blank line between a statement with a body (e.g an
<code>if sys.version_info &gt; (3, x):</code>) and a function definition on the same level (<a href=""https://redirect.github.com/psf/black/issues/3862"">#3862</a>)</li>
<li>Fix a bug whereby spaces were removed from walrus operators within subscript(<a href=""https://redirect.github.com/psf/black/issues/3823"">#3823</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Black now applies exclusion and ignore logic before resolving symlinks (<a href=""https://redirect.github.com/psf/black/issues/3846"">#3846</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Avoid importing <code>IPython</code> if notebook cells do not contain magics (<a href=""https://redirect.github.com/psf/black/issues/3782"">#3782</a>)</li>
<li>Improve caching by comparing file hashes as fallback for mtime and size (<a href=""https://redirect.github.com/psf/black/issues/3821"">#3821</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Fix an issue in <code>blackd</code> with single character input (<a href=""https://redirect.github.com/psf/black/issues/3558"">#3558</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Black now has an
<a href=""https://github.com/psf/black-pre-commit-mirror"">official pre-commit mirror</a>. Swapping
<code>https://github.com/psf/black</code> to <code>https://github.com/psf/black-pre-commit-mirror</code> in
your <code>.pre-commit-config.yaml</code> will make Black about 2x faster (<a href=""https://redirect.github.com/psf/black/issues/3828"">#3828</a>)</li>
<li>The <code>.black.env</code> folder specified by <code>ENV_PATH</code> will now be removed on the completion
of the GitHub Action (<a href=""https://redirect.github.com/psf/black/issues/3759"">#3759</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>23.9.1</h2>
<p>Due to various issues, the previous release (23.9.0) did not include compiled mypyc
wheels, which make Black significantly faster. These issues have now been fixed, and
this release should come with compiled wheels once again.</p>
<p>There will be no wheels for Python 3.12 due to a bug in mypyc. We will provide 3.12
wheels in a future release as soon as the mypyc bug is fixed.</p>
<h3>Packaging</h3>
<ul>
<li>Upgrade to mypy 1.5.1 (<a href=""https://redirect.github.com/psf/black/issues/3864"">#3864</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Store raw tuples instead of NamedTuples in Black's cache, improving performance and
decreasing the size of the cache (<a href=""https://redirect.github.com/psf/black/issues/3877"">#3877</a>)</li>
</ul>
<h2>23.9.0</h2>
<h3>Preview style</h3>
<ul>
<li>More concise formatting for dummy implementations (<a href=""https://redirect.github.com/psf/black/issues/3796"">#3796</a>)</li>
<li>In stub files, add a blank line between a statement with a body (e.g an
<code>if sys.version_info &gt; (3, x):</code>) and a function definition on the same level (<a href=""https://redirect.github.com/psf/black/issues/3862"">#3862</a>)</li>
<li>Fix a bug whereby spaces were removed from walrus operators within subscript(<a href=""https://redirect.github.com/psf/black/issues/3823"">#3823</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>Black now applies exclusion and ignore logic before resolving symlinks (<a href=""https://redirect.github.com/psf/black/issues/3846"">#3846</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Avoid importing <code>IPython</code> if notebook cells do not contain magics (<a href=""https://redirect.github.com/psf/black/issues/3782"">#3782</a>)</li>
<li>Improve caching by comparing file hashes as fallback for mtime and size (<a href=""https://redirect.github.com/psf/black/issues/3821"">#3821</a>)</li>
</ul>
<h3><em>Blackd</em></h3>
<ul>
<li>Fix an issue in <code>blackd</code> with single character input (<a href=""https://redirect.github.com/psf/black/issues/3558"">#3558</a>)</li>
</ul>
<h3>Integrations</h3>
<ul>
<li>Black now has an
<a href=""https://github.com/psf/black-pre-commit-mirror"">official pre-commit mirror</a>. Swapping
<code>https://github.com/psf/black</code> to <code>https://github.com/psf/black-pre-commit-mirror</code> in
your <code>.pre-commit-config.yaml</code> will make Black about 2x faster (<a href=""https://redirect.github.com/psf/black/issues/3828"">#3828</a>)</li>
<li>The <code>.black.env</code> folder specified by <code>ENV_PATH</code> will now be removed on the completion
of the GitHub Action (<a href=""https://redirect.github.com/psf/black/issues/3759"">#3759</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/psf/black/commit/e87737140f32d3cd7c44ede75f02dcd58e55820e""><code>e877371</code></a> Prepare release 23.9.1 (<a href=""https://redirect.github.com/psf/black/issues/3878"">#3878</a>)</li>
<li><a href=""https://github.com/psf/black/commit/62dca32dc55a850f39d78ba8b9c21cc4261a2ddf""><code>62dca32</code></a> mypyc builds on PRs, skip mypyc wheels for 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3870"">#3870</a>)</li>
<li><a href=""https://github.com/psf/black/commit/751583a1dfc691423d96d7711a4c8d9cfe3ee9c8""><code>751583a</code></a> Pickle raw tuples in FileData cache (<a href=""https://redirect.github.com/psf/black/issues/3877"">#3877</a>)</li>
<li><a href=""https://github.com/psf/black/commit/f7917453c99f8183ffd0397affcccb2c37594771""><code>f791745</code></a> Re-export black.Mode (<a href=""https://redirect.github.com/psf/black/issues/3875"">#3875</a>)</li>
<li><a href=""https://github.com/psf/black/commit/0b62b9c9a44a995e44d64ecf7cc08d8d7037642d""><code>0b62b9c</code></a> Ignore aiohttp DeprecationWarning for 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3876"">#3876</a>)</li>
<li><a href=""https://github.com/psf/black/commit/c83ad6c077e7bb281cfd3fbdd89bbeb4c980e563""><code>c83ad6c</code></a> Upgrade to Furo 2023.9.10 to fix docs build (<a href=""https://redirect.github.com/psf/black/issues/3873"">#3873</a>)</li>
<li><a href=""https://github.com/psf/black/commit/4eebfd1a7a4aa2652cfc674cf301d1f2f48098aa""><code>4eebfd1</code></a> Add mypyc test marks to new tests that patch (<a href=""https://redirect.github.com/psf/black/issues/3871"">#3871</a>)</li>
<li><a href=""https://github.com/psf/black/commit/add161b367a0d5a3cc395ec8e045f7b965edaef8""><code>add161b</code></a> Bump RTD Python version from 3.8 to 3.11 (<a href=""https://redirect.github.com/psf/black/issues/3868"">#3868</a>)</li>
<li><a href=""https://github.com/psf/black/commit/4e93f2aa01606154dc6af6e494b9f2b7e4c8c7fa""><code>4e93f2a</code></a> Add classifier for 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3866"">#3866</a>)</li>
<li><a href=""https://github.com/psf/black/commit/716fa08090b6a51e4c72dd0a14b6c45f7da4a9d0""><code>716fa08</code></a> Upgrade mypy (<a href=""https://redirect.github.com/psf/black/issues/3864"">#3864</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/psf/black/compare/23.7.0...23.9.1"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2182.org.readthedocs.build/en/2182/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1712897194,IC_kwDOBm6k_c5mGLyq,,9599,2023-09-10T17:54:07Z,"This looks relevant:

https://pluggy.readthedocs.io/en/stable/#wrappers

> A *hookimpl* can be marked with the `""wrapper""` option, which indicates that the function will be called to *wrap* (or surround) all other normal *hookimpl* calls. A *hook wrapper* can thus execute some code ahead and after the execution of all corresponding non-wrappper *hookimpls*.

This could be the perfect mechanism for implementing this hook, although I still need to figure out how it interacts with streaming.",107914493
issue_comments,1712895580,IC_kwDOCGYnMM5mGLZc,,9599,2023-09-10T17:46:41Z,"In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`:

https://www.sqlite.org/lang_vacuum.html#how_vacuum_works

> The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid).

So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.",140912432
issue_comments,1712895084,IC_kwDOBm6k_c5mGLRs,,9599,2023-09-10T17:44:14Z,Used by `datasette-short-links` here: https://github.com/datasette/datasette-short-links/blob/468c3e25dbe06a8dcba8edda59bc16a18e126a51/datasette_short_links/__init__.py#L108-L115,107914493
issues,1888477283,I_kwDOC8SPRc5wj-Bj,Run `rebuild_fts` after building the index,9599,2023-09-08T23:17:45Z,"In:
- https://github.com/simonw/datasette.io/issues/152#issuecomment-1712323347

This turned out to be the fix:

```bash
dogsheep-beta index dogsheep-index.db templates/dogsheep-beta.yml
sqlite-utils rebuild-fts dogsheep-index.db
```",197431109
issue_comments,1711057080,IC_kwDOBm6k_c5l_Ki4,,9599,2023-09-08T04:28:03Z,"Landed:

- https://docs.datasette.io/en/latest/plugin_hooks.html#actors-from-ids-datasette-actor-ids
- https://docs.datasette.io/en/latest/internals.html#await-actors-from-ids-actor-ids",107914493
issue_comments,1711054840,IC_kwDOBm6k_c5l_J_4,,9599,2023-09-08T04:23:47Z,"I've implemented this hook once now in:
- https://github.com/datasette/datasette-remote-actors

And built and tested a debug tool for it in:
- https://github.com/datasette/datasette-debug-actors-from-ids

I'm now confident in the design of this plugin hook, I'm going to land it.",107914493
issue_comments,1711054624,IC_kwDOBm6k_c5l_J8g,,9599,2023-09-08T04:23:23Z,"I've implemented this hook once now in:
- https://github.com/datasette/datasette-remote-actors

And built and tested a debug tool for it in:
- https://github.com/datasette/datasette-debug-actors-from-ids

I'm now confident in the design of this plugin hook, I'm going to land it.",107914493
issue_comments,1711028355,IC_kwDOBm6k_c5l_DiD,,9599,2023-09-08T03:36:02Z,"I shipped the first version of 

- https://github.com/datasette/datasette-remote-actors/issues/1

When I land this plugin in a Datasette release I need to update that repo to depend on the new alpha.",107914493
issue_comments,1710972324,IC_kwDOBm6k_c5l-12k,,22429695,2023-09-08T01:58:44Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and no project coverage change.
> Comparison is base [(`c263704`)](https://app.codecov.io/gh/simonw/datasette/commit/c26370485a4fd4bf130da051be9163d92c57f24f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`e1c5a9d`)](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2181   +/-   ##
=======================================
  Coverage   92.68%   92.69%           
=======================================
  Files          40       40           
  Lines        6017     6025    +8     
=======================================
+ Hits         5577     5585    +8     
  Misses        440      440           
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.43% <100.00%> (+0.03%)` | :arrow_up: |
| [datasette/hookspecs.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   

:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).",107914493
issue_comments,1710969448,IC_kwDOBm6k_c5l-1Jo,,9599,2023-09-08T01:53:27Z,"Documentation preview:
- https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids
- https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids",107914493
issue_comments,1710969339,IC_kwDOBm6k_c5l-1H7,,9599,2023-09-08T01:53:16Z,"Documentation preview:
- https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids
- https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids",107914493
issues,1886812002,PR_kwDOBm6k_c5Z1N2L,actors_from_ids plugin hook and datasette.actors_from_ids() method,9599,2023-09-08T01:51:07Z,"Refs:
- #2180

This plugin hook is feature complete - including documentation and tests.

I'm not going to land it in Datasette `main` until we've used it at least once though, which should happen promptly in development for [Datasette Cloud](https://www.datasette.cloud/).

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2181.org.readthedocs.build/en/2181/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1710950671,IC_kwDODFE5qs5l-wkP,,150855,2023-09-08T01:22:49Z,"Makes sense, thanks for explaining!",206649770
issue_comments,1710947637,IC_kwDOBm6k_c5l-v01,,9599,2023-09-08T01:17:28Z,I think this is both a plugin hook and a `await datasette.actors_from_ids(actor_ids)` internal API function that calls it.,107914493
issues,1886791100,I_kwDOBm6k_c5wdiW8,Plugin hook: `actors_from_ids()`,9599,2023-09-08T01:16:41Z,"In building Datasette Cloud we realized that a bunch of the features we are building need a way of resolving an actor ID to the actual actor, in order to display something more interesting than just an integer ID.

Social plugins in particular need this - comments by X, CSV uploaded by X, that kind of thing.

I think the solution is a new plugin hook: `actors_from_ids(datasette, ids)` which can return a list of actor dictionaries.

The default implementation can return `[{""id"": ""...""}]` for the IDs passed to it.

Pluggy has a `firstresult=True` option which is relevant here, since this is the first plugin hook we will have implemented where only one plugin should provide an answer.",107914493
issue_comments,1710939868,IC_kwDOCGYnMM5l-t7c,,22429695,2023-09-08T01:03:40Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and no project coverage change.
> Comparison is base [(`5d123f0`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/5d123f031fc4fadc98f508e0ef6b7b6671e86155?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b86374f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #593   +/-   ##
=======================================
  Coverage   95.77%   95.77%           
=======================================
  Files           8        8           
  Lines        2837     2840    +3     
=======================================
+ Hits         2717     2720    +3     
  Misses        120      120           
```


| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <100.00%> (+<0.01%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issues,1886783150,PR_kwDOCGYnMM5Z1H1d,".transform() now preserves rowid values, refs #592",9599,2023-09-08T01:02:28Z,"Refs:
- #592

- [x] Tests against weird shaped tables

I need to test that this works against:

- `rowid` tables
- Tables that have a column called `rowid` even though they are not rowid tables


<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--593.org.readthedocs.build/en/593/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1710935270,IC_kwDOCGYnMM5l-szm,,9599,2023-09-08T00:55:30Z,"Yes! That recreated the bug:
```
>       assert previous_rows == next_rows
E       AssertionError: assert equals failed
E         [                                                                [                                                               
E           (1, '1', 'Paris'),                                               (1, '1', 'Paris'),                                            
E           (3, '3', 'New York'),                                            (2, '3', 'New York'),                                         
E           (4, '4', 'London'),                                              (3, '4', 'London'),                                           
E         ]                                                       ...
E         
```",140912432
issue_comments,1710934448,IC_kwDOCGYnMM5l-smw,,9599,2023-09-08T00:54:03Z,"Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3.

If I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.",140912432
issue_comments,1710933716,IC_kwDOCGYnMM5l-sbU,,9599,2023-09-08T00:52:42Z,"I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key.

The reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows.

But I tried this test and it passed, too:
```python
def test_transform_preserves_rowids(fresh_db):
    fresh_db[""places""].insert_all(
        [
            {""id"": ""1"", ""name"": ""Paris"", ""country"": ""France""},
            {""id"": ""2"", ""name"": ""London"", ""country"": ""UK""},
            {""id"": ""3"", ""name"": ""New York"", ""country"": ""USA""},
        ],
        pk=""id"",
    )
    previous_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"")
    )
    # Transform it
    fresh_db[""places""].transform(column_order=(""country"", ""name""))
    # Should be the same
    next_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"")
    )
    assert previous_rows == next_rows
```",140912432
issue_comments,1710931605,IC_kwDOCGYnMM5l-r6V,,9599,2023-09-08T00:49:02Z,"I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.",140912432
issue_comments,1710930934,IC_kwDOCGYnMM5l-rv2,,9599,2023-09-08T00:47:57Z,"That's odd, I wrote a test for this just now and it passes already:
```python
def test_transform_preserves_rowids(fresh_db):
    # Create a rowid table
    fresh_db[""places""].insert_all(
        (
            {""name"": ""Paris"", ""country"": ""France""},
            {""name"": ""London"", ""country"": ""UK""},
            {""name"": ""New York"", ""country"": ""USA""},
        ),
    )
    assert fresh_db[""places""].use_rowid
    previous_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, name from places"")
    )
    # Transform it
    fresh_db[""places""].transform(column_order=(""country"", ""name""))
    # Should be the same
    next_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, name from places"")
    )
    assert previous_rows == next_rows
```
So maybe I'm wrong about the cause of that bug?",140912432
issues,1886771493,I_kwDOCGYnMM5wddkl,`table.transform()` should preserve `rowid` values,9599,2023-09-08T00:42:38Z,"I just spotted a bug when using https://datasette.io/plugins/datasette-configure-fts and https://datasette.io/plugins/datasette-edit-schema at the same time.

Steps to reproduce:

- Configure FTS for a table, then run a test search
- Edit the schema for that table and change the order of columns
- Run the test search again

I got the wrong search results, which I think is because the `_fts` table pointed to the first table by `rowid` but those `rowid` values were entirely rewritten as a consequence of running `table.transform()` on the table.

Reconfiguring FTS on the table fixed the problem.

I think `table.transform()` should be able to preserve `rowid` values.",140912432
issue_comments,1710879239,IC_kwDOBm6k_c5l-fIH,,9599,2023-09-07T23:20:32Z,"To test that locally, use this YAML instead:
```yaml
databases:
  content:
    allow:
      id: root
    tables:
      releases:
        allow: true
```
And:
```yaml
allow:
  id: root
databases:
  content:
    tables:
      releases:
        allow: true",107914493
issue_comments,1710878391,IC_kwDOBm6k_c5l-e63,,9599,2023-09-07T23:19:05Z,"This fix didn't work on Datasette Cloud. I used `/-/permissions` to debug it and saw this:

![image](https://github.com/simonw/datasette/assets/9599/61d2bc5f-1f96-41ea-8658-91dfbcb6610c)

Only checking `view-table` is not enough: for my instance on Datasette Cloud the view permission check that should have failed was for the database or instance.",107914493
issue_comments,1710871095,IC_kwDOBm6k_c5l-dI3,,9599,2023-09-07T23:07:16Z,"I ran this:

    datasette content.db -p 8043 -m fk-auth.yml --root

Against this YAML:
```yaml
databases:
  content:
    tables:
      users:
        allow:
          id: root
```
And it worked as it should - here's a screenshot of an anonymous user and a root user viewing the same page:

![CleanShot 2023-09-07 at 16 05 34@2x](https://github.com/simonw/datasette/assets/9599/3e91da08-107c-421c-8a00-aa650b960c58)
",107914493
issues,1886649402,I_kwDOBm6k_c5wc_w6,Flaky test: test_hidden_sqlite_stat1_table,9599,2023-09-07T22:48:43Z,"This test here: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/tests/test_api.py#L1011-L1020

It failed for me like this:

`E       AssertionError: assert [('normal', False), ('sqlite_stat1', True), ('sqlite_stat4', True)] in ([('normal', False), ('sqlite_stat1', True)],)`

Looks like some builds of SQLite include a `sqlite_stat4` table.",107914493
issue_comments,1710567329,IC_kwDOBm6k_c5l9S-h,,9599,2023-09-07T17:59:59Z,Should I put the permission check in that undocumented `datasette.expand_foreign_keys()` method? I think so - it should accept `request.actor` as one of its arguments.,107914493
issue_comments,1710565268,IC_kwDOBm6k_c5l9SeU,,9599,2023-09-07T17:58:04Z,"Relevant code: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/views/table.py#L1132-L1149

Which calls this undocumented method:

https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/app.py#L938-L973",107914493
issues,1886350562,I_kwDOBm6k_c5wb2zi,Don't show foreign key links to tables the user cannot access,9599,2023-09-07T17:56:41Z,"Spotted this problem while working on this plugin:
- https://github.com/simonw/datasette-public

It's possible to make a table public to any users - but then you may end up with situations like this:

<img width=""1095"" alt=""CleanShot 2023-09-07 at 10 55 02@2x"" src=""https://github.com/simonw/datasette/assets/9599/514a00be-f399-4b08-aa5e-f07d603b5ffe"">

That table is public, but the foreign key links go to tables that are NOT public.

We're also leaking the names of the values in those private tables here, which we shouldn't do. So this is a tiny bit of an information leak.

Since this only affects people who have configured a table to be public that has foreign keys to a table that is private I don't think this is worth issuing a vulnerability report about - I very much doubt anyone is running Datasette configured in a way that could result in problems because of this.",107914493
issue_comments,1710380941,IC_kwDODFE5qs5l8leN,,28565,2023-09-07T15:39:59Z,"> @maxhawkins curious why you didn't use the stdlib `mailbox` to parse the `mbox` files?

Mailbox parses the entire mbox into memory. Using the lower level library lets us stream the emails in one at a time to support larger archives. Both libraries are in the stdlib.",206649770
issue_comments,1709373304,IC_kwDOBm6k_c5l4vd4,,363004,2023-09-07T02:14:15Z,"I ran into the same issue on Windows using 

`datasette publish cloudrun mydatabase.db --service=my-database`
do do a [google cloud publish](https://docs.datasette.io/en/stable/publish.html).

@Rik-de-Kort your fix worked perfectly! Thanks!

I can always go back and delete the temp directories myself :)
",107914493
issue_comments,1708945716,IC_kwDODFE5qs5l3HE0,,150855,2023-09-06T19:12:33Z,@maxhawkins curious why you didn't use the stdlib `mailbox` to parse the `mbox` files?,206649770
issues,1884499674,PR_kwDODFE5qs5ZtYMc,"use poetry for packages, asdf for versioning, and gh actions for ci",150855,2023-09-06T17:59:16Z,"- build: use poetry for package management, asdf for python version
- build: cleanup poetry config, add keywords, ignore dist
- ci: migrate circleci to gh actions
- fix: dup method definition
",206649770
issue_comments,1708777964,IC_kwDOBm6k_c5l2eHs,,9599,2023-09-06T17:04:30Z,"Here's the thinking: setting the`DATASETTE_INTERNAL` environment variable for your entire machine will cause any time you run `datasette x.db` to result in a process that shares the same `internal.db` database as other instances.

You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:

https://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62

The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.

This convinced us that actually we should move those `catalog_` tables OUT of `internal.db`. The `_internal` database will be reserved for plugins that want to use it for caching, storing progress, etc.

I think we move them to an in-memory `_catalog` database which is excluded from `ds.databases` (like `_internal` is ) but can be accessed using `datasette.get_catalog_database()` - similar to `datasette.get_internal_database()`.

So each instance of Datasette gets its own truly private `_catalog`, which is in-memory and so gets cleared at the end of each process.

An interesting thing that came up about a shared `_internal` database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.",107914493
issues,1884408624,I_kwDOBm6k_c5wUcsw,Move schema tables from _internal to _catalog,9599,2023-09-06T16:58:33Z,"This came up in discussion over:
- https://github.com/simonw/datasette/pull/2174

",107914493
issue_comments,1708728926,IC_kwDOBm6k_c5l2SJe,,9599,2023-09-06T16:36:56Z,`DATASETTE_INTERNAL` would be more consistent with `DATASETTE_SECRET`.,107914493
issue_comments,1708727204,IC_kwDOBm6k_c5l2Ruk,,9599,2023-09-06T16:35:39Z,"We can use this here: https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables
```python
@click.option(..., envvar=""DATASETTE_INTERNAL"")
```",107914493
issue_comments,1708699926,IC_kwDOBm6k_c5l2LEW,,22429695,2023-09-06T16:18:13Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`d75b519`)](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
> Report is 7 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2174   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          40       40           
  Lines        6012     6017    +5     
=======================================
+ Hits         5572     5577    +5     
  Misses        440      440           
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <ø> (ø)` | |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2174/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   

:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).",107914493
issue_comments,1708695907,IC_kwDOCGYnMM5l2KFj,,9599,2023-09-06T16:15:59Z,"The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant:
- https://github.com/pypa/setuptools/issues/3661

```
25h  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
...
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py"", line 697, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py"", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py"", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""<frozen importlib._bootstrap>"", line 1381, in _gcd_import
  File ""<frozen importlib._bootstrap>"", line 1354, in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1304, in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 488, in _call_with_frames_removed
  File ""<frozen importlib._bootstrap>"", line 1381, in _gcd_import
  File ""<frozen importlib._bootstrap>"", line 1354, in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1325, in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 929, in _load_unlocked
  File ""<frozen importlib._bootstrap_external>"", line 994, in exec_module
  File ""<frozen importlib._bootstrap>"", line 488, in _call_with_frames_removed
  File ""/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py"", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'
```",140912432
issue_comments,1708693020,IC_kwDOCGYnMM5l2JYc,,22429695,2023-09-06T16:14:03Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`347fdc8`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/347fdc865e91b8d3410f49a5c9d5b499fbb594c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74% compared to head [(`1f14df1`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #591   +/-   ##
=======================================
  Coverage   95.74%   95.74%           
=======================================
  Files           8        8           
  Lines        2842     2842           
=======================================
  Hits         2721     2721           
  Misses        121      121           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issues,1884335789,PR_kwDOCGYnMM5Zs0KB,Test against Python 3.12 preview,9599,2023-09-06T16:10:00Z,"https://dev.to/hugovk/help-test-python-312-beta-1508/

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--591.org.readthedocs.build/en/591/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issues,1884333600,PR_kwDOBm6k_c5Zszqk,Test against Python 3.12 preview,9599,2023-09-06T16:09:05Z,"https://dev.to/hugovk/help-test-python-312-beta-1508/

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2175.org.readthedocs.build/en/2175/

<!-- readthedocs-preview datasette end -->",107914493
issues,1884330740,PR_kwDOBm6k_c5ZszDF,Use $DATASETTE_INTERNAL in absence of --internal,15178711,2023-09-06T16:07:15Z,"#refs 2157, specifically [this comment](https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967)

Passing in `--internal my_internal.db` over and over again can get repetitive. 

This PR adds a new configurable env variable `DATASETTE_INTERNAL_DB_PATH`. If it's defined, then it takes place as the path to the internal database. Users can still overwrite this behavior by passing in their own `--internal internal.db` flag.

In draft mode for now, needs tests and documentation. 

Side note: Maybe we can have a sections in the docs that lists all the ""configuration environment variables"" that Datasette respects? I did a quick grep and found:

- `DATASETTE_LOAD_PLUGINS`
- `DATASETTE_SECRETS`


<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2174.org.readthedocs.build/en/2174/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1707570378,IC_kwDOBm6k_c5lx3TK,,9599,2023-09-06T02:40:31Z,"Surprising error:
```
  blacken-docs -l 60 docs/*.rst
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib
docs/writing_plugins.rst:365: code block parse error Cannot parse: 8:0: <line number missing in source>
```",107914493
issue_comments,1707570044,IC_kwDOBm6k_c5lx3N8,,22429695,2023-09-06T02:39:59Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`fd083e3`)](https://app.codecov.io/gh/simonw/datasette/commit/fd083e37ec53e7e625111168d324a572344a3b19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`a481ebd`)](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2173   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          40       40           
  Lines        6012     6012           
=======================================
  Hits         5572     5572           
  Misses        440      440           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1707565495,IC_kwDOBm6k_c5lx2G3,,9599,2023-09-06T02:33:56Z,Running tests to see if Pyodide works - that was the reason I switched to my `click-default-group-wheel` package originally.,107914493
issues,1883055640,PR_kwDOBm6k_c5ZociX,click-default-group>=1.2.3,9599,2023-09-06T02:33:28Z,"Now available as a wheel:
- https://github.com/click-contrib/click-default-group/issues/21

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2173.org.readthedocs.build/en/2173/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1706701195,IC_kwDOBm6k_c5lujGL,,7983005,2023-09-05T14:10:39Z,"Hey @simonw I hit the same issue as mentionned by @chmaynard on a fresh install, ""/mnt/fixtures.db"" doesn't seem to exist in the docker image",107914493
issues,1880968405,PR_kwDOJHON9s5ZhYny,fix: fix the problem of Chinese character garbling,2698003,2023-09-04T23:48:28Z,"1. The code uses two different ways of writing encoding formats, `mac_roman` and `macroman`. It is uncertain whether there are any typo errors.
2. When there are Chinese characters in the content, exporting it results in garbled code. Changing it to `utf8` can fix the issue.",611552758
issue_comments,1704387161,IC_kwDOCGYnMM5lluJZ,,9599,2023-09-03T19:50:36Z,"Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.",140912432
issues,1879214365,I_kwDOCGYnMM5wAokd,Ability to tell if a Database is an in-memory one,9599,2023-09-03T19:50:15Z,"Currently the constructor accepts `memory=True` or `memory_name=...` and uses those to create a connection, but does not record what those values were:

https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L307-L349

This makes it hard to tell if a database object is to an in-memory or a file-based database, which is sometimes useful to know.",140912432
issue_comments,1704384393,IC_kwDOCGYnMM5llteJ,,9599,2023-09-03T19:36:34Z,"Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b

Still needs tests and documentation (and some more thought to make sure it's doing the right thing).",140912432
issue_comments,1704384111,IC_kwDOCGYnMM5lltZv,,9599,2023-09-03T19:35:03Z,"Normally in Python/`sqlite3` you de-register a function by passing `None` to it.

You can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else:

https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464",140912432
issue_comments,1704383901,IC_kwDOCGYnMM5lltWd,,9599,2023-09-03T19:34:05Z,"For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager.

I played with this idea a bit:

```python
with db.register_functions(md5, md5_random):
    db.query(...)
```
",140912432
issues,1879209560,I_kwDOCGYnMM5wAnZY,Mechanism for de-registering registered SQL functions,9599,2023-09-03T19:32:39Z,I used a custom SQL function in a migration script and then realized that it should be de-registered before the end of the script to avoid leaking into the calling code.,140912432
issue_comments,1701894468,IC_kwDOBm6k_c5lcNlE,,9599,2023-08-31T23:08:24Z,https://docs.datasette.io/en/latest/writing_plugins.html#plugins-that-define-new-plugin-hooks,107914493
issues,1876407598,I_kwDOBm6k_c5v17Uu,execute-sql on a database should imply view-database/view-permission,9599,2023-08-31T22:45:56Z,"I noticed that a token with `execute-sql` permission alone did not work, because it was not allowed to view the instance of the database.",107914493
issue_comments,1701831013,IC_kwDOBm6k_c5lb-Fl,,9599,2023-08-31T21:51:12Z,"Need to make sure the design of this takes streaming responses into account. Those could be pretty tricky here.

I nice thing about `asgi_wrapper()` is that it handles those already.",107914493
issue_comments,1701830241,IC_kwDOBm6k_c5lb95h,,9599,2023-08-31T21:50:18Z,"The hook could be called `register_middleware()` and could work like `register_routes()` and `register_commands()`:

```python
@hookspec
def register_middleware(datasette):
    """"""Register middleware: returns a list of async def middleware functions""""""

```",107914493
issue_comments,1701828197,IC_kwDOBm6k_c5lb9Zl,,9599,2023-08-31T21:48:00Z,"A pattern like this could be interesting:
```python
async def my_middleware(datasette, request, get_response):
    # Mess with request here if neccessary
    response = await get_response(request)
    # mess with response
    return response
```
The Django pattern is more complicated but does have that mechanism for running one-time configuration prior to defining the `middleware()` function, which is neat.",107914493
issue_comments,1701826521,IC_kwDOBm6k_c5lb8_Z,,9599,2023-08-31T21:46:13Z,"This could even be a pair of hooks - `process_request()` and `process_response()`.

Or could take a leaf from Django, which redesigned middleware to use this pattern instead:

```python
def simple_middleware(get_response):
    # One-time configuration and initialization.
    def middleware(request):
        # Code to be executed for each request before
        # the view (and later middleware) are called.
        response = get_response(request)
        # Code to be executed for each request/response after
        # the view is called.
        return response
    return middleware
```
Or even borrow an idea from `pytest` where fixtures can `yield` in the middle, like this:
```python
@pytest.fixture
def sending_user(mail_admin):
    user = mail_admin.create_user()
    yield user
    mail_admin.delete_user(user)
```",107914493
issue_comments,1701823609,IC_kwDOBm6k_c5lb8R5,,9599,2023-08-31T21:43:06Z,"Not sure what to call this. Maybe `app_wrapper()`?

Or perhaps it's simpler than that, something like this:

```python
@hookspec
def process_response(datasette, request, response):
    """"""Last chance to modify the response before it is returned to the client""""""
```",107914493
issues,1876353656,I_kwDOBm6k_c5v1uJ4,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),9599,2023-08-31T21:42:04Z,"There's a long justification for why this might be needed here:
- https://github.com/simonw/datasette-auth-tokens/issues/10#issuecomment-1701820001

Short version: it would be neat if it was possible to stash some data on the `request` object such that a later plugin/middleware-type-thing could use that to influence the final returned response - similar to the kinds of things you can do with Django middleware.

The `asgi_wrapper()` mechanism doesn't have access to the request or response objects - it gets `scope` and can mess around with `receive` and `send`, but those are pretty low-level primitives.

Since Datasette has well-defined `request` and `response` objects now it might be nice to have a middleware layer that can manipulate those directly.",107914493
issues,1875739055,I_kwDOBm6k_c5vzYGv,Document return type of await ds.permission_allowed(),9599,2023-08-31T15:14:23Z,"The return type isn't documented here: https://github.com/simonw/datasette/blob/4c3ef033110407f3b3dbce501659d523724985e0/docs/internals.rst#L327-L350

On inspecting the code I'm not 100% sure if it's possible for this. method to return `None`, or if it can only return `True` or `False`. Need to confirm that.

https://github.com/simonw/datasette/blob/4c3ef033110407f3b3dbce501659d523724985e0/datasette/app.py#L822C15-L853",107914493
issue_comments,1701045404,IC_kwDOBm6k_c5lY-Sc,,22429695,2023-08-31T13:31:15Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`7c9df6e`)](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2166   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          40       40           
  Lines        6012     6012           
=======================================
  Hits         5572     5572           
  Misses        440      440           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1875519316,PR_kwDOBm6k_c5ZPO5y,Bump the python-packages group with 1 update,49699333,2023-08-31T13:19:57Z,"Bumps the python-packages group with 1 update: [sphinx](https://github.com/sphinx-doc/sphinx).

<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.2.5</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.2.5 (released Aug 30, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11645"">#11645</a>: Fix a regression preventing autodoc from importing modules within
packages that make use of <code>if typing.TYPE_CHECKING:</code> to guard circular
imports needed by type checkers.
Patch by Matt Wozniski.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11634"">#11634</a>: Fixed inheritance diagram relative link resolution
for sibling files in a subdirectory.
Patch by Albert Shih.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11659"">#11659</a>: Allow <code>?config=...</code> in :confval:<code>mathjax_path</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11654"">#11654</a>: autodoc: Fail with a more descriptive error message
when an object claims to be an instance of <code>type</code>,
but is not a class.
Patch by James Braza.</li>
<li>11620: Cease emitting :event:<code>source-read</code> events for files read via
the :dudir:<code>include</code> directive.</li>
<li>11620: Add a new :event:<code>include-read</code> for observing and transforming
the content of included files via the :dudir:<code>include</code> directive.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11627"">#11627</a>: Restore support for copyright lines of the form <code>YYYY</code>
when <code>SOURCE_DATE_EPOCH</code> is set.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/fcc38997f1d9b728bb4ffc64fc362c7763a4ee25""><code>fcc3899</code></a> Bump to 7.2.5 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/2a631f97ef7f4b0edc2470c9cc2b7d809dd75a87""><code>2a631f9</code></a> Restore support for YYYY copyright lines</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/2730cc3353b06f16786d697c61c8b847a91ddd30""><code>2730cc3</code></a> Remove double spaces in CHANGES</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ff18318613db56d0000db47e5c8f0140556cef0c""><code>ff18318</code></a> Add an <code>'include-read'</code> event (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11657"">#11657</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/74329d9c51c16be54e1f3dece07461f67cfb8ab0""><code>74329d9</code></a> Fail better in <code>ExceptionDocumenter.can_document_member</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11660"">#11660</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/7d046a862f06631367984b87c1e8c2acb518de4f""><code>7d046a8</code></a> Allow <code>?config=...</code> in <code>mathjax_path</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11659"">#11659</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4692208fde90f5a651f303c0550bcf2b7a8a049c""><code>4692208</code></a> Fix two relative link bugs in inheritance diagrams (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11634"">#11634</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ca0fc7aa86cb8acfb4445aab17666a836abaee43""><code>ca0fc7a</code></a> Add git .mailmap file</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/8248be31dbd6f235b9393f39ff16fa9e67dec67c""><code>8248be3</code></a> autodoc: Reset <code>sys.modules</code> on partial import failure (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11645"">#11645</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e494baa9dd3f43ceeab29454f6b433456339da78""><code>e494baa</code></a> Recommend correct replacement names for deprecated APIs (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11655"">#11655</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v7.2.4...v7.2.5"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=7.2.4&new-version=7.2.5)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2166.org.readthedocs.build/en/2166/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1700291967,IC_kwDOBm6k_c5lWGV_,,15178711,2023-08-31T02:45:56Z,"@simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)

My rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH=""/path/to/internal.db""` to their bashrc/zshrc/whereever to not have to worry about `--internal`",107914493
issue_comments,1699926384,IC_kwDOBm6k_c5lUtFw,,9599,2023-08-30T22:17:42Z,"This is implemented now:
- https://docs.datasette.io/en/stable/internals.html#await-ensure-permissions-actor-permissions",107914493
issue_comments,1699925224,IC_kwDOBm6k_c5lUszo,,9599,2023-08-30T22:16:38Z,"This is going to happen in this tool instead:
- https://github.com/simonw/dclient",107914493
issue_comments,1699910555,IC_kwDOBm6k_c5lUpOb,,9599,2023-08-30T22:05:14Z,Documentation preview: https://github.com/simonw/datasette/blob/6321c9c055a640ed6ea98e231dc5813dcde1f773/docs/plugins.rst#controlling-which-plugins-are-loaded,107914493
issue_comments,1699884314,IC_kwDOBm6k_c5lUi0a,,9599,2023-08-30T21:38:29Z,"Here's the reason for that name disparity:

https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L54-L65

Note how the `distinfo.project_name` name is used when available. That seems to work for regularly installed plugins but not for plugins loaded via `DATASETTE_LOAD_PLUGINS`.

And that's looking things up in `plugin_to_distinfo` which is populated here:

https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L37",107914493
issue_comments,1699812599,IC_kwDOBm6k_c5lURT3,,22429695,2023-08-30T20:43:19Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`26.66%`** and project coverage change: **`-0.17%`** :warning:
> Comparison is base [(`30b28c8`)](https://app.codecov.io/gh/simonw/datasette/commit/30b28c8367a9c6870386ea10a202705b40862457?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.84% compared to head [(`5eddf34`)](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.

> :exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2165      +/-   ##
==========================================
- Coverage   92.84%   92.68%   -0.17%     
==========================================
  Files          40       40              
  Lines        5998     6012      +14     
==========================================
+ Hits         5569     5572       +3     
- Misses        429      440      +11     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/plugins.py](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `64.58% <26.66%> (-17.77%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1699811810,IC_kwDOBm6k_c5lURHi,,9599,2023-08-30T20:42:41Z,"The `load_setuptools_entrypoints()` function in Pluggy [does this](https://github.com/pytest-dev/pluggy/blob/0b41c9766508a46ae666cf281684df3164b3e2a9/src/pluggy/_manager.py#L376):

```python
            for ep in dist.entry_points:
                if (
                    ep.group != group
                    or (name is not None and ep.name != name)
                    # already registered
                    or self.get_plugin(ep.name)
                    or self.is_blocked(ep.name)
                ):
                    continue
                plugin = ep.load()
                self.register(plugin, name=ep.name)
```",107914493
issue_comments,1699809688,IC_kwDOBm6k_c5lUQmY,,9599,2023-08-30T20:41:06Z,"Slight weirdness: I noticed that the output from the `datasette plugins` command looks like this for plugins loaded with the new environment variable:
```json
    {
        ""name"": ""datasette_pretty_json"",
        ""static"": false,
        ""templates"": false,
        ""version"": null,
        ""hooks"": [
            ""render_cell""
        ]
    },
```
That should ideally be `datasette-pretty-json`, not `datasette_pretty_json`.",107914493
issue_comments,1699802028,IC_kwDOBm6k_c5lUOus,,9599,2023-08-30T20:35:02Z,"Testing this is going to be a bit of a pain.

I think I'll add a whole separate test block to CI which installs a couple of plugins and then exercises this feature using `datasette plugins`.

I'll use `datasette-init` and `datasette-json-html` just because they are small and simple.",107914493
issues,1874327336,PR_kwDOBm6k_c5ZLMSe,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,9599,2023-08-30T20:33:30Z,"- #2164

TODO:

- [x] Automated tests
- [ ] Documentation
- [x] Make sure `DATASETTE_LOAD_PLUGINS=''` works for loading zero plugins",107914493
issue_comments,1699728102,IC_kwDOBm6k_c5lT8rm,,9599,2023-08-30T19:35:32Z,"Figured this out so far with the help of GPT-4: https://chat.openai.com/share/0e785865-621b-4fb3-ba05-7449e57c8496

Now this works:
```bash
DATASETTE_LOAD_PLUGINS=datasette-write-ui datasette plugins
```
```json
[
    {
        ""name"": ""datasette_write_ui"",
        ""static"": true,
        ""templates"": true,
        ""version"": null,
        ""hooks"": [
            ""extra_template_vars"",
            ""register_routes""
        ]
    }
]
```
Or multiple plugins:
```bash
DATASETTE_LOAD_PLUGINS=datasette-write-ui,datasette-pretty-json datasette plugins
```
Outputs:
```json
[
    {
        ""name"": ""datasette_pretty_json"",
        ""static"": false,
        ""templates"": false,
        ""version"": null,
        ""hooks"": [
            ""render_cell""
        ]
    },
    {
        ""name"": ""datasette_write_ui"",
        ""static"": true,
        ""templates"": true,
        ""version"": null,
        ""hooks"": [
            ""extra_template_vars"",
            ""register_routes""
        ]
    }
]
```",107914493
issues,1874255116,I_kwDOBm6k_c5vtt0M,Ability to only load a specific list of plugins,9599,2023-08-30T19:33:41Z,"I'm going to try and get this working through an environment variable, so that you can start Datasette and it will only load a subset of plugins including those that use the `register_commands()` hook.

Initial research on this:
- https://github.com/pytest-dev/pluggy/issues/422",107914493
issue_comments,1697818917,IC_kwDOBm6k_c5lMqkl,,9599,2023-08-29T16:56:20Z,https://github.com/simonw/datasette/blob/50da908213a0fc405ecd7a40090dfea7a2e7395c/datasette/utils/internal_db.py#L8-L62,107914493
issues,1872043170,I_kwDOBm6k_c5vlRyi,Rename core_X to catalog_X in the internals,9599,2023-08-29T16:45:00Z,"Discussed with Alex this morning. We think the American spelling is fine here (it's shorter than `catalogue`) and that it's a slightly less lazy name than `core_`.

Follows:
- https://github.com/simonw/datasette/issues/2157",107914493
issue_comments,1697725150,IC_kwDOBm6k_c5lMTre,,9599,2023-08-29T15:54:12Z,"In that last commit I also upgraded `Permission` from a named tuple to a dataclass, and added a `implies_can_view=True` private (not documented) option to it.

Last step is to refactor the code to use that new property.",107914493
issues,1871935751,I_kwDOD079W85vk3kH, ImportError: cannot import name 'formatargspec' from 'inspect',36752421,2023-08-29T15:36:31Z,"I get the following error when running ""pip3 install dogsheep-photos""
""  from inspect import ismethod, isclass, formatargspec
      ImportError: cannot import name 'formatargspec' from 'inspect' (/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/inspect.py). Did you mean: 'formatargvalues'?""
    
Python 3.12.0rc1
sqlite 3.43.0
datasette, version 0.64.3",256834907
issue_comments,1697612168,IC_kwDOBm6k_c5lL4GI,,9599,2023-08-29T14:57:54Z,"The code to refactor is this: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L181-L280

I'm going to turn that into a more general `restrictions_allow_action` function.",107914493
issue_comments,1696710911,IC_kwDOBm6k_c5lIcD_,,9599,2023-08-29T03:23:44Z,I'm going to merge this so we can see how it feels.,107914493
issue_comments,1696709110,IC_kwDOBm6k_c5lIbn2,,9599,2023-08-29T03:20:40Z,"> However, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.

I'm completely happy for the `core_*` tables (or `datasette_*` or some other name) to live in the persisted-to-disk `internal.db` database, even though they're effectively meant to be an in-memory cache.

I don't think it causes any harm, and it could even be quite useful to have them visible on disk - other applications could read the `internal.db` database while Datasette itself is running, should they have some weird reason to want to do that!

Having those tables stick around in `internal.db` after Datasette shuts down could be useful for other debugging activities as well.",107914493
issue_comments,1696707458,IC_kwDOBm6k_c5lIbOC,,9599,2023-08-29T03:17:54Z,Documentation preview: https://datasette--2162.org.readthedocs.build/en/2162/internals.html#datasette-s-internal-database,107914493
issue_comments,1696644066,IC_kwDOBm6k_c5lILvi,,9599,2023-08-29T01:37:47Z,"Code for this might be cleaner with a `Restrictions()` class that takes a `""_r""` dictionary to the constructor and can then answer questions like `.any_resource_has_permission(""view-table"")` - where it can resolve aliases etc as well.",107914493
issue_comments,1696642671,IC_kwDOBm6k_c5lILZv,,9599,2023-08-29T01:35:37Z,"Reminder that I also need to confirm that `insert-row` works if you have it at the instance level, the database level or the resource level in `_r`.",107914493
issue_comments,1696618784,IC_kwDOBm6k_c5lIFkg,,9599,2023-08-29T00:58:34Z,Should this have `implies_can_view=True` too? Probably: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L20-L22,107914493
issue_comments,1696595326,IC_kwDOBm6k_c5lH_1-,,9599,2023-08-29T00:20:08Z,Cog failed!,107914493
issue_comments,1696594855,IC_kwDOBm6k_c5lH_un,,22429695,2023-08-29T00:19:29Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning:
> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`73489ca`)](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.79%.
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2162      +/-   ##
==========================================
- Coverage   92.82%   92.79%   -0.03%     
==========================================
  Files          40       40              
  Lines        5948     5953       +5     
==========================================
+ Hits         5521     5524       +3     
- Misses        427      429       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `96.87% <ø> (-0.04%)` | :arrow_down: |
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.32% <100.00%> (+0.03%)` | :arrow_up: |
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (-0.06%)` | :arrow_down: |
| [datasette/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.72% <100.00%> (+0.04%)` | :arrow_up: |
| [datasette/utils/internal\_db.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2ludGVybmFsX2RiLnB5) | `100.00% <100.00%> (ø)` | |
| [datasette/views/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.45% <100.00%> (ø)` | |
| [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <100.00%> (-0.85%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1696592763,IC_kwDOBm6k_c5lH_N7,,9599,2023-08-29T00:16:38Z,"Since this bumps Sphinx I'm manually reviewing the ReadTheDocs preview a bit, looks good to me: https://datasette--2160.org.readthedocs.build/en/2160/",107914493
issue_comments,1696591957,IC_kwDOBm6k_c5lH_BV,,49699333,2023-08-29T00:15:29Z,This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.,107914493
issue_comments,1696591943,IC_kwDOBm6k_c5lH_BH,,9599,2023-08-29T00:15:27Z,"Now solving this here:
- #2160 ",107914493
issue_comments,1696586767,IC_kwDOBm6k_c5lH9wP,,9599,2023-08-29T00:08:26Z,"https://github.com/simonw/datasette/issues/2148#issuecomment-1689177556

> Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.
",107914493
issue_comments,1696586213,IC_kwDOBm6k_c5lH9nl,,9599,2023-08-29T00:07:46Z,"I figured out why this was failing in:
- https://github.com/simonw/datasette/pull/2148

It's because Sphinx dropped support for Python 3.8.
",107914493
issues,1870672704,PR_kwDOBm6k_c5Y-7Em,"Add new `--internal internal.db` option, deprecate legacy `_internal` database",15178711,2023-08-29T00:05:07Z,"refs #2157 

This PR adds a new `--internal` option to datasette serve. If provided, it is the path to a persistent internal database that Datasette core and Datasette plugins can use to store data, as discussed in the proposal issue. 

This PR also removes and deprecates the previous in-memory `_internal` database. Those tables now appear in the `internal` database, with `core_` prefixes (ex `tables` in `_internal` is now `core_tables` in `internal`).


## A note on the new `core_` tables
However, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.

I tried to put those `core_` tables as `TEMP` tables - after all, there's always one 1 `internal` DB connection at a time, so I figured it would work. But, since we use the `Database()` wrapper for the internal DB, it has two separate connections: a default read-only connection and a write connection that is created when a write operation occurs. Which meant the `TEMP` tables would be created by the write connection, but not available in the read-only connection. 

So I had a brillant idea: Attach an in-memory named database with `cache=shared`, and create those tables there! 

```sql
ATTACH DATABASE 'file:datasette_internal_core?mode=memory&cache=shared' AS core;
```

We'd run this on both the read-only connection and the write-only connection. That way, those tables would stay in memory, they'd communicate with the `cache=shared` feature, and we'd be good to go.


However, I couldn't find an easy way to run a `ATTACH DATABASE` command on the read-only query. 

Using `Database()` as a wrapper for the internal DB is pretty limiting - it's meant for Datasette ""data"" databases, where we want multiple readers and possibly 1 write connection at a time. But the internal database doesn't really require that kind of support - I think we could get away with a single read/write connection, but it seemed like too big of a rabbithole to go through now. 


<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2162.org.readthedocs.build/en/2162/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1696378239,IC_kwDOBm6k_c5lHK1_,,9599,2023-08-28T20:38:01Z,"I want to test ""for this set of restrictions, does a GET/POST to this path return 200 or 403""?",107914493
issue_comments,1696361304,IC_kwDOBm6k_c5lHGtY,,9599,2023-08-28T20:23:47Z,"Here's an existing relevant test:

https://github.com/simonw/datasette/blob/2e2825869fc2655b5fcadc743f6f9dec7a49bc65/tests/test_permissions.py#L616-L666

It's not quite right for this new set of tests though, since they need to be exercising actual endpoints (`/.json` etc) in order to check that this works correctly.",107914493
issue_comments,1696355634,IC_kwDOBm6k_c5lHFUy,,9599,2023-08-28T20:19:15Z,"Documentation:
- https://docs.datasette.io/en/latest/cli-reference.html#datasette-get
",107914493
issue_comments,1696267473,IC_kwDOBm6k_c5lGvzR,,22429695,2023-08-28T19:38:28Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`75.00%`** and project coverage change: **`+0.05%`** :tada:
> Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`a5cbf80`)](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.81%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2161      +/-   ##
==========================================
+ Coverage   92.76%   92.81%   +0.05%     
==========================================
  Files          40       40              
  Lines        5943     5944       +1     
==========================================
+ Hits         5513     5517       +4     
+ Misses        430      427       -3     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.37% <75.00%> (+1.02%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1696260446,IC_kwDOBm6k_c5lGuFe,,9599,2023-08-28T19:33:05Z,"Now a PR:
- #2161 ",107914493
issues,1870345352,PR_kwDOBm6k_c5Y90K9,"-s/--setting x y gets merged into datasette.yml, refs #2143, #2156",9599,2023-08-28T19:30:42Z,"This change updates the `-s/--setting` option to `datasette serve` to allow it to be used to set arbitrarily complex nested settings in a way that is compatible with the new `-c datasette.yml` work happening in:
- #2143

It will enable things like this:
```
datasette data.db --setting plugins.datasette-ripgrep.path ""/home/simon/code""
```
For the moment though it just affects [settings](https://docs.datasette.io/en/1.0a4/settings.html) - so you can do this:
```
datasette data.db --setting settings.sql_time_limit_ms 3500
```
I've also implemented a backwards compatibility mechanism, so if you use it this way (the old way):
```
datasette data.db --setting sql_time_limit_ms 3500
```
It will notice that the setting you passed is one of Datasette's core settings, and will treat that as if you said `settings.sql_time_limit_ms` instead.

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2161.org.readthedocs.build/en/2161/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1695754277,IC_kwDOBm6k_c5lEygl,,22429695,2023-08-28T14:00:17Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`c81b024`)](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82%.

> :exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2160   +/-   ##
=======================================
  Coverage   92.82%   92.82%           
=======================================
  Files          40       40           
  Lines        5948     5948           
=======================================
  Hits         5521     5521           
  Misses        427      427           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1695736691,IC_kwDOBm6k_c5lEuNz,,49699333,2023-08-28T13:49:35Z,Superseded by #2160.,107914493
issues,1869807874,PR_kwDOBm6k_c5Y8AN0,"Bump sphinx, furo, blacken-docs dependencies",49699333,2023-08-28T13:49:31Z,"Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [blacken-docs](https://github.com/asottile/blacken-docs).

Updates `sphinx` from 7.1.2 to 7.2.4
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.2.4</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.3</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.2</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.2.4 (released Aug 28, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11618"">#11618</a>: Fix a regression in the MoveModuleTargets transform,
introduced in <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10478"">#10478</a> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/9662"">#9662</a>).</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11649"">#11649</a>: linkcheck: Resolve hanging tests for timezones west of London
and incorrect conversion from UTC to offsets from the UNIX epoch.
Patch by Dmitry Shachnev and Adam Turner.</li>
</ul>
<h1>Release 7.2.3 (released Aug 23, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11576"">#11576</a>: Require sphinxcontrib-serializinghtml 1.1.9.</li>
</ul>
<h2>Bugs fixed</h2>
<ul>
<li>Fix regression in <code>autodoc.Documenter.parse_name()</code>.</li>
<li>Fix regression in JSON serialisation.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11543"">#11543</a>: autodoc: Support positional-only parameters in <code>classmethod</code> methods
when <code>autodoc_preserve_defaults</code> is <code>True</code>.</li>
<li>Restore support string methods on path objects.
This is deprecated and will be removed in Sphinx 8.
Use :py:func:<code>os.fspath</code> to convert :py:class:<code>~pathlib.Path</code> objects to strings,
or :py:class:<code>~pathlib.Path</code>'s methods to work with path objects.</li>
</ul>
<h1>Release 7.2.2 (released Aug 17, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>Fix the signature of the <code>StateMachine.insert_input()</code> patch,
for when calling with keyword arguments.</li>
<li>Fixed membership testing (<code>in</code>) for the :py:class:<code>str</code> interface
of the asset classes (<code>_CascadingStyleSheet</code> and <code>_JavaScript</code>),
which several extensions relied upon.</li>
<li>Fixed a type error in <code>SingleFileHTMLBuilder._get_local_toctree</code>,
<code>includehidden</code> may be passed as a string or a boolean.</li>
<li>Fix <code>:noindex:</code> for <code>PyModule</code> and <code>JSModule</code>.</li>
</ul>
<h1>Release 7.2.1 (released Aug 17, 2023)</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/3256f1f22092a3e9c2925472f6f0b41e4f09e902""><code>3256f1f</code></a> Bump to 7.2.4 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/2f025a4b226610a170d008a24eeba4f90719fa76""><code>2f025a4</code></a> linkcheck: Fix conversion from UTC time to the UNIX epoch (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11649"">#11649</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/1567281287f780d83d5c1b1202a7e551fd0b3998""><code>1567281</code></a> autodoc: Fix UnboundLocalError in <code>filter_members</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11651"">#11651</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/5e88b9f88665dd8782b9feb381b499ae580cfa1a""><code>5e88b9f</code></a> Fix the MoveModuleTargets transform (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11647"">#11647</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/694fcee13d4298ca39ff08837c28b12b2485fbfd""><code>694fcee</code></a> Fix markup in CHANGES (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11639"">#11639</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/c503c90090efb54eab81621dde73e4c57e1af1e6""><code>c503c90</code></a> Improve <code>pathlib</code> type annotations (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11646"">#11646</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/bf339b123e9f01bfc62663849563d52a7099701a""><code>bf339b1</code></a> Bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/2f6ea1422b08f9494f34b80c2ff6c96b5f396fdc""><code>2f6ea14</code></a> Bump to 7.2.3 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/511e4070aa429a41eac04e88d96806c8fd0a5cc7""><code>511e407</code></a> Implement <code>bool()</code> for string paths</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/494de7327f23e1b8a09f4e327f6d4cdf4e351013""><code>494de73</code></a> Implement <code>hash()</code> for string paths</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v7.1.2...v7.2.4"">compare view</a></li>
</ul>
</details>
<br />

Updates `furo` from 2023.7.26 to 2023.8.19
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.08.19 -- Xenolithic Xanadu</h2>
<ul>
<li>Fix missing search context with Sphinx 7.2, for dirhtml builds.</li>
<li>Drop support for Python 3.7.</li>
<li>Present configuration errors in a better format -- thanks <a href=""https://github.com/AA-Turner""><code>@​AA-Turner</code></a>!</li>
<li>Bump <code>require_sphinx()</code> to Sphinx 6.0, in line with dependency changes in Unassuming Ultramarine.</li>
</ul>
<h2>2023.08.17 -- Wonderous White</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.2.0 and 7.2.1.</li>
</ul>
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<ul>
<li>Update Python version classifiers.</li>
<li>Increase the icon size in mobile header.</li>
<li>Increase admonition title bg opacity.</li>
<li>Change the default API background to transparent.</li>
<li>Transition the API background change.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/0766bb2d6d0392bf90d4e4ac447ae721fa673ee9""><code>0766bb2</code></a> Prepare release: 2023.08.19</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/807d73cb1080a1c9b8321a1dcffd84aee73e859a""><code>807d73c</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/364b2611f27e463cd1f879bf348975c01294ffc0""><code>364b261</code></a> Accomodate for the required <code>data-content_root</code> for search</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/0d38bc6cffc528ce3cf742fe2bd20e83c46d7891""><code>0d38bc6</code></a> Simplify retrieval of <code>pygments_dark_style</code> value (<a href=""https://redirect.github.com/pradyunsg/furo/issues/699"">#699</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/3631ffc4ea2f241b89bbd9b6c9d76e52a4226c56""><code>3631ffc</code></a> Use <code>sphinx.errors.ConfigError</code> (<a href=""https://redirect.github.com/pradyunsg/furo/issues/697"">#697</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/d2e2448852752ec570313bc1daea3bda56069816""><code>d2e2448</code></a> Remove duplicate HTML builder check (<a href=""https://redirect.github.com/pradyunsg/furo/issues/698"">#698</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/7b4f1308832e99d8e49c80f24665e938673c1644""><code>7b4f130</code></a> Drop Python 3.7 (<a href=""https://redirect.github.com/pradyunsg/furo/issues/701"">#701</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/e322b71b10c959cce0af80d3ac6934725ce728bb""><code>e322b71</code></a> Remove pointless assert (<a href=""https://redirect.github.com/pradyunsg/furo/issues/702"">#702</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/ee2097a797e92f1155a17287d063b3181f137115""><code>ee2097a</code></a> Bump <code>require_sphinx()</code> to Sphinx 6.0 (<a href=""https://redirect.github.com/pradyunsg/furo/issues/700"">#700</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c1ff10b768aaa2a2e479bbf8d58187e12a2731ac""><code>c1ff10b</code></a> Back to development</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.07.26...2023.08.19"">compare view</a></li>
</ul>
</details>
<br />

Updates `blacken-docs` from 1.15.0 to 1.16.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.16.0 (2023-08-16)</h2>
<ul>
<li>
<p>Allow Markdown fence options.</p>
<p>Thanks to initial work from Matthew Anderson in <code>PR [#246](https://github.com/asottile/blacken-docs/issues/246) &lt;https://github.com/adamchainz/blacken-docs/pull/246&gt;</code>__.</p>
</li>
<li>
<p>Expand Markdown detection to all Python language names from Pygments: <code>py</code>, <code>sage</code>, <code>python3</code>, <code>py3</code>, and <code>numpy</code>.</p>
</li>
<li>
<p>Preserve leading whitespace lines in reStructuredText code blocks.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#217](https://github.com/asottile/blacken-docs/issues/217) &lt;https://github.com/adamchainz/blacken-docs/issues/217&gt;</code>__.</p>
</li>
<li>
<p>Use exit code 2 to indicate errors from Black, whilst exit code 1 remains for “files have been formatted”.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#218](https://github.com/asottile/blacken-docs/issues/218) &lt;https://github.com/adamchainz/blacken-docs/issues/218&gt;</code>__.</p>
</li>
<li>
<p>Support passing the <code>--preview</code> option through to Black, to select the future style.</p>
</li>
<li>
<p>Remove <code>language_version</code> from <code>.pre-commit-hooks.yaml</code>.
This change allows <code>default_language_version</code> in ``.pre-commit-config.yaml` to take precedence.</p>
<p>Thanks to Aneesh Agrawal in <code>PR [#258](https://github.com/asottile/blacken-docs/issues/258) &lt;https://github.com/adamchainz/blacken-docs/pull/258&gt;</code>__.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/960ead214cd1184149d366c6d27ca6c369ce46b6""><code>960ead2</code></a> Version 1.16.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8f0ed18ce69d35e7e631d2b01983ecf8248667a0""><code>8f0ed18</code></a> Support passing --preview through to Black (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/273"">#273</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/4eb4e4cdbff31cde3d9ffdd7693d5b7dce4dc506""><code>4eb4e4c</code></a> Tweak changelog note</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/6c7450cfa90581b482d71b451b2d007361aa47c1""><code>6c7450c</code></a> Use exit code 2 to indicate errors (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/272"">#272</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/99dfc8d0b402b9b7b0f0e723fc1d030063249eac""><code>99dfc8d</code></a> Preserve leading whitespace lines in rST (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/271"">#271</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/94465e83e7253b4caef1e00fcb30b74ced33f08e""><code>94465e8</code></a> Reformat markdown tests with dedent() (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/270"">#270</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7cd5f30f0f9e014bda566a1893df3b9c7f9a2c48""><code>7cd5f30</code></a> Use .md in glob example</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f97e56994178e079ab807cebe54c2044f7b2ae55""><code>f97e569</code></a> Document applying to many files (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/269"">#269</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/ae612b095b5005eee8c3ab9fe65a248d7654582d""><code>ae612b0</code></a> Expand Markdown detection to all Python language names (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/268"">#268</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/da9b4557afc9b820c40b8d6bdb8f515c6cdec0f5""><code>da9b455</code></a> Replace NamedTuple with plain class (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/267"">#267</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/asottile/blacken-docs/compare/1.15.0...1.16.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2160.org.readthedocs.build/en/2160/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1694823972,IC_kwDOCGYnMM5lBPYk,,9599,2023-08-28T00:41:54Z,Tips on typing `**kwargs`: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/,140912432
issues,1868713944,I_kwDOCGYnMM5vYk_Y,`table.get(column=value)` option for retrieving things not by their primary key,9599,2023-08-28T00:41:23Z,"This came up working on this feature:
- https://github.com/simonw/llm/pull/186

I have a table with this schema:
```sql
CREATE TABLE [collections] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [model] TEXT
);
CREATE UNIQUE INDEX [idx_collections_name]
    ON [collections] ([name]);
```
So the primary key is an integer (because it's going to have a huge number of rows foreign key related to it, and I don't want to store a larger text value thousands of times), but there is a unique constraint on the `name` - that would be the primary key column if not for all of those foreign keys.

Problem is, fetching the collection by name is actually pretty inconvenient.

Fetch by numeric ID:

```python
try:
    table[""collections""].get(1)
except NotFoundError:
    # It doesn't exist
```
Fetching by name:
```python
def get_collection(db, collection):
    rows = db[""collections""].rows_where(""name = ?"", [collection])
    try:
        return next(rows)
    except StopIteration:
        raise NotFoundError(""Collection not found: {}"".format(collection))
```
It would be neat if, for columns where we know that we should always get 0 or one result, we could do this instead:
```python
try:
    collection = table[""collections""].get(name=""entries"")
except NotFoundError:
    # It doesn't exist
```
The existing `.get()` method doesn't have any non-positional arguments, so using `**kwargs` like that should work:

https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L1495",140912432
issues,1866815458,PR_kwDOBm6k_c5YyF-C,Implement Dark Mode colour scheme,3315059,2023-08-25T10:46:23Z,"Closes #2095.


<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2159.org.readthedocs.build/en/2159/

<!-- readthedocs-preview datasette end -->",107914493
issues,1865983069,PR_kwDOBm6k_c5YvQSi,add brand option to metadata.json.,52261150,2023-08-24T22:37:41Z,"This adds a brand link to the top navbar if 'brand' key is populated in metadata.json. The link will be either '#' or use the contents of 'brand_url' in metadata.json for href.

I was able to get this done on my own site by replacing `templates/_crumbs.html` with a custom version, but I thought it would be nice to incorporate this in the tool directly.

![image](https://github.com/simonw/datasette/assets/52261150/fdfe9bb5-fee4-466c-8074-6132071d94e6)


<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2158.org.readthedocs.build/en/2158/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1692494455,IC_kwDOBm6k_c5k4Wp3,,9599,2023-08-24T22:26:08Z,"Closing this issue in favour of this one:
- #2157 ",107914493
issue_comments,1692465763,IC_kwDOBm6k_c5k4Ppj,,9599,2023-08-24T21:54:29Z,"But yes, I'm a big +1 on this whole plan.",107914493
issue_comments,1692465334,IC_kwDOBm6k_c5k4Pi2,,9599,2023-08-24T21:54:09Z,"We discussed this in-person this morning and these notes reflect what we talked about perfectly.

I've had so many bugs with plugins that I've written myself that have forgotten to special-case the `_internal` database when looping through `datasette.databases.keys()` - removing it from there entirely would help a lot.

Just one tiny disagreement: for `datasette-comments` I think having it store things in `_internal` could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful.

Show me all rows in `foia_requests` with at least one associated comment in `datasette_comments.comments` kind of tihng.",107914493
issues,1865869205,I_kwDOBm6k_c5vNueV,"Proposal: Make the `_internal` database persistent, customizable, and hidden",15178711,2023-08-24T20:54:29Z,"The current `_internal` database is used by Datasette core to cache info about databases/tables/columns/foreign keys of databases in a Datasette instance. It's a temporary database created at startup, that can only be seen by the root user. See an [example `_internal` DB here](https://latest.datasette.io/_internal), after [logging in as root](https://latest.datasette.io/login-as-root).

The current `_internal` database has a few rough edges:

- It's part of `datasette.databases`, so many plugins have to specifically exclude `_internal` from their queries [examples here](https://github.com/search?q=datasette+hookimpl+%22_internal%22+language%3APython+-path%3Adatasette%2F&ref=opensearch&type=code)
- It's only used by Datasette core and can't be used by plugins or 3rd parties
- It's created from scratch at startup and stored in memory. Why is fine, the performance is great, but persistent storage would be nice.

Additionally, it would be really nice if plugins could use this `_internal` database to store their own configuration, secrets, and settings. For example:

- `datasette-auth-tokens` [creates a `_datasette_auth_tokens` table](https://github.com/simonw/datasette-auth-tokens/blob/main/datasette_auth_tokens/__init__.py#L15) to store auth token metadata. This could be moved into the `_internal` database to avoid writing to the gues database
- `datasette-socrata` [creates a `socrata_imports`](https://github.com/simonw/datasette-socrata/blob/1409aa9b4d2fc3aff286b52e73af33b5786d56d0/datasette_socrata/__init__.py#L190-L198) table, which also can be in `_internal`
- `datasette-upload-csvs` [creates a `_csv_progress_`](https://github.com/simonw/datasette-upload-csvs/blob/main/datasette_upload_csvs/__init__.py#L154) table, which can be in `_internal`
- `datasette-write-ui` wants to have the ability for users to toggle whether a table appears editable, which can be either in `datasette.yaml` or on-the-fly by storing config in `_internal`


In general, these are specific features that Datasette plugins would have access to if there was a central internal database they could read/write to:

- **Dynamic configuration**. Changing the `datasette.yaml` file works, but can be tedious to restart the server every time. Plugins can define their own configuration table in `_internal`, and could read/write to it to store configuration based on user actions (cell menu click, API access, etc.)
- **Caching**. If a plugin or Datasette Core needs to cache some expensive computation, they can store it inside `_internal` (possibly as a temporary table) instead of managing their own caching solution.
- **Audit logs**. If a plugin performs some sensitive operations, they can log usage info to `_internal` for others to audit later. 
- **Long running process status**. Many plugins (`datasette-upload-csvs`, `datasette-litestream`, `datasette-socrata`) perform tasks that run for a really long time, and want to give continue status updates to the user. They can store this info inside` _internal`
- **Safer authentication**. Passwords and authentication plugins usually store credentials/hashed secrets in configuration files or environment variables, which can be difficult to handle. Now, they can store them in `_internal` 

## Proposal

- We remove `_internal` from [`datasette.databases`](https://docs.datasette.io/en/latest/internals.html#databases) property.
- We add new `datasette.get_internal_db()` method that returns the `_internal` database, for plugins to use
- We add a new `--internal internal.db` flag. If provided, then the `_internal` DB will be sourced from that file, and further updates will be persisted to that file (instead of an in-memory database)
- When creating internal.db, create a new `_datasette_internal` table to mark it a an ""datasette internal database""
- In `datasette serve`, we check for the existence of the `_datasette_internal` table. If it exists, we assume the user provided that file in error and raise an error. This is to limit the chance that someone accidentally publishes their internal database to the internet. We could optionally add a `--unsafe-allow-internal` flag (or database plugin) that allows someone to do this if they really want to.


## New features unlocked with this

These features don't really need a standardized `_internal` table per-say (plugins could currently configure their own long-time storage features if they really wanted to), but it would make it much simpler to create these kinds of features with a persistent application database.

- **`datasette-comments`** : A plugin for commenting on rows or specific values in a database. Comment contents + threads + email notification info can be stored in `_internal`
- **Bookmarks**: ""Bookmarking"" an SQL query could be stored in `_internal`, or a URL link shortener
- **Webhooks**: If a plugin wants to either consume a webhook or create a new one, they can store hashed credentials/API endpoints in `_internal`",107914493
issue_comments,1692322342,IC_kwDOBm6k_c5k3som,,52261150,2023-08-24T19:56:15Z,"Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.

It's not uncommon to have things like this currently:

```https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101```

I'm thinking a plugin like [https://datasette.io/plugins/datasette-query-files](https://datasette.io/plugins/datasette-query-files), but could be created and managed from the UI (with the right permissions).",107914493
issue_comments,1692210044,IC_kwDOBm6k_c5k3RN8,,9599,2023-08-24T18:28:27Z,"Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360

Looks to me like that second bit of code doesn't yet handle `datasette.yml`

This code does though:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335

`parse_metadata()` is clearly a bad name for this function:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L980-L990

That ` @documented` decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content

So we should rename it to something better like `parse_json_or_yaml()` but keep `parse_metadata` as an undocumented alias for that to avoid any unnecessary plugin breaks.",107914493
issue_comments,1692206200,IC_kwDOBm6k_c5k3QR4,,9599,2023-08-24T18:25:23Z,"Ran out of time for this, I'll look at the next step next week.",107914493
issue_comments,1692201647,IC_kwDOBm6k_c5k3PKv,,9599,2023-08-24T18:21:53Z,"Oops, that was meant to be a PR. It's just a utility function though so it's safe to land already. I'll do a PR for the actual integration of it.",107914493
issue_comments,1692186522,IC_kwDOBm6k_c5k3Lea,,9599,2023-08-24T18:10:04Z,I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.,107914493
issues,1865649347,I_kwDOBm6k_c5vM4zD,datasette -s/--setting option for setting nested configuration options,9599,2023-08-24T18:09:27Z,"> I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax.
>
> Here's what I've come up with:
> ```
> datasette \
>   -s settings.sql_time_limit_ms 1000 \
>   -s plugins.datasette-auth-tokens.manage_tokens true \
>   -s plugins.datasette-auth-tokens.manage_tokens_database tokens \
>  mydatabase.db tokens.db
> ```
> Which would be equivalent to `datasette.yml` containing this:
> ```yaml
> plugins:
>   datasette-auth-tokens:
>     manage_tokens: true
>     manage_tokens_database: tokens
> settings:
>   sql_time_limit_ms: 1000
> ```
More details in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514
            ",107914493
issue_comments,1692182910,IC_kwDOBm6k_c5k3Kl-,,9599,2023-08-24T18:06:57Z,"The other thing that could work is something like this:
```bash
export AUTH_TOKENS_DB=""tokens""
datasette \
  -s settings.sql_time_limit_ms 1000 \
  -s plugins.datasette-auth-tokens.manage_tokens true \
  -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB
```
So `-e` is an alternative version of `-s` which reads from the named environment variable instead of having the value provided directly as the second value in the pair.

I quite like this, because it could replace the really ugly `$ENV` pattern we have in plugin configuration at the moment:  https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values
```yaml
plugins:
  datasette-auth-github:
    client_secret:
      $env: GITHUB_CLIENT_SECRET
```",107914493
issue_comments,1692180683,IC_kwDOBm6k_c5k3KDL,,9599,2023-08-24T18:05:17Z,"That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.

Need to figure out the right syntax for that. Something like this perhaps:

```bash
DATASETTE_CONFIG_PLUGINS='{""datasette-ripgrep"": ...}'
```
Hard to know how to make this nestable though. I considered this:
```bash
DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/'
```
But that doesn't work, because how does the processing code know that it should split on `_` for most of the tokens but NOT split `DATASETTE_RIPGREP`, instead treating that as `datasette-ripgrep`?

I checked and `-` is not a valid character in an environment variable, at least in zsh on macOS:
```
% export FOO_BAR-BAZ=1
export: not valid in this context: FOO_BAR-BAZ
```",107914493
issues,1865572575,PR_kwDOBm6k_c5Yt2eO,Fix hupper.start_reloader entry point,79087,2023-08-24T17:14:08Z,"Update hupper's entry point so that click commands are processed properly.

Fixes #2123

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2155.org.readthedocs.build/en/2155/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1691845306,IC_kwDOBm6k_c5k14K6,,9599,2023-08-24T14:57:39Z,"Notes on manual testing so far - it looks like this might be working!
- https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713",107914493
issue_comments,1691842259,IC_kwDOBm6k_c5k13bT,,9599,2023-08-24T14:55:54Z,"So what's needed to finish this is:
- Tests that demonstrate that nothing is revealed that shouldn't be by tokens restricted in this way
- Similar tests for other permissions like `create-table` that check that they work (and don't also need `view-instance` etc).
- Documentation",107914493
issue_comments,1691824713,IC_kwDOBm6k_c5k1zJJ,,9599,2023-08-24T14:45:49Z,"I tested this out against a Datasette Cloud instance. I created a restricted token and tested it like this:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/-/actor.json' | jq
```
```json
{
  ""actor"": {
    ""id"": ""245"",
    ""token"": ""dsatok"",
    ""token_id"": 2,
    ""_r"": {
      ""r"": {
        ""data"": {
          ""all_stocks"": [
            ""vt""
          ]
        }
      }
    }
  }
}
```
It can access the `all_stocks` demo table:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/data/all_stocks.json?_size=1' | jq
```
```json
{
  ""ok"": true,
  ""next"": ""1"",
  ""rows"": [
    {
      ""rowid"": 1,
      ""Date"": ""2013-01-02"",
      ""Open"": 79.12,
      ""High"": 79.29,
      ""Low"": 77.38,
      ""Close"": 78.43,
      ""Volume"": 140124866,
      ""Name"": ""AAPL""
    }
  ],
  ""truncated"": false
}
```
Accessing the database returns just information about that table, even though other tables exist:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/data.json?_size=1'
```
```json
{
  ""database"": ""data"",
  ""private"": true,
  ""path"": ""/data"",
  ""size"": 3796992,
  ""tables"": [
    {
      ""name"": ""all_stocks"",
      ""columns"": [
        ""Date"",
        ""Open"",
        ""High"",
        ""Low"",
        ""Close"",
        ""Volume"",
        ""Name""
      ],
      ""primary_keys"": [],
      ""count"": 8813,
      ""hidden"": false,
      ""fts_table"": null,
      ""foreign_keys"": {
        ""incoming"": [],
        ""outgoing"": []
      },
      ""private"": true
    }
  ],
  ""hidden_count"": 0,
  ""views"": [],
  ""queries"": [],
  ""allow_execute_sql"": false,
  ""table_columns"": {}
}
```
And hitting the top-level `/.json` thing does the same - it reveals that table but not any of the other tables or databases:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/.json?_size=1'
```
```json
{
  ""data"": {
    ""name"": ""data"",
    ""hash"": null,
    ""color"": ""8d777f"",
    ""path"": ""/data"",
    ""tables_and_views_truncated"": [
      {
        ""name"": ""all_stocks"",
        ""columns"": [
          ""Date"",
          ""Open"",
          ""High"",
          ""Low"",
          ""Close"",
          ""Volume"",
          ""Name""
        ],
        ""primary_keys"": [],
        ""count"": null,
        ""hidden"": false,
        ""fts_table"": null,
        ""num_relationships_for_sorting"": 0,
        ""private"": false
      }
    ],
    ""tables_and_views_more"": false,
    ""tables_count"": 1,
    ""table_rows_sum"": 0,
    ""show_table_row_counts"": false,
    ""hidden_table_rows_sum"": 0,
    ""hidden_tables_count"": 0,
    ""views_count"": 0,
    ""private"": false
  }
}
```",107914493
issue_comments,1691798722,IC_kwDOBm6k_c5k1szC,,22429695,2023-08-24T14:31:42Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`+0.03%`** :tada:
> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`3e49fd3`)](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.85%.
> Report is 3 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2154      +/-   ##
==========================================
+ Coverage   92.82%   92.85%   +0.03%     
==========================================
  Files          40       40              
  Lines        5948     6008      +60     
==========================================
+ Hits         5521     5579      +58     
- Misses        427      429       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <ø> (-0.85%)` | :arrow_down: |
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.37% <100.00%> (+0.07%)` | :arrow_up: |
| [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.48% <100.00%> (+0.57%)` | :arrow_up: |
| [datasette/permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3Blcm1pc3Npb25zLnB5) | `100.00% <100.00%> (ø)` | |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2154/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1691788400,IC_kwDOBm6k_c5k1qRw,,9599,2023-08-24T14:25:56Z,"Can be tested with:
```bash
pip install https://github.com/simonw/datasette/archive/6d57a8c23043e99b27f7a2afbe58f4d58815fd51.zip
```",107914493
issues,1865281760,PR_kwDOBm6k_c5Ys3C5,Cascade for restricted token view-table/view-database/view-instance operations,9599,2023-08-24T14:24:23Z,"Refs:
- #2102

Also includes a prototype implementation of `--actor option` which I'm using for testing this, from:
- #2153

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2154.org.readthedocs.build/en/2154/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1691779180,IC_kwDOBm6k_c5k1oBs,,9599,2023-08-24T14:21:03Z,"`datasette serve` currently only has a `--get` - for this to be really useful it needs to grow `--post` and maybe other verbs too.

Which is a good argument for moving this functionality to `datasette client get ...` instead.",107914493
issue_comments,1691767797,IC_kwDOBm6k_c5k1lP1,,9599,2023-08-24T14:15:10Z,This is broken because Sphinx no longer supports Python 3.8.,107914493
issue_comments,1691763427,IC_kwDOBm6k_c5k1kLj,,9599,2023-08-24T14:12:43Z,Annoying that `datasette client ...` makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing `datasette.client` Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance!,107914493
issue_comments,1691761685,IC_kwDOBm6k_c5k1jwV,,9599,2023-08-24T14:11:41Z,"Another option: implement this as a plugin, providing a new command like `datasette get ...`

Or implement `datasette client get ...` as core commands or a plugin - except that clashes with the `datasette client` command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one).",107914493
issue_comments,1691758168,IC_kwDOBm6k_c5k1i5Y,,9599,2023-08-24T14:09:45Z,I'm going to implement this in a branch to make it easier to test out.,107914493
issue_comments,1691753489,IC_kwDOBm6k_c5k1hwR,,9599,2023-08-24T14:07:25Z,"Building that `""_r""` array is the main reason this would be useful, but it's also fiddly to get right.

`datasette create-token` has a design for that already: https://docs.datasette.io/en/1.0a4/authentication.html#datasette-create-token
```
datasette create-token root \
    --secret mysecret \
    --all view-instance \
    --all view-table \
    --database docs view-query \
    --resource docs documents insert-row \
    --resource docs documents update-row
```
Adding imitations of those options (excluding `--secret`, not needed here) to `datasette serve` would add a LOT of extra options, but it would also make it really convenient to attempt a request with a specific set of restrictions. Not sure if that would be worth the extra `--help` output or not.

I feel like the names would have to have a common prefix though. Maybe something like this:

```bash
datasette serve data.db --get `/data/mytable.json' \
  --actor-id root \
  --r-all view-instance \
  --r-database data view-query \
  --r-resource data documents update-row
```
Other options could be the longer `--restrict-all/--restrict-database/--restrict-resource`.",107914493
issues,1865232341,I_kwDOBm6k_c5vLS_V,Datasette --get --actor option,9599,2023-08-24T14:00:03Z,"I experimented with a prototype of this here:
- https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971_

Which lets me run requests as if they belonged to a specific actor like this:

```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```

Really useful for testing actors an `_r` options. Is this worth adding as a feature?",107914493
issue_comments,1691710474,IC_kwDOBm6k_c5k1XQK,,22429695,2023-08-24T13:45:21Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`2e45686`)](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76%.

> :exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2152   +/-   ##
=======================================
  Coverage   92.76%   92.76%           
=======================================
  Files          40       40           
  Lines        5943     5919   -24     
=======================================
- Hits         5513     5491   -22     
+ Misses        430      428    -2     
```


[see 3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2152/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1865174661,PR_kwDOBm6k_c5YsfZ7,Bump the python-packages group with 3 updates,49699333,2023-08-24T13:34:44Z,"Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [blacken-docs](https://github.com/asottile/blacken-docs).

Updates `sphinx` from 7.1.2 to 7.2.3
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.2.3</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.2</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.2.3 (released Aug 23, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11576"">#11576</a>: Require sphinxcontrib-serializinghtml 1.1.9.</li>
</ul>
<h2>Bugs fixed</h2>
<ul>
<li>Fix regression in <code>autodoc.Documenter.parse_name()</code>.</li>
<li>Fix regression in JSON serialisation.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11543"">#11543</a>: autodoc: Support positional-only parameters in <code>classmethod</code> methods
when <code>autodoc_preserve_defaults</code> is <code>True</code>.</li>
<li>Restore support string methods on path objects.
This is deprecated and will be removed in Sphinx 8.
Use :py:func<code>os.fspath</code> to convert :py:class:<del><code>pathlib.Path</code> objects to strings,
or :py:class:</del><code>pathlib.Path</code>'s methods to work with path objects.</li>
</ul>
<h1>Release 7.2.2 (released Aug 17, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>Fix the signature of the <code>StateMachine.insert_input()</code> patch,
for when calling with keyword arguments.</li>
<li>Fixed membership testing (<code>in</code>) for the :py:class:<code>str</code> interface
of the asset classes (<code>_CascadingStyleSheet</code> and <code>_JavaScript</code>),
which several extensions relied upon.</li>
<li>Fixed a type error in <code>SingleFileHTMLBuilder._get_local_toctree</code>,
<code>includehidden</code> may be passed as a string or a boolean.</li>
<li>Fix <code>:noindex:</code> for <code>PyModule</code> and JSModule``.</li>
</ul>
<h1>Release 7.2.1 (released Aug 17, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>Restored the the :py:class:<code>str</code> interface of the asset classes
(<code>_CascadingStyleSheet</code> and <code>_JavaScript</code>), which several extensions relied upon.
This will be removed in Sphinx 9.</li>
<li>Restored calls to <code>Builder.add_{css,js}_file()</code>,
which several extensions relied upon.</li>
<li>Restored the private API <code>TocTree.get_toctree_ancestors()</code>,
which several extensions relied upon.</li>
</ul>
<p>Release 7.2.0 (released Aug 17, 2023)</p>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/2f6ea1422b08f9494f34b80c2ff6c96b5f396fdc""><code>2f6ea14</code></a> Bump to 7.2.3 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/511e4070aa429a41eac04e88d96806c8fd0a5cc7""><code>511e407</code></a> Implement <code>bool()</code> for string paths</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/494de7327f23e1b8a09f4e327f6d4cdf4e351013""><code>494de73</code></a> Implement <code>hash()</code> for string paths</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/2986aa113c61c23346e4097f1055172a363e66cc""><code>2986aa1</code></a> Override special methods for string paths</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/07b87e9512e88597894db31178c434cf3e3b0c28""><code>07b87e9</code></a> Update CHANGES for 7.2.3</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/6b17dd1b282def42aa868a4d04d56bea00f473ee""><code>6b17dd1</code></a> Support string methods on path objects (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11619"">#11619</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/a73fb59a71f205b9bae90ac28f65dd5c1e48ed9b""><code>a73fb59</code></a> Support positional-only parameters in classmethods (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11635"">#11635</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/02cb02ccfb3736e9bc95df9e7a774f4799cf5f9f""><code>02cb02c</code></a> Fix invocation of <code>python -m sphinx build</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/6183b6abee561fc71c88fc66416d31a8f8ca009f""><code>6183b6a</code></a> Require sphinxcontrib-serializinghtml 1.1.9 or later</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/1e16f21019833d672e9af8ce8b65fe21a40246b2""><code>1e16f21</code></a> Fix regression in <code>autodoc.Documenter.parse_name</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11613"">#11613</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v7.1.2...v7.2.3"">compare view</a></li>
</ul>
</details>
<br />

Updates `furo` from 2023.7.26 to 2023.8.19
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.08.19 -- Xenolithic Xanadu</h2>
<ul>
<li>Fix missing search context with Sphinx 7.2, for dirhtml builds.</li>
<li>Drop support for Python 3.7.</li>
<li>Present configuration errors in a better format -- thanks <a href=""https://github.com/AA-Turner""><code>@​AA-Turner</code></a>!</li>
<li>Bump <code>require_sphinx()</code> to Sphinx 6.0, in line with dependency changes in Unassuming Ultramarine.</li>
</ul>
<h2>2023.08.17 -- Wonderous White</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.2.0 and 7.2.1.</li>
</ul>
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<ul>
<li>Update Python version classifiers.</li>
<li>Increase the icon size in mobile header.</li>
<li>Increase admonition title bg opacity.</li>
<li>Change the default API background to transparent.</li>
<li>Transition the API background change.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/0766bb2d6d0392bf90d4e4ac447ae721fa673ee9""><code>0766bb2</code></a> Prepare release: 2023.08.19</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/807d73cb1080a1c9b8321a1dcffd84aee73e859a""><code>807d73c</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/364b2611f27e463cd1f879bf348975c01294ffc0""><code>364b261</code></a> Accomodate for the required <code>data-content_root</code> for search</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/0d38bc6cffc528ce3cf742fe2bd20e83c46d7891""><code>0d38bc6</code></a> Simplify retrieval of <code>pygments_dark_style</code> value (<a href=""https://redirect.github.com/pradyunsg/furo/issues/699"">#699</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/3631ffc4ea2f241b89bbd9b6c9d76e52a4226c56""><code>3631ffc</code></a> Use <code>sphinx.errors.ConfigError</code> (<a href=""https://redirect.github.com/pradyunsg/furo/issues/697"">#697</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/d2e2448852752ec570313bc1daea3bda56069816""><code>d2e2448</code></a> Remove duplicate HTML builder check (<a href=""https://redirect.github.com/pradyunsg/furo/issues/698"">#698</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/7b4f1308832e99d8e49c80f24665e938673c1644""><code>7b4f130</code></a> Drop Python 3.7 (<a href=""https://redirect.github.com/pradyunsg/furo/issues/701"">#701</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/e322b71b10c959cce0af80d3ac6934725ce728bb""><code>e322b71</code></a> Remove pointless assert (<a href=""https://redirect.github.com/pradyunsg/furo/issues/702"">#702</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/ee2097a797e92f1155a17287d063b3181f137115""><code>ee2097a</code></a> Bump <code>require_sphinx()</code> to Sphinx 6.0 (<a href=""https://redirect.github.com/pradyunsg/furo/issues/700"">#700</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c1ff10b768aaa2a2e479bbf8d58187e12a2731ac""><code>c1ff10b</code></a> Back to development</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.07.26...2023.08.19"">compare view</a></li>
</ul>
</details>
<br />

Updates `blacken-docs` from 1.15.0 to 1.16.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.16.0 (2023-08-16)</h2>
<ul>
<li>
<p>Allow Markdown fence options.</p>
<p>Thanks to initial work from Matthew Anderson in <code>PR [#246](https://github.com/asottile/blacken-docs/issues/246) &lt;https://github.com/adamchainz/blacken-docs/pull/246&gt;</code>__.</p>
</li>
<li>
<p>Expand Markdown detection to all Python language names from Pygments: <code>py</code>, <code>sage</code>, <code>python3</code>, <code>py3</code>, and <code>numpy</code>.</p>
</li>
<li>
<p>Preserve leading whitespace lines in reStructuredText code blocks.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#217](https://github.com/asottile/blacken-docs/issues/217) &lt;https://github.com/adamchainz/blacken-docs/issues/217&gt;</code>__.</p>
</li>
<li>
<p>Use exit code 2 to indicate errors from Black, whilst exit code 1 remains for “files have been formatted”.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#218](https://github.com/asottile/blacken-docs/issues/218) &lt;https://github.com/adamchainz/blacken-docs/issues/218&gt;</code>__.</p>
</li>
<li>
<p>Support passing the <code>--preview</code> option through to Black, to select the future style.</p>
</li>
<li>
<p>Remove <code>language_version</code> from <code>.pre-commit-hooks.yaml</code>.
This change allows <code>default_language_version</code> in ``.pre-commit-config.yaml` to take precedence.</p>
<p>Thanks to Aneesh Agrawal in <code>PR [#258](https://github.com/asottile/blacken-docs/issues/258) &lt;https://github.com/adamchainz/blacken-docs/pull/258&gt;</code>__.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/960ead214cd1184149d366c6d27ca6c369ce46b6""><code>960ead2</code></a> Version 1.16.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8f0ed18ce69d35e7e631d2b01983ecf8248667a0""><code>8f0ed18</code></a> Support passing --preview through to Black (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/273"">#273</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/4eb4e4cdbff31cde3d9ffdd7693d5b7dce4dc506""><code>4eb4e4c</code></a> Tweak changelog note</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/6c7450cfa90581b482d71b451b2d007361aa47c1""><code>6c7450c</code></a> Use exit code 2 to indicate errors (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/272"">#272</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/99dfc8d0b402b9b7b0f0e723fc1d030063249eac""><code>99dfc8d</code></a> Preserve leading whitespace lines in rST (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/271"">#271</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/94465e83e7253b4caef1e00fcb30b74ced33f08e""><code>94465e8</code></a> Reformat markdown tests with dedent() (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/270"">#270</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7cd5f30f0f9e014bda566a1893df3b9c7f9a2c48""><code>7cd5f30</code></a> Use .md in glob example</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f97e56994178e079ab807cebe54c2044f7b2ae55""><code>f97e569</code></a> Document applying to many files (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/269"">#269</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/ae612b095b5005eee8c3ab9fe65a248d7654582d""><code>ae612b0</code></a> Expand Markdown detection to all Python language names (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/268"">#268</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/da9b4557afc9b820c40b8d6bdb8f515c6cdec0f5""><code>da9b455</code></a> Replace NamedTuple with plain class (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/267"">#267</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/asottile/blacken-docs/compare/1.15.0...1.16.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2152.org.readthedocs.build/en/2152/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1691094870,IC_kwDOBm6k_c5kzA9W,,1238873,2023-08-24T06:43:40Z,"If I may, the ""path-like"" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.

For instance:

```
datasette -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}'
```

could also be provided using:

```
export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{""foo"": [1,2,3], ""bar"": ""baz""}'
datasette
```

(I do not like mixing `-` and `_` in env vars but I do not have a best easily reversible example at the moment)

FYI, you could take some inspiration from another great open source data project, Metabase:
https://www.metabase.com/docs/latest/configuring-metabase/config-file
https://www.metabase.com/docs/latest/configuring-metabase/environment-variables",107914493
issue_comments,1691045051,IC_kwDOBm6k_c5ky0y7,,9599,2023-08-24T05:51:59Z,"With that fix in place, this works:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```
But this fails, because it's for a table not explicitly listed:
```bash
datasette fixtures.db --get '/fixtures/searchable.json' --actor '{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```",107914493
issue_comments,1691044283,IC_kwDOBm6k_c5ky0m7,,9599,2023-08-24T05:51:02Z,"Also need to confirm that permissions like `insert-row`, `delete-row`, `create-table` etc don't also need special cases to ensure they get through the `view-instance` etc checks, if those exist for those actions.",107914493
issue_comments,1691043475,IC_kwDOBm6k_c5ky0aT,,9599,2023-08-24T05:50:04Z,"On first test this seems to work!

```diff
diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py
index 63a66c3c..9303dac8 100644
--- a/datasette/default_permissions.py
+++ b/datasette/default_permissions.py
@@ -187,6 +187,30 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource):
         return None
     _r = actor.get(""_r"")
 
+    # Special case for view-instance: it's allowed if there are any view-database
+    # or view-table permissions defined
+    if action == ""view-instance"":
+        database_rules = _r.get(""d"") or {}
+        for rules in database_rules.values():
+            if ""vd"" in rules or ""view-database"" in rules:
+                return None
+        # Now check resources
+        resource_rules = _r.get(""r"") or {}
+        for _database, resources in resource_rules.items():
+            for rules in resources.values():
+                if ""vt"" in rules or ""view-table"" in rules:
+                    return None
+
+    # Special case for view-database: it's allowed if there are any view-table permissions
+    # defined within that database
+    if action == ""view-database"":
+        database_name = resource
+        resource_rules = _r.get(""r"") or {}
+        resources_in_database = resource_rules.get(database_name) or {}
+        for rules in resources_in_database.values():
+            if ""vt"" in rules or ""view-table"" in rules:
+                return None
+
     # Does this action have an abbreviation?
     to_check = {action}
     permission = datasette.permissions.get(action)
```
Needs a LOT of testing to make sure what it's doing is sensible though.",107914493
issue_comments,1691037971,IC_kwDOBm6k_c5kyzET,,9599,2023-08-24T05:42:47Z,"I applied a fun trick to help test this out:
```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 58f89c1c..830f47ef 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -445,6 +445,10 @@ def uninstall(packages, yes):
     ""--token"",
     help=""API token to send with --get requests"",
 )
+@click.option(
+    ""--actor"",
+    help=""Actor to use for --get requests"",
+)
 @click.option(""--version-note"", help=""Additional note to show on /-/versions"")
 @click.option(""--help-settings"", is_flag=True, help=""Show available settings"")
 @click.option(""--pdb"", is_flag=True, help=""Launch debugger on any errors"")
@@ -499,6 +503,7 @@ def serve(
     root,
     get,
     token,
+    actor,
     version_note,
     help_settings,
     pdb,
@@ -611,7 +616,10 @@ def serve(
         headers = {}
         if token:
             headers[""Authorization""] = ""Bearer {}"".format(token)
-        response = client.get(get, headers=headers)
+        cookies = {}
+        if actor:
+            cookies[""ds_actor""] = client.actor_cookie(json.loads(actor))
+        response = client.get(get, headers=headers, cookies=cookies)
         click.echo(response.text)
         exit_code = 0 if response.status == 200 else 1
         sys.exit(exit_code)
```
This adds a `--actor` option to `datasette ... --get /path` which makes it easy to test an API endpoint using a fake actor with a set of `_r` restrictions.

With that in place I can try this, with a token that has view-instance and view-database and view-table:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
    ""_r"": {
        ""a"": [
            ""vi""
        ],
        ""d"": {
            ""fixtures"": [
                ""vd""
            ]
        },
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```
Or this, with a token that just has view-table but is missing the view-database and view-instance:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```",107914493
issue_comments,1691036559,IC_kwDOBm6k_c5kyyuP,,9599,2023-08-24T05:40:53Z,"There might be an easier way to solve this. Here's some permission checks that run when hitting `/fixtures/facetable.json`:

```
permission_allowed: action=view-table, resource=('fixtures', 'facetable'), actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

  File ""/datasette/views/table.py"", line 727, in table_view_traced
    view_data = await table_view_data(

  File ""/datasette/views/table.py"", line 875, in table_view_data
    visible, private = await datasette.check_visibility(

  File ""/datasette/app.py"", line 890, in check_visibility
    await self.ensure_permissions(actor, permissions)

permission_allowed: action=view-database, resource=fixtures, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

  File ""/datasette/views/table.py"", line 727, in table_view_traced
    view_data = await table_view_data(

  File ""/datasette/views/table.py"", line 875, in table_view_data
    visible, private = await datasette.check_visibility(

  File ""/datasette/app.py"", line 890, in check_visibility
    await self.ensure_permissions(actor, permissions)

permission_allowed: action=view-instance, resource=<None>, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

  File ""/datasette/views/table.py"", line 727, in table_view_traced
    view_data = await table_view_data(

  File ""/datasette/views/table.py"", line 875, in table_view_data
    visible, private = await datasette.check_visibility(

  File ""/datasette/app.py"", line 890, in check_visibility
    await self.ensure_permissions(actor, permissions)
```
That's with a token that has the view instance, view database and view table permissions required.

But... what if the restrictions logic said that if you have view-table you automatically also get view-database and view-instance?

Would that actually let people do anything they shouldn't be able to do? I don't think it would even let them see a list of tables that they weren't allowed to visit, so it might be OK.

I'll try that and see how it works.",107914493
issue_comments,1690955706,IC_kwDOBm6k_c5kye-6,,18899,2023-08-24T03:54:35Z,"That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.

But without a real motivating example, I see why you don't want to add that.",107914493
issue_comments,1690800641,IC_kwDOBm6k_c5kx5IB,,9599,2023-08-24T00:11:16Z,"> @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

That's a neat example thanks!",107914493
issue_comments,1690800119,IC_kwDOBm6k_c5kx4_3,,9599,2023-08-24T00:10:32Z,"Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin):

```bash
datasette data.db \
  -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}'
```
Which would be equivalent to:
```yaml
plugins:
  datasette-complex-plugin:
    configs:
      foo:
        - 1
        - 2
        - 3
      bar: baz
```
This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex.

That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again.

For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.",107914493
issue_comments,1690799608,IC_kwDOBm6k_c5kx434,,77071,2023-08-24T00:09:47Z,"@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.",107914493
issue_comments,1690792514,IC_kwDOBm6k_c5kx3JC,,9599,2023-08-24T00:00:16Z,"I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax.

Here's what I've come up with:
```
datasette \
  -s settings.sql_time_limit_ms 1000 \
  -s plugins.datasette-auth-tokens.manage_tokens true \
  -s plugins.datasette-auth-tokens.manage_tokens_database tokens \
  -s plugins.datasette-ripgrep.path ""/home/simon/code-to-search"" \
  -s databases.mydatabase.tables.example_table.sort created \
  mydatabase.db tokens.db
```
Which would be equivalent to `datasette.yml` containing this:
```yaml
plugins:
  datasette-auth-tokens:
    manage_tokens: true
    manage_tokens_database: tokens
  datasette-ripgrep:
    path: /home/simon/code-to-search
databases:
  mydatabase:
    tables:
      example_table:
        sort: created
settings:
  sql_time_limit_ms: 1000
```
Here's a prototype implementation of this:
```python
import json
from typing import Any, List, Tuple

def _handle_pair(key: str, value: str) -> dict:
    """"""
    Turn a key-value pair into a nested dictionary.
    foo, bar => {'foo': 'bar'}
    foo.bar, baz => {'foo': {'bar': 'baz'}}
    foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}}
    foo.bar, ""baz"" => {'foo': {'bar': 'baz'}}
    foo.bar, '{""baz"": ""qux""}' => {'foo': {'bar': ""{'baz': 'qux'}""}}
    """"""
    try:
        value = json.loads(value)
    except json.JSONDecodeError:
        # If it doesn't parse as JSON, treat it as a string
        pass

    keys = key.split('.')
    result = current_dict = {}
    
    for k in keys[:-1]:
        current_dict[k] = {}
        current_dict = current_dict[k]
    
    current_dict[keys[-1]] = value
    return result


def _combine(base: dict, update: dict) -> dict:
    """"""
    Recursively merge two dictionaries.
    """"""
    for key, value in update.items():
        if isinstance(value, dict) and key in base and isinstance(base[key], dict):
            base[key] = _combine(base[key], value)
        else:
            base[key] = value
    return base

def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict:
    """"""
    Parse a list of key-value pairs into a nested dictionary.
    """"""
    result = {}
    for key, value in pairs:
        parsed_pair = _handle_pair(key, value)
        result = _combine(result, parsed_pair)
    return result
```
Exercised like this:
```python
print(json.dumps(handle_pairs([
    (""settings.sql_time_limit_ms"", ""1000""),
    (""plugins.datasette-auth-tokens.manage_tokens"", ""true""),
    (""plugins.datasette-auth-tokens.manage_tokens_database"", ""tokens""),
    (""plugins.datasette-ripgrep.path"", ""/home/simon/code-to-search""),
    (""databases.mydatabase.tables.example_table.sort"", ""created""),
]), indent=4))
```
Output:
```json
{
    ""settings"": {
        ""sql_time_limit_ms"": 1000
    },
    ""plugins"": {
        ""datasette-auth-tokens"": {
            ""manage_tokens"": true,
            ""manage_tokens_database"": ""tokens""
        },
        ""datasette-ripgrep"": {
            ""path"": ""/home/simon/code-to-search""
        }
    },
    ""databases"": {
        ""mydatabase"": {
            ""tables"": {
                ""example_table"": {
                    ""sort"": ""created""
                }
            }
        }
    }
}
```
Note that `-s` isn't currently an option for `datasette serve`.

`--setting key value` IS an existing option, but it isn't completely compatible with this because it maps directly just to settings.

Although... we could keep compatibility by saying that if you call `--setting known_setting value` and that `known_setting` is in this list then we treat it as if you said `-s settings.known_setting value` instead:

https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204",107914493
issue_comments,1690787394,IC_kwDOBm6k_c5kx15C,,9599,2023-08-23T23:52:02Z,"> This also makes it simple to separate out secrets.
> 
> `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml`

Having multiple configs that combine in that way is a really interesting direction.

> To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight.

I'm very keen on separating out the ""metadata"" - where metadata is the slimmest possible set of things, effectively the data license and the source and the column and table descriptions - from everything else, mainly because I want metadata to be able to travel with the data.

One idea that's been discussed before is having an optional mechanism for storing metadata in the SQLite database file itself - potentially in a `_datasette_metadata` table. That way you could distribute a DB file and anyone who opened it in Datasette would also see the correct metadata about it.

That's why I'm so keen on splitting out metadata from all of the other stuff - settings and plugin configuration and authentication rules.

So really it becomes ""true metadata"" v.s. ""all of the other junk that's accumulated in metadata and `settings.json`"".",107914493
issue_comments,1690746371,IC_kwDOBm6k_c5kxr4D,,22429695,2023-08-23T22:51:57Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`bdf59eb`)](https://app.codecov.io/gh/simonw/datasette/commit/bdf59eb7db42559e538a637bacfe86d39e5d17ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73% compared to head [(`b895cd2`)](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2151   +/-   ##
=======================================
  Coverage   92.73%   92.73%           
=======================================
  Files          40       40           
  Lines        5919     5919           
=======================================
  Hits         5489     5489           
  Misses        430      430           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1864112887,PR_kwDOBm6k_c5Yo7bk,Test Datasette on multiple SQLite versions,15178711,2023-08-23T22:42:51Z,"still testing, hope it works!

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2151.org.readthedocs.build/en/2151/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1690705243,IC_kwDOBm6k_c5kxh1b,,9599,2023-08-23T22:03:54Z,Idea: `datasette-permissions-debug` plugin which simply prints out a stacktrace for every permission check so you can see where in the code they are.,107914493
issue_comments,1690703764,IC_kwDOBm6k_c5kxheU,,9599,2023-08-23T22:02:14Z,"Built this new test:
```python
@pytest.mark.asyncio
async def test_view_table_token_can_access_table(perms_ds):
    actor = {
        ""id"": ""restricted-token"",
        ""token"": ""dstok"",
        # Restricted to just view-table on perms_ds_two/t1
        ""_r"": {""r"": {""perms_ds_two"": {""t1"": [""vt""]}}},
    }
    cookies = {""ds_actor"": perms_ds.client.actor_cookie(actor)}
    response = await perms_ds.client.get(""/perms_ds_two/t1.json"", cookies=cookies)
    assert response.status_code == 200
```
The test fails. Running it with `pytest --pdb` let me do this:
```
(Pdb) from pprint import pprint
(Pdb) pprint(perms_ds._permission_checks)
deque([{'action': 'view-table',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': ('perms_ds_two', 't1'),
        'result': None,
        'used_default': True,
        'when': '2023-08-23T21:59:45.117155'},
       {'action': 'view-database',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': 'perms_ds_two',
        'result': False,
        'used_default': False,
        'when': '2023-08-23T21:59:45.117189'},
       {'action': 'view-instance',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': None,
        'result': False,
        'used_default': False,
        'when': '2023-08-23T21:59:45.126751'},
       {'action': 'debug-menu',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': None,
        'result': False,
        'used_default': False,
        'when': '2023-08-23T21:59:45.126777'}],
      maxlen=200)
```",107914493
issue_comments,1690693830,IC_kwDOBm6k_c5kxfDG,,9599,2023-08-23T21:51:52Z,"This is the hook in question: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/hookspecs.py#L108-L110

- `True` means they are allowed to access it. You only need a single`True` from a plugin to allow it.
- `False` means they are not, and just one `False` from a plugin will deny it (even if another one returned `True` I think)
- `None` means that the plugin has no opinion on this question.",107914493
issue_comments,1690438270,IC_kwDOBm6k_c5kwgp-,,9599,2023-08-23T18:27:47Z,"I added `outline: 3px dotted pink` to that `form label` block to help spot where else it's being used.

Oh interesting, looks like it's over-ridden here too:

<img width=""950"" alt=""CleanShot 2023-08-23 at 11 27 15@2x"" src=""https://github.com/simonw/datasette/assets/9599/f3f3e995-2d6c-4afe-8370-01e0b5141a64"">
",107914493
issue_comments,1690435866,IC_kwDOBm6k_c5kwgEa,,9599,2023-08-23T18:25:51Z,"Looks like that affects a few forms:

<img width=""701"" alt=""CleanShot 2023-08-23 at 11 24 46@2x"" src=""https://github.com/simonw/datasette/assets/9599/7145aec1-832e-4eea-84a6-a735df4dfd6e"">

The search form on the table page over-rides it already:

<img width=""1243"" alt=""CleanShot 2023-08-23 at 11 25 26@2x"" src=""https://github.com/simonw/datasette/assets/9599/e002089e-e992-4cc5-bb6b-c54cf614ec95"">

",107914493
issue_comments,1690432928,IC_kwDOBm6k_c5kwfWg,,9599,2023-08-23T18:23:26Z,"That should be scoped just to the query filters form on the table page.

I'll fix it in `main` but I'm still going to ship a fix for those plugins separately so they work well before people upgrade to the next Datasette release with this change in it.",107914493
issue_comments,1690431509,IC_kwDOBm6k_c5kwfAV,,9599,2023-08-23T18:22:47Z,https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/static/app.css#L485-L489,107914493
issues,1863810783,I_kwDOBm6k_c5vF37f,form label { width: 15% } is a bad default,9599,2023-08-23T18:22:27Z,"See:
- https://github.com/simonw/datasette-configure-fts/issues/14
- https://github.com/simonw/datasette-auth-tokens/issues/12",107914493
issue_comments,1690423878,IC_kwDOBm6k_c5kwdJG,,9599,2023-08-23T18:18:18Z,"Dupe of:
- #2097 ",107914493
issue_comments,1689207309,IC_kwDOBm6k_c5kr0IN,,9599,2023-08-23T03:07:27Z,"> I'm happy to debug and land a patch if it's welcome.

Yes please! What an odd bug.",107914493
issue_comments,1689206768,IC_kwDOBm6k_c5krz_w,,9599,2023-08-23T03:06:32Z,"I'm less convinced by the ""rewrite the query in some way"" optional idea. What kind of use-cases can you imagine for that?

My hunch is that it's much more likely to cause weird breakages than it is to allow for useful plugin extensions, but I'm willing to be convinced otherwise.",107914493
issue_comments,1689206170,IC_kwDOBm6k_c5krz2a,,9599,2023-08-23T03:05:32Z,"Interestingly enough there's actually a mechanism that looks like that a bit already: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/views/database.py#L496-L508

That `validate_sql_select()` function is defined here: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L256-L265

Against these constants:

https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L223-L253

Which isn't a million miles away from your suggestion to have a hook that can say if the query should be executed or not.",107914493
issue_comments,1689198413,IC_kwDOBm6k_c5krx9N,,49699333,2023-08-23T02:57:55Z,"Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry!

If you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`.
",107914493
issue_comments,1689198368,IC_kwDOBm6k_c5krx8g,,9599,2023-08-23T02:57:53Z,@dependabot rebase,107914493
issue_comments,1689177556,IC_kwDOBm6k_c5krs3U,,9599,2023-08-23T02:44:29Z,"Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.",107914493
issue_comments,1689175062,IC_kwDOBm6k_c5krsQW,,9599,2023-08-23T02:40:46Z,"Here's why the tests are failing:

https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/.github/workflows/test.yml#L30-L46

It looks like those tests don't actually need Sphinx installed - they install `pip install -e '.[test,docs]'` to get the other docs dependencies:

https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/setup.py#L70-L80",107914493
issue_comments,1689173748,IC_kwDOBm6k_c5krr70,,9599,2023-08-23T02:38:31Z,"Sphinx dropped support for Python 3.8 in their version 7.2.0:

- https://www.sphinx-doc.org/en/master/changes.html#release-7-2-0-released-aug-17-2023
- https://www.sphinx-doc.org/en/master/internals/release-process.html#python-version-support-policy
- https://github.com/sphinx-doc/sphinx/pull/11511
- ",107914493
issue_comments,1689159200,IC_kwDOBm6k_c5kroYg,,9599,2023-08-23T02:15:36Z,This could play havoc with unmerged PRs. I should merge any big ones (like the JavaScript plugins work) first.,107914493
issue_comments,1689158712,IC_kwDOBm6k_c5kroQ4,,9599,2023-08-23T02:14:45Z,"Thinking about this again today. Posted about it on Discord: https://discord.com/channels/823971286308356157/823971286941302908/1143729300349132933

I won't enforce it in a `pytest` test, I'll enforce it with a CI lint check that's also in the `Justfile` here instead:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/Justfile#L19-L23",107914493
issue_comments,1689154837,IC_kwDOBm6k_c5krnUV,,9599,2023-08-23T02:08:33Z,Browse this commit to see the result: https://github.com/simonw/datasette/tree/59a5d336bd4336bc53103922ada4bf726f4336c9,107914493
issue_comments,1689153446,IC_kwDOBm6k_c5krm-m,,9599,2023-08-23T02:06:35Z,"I just tried this again today by dropping this into `.isort.cfg`:
```ini
[settings]
multi_line_output=3
include_trailing_comma=True
force_grid_wrap=0
use_parentheses=True
line_length=88
known_first_party=datasette
```
And running this in the root of the project:
```
isort .
```
It produced a huge diff, but when I ran the tests:
```bash
pytest -n auto
```
The tests all passed. I'll push a PR.",107914493
issue_comments,1689133247,IC_kwDOBm6k_c5kriC_,,22429695,2023-08-23T01:36:42Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage has no change and project coverage change: **`+0.01%`** :tada:
> Comparison is base [(`17ec309`)](https://app.codecov.io/gh/simonw/datasette/commit/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.71% compared to head [(`a651d3c`)](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%.
> Report is 2 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2148      +/-   ##
==========================================
+ Coverage   92.71%   92.73%   +0.01%     
==========================================
  Files          40       40              
  Lines        5919     5919              
==========================================
+ Hits         5488     5489       +1     
+ Misses        431      430       -1     
```


[see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2148/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1689130061,IC_kwDOBm6k_c5krhRN,,9599,2023-08-23T01:31:08Z,This branch is WAY out of date.,107914493
issue_comments,1689128911,IC_kwDOBm6k_c5krg_P,,9599,2023-08-23T01:29:20Z,"It's going to be called `datasette.json` and the concept of metadata will be split out separately. See:

- #2149 ",107914493
issue_comments,1689128553,IC_kwDOBm6k_c5krg5p,,9599,2023-08-23T01:28:37Z,"This is obsoleted by the work happening here:
- #2093 ",107914493
issue_comments,1689127479,IC_kwDOBm6k_c5krgo3,,9599,2023-08-23T01:26:53Z,@dependabot recreate,107914493
issue_comments,1689125244,IC_kwDOBm6k_c5krgF8,,9599,2023-08-23T01:23:27Z,"This is a really great start - tests pass, implementation looks clean, the new stubbed documentation page makes sense. Let's land it in `main` and iterate on it further in future PRs.",107914493
issue_comments,1688547401,IC_kwDOBm6k_c5kpTBJ,,22429695,2023-08-22T16:30:41Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`76.47%`** and project coverage change: **`+0.10%`** :tada:
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`db720cd`)](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.17%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2149      +/-   ##
==========================================
+ Coverage   92.06%   92.17%   +0.10%     
==========================================
  Files          40       40              
  Lines        5937     5916      -21     
==========================================
- Hits         5466     5453      -13     
+ Misses        471      463       -8     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `81.02% <25.00%> (+1.20%)` | :arrow_up: |
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.29% <92.30%> (-0.09%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1861812208,PR_kwDOBm6k_c5YhH-W,"Start a new `datasette.yaml` configuration file, with settings support",15178711,2023-08-22T16:24:16Z,"refs #2093 #2143 

This is the first step to implementing the new `datasette.yaml`/`datasette.json` configuration file. 

- The old `--config` argument is now back, and is the path to a `datasette.yaml` file. Acts like the `--metadata` flag.
- The old `settings.json` behavior has been removed.
- The `""settings""` key inside `datasette.yaml` defines the same `--settings` flags
- Values passed in `--settings` will over-write values in `datasette.yaml`

Docs for the Config file is pretty light, not much to add until we add more config to the file.

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2149.org.readthedocs.build/en/2149/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1688532012,IC_kwDOBm6k_c5kpPQs,,15178711,2023-08-22T16:21:40Z,"OK Here's the gameplan for this, which is closely tied to #2143 :

- We will add a new `datasette.json`/`datasette.yaml` configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format
- Metadata will NOT be a part of this file
- TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML
- The `settings.json` file will be deprecated, and the `--config` arg will be brought back.
- Command line arguments can still be used to overwrite values (ex `--setting` will overwrite settings in `datasette.yaml`

The format of `datasette.json` will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426

Here's the current implementation plan:

1. Add a new `--config` flag and port over `""settings""` into a new datasette.json config file, remove settings.json
2. Add top-level plugin config support to `datasette.json`
3. Figure out database/table structure of config `datasette.json`
4. Port over database/table level plugin config support `datasette.json`
5. Port over permissions/auth settings to `datasette.json`
6. Deprecate non-metadata values in `metadata.json`",107914493
issue_comments,1687433388,IC_kwDOBm6k_c5klDCs,,18899,2023-08-22T05:05:33Z,"Thanks for all this! You're totally right that the ASGI option is doable, if a bit low level and coupled to the current URI design. I'm totally fine with that being the final answer.

process_view is interesting and in the general direction of what I had in mind.

A somewhat less powerful idea: Is there value in giving a hook for just the query that's about to be run? Maybe I'm thinking a little narrowly about this problem I decided I wanted to solve, but I could see other uses for a hook of the sketch below:

```
def prepare_query(database, table, query):
   """"""Modify query that is about to be run in some way. Return the (possibly rewritten) query to run, or None to disallow running the query""""""
```
(Maybe you actually want to return a tuple so there can be an error message when you disallow, or something.)

Maybe it's too narrowly useful and some of the other pieces of datasette obviate some of these ideas, but off the cuff I could imagine using it to:
* Require a LIMIT. Either fail the query or add the limit if it's not there.
* Do logging, like my usecase.
* Do other analysis on whether you want to allow the query to run; a linter? query complexity? 

Definitely feel free to say no, or not now. This is all me just playing around with what datasette and its plugin architecture can do with toy ideas, so don't let me push you to commit to a hook you don't feel confident fits well in the design.",107914493
issue_comments,1686749342,IC_kwDOBm6k_c5kicCe,,9599,2023-08-21T17:33:11Z,"I'm definitely open to suggestions for plugin hooks that might make this kind of thing easier.

One idea I've been mulling is whether there should be a plugin hook that files on arbitrary views - similar to Django's `process_view` mechanism: https://docs.djangoproject.com/en/4.2/topics/http/middleware/#process-view

That would allow people to setup code that runs before or after any of the default views in Datasette.

I'm not yet 100% sold on the idea, because I worry about implementing it in a way that guarantees plugins won't break on future releases. But I'm open to considering it.",107914493
issue_comments,1686747420,IC_kwDOBm6k_c5kibkc,,9599,2023-08-21T17:31:42Z,"Are you talking just about queries submitted to `/database?sql=` using the interface on https://latest.datasette.io/fixtures?sql=select+*+from+facetable or are you interested in queries that are run to power other pages like https://latest.datasette.io/fixtures/facetable as well? I'll assume the former.

There are a few ways you could solve this at the moment.

The easiest would be with a piece of ASGI middleware that looks for URLs matching `/dbname?sql=...` and logs those. I played with a version of that a few years ago: https://simonwillison.net/2019/Dec/16/logging-sqlite-asgi-middleware/ - see also https://github.com/simonw/asgi-log-to-sqlite

Then you can load that middleware from a plugin using https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette

That feels a bit delicate because it's relying on the URL design not changing, but I'm happy to confirm that URL is going to stay the same for Datasette 1.0 and I have no plans to change it ever.

There's also a tracing mechanism built into Datasette itself that you could hook into. The internals of that are documented here: https://docs.datasette.io/en/stable/internals.html#datasette-tracer - but I don't yet consider it a 100% stable API. I don't plan to change it but I won't promise not to either.

I used that mechanism in this plugin: https://datasette.io/plugins/datasette-pretty-traces - demonstrated here: https://latest-with-plugins.datasette.io/github/commits?_trace=1

The hackiest way to do this would be to patch Datasette itself and try to replace the `query_view`. This definitely isn't a documented, stable API though and would be very likely to break at arbitrary points in the future.

So my recommendation for the moment is the ASGI middleware option.",107914493
issue_comments,1686745094,IC_kwDOBm6k_c5kibAG,,15178711,2023-08-21T17:30:01Z,"Another point: The new Datasette write API should refuse to insert a row with a NULL primary key. That will likely decrease the likelihood someone find themselves with NULLs in their primary keys, at least with Datasette users. Especially buggy code that uses the write API, like our `datasette-write-ui` bug that led to this issue.
",107914493
issue_comments,1686683596,IC_kwDOBm6k_c5kiL_M,,9599,2023-08-21T16:49:12Z,"Suggestion from @asg017 is that we say that if your row has a null primary key you don't get a link to a row page for that row.

Which has some precedent, because our SQL view display doesn't link to row pages at all (since they don't make sense for views): https://latest.datasette.io/fixtures/simple_view",107914493
issue_comments,1686366557,IC_kwDOBm6k_c5kg-ld,,49699333,2023-08-21T13:48:15Z,Superseded by #2148.,107914493
issues,1859415334,PR_kwDOBm6k_c5YY5Ea,"Bump sphinx, furo, blacken-docs dependencies",49699333,2023-08-21T13:48:11Z,"Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [blacken-docs](https://github.com/asottile/blacken-docs).

Updates `sphinx` from 7.1.2 to 7.2.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.2.2</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.2.2 (released Aug 17, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>Fix the signature of the <code>StateMachine.insert_input()</code> patch,
for when calling with keyword arguments.</li>
<li>Fixed membership testing (<code>in</code>) for the :py:class:<code>str</code> interface
of the asset classes (<code>_CascadingStyleSheet</code> and <code>_JavaScript</code>),
which several extensions relied upon.</li>
<li>Fixed a type error in <code>SingleFileHTMLBuilder._get_local_toctree</code>,
<code>includehidden</code> may be passed as a string or a boolean.</li>
<li>Fix <code>:noindex:</code> for <code>PyModule</code> and JSModule``.</li>
</ul>
<h1>Release 7.2.1 (released Aug 17, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>Restored the the :py:class:<code>str</code> interface of the asset classes
(<code>_CascadingStyleSheet</code> and <code>_JavaScript</code>), which several extensions relied upon.
This will be removed in Sphinx 9.</li>
<li>Restored calls to <code>Builder.add_{css,js}_file()</code>,
which several extensions relied upon.</li>
<li>Restored the private API <code>TocTree.get_toctree_ancestors()</code>,
which several extensions relied upon.</li>
</ul>
<h1>Release 7.2.0 (released Aug 17, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11511"">#11511</a>: Drop Python 3.8 support.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11576"">#11576</a>: Require Pygments 2.14 or later.</li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11512"">#11512</a>: Deprecate <code>sphinx.util.md5</code> and <code>sphinx.util.sha1</code>.
Use <code>hashlib</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11526"">#11526</a>: Deprecate <code>sphinx.testing.path</code>.
Use <code>os.path</code> or <code>pathlib</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11528"">#11528</a>: Deprecate <code>sphinx.util.split_index_msg</code> and <code>sphinx.util.split_into</code>.
Use <code>sphinx.util.index_entries.split_index_msg</code> instead.</li>
<li>Deprecate <code>sphinx.builders.html.Stylesheet</code>
and <code>sphinx.builders.html.Javascript</code>.
Use <code>sphinx.application.Sphinx.add_css_file()</code></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ed84d63e6f2c4fd43b97fc43ee8be4156a13af9e""><code>ed84d63</code></a> Bump to 7.2.2 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ea4a73e148022244bba027b86a8c2bf7067e97be""><code>ea4a73e</code></a> [bot]: Update message catalogues (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11612"">#11612</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e47846a812f09ae2ca9e0444ed227934c2eb9c6e""><code>e47846a</code></a> Fix <code>:noindex:</code> for <code>PyModule</code> and JSModule``</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/b2fc47fe3dd0adba46edbb21ef03102ecb997f89""><code>b2fc47f</code></a> Add CHANGES entry for renaming the <code>StateMachine.insert_input()</code> parameter</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/0835c3e8ea6b23fb12b29d9c5104e06595f400b6""><code>0835c3e</code></a> Fix regression in <code>SingleFileHTMLBuilder._get_local_toctree</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/49dc0dd3999f66f27728b974394585422f10d947""><code>49dc0dd</code></a> Fix asset class string interface membership testing</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/8512855776ddd4790547d7e46ad98fd3ee48c2fb""><code>8512855</code></a> Fix signature of docutils include_source monkeypatch (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11610"">#11610</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e1d9068c60b43cf8dcb8dbd62ba26900eb63ff3b""><code>e1d9068</code></a> Bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/441a9e4d622e5f00f9fed8f59b9cac251cff9f80""><code>441a9e4</code></a> Bump to 7.2.1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ec3185373044c120298501dbe91444bdb6ccb04c""><code>ec31853</code></a> Restore <code>TocTree.get_toctree_ancestors()</code></li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v7.1.2...v7.2.2"">compare view</a></li>
</ul>
</details>
<br />

Updates `furo` from 2023.7.26 to 2023.8.19
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.08.19 -- Xenolithic Xanadu</h2>
<ul>
<li>Fix missing search context with Sphinx 7.2, for dirhtml builds.</li>
<li>Drop support for Python 3.7.</li>
<li>Present configuration errors in a better format -- thanks <a href=""https://github.com/AA-Turner""><code>@​AA-Turner</code></a>!</li>
<li>Bump <code>require_sphinx()</code> to Sphinx 6.0, in line with dependency changes in Unassuming Ultramarine.</li>
</ul>
<h2>2023.08.17 -- Wonderous White</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.2.0 and 7.2.1.</li>
</ul>
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<ul>
<li>Update Python version classifiers.</li>
<li>Increase the icon size in mobile header.</li>
<li>Increase admonition title bg opacity.</li>
<li>Change the default API background to transparent.</li>
<li>Transition the API background change.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/0766bb2d6d0392bf90d4e4ac447ae721fa673ee9""><code>0766bb2</code></a> Prepare release: 2023.08.19</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/807d73cb1080a1c9b8321a1dcffd84aee73e859a""><code>807d73c</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/364b2611f27e463cd1f879bf348975c01294ffc0""><code>364b261</code></a> Accomodate for the required <code>data-content_root</code> for search</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/0d38bc6cffc528ce3cf742fe2bd20e83c46d7891""><code>0d38bc6</code></a> Simplify retrieval of <code>pygments_dark_style</code> value (<a href=""https://redirect.github.com/pradyunsg/furo/issues/699"">#699</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/3631ffc4ea2f241b89bbd9b6c9d76e52a4226c56""><code>3631ffc</code></a> Use <code>sphinx.errors.ConfigError</code> (<a href=""https://redirect.github.com/pradyunsg/furo/issues/697"">#697</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/d2e2448852752ec570313bc1daea3bda56069816""><code>d2e2448</code></a> Remove duplicate HTML builder check (<a href=""https://redirect.github.com/pradyunsg/furo/issues/698"">#698</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/7b4f1308832e99d8e49c80f24665e938673c1644""><code>7b4f130</code></a> Drop Python 3.7 (<a href=""https://redirect.github.com/pradyunsg/furo/issues/701"">#701</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/e322b71b10c959cce0af80d3ac6934725ce728bb""><code>e322b71</code></a> Remove pointless assert (<a href=""https://redirect.github.com/pradyunsg/furo/issues/702"">#702</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/ee2097a797e92f1155a17287d063b3181f137115""><code>ee2097a</code></a> Bump <code>require_sphinx()</code> to Sphinx 6.0 (<a href=""https://redirect.github.com/pradyunsg/furo/issues/700"">#700</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c1ff10b768aaa2a2e479bbf8d58187e12a2731ac""><code>c1ff10b</code></a> Back to development</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.07.26...2023.08.19"">compare view</a></li>
</ul>
</details>
<br />

Updates `blacken-docs` from 1.15.0 to 1.16.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.16.0 (2023-08-16)</h2>
<ul>
<li>
<p>Allow Markdown fence options.</p>
<p>Thanks to initial work from Matthew Anderson in <code>PR [#246](https://github.com/asottile/blacken-docs/issues/246) &lt;https://github.com/adamchainz/blacken-docs/pull/246&gt;</code>__.</p>
</li>
<li>
<p>Expand Markdown detection to all Python language names from Pygments: <code>py</code>, <code>sage</code>, <code>python3</code>, <code>py3</code>, and <code>numpy</code>.</p>
</li>
<li>
<p>Preserve leading whitespace lines in reStructuredText code blocks.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#217](https://github.com/asottile/blacken-docs/issues/217) &lt;https://github.com/adamchainz/blacken-docs/issues/217&gt;</code>__.</p>
</li>
<li>
<p>Use exit code 2 to indicate errors from Black, whilst exit code 1 remains for “files have been formatted”.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#218](https://github.com/asottile/blacken-docs/issues/218) &lt;https://github.com/adamchainz/blacken-docs/issues/218&gt;</code>__.</p>
</li>
<li>
<p>Support passing the <code>--preview</code> option through to Black, to select the future style.</p>
</li>
<li>
<p>Remove <code>language_version</code> from <code>.pre-commit-hooks.yaml</code>.
This change allows <code>default_language_version</code> in ``.pre-commit-config.yaml` to take precedence.</p>
<p>Thanks to Aneesh Agrawal in <code>PR [#258](https://github.com/asottile/blacken-docs/issues/258) &lt;https://github.com/adamchainz/blacken-docs/pull/258&gt;</code>__.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/960ead214cd1184149d366c6d27ca6c369ce46b6""><code>960ead2</code></a> Version 1.16.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8f0ed18ce69d35e7e631d2b01983ecf8248667a0""><code>8f0ed18</code></a> Support passing --preview through to Black (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/273"">#273</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/4eb4e4cdbff31cde3d9ffdd7693d5b7dce4dc506""><code>4eb4e4c</code></a> Tweak changelog note</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/6c7450cfa90581b482d71b451b2d007361aa47c1""><code>6c7450c</code></a> Use exit code 2 to indicate errors (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/272"">#272</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/99dfc8d0b402b9b7b0f0e723fc1d030063249eac""><code>99dfc8d</code></a> Preserve leading whitespace lines in rST (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/271"">#271</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/94465e83e7253b4caef1e00fcb30b74ced33f08e""><code>94465e8</code></a> Reformat markdown tests with dedent() (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/270"">#270</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7cd5f30f0f9e014bda566a1893df3b9c7f9a2c48""><code>7cd5f30</code></a> Use .md in glob example</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f97e56994178e079ab807cebe54c2044f7b2ae55""><code>f97e569</code></a> Document applying to many files (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/269"">#269</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/ae612b095b5005eee8c3ab9fe65a248d7654582d""><code>ae612b0</code></a> Expand Markdown detection to all Python language names (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/268"">#268</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/da9b4557afc9b820c40b8d6bdb8f515c6cdec0f5""><code>da9b455</code></a> Replace NamedTuple with plain class (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/267"">#267</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/asottile/blacken-docs/compare/1.15.0...1.16.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name> dependency` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name> dependency` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2148.org.readthedocs.build/en/2148/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1685471752,IC_kwDOBm6k_c5kdkII,,77071,2023-08-21T01:07:23Z,"@simonw, since you're referencing ""rowid"" column by name, I just want to note that there may be an existing rowid column with completely different semantics (https://www.sqlite.org/lang_createtable.html#rowid), which is likely to break this logic. I don't see a good way to detect a proper ""rowid"" name short of checking if there is a field with that name and using the alternative (`_rowid_` or `oid`), which is not ideal, but may work.

In terms of the original issue, maybe a way to deal with it is to use rowid by default and then use primary key for WITHOUT ROWID tables (as they are guaranteed to be not null), but I suspect it may require significant changes to the API (and doesn't fully address the issue of what value to pass to indicate NULL when editing records). Would it make sense to generate a random string to indicate NULL values when editing?",107914493
issues,1858228057,I_kwDOBm6k_c5uwk9Z,Plugin hook for database queries that are run,18899,2023-08-20T18:43:50Z,"I'm interested in making a plugin that saves every query that gets run to a table in the database. (I know about datasette-query-history but thought it would be good to have a server-side option.)

As far as I can tell reading the docs, there isn't really a hook setup to allow this.

Maybe I could hack it with some of the hooks that are passed requests, but that doesn't seem good.

I'm a little surprised this isn't possible, so I thought I would open an issue and see if that's a deeply considered decision or just ""haven't needed it yet."" I'm potentially interested in implementing the hook if the latter.",107914493
issue_comments,1685263948,IC_kwDOBm6k_c5kcxZM,,11784304,2023-08-20T11:50:10Z,"This also makes it simple to separate out secrets.

`datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml`

settings.yaml
```
settings:
  default_page_size: 10
  max_returned_rows: 3000
  sql_time_limit_ms"": 8000
plugins:
  datasette-ripgrep:
    path: /usr/local/lib/python3.11/site-packages
```

secrets.yaml
```
plugins:
  datasette-auth-github:
            client_secret: SUCH_SECRET 
```


db-docs.yaml
```
databases:
  docs:
    permissions:
      create-table:
        id: editor
```

db-fixtures.yaml
```
databases:
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search
```",107914493
issue_comments,1685260624,IC_kwDOBm6k_c5kcwlQ,,11784304,2023-08-20T11:31:16Z,https://pypi.org/project/deep-chainmap/,107914493
issue_comments,1685260244,IC_kwDOBm6k_c5kcwfU,,11784304,2023-08-20T11:29:00Z,https://docs.python.org/3/library/collections.html#collections.ChainMap,107914493
issue_comments,1685259985,IC_kwDOBm6k_c5kcwbR,,11784304,2023-08-20T11:27:21Z,"To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight. Even separating db/table-scope settings from pure descriptive metadata is not always easy. Like, do canned queries belong to database metadata or to settings? Do I need two separate files for this?

One pragmatic solution I used in a project is stacking yaml configuration files. Basically, have an arbitrary number of yaml or json settings files that you load in a specified order. Every file adds to the corresponding settings in the earlier-loaded file (if it already existed). I implemented this myself but found later that there is an existing Python ""cascading dict"" type of thing, I forget what it's called. There is a bit of a challenge deciding whether there is ""replacement"" or ""addition"" (I think I pragmatically ran `update` on the second level of the dict but better solutions are certainly possible). 

This way, one allows separation of settings into different blocks, while not imposing a specific idea of what belongs where that might not apply equally to all cases.
",107914493
issue_comments,1685096381,IC_kwDOCGYnMM5kcIe9,,9599,2023-08-19T20:04:32Z,I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here.,140912432
issue_comments,1685096284,IC_kwDOCGYnMM5kcIdc,,9599,2023-08-19T20:03:59Z,"Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in:
- #577",140912432
issue_comments,1685096129,IC_kwDOCGYnMM5kcIbB,,9599,2023-08-19T20:03:00Z,"Simplest possible recreation of the bug:
```bash
python -c '
import sqlite_utils
db = sqlite_utils.Database(memory=True)
db.execute(""""""
CREATE TABLE ""logs"" (
   [id] INTEGER PRIMARY KEY,
   [chat_id] INTEGER REFERENCES [log]([id]),
   [reply_to_id] INTEGER
);
"""""")
db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"")
'
```
That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists!",140912432
issues,1857851384,I_kwDOCGYnMM5uvI_4,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference,9599,2023-08-19T20:01:26Z,"Extremely detailed story of how I got to this point:

- https://github.com/simonw/llm/issues/162

Steps to reproduce (only if that pragma is on though):
```bash
python -c '
import sqlite_utils
db = sqlite_utils.Database(memory=True)
db.execute(""""""
CREATE TABLE ""logs"" (
   [id] INTEGER PRIMARY KEY,
   [model] TEXT,
   [prompt] TEXT,
   [system] TEXT,
   [prompt_json] TEXT,
   [options_json] TEXT,
   [response] TEXT,
   [response_json] TEXT,
   [reply_to_id] INTEGER,
   [chat_id] INTEGER REFERENCES [log]([id]),
   [duration_ms] INTEGER,
   [datetime_utc] TEXT
);
"""""")
db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"")
'
```
This succeeds in some environments, fails in others.",140912432
issue_comments,1684530060,IC_kwDOBm6k_c5kZ-OM,,9599,2023-08-18T23:09:03Z,"Ran a quick benchmark on ChatGPT Code Interpreter: https://chat.openai.com/share/8357dc01-a97e-48ae-b35a-f06249935124

Conclusion from there is that this query returns fast no matter how much the table grows:

```sql
SELECT EXISTS(SELECT 1 FROM ""nasty"" WHERE ""id"" IS NULL)
```
So detecting if a table contains any null primary keys is definitely feasible without a performance hit.",107914493
issue_comments,1684526447,IC_kwDOBm6k_c5kZ9Vv,,9599,2023-08-18T23:05:02Z,How expensive is it to detect if a SQLite table contains at least one `null` primary key?,107914493
issue_comments,1684525943,IC_kwDOBm6k_c5kZ9N3,,9599,2023-08-18T23:04:14Z,"This is hard. I tried this:
```python
def path_from_row_pks(row, pks, use_rowid, quote=True):
    """"""Generate an optionally tilde-encoded unique identifier
    for a row from its primary keys.""""""
    if use_rowid or any(row[pk] is None for pk in pks):
        bits = [row[""rowid""]]
    else:
        bits = [
            row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks
        ]
    if quote:
        bits = [tilde_encode(str(bit)) for bit in bits]
    else:
        bits = [str(bit) for bit in bits]

    return "","".join(bits)
```
The `    if use_rowid or any(row[pk] is None for pk in pks)` bit is new.

But I got this error on http://127.0.0.1:8003/nulls/nasty :

```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 1364, in run_display_columns_and_rows
    display_columns, display_rows = await display_columns_and_rows(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 186, in display_columns_and_rows
    pk_path = path_from_row_pks(row, pks, not pks, False)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 124, in path_from_row_pks
    bits = [row[""rowid""]]
IndexError: No item with that key
```
Because the SQL query I ran to populate the page didn't know that it would need to select `rowid` as well.",107914493
issue_comments,1684525054,IC_kwDOBm6k_c5kZ8_-,,9599,2023-08-18T23:02:26Z,"Creating a quick test database:
```bash
sqlite-utils create-table nulls.db nasty id text --pk id
sqlite-utils nulls.db 'insert into nasty (id) values (null)'
```",107914493
issue_comments,1684523322,IC_kwDOBm6k_c5kZ8k6,,9599,2023-08-18T22:59:14Z,"Except it looks like the Links from other tables section is broken:

<img width=""649"" alt=""CleanShot 2023-08-18 at 15 58 54@2x"" src=""https://github.com/simonw/datasette/assets/9599/ddebf0a7-9855-4cf0-b283-f06f8c25414d"">
",107914493
issue_comments,1684522567,IC_kwDOBm6k_c5kZ8ZH,,9599,2023-08-18T22:58:07Z,"Here's a prototype of that:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index b2644ace..acc55249 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -1386,7 +1386,7 @@ class Datasette:
         )
         add_route(
             RowView.as_view(self),
-            r""/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[^/]+?)(\.(?P<format>\w+))?$"",
+            r""/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[A-Za-z0-9\_\-\~]+|\.\d+)(\.(?P<format>\w+))?$"",
         )
         add_route(
             TableInsertView.as_view(self),
@@ -1440,7 +1440,15 @@ class Datasette:
     async def resolve_row(self, request):
         db, table_name, _ = await self.resolve_table(request)
         pk_values = urlsafe_components(request.url_vars[""pks""])
-        sql, params, pks = await row_sql_params_pks(db, table_name, pk_values)
+
+        if len(pk_values) == 1 and pk_values[0].startswith("".""):
+            # It's a special .rowid value
+            pk_values = (pk_values[0][1:],)
+            sql, params, pks = await row_sql_params_pks(
+                db, table_name, pk_values, rowid=True
+            )
+        else:
+            sql, params, pks = await row_sql_params_pks(db, table_name, pk_values)
         results = await db.execute(sql, params, truncate=True)
         row = results.first()
         if row is None:
diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py
index c388673d..96669281 100644
--- a/datasette/utils/__init__.py
+++ b/datasette/utils/__init__.py
@@ -1206,9 +1206,12 @@ def truncate_url(url, length):
     return url[: length - 1] + ""…""
 
 
-async def row_sql_params_pks(db, table, pk_values):
+async def row_sql_params_pks(db, table, pk_values, rowid=False):
     pks = await db.primary_keys(table)
-    use_rowid = not pks
+    if rowid:
+        use_rowid = True
+    else:
+        use_rowid = not pks
     select = ""*""
     if use_rowid:
         select = ""rowid, *""
```
It works:

<img width=""649"" alt=""CleanShot 2023-08-18 at 15 57 36@2x"" src=""https://github.com/simonw/datasette/assets/9599/93803f41-d194-4193-8320-7c47d0373372"">

<img width=""649"" alt=""CleanShot 2023-08-18 at 15 57 53@2x"" src=""https://github.com/simonw/datasette/assets/9599/28d28efa-5ee0-469b-a31d-ce00f2d1f77a"">
",107914493
issue_comments,1684505071,IC_kwDOBm6k_c5kZ4Hv,,9599,2023-08-18T22:44:35Z,"Also relevant:
https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L1147-L1153",107914493
issue_comments,1684504398,IC_kwDOBm6k_c5kZ39O,,9599,2023-08-18T22:43:31Z,"`(?P<pks>[^/]+?)` could instead be a regex that is restricted to the tilde-encoded set of characters, or `\.\d+`.",107914493
issue_comments,1684504051,IC_kwDOBm6k_c5kZ33z,,9599,2023-08-18T22:43:06Z,Here's the regex in question at the moment: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/app.py#L1387-L1390,107914493
issue_comments,1684503587,IC_kwDOBm6k_c5kZ3wj,,9599,2023-08-18T22:42:28Z,"I could set a rule that extensions (including custom render extensions set by plugins) must not be valid integers, and teach Datasette that `/\.\d+` is the indication of a `rowid`.",107914493
issue_comments,1684503189,IC_kwDOBm6k_c5kZ3qV,,9599,2023-08-18T22:41:51Z,"```pycon
>>> tilde_encode(""~"")
'~7E'
>>> tilde_encode(""."")
'~2E'
>>> tilde_encode(""-"")
'-'
```
I think `.` might be the way to do this:

     /database/table/.4

But... I worry about that colliding with my URL routing code that spots the difference between these:

     /database/table/.4
     /database/table/.4.json
     /database/table/.4.csv

etc.",107914493
issue_comments,1684502278,IC_kwDOBm6k_c5kZ3cG,,9599,2023-08-18T22:40:20Z,"From reviewing https://simonwillison.net/2022/Mar/19/weeknotes/

    unreserved  = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~""

That's how I chose the tilde character - but it also suggests that I could use `-` or `.` or `_` for my new `rowid` encoding.

So maybe `/database/table/_4` could indicate ""the row with `rowid` of 4"".

No, that doesn't work:
```pycon
>>> from datasette.utils import tilde_encode
>>> tilde_encode(""_"")
'_'
```
I need a character which tilde-encoding does indeed encode.",107914493
issue_comments,1684500540,IC_kwDOBm6k_c5kZ3A8,,9599,2023-08-18T22:37:37Z,"I just found this and panicked, thinking maybe tilde encoding is a bad idea after all! https://jkorpela.fi/tilde.html

But... ""Date of last update: 1999-08-27"" - I think I'm OK.",107914493
issue_comments,1684500172,IC_kwDOBm6k_c5kZ27M,,9599,2023-08-18T22:37:04Z,"Looking at the way these URLs work: because the components themselves in `a~2Fb,~2Ec-d` are tilde-encoded, any character that's ""safe"" in tilde-encoding could be used to indicate ""this is actually a rowid"".

",107914493
issue_comments,1684498947,IC_kwDOBm6k_c5kZ2oD,,9599,2023-08-18T22:35:04Z,"The most interesting row URL in the fixtures database right now is this one:

https://latest.datasette.io/fixtures/compound_primary_key/a~2Fb,~2Ec-d

<img width=""163"" alt=""CleanShot 2023-08-18 at 15 34 52@2x"" src=""https://github.com/simonw/datasette/assets/9599/dd79b0b6-e1c3-4e30-b64b-973b062bfe70"">
",107914493
issue_comments,1684497642,IC_kwDOBm6k_c5kZ2Tq,,9599,2023-08-18T22:32:53Z,"Here's a potential solution: make it so ALL `rowid` tables in SQLite can be optionally addressed by their `rowid` instead of by their primary key.

Then teach the code that outputs the URL to a row page to spot if there are `null` primary keys and switch to that alternative addressing mechanism instead.",107914493
issue_comments,1684497000,IC_kwDOBm6k_c5kZ2Jo,,9599,2023-08-18T22:31:53Z,"So it sounds like SQLite does ensure that a `rowid` before it allows a primary key to be null.

So one solution here would be to detect a null primary key and switch that table over to using `rowid` URLs instead. The key problem we're trying to solve here after all is how to link to a row:

https://latest.datasette.io/fixtures/infinity/1

But when would we run that check? And does every row in the table get a new `/rowid/` URL just because someone messed up and inserted a `null` by mistake?",107914493
issue_comments,1684496274,IC_kwDOBm6k_c5kZ1-S,,15178711,2023-08-18T22:30:45Z,"> That said, I do really like a bias towards settings that can be changed at runtime

Does this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff. 

Well it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. ",107914493
issue_comments,1684495674,IC_kwDOBm6k_c5kZ106,,9599,2023-08-18T22:29:47Z,"https://www.sqlite.org/lang_createtable.html#the_primary_key says:

>According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid) or the table is a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table or a [STRICT](https://www.sqlite.org/stricttables.html) table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns.",107914493
issue_comments,1684494464,IC_kwDOBm6k_c5kZ1iA,,9599,2023-08-18T22:27:51Z,"Oh wow, null primary keys are bad news... SQLite lets you insert multiple rows with the same `null` value!
```pycon
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
<Table foo (id, name)>
>>> db.schema
'CREATE TABLE [foo] (\n   [id] TEXT PRIMARY KEY,\n   [name] TEXT\n);'
>>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
<Table foo (id, name)>
>>> db.schema
'CREATE TABLE [foo] (\n   [id] TEXT PRIMARY KEY,\n   [name] TEXT\n);'
>>> list(db[""foo""].rows)
[{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}]
>>> list(db.query('select * from foo where id = null'))
[]
>>> list(db.query('select * from foo where id is null'))
[{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}]
```",107914493
issue_comments,1684488526,IC_kwDOBm6k_c5kZ0FO,,9599,2023-08-18T22:18:39Z,"> Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

> Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

I'm not a fan of that. I feel like software history is full of examples of projects that implemented configuration-as-code and then later regretted it - the most recent example is `setup.py` in Python turning into `pyproject.yaml`, but I feel like I've seen that pattern play out elsewhere too.

I don't think having people dynamically generate JSON/YAML for their configuration is a big burden. I'd have to see some very compelling use-cases to convince me otherwise.

That said, I do really like a bias towards settings that can be changed at runtime. Datasette has suffered a bit from some settings that can't be easily changed at runtime already - hence my gnarly https://github.com/simonw/datasette-remote-metadata plugin.

For things like Datasette Cloud for example the more people can configure without rebooting their container the better!

I don't think live reconfiguration at runtime is incompatible with JSON/YAML configuration though. Caddy is one of my favourite examples of software that can be entirely re-configured at runtime by POSTING a big blob of JSON to it: https://caddyserver.com/docs/quick-starts/api
",107914493
issue_comments,1684485591,IC_kwDOBm6k_c5kZzXX,,9599,2023-08-18T22:14:35Z,"Actually there is one thing that I'm not comfortable about with respect to the existing design: the way the database / tables stuff is nested.

They assume that the user will attach the database to Datasette using a fixed name - `docs.db` or whatever.

But what if we want to support users downloading databases from each other and attaching them to Datasette where those DBs might carry some of their own configuration?

Moving metadata into the databases makes sense there, but what about database-specific settings like the default sort order for a table, or configured canned queries?

Having those tied to the filename of the database itself feels unpleasant to me. But how else could we handle this?",107914493
issue_comments,1684484426,IC_kwDOBm6k_c5kZzFK,,9599,2023-08-18T22:12:52Z,"Yeah, I'm convinced by that. There's not point in having both `settings.json` and `datasette.json`.

I like `datasette.json` ( / `datasette.yml`) as a name. That can be the file that lives in your config directory too, so if you run `datasette .` in a folder containing `datasette.yml` all of those settings get picked up.

Here's a thought for how it could look - I'll go with the YAML format because I expect that to be the default most people use, just because it supports multi-line strings better.

I based this on the big example at https://docs.datasette.io/en/1.0a3/metadata.html#using-yaml-for-metadata - and combined some bits from https://docs.datasette.io/en/1.0a3/authentication.html as well.

```yaml
title: Demonstrating Metadata from YAML
description_html: |-
  <p>This description includes a long HTML string</p>
  <ul>
    <li>YAML is better for embedding HTML strings than JSON!</li>
  </ul>

settings:
  default_page_size: 10
  max_returned_rows: 3000
  sql_time_limit_ms"": 8000

databases:
  docs:
    permissions:
      create-table:
        id: editor
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search

permissions:
  debug-menu:
    id: '*'

plugins:
  datasette-ripgrep:
    path: /usr/local/lib/python3.11/site-packages
```
I'm inclined to say we try to be a super-set of the existing `metadata.yml` format, at least where it makes sense to do so. That way the upgrade path is smooth for people. Also, I don't think the format itself is terrible - it's the name that's the big problem.

In this example I've mixed in one extra concept: that `settings:` block with a bunch of settings in it.

There are some things in there that look a little bit like metadata - the `title` and `description_html` fields.

But _are they_ metadata? The title and description of the overall instance feels like it could be described as general configuration. The stuff for the `query` should live where the query itself is defined.

Note that queries can be defined by a plugin hook too: https://docs.datasette.io/en/1.0a3/plugin_hooks.html#canned-queries-datasette-database-actor

What do you think? Is this the right direction, or are you thinking there's a more radical redesign that would make sense here?",107914493
issue_comments,1684384750,IC_kwDOBm6k_c5kZavu,,9599,2023-08-18T20:07:18Z,The big challenge here is what the URL to that row page should look like. How can I encode a `None` in a form that can be encoded and decoded without clashing with primary keys that are the string `None` or `null`?,107914493
issues,1857234285,I_kwDOBm6k_c5usyVt,If a row has a primary key of `null` various things break,9599,2023-08-18T20:06:28Z,"Stumbled across this while experimenting with `datasette-write-ui`. The error I got was a 500 on the `/db` page:

> `'NoneType' object has no attribute 'encode'`

Tracked it down to this code, which assembles the URL for a row page:

https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L120-L134

That's because `tilde_encode` can't handle `None`: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L1175-L1178

",107914493
issue_comments,1684235760,IC_kwDOCGYnMM5kY2Xw,,9599,2023-08-18T17:43:11Z,"Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it:

https://github.com/simonw/sqlite-utils-fast-fks",140912432
issue_comments,1684205563,IC_kwDOBm6k_c5kYu_7,,15178711,2023-08-18T17:12:54Z,"Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

Though I imagine Python imports might make this complex to do, and json/yaml is already supported and pretty easy to write
",107914493
issue_comments,1684202932,IC_kwDOBm6k_c5kYuW0,,15178711,2023-08-18T17:10:21Z,"I agree with all your points!

I think the best solution would be having a `datasette.json` config file, where you ""configure"" your datasette instances, with settings, permissions/auth, plugin configuration, and table settings (sortable column, label columns, etc.). Which #2093 would do.

Then optionally, you have a `metadata.json`, or use `datasette_metadata`, or some other plugin to define metadata (ex the future [sqlite-docs](https://github.com/asg017/sqlite-docs) plugin).

Everything in `datasette.json` could also be overwritten by CLI flags, like `--setting key value`, `--plugin xxxx key value`.

We could even completely remove `settings.json` in favor or just `datasette.json`. Mostly because I think the less files the better, especially if they have generic names like `settings.json` or `config.json`. ",107914493
issue_comments,1683963463,IC_kwDOBm6k_c5kXz5H,,22429695,2023-08-18T13:58:39Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`3a97755`)](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2144   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          40       40           
  Lines        5937     5937           
=======================================
  Hits         5466     5466           
  Misses        471      471           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1856760386,PR_kwDOBm6k_c5YQGcc,Bump the python-packages group with 3 updates,49699333,2023-08-18T13:49:37Z,"Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [blacken-docs](https://github.com/asottile/blacken-docs).

Updates `sphinx` from 7.1.2 to 7.2.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.2.2</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.2.2 (released Aug 17, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>Fix the signature of the <code>StateMachine.insert_input()</code> patch,
for when calling with keyword arguments.</li>
<li>Fixed membership testing (<code>in</code>) for the :py:class:<code>str</code> interface
of the asset classes (<code>_CascadingStyleSheet</code> and <code>_JavaScript</code>),
which several extensions relied upon.</li>
<li>Fixed a type error in <code>SingleFileHTMLBuilder._get_local_toctree</code>,
<code>includehidden</code> may be passed as a string or a boolean.</li>
<li>Fix <code>:noindex:</code> for <code>PyModule</code> and JSModule``.</li>
</ul>
<h1>Release 7.2.1 (released Aug 17, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li>Restored the the :py:class:<code>str</code> interface of the asset classes
(<code>_CascadingStyleSheet</code> and <code>_JavaScript</code>), which several extensions relied upon.
This will be removed in Sphinx 9.</li>
<li>Restored calls to <code>Builder.add_{css,js}_file()</code>,
which several extensions relied upon.</li>
<li>Restored the private API <code>TocTree.get_toctree_ancestors()</code>,
which several extensions relied upon.</li>
</ul>
<h1>Release 7.2.0 (released Aug 17, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11511"">#11511</a>: Drop Python 3.8 support.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11576"">#11576</a>: Require Pygments 2.14 or later.</li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11512"">#11512</a>: Deprecate <code>sphinx.util.md5</code> and <code>sphinx.util.sha1</code>.
Use <code>hashlib</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11526"">#11526</a>: Deprecate <code>sphinx.testing.path</code>.
Use <code>os.path</code> or <code>pathlib</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11528"">#11528</a>: Deprecate <code>sphinx.util.split_index_msg</code> and <code>sphinx.util.split_into</code>.
Use <code>sphinx.util.index_entries.split_index_msg</code> instead.</li>
<li>Deprecate <code>sphinx.builders.html.Stylesheet</code>
and <code>sphinx.builders.html.Javascript</code>.
Use <code>sphinx.application.Sphinx.add_css_file()</code></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ed84d63e6f2c4fd43b97fc43ee8be4156a13af9e""><code>ed84d63</code></a> Bump to 7.2.2 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ea4a73e148022244bba027b86a8c2bf7067e97be""><code>ea4a73e</code></a> [bot]: Update message catalogues (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11612"">#11612</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e47846a812f09ae2ca9e0444ed227934c2eb9c6e""><code>e47846a</code></a> Fix <code>:noindex:</code> for <code>PyModule</code> and JSModule``</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/b2fc47fe3dd0adba46edbb21ef03102ecb997f89""><code>b2fc47f</code></a> Add CHANGES entry for renaming the <code>StateMachine.insert_input()</code> parameter</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/0835c3e8ea6b23fb12b29d9c5104e06595f400b6""><code>0835c3e</code></a> Fix regression in <code>SingleFileHTMLBuilder._get_local_toctree</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/49dc0dd3999f66f27728b974394585422f10d947""><code>49dc0dd</code></a> Fix asset class string interface membership testing</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/8512855776ddd4790547d7e46ad98fd3ee48c2fb""><code>8512855</code></a> Fix signature of docutils include_source monkeypatch (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11610"">#11610</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e1d9068c60b43cf8dcb8dbd62ba26900eb63ff3b""><code>e1d9068</code></a> Bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/441a9e4d622e5f00f9fed8f59b9cac251cff9f80""><code>441a9e4</code></a> Bump to 7.2.1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ec3185373044c120298501dbe91444bdb6ccb04c""><code>ec31853</code></a> Restore <code>TocTree.get_toctree_ancestors()</code></li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v7.1.2...v7.2.2"">compare view</a></li>
</ul>
</details>
<br />

Updates `furo` from 2023.7.26 to 2023.8.17
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.08.17 -- Wonderous White</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.2.0 and 7.2.1.</li>
</ul>
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<ul>
<li>Update Python version classifiers.</li>
<li>Increase the icon size in mobile header.</li>
<li>Increase admonition title bg opacity.</li>
<li>Change the default API background to transparent.</li>
<li>Transition the API background change.</li>
<li>Remove the &quot;indent&quot; of API entries which have a background.</li>
<li>Break long inline code literals.</li>
</ul>
<h2>2022.12.07 -- Reverent Raspberry</h2>
<ul>
<li>✨ Add support for Sphinx 6.</li>
<li>✨ Improve footnote presentation with docutils 0.18+.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/99f4e6df480ac3d043c09d472acd69390f47e402""><code>99f4e6d</code></a> Prepare release: 2023.08.17</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/59ad807bc6f8c8b36bf5a0bad2609e7781d8aaf9""><code>59ad807</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/9b94420776972613c3990a31467855ceceefcff7""><code>9b94420</code></a> Accomodate for Sphinx 7.2 behavior change</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/7aeb6d204f808ae0b0da8a5dee6f6035d9cd04cf""><code>7aeb6d2</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/pradyunsg/furo/issues/675"">#675</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/0c9be3f65d6ae84ff56d7679b7d650323b355be3""><code>0c9be3f</code></a> Back to development</li>
<li>See full diff in <a href=""https://github.com/pradyunsg/furo/compare/2023.07.26...2023.08.17"">compare view</a></li>
</ul>
</details>
<br />

Updates `blacken-docs` from 1.15.0 to 1.16.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.16.0 (2023-08-16)</h2>
<ul>
<li>
<p>Allow Markdown fence options.</p>
<p>Thanks to initial work from Matthew Anderson in <code>PR [#246](https://github.com/asottile/blacken-docs/issues/246) &lt;https://github.com/adamchainz/blacken-docs/pull/246&gt;</code>__.</p>
</li>
<li>
<p>Expand Markdown detection to all Python language names from Pygments: <code>py</code>, <code>sage</code>, <code>python3</code>, <code>py3</code>, and <code>numpy</code>.</p>
</li>
<li>
<p>Preserve leading whitespace lines in reStructuredText code blocks.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#217](https://github.com/asottile/blacken-docs/issues/217) &lt;https://github.com/adamchainz/blacken-docs/issues/217&gt;</code>__.</p>
</li>
<li>
<p>Use exit code 2 to indicate errors from Black, whilst exit code 1 remains for “files have been formatted”.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#218](https://github.com/asottile/blacken-docs/issues/218) &lt;https://github.com/adamchainz/blacken-docs/issues/218&gt;</code>__.</p>
</li>
<li>
<p>Support passing the <code>--preview</code> option through to Black, to select the future style.</p>
</li>
<li>
<p>Remove <code>language_version</code> from <code>.pre-commit-hooks.yaml</code>.
This change allows <code>default_language_version</code> in ``.pre-commit-config.yaml` to take precedence.</p>
<p>Thanks to Aneesh Agrawal in <code>PR [#258](https://github.com/asottile/blacken-docs/issues/258) &lt;https://github.com/adamchainz/blacken-docs/pull/258&gt;</code>__.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/960ead214cd1184149d366c6d27ca6c369ce46b6""><code>960ead2</code></a> Version 1.16.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8f0ed18ce69d35e7e631d2b01983ecf8248667a0""><code>8f0ed18</code></a> Support passing --preview through to Black (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/273"">#273</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/4eb4e4cdbff31cde3d9ffdd7693d5b7dce4dc506""><code>4eb4e4c</code></a> Tweak changelog note</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/6c7450cfa90581b482d71b451b2d007361aa47c1""><code>6c7450c</code></a> Use exit code 2 to indicate errors (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/272"">#272</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/99dfc8d0b402b9b7b0f0e723fc1d030063249eac""><code>99dfc8d</code></a> Preserve leading whitespace lines in rST (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/271"">#271</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/94465e83e7253b4caef1e00fcb30b74ced33f08e""><code>94465e8</code></a> Reformat markdown tests with dedent() (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/270"">#270</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7cd5f30f0f9e014bda566a1893df3b9c7f9a2c48""><code>7cd5f30</code></a> Use .md in glob example</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f97e56994178e079ab807cebe54c2044f7b2ae55""><code>f97e569</code></a> Document applying to many files (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/269"">#269</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/ae612b095b5005eee8c3ab9fe65a248d7654582d""><code>ae612b0</code></a> Expand Markdown detection to all Python language names (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/268"">#268</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/da9b4557afc9b820c40b8d6bdb8f515c6cdec0f5""><code>da9b455</code></a> Replace NamedTuple with plain class (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/267"">#267</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/asottile/blacken-docs/compare/1.15.0...1.16.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name> dependency` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name> dependency` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2144.org.readthedocs.build/en/2144/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1683950031,IC_kwDOBm6k_c5kXwnP,,49699333,2023-08-18T13:49:24Z,"Looks like these dependencies are updatable in another way, so this is no longer needed.",107914493
issue_comments,1683443891,IC_kwDOBm6k_c5kV1Cz,,9599,2023-08-18T06:58:15Z,"Hah, that `--plugin-secret` thing was a messy solution I came up with to the problem that all metadata is visible at `/-/metadata` - so if you need to stash a secret you need a way to keep it not-visible in there!

Hence the whole `$env` mess: https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values

```json
{
    ""plugins"": {
        ""datasette-auth-github"": {
            ""client_secret"": {
                ""$env"": ""GITHUB_CLIENT_SECRET""
            }
        }
    }
}
```

If configuration and metadata were separate we could ditch that whole messy situation - configuration can stay hidden, metadata can stay public.

Though I have been thinking that Datasette might benefit from a ""secrets"" mechanism that's separate from configuration and metadata... kind of like what LLM has: https://llm.datasette.io/en/stable/help.html#llm-keys-help",107914493
issue_comments,1683440597,IC_kwDOBm6k_c5kV0PV,,9599,2023-08-18T06:54:49Z,"A related point that I've been considering a lot recently: it turns out that sometimes I really want to define settings on the CLI instead of in a file, purely for convenience.

It's pretty annoying when I want to try out a new plugin but I have to create a dedicated `metadata.yml` file for it just to setup a single option - I'd love to have the option to be able to run this instead:

```bash
datasette data.db --plugin-setting datasette-upload-csvs default-database data
```

So maybe there's a world in which all of the settings can be applied in a `datasette.yml` file OR with command-line options.

That gets trickier when you need to pass a nested structure or similar, but we could always support those as JSON:

```bash
datasette data.db --plugin-setting datasette-emoji-reactions emoji '[""😼"", ""🐺""]'
```
Note that we kind of have precedent for this in `datasette publish`: https://docs.datasette.io/en/stable/publish.html#custom-metadata-and-plugins

```bash
datasette publish heroku my_database.db \
    --name my-heroku-app-demo \
    --install=datasette-auth-github \
    --plugin-secret datasette-auth-github client_id your_client_id \
    --plugin-secret datasette-auth-github client_secret your_client_secret
```",107914493
issue_comments,1683435579,IC_kwDOBm6k_c5kVzA7,,9599,2023-08-18T06:49:39Z,"My ideal situation then would be something like this:

- Metadata itself is VERY clearly described, including sensible rules for metadata inheritance where it makes sense. There is a `datasette.X` method for accessing it which is much more intuitive than `datasette.metadata()`.
- It's possible that method should be an `async` method, because that would support things like plugins that lookup metadata in database tables better.
- All templates etc switch to the new, clean, intuitive metadata mechanism before 1.0.
- I'm interested in the option of metadata being able to live in a `_datasette_metadata` table in the databases themselves - either as a plugin or as a core feature. I think it makes a lot of sense for metadata to optionally live with the data that it describes.
- Configuration gets split from metadata. The stuff that configures Datasette no longer lives in the `metadata.yml` file - it lives in `config.yml` (or even `datasette.yml`).

Currently we have three types of things:

- Metadata - information about the data
- Configuration - stuff like ""these columns should be sortable"" and ""this is configured as `fts_table`"" and suchlike
- Settings - the stuff that you pass to `datasette --setting x y` on server start.

Should settings and configuration be separate? I'm not 100% sure that they should - maybe those two concepts should be combined somehow.

Configuration directory mode needs to be considered too: https://docs.datasette.io/en/stable/settings.html#configuration-directory-mode - interestingly it already has a thing where it can pick up settings from a `settings.json` file - where settings are things like `datasette --setting sql_time_limit_ms 4000`.",107914493
issue_comments,1683429959,IC_kwDOBm6k_c5kVxpH,,9599,2023-08-18T06:43:33Z,"The single biggest design challenge I've had with metadata relates to how it should or should not be inherited.

If you apply a license to a Datasette instance, it feels like that should flow down to cover all of the databases and all of the tables within those databases.

If the license is at the database level, it should cover all tables.

But... should source do the same thing? I made it behave the same way as license, but it's presumably common for one database to have a single license but multiple different sources of data.

Then there's title - should that inherit? It feels like title should apply to only one level - you may want a title that applies to the instance, then a different custom title for databases and tables.

Here's the current state of play for metadata: https://docs.datasette.io/en/1.0a3/metadata.html

So there's `title` and `description` - and I'll be honest, I'm not 100% sure even I understand how those should be inherited down by tables/etc.

There's `description_html` which over-rides the `description` if it is set. It's a useful customization hack, but a bit surprising.

Then there are these six:

- `license`
- `license_url`
- `source`
- `source_url`
- `about`
- `about_url`

I added `about` later than the others, because I realized that plenty of my own projects needed a link to an article explaining them somewhere - e.g. https://scotrail.datasette.io/

Tables can also have column descriptions - just a string for each column. There's a demo of those here: https://latest.datasette.io/fixtures/roadside_attractions

And then there's all of the other stuff, most of which feels much more like ""settings"" than ""metadata"":

- `sort: created` - the custom sort order
- `size: 10` for a custom page size for a specific table
- `sortable_columns` to set which columns can be used to sort
- `hidden: true` to hide a table
- `label_column: title` is an interesting one - it lets you hint to Datasette which column should be displayed when there is a foreign key relationship. It's sort-of-metadata and sort-of-a-setting.
- `facets` sets default facets, see https://docs.datasette.io/en/1.0a3/facets.html#facets-in-metadata
- `facet_size` sets the number of facets to display
- `fts_table` and `fts_pk` can be used to configure FTS, especially for views: https://docs.datasette.io/en/1.0a3/full_text_search.html

And the authentication stuff! `allow` and `allow_sql` blocks: https://docs.datasette.io/en/1.0a3/authentication.html#defining-permissions-with-allow-blocks

And the new `permissions` key in the 1.0 alphas: https://docs.datasette.io/en/1.0a3/authentication.html#other-permissions-in-metadata

I think that might be everything (excluding the `plugins` settings stuff, which is also a bad fit for metadata.)

And to make things even more confusing... I believe you can add arbitrary key/value pairs to your metadata and then use them in your templates! I think I've heard from at least one person who uses that ability.",107914493
issue_comments,1683420879,IC_kwDOBm6k_c5kVvbP,,9599,2023-08-18T06:33:24Z,"I completely agree: metadata is a mess, and it deserves our attention.

> 1. Metadata cannot be updated without re-starting the entire Datasette instance.

That's not completely true - there are hacks around that. I have a plugin that applies one set of gnarly hacks for that here: https://github.com/simonw/datasette-remote-metadata - it's pretty grim though!

> 2. The `metadata.json`/`metadata.yaml` has become a kitchen sink of unrelated (imo) features like plugin config, authentication config, canned queries

100% this: it's a complete mess.

Datasette used to have a `datasette --config foo:bar` mechanism, which I deprecated in favour of `datasette --setting foo bar` partly because I wanted to free up `--config` for pointing at a real config file, so we could stop dropping everything in `--metadata metadata.yml`.

> 3. The Python APIs for defining extra metadata are a bit awkward (the `datasette.metadata()` class, `get_metadata()` hook, etc.)

Yes, they're not pretty at all.",107914493
issue_comments,1683404978,IC_kwDOCGYnMM5kVriy,,9599,2023-08-18T06:13:46Z,"I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues.",140912432
issue_comments,1683398866,IC_kwDOCGYnMM5kVqDS,,9599,2023-08-18T06:05:50Z,"Options:
- Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default?
- Read that `PRAGMA` and automatically do that view workaround if it's turned on
- Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection?
- Try the `transform()` in a transaction, detect the `""error in view""`, `""no such table""`error, if spotted then do the VIEW workaround and try again

I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler.",140912432
issue_comments,1683396150,IC_kwDOCGYnMM5kVpY2,,9599,2023-08-18T06:02:18Z,"More notes in here:
- https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873

Not all Python/SQLite installations exhibit this problem by default!

It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table

If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view.

Here's a one-liner to test if it is on or not:

```bash
python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""PRAGMA legacy_alter_table"").fetchall())'
```",140912432
issues,1856075668,I_kwDOCGYnMM5uoXeU,.transform() fails to drop column if table is part of a view,9599,2023-08-18T05:25:22Z,"I got this error trying to drop a column from a table that was part of a SQL view:

> error in view plugins: no such table: main.pypi_releases

Upon further investigation I found that this pattern seemed to fix it:
```python
def transform_the_table(conn):
    # Run this in a transaction:
    with conn:
        # We have to read all the views first, because we need to drop and recreate them
        db = sqlite_utils.Database(conn)
        views = {v.name: v.schema for v in db.views if table.lower() in v.schema.lower()}
        for view in views.keys():
            db[view].drop()
        db[table].transform(
            types=types,
            rename=rename,
            drop=drop,
            column_order=[p[0] for p in order_pairs],
        )
        # Now recreate the views
        for name, schema in views.items():
            db.create_view(name, schema)
```
So grab a copy of any view that might reference this table, start a transaction, drop those views, run the transform, recreate the views again.

> I wonder if this should become an option in `sqlite-utils`? Maybe a `recreate_views=True` argument for `table.tranform(...)`? Should it be opt-in or opt-out?

_Originally posted by @simonw in https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683370548_
            ",140912432
issue_comments,1683217284,IC_kwDOCGYnMM5kU9uE,,9599,2023-08-18T01:50:21Z,"And a test of the `--sql` option:
```bash
sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
sqlite-utils create-table /tmp/t.db country id integer name text
sqlite-utils create-table /tmp/t.db city id integer name text
sqlite-utils create-table /tmp/t.db continent id integer name text
sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql
```
Outputs:
```sql
CREATE TABLE [places_new_6a705d2f5a13] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER REFERENCES [country]([id]),
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
);
INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent])
   SELECT [id], [name], [country], [city], [continent] FROM [places];
DROP TABLE [places];
ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places];
```",140912432
issue_comments,1683212074,IC_kwDOCGYnMM5kU8cq,,9599,2023-08-18T01:43:54Z,"Some manual testing:
```bash
sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER
);
```
```bash
sqlite-utils create-table /tmp/t.db country id integer name text
sqlite-utils create-table /tmp/t.db city id integer name text
sqlite-utils create-table /tmp/t.db continent id integer name text
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER
);
CREATE TABLE [country] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [city] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [continent] (
   [id] INTEGER,
   [name] TEXT
);
```
```bash
sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [country] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [city] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [continent] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE ""places"" (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER REFERENCES [country]([id]),
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
);
```
```bash
sqlite-utils transform /tmp/t.db places --drop-foreign-key country
sqlite-utils schema /tmp/t.db places
```
```sql
CREATE TABLE ""places"" (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
)
```",140912432
issue_comments,1683201239,IC_kwDOCGYnMM5kU5zX,,9599,2023-08-18T01:30:46Z,"Help can now look like this:
```
  --drop-foreign-key TEXT         Drop foreign key constraint for this column
  --add-foreign-key <TEXT TEXT TEXT>...
                                  Add a foreign key constraint from a column
                                  to another table with another column
```",140912432
issue_comments,1683200128,IC_kwDOCGYnMM5kU5iA,,9599,2023-08-18T01:29:00Z,I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option.,140912432
issue_comments,1683198740,IC_kwDOCGYnMM5kU5MU,,9599,2023-08-18T01:26:47Z,"The only CLI feature that supports providing just the column name appears to be this:
```bash
sqlite-utils add-foreign-key --help
```
```
Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE]
                                    [OTHER_COLUMN]

  Add a new foreign key constraint to an existing table

  Example:

      sqlite-utils add-foreign-key my.db books author_id authors id

  WARNING: Could corrupt your database! Back up your database file first.
```
I can drop that WARNING now since I'm not writing to `sqlite_master` any more.",140912432
issue_comments,1683197882,IC_kwDOCGYnMM5kU4-6,,9599,2023-08-18T01:25:53Z,"Probably most relevant here is this snippet from:
```bash
sqlite-utils create-table --help
```
```
  --default <TEXT TEXT>...  Default value that should be set for a column
  --fk <TEXT TEXT TEXT>...  Column, other table, other column to set as a
                            foreign key
```",140912432
issue_comments,1683195669,IC_kwDOCGYnMM5kU4cV,,9599,2023-08-18T01:24:57Z,"Currently:
```bash
sqlite-utils transform --help
```
```
Usage: sqlite-utils transform [OPTIONS] PATH TABLE

  Transform a table beyond the capabilities of ALTER TABLE

  Example:

      sqlite-utils transform mydb.db mytable \
          --drop column1 \
          --rename column2 column_renamed

Options:
  --type <TEXT CHOICE>...   Change column type to INTEGER, TEXT, FLOAT or BLOB
  --drop TEXT               Drop this column
  --rename <TEXT TEXT>...   Rename this column to X
  -o, --column-order TEXT   Reorder columns
  --not-null TEXT           Set this column to NOT NULL
  --not-null-false TEXT     Remove NOT NULL from this column
  --pk TEXT                 Make this column the primary key
  --pk-none                 Remove primary key (convert to rowid table)
  --default <TEXT TEXT>...  Set default value for this column
  --default-none TEXT       Remove default from this column
  --drop-foreign-key TEXT   Drop foreign key constraint for this column
  --sql                     Output SQL without executing it
  --load-extension TEXT     Path to SQLite extension, with optional
                            :entrypoint
  -h, --help                Show this message and exit.
```",140912432
issues,1855894222,I_kwDOCGYnMM5unrLO,CLI equivalents to `transform(add_foreign_keys=)`,9599,2023-08-18T01:07:15Z,"The new options added in:
- #577
Deserve consideration in the CLI as well.

https://github.com/simonw/sqlite-utils/blob/d2bcdc00c6ecc01a6e8135e775ffdb87572b802b/sqlite_utils/db.py#L1706-L1708",140912432
issues,1855885427,I_kwDOBm6k_c5unpBz,De-tangling Metadata before Datasette 1.0,15178711,2023-08-18T00:51:50Z,"Metadata in Datasette is a really powerful feature, but is a bit difficult to work with. It was initially a way to add ""metadata"" about your ""data"" in Datasette instances, like descriptions for databases/tables/columns, titles, source URLs, licenses, etc. But it later became the go-to spot for other Datasette features that have nothing to do with metadata, like permissions/plugins/canned queries. 

Specifically, I've found the following problems when working with Datasette metadata:

1. Metadata cannot be updated without re-starting the entire Datasette instance.
2. The `metadata.json`/`metadata.yaml` has become a kitchen sink of unrelated (imo) features like plugin config, authentication config, canned queries
3. The Python APIs for defining extra metadata are a bit awkward (the `datasette.metadata()` class, `get_metadata()` hook, etc.)

## Possible solutions

Here's a few ideas of Datasette core changes we can make to address these problems. 

### Re-vamp the Datasette Python metadata APIs

The Datasette object has a single `datasette.metadata()` method that's a bit difficult to work with. There's also no Python API for inserted new metadata, so plugins have to rely on the `get_metadata()` hook.

The `get_metadata()` hook can also be improved - it doesn't work with async functions yet, so you're quite limited to what you can do.

(I'm a bit fuzzy on what to actually do here, but I imagine it'll be very small breaking changes to a few Python methods)

### Add an optional `datasette_metadata` table

Datasette should detect and use metadata stored in a new special table called `datasette_metadata`. This would be a regular table that a user can edit on their own, and would serve as a ""live updating"" source of metadata, than can be changed while the Datasette instance is running.

Not too sure what the schema would look like, but I'd imagine:

```sql
CREATE TABLE datasette_metadata(
  level text,
  target any,
  key text,
  value any,
  primary key (level, target)
)
```

Every row in this table would map to a single metadata ""entry"".

- `level` would be one of ""datasette"", ""database"", ""table"", ""column"", which is the ""level"" the entry describes. For example, `level=""table""` means it is metadata about a specific table, `level=""database""` for a specific database, or `level=""datasette""` for the entire Datasette instance.
- `target` would ""point"" to the specific object the entry metadata is about, and would depend on what `level` is specific. 
  - `level=""database""`: `target` would be the string name of the database that the metadata entry is about. ex `""fixtures""`
  - `level=""table""`: `target` would be a JSON array of two strings. The first element would be the database name, and the second would be the table name. ex `[""fixtures"", ""students""]`
  - `level=""column""`: `target` would be a JSON array of 3 strings: The database name, table name, and column name. Ex `[""fixtures"", ""students"", ""student_id""`]
- `key` would be the type of metadata entry the row has, similar to the current ""keys"" that exist in `metadata.json`. Ex `""about_url""`, `""source""`, `""description""`, etc
- `value` would be the text value of be metadata entry. The literal text value of a description, about_url, column_label, etc

A quick sample:

level | target | key | value
-- | -- | -- | --
datasette | NULL | title | my datasette title...
db | fixtures | source | <description of my database source>
table | [""fixtures"", ""students""] | label_column | student_name
column | [""fixtures"", ""students"", ""birthdate""] | description | <description of the fixtures.students.birthdate column>

This `datasette_metadata` would be configured with other tools, and hopefully not manually by end users. Datasette Core could also offer a UI for editing entries in `datasette_metadata`, to update descriptions/columns on the fly.

### Re-vamp `metadata.json` and move non-metadata config to another place

The motivation behind this is that it's awkward that `metadata.json` contains config about things that are not strictly metadata, including:

- Plugin configuration
- [Authentication/permissions](https://docs.datasette.io/en/latest/authentication.html#access-permissions-in-metadata) (ex the `allow` key on datasettes/databases/tables
- Canned queries. might be controversial, but in my mind, canned queries are application-specific code and configuration, and don't describe the data that exists in SQLite databases. 

I think we should move these outside of `metadata.json` and into a different file. The `datasette.json` idea in  #2093 may be a good solution here: plugin/permissions/canned queries can be defined in `datasette.json`, while `metadata.json`/`datasette_metadata` will strictly be about documenting databases/tables/columns. 
",107914493
issue_comments,1683164661,IC_kwDOCGYnMM5kUw31,,9599,2023-08-18T00:45:53Z,"More updated documentation:
- https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform
- https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints",140912432
issue_comments,1683145819,IC_kwDOCGYnMM5kUsRb,,9599,2023-08-18T00:17:26Z,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,140912432
issue_comments,1683145110,IC_kwDOCGYnMM5kUsGW,,9599,2023-08-18T00:16:28Z,"One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`:

https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708

I should write tests for them too.",140912432
issue_comments,1683143723,IC_kwDOCGYnMM5kUrwr,,9599,2023-08-18T00:14:52Z,"Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times:

> The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries.
> 
> `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up.
> 
> Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`.

That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.",140912432
issue_comments,1683139304,IC_kwDOCGYnMM5kUqro,,9599,2023-08-18T00:09:56Z,"Upgrading `flake8` locally replicated the error:
```
pip install -U flake8
flake8
```
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```",140912432
issue_comments,1683138953,IC_kwDOCGYnMM5kUqmJ,,9599,2023-08-18T00:09:20Z,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop:
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```",140912432
issue_comments,1683137259,IC_kwDOCGYnMM5kUqLr,,9599,2023-08-18T00:06:59Z,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints",140912432
issue_comments,1683122767,IC_kwDOCGYnMM5kUmpP,,9599,2023-08-17T23:46:09Z,"Oops, `mypy` failures:

```
sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"")  [assignment]
sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[<type>, <type>] = ..."")  [var-annotated]
sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform""  [union-attr]
sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected <nothing>  [arg-type]
sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected <nothing>  [arg-type]
Found 5 errors in 1 file (checked 56 source files)
```",140912432
issue_comments,1683122490,IC_kwDOCGYnMM5kUmk6,,22429695,2023-08-17T23:45:44Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`92.85%`** and project coverage change: **`-0.07%`** :warning:
> Comparison is base [(`1dc6b5a`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/1dc6b5aa644a92d3654f7068110ed7930989ce71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.82% compared to head [(`2915050`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.76%.
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #584      +/-   ##
==========================================
- Coverage   95.82%   95.76%   -0.07%     
==========================================
  Files           8        8              
  Lines        2829     2834       +5     
==========================================
+ Hits         2711     2714       +3     
- Misses        118      120       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <92.85%> (-0.15%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issue_comments,1683118376,IC_kwDOCGYnMM5kUlko,,9599,2023-08-17T23:41:10Z,"The problem here is that the table created by this line:
```python
fresh_db.create_table(""breeds"", {""name"": str})
```
Has this schema:
```sql
CREATE TABLE [breeds] (
   [name] TEXT
);
```
SQLite creates an invisible `rowid` column for it automatically.

On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added:
```sql
(Pdb) print(fresh_db.schema)
CREATE TABLE [dogs] (
   [name] TEXT
, [breed_id] INTEGER,
   FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid])
);
CREATE TABLE [breeds] (
   [name] TEXT
);
```
But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884

Here's what the debugger reveals about this code:
```python
        for fk in foreign_keys:
            if fk.other_table == name and columns.get(fk.other_column):
                continue
            if not any(
                c for c in self[fk.other_table].columns if c.name == fk.other_column
            ):
                raise AlterError(
                    ""No such column: {}.{}"".format(fk.other_table, fk.other_column)
                )
```
```
(Pdb) fk
ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid')
(Pdb) self[fk.other_table].columns
[Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]
```",140912432
issue_comments,1683114719,IC_kwDOCGYnMM5kUkrf,,9599,2023-08-17T23:36:02Z,"Just these three lines recreate the problem:
```python
from sqlite_utils import Database

fresh_db = Database(memory=True)

fresh_db.create_table(""dogs"", {""name"": str})
fresh_db.create_table(""breeds"", {""name"": str})
fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"")
```
Traceback:
```
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column
    self.add_foreign_key(col_name, fk, fk_col)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key
    self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys
    self[table].transform(add_foreign_keys=fks)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform
    sqls = self.transform_sql(
           ^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql
    self.db.create_table_sql(
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql
    raise AlterError(
sqlite_utils.db.AlterError: No such column: breeds.rowid
```",140912432
issue_comments,1683112857,IC_kwDOCGYnMM5kUkOZ,,9599,2023-08-17T23:33:58Z,"Full test:

https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484",140912432
issue_comments,1683112298,IC_kwDOCGYnMM5kUkFq,,9599,2023-08-17T23:33:14Z,"Just one failing test left:

```
        # Soundness check foreign_keys point to existing tables
        for fk in foreign_keys:
            if fk.other_table == name and columns.get(fk.other_column):
                continue
            if not any(
                c for c in self[fk.other_table].columns if c.name == fk.other_column
            ):
>               raise AlterError(
                    ""No such column: {}.{}"".format(fk.other_table, fk.other_column)
                )
E               sqlite_utils.db.AlterError: No such column: breeds.rowid

sqlite_utils/db.py:882: AlterError
==== short test summary info ====
FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid
==== 1 failed, 378 deselected in 0.49s ====
```",140912432
issues,1855838223,PR_kwDOCGYnMM5YM-I3,.transform() instead of modifying sqlite_master for add_foreign_keys,9599,2023-08-17T23:32:45Z,"Refs:
- #577

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--584.org.readthedocs.build/en/584/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1683110636,IC_kwDOCGYnMM5kUjrs,,9599,2023-08-17T23:31:27Z,"Spotted this while working on:
- #577 ",140912432
issues,1855836914,I_kwDOCGYnMM5undLy,Get rid of test.utils.collapse_whitespace,9599,2023-08-17T23:31:09Z,"I have a neater pattern for this now - instead of: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L472-L475

I now prefer:

https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_create.py#L1163-L1171",140912432
issue_comments,1683098094,IC_kwDOCGYnMM5kUgnu,,9599,2023-08-17T23:15:36Z,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133

Needs updating like so:
```diff
diff --git a/tests/test_extract.py b/tests/test_extract.py
index 70ad0cf..fd52534 100644
--- a/tests/test_extract.py
+++ b/tests/test_extract.py
@@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db):
     assert fresh_db[""tree""].schema == (
         'CREATE TABLE ""tree"" (\n'
         ""   [name] TEXT,\n""
-        ""   [common_name_latin_name_id] INTEGER,\n""
-        ""   FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n""
+        ""   [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n""
         "")""
     )
     assert (
```
Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in.

I don't think this should count as a breaking change release though, but it's still worth noting.",140912432
issue_comments,1683076325,IC_kwDOCGYnMM5kUbTl,,9599,2023-08-17T22:48:36Z,"I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism.

It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.",140912432
issue_comments,1683074857,IC_kwDOCGYnMM5kUa8p,,9599,2023-08-17T22:46:40Z,"As a reminder:
https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165",140912432
issue_comments,1683074546,IC_kwDOCGYnMM5kUa3y,,9599,2023-08-17T22:46:18Z,"Maybe this:
```python
        drop_foreign_keys: Optional[Iterable] = None,
```
Should be this:
```python
        drop_foreign_keys: Optional[Iterable[str]] = None,
```
Because it takes a list of column names that should have their foreign keys dropped.",140912432
issue_comments,1683074009,IC_kwDOCGYnMM5kUavZ,,9599,2023-08-17T22:45:29Z,"Actually I think `table.transform()` might get the following optional arguments:
```python
    def transform(
        self,
        *,
        # ...
        # This one exists already:
        drop_foreign_keys: Optional[Iterable] = None,
        # These two are new. This one specifies keys to add:
        add_foreign_keys: Optional[ForeignKeysType] = None,
        # Or this one causes them all to be replaced with the new definitions:
        foreign_keys: Optional[ForeignKeysType] = None,
```
There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go.

",140912432
issue_comments,1683071519,IC_kwDOCGYnMM5kUaIf,,9599,2023-08-17T22:42:28Z,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation:

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149

At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`.

Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`:

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177",140912432
issue_comments,1683068505,IC_kwDOCGYnMM5kUZZZ,,9599,2023-08-17T22:39:17Z,"This would help address these issues, among potentially many others:
- https://github.com/simonw/llm/issues/60
- https://github.com/simonw/llm/issues/116
- https://github.com/simonw/llm/issues/123",140912432
issue_comments,1683066934,IC_kwDOCGYnMM5kUZA2,,9599,2023-08-17T22:37:18Z,"I'm certain this could work.

It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter:

https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872

Improving this code to support adding foreign keys as well would be pretty simple.

And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.",140912432
issue_comments,1682272492,IC_kwDOBm6k_c5kRXDs,,22429695,2023-08-17T13:18:08Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`6b617f7`)](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2142   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          40       40           
  Lines        5937     5937           
=======================================
  Hits         5466     5466           
  Misses        471      471           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1854970601,PR_kwDOBm6k_c5YKAZ4,Bump the python-packages group with 2 updates,49699333,2023-08-17T13:07:53Z,"Bumps the python-packages group with 2 updates: [sphinx](https://github.com/sphinx-doc/sphinx) and [blacken-docs](https://github.com/asottile/blacken-docs).

Updates `sphinx` from 7.1.2 to 7.2.0
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.2.0 (released Aug 17, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11511"">#11511</a>: Drop Python 3.8 support.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11576"">#11576</a>: Require Pygments 2.14 or later.</li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11512"">#11512</a>: Deprecate <code>sphinx.util.md5</code> and <code>sphinx.util.sha1</code>.
Use <code>hashlib</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11526"">#11526</a>: Deprecate <code>sphinx.testing.path</code>.
Use <code>os.path</code> or <code>pathlib</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11528"">#11528</a>: Deprecate <code>sphinx.util.split_index_msg</code> and <code>sphinx.util.split_into</code>.
Use <code>sphinx.util.index_entries.split_index_msg</code> instead.</li>
<li>Deprecate <code>sphinx.builders.html.Stylesheet</code>
and <code>sphinx.builders.html.Javascript</code>.
Use <code>sphinx.application.Sphinx.add_css_file()</code>
and <code>sphinx.application.Sphinx.add_js_file()</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11582"">#11582</a>: Deprecate <code>sphinx.builders.html.StandaloneHTMLBuilder.css_files</code> and
<code>sphinx.builders.html.StandaloneHTMLBuilder.script_files</code>.
Use <code>sphinx.application.Sphinx.add_css_file()</code>
and <code>sphinx.application.Sphinx.add_js_file()</code> instead.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11459"">#11459</a>: Deprecate <code>sphinx.ext.autodoc.preserve_defaults.get_function_def()</code>.
Patch by Bénédikt Tran.</li>
</ul>
<h2>Features added</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11526"">#11526</a>: Support <code>os.PathLike</code> types and <code>pathlib.Path</code> objects
in many more places.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/5474"">#5474</a>: coverage: Print summary statistics tables.
Patch by Jorge Leitao.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/6319"">#6319</a>: viewcode: Add :confval:<code>viewcode_line_numbers</code> to control
whether line numbers are added to rendered source code.
Patch by Ben Krikler.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/9662"">#9662</a>: Add the <code>:no-typesetting:</code> option to suppress textual output
and only create a linkable anchor.
Patch by Latosha Maltba.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11221"">#11221</a>: C++: Support domain objects in the table of contents.
Patch by Rouslan Korneychuk.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10938"">#10938</a>: doctest: Add :confval:<code>doctest_show_successes</code> option.
Patch by Trey Hunner.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11533"">#11533</a>: Add <code>:no-index:</code>, <code>:no-index-entry:</code>, and <code>:no-contents-entry:</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11572"">#11572</a>: Improve <code>debug</code> logging of reasons why files are detected as out of
date.
Patch by Eric Larson.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/da9f8a5c33ad5eeef05dd780f2988f3ff7351ef3""><code>da9f8a5</code></a> Bump to 7.2.0 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/794324ac000095aa6409a4958b44a17dc25d3b49""><code>794324a</code></a> Declare support for Python 3.13</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/03bceac43ff1e69a33fe973612504738182cf217""><code>03bceac</code></a> [bot]: Update message catalogues (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11538"">#11538</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/aecebcea9a3f6cbd3f379bc34f99d2ffb0f87220""><code>aecebce</code></a> Make <code>_resolve_toctree()</code> keyword-only</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/76658c49a931745da45fa4d5c8d482b7c6facd4f""><code>76658c4</code></a> Fix <code>sphinx.ext.autodoc.preserve_defaults</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11550"">#11550</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4dee1629901f5e5c2aefc274da42483e16770e50""><code>4dee162</code></a> Fix layout issues in the agogo theme for smaller viewports (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11594"">#11594</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4ddbee4a7cdf14a7d064ffaea6cbb0c4fdb63f23""><code>4ddbee4</code></a> Fix <code>console_scripts</code> entry points</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4add50a5f0c3ff7954edcec3d33dd1c0fa40a1a6""><code>4add50a</code></a> Remove unneeded type annotation</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/9d8ef833dbf9d7bb04771e622d6ceea738ac9d18""><code>9d8ef83</code></a> Fix <code>test_inspect_main_noargs</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ddf8a8e7d4756170d5ee6dff7fc88ecd12912e59""><code>ddf8a8e</code></a> Add CHANGES entry for <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11533"">GH-11533</a></li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v7.1.2...v7.2.0"">compare view</a></li>
</ul>
</details>
<br />

Updates `blacken-docs` from 1.15.0 to 1.16.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.16.0 (2023-08-16)</h2>
<ul>
<li>
<p>Allow Markdown fence options.</p>
<p>Thanks to initial work from Matthew Anderson in <code>PR [#246](https://github.com/asottile/blacken-docs/issues/246) &lt;https://github.com/adamchainz/blacken-docs/pull/246&gt;</code>__.</p>
</li>
<li>
<p>Expand Markdown detection to all Python language names from Pygments: <code>py</code>, <code>sage</code>, <code>python3</code>, <code>py3</code>, and <code>numpy</code>.</p>
</li>
<li>
<p>Preserve leading whitespace lines in reStructuredText code blocks.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#217](https://github.com/asottile/blacken-docs/issues/217) &lt;https://github.com/adamchainz/blacken-docs/issues/217&gt;</code>__.</p>
</li>
<li>
<p>Use exit code 2 to indicate errors from Black, whilst exit code 1 remains for “files have been formatted”.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#218](https://github.com/asottile/blacken-docs/issues/218) &lt;https://github.com/adamchainz/blacken-docs/issues/218&gt;</code>__.</p>
</li>
<li>
<p>Support passing the <code>--preview</code> option through to Black, to select the future style.</p>
</li>
<li>
<p>Remove <code>language_version</code> from <code>.pre-commit-hooks.yaml</code>.
This change allows <code>default_language_version</code> in ``.pre-commit-config.yaml` to take precedence.</p>
<p>Thanks to Aneesh Agrawal in <code>PR [#258](https://github.com/asottile/blacken-docs/issues/258) &lt;https://github.com/adamchainz/blacken-docs/pull/258&gt;</code>__.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/960ead214cd1184149d366c6d27ca6c369ce46b6""><code>960ead2</code></a> Version 1.16.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8f0ed18ce69d35e7e631d2b01983ecf8248667a0""><code>8f0ed18</code></a> Support passing --preview through to Black (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/273"">#273</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/4eb4e4cdbff31cde3d9ffdd7693d5b7dce4dc506""><code>4eb4e4c</code></a> Tweak changelog note</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/6c7450cfa90581b482d71b451b2d007361aa47c1""><code>6c7450c</code></a> Use exit code 2 to indicate errors (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/272"">#272</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/99dfc8d0b402b9b7b0f0e723fc1d030063249eac""><code>99dfc8d</code></a> Preserve leading whitespace lines in rST (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/271"">#271</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/94465e83e7253b4caef1e00fcb30b74ced33f08e""><code>94465e8</code></a> Reformat markdown tests with dedent() (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/270"">#270</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7cd5f30f0f9e014bda566a1893df3b9c7f9a2c48""><code>7cd5f30</code></a> Use .md in glob example</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f97e56994178e079ab807cebe54c2044f7b2ae55""><code>f97e569</code></a> Document applying to many files (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/269"">#269</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/ae612b095b5005eee8c3ab9fe65a248d7654582d""><code>ae612b0</code></a> Expand Markdown detection to all Python language names (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/268"">#268</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/da9b4557afc9b820c40b8d6bdb8f515c6cdec0f5""><code>da9b455</code></a> Replace NamedTuple with plain class (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/267"">#267</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/asottile/blacken-docs/compare/1.15.0...1.16.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name> dependency` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name> dependency` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2142.org.readthedocs.build/en/2142/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1682256251,IC_kwDOBm6k_c5kRTF7,,49699333,2023-08-17T13:07:43Z,"Looks like blacken-docs is updatable in another way, so this is no longer needed.",107914493
issue_comments,1680660343,IC_kwDOBm6k_c5kLNd3,,22429695,2023-08-16T13:55:22Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`e2be94a`)](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2141   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          40       40           
  Lines        5937     5937           
=======================================
  Hits         5466     5466           
  Misses        471      471           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1853289039,PR_kwDOBm6k_c5YEUBK,Bump the python-packages group with 1 update,49699333,2023-08-16T13:47:35Z,"Bumps the python-packages group with 1 update: [blacken-docs](https://github.com/asottile/blacken-docs).

<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.16.0 (2023-08-16)</h2>
<ul>
<li>
<p>Allow Markdown fence options.</p>
<p>Thanks to initial work from Matthew Anderson in <code>PR [#246](https://github.com/asottile/blacken-docs/issues/246) &lt;https://github.com/adamchainz/blacken-docs/pull/246&gt;</code>__.</p>
</li>
<li>
<p>Expand Markdown detection to all Python language names from Pygments: <code>py</code>, <code>sage</code>, <code>python3</code>, <code>py3</code>, and <code>numpy</code>.</p>
</li>
<li>
<p>Preserve leading whitespace lines in reStructuredText code blocks.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#217](https://github.com/asottile/blacken-docs/issues/217) &lt;https://github.com/adamchainz/blacken-docs/issues/217&gt;</code>__.</p>
</li>
<li>
<p>Use exit code 2 to indicate errors from Black, whilst exit code 1 remains for “files have been formatted”.</p>
<p>Thanks to Julianus Pfeuffer for the report in <code>Issue [#218](https://github.com/asottile/blacken-docs/issues/218) &lt;https://github.com/adamchainz/blacken-docs/issues/218&gt;</code>__.</p>
</li>
<li>
<p>Support passing the <code>--preview</code> option through to Black, to select the future style.</p>
</li>
<li>
<p>Remove <code>language_version</code> from <code>.pre-commit-hooks.yaml</code>.
This change allows <code>default_language_version</code> in ``.pre-commit-config.yaml` to take precedence.</p>
<p>Thanks to Aneesh Agrawal in <code>PR [#258](https://github.com/asottile/blacken-docs/issues/258) &lt;https://github.com/adamchainz/blacken-docs/pull/258&gt;</code>__.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/960ead214cd1184149d366c6d27ca6c369ce46b6""><code>960ead2</code></a> Version 1.16.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8f0ed18ce69d35e7e631d2b01983ecf8248667a0""><code>8f0ed18</code></a> Support passing --preview through to Black (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/273"">#273</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/4eb4e4cdbff31cde3d9ffdd7693d5b7dce4dc506""><code>4eb4e4c</code></a> Tweak changelog note</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/6c7450cfa90581b482d71b451b2d007361aa47c1""><code>6c7450c</code></a> Use exit code 2 to indicate errors (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/272"">#272</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/99dfc8d0b402b9b7b0f0e723fc1d030063249eac""><code>99dfc8d</code></a> Preserve leading whitespace lines in rST (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/271"">#271</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/94465e83e7253b4caef1e00fcb30b74ced33f08e""><code>94465e8</code></a> Reformat markdown tests with dedent() (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/270"">#270</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7cd5f30f0f9e014bda566a1893df3b9c7f9a2c48""><code>7cd5f30</code></a> Use .md in glob example</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f97e56994178e079ab807cebe54c2044f7b2ae55""><code>f97e569</code></a> Document applying to many files (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/269"">#269</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/ae612b095b5005eee8c3ab9fe65a248d7654582d""><code>ae612b0</code></a> Expand Markdown detection to all Python language names (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/268"">#268</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/da9b4557afc9b820c40b8d6bdb8f515c6cdec0f5""><code>da9b455</code></a> Replace NamedTuple with plain class (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/267"">#267</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/asottile/blacken-docs/compare/1.15.0...1.16.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=blacken-docs&package-manager=pip&previous-version=1.15.0&new-version=1.16.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name> dependency` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name> dependency` will remove all of the ignore conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2141.org.readthedocs.build/en/2141/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1675157445,IC_kwDOBm6k_c5j2N_F,,9599,2023-08-11T17:45:54Z,"Affected pages:

- https://docs.datasette.io/en/latest/authentication.html
- https://docs.datasette.io/en/latest/changelog.html
- https://docs.datasette.io/en/latest/cli-reference.html
- https://docs.datasette.io/en/latest/contributing.html
- https://docs.datasette.io/en/latest/custom_templates.html
- https://docs.datasette.io/en/latest/deploying.html
- https://docs.datasette.io/en/latest/facets.html
- https://docs.datasette.io/en/latest/full_text_search.html
- https://docs.datasette.io/en/latest/installation.html
- https://docs.datasette.io/en/latest/plugin_hooks.html
- https://docs.datasette.io/en/latest/publish.html
- https://docs.datasette.io/en/latest/settings.html
- https://docs.datasette.io/en/latest/spatialite.html
- https://docs.datasette.io/en/latest/sql_queries.html

e.g. on https://docs.datasette.io/en/latest/authentication.html#using-the-root-actor

<img width=""759"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/aa3ea5bd-1b4b-4ecf-a5ed-a62508dcde31"">
",107914493
issues,1847201263,I_kwDOBm6k_c5uGg3v,Remove all remaining documentation instances of '$ ',9599,2023-08-11T17:42:13Z,"For example this: https://github.com/simonw/datasette/blob/4535568f2ce907af646304d0ebce2500ebd55677/docs/authentication.rst?plain=1#L33-L35

The problem with that `$ ` prefix is that it prevents users from copying and pasting the raw command.

<img width=""773"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/9414437d-1eb9-416b-ae51-0f4f5d665b80"">

https://docs.datasette.io/en/stable/authentication.html#using-the-root-actor",107914493
issue_comments,1674242356,IC_kwDOBm6k_c5jyuk0,,36199671,2023-08-11T05:52:29Z,"I see :) yeah, I’m on the stable version installed from homebrew on macOS",107914493
issue_comments,1674227353,IC_kwDOBm6k_c5jyq6Z,,9599,2023-08-11T05:25:45Z,"Demo:

- https://latest.datasette.io/
- https://latest.datasette.io/fixtures
- https://latest.datasette.io/fixtures/facetable
- https://latest.datasette.io/fixtures/pragma_cache_size
- https://latest.datasette.io/fixtures/attraction_characteristic/2",107914493
issue_comments,1674223722,IC_kwDOBm6k_c5jyqBq,,9599,2023-08-11T05:19:42Z,"I decided not to implement this yet, because making `database.color` dependent on metadata interferes with my goal to eventually make metadata lookups `async` in order to serve them better from plugins.

I'll leave this issue open though as it may get solved in future redesigns of the metadata system, refs:

- #1384",107914493
issue_comments,1674211804,IC_kwDOBm6k_c5jynHc,,9599,2023-08-11T04:56:28Z,"That `database.color` is only available in `index.html` at the moment thanks to this:

https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/views/index.py#L108-L110",107914493
issue_comments,1674210215,IC_kwDOBm6k_c5jymun,,9599,2023-08-11T04:53:11Z,"Relevant template snippet:

https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13",107914493
issue_comments,1674210009,IC_kwDOBm6k_c5jymrZ,,9599,2023-08-11T04:52:44Z,"I'll fix these too:
- https://github.com/simonw/datasette/issues/2119
- https://github.com/simonw/datasette/issues/568",107914493
issue_comments,1674115279,IC_kwDOBm6k_c5jyPjP,,9599,2023-08-11T01:27:30Z,"https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/database.html#L13

Looks like I should be doing this instead:

https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13",107914493
issues,1846076261,I_kwDOBm6k_c5uCONl,border-color: ##ff0000 bug - two hashes,9599,2023-08-11T01:22:58Z,"Spotted this on https://latest.datasette.io/extra_database

```html
<div class=""page-header"" style=""border-color: ##ff0000"">
```",107914493
issue_comments,1672404276,IC_kwDOBm6k_c5jrt00,,9599,2023-08-10T01:30:25Z,"Here's the prototype of adding documentation for these using Cog: https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875

![](https://user-images.githubusercontent.com/9599/249324964-79210a43-716c-4470-a62d-01aa871973ec.png)",107914493
issue_comments,1672385674,IC_kwDOBm6k_c5jrpSK,,9599,2023-08-10T01:07:43Z,"What version of Datasette are you running?

That feature was added in Datasette 1.0a2, so if you're on the current stable release you won't have it yet.",107914493
issue_comments,1672384439,IC_kwDOBm6k_c5jro-3,,9599,2023-08-10T01:05:56Z,"Now live at https://latest.datasette.io/counters

![increment](https://github.com/simonw/datasette/assets/9599/ffb3625b-b61b-4189-80e3-7b4e8b411f14)
",107914493
issue_comments,1672366410,IC_kwDOBm6k_c5jrklK,,9599,2023-08-10T00:39:16Z,"Here's what it looks like when it works:

<img width=""715"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/a53efc81-8ceb-44a2-b083-2efb27d20cf7"">

And here's what you get if there's a SQL error in the `on_success_message_sql` query itself:

<img width=""715"" alt=""CleanShot 2023-08-09 at 17 38 12@2x"" src=""https://github.com/simonw/datasette/assets/9599/e211cee2-3edb-475a-906d-b0bb5a118d77"">
",107914493
issue_comments,1672360472,IC_kwDOBm6k_c5jrjIY,,54462,2023-08-10T00:31:24Z,"It looks very nice now.
Finally, no more manual installation of plugins one by one. Thank you, Simon!  ❤️ 
",107914493
issue_comments,1672353428,IC_kwDOBm6k_c5jrhaU,,9599,2023-08-10T00:20:56Z,"Goal here is to customize the message displayed after a query has succeeded. e.g.

```yaml
databases:
  mydatabase:
    queries:
      add_name:
        sql: INSERT INTO names (name) VALUES (:name)
        write: true
        on_success_message_sql: 'select ''Name inserted: '' || :name'
        on_success_redirect: /mydatabase/names
```",107914493
issues,1844213115,I_kwDOBm6k_c5t7HV7,on_success_message_sql option for writable canned queries,9599,2023-08-10T00:20:14Z,"> Or... how about if the `on_success_message` option could define a SQL query to be executed to generate that message? Maybe `on_success_message_sql`.

- https://github.com/simonw/datasette/issues/2134",107914493
issue_comments,1672340556,IC_kwDOBm6k_c5jreRM,,9599,2023-08-10T00:07:17Z,Most of this is shipped in https://docs.datasette.io/en/1.0a3/changelog.html#a3-2023-08-09,107914493
issue_comments,1672224611,IC_kwDOBm6k_c5jrB9j,,9599,2023-08-09T22:07:43Z,Documentation: https://docs.datasette.io/en/latest/plugins.html#seeing-what-plugins-are-installed,107914493
issue_comments,1672076671,IC_kwDOBm6k_c5jqd1_,,9599,2023-08-09T20:09:31Z,I'm going to have it only return plugins with a version other than `None`.,107914493
issue_comments,1672076284,IC_kwDOBm6k_c5jqdv8,,9599,2023-08-09T20:09:12Z,"Problem - if there are plugins in the plugins directory it was returning this:
```
messages_output_renderer.py==None
my_plugin.py==None
my_plugin_2.py==None
register_output_renderer.py==None
sleep_sql_function.py==None
view_name.py==None
```",107914493
issue_comments,1671983228,IC_kwDOBm6k_c5jqHB8,,9599,2023-08-09T19:02:47Z,"As part of this I'd like to refactor the JSON renderer code - I started attempting that here: https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2

In particular I want to get rid of that `data` argument.",107914493
issue_comments,1671982215,IC_kwDOBm6k_c5jqGyH,,9599,2023-08-09T19:02:19Z,"I'm going with a bit of a hacky workaround for the moment, until I get a chance to refactor this:
- #2137 ",107914493
issue_comments,1671964003,IC_kwDOBm6k_c5jqCVj,,9599,2023-08-09T18:53:10Z,"I made an initial attempt at this in a branch, but it broke too much other stuff.  https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2",107914493
issues,1843821954,I_kwDOBm6k_c5t5n2C,Redesign row default JSON,9599,2023-08-09T18:49:11Z,"This URL here:

https://latest.datasette.io/fixtures/simple_primary_key/1.json?_extras=foreign_key_tables

```json
{
  ""database"": ""fixtures"",
  ""table"": ""simple_primary_key"",
  ""rows"": [
    {
      ""id"": ""1"",
      ""content"": ""hello""
    }
  ],
  ""columns"": [
    ""id"",
    ""content""
  ],
  ""primary_keys"": [
    ""id""
  ],
  ""primary_key_values"": [
    ""1""
  ],
  ""units"": {},
  ""foreign_key_tables"": [
    {
      ""other_table"": ""foreign_key_references"",
      ""column"": ""id"",
      ""other_column"": ""foreign_key_with_blank_label"",
      ""count"": 0,
      ""link"": ""/fixtures/foreign_key_references?foreign_key_with_blank_label=1""
    },
    {
      ""other_table"": ""foreign_key_references"",
      ""column"": ""id"",
      ""other_column"": ""foreign_key_with_label"",
      ""count"": 1,
      ""link"": ""/fixtures/foreign_key_references?foreign_key_with_label=1""
    },
    {
      ""other_table"": ""complex_foreign_keys"",
      ""column"": ""id"",
      ""other_column"": ""f3"",
      ""count"": 1,
      ""link"": ""/fixtures/complex_foreign_keys?f3=1""
    },
    {
      ""other_table"": ""complex_foreign_keys"",
      ""column"": ""id"",
      ""other_column"": ""f2"",
      ""count"": 0,
      ""link"": ""/fixtures/complex_foreign_keys?f2=1""
    },
    {
      ""other_table"": ""complex_foreign_keys"",
      ""column"": ""id"",
      ""other_column"": ""f1"",
      ""count"": 1,
      ""link"": ""/fixtures/complex_foreign_keys?f1=1""
    }
  ],
  ""query_ms"": 4.226590999678592,
  ""source"": ""tests/fixtures.py"",
  ""source_url"": ""https://github.com/simonw/datasette/blob/main/tests/fixtures.py"",
  ""license"": ""Apache License 2.0"",
  ""license_url"": ""https://github.com/simonw/datasette/blob/main/LICENSE"",
  ""ok"": true,
  ""truncated"": false
}
```

That `?_extras=` should be `?_extra=` - plus the row JSON should be redesigned to fit the new default JSON representation.",107914493
issue_comments,1671862711,IC_kwDOBm6k_c5jppm3,,9599,2023-08-09T17:35:22Z,"Need `?_extra=columns` too since I'm removing `""columns""` from the default query JSON in https://github.com/simonw/datasette/issues/2136",107914493
issue_comments,1671861623,IC_kwDOBm6k_c5jppV3,,9599,2023-08-09T17:34:35Z,"I'll make `?_extra=columns` work in:
- #2113",107914493
issue_comments,1671848664,IC_kwDOBm6k_c5jpmLY,,9599,2023-08-09T17:24:37Z,"It also bothers me that `ok` comes first in one but not first in the other - I'd like the order of keys in the JSON to be consistent, even though order isn't meant to mean anything.",107914493
issues,1843710170,I_kwDOBm6k_c5t5Mja,Query view shouldn't return `columns`,9599,2023-08-09T17:23:57Z,"I just noticed that https://latest.datasette.io/fixtures/roadside_attraction_characteristics.json?_labels=on&_size=1 returns:
```json
{
  ""ok"": true,
  ""next"": ""1"",
  ""rows"": [
    {
      ""rowid"": 1,
      ""attraction_id"": {
        ""value"": 1,
        ""label"": ""The Mystery Spot""
      },
      ""characteristic_id"": {
        ""value"": 2,
        ""label"": ""Paranormal""
      }
    }
  ],
  ""truncated"": false
}
```
But https://latest.datasette.io/fixtures.json?sql=select+rowid%2C+attraction_id%2C+characteristic_id+from+roadside_attraction_characteristics+order+by+rowid+limit+1 returns:
```json
{
  ""rows"": [
    {
      ""rowid"": 1,
      ""attraction_id"": 1,
      ""characteristic_id"": 2
    }
  ],
  ""columns"": [
    ""rowid"",
    ""attraction_id"",
    ""characteristic_id""
  ],
  ""ok"": true,
  ""truncated"": false
}
```
The `columns` key in the query response is inconsistent with the table response.",107914493
issue_comments,1671844664,IC_kwDOBm6k_c5jplM4,,9599,2023-08-09T17:21:22Z,"OK, I've done the small changes - still need to document the big JSON changes though.

Those aren't covered by the regular documentation either so I need to update that.",107914493
issue_comments,1671836347,IC_kwDOBm6k_c5jpjK7,,9599,2023-08-09T17:15:03Z,Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette-request,107914493
issue_comments,1671753753,IC_kwDOBm6k_c5jpPAZ,,9599,2023-08-09T16:26:17Z,"- Dropped support for Python 3.7
- New `Justfile`
- #1970
- `datasette.render_template()` now accepts a `Context` subclass as an alternative to a dictionary
- #2106
- #2007
- https://github.com/simonw/datasette/issues/2130
- https://github.com/simonw/datasette/issues/2079
- https://github.com/simonw/datasette/issues/1153",107914493
issue_comments,1671750913,IC_kwDOBm6k_c5jpOUB,,9599,2023-08-09T16:24:26Z,"Remember not to include changes from this:

https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/docs/changelog.rst?plain=1#L9-L31
",107914493
issues,1843600087,I_kwDOBm6k_c5t4xrX,Release notes for 1.0a3,9599,2023-08-09T16:09:26Z,118 commits! https://github.com/simonw/datasette/compare/1.0a2...26be9f0445b753fb84c802c356b0791a72269f25,107914493
issue_comments,1671725144,IC_kwDOBm6k_c5jpIBY,,9599,2023-08-09T16:07:58Z,Punting this out of the 1.0a3 alpha for the moment.,107914493
issue_comments,1671722482,IC_kwDOBm6k_c5jpHXy,,9599,2023-08-09T16:06:28Z,"Plugin configuration itself currently lives in `metadata.json` which I don't like - that file was supposed to be for metadata about your databases, it's a bit of an accident that it grew configuration options as well.

I'm hoping to fix that for Datasette 1.0, but I'm still figuring out how to do that.",107914493
issue_comments,1671720761,IC_kwDOBm6k_c5jpG85,,9599,2023-08-09T16:05:33Z,"Honestly the `--all` option isn't particularly useful - it's really more of a debugging tool for myself than something that I expect anyone else to use.

The purpose of `datasette plugins` generally is to help people confirm that plugins installed correctly. It's effectively the CLI alternative to hitting https://datasette.io/-/plugins",107914493
issue_comments,1671649530,IC_kwDOBm6k_c5jo1j6,,54462,2023-08-09T15:41:14Z,"Yes,  using this approach(`datasette install -r requirements.txt`) will result in more consistency. 

I'm curious about the results of the `datasette plugins --all` command. Where will we use the output of this command? Will it include configuration information for these plugins in the future? If so, will we need to consider the configuration of these plugins in addition to installing them on different computers?",107914493
issue_comments,1671634654,IC_kwDOBm6k_c5jox7e,,9599,2023-08-09T15:32:14Z,"```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 32266888..370b722c 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -223,15 +223,22 @@ pm.hook.publish_subcommand(publish=publish)
 
 @cli.command()
 @click.option(""--all"", help=""Include built-in default plugins"", is_flag=True)
+@click.option(
+    ""--requirements"", help=""Output requirements.txt of installed plugins"", is_flag=True
+)
 @click.option(
     ""--plugins-dir"",
     type=click.Path(exists=True, file_okay=False, dir_okay=True),
     help=""Path to directory containing custom plugins"",
 )
-def plugins(all, plugins_dir):
+def plugins(all, requirements, plugins_dir):
     """"""List currently installed plugins""""""
     app = Datasette([], plugins_dir=plugins_dir)
-    click.echo(json.dumps(app._plugins(all=all), indent=4))
+    if requirements:
+        for plugin in app._plugins():
+            click.echo(""{}=={}"".format(plugin[""name""], plugin[""version""]))
+    else:
+        click.echo(json.dumps(app._plugins(all=all), indent=4))
 
 
 @cli.command()
```
Output:
```
% datasette plugins --requirements  
datasette-codespaces==0.1.1
datasette-graphql==2.2
datasette-json-html==1.0.1
datasette-pretty-json==0.2.2
datasette-x-forwarded-host==0.1
```",107914493
issue_comments,1671628602,IC_kwDOBm6k_c5jowc6,,9599,2023-08-09T15:28:42Z,"Neat idea! We already have this command:

    datasette install -r requirements.txt

So maybe `datasetet plugins --requirements` could generate a `requirements.txt` file which could be used there.",107914493
issue_comments,1671626876,IC_kwDOBm6k_c5jowB8,,9599,2023-08-09T15:27:40Z,Another place that needs fixing: https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/datasette/views/database.py#L244-L249,107914493
issue_comments,1671581610,IC_kwDOBm6k_c5jok-q,,9599,2023-08-09T15:01:14Z,Or... how about if the `on_success_message` option could define a SQL query to be executed to generate that message? Maybe `on_success_message_sql`.,107914493
issue_comments,1671526372,IC_kwDOBm6k_c5joXfk,,9599,2023-08-09T14:39:54Z,"Alternatively, what about if there was a custom SQL function available during canned write queries for setting the output message? Then the fact that canned queries don't return a table view wouldn't be a problem. Bit weird though.",107914493
issue_comments,1671503163,IC_kwDOBm6k_c5joR07,,9599,2023-08-09T14:32:30Z,"Annoying thing about this plugin is that you don't see the new counter value when you submit the increment or decrement query.

Maybe canned queries should support SQL multiple statements? Could return the result of the last one.",107914493
issue_comments,1671500965,IC_kwDOBm6k_c5joRSl,,9599,2023-08-09T14:31:43Z,"Here's a simple plugin that can do this:
```python
from datasette import hookimpl

@hookimpl
def startup(datasette):
    db = datasette.add_memory_database(""counters"")

    async def inner():
        await db.execute_write(""create table if not exists counters (name text primary key, value integer)"")
        await db.execute_write(""insert or ignore into counters (name, value) values ('counter', 0)"")

    return inner


@hookimpl
def canned_queries(database):
    if database == ""counters"":
        return {
            ""increment"": {
                ""sql"": ""update counters set value = value + 1 where name = 'counter'"",
                ""write"": True,
            },
            ""decrement"": {
                ""sql"": ""update counters set value = value - 1 where name = 'counter'"",
                ""write"": True,
            },
        }
```",107914493
issues,1843391585,I_kwDOBm6k_c5t3-xh,Add writable canned query demo to latest.datasette.io,9599,2023-08-09T14:31:30Z,"This would be useful while working on:
- #2114",107914493
issue_comments,1670470516,IC_kwDOBm6k_c5jkVt0,,9599,2023-08-09T00:09:55Z,"I decided to figure out where the `_size=` parameter here comes from: https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/database.py#L576-L588

It took some spelunking, but it looks like I added that five years ago as part of CSV streaming in https://github.com/simonw/datasette/commit/fc3660cfad7668dbce6ead12766e048fc1f78b11

It's there purely to allow CSV streaming to set `_size=""max""`.

And since CSV streaming is still just a table thing, not a query thing, I can drop it.",107914493
issue_comments,1670454222,IC_kwDOBm6k_c5jkRvO,,9599,2023-08-08T23:44:42Z,"The tests for this are still passing:

```bash
pytest tests/test_canned_queries.py
```
Because I left in the old code:

https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L752-L767

https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L701-L727",107914493
issue_comments,1669877769,IC_kwDOD079W85jiFAJ,,22996,2023-08-08T15:52:52Z,"You can also install this with pip using this oneliner:

```
pip install git+https://github.com/RhetTbull/dogsheep-photos.git@update_for_bigsur
```",256834907
issues,1841501975,I_kwDOBm6k_c5twxcX,[feature request]`datasette install plugins.json` options,54462,2023-08-08T15:06:50Z,"Hi, simon ❤️ 

`datasette plugins  --all > plugins.json` could generate all plugins info. On another machine, it would be great to install all plugins just by `datasette install plugins.json`",107914493
issue_comments,1669640626,IC_kwDOBm6k_c5jhLGy,,9599,2023-08-08T13:39:54Z,https://github.com/simonw/datasette/blob/1377a290cd85ba8d3338b1da47c4665ed4c6c625/datasette/views/database.py#L509-L517,107914493
issues,1841343173,I_kwDOBm6k_c5twKrF,Get form fields on query page working again ,9599,2023-08-08T13:39:05Z,"Caused by:
- #2112

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%22pk1%22+%3D+%3Ap0+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=b

<img width=""1100"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/7ca20120-923b-4b7d-988d-cdb0e7c30afb"">

The `:p0` form field is missing. Submitting the form results in this error:

<img width=""687"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/f900ee16-71fd-42aa-86d7-0058af8457aa"">
",107914493
issue_comments,1668983779,IC_kwDOBm6k_c5jeqvj,,9599,2023-08-08T06:27:42Z,"I've now shipped the new query JSON without any extras: https://latest.datasette.io/fixtures.json?sql=select+*+from+facetable

There aren't actually that many extras that make sense for the query page. These are the most obvious:

- `query` - adding a `{""sql"": ""..."", ""params"": ""..""}` block, as seen on https://latest.datasette.io/fixtures/facetable.json?_extra=query&_size=0
- `duration_ms` to show the duration - which isn't an extra on table right now but should be.
- `metadata` - https://latest.datasette.io/fixtures/facetable.json?_extra=metadata&_size=0",107914493
issue_comments,1668799233,IC_kwDOBm6k_c5jd9sB,,9599,2023-08-08T02:02:07Z,I just need to pass `view_name` to `datasette.render_template()`.,107914493
issue_comments,1668798642,IC_kwDOBm6k_c5jd9iy,,9599,2023-08-08T02:01:01Z,Shipped: https://latest.datasette.io/fixtures?sql=select+*+from+facetable,107914493
issue_comments,1668798261,IC_kwDOBm6k_c5jd9c1,,9599,2023-08-08T02:00:22Z,"Shipped: https://latest.datasette.io/_memory.json?sql=select+sqlite_version()
```json
{
  ""rows"": [
    {
      ""sqlite_version()"": ""3.34.1""
    }
  ],
  ""columns"": [
    ""sqlite_version()""
  ],
  ""ok"": true,
  ""truncated"": false
}
```",107914493
issue_comments,1668793100,IC_kwDOBm6k_c5jd8MM,,9599,2023-08-08T01:51:54Z,Updated documentation is here: https://docs.datasette.io/en/latest/plugin_hooks.html#register-output-renderer-datasette,107914493
issue_comments,1668792576,IC_kwDOBm6k_c5jd8EA,,9599,2023-08-08T01:51:14Z,"OK, I've landed the first example of this now. I'm cautiously optimistic about it, but I think I need to use it in a few more places AND add the actual code that uses it for documentation before I'm completely happy and close this issue.",107914493
issue_comments,1668789134,IC_kwDOBm6k_c5jd7OO,,9599,2023-08-08T01:46:12Z,"I'm going to squash merge this even though it's a bit big, because I want a clean passing commit in the `main` branch.",107914493
issue_comments,1668786770,IC_kwDOBm6k_c5jd6pS,,9599,2023-08-08T01:42:55Z,"> UPDATE: Fixed the `_size=max` bit, but I've not replicated the behaviour where it adds `?_labels=on` if there are expandable columns yet.

It looks like that behaviour is only relevant to table views, and it's already implemented - https://latest.datasette.io/fixtures/roadside_attraction_characteristics links to https://latest.datasette.io/fixtures/roadside_attraction_characteristics.csv?_labels=on&_size=max",107914493
issue_comments,1668781558,IC_kwDOBm6k_c5jd5X2,,9599,2023-08-08T01:34:42Z,"3 failures left:
```
=================================================================== short test summary info ====================================================================
FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href=""/fixtures.csv?sql=select+1&amp;_size=max"">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link r...
FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - KeyError: 'Access-Control-Allow-Origin'
FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
=============================================== 3 failed, 1309 passed, 2 skipped, 1 xfailed in 61.34s (0:01:01) ================================================
```",107914493
issues,1840417903,I_kwDOBm6k_c5tsoxv,Refactor code that supports templates_considered comment,9599,2023-08-08T01:28:36Z,"I ended up duplicating it here: https://github.com/simonw/datasette/blob/7532feb424b1dce614351e21b2265c04f9669fe2/datasette/views/database.py#L164-L167

I think it should move to `datasette.render_template()` - and maybe have a renamed template variable too.",107914493
issue_comments,1668720671,IC_kwDOBm6k_c5jdqgf,,9599,2023-08-07T23:58:21Z,"11 tests left:
```
FAILED tests/test_html.py::test_alternate_url_json[/fixtures-http://localhost/fixtures.json] - KeyError: 'link'
FAILED tests/test_html.py::test_alternate_url_json[/fixtures?sql=select+*+from+facetable-http://localhost/fixtures.json?sql=select+*+from+facetable] - assert '<link rel=""alternate"" type=""application/json+datasette"" href=""http://localhost/fixtures.json?sql=select+*+from+facetable"">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select * from...
FAILED tests/test_html.py::test_query_page_truncates - assert equals failed
FAILED tests/test_html.py::test_templates_considered[/fixtures-database-fixtures.html, *database.html] - assert '<!-- Templates considered: database-fixtures.html, *database.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d599......
FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href=""/fixtures.csv?sql=select+1&amp;_size=max"">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link rel=""stylesheet"" href=""/-/static/ap...se""}, ""view...
FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures-expected_decoded_object1] - AssertionError: assert equals failed
FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - KeyError: 'Access-Control-Allow-Origin'
FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures-expected_extra_body_script1] - AssertionError: assert equals failed
FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
FAILED tests/test_plugins.py::test_view_names[/fixtures-database] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_view_names[/fixtures?sql=select+1-database] - AssertionError: assert equals failed
======================================================================= 11 failed, 1301 passed, 2 skipped, 1 xfailed in 64.10s (0:01:04) =======================================================================
```",107914493
issue_comments,1668715803,IC_kwDOBm6k_c5jdpUb,,9599,2023-08-07T23:50:26Z,"This bit needs some more work: https://github.com/simonw/datasette/blob/a791115e31e64e9bbd1d36f4d46887b3e42e752a/datasette/views/database.py#L514-L518

The original logic for that is from here, I need to figure out how to replicate it:

https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L368

https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L342-L344

UPDATE: Fixed the `_size=max` bit, but I've not replicated the behaviour where it adds `?_labels=on` if there are expandable columns yet.",107914493
issue_comments,1668695958,IC_kwDOBm6k_c5jdkeW,,9599,2023-08-07T23:21:07Z,"Main place the code needs updating is here: https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L301-L310

Plus the documentation.",107914493
issues,1840329615,I_kwDOBm6k_c5tsTOP,Render plugin mechanism needs `error` and `truncated` fields,9599,2023-08-07T23:19:19Z,"While working on:
- https://github.com/simonw/datasette/pull/2118

It became clear that the `render` callback function documented here: https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-output-renderer-datasette

Needs to grow the ability to be told if an error occurred (an `error` string) and if the results were truncated (a `truncated` boolean).",107914493
issue_comments,1668691898,IC_kwDOBm6k_c5jdje6,,9599,2023-08-07T23:15:14Z,"New error design:
```json
{""ok"": false, ""error"": ""Error message here"", ""rows"": [], ""columns"": [], ""truncated"": false}
```
I guess that means I need to add `""ok"": true` to the default format as well - which makes sense because table JSON has that already:

https://latest.datasette.io/fixtures/facetable.json

```json
{
  ""ok"": true,
  ""next"": null,
  ""rows"": [...]
}
```",107914493
issue_comments,1668690232,IC_kwDOBm6k_c5jdjE4,,9599,2023-08-07T23:13:24Z,This may end up being a renderer plugin hook design question too.,107914493
issues,1840324765,I_kwDOBm6k_c5tsSCd,CSV ?sql= should indicate errors,9599,2023-08-07T23:13:04Z,"> https://latest.datasette.io/_memory.csv?sql=select+blah is a blank page right now:

```bash
curl -I 'https://latest.datasette.io/_memory.csv?sql=select+blah'
```
```
HTTP/2 200 
access-control-allow-origin: *
access-control-allow-headers: Authorization, Content-Type
access-control-expose-headers: Link
access-control-allow-methods: GET, POST, HEAD, OPTIONS
access-control-max-age: 3600
content-type: text/plain; charset=utf-8
x-databases: _memory, _internal, fixtures, fixtures2, extra_database, ephemeral
date: Mon, 07 Aug 2023 23:12:15 GMT
server: Google Frontend
```

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/2118#issuecomment-1668688947_",107914493
issue_comments,1668688947,IC_kwDOBm6k_c5jdiwz,,9599,2023-08-07T23:12:37Z,"https://latest.datasette.io/_memory.csv?sql=select+blah is a blank page right now:

```bash
curl -I 'https://latest.datasette.io/_memory.csv?sql=select+blah'
```
```
HTTP/2 200 
access-control-allow-origin: *
access-control-allow-headers: Authorization, Content-Type
access-control-expose-headers: Link
access-control-allow-methods: GET, POST, HEAD, OPTIONS
access-control-max-age: 3600
content-type: text/plain; charset=utf-8
x-databases: _memory, _internal, fixtures, fixtures2, extra_database, ephemeral
date: Mon, 07 Aug 2023 23:12:15 GMT
server: Google Frontend
```",107914493
issue_comments,1668687891,IC_kwDOBm6k_c5jdigT,,9599,2023-08-07T23:11:04Z,"One of the failing tests is because this URL on my local environment returns this:

http://127.0.0.1:8001/_memory.json?sql=select+blah
```json
{""rows"": [], ""columns"": [], ""truncated"": false}
```
With a 200 status code. It should be an error.

https://latest.datasette.io/_memory.json?sql=select+blah returns a 400 status code with the following JSON:

```json
{
  ""ok"": false,
  ""database"": ""_memory"",
  ""query_name"": null,
  ""rows"": [],
  ""truncated"": false,
  ""columns"": [],
  ""query"": {
    ""sql"": ""select blah"",
    ""params"": {}
  },
  ""error"": ""no such column: blah"",
  ""private"": false,
  ""allow_execute_sql"": true,
  ""query_ms"": 2.68310698447749,
  ""source"": ""tests/fixtures.py"",
  ""source_url"": ""https://github.com/simonw/datasette/blob/main/tests/fixtures.py"",
  ""license"": ""Apache License 2.0"",
  ""license_url"": ""https://github.com/simonw/datasette/blob/main/LICENSE""
}
```
So I need a design for what this should look like for the error case.

I should consider what errors look like for alternative formats too.",107914493
issue_comments,1668578107,IC_kwDOBm6k_c5jdHs7,,9599,2023-08-07T21:06:50Z,"15:
```
=========================================================================================== short test summary info ============================================================================================
FAILED tests/test_cli.py::test_sql_errors_logged_to_stderr - assert 0 == 1
FAILED tests/test_html.py::test_show_hide_sql_query - AssertionError: assert False
FAILED tests/test_canned_queries.py::test_magic_parameters_cannot_be_used_in_arbitrary_queries - assert 400 == 200
FAILED tests/test_html.py::test_templates_considered[/fixtures-database-fixtures.html, *database.html] - assert '<!-- Templates considered: database-fixtures.html, *database.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d599......
FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href=""/fixtures.csv?sql=select+1&amp;_size=max"">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link rel=""stylesheet"" href=""/-/static/ap...se""}, ""view...
FAILED tests/test_html.py::test_base_url_config[True-/fixtures?sql=select+1] - AssertionError: {
FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - assert 200 == 400
FAILED tests/test_html.py::test_alternate_url_json[/fixtures?sql=select+*+from+facetable-http://localhost/fixtures.json?sql=select+*+from+facetable] - assert '<link rel=""alternate"" type=""application/json+datasette"" href=""http://localhost/fixtures.json?sql=select+*+from+facetable"">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select * from...
FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures-expected_decoded_object1] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_view_names[/fixtures-database] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_view_names[/fixtures?sql=select+1-database] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures-expected_extra_body_script1] - AssertionError: assert equals failed
FAILED tests/test_html.py::test_base_url_config[False-/fixtures?sql=select+1] - AssertionError: {
FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
FAILED tests/test_html.py::test_alternate_url_json[/fixtures-http://localhost/fixtures.json] - KeyError: 'link'
============================================================================ 15 failed, 1297 passed, 2 skipped, 1 xfailed in 58.15s ============================================================================
```",107914493
issue_comments,1668568445,IC_kwDOBm6k_c5jdFV9,,9599,2023-08-07T20:57:46Z,17 failing tests now.,107914493
issue_comments,1668187546,IC_kwDOBm6k_c5jboWa,,49699333,2023-08-07T16:20:26Z,"Looks like sphinx is up-to-date now, so this is no longer needed.",107914493
issue_comments,1668186872,IC_kwDOBm6k_c5jboL4,,49699333,2023-08-07T16:20:19Z,"Looks like furo is up-to-date now, so this is no longer needed.",107914493
issue_comments,1668186815,IC_kwDOBm6k_c5jboK_,,49699333,2023-08-07T16:20:18Z,"Looks like blacken-docs is up-to-date now, so this is no longer needed.",107914493
issue_comments,1668143992,IC_kwDOBm6k_c5jbdt4,,22429695,2023-08-07T15:58:08Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`adf54f5`)](https://app.codecov.io/gh/simonw/datasette/commit/adf54f5c808c16e9ad812c3a4376f35cf829a93c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`935d336`)](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2128   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1839766197,PR_kwDOBm6k_c5XWhWF,"Bump blacken-docs, furo, blacken-docs",49699333,2023-08-07T15:50:40Z,"Bumps the python-packages group with 3 updates: [sphinx](https://github.com/sphinx-doc/sphinx), [furo](https://github.com/pradyunsg/furo) and [blacken-docs](https://github.com/asottile/blacken-docs).

Updates `sphinx` from 6.1.3 to 7.1.2
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.1.2</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.1.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.1.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0rc1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.1.2 (released Aug 02, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11542"">#11542</a>: linkcheck: Properly respect :confval:<code>linkcheck_anchors</code>
and do not spuriously report failures to validate anchors.
Patch by James Addison.</li>
</ul>
<h1>Release 7.1.1 (released Jul 27, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11514"">#11514</a>: Fix <code>SOURCE_DATE_EPOCH</code> in multi-line copyright footer.
Patch by Bénédikt Tran.</li>
</ul>
<h1>Release 7.1.0 (released Jul 24, 2023)</h1>
<h2>Incompatible changes</h2>
<ul>
<li>
<p>Releases are no longer signed, given the <code>change in PyPI policy</code>_.</p>
<p>.. _change in PyPI policy: <a href=""https://blog.pypi.org/posts/2023-05-23-removing-pgp/"">https://blog.pypi.org/posts/2023-05-23-removing-pgp/</a></p>
</li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11412"">#11412</a>: Emit warnings on using a deprecated Python-specific index entry type
(namely, <code>module</code>, <code>keyword</code>, <code>operator</code>, <code>object</code>, <code>exception</code>,
<code>statement</code>, and <code>builtin</code>) in the :rst:dir:<code>index</code> directive, and
set the removal version to Sphinx 9. Patch by Adam Turner.</li>
</ul>
<h2>Features added</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11415"">#11415</a>: Add a checksum to JavaScript and CSS asset URIs included within
generated HTML, using the CRC32 algorithm.</li>
<li>:meth:<code>~sphinx.application.Sphinx.require_sphinx</code> now allows the version
requirement to be specified as <code>(major, minor)</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11011"">#11011</a>: Allow configuring a line-length limit for object signatures, via
:confval:<code>maximum_signature_line_length</code> and the domain-specific variants.
If the length of the signature (in characters) is greater than the configured
limit, each parameter in the signature will be split to its own logical line.
This behaviour may also be controlled by options on object description
directives, for example :rst:dir:<code>py:function:single-line-parameter-list</code>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/320d0b631cc7d9617f5167cc61c9284674f5b2df""><code>320d0b6</code></a> Bump to 7.1.2 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/06c1339bb36378512d2e27bcff86b6bd5db0dc0a""><code>06c1339</code></a> Ensure that <code>linkcheck_anchors</code> is properly respected (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11544"">#11544</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/89808c6f49e1738765d18309244dca0156ee28f6""><code>89808c6</code></a> Emend attribution for <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11431"">GH-11431</a> in CHANGES (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11547"">#11547</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4958d2abf1abb1c3a10e77d8297a746cc90d5234""><code>4958d2a</code></a> Fix flake8 E231 reported linting failure</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4862da883f8c984e8f15a8643c61702f90947d6c""><code>4862da8</code></a> Bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d8d73c16b8b86df8f1a32698229a2b818fea2e98""><code>d8d73c1</code></a> Bump to 7.1.1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/a54e630a21a6d9b7e80a22e969e3a86e35678d2d""><code>a54e630</code></a> [internationalisation] Update message catalogues</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/8452300d54dce2da751941d9547dd54dc03e69bf""><code>8452300</code></a> Fix multi-line copyright when <code>SOURCE_DATE_EPOCH</code> is set (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11524"">#11524</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/fe08cec019c52c15b7f7365013134e3f3d1b3739""><code>fe08cec</code></a> bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e560f6334aa87af07304415e6aeaf42d03d1c525""><code>e560f63</code></a> Bump to 7.1.0 final</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v7.1.2"">compare view</a></li>
</ul>
</details>
<br />

Updates `furo` from 2023.3.27 to 2023.7.26
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<ul>
<li>Update Python version classifiers.</li>
<li>Increase the icon size in mobile header.</li>
<li>Increase admonition title bg opacity.</li>
<li>Change the default API background to transparent.</li>
<li>Transition the API background change.</li>
<li>Remove the &quot;indent&quot; of API entries which have a background.</li>
<li>Break long inline code literals.</li>
</ul>
<h2>2022.12.07 -- Reverent Raspberry</h2>
<ul>
<li>✨ Add support for Sphinx 6.</li>
<li>✨ Improve footnote presentation with docutils 0.18+.</li>
<li>Drop support for Sphinx 4.</li>
<li>Improve documentation about what the edit button does.</li>
<li>Improve handling of empty-flexboxes for better print experience on Chrome.</li>
<li>Improve styling for inline signatures.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/35f5307acc66d63ddb2e50eebc79a389518e881a""><code>35f5307</code></a> Prepare release: 2023.07.26</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/0a8bedc51399777be2d01f2401fd52a32d9f6673""><code>0a8bedc</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/a92dd0c2bfa398825e239171925a81bdc428e90a""><code>a92dd0c</code></a> Make <code>_add_asset_hashes</code> a no-op with Sphinx 7.1</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/f8db95bed5d1c57b4ea6bf536c65560fa9ae33a9""><code>f8db95b</code></a> Improve literals with inline code are handled</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/1680dbefd61766a28e43880ae05a9f5cc0e7f19b""><code>1680dbe</code></a> Document the use of <code>figclass</code> with <code>figure</code> directive</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/beebd7e676d1538891fef4ead30f79d866fe78c2""><code>beebd7e</code></a> Increase the specificity of the admonition title selector</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/834e951f5567fc4b0aa3564d33c0f0011d459263""><code>834e951</code></a> Setup uploads to Percy</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/27bf2c0e6f1b2c3ae9b0a7beed689c4df9bed1b2""><code>27bf2c0</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/pradyunsg/furo/issues/672"">#672</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c8b51d09af3dcaac3046f7e761119e9d1b7c9e37""><code>c8b51d0</code></a> Fix how content overflow is handled</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/80afa27a9a7bd0ac9259636239e349656c4726ab""><code>80afa27</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/pradyunsg/furo/issues/652"">#652</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.03.27...2023.07.26"">compare view</a></li>
</ul>
</details>
<br />

Updates `blacken-docs` from 1.14.0 to 1.15.0
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.15.0 (2023-07-09)</h2>
<ul>
<li>Drop Python 3.7 support.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/2e74dc4c30c336176e76e0426c66f09a4e79df9c""><code>2e74dc4</code></a> Version 1.15.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8ddb2f1269b7652a9b668faa92c9d63d764696c0""><code>8ddb2f1</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/257"">#257</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/64fbca6cad36db1822528ece7f97a8152d29e05f""><code>64fbca6</code></a> Drop Python 3.7 support (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/256"">#256</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f6a0817dbb7957f96ea6cc251edf3a50bf1c3c08""><code>f6a0817</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/255"">#255</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/cbdcb50b125b4f2fdb470122b2ba5e3c15de241a""><code>cbdcb50</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/254"">#254</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/37fc70ee00c7063777ed7343c1bd26fceb5422a6""><code>37fc70e</code></a> Upgrade requirements (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/253"">#253</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7bbe434415ce9f590cb057010590a648e7bea072""><code>7bbe434</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/250"">#250</a>)</li>
<li>See full diff in <a href=""https://github.com/asottile/blacken-docs/compare/1.14.0...1.15.0"">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2128.org.readthedocs.build/en/2128/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1668113177,IC_kwDOCGYnMM5jbWMZ,,25778,2023-08-07T15:41:49Z,I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.,140912432
issues,1839344979,I_kwDOCGYnMM5toi1T,Handling CSV/file input that contains NUL bytes,1448859,2023-08-07T12:24:14Z,"I was using sqlite-utils to create a DB from a CSV and it turns out the CSV contains a NUL byte.

When the processing reaches the line that contains the NUL an exception is raised.

I'm wondering if there is something that can be done in `sqlite-utils` to say ""skip lines with encoding errors"" or some such. I think it isn't super straightforward though as the exception comes from inside the `csv` module that does all the parsing.

Concretely the file is the `KernelVersions.csv` from https://www.kaggle.com/datasets/kaggle/meta-kaggle

This is the command and output:
```
$ sqlite-utils insert --csv kaggle.db kaggle KernelVersions.csv
  [------------------------------------]    0%
  [#####################---------------]   60%  00:04:24Traceback (most recent call last):
  File ""/home/foobar/miniconda/envs/meta-kaggle/bin/sqlite-utils"", line 10, in <module>
    sys.exit(cli())
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py"", line 1128, in __call__
    return self.main(*args, **kwargs)
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py"", line 1053, in main
    rv = self.invoke(ctx)
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py"", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py"", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/click/core.py"", line 754, in invoke
    return __callback(*args, **kwargs)
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py"", line 1223, in insert
    insert_upsert_implementation(
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py"", line 1085, in insert_upsert_implementation
    db[table].insert_all(
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/db.py"", line 3198, in insert_all
    chunk = list(chunk)
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/db.py"", line 3742, in fix_square_braces
    for record in records:
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py"", line 1071, in <genexpr>
    docs = (decode_base64_values(doc) for doc in docs)
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py"", line 1068, in <genexpr>
    docs = (verify_is_dict(doc) for doc in docs)
  File ""/home/foobar/miniconda/envs/meta-kaggle/lib/python3.10/site-packages/sqlite_utils/cli.py"", line 1003, in <genexpr>
    docs = (dict(zip(headers, row)) for row in reader)
_csv.Error: line contains NUL
```",140912432
issue_comments,1667011607,IC_kwDOBm6k_c5jXJQX,,9599,2023-08-07T00:02:49Z,"This is pretty neat:

https://github.com/simonw/datasette/blob/2f9038a831a3510d4c9ab39a12d96259b3a55bc7/datasette/views/database.py#L157-L206",107914493
issues,1838469176,I_kwDOBm6k_c5tlNA4,Context base class to support documenting the context,9599,2023-08-07T00:01:02Z,"This idea first came up here:
- https://github.com/simonw/datasette/issues/2112#issuecomment-1652751140

If `datasette.render_template(...)` takes an optional `Context` subclass as an alternative to a context dictionary, I could then use dataclasses to define the context made available to specific templates - which then gives me something I can use to help document what they are.

Also refs:
- https://github.com/simonw/datasette/issues/1510",107914493
issue_comments,1666912107,IC_kwDOBm6k_c5jWw9r,,36199671,2023-08-06T16:27:34Z,"And in similar fashion, how can I assign the `edit-tiddlywiki` permission to my user `myuser` in `metadata.yml` / `metadata.json`?",107914493
issues,1838266862,I_kwDOBm6k_c5tkbnu,Permissions in metadata.yml / metadata.json,36199671,2023-08-06T16:24:10Z,"https://docs.datasette.io/en/latest/authentication.html#other-permissions-in-metadata says the following:

> For all other permissions, you can use one or more ""permissions"" blocks in your metadata.

> To grant access to the permissions debug tool to all signed in users you can grant permissions-debug to any actor with an id matching the wildcard * by adding this a the root of your metadata:

```yaml
permissions:
  debug-menu:
    id: '*'
```

I tried this.

My `metadata.yml` file looks like:

```yaml
permissions:
  debug-menu:
    id: '*'
  permissions-debug:
    id: '*'
plugins:
  datasette-auth-passwords:
    myuser_password_hash:
      $env: ""PASSWORD_HASH_MYUSER""
```

And then I run

```zsh
datasette -m metadata.yml tiddlywiki.db --root
```

And I open a session for the ""root"" user of datasette with the link given.

I open a private browser session and log in as ""myuser"" from http://127.0.0.1:8001/-/login

Then I check http://127.0.0.1:8001/-/actor which confirms that I am logged in as the ""myuser"" actor

```json
{
    ""actor"": {
        ""id"": ""myuser""
    }
}
```

In the session where I am logged in as ""myuser"" I then try to go to http://127.0.0.1:8001/-/permissions

But all I get there as the logged in user ""myuser"" is

> Forbidden
>
> Permission denied

And then if I check the http://127.0.0.1:8001/-/permissions as the datasette ""root"" user from another browser session, I see:

> permissions-debug checked at 2023-08-06T16:22:58.997841 ✗ (used default)
>
> Actor: {""id"": ""myuser""}

It seems that in spite of having tried to give the `permissions-debug` permission to the ""myuser"" user in my `metadata.yml` file, datasette does not agree that ""myuser"" has permission `permissions-debug`..

What do I need to do differently so that my ""myuser"" user is able to access http://127.0.0.1:8001/-/permissions ?",107914493
issue_comments,1662228666,IC_kwDOBm6k_c5jE5i6,,22429695,2023-08-02T13:36:22Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`730f554`)](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2125   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1662215579,IC_kwDOBm6k_c5jE2Wb,,49699333,2023-08-02T13:28:43Z,Superseded by #2125.,107914493
issues,1833193570,PR_kwDOBm6k_c5XArm3,Bump sphinx from 6.1.3 to 7.1.2,49699333,2023-08-02T13:28:39Z,"Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 7.1.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.1.2</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.1.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.1.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0rc1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.1.2 (released Aug 02, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11542"">#11542</a>: linkcheck: Properly respect :confval:<code>linkcheck_anchors</code>
and do not spuriously report failures to validate anchors.
Patch by James Addison.</li>
</ul>
<h1>Release 7.1.1 (released Jul 27, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11514"">#11514</a>: Fix <code>SOURCE_DATE_EPOCH</code> in multi-line copyright footer.
Patch by Bénédikt Tran.</li>
</ul>
<h1>Release 7.1.0 (released Jul 24, 2023)</h1>
<h2>Incompatible changes</h2>
<ul>
<li>
<p>Releases are no longer signed, given the <code>change in PyPI policy</code>_.</p>
<p>.. _change in PyPI policy: <a href=""https://blog.pypi.org/posts/2023-05-23-removing-pgp/"">https://blog.pypi.org/posts/2023-05-23-removing-pgp/</a></p>
</li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11412"">#11412</a>: Emit warnings on using a deprecated Python-specific index entry type
(namely, <code>module</code>, <code>keyword</code>, <code>operator</code>, <code>object</code>, <code>exception</code>,
<code>statement</code>, and <code>builtin</code>) in the :rst:dir:<code>index</code> directive, and
set the removal version to Sphinx 9. Patch by Adam Turner.</li>
</ul>
<h2>Features added</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11415"">#11415</a>: Add a checksum to JavaScript and CSS asset URIs included within
generated HTML, using the CRC32 algorithm.</li>
<li>:meth:<code>~sphinx.application.Sphinx.require_sphinx</code> now allows the version
requirement to be specified as <code>(major, minor)</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11011"">#11011</a>: Allow configuring a line-length limit for object signatures, via
:confval:<code>maximum_signature_line_length</code> and the domain-specific variants.
If the length of the signature (in characters) is greater than the configured
limit, each parameter in the signature will be split to its own logical line.
This behaviour may also be controlled by options on object description
directives, for example :rst:dir:<code>py:function:single-line-parameter-list</code>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/320d0b631cc7d9617f5167cc61c9284674f5b2df""><code>320d0b6</code></a> Bump to 7.1.2 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/06c1339bb36378512d2e27bcff86b6bd5db0dc0a""><code>06c1339</code></a> Ensure that <code>linkcheck_anchors</code> is properly respected (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11544"">#11544</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/89808c6f49e1738765d18309244dca0156ee28f6""><code>89808c6</code></a> Emend attribution for <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11431"">GH-11431</a> in CHANGES (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11547"">#11547</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4958d2abf1abb1c3a10e77d8297a746cc90d5234""><code>4958d2a</code></a> Fix flake8 E231 reported linting failure</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4862da883f8c984e8f15a8643c61702f90947d6c""><code>4862da8</code></a> Bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d8d73c16b8b86df8f1a32698229a2b818fea2e98""><code>d8d73c1</code></a> Bump to 7.1.1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/a54e630a21a6d9b7e80a22e969e3a86e35678d2d""><code>a54e630</code></a> [internationalisation] Update message catalogues</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/8452300d54dce2da751941d9547dd54dc03e69bf""><code>8452300</code></a> Fix multi-line copyright when <code>SOURCE_DATE_EPOCH</code> is set (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11524"">#11524</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/fe08cec019c52c15b7f7365013134e3f3d1b3739""><code>fe08cec</code></a> bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e560f6334aa87af07304415e6aeaf42d03d1c525""><code>e560f63</code></a> Bump to 7.1.0 final</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v7.1.2"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=6.1.3&new-version=7.1.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2125.org.readthedocs.build/en/2125/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1658968449,IC_kwDOBm6k_c5i4dmB,,9599,2023-07-31T18:57:58Z,"Added to the 1.0 milestone because: https://news.ycombinator.com/item?id=36932876#36939734

>  Hah, wow, it looks like I've been procrastinating on making a decision if I like that or not for three years!
>
> I'll add it to the Datasette 1.0 milestone so it definitely gets my attention before shipping that release. ",107914493
issues,1827436260,PR_kwDOD079W85WtVyk,Missing option in datasette instructions,319473,2023-07-29T10:34:48Z,Gotta tell it where to look,256834907
issue_comments,1656696679,IC_kwDOD079W85ivy9n,,319473,2023-07-29T10:10:29Z,"+1 to getting this merged down.

For future googlers, I installed by...
```
git clone git@github.com:RhetTbull/dogsheep-photos.git
cd dogsheep-photos
git checkout update_for_bigsur
python setup.py install
```",256834907
issue_comments,1656694944,IC_kwDOD079W85ivyig,,319473,2023-07-29T10:01:19Z,Duplicate of https://github.com/dogsheep/dogsheep-photos/pull/36 - closing.,256834907
issue_comments,1656694854,IC_kwDOD079W85ivyhG,,319473,2023-07-29T10:00:45Z,Ran across https://github.com/dogsheep/dogsheep-photos/issues/33 which is the same subject.  My PR just fixes docs,256834907
issues,1827427757,PR_kwDOD079W85WtUKG,photos-to-sql not found?,319473,2023-07-29T09:59:42Z,"I wonder if `photos-to-sql` is an old name for `dogsheep-photos`, because I can't find it anywhere.

I can't actually get this command to work (`sqlite3.OperationalError: no such table: attached.ZGENERICASSET` thrown) but I don't think that's related",256834907
issue_comments,1655691977,IC_kwDOBm6k_c5ir9rJ,,22429695,2023-07-28T13:33:54Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`9388102`)](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2124   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1655678215,IC_kwDOBm6k_c5ir6UH,,49699333,2023-07-28T13:23:16Z,Superseded by #2124.,107914493
issues,1826424151,PR_kwDOBm6k_c5Wp6Hs,Bump sphinx from 6.1.3 to 7.1.1,49699333,2023-07-28T13:23:11Z,"Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 7.1.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.1.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>Sphinx 7.1.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0rc1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.1.1 (released Jul 27, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11514"">#11514</a>: Fix <code>SOURCE_DATE_EPOCH</code> in multi-line copyright footer.
Patch by Bénédikt Tran.</li>
</ul>
<h1>Release 7.1.0 (released Jul 24, 2023)</h1>
<h2>Incompatible changes</h2>
<ul>
<li>
<p>Releases are no longer signed, given the <code>change in PyPI policy</code>_.</p>
<p>.. _change in PyPI policy: <a href=""https://blog.pypi.org/posts/2023-05-23-removing-pgp/"">https://blog.pypi.org/posts/2023-05-23-removing-pgp/</a></p>
</li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11412"">#11412</a>: Emit warnings on using a deprecated Python-specific index entry type
(namely, <code>module</code>, <code>keyword</code>, <code>operator</code>, <code>object</code>, <code>exception</code>,
<code>statement</code>, and <code>builtin</code>) in the :rst:dir:<code>index</code> directive, and
set the removal version to Sphinx 9. Patch by Adam Turner.</li>
</ul>
<h2>Features added</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11415"">#11415</a>: Add a checksum to JavaScript and CSS asset URIs included within
generated HTML, using the CRC32 algorithm.</li>
<li>:meth:<code>~sphinx.application.Sphinx.require_sphinx</code> now allows the version
requirement to be specified as <code>(major, minor)</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11011"">#11011</a>: Allow configuring a line-length limit for object signatures, via
:confval:<code>maximum_signature_line_length</code> and the domain-specific variants.
If the length of the signature (in characters) is greater than the configured
limit, each parameter in the signature will be split to its own logical line.
This behaviour may also be controlled by options on object description
directives, for example :rst:dir:<code>py:function:single-line-parameter-list</code>.
Patch by Thomas Louf, Adam Turner, and Jean-François B.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10983"">#10983</a>: Support for multiline copyright statements in the footer block.
Patch by Stefanie Molin</li>
<li><code>sphinx.util.display.status_iterator</code> now clears the current line
with ANSI control codes, rather than overprinting with space characters.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11431"">#11431</a>: linkcheck: Treat SSL failures as broken links.
Patch by Bénédikt Tran</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11157"">#11157</a>: Keep the <code>translated</code> attribute on translated nodes.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11451"">#11451</a>: Improve the traceback displayed when using :option:<code>sphinx-build -T</code>
in parallel builds. Patch by Bénédikt Tran</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d8d73c16b8b86df8f1a32698229a2b818fea2e98""><code>d8d73c1</code></a> Bump to 7.1.1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/a54e630a21a6d9b7e80a22e969e3a86e35678d2d""><code>a54e630</code></a> [internationalisation] Update message catalogues</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/8452300d54dce2da751941d9547dd54dc03e69bf""><code>8452300</code></a> Fix multi-line copyright when <code>SOURCE_DATE_EPOCH</code> is set (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11524"">#11524</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/fe08cec019c52c15b7f7365013134e3f3d1b3739""><code>fe08cec</code></a> bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e560f6334aa87af07304415e6aeaf42d03d1c525""><code>e560f63</code></a> Bump to 7.1.0 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/066e0fabc60c55072b3f75876637a44fdcb51332""><code>066e0fa</code></a> Add translation progress information (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11509"">#11509</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/0882914d5c5057ad3cec280c0142cfe5f6965844""><code>0882914</code></a> Target PyPI in create-release.yml</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/21fbee5e83a3e9d34e589d06c66d928f3a67923c""><code>21fbee5</code></a> Fix OIDC token payload</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/1a403e4f9ad4d8d7a3e4c7c34f55d9a45ed600bd""><code>1a403e4</code></a> Add informational log messaging</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/258b0ea9fcdd05221ebead5e5c162a7ad37c4412""><code>258b0ea</code></a> Revert &quot;Switch to using <code>github.request</code>&quot;</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v7.1.1"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=6.1.3&new-version=7.1.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2124.org.readthedocs.build/en/2124/

<!-- readthedocs-preview datasette end -->",107914493
issues,1825007061,I_kwDOBm6k_c5sx2XV,datasette serve when invoked with --reload interprets the serve command as a file,79087,2023-07-27T19:07:22Z,"When running `datasette serve` with the `--reload` flag, the serve command is picked up as a file argument:

```
$ datasette serve --reload test_db
Starting monitor for PID 13574.
Error: Invalid value for '[FILES]...': Path 'serve' does not exist.
Press ENTER or change a file to reload.
```

If a 'serve' file is created it launches properly (albeit with an empty database called serve):

```
$ touch serve; datasette serve --reload test_db
Starting monitor for PID 13628.
INFO:     Started server process [13628]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
```

Version (running from HEAD on main):

```
$ datasette --version
datasette, version 1.0a2
```

This issue appears to have existed for awhile as https://github.com/simonw/datasette/issues/1380#issuecomment-953366110 mentions the error in a different context.

I'm happy to debug and land a patch if it's welcome.",107914493
issues,1824457306,I_kwDOBm6k_c5svwJa,Parameters on canned queries: fixed or query-generated list?,1563881,2023-07-27T14:07:07Z,"Hi,

currently parameters in canned queries are just text fields. It would be cool to have one of the options below. Would you accept a PR doing something in this direction? (Possibly this could even work as a plugin.)

* adding facets, which would work like facets on tables or views, giving a list of selectable options (and leaving parameters as is)
* making it possible to provide a query which returns selectable values for a parameter, e.g.
```     
calendar_entries_current_instrument:
        sql: |      
          select * from calendar_entries 
              where 
                    DTEND_UNIX > UNIXEPOCH() and
                    DTSTART_UNIX < UNIXEPOCH() + :days *24*60*60 and
                    current = 1 and
                    MACHINE = :instrument
              order by
                    DTSTART_UNIX
        params:
            days: 
              sql: ""SELECT VALUE FROM generate_series(1, 30, 1)""
              # this obviously requires the corresponding sqlite extension
            instrument:
              sql: ""SELECT DISTINCT MACHINE FROM calendar_entries""
```
* making it possible to provide a fixed list of parameters
```     
calendar_entries_current_instrument:
        sql: |      
          select * from calendar_entries 
              where 
                    DTEND_UNIX > UNIXEPOCH() and
                    DTSTART_UNIX < UNIXEPOCH() + :days *24*60*60 and
                    current = 1 and
                    MACHINE = :instrument
              order by
                    DTSTART_UNIX
        params:
            days: 
              values: [1, 2, 3, 5, 10, 20, 30]
            instrument:
              values: [supermachine, crappymachine, boringmachine]
```",107914493
issue_comments,1653666188,IC_kwDOBm6k_c5ikPGM,,22429695,2023-07-27T13:48:43Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`3aca01f`)](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2121   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1653652665,IC_kwDOBm6k_c5ikLy5,,49699333,2023-07-27T13:40:52Z,Superseded by #2121.,107914493
issues,1824399610,PR_kwDOBm6k_c5WjCS8,Bump furo from 2023.3.27 to 2023.7.26,49699333,2023-07-27T13:40:48Z,"Bumps [furo](https://github.com/pradyunsg/furo) from 2023.3.27 to 2023.7.26.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.07.26 -- Vigilant Volt</h2>
<ul>
<li>Fix compatiblity with Sphinx 7.1.</li>
<li>Improve how content overflow is handled.</li>
<li>Improve how literal blocks containing inline code are handled.</li>
</ul>
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<ul>
<li>Update Python version classifiers.</li>
<li>Increase the icon size in mobile header.</li>
<li>Increase admonition title bg opacity.</li>
<li>Change the default API background to transparent.</li>
<li>Transition the API background change.</li>
<li>Remove the &quot;indent&quot; of API entries which have a background.</li>
<li>Break long inline code literals.</li>
</ul>
<h2>2022.12.07 -- Reverent Raspberry</h2>
<ul>
<li>✨ Add support for Sphinx 6.</li>
<li>✨ Improve footnote presentation with docutils 0.18+.</li>
<li>Drop support for Sphinx 4.</li>
<li>Improve documentation about what the edit button does.</li>
<li>Improve handling of empty-flexboxes for better print experience on Chrome.</li>
<li>Improve styling for inline signatures.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/35f5307acc66d63ddb2e50eebc79a389518e881a""><code>35f5307</code></a> Prepare release: 2023.07.26</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/0a8bedc51399777be2d01f2401fd52a32d9f6673""><code>0a8bedc</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/a92dd0c2bfa398825e239171925a81bdc428e90a""><code>a92dd0c</code></a> Make <code>_add_asset_hashes</code> a no-op with Sphinx 7.1</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/f8db95bed5d1c57b4ea6bf536c65560fa9ae33a9""><code>f8db95b</code></a> Improve literals with inline code are handled</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/1680dbefd61766a28e43880ae05a9f5cc0e7f19b""><code>1680dbe</code></a> Document the use of <code>figclass</code> with <code>figure</code> directive</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/beebd7e676d1538891fef4ead30f79d866fe78c2""><code>beebd7e</code></a> Increase the specificity of the admonition title selector</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/834e951f5567fc4b0aa3564d33c0f0011d459263""><code>834e951</code></a> Setup uploads to Percy</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/27bf2c0e6f1b2c3ae9b0a7beed689c4df9bed1b2""><code>27bf2c0</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/pradyunsg/furo/issues/672"">#672</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c8b51d09af3dcaac3046f7e761119e9d1b7c9e37""><code>c8b51d0</code></a> Fix how content overflow is handled</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/80afa27a9a7bd0ac9259636239e349656c4726ab""><code>80afa27</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/pradyunsg/furo/issues/652"">#652</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.03.27...2023.07.26"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=furo&package-manager=pip&previous-version=2023.3.27&new-version=2023.7.26)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2121.org.readthedocs.build/en/2121/

<!-- readthedocs-preview datasette end -->",107914493
issues,1823428714,I_kwDOBm6k_c5sr1Bq,Add __all__ to datasette/__init__.py,9599,2023-07-27T01:07:10Z,"Currently looks like this: https://github.com/simonw/datasette/blob/08181823990a71ffa5a1b57b37259198eaa43e06/datasette/__init__.py#L1-L6

Adding `__all__ = [""Permission"", ""Forbidden""...]` would let me get rid of those `# noqa` comments.",107914493
issue_comments,1652751140,IC_kwDOBm6k_c5igvsk,,9599,2023-07-27T00:52:53Z,"Maybe I teach `datasette.render_template()` to take a dataclass instance as an optional alternative to a dictionary, and if it gets one it turns it into something that works well in the template context.",107914493
issue_comments,1652746185,IC_kwDOBm6k_c5igufJ,,9599,2023-07-27T00:46:37Z,"Got Claude to generate this code:
```python
from dataclasses import dataclass, field

@dataclass
class QueryContext:
    database: str = field(metadata={""help"": ""The name of the database being queried""})
    query: dict = field(metadata={""help"": ""The SQL query object containing the `sql` string""})
    canned_query: str = field(metadata={""help"": ""The name of the canned query if this is a canned query""})
    private: bool = field(metadata={""help"": ""Boolean indicating if this is a private database""})
    urls: dict = field(metadata={""help"": ""Object containing URL helpers like `database()`""})
    canned_write: bool = field(metadata={""help"": ""Boolean indicating if this canned query allows writes""})
    db_is_immutable: bool = field(metadata={""help"": ""Boolean indicating if this database is immutable""})
    error: str = field(metadata={""help"": ""Any query error message""})
    hide_sql: bool = field(metadata={""help"": ""Boolean indicating if the SQL should be hidden""})
    show_hide_link: str = field(metadata={""help"": ""The URL to toggle showing/hiding the SQL""})
    show_hide_text: str = field(metadata={""help"": ""The text for the show/hide SQL link""})
    editable: bool = field(metadata={""help"": ""Boolean indicating if the SQL can be edited""})
    allow_execute_sql: bool = field(metadata={""help"": ""Boolean indicating if custom SQL can be executed""})
    tables: list = field(metadata={""help"": ""List of table objects in the database""})
    named_parameter_values: dict = field(metadata={""help"": ""Dictionary of parameter names/values""})
    csrftoken: callable = field(metadata={""help"": ""Function to generate a CSRF token""})
    edit_sql_url: str = field(metadata={""help"": ""URL to edit the SQL for a canned query""})
    display_rows: list = field(metadata={""help"": ""List of result rows to display""})
    columns: list = field(metadata={""help"": ""List of column names""})
    renderers: dict = field(metadata={""help"": ""Dictionary of renderer name to URL""})
    url_csv: str = field(metadata={""help"": ""URL for CSV export""})
    metadata: dict = field(metadata={""help"": ""Metadata about the query/database""})
```",107914493
issue_comments,1652723029,IC_kwDOBm6k_c5igo1V,,9599,2023-07-27T00:25:49Z,"Getting the query page working with all of the necessary HTML context variables is a slog.

I uploaded `query.html` to Claude and asked it ""List of every template context variable with a detailed description of each one"":

<img width=""518"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/9c7f3292-d3f3-49bc-9263-dc39bed5aa58"">

> Based on the template code provided, here is a list of the template context variables and a description of each one:
> 
> - `database` - The name of the database being queried.
> - `query` - The SQL query object containing the `sql` string.
> - `canned_query` - The name of the canned query if this is a canned query. 
> - `private` - Boolean indicating if this is a private database.
> - `urls` - Object containing URL helpers like `database()`.
> - `canned_write` - Boolean indicating if this canned query allows writes.
> - `db_is_immutable` - Boolean indicating if this database is immutable. 
> - `error` - Any query error message.
> - `hide_sql` - Boolean indicating if the SQL should be hidden. 
> - `show_hide_link` - The URL to toggle showing/hiding the SQL.
> - `show_hide_text` - The text for the show/hide SQL link. 
> - `editable` - Boolean indicating if the SQL can be edited.
> - `allow_execute_sql` - Boolean indicating if custom SQL can be executed.
> - `tables` - List of table objects in the database.
> - `named_parameter_values` - Dictionary of parameter names/values.
> - `csrftoken()` - Function to generate a CSRF token.
> - `edit_sql_url` - URL to edit the SQL for a canned query.
> - `display_rows` - List of result rows to display.
> - `columns` - List of column names. 
> - `renderers` - Dictionary of renderer name to URL.
> - `url_csv` - URL for CSV export.
> - `metadata` - Metadata about the query/database.
> - `crumbs` - Breadcrumb navigation object.
> 
> So in summary, it includes variables for the database, query, results, URLs, permissions, and metadata to power the template.",107914493
issue_comments,1652722208,IC_kwDOBm6k_c5igoog,,9599,2023-07-27T00:24:50Z,"I changed my mind, I'm not going to do this with a format hook.

Also: https://github.com/simonw/datasette/issues/2109#issuecomment-1652715823

I've decided to let HTML have its own special context. I'm going to try formalizing that as a `@dataclass` with fields with `metadata={""help"": ...}` to see what that looks like.",107914493
issue_comments,1652721111,IC_kwDOBm6k_c5igoXX,,9599,2023-07-27T00:23:28Z,"Once again I'm tempted to formalize the extra HTML context as a `dataclass` so I can ensure it is documented correctly.

Here's an example of Hugging Face doing that: https://github.com/huggingface/transformers/blob/1689aea73346816b936b84932e12b774974e61a6/src/transformers/training_args.py#L622C1-L624

```python
from dataclasses import dataclass, field

@dataclass
class TrainingArguments:
    output_dir: str = field(
        metadata={""help"": ""The output directory where the model predictions and checkpoints will be written.""},
    )
```",107914493
issues,1823393475,I_kwDOBm6k_c5srsbD,"database color shows only on index page, not other pages",9599,2023-07-27T00:19:39Z,"I think this has been a bug for a long time.

https://latest.datasette.io/ currently shows:

<img width=""827"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/fcd08d0c-5b07-40be-bc9f-3ba9199ad858"">

Those colors are based on a hash of the database name. But when you click through to https://latest.datasette.io/fixtures

<img width=""692"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/7c9dfd66-06e8-47fc-9ed7-ce425da22cb4"">

It's red on all sub-pages too.",107914493
issue_comments,1652715823,IC_kwDOBm6k_c5ignEv,,9599,2023-07-27T00:17:04Z,"New decision: I had originally decided that the HTML view would just use data that could otherwise be extracted from the JSON view if you fed in enough extras.

I've changed my mind. I'm OK with the HTML view getting a few bonus things available in its context, provided those are clearly documented for template authors.

This should help avoid me having to make many changes to the templates themselves.

HTML-specific stuff will be things like `csrftoken()` and `edit_sql_url` and `show_hide_link` and `database_color` and suchlike.",107914493
issue_comments,1652681136,IC_kwDOBm6k_c5igemw,,9599,2023-07-26T23:30:44Z,"The `_shape=` stuff should use `json_renderer` instead - that's how the table view did it:
https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d#diff-5c9ef29c33ed0fde413565b23fa258d60fc3a2bb205b016db9e915c9bd5ecfb3

https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/table.py#L822-L850

Instead of:

https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/database.py#L239-L288",107914493
issues,1823352380,PR_kwDOBm6k_c5Wfgd9,New JSON design for query views,9599,2023-07-26T23:29:21Z,"WIP. Refs:
- #2109 

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2118.org.readthedocs.build/en/2118/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1652496702,IC_kwDOCGYnMM5ifxk-,,9599,2023-07-26T21:07:45Z,Docs: https://sqlite-utils.datasette.io/en/latest/cli.html#using-the-debugger,140912432
issues,1823160748,I_kwDOCGYnMM5sqzms,`sqlite-utils convert --pdb` option,9599,2023-07-26T21:02:50Z,While using `sqlite-utils convert` I realized it would be handy if you could pass `--pdb` to have it open the debugger at the first instance of a failed conversion.,140912432
issue_comments,1652407208,IC_kwDOBm6k_c5ifbuo,,9599,2023-07-26T19:54:20Z,"I implemented `/content` and `/content.json` but I left `/content?sql=...` as this:

https://github.com/simonw/datasette/blob/2e40a506ad45b44fd7642474f630a31ef18b5911/datasette/views/database.py#L220-L221",107914493
issue_comments,1652354606,IC_kwDOBm6k_c5ifO4u,,9599,2023-07-26T19:16:35Z,"I just noticed that this URL: https://latest.datasette.io/fixtures.csv

Returns a 500 error right now!

<img width=""275"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/bd560821-07ba-4e8f-8472-9e677294687d"">

It's fine with a `?sql=` query: https://latest.datasette.io/fixtures.csv?sql=select+*+from+facetable",107914493
issues,1822982933,I_kwDOBm6k_c5sqIMV,Figure out what to do about `DatabaseView.name`,9599,2023-07-26T18:58:06Z,"In the old code:

https://github.com/simonw/datasette/blob/08181823990a71ffa5a1b57b37259198eaa43e06/datasette/views/database.py#L34-L35

This `name` class attribute was later used by some of the plugin hooks, passed as `view_name`: https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/hookspecs.py#L50-L54

Figure out how that should work once I've refactored those classes to view functions instead.

Refs:
- #2109 ",107914493
issue_comments,1652330111,IC_kwDOBm6k_c5ifI5_,,9599,2023-07-26T18:55:31Z,Changed my mind on this - I'm going to have the `query_view` mapped to `/db` but have the first code on there notice if `?sql=` is missing and return a `database_view()` function instead.,107914493
issue_comments,1652325193,IC_kwDOBm6k_c5ifHtJ,,9599,2023-07-26T18:51:15Z,"Here's the code I'm going to be entirely replacing:

https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L213-L530

Plus this weird class in `views/table.py`:

https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/table.py#L701-L749",107914493
issue_comments,1652321419,IC_kwDOBm6k_c5ifGyL,,9599,2023-07-26T18:48:03Z,"This is also where I'll bring back writable canned queries:

https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L281-L334",107914493
issue_comments,1652318269,IC_kwDOBm6k_c5ifGA9,,9599,2023-07-26T18:45:23Z,"To avoid confusion I'll start by having `/content` return a HTML ""TODO: implement this"" message.",107914493
issue_comments,1652315926,IC_kwDOBm6k_c5ifFcW,,9599,2023-07-26T18:43:17Z,"Tests pass, and manually tested like this too:
```bash
datasette -i pelicans.db
```",107914493
issue_comments,1652304289,IC_kwDOBm6k_c5ifCmh,,9599,2023-07-26T18:33:07Z,"This runs three tests:
```bash
pytest -k test_database_download
```",107914493
issue_comments,1652303471,IC_kwDOBm6k_c5ifCZv,,9599,2023-07-26T18:32:24Z,https://github.com/simonw/datasette/blob/dc5171eb1b1d9f1d55e367f8a4d93edb55a43351/datasette/views/database.py#L172-L212,107914493
issues,1822949756,I_kwDOBm6k_c5sqAF8,Turn DatabaseDownload into an async view function,9599,2023-07-26T18:31:59Z,"A minor refactor, but it is a good starting point for this new branch. Refs:
- #2109",107914493
issue_comments,1652298879,IC_kwDOBm6k_c5ifBR_,,9599,2023-07-26T18:28:33Z,"Applied the same fix as here:
- https://github.com/simonw/llm/issues/136",107914493
issue_comments,1652296467,IC_kwDOBm6k_c5ifAsT,,9599,2023-07-26T18:26:44Z,"I'm abandoning this branch in favour of a fresh attempt, described here:
- https://github.com/simonw/datasette/issues/2109

I'll copy bits and pieces of this branch across as-needed.",107914493
issue_comments,1652295866,IC_kwDOBm6k_c5ifAi6,,9599,2023-07-26T18:26:18Z,"I'm going to do this work in a fresh branch, copying bits over from here as needed:
- https://github.com/simonw/datasette/pull/2053",107914493
issue_comments,1652294920,IC_kwDOBm6k_c5ifAUI,,9599,2023-07-26T18:25:34Z,"OK, these issues will do for the plan.",107914493
issues,1822940964,I_kwDOBm6k_c5sp98k,Ensure all tests pass against new query view JSON,9599,2023-07-26T18:25:20Z,- #2109 ,107914493
issues,1822940263,I_kwDOBm6k_c5sp9xn,Implement canned queries against new query JSON work,9599,2023-07-26T18:24:50Z,- #2109 ,107914493
issues,1822939274,I_kwDOBm6k_c5sp9iK,Implement and document extras for the new query view page,9599,2023-07-26T18:24:01Z,- #2109 ,107914493
issues,1822938661,I_kwDOBm6k_c5sp9Yl,Build HTML version of /content?sql=...,9599,2023-07-26T18:23:34Z,"This will help make the hook as robust as possible.
- #2109 ",107914493
issues,1822937426,I_kwDOBm6k_c5sp9FS,Implement new /content.json?sql=...,9599,2023-07-26T18:22:39Z,"This will be the base that the remaining work builds on top of. Refs:
- #2109 ",107914493
issues,1822936521,I_kwDOBm6k_c5sp83J,Merge database index page and query view,9599,2023-07-26T18:21:57Z,"Refs:
- #2109

The idea here is that hitting `/content` without a `?sql=` will show an empty result set AND default to including a bunch of extras about the list of tables in the database.

Then I won't have to think about `/content` and `/content?sql=` as separate pages any more.",107914493
issues,1822934563,I_kwDOBm6k_c5sp8Yj,Plan for getting the new JSON format query views working,9599,2023-07-26T18:20:18Z,"I've been stuck on this for too long. I'm breaking it down into a full milestone:

https://github.com/simonw/datasette/milestone/29",107914493
issues,1822918995,I_kwDOCGYnMM5sp4lT,Add way to export to a csv file using the Python library,44324811,2023-07-26T18:09:26Z,"According to the documentation, we can make a csv output using the CLI tool, but not the Python library. Could we have the latter?",140912432
issues,1822813627,I_kwDOBm6k_c5spe27,some (many?) SQL syntax errors are not throwing errors with a .csv endpoint,536941,2023-07-26T16:57:45Z,"here's a CTE query that should always fail with a syntax error:

```sql
with foo as (nonsense)
select
  *
from
  foo;
```

when we make this query against the default endpoint, we do indeed get a 400 status code the problem is returned to the user: https://global-power-plants.datasettes.com/global-power-plants?sql=with+foo+as+%28nonsense%29+select+*+from+foo%3B

but, if we use the csv endpoint, we get a 200 status code and no indication of a problem: https://global-power-plants.datasettes.com/global-power-plants.csv?sql=with+foo+as+%28nonsense%29+select+*+from+foo%3B

same with this bad sql

```sql
select
  a,
from
  foo;
```

https://global-power-plants.datasettes.com/global-power-plants?sql=select%0D%0A++a%2C%0D%0Afrom%0D%0A++foo%3B

vs 

https://global-power-plants.datasettes.com/global-power-plants.csv?sql=select%0D%0A++a%2C%0D%0Afrom%0D%0A++foo%3B

but, datasette catches this bad sql at both endpoints:

```sql
slect
  a
from
  foo;
```

https://global-power-plants.datasettes.com/global-power-plants?sql=slect%0D%0A++a%0D%0Afrom%0D%0A++foo%3B
https://global-power-plants.datasettes.com/global-power-plants.csv?sql=slect%0D%0A++a%0D%0Afrom%0D%0A++foo%3B


",107914493
issue_comments,1651904060,IC_kwDOBm6k_c5idg48,,9599,2023-07-26T14:19:00Z,"Worth noting that the `register_output_renderer()` is actually pretty easy to extend, because it returns a dictionary which could have more keys (like the required set of extras) added to it:
```python
@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""test"",
        ""render"": render_demo,
        ""can_render"": can_render_demo,  # Optional
    }
```
https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-output-renderer-datasette

One slight hiccup with that plugin hook is this:

> rows - list of `sqlite3.Row` objects

I could turn that into a Datasette defined object that behaves like a [sqlite3.Row](https://docs.python.org/3/library/sqlite3.html#sqlite3.Row) though, which would give me extra flexibility in the future.

A bit tricky though since it's implemented in C for performance: https://github.com/python/cpython/blob/b0202a4e5d6b629ba5acbc703e950f08ebaf07df/Modules/_sqlite/row.c

Pasted that into Claude for the following explanation:

> -   pysqlite_Row is the structure defining the Row object. It contains the tuple of data (self->data) and description of columns (self->description).
> -   pysqlite_row_new() is the constructor which creates a new Row object given a cursor and tuple of data.
> -   pysqlite_row_dealloc() frees the memory when Row object is deleted.
> -   pysqlite_row_keys() returns the column names of the row.
> -   pysqlite_row_length() and pysqlite_row_subscript() implement sequence like behavior to access row elements by index.
> -   pysqlite_row_subscript() also allows accessing by column name by doing a lookup in description.
> -   pysqlite_row_hash() and pysqlite_row_richcompare() implement equality checks and hash function.

I could use protocols in Python to make my own `DatasetteRow` which can be used interchangeably with `sqlite3.Row` - https://docs.python.org/3/library/typing.html#typing.Protocol

Turned this into a TIL: https://til.simonwillison.net/python/protocols",107914493
issue_comments,1651894668,IC_kwDOBm6k_c5idemM,,9599,2023-07-26T14:14:34Z,"Another point of confusion is how `/content` sometimes serves the database index page (with a list of tables) and sometimes solves the results of a query.

I could resolve this by turning the information on the index page into extras, which can optionally be requested any time a query is run but default to being shown if there is no query.",107914493
issue_comments,1651883505,IC_kwDOBm6k_c5idb3x,,9599,2023-07-26T14:08:20Z,"I think the hardest part of getting this working is dealing with the different formats.

Idea: refactor `.html` as a format (since it's by far the most complex) and tweak the plugin hook a bit as part of that, then use what I learn from that to get the other formats working.",107914493
issue_comments,1651874649,IC_kwDOBm6k_c5idZtZ,,9599,2023-07-26T14:03:37Z,"Big chunk of commented-out code I just removed:
```python

    import pdb

    pdb.set_trace()

    if isinstance(output, dict) and output.get(""ok"") is False:
        # TODO: Other error codes?

        response.status_code = 400

    if datasette.cors:
        add_cors_headers(response.headers)

    return response

    # registry = Registry(
    #     extra_count,
    #     extra_facet_results,
    #     extra_facets_timed_out,
    #     extra_suggested_facets,
    #     facet_instances,
    #     extra_human_description_en,
    #     extra_next_url,
    #     extra_columns,
    #     extra_primary_keys,
    #     run_display_columns_and_rows,
    #     extra_display_columns,
    #     extra_display_rows,
    #     extra_debug,
    #     extra_request,
    #     extra_query,
    #     extra_metadata,
    #     extra_extras,
    #     extra_database,
    #     extra_table,
    #     extra_database_color,
    #     extra_table_actions,
    #     extra_filters,
    #     extra_renderers,
    #     extra_custom_table_templates,
    #     extra_sorted_facet_results,
    #     extra_table_definition,
    #     extra_view_definition,
    #     extra_is_view,
    #     extra_private,
    #     extra_expandable_columns,
    #     extra_form_hidden_args,
    # )

    results = await registry.resolve_multi(
        [""extra_{}"".format(extra) for extra in extras]
    )
    data = {
        ""ok"": True,
        ""next"": next_value and str(next_value) or None,
    }
    data.update(
        {
            key.replace(""extra_"", """"): value
            for key, value in results.items()
            if key.startswith(""extra_"") and key.replace(""extra_"", """") in extras
        }
    )
    raw_sqlite_rows = rows[:page_size]
    data[""rows""] = [dict(r) for r in raw_sqlite_rows]

    private = False
    if canned_query:
        # Respect canned query permissions
        visible, private = await datasette.check_visibility(
            request.actor,
            permissions=[
                (""view-query"", (database, canned_query)),
                (""view-database"", database),
                ""view-instance"",
            ],
        )
        if not visible:
            raise Forbidden(""You do not have permission to view this query"")

    else:
        await datasette.ensure_permissions(request.actor, [(""execute-sql"", database)])

    # If there's no sql, show the database index page
    if not sql:
        return await database_index_view(request, datasette, db)

    validate_sql_select(sql)

    # Extract any :named parameters
    named_parameters = named_parameters or await derive_named_parameters(db, sql)
    named_parameter_values = {
        named_parameter: params.get(named_parameter) or """"
        for named_parameter in named_parameters
        if not named_parameter.startswith(""_"")
    }

    # Set to blank string if missing from params
    for named_parameter in named_parameters:
        if named_parameter not in params and not named_parameter.startswith(""_""):
            params[named_parameter] = """"

    extra_args = {}
    if params.get(""_timelimit""):
        extra_args[""custom_time_limit""] = int(params[""_timelimit""])
    if _size:
        extra_args[""page_size""] = _size

    templates = [f""query-{to_css_class(database)}.html"", ""query.html""]
    if canned_query:
        templates.insert(
            0,
            f""query-{to_css_class(database)}-{to_css_class(canned_query)}.html"",
        )

    query_error = None

    # Execute query - as write or as read
    if write:
        raise NotImplementedError(""Write queries not yet implemented"")
        # if request.method == ""POST"":
        #     # If database is immutable, return an error
        #     if not db.is_mutable:
        #         raise Forbidden(""Database is immutable"")
        #     body = await request.post_body()
        #     body = body.decode(""utf-8"").strip()
        #     if body.startswith(""{"") and body.endswith(""}""):
        #         params = json.loads(body)
        #         # But we want key=value strings
        #         for key, value in params.items():
        #             params[key] = str(value)
        #     else:
        #         params = dict(parse_qsl(body, keep_blank_values=True))
        #     # Should we return JSON?
        #     should_return_json = (
        #         request.headers.get(""accept"") == ""application/json""
        #         or request.args.get(""_json"")
        #         or params.get(""_json"")
        #     )
        #     if canned_query:
        #         params_for_query = MagicParameters(params, request, self.ds)
        #     else:
        #         params_for_query = params
        #     ok = None
        #     try:
        #         cursor = await self.ds.databases[database].execute_write(
        #             sql, params_for_query
        #         )
        #         message = metadata.get(
        #             ""on_success_message""
        #         ) or ""Query executed, {} row{} affected"".format(
        #             cursor.rowcount, """" if cursor.rowcount == 1 else ""s""
        #         )
        #         message_type = self.ds.INFO
        #         redirect_url = metadata.get(""on_success_redirect"")
        #         ok = True
        #     except Exception as e:
        #         message = metadata.get(""on_error_message"") or str(e)
        #         message_type = self.ds.ERROR
        #         redirect_url = metadata.get(""on_error_redirect"")
        #         ok = False
        #     if should_return_json:
        #         return Response.json(
        #             {
        #                 ""ok"": ok,
        #                 ""message"": message,
        #                 ""redirect"": redirect_url,
        #             }
        #         )
        #     else:
        #         self.ds.add_message(request, message, message_type)
        #         return self.redirect(request, redirect_url or request.path)
        # else:

        #     async def extra_template():
        #         return {
        #             ""request"": request,
        #             ""db_is_immutable"": not db.is_mutable,
        #             ""path_with_added_args"": path_with_added_args,
        #             ""path_with_removed_args"": path_with_removed_args,
        #             ""named_parameter_values"": named_parameter_values,
        #             ""canned_query"": canned_query,
        #             ""success_message"": request.args.get(""_success"") or """",
        #             ""canned_write"": True,
        #         }

        #     return (
        #         {
        #             ""database"": database,
        #             ""rows"": [],
        #             ""truncated"": False,
        #             ""columns"": [],
        #             ""query"": {""sql"": sql, ""params"": params},
        #             ""private"": private,
        #         },
        #         extra_template,
        #         templates,
        #     )

    # Not a write
    rows = []
    if canned_query:
        params_for_query = MagicParameters(params, request, datasette)
    else:
        params_for_query = params
    try:
        results = await datasette.execute(
            database, sql, params_for_query, truncate=True, **extra_args
        )
        columns = [r[0] for r in results.description]
        rows = list(results.rows)
    except sqlite3.DatabaseError as e:
        query_error = e
        results = None
        columns = []

    allow_execute_sql = await datasette.permission_allowed(
        request.actor, ""execute-sql"", database
    )

    format_ = request.url_vars.get(""format"") or ""html""

    if format_ == ""csv"":
        raise NotImplementedError(""CSV format not yet implemented"")
    elif format_ in datasette.renderers.keys():
        # Dispatch request to the correct output format renderer
        # (CSV is not handled here due to streaming)
        result = call_with_supported_arguments(
            datasette.renderers[format_][0],
            datasette=datasette,
            columns=columns,
            rows=rows,
            sql=sql,
            query_name=None,
            database=db.name,
            table=None,
            request=request,
            view_name=""table"",  # TODO: should this be ""query""?
            # These will be deprecated in Datasette 1.0:
            args=request.args,
            data={
                ""rows"": rows,
            },  # TODO what should this be?
        )
        result = await await_me_maybe(result)
        if result is None:
            raise NotFound(""No data"")
        if isinstance(result, dict):
            r = Response(
                body=result.get(""body""),
                status=result.get(""status_code"") or 200,
                content_type=result.get(""content_type"", ""text/plain""),
                headers=result.get(""headers""),
            )
        elif isinstance(result, Response):
            r = result
            # if status_code is not None:
            #     # Over-ride the status code
            #     r.status = status_code
        else:
            assert False, f""{result} should be dict or Response""
    elif format_ == ""html"":
        headers = {}
        templates = [f""query-{to_css_class(database)}.html"", ""query.html""]
        template = datasette.jinja_env.select_template(templates)
        alternate_url_json = datasette.absolute_url(
            request,
            datasette.urls.path(path_with_format(request=request, format=""json"")),
        )
        headers.update(
            {
                ""Link"": '{}; rel=""alternate""; type=""application/json+datasette""'.format(
                    alternate_url_json
                )
            }
        )
        r = Response.html(
            await datasette.render_template(
                template,
                dict(
                    data,
                    append_querystring=append_querystring,
                    path_with_replaced_args=path_with_replaced_args,
                    fix_path=datasette.urls.path,
                    settings=datasette.settings_dict(),
                    # TODO: review up all of these hacks:
                    alternate_url_json=alternate_url_json,
                    datasette_allow_facet=(
                        ""true"" if datasette.setting(""allow_facet"") else ""false""
                    ),
                    is_sortable=any(c[""sortable""] for c in data[""display_columns""]),
                    allow_execute_sql=await datasette.permission_allowed(
                        request.actor, ""execute-sql"", resolved.db.name
                    ),
                    query_ms=1.2,
                    select_templates=[
                        f""{'*' if template_name == template.name else ''}{template_name}""
                        for template_name in templates
                    ],
                ),
                request=request,
                view_name=""table"",
            ),
            headers=headers,
        )
    else:
        assert False, ""Invalid format: {}"".format(format_)
    # if next_url:
    #     r.headers[""link""] = f'<{next_url}>; rel=""next""'
    return r
```",107914493
issues,1821108702,I_kwDOCGYnMM5si-ne,Special handling for SQLite column of type `JSON`,15178711,2023-07-25T20:37:23Z,"`sqlite-utils` should detect and have specially handling for column with a `JSON` column. For example:

```sql
CREATE TABLE ""dogs"" (
  id INTEGER PRIMARY KEY,
  name TEXT,
  friends JSON 
);
```

## Automatic Nesting

According to [""Nested JSON Values""](https://sqlite-utils.datasette.io/en/stable/cli.html#nested-json-values), sqlite-utils will only expand JSON if the `--json-cols` flag is passed. It looks like it'll try to `json.load` all text column to test if its JSON, which can get expensive on non-json columns. 

Instead, `sqlite-utils` should be default (ie without the `--json-cols` flags) do the `maybe_json()` operation on columns with a declared `JSON` type. So the above table would expand the `""friends""` column as expected, withoutthe `--json-cols` flag:

```bash
sqlite-utils dogs.db ""select * from dogs"" | python -mjson.tool
```

```
[
    {
        ""id"": 1,
        ""name"": ""Cleo"",
        ""friends"": [
            {
                ""name"": ""Pancakes""
            },
            {
                ""name"": ""Bailey""
            }
        ]
    }
]
```

---

I'm sure there's other ways `sqlite-utils` can specially handle JSON columns, so keeping this open while I think of more",140912432
issue_comments,1649863533,IC_kwDOBm6k_c5iVutt,,22429695,2023-07-25T13:37:07Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`278ac91`)](https://app.codecov.io/gh/simonw/datasette/commit/278ac91a4d68da333a6ed74a2c75b3ec9db8bd19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`7fc79ba`)](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2107   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5805     5805           
=======================================
  Hits         5363     5363           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1649849249,IC_kwDOBm6k_c5iVrOh,,49699333,2023-07-25T13:28:35Z,Superseded by #2107.,107914493
issues,1820346348,PR_kwDOBm6k_c5WVYor,Bump sphinx from 6.1.3 to 7.1.0,49699333,2023-07-25T13:28:30Z,"Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 7.1.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>Sphinx 7.1.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0rc1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.1.0 (released Jul 24, 2023)</h1>
<h2>Incompatible changes</h2>
<ul>
<li>
<p>Releases are no longer signed, given the <code>change in PyPI policy</code>_.</p>
<p>.. _change in PyPI policy: <a href=""https://blog.pypi.org/posts/2023-05-23-removing-pgp/"">https://blog.pypi.org/posts/2023-05-23-removing-pgp/</a></p>
</li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11412"">#11412</a>: Emit warnings on using a deprecated Python-specific index entry type
(namely, <code>module</code>, <code>keyword</code>, <code>operator</code>, <code>object</code>, <code>exception</code>,
<code>statement</code>, and <code>builtin</code>) in the :rst:dir:<code>index</code> directive, and
set the removal version to Sphinx 9. Patch by Adam Turner.</li>
</ul>
<h2>Features added</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11415"">#11415</a>: Add a checksum to JavaScript and CSS asset URIs included within
generated HTML, using the CRC32 algorithm.</li>
<li>:meth:<code>~sphinx.application.Sphinx.require_sphinx</code> now allows the version
requirement to be specified as <code>(major, minor)</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11011"">#11011</a>: Allow configuring a line-length limit for object signatures, via
:confval:<code>maximum_signature_line_length</code> and the domain-specific variants.
If the length of the signature (in characters) is greater than the configured
limit, each parameter in the signature will be split to its own logical line.
This behaviour may also be controlled by options on object description
directives, for example :rst:dir:<code>py:function:single-line-parameter-list</code>.
Patch by Thomas Louf, Adam Turner, and Jean-François B.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10983"">#10983</a>: Support for multiline copyright statements in the footer block.
Patch by Stefanie Molin</li>
<li><code>sphinx.util.display.status_iterator</code> now clears the current line
with ANSI control codes, rather than overprinting with space characters.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11431"">#11431</a>: linkcheck: Treat SSL failures as broken links.
Patch by Bénédikt Tran</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11157"">#11157</a>: Keep the <code>translated</code> attribute on translated nodes.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11451"">#11451</a>: Improve the traceback displayed when using :option:<code>sphinx-build -T</code>
in parallel builds. Patch by Bénédikt Tran</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11324"">#11324</a>: linkcheck: Use session-basd HTTP requests.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11438"">#11438</a>: Add support for the :rst:dir:<code>py:class</code> and :rst:dir:<code>py:function</code>
directives for PEP 695 (generic classes and functions declarations) and
PEP 696 (default type parameters).  Multi-line support (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11011"">#11011</a>) is enabled
for type parameters list and can be locally controlled on object description
directives, e.g., :rst:dir:<code>py:function:single-line-type-parameter-list</code>.
Patch by Bénédikt Tran.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11484"">#11484</a>: linkcheck: Allow HTML anchors to be ignored on a per-URL basis
via :confval:<code>linkcheck_anchors_ignore_for_url</code> while</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e560f6334aa87af07304415e6aeaf42d03d1c525""><code>e560f63</code></a> Bump to 7.1.0 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/066e0fabc60c55072b3f75876637a44fdcb51332""><code>066e0fa</code></a> Add translation progress information (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11509"">#11509</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/0882914d5c5057ad3cec280c0142cfe5f6965844""><code>0882914</code></a> Target PyPI in create-release.yml</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/21fbee5e83a3e9d34e589d06c66d928f3a67923c""><code>21fbee5</code></a> Fix OIDC token payload</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/1a403e4f9ad4d8d7a3e4c7c34f55d9a45ed600bd""><code>1a403e4</code></a> Add informational log messaging</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/258b0ea9fcdd05221ebead5e5c162a7ad37c4412""><code>258b0ea</code></a> Revert &quot;Switch to using <code>github.request</code>&quot;</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/f9c89e5d8431155caa7be57d923f96004a2dd4bd""><code>f9c89e5</code></a> Switch to using <code>github.request</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/52c7f66ce172f723e8227896fe02165d288cb28f""><code>52c7f66</code></a> Use the correct token minting URL for TestPyPI</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/6079f28faa2a60d00f62b02786f23cd489019cdb""><code>6079f28</code></a> Install twine in PyPI publish workflow</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/3d43b9efdb49706cad6947f0c4d877d603781fe6""><code>3d43b9e</code></a> Fix github-script syntax in create-release.yml</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v7.1.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=6.1.3&new-version=7.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2107.org.readthedocs.build/en/2107/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1648339661,IC_kwDOCGYnMM5iP6rN,,25778,2023-07-24T17:44:30Z,"> A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV.

This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this:

```sh
sqlite-utils insert data.db table file.geojson --format geojson
# ... explore and manipulate in Datasette
sqlite-utils query data.db ... --format geojson > output.geojson
```

This would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files.
",140912432
issue_comments,1648325682,IC_kwDOCGYnMM5iP3Qy,,9599,2023-07-24T17:33:10Z,"A related feature would be support for plugins to add new ways of _ingesting_ data - currently `sqlite-utils insert` works against JSON, newline-JSON, CSV and TSV.",140912432
issue_comments,1648324312,IC_kwDOCGYnMM5iP27Y,,9599,2023-07-24T17:32:21Z,I'm currently leaning very slightly towards `--format geojson` - and explaining in the docs that `--format` is for formats added by plugins.,140912432
issue_comments,1648323482,IC_kwDOCGYnMM5iP2ua,,9599,2023-07-24T17:31:56Z,"The main blocker here is coming up with a design. The challenge is cleanly integrating it with the existing format options:

https://github.com/simonw/sqlite-utils/blob/8bee14588687b66c54c7a3dfae5de2f9cc7cac3f/docs/cli-reference.rst#L114-L132

I'd like to avoid breaking backwards compatibility.

The most obvious solution is to add a `--format x` option. This is slightly confusing as `--fmt` already exists.

Or... `--fmt` could be enhanced to handle plugins too - and the existing set of formats (from the `tabulate` integration) could be refactored to use the new hook.

If I ever do release `sqlite-utils 4` I'm tempted to move that tabulate stuff into a plugin.",140912432
issues,1818838294,I_kwDOCGYnMM5saUUW,Plugin hook for adding new output formats,9599,2023-07-24T17:29:18Z,"> What would it take to add a format hook? I'm still thinking about my GIS workflow, and being able to do `sqlite-utils query ... --geojson` would be nice. It's the one place my Datasette workflow is messy, having to do `datasette . --get /path/to/query.geojson --setting max_rows_returned 10000 --load-extension spatialite`.
> I know the current pattern is `--csv`, but maybe `--format geojson` is more future-proof.

https://discord.com/channels/823971286308356157/997738192360964156/1133076679011602432",140912432
issues,1817289521,I_kwDOCGYnMM5sUaMx,Get `add_foreign_keys()` to work without modifying `sqlite_master`,9599,2023-07-23T20:40:18Z,"https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1165-L1174

This is the only place in the code that attempts to modify `sqlite_master` directly, which fails on some Python installations.

Could this use the `.transform()` trick instead?

Or automatically switch to that trick if it hits an error?",140912432
issue_comments,1646950438,IC_kwDOC8SPRc5iKngm,,10352819,2023-07-23T20:18:26Z,"My bad, although I could not find how to use `url`s, I was using a function call syntax for the filters instead of jinja's pipe syntax; closing this issue.",197431109
issues,1817281557,I_kwDOC8SPRc5sUYQV,cannot use jinja filters in display?,10352819,2023-07-23T20:09:54Z,"Hi, I'm trying to have a display function in Dogsheep's `config.yml` that includes something like this:

```         
<h3>
  <a href=""{{ urls.row('my_database', 'my_table', key) }}"">{{ display.title }}</a>
  <a href=""{{ display.url }}🔗"" target=""_blank"">(source)</a>
</h3>
<p>{{ display.snippet|safe }}</p>
```

Unfortunately, rendering fails with a message 'urls is undefined'.

The same happens if I'm trying to build a row URL manually, using filters like `quote_plus` (as my keys are URLs).

Any hints?

Thanks!",197431109
issue_comments,1646753678,IC_kwDOCGYnMM5iJ3eO,,9599,2023-07-23T05:49:51Z,Done - bottom of https://sqlite-utils.datasette.io/en/latest/changelog.html#id116,140912432
issue_comments,1646752814,IC_kwDOCGYnMM5iJ3Qu,,9599,2023-07-23T05:43:13Z,"Using `llm` in my `sqlite-utils` local checkout:
```bash
git diff 0.2 0.3 | llm -s 'release notes as markdown bullet points'
```
> - Added a new `Table` class in the `db` module which represents a table in a SQLite database
> - The `Table` class has properties that provide information about the table, such as column names, foreign keys, and schema
> - The `Table` class also has a `count` property that returns the number of rows in the table
> - The `Table` class has a `create` method that can be used to create a new table with specified columns, primary key, and foreign keys
> - Updated the `Database` class in the `__init__` module to include the `Table` class and expose it via the `__all__` variable
> - Added new tests for creating tables and introspecting tables

```bash
git diff 0.3 0.3.1 | llm -s 'release notes as markdown bullet points'
```

> - Updated PyPI version badge to display the latest version
> - Updated Travis CI badge to link to the master branch
> - Added a badge for documentation status
> - Added a license badge
> - Added a description of the package
> - Added installation instructions
> - Added a link to the documentation
> - Added a ""Related projects"" section and linked to Datasette and csvs-to-sqlite
> - Updated the version in setup.py to 0.3.1

```bash
git diff 0.3.1 0.4 | llm -s 'release notes as markdown bullet points'
```

> - Added an `enable_fts` method to the `Table` class that enables full-text search on specified columns
> - Added a `populate_fts` method to the `Table` class that refreshes the search index for full-text search
> - Added a `search` method to the `Table` class that allows searching the table using full-text search
> - Updated the `setup.py` script to version 0.4
> - Updated the `db.py` file to include the `enable_fts`, `populate_fts`, and `search` methods
> - Added test cases for the `enable_fts`, `populate_fts`, and `search` methods in the `test_enable_fts.py` file
> - Added test cases for the `count`, `columns`, and `schema` properties in the `test_introspect.py` file
> ",140912432
issues,1816997390,I_kwDOCGYnMM5sTS4O,Backfill the release notes prior to 0.4,9599,2023-07-23T05:41:42Z,"Currently the changelog starts at 0.4:

https://sqlite-utils.datasette.io/en/3.34/changelog.html#id115

I want the other releases - according to https://pypi.org/project/sqlite-utils/#history there are three missing:

<img width=""663"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/4ebc036b-7bb1-477c-95c1-a2c7e26bcb62"">
",140912432
issue_comments,1646688748,IC_kwDOCGYnMM5iJnns,,9599,2023-07-22T23:12:03Z,"Just tried this out by adding the example from the docs to my `sqlite-utils-hello-world` plugin and running:
```bash
sqlite-utils install -e ../sqlite-utils-hello-world 
```
```
Obtaining file:///Users/simon/Dropbox/Development/sqlite-utils-hello-world
...
```
Then:
```bash
sqlite-utils memory ""select hello('simon')""
```
```json
[{""hello('simon')"": ""Hello, simon!""}]
```
Also:
```pycon
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> list(db.query(""select hello('simon')""))
[{""hello('simon')"": 'Hello, simon!'}]
>>> db2 = sqlite_utils.Database(memory=True, execute_plugins=False)
>>> list(db2.query(""select hello('simon')""))
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 494, in query
    cursor = self.execute(sql, params or tuple())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 512, in execute
    return self.conn.execute(sql, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlean.dbapi2.OperationalError: no such function: hello
```",140912432
issue_comments,1646688339,IC_kwDOCGYnMM5iJnhT,,9599,2023-07-22T23:08:49Z,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#prepare-connection-conn,140912432
issue_comments,1646688288,IC_kwDOCGYnMM5iJngg,,9599,2023-07-22T23:08:22Z,Documented here: https://sqlite-utils.datasette.io/en/latest/plugins.html,140912432
issue_comments,1646687461,IC_kwDOCGYnMM5iJnTl,,9599,2023-07-22T23:01:44Z,Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346,140912432
issues,1816919568,I_kwDOCGYnMM5sS_4Q,Python API ability to opt-out of connection plugins,9599,2023-07-22T23:01:13Z,"Plugins affecting the CLI by default makes sense to me.

I'm less confident about them _always_ affecting users of the Python API.

I'm going to have them apply by default, but I'm going to add a mechanism to opt-out on an individual database basis. Basically this:

```python
from sqlite_utils import Database
db = Database(memory=True, execute_plugins=False)
# Anything using db from here on will not execute plugins
```
cc @asg017 

Refs:
- #567 
- #574 ",140912432
issue_comments,1646687219,IC_kwDOCGYnMM5iJnPz,,9599,2023-07-22T22:59:36Z,Now that we have two plugin hooks I'm closing this issue (we can open other issues for further hooks).,140912432
issue_comments,1646687103,IC_kwDOCGYnMM5iJnN_,,9599,2023-07-22T22:58:35Z,"https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn

<img width=""773"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/3211a37a-c30d-47f9-bc10-c2c3240b6f6e"">

",140912432
issue_comments,1646686675,IC_kwDOCGYnMM5iJnHT,,9599,2023-07-22T22:54:38Z,"<img width=""779"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/c4e5769d-f1a3-427b-b260-882fce5c7205"">

Glitch in the rendered documentation from https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn",140912432
issue_comments,1646686477,IC_kwDOCGYnMM5iJnEN,,9599,2023-07-22T22:52:56Z,"Alex built this in:
- #573",140912432
issues,1816918185,I_kwDOCGYnMM5sS_ip,`prepare_connection()` plugin hook,9599,2023-07-22T22:52:47Z,"> Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424_
            ",140912432
issue_comments,1646686424,IC_kwDOCGYnMM5iJnDY,,9599,2023-07-22T22:52:34Z,Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!,140912432
issue_comments,1646686382,IC_kwDOCGYnMM5iJnCu,,22429695,2023-07-22T22:52:22Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00`**% and project coverage change: **`+0.03`** :tada:
> Comparison is base [(`86a352f`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/86a352f8b713ca30a65a2048170bd510d529d8c4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.09% compared to head [(`faf398f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.12%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #573      +/-   ##
==========================================
+ Coverage   96.09%   96.12%   +0.03%     
==========================================
  Files           8        8              
  Lines        2794     2816      +22     
==========================================
+ Hits         2685     2707      +22     
  Misses        109      109              
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.36% <100.00%> (+0.01%)` | :arrow_up: |
| [sqlite\_utils/hookspecs.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issue_comments,1646686332,IC_kwDOCGYnMM5iJnB8,,9599,2023-07-22T22:52:01Z,I was literally seconds away from shipping version 3.34 but I this looks good so I'm going to try and get it in there.,140912432
issues,1816917522,PR_kwDOCGYnMM5WJ6Jm,feat: Implement a prepare_connection plugin hook,15178711,2023-07-22T22:48:44Z,"Just like the [Datasette prepare_connection hook](https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette), this PR adds a similar hook for the `sqlite-utils` plugin system. 

The sole argument is `conn`, since I don't believe a `database` or `datasette` argument would be relevant here. 

I want to do this so I can release `sqlite-utils` plugins for my [SQLite extensions](https://github.com/asg017/sqlite-ecosystem), similar to the Datasette plugins I've release for them. 

An example plugin: https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c

```bash
$ sqlite-utils install https://gist.github.com/asg017/d7cdf0d56e2be87efda28cebee27fa3c/archive/5f5ad549a40860787629c69ca120a08c32519e99.zip

$ sqlite-utils memory 'select hello(""alex"") as response'
[{""response"": ""Hello, alex!""}]
```
Refs:
- #574 

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--573.org.readthedocs.build/en/573/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1646682686,IC_kwDOCGYnMM5iJmI-,,9599,2023-07-22T22:23:47Z,"Demo:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True)
>>> db[""foo""].insert({""name"": ""Cleo""})
<Table foo (name)>
>>> print(db[""foo""].schema)
CREATE TABLE [foo] (
   [name] TEXT
)
>>> db[""foo""].transform(not_null={""name""})
<Table foo (name)>
>>> print(db[""foo""].schema)
CREATE TABLE ""foo"" (
   [name] TEXT NOT NULL
)
>>> db[""foo""].transform(types={""name"": int}, keep_table=""kept"")
<Table foo (name)>
>>> print(db.schema)
CREATE TABLE ""kept"" (
   [name] TEXT NOT NULL
);
CREATE TABLE ""foo"" (
   [name] INTEGER NOT NULL
);
```",140912432
issue_comments,1646681386,IC_kwDOCGYnMM5iJl0q,,9599,2023-07-22T22:13:22Z,I'm happy with how this works on https://sqlite-utils.datasette.io/en/stable/cli-reference.html,140912432
issue_comments,1646681192,IC_kwDOCGYnMM5iJlxo,,9599,2023-07-22T22:12:08Z,Found it: https://github.com/simonw/sqlite-utils/blob/18f190e28334d821be78a1dbbf31d7610fc1f9c1/.github/workflows/test.yml#L29-L31,140912432
issue_comments,1646660777,IC_kwDOCGYnMM5iJgyp,,9599,2023-07-22T19:59:13Z,I don't know where that is coming from! I can't see `textual` as a dependency we are pulling in anywhere.,140912432
issues,1816877910,I_kwDOCGYnMM5sS1tW,Don't test Python 3.7 against textual,9599,2023-07-22T19:57:03Z,"Spotted this in the GitHub Actions logs:

![IMG_5046](https://github.com/simonw/sqlite-utils/assets/9599/81fb1093-cd8a-4019-a612-2e49b500c933)
",140912432
issue_comments,1646659809,IC_kwDOCGYnMM5iJgjh,,9599,2023-07-22T19:53:56Z,"CLI documentation:

- https://sqlite-utils.datasette.io/en/latest/cli.html#renaming-a-table
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#rename-table",140912432
issues,1816876211,I_kwDOCGYnMM5sS1Sz,`.transform(keep_table=...)` option,9599,2023-07-22T19:49:29Z,">> Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.
>
> I think `keep_table=""name_of_table""` is good for this.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324_
            ",140912432
issue_comments,1646658978,IC_kwDOCGYnMM5iJgWi,,9599,2023-07-22T19:48:54Z,"Python method documentation:
- https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.rename_table
- https://sqlite-utils.datasette.io/en/latest/python-api.html#renaming-a-table",140912432
issue_comments,1646657849,IC_kwDOCGYnMM5iJgE5,,9599,2023-07-22T19:42:02Z,"Manually testing new `rename-table` command:
```
$ sqlite-utils schema /tmp/f.db 
CREATE TABLE [foo] (
   [id] INTEGER,
   [name] TEXT
);
$ sqlite-utils rename-table /tmp/f.db bad-table hi
Error: Table ""bad-table"" could not be renamed. no such table: bad-table
$ sqlite-utils rename-table /tmp/f.db foo foo     
Error: Table ""foo"" could not be renamed. there is already another table or index with this name: foo
$ sqlite-utils rename-table /tmp/f.db foo bar
$ sqlite-utils schema /tmp/f.db                   
CREATE TABLE ""bar"" (
   [id] INTEGER,
   [name] TEXT
);
```",140912432
issue_comments,1646657324,IC_kwDOCGYnMM5iJf8s,,9599,2023-07-22T19:39:06Z,"> Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.

I think `keep_table=""name_of_table""` is good for this.",140912432
issue_comments,1646656283,IC_kwDOCGYnMM5iJfsb,,25778,2023-07-22T19:32:24Z,Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. ,140912432
issue_comments,1646656246,IC_kwDOCGYnMM5iJfr2,,9599,2023-07-22T19:32:07Z,"Trying out a simple first implementation:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True, tracer=print)
PRAGMA recursive_triggers=on; None
>>> db[""foo""].insert({""id"": 1})
select name from sqlite_master where type = 'view' None
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'view' None
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'view' None
CREATE TABLE [foo] (
   [id] INTEGER
);
         None
select name from sqlite_master where type = 'view' None
INSERT INTO [foo] ([id]) VALUES (?); [1]
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'table' None
PRAGMA table_info([foo]) None
<Table foo (id)>
>>> db.rename_table(""foo"", ""baz"")
ALTER TABLE [foo] RENAME TO [baz] None
>>> print(db.schema)
select sql from sqlite_master where sql is not null None
CREATE TABLE ""baz"" (
   [id] INTEGER
);
```",140912432
issue_comments,1646655272,IC_kwDOCGYnMM5iJfco,,9599,2023-07-22T19:25:35Z,"Here's why that test broke: https://github.com/simonw/sqlite-utils/blob/58b577279fcd5ef6ce88f88b28668dffebfe7f44/sqlite_utils/db.py#L960-L964

I added an extra `if self[name].exists()` check to the `db.create_table()` method.",140912432
issue_comments,1646654818,IC_kwDOCGYnMM5iJfVi,,9599,2023-07-22T19:22:40Z,"I think this broke a test:
```
$ pytest tests/test_tracer.py                           
=============================================== test session starts ================================================
platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/simon/Dropbox/Development/sqlite-utils
plugins: icdiff-0.6, hypothesis-6.68.2
collected 2 items                                                                                                  

tests/test_tracer.py F.                                                                                      [100%]

===================================================== FAILURES =====================================================
___________________________________________________ test_tracer ____________________________________________________

    def test_tracer():
        collected = []
        db = Database(
            memory=True, tracer=lambda sql, params: collected.append((sql, params))
        )
        db[""dogs""].insert({""name"": ""Cleopaws""})
        db[""dogs""].enable_fts([""name""])
        db[""dogs""].search(""Cleopaws"")
>       assert collected == [
            (""PRAGMA recursive_triggers=on;"", None),
            (""select name from sqlite_master where type = 'view'"", None),
            (""select name from sqlite_master where type = 'table'"", None),
            (""select name from sqlite_master where type = 'view'"", None),
            (""CREATE TABLE [dogs] (\n   [name] TEXT\n);\n        "", None),
            (""select name from sqlite_master where type = 'view'"", None),
            (""INSERT INTO [dogs] ([name]) VALUES (?);"", [""Cleopaws""]),
            (""select name from sqlite_master where type = 'view'"", None),
            (
                ""CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n    [name],\n    content=[dogs]\n)"",
                None,
            ),
            (
                ""INSERT INTO [dogs_fts] (rowid, [name])\n    SELECT rowid, [name] FROM [dogs];"",
                None,
            ),
            (""select name from sqlite_master where type = 'view'"", None),
        ]
E       assert equals failed
E         [                                                  [                                                 
E           ('PRAGMA recursive_triggers=on;', None),           ('PRAGMA recursive_triggers=on;', None),        
E           (                                                                                                  
E             ""select name from sqlite_master where type =                                                     
E         'view'"",                                                                                             
E             None,                                                       ...
E         
E         ...Full output truncated (13 lines hidden), use '-vv' to show

tests/test_tracer.py:12: AssertionError
============================================= short test summary info ==============================================
FAILED tests/test_tracer.py::test_tracer - assert equals failed
=========================================== 1 failed, 1 passed in 0.05s ============================================
```",140912432
issue_comments,1646654383,IC_kwDOCGYnMM5iJfOv,,9599,2023-07-22T19:20:16Z,"Here's documentation for the new plugins mechanism, including a very short tutorial on writing a new plugin (inspired by https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html):

https://sqlite-utils.datasette.io/en/latest/plugins.html",140912432
issue_comments,1646654275,IC_kwDOCGYnMM5iJfND,,9599,2023-07-22T19:19:35Z,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#register-commands-cli,140912432
issue_comments,1646653610,IC_kwDOCGYnMM5iJfCq,,9599,2023-07-22T19:14:56Z,"Manual testing of CLI command as well:
```
$ sqlite-utils create-table /tmp/f.db foo id integer                         
$ sqlite-utils create-table /tmp/f.db foo id integer
Error: Table ""foo"" already exists. Use --replace to delete and replace it.
$ sqlite-utils create-table /tmp/f.db foo id integer --replace
$ sqlite-utils create-table /tmp/f.db foo id                  
$ sqlite-utils schema /tmp/f.db
CREATE TABLE [foo] (
   [id] INTEGER
);
$ sqlite-utils create-table /tmp/f.db foo id integer name str --transform
Error: column types must be one of ('INTEGER', 'TEXT', 'FLOAT', 'BLOB')
$ sqlite-utils create-table /tmp/f.db foo id integer name text --transform
$ sqlite-utils schema /tmp/f.db                                          
CREATE TABLE ""foo"" (
   [id] INTEGER,
   [name] TEXT
);
$ sqlite-utils create-table /tmp/f.db foo id integer name text --ignore   
$ sqlite-utils create-table /tmp/f.db foo id integer name text --replace
$ sqlite-utils schema /tmp/f.db                                        
CREATE TABLE [foo] (
   [id] INTEGER,
   [name] TEXT
);                                                                    
```",140912432
issue_comments,1646653382,IC_kwDOCGYnMM5iJe_G,,9599,2023-07-22T19:13:20Z,"Demo:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True)
>>> db[""foo""].create({""id"": int})
<Table foo (id)>
>>> db[""foo""].create({""id"": int})
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1647, in create
    self.db.create_table(
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1030, in create_table
    self.execute(sql)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 510, in execute
    return self.conn.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^
sqlean.dbapi2.OperationalError: table [foo] already exists
>>> db[""foo""].create({""id"": int}, ignore=True)
<Table foo (id)>
>>> db[""foo""].create({""id"": int, ""name"": str}, replace=True)
<Table foo (id, name)>
>>> db[""foo""].create({""id"": int, ""name"": str, ""age"": int}, transform=True)
<Table foo (id, name, age)>
```",140912432
issue_comments,1646652105,IC_kwDOCGYnMM5iJerJ,,9599,2023-07-22T19:05:13Z,I think this is `replace=True` and `ignore=True` to match the CLI. And refactoring the CLI to use them.,140912432
issue_comments,1646648262,IC_kwDOBm6k_c5iJdvG,,9599,2023-07-22T18:42:20Z,"Tested this locally with:
```bash
datasette install -e ../datasette-graphql
```
Running `datasette plugins` confirmed that the plugin had been installed.",107914493
issue_comments,1646646931,IC_kwDOBm6k_c5iJdaT,,9599,2023-07-22T18:34:04Z,Here's the diff for adding it to `sqlite-utils`: https://github.com/simonw/sqlite-utils/commit/ef31210bf06f920e0890e171c3198f0b0dc8d72d,107914493
issues,1816857442,I_kwDOBm6k_c5sSwti,`datasette install -e` option,9599,2023-07-22T18:33:42Z,"As seen in LLM and now in `sqlite-utils` too:
- https://github.com/simonw/sqlite-utils/issues/570

Useful for developing plugins, see tutorial at https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html",107914493
issues,1816857105,I_kwDOCGYnMM5sSwoR,`sqlite-utils install -e` option,9599,2023-07-22T18:32:23Z,"As seen in LLM.

Needed while working on:
- #567",140912432
issue_comments,1646645990,IC_kwDOCGYnMM5iJdLm,,9599,2023-07-22T18:29:40Z,"Example plugin: https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf

```bash
sqlite-utils install https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf/archive/cd7960e476c441a3c5f619e2a44a641a39b91467.zip
```
`pyproject.toml`:
```toml
[project]
name = ""sqlite-utils-hello-world""
version = ""0.1""

[project.entry-points.sqlite_utils]
hello_world = ""sqlite_utils_hello_world""
```
`sqlite_utils_hello_world.py`:
```python
import click
import sqlite_utils


@sqlite_utils.hookimpl
def register_commands(cli):
    @cli.command()
    def hello_world():
        ""Say hello world""
        click.echo(""Hello world!"")
```",140912432
issue_comments,1646643676,IC_kwDOCGYnMM5iJcnc,,9599,2023-07-22T18:18:24Z,"Here's where I added that to LLM: https://github.com/simonw/llm/commit/a396950f7934e82a9968703bb3ce9ab7ab62f7f8

- https://github.com/simonw/llm/issues/49",140912432
issues,1816852402,I_kwDOCGYnMM5sSvey,register_command plugin hook,9599,2023-07-22T18:17:27Z,"> I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.

_Originally posted by @simonw in https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450_
            ",140912432
issue_comments,1646643450,IC_kwDOCGYnMM5iJcj6,,9599,2023-07-22T18:17:18Z,I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.,140912432
issue_comments,1646643379,IC_kwDOCGYnMM5iJciz,,9599,2023-07-22T18:16:54Z,"> Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?

Yeah, it could do! That's not a terrible idea to be honest, those things have really been proliferating.

Alternatively, they could each register themselves as plugins in addition - so if you install e.g. `pocket-to-sqlite` you could then optionally also run it as `sqlite-utils pocket-to-sqlite ...`

The benefit there is for people who install `sqlite-utils` from Homebrew, where it gets its own virtual environment. They could run:

```bash
brew install sqlite-utils
sqlite-utils install pocket-to-sqlite

sqlite-utils pocket-to-sqlite ...
```",140912432
issue_comments,1646642959,IC_kwDOCGYnMM5iJccP,,9599,2023-07-22T18:14:49Z,Here's where those are implemented for the `create-table` CLI command: https://github.com/simonw/sqlite-utils/blob/f7af23837deab5c98dae9441d1f68318065d7d8c/sqlite_utils/cli.py#L1543-L1564,140912432
issue_comments,1646642666,IC_kwDOCGYnMM5iJcXq,,9599,2023-07-22T18:13:19Z,"https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

```bash
sqlite-utils create-table ... --replace
```
That also has `--ignore`:

```
  --ignore                  If table already exists, do nothing
  --replace                 If table already exists, replace it
  --transform               If table already exists, try to transform the schema
```",140912432
issues,1816851056,I_kwDOCGYnMM5sSvJw,"table.create(..., replace=True)",9599,2023-07-22T18:12:22Z,"Found myself using this pattern to quickly prototype a schema:

```python
import sqlite_utils
db = sqlite_utils.Database(memory=True)

print(db[""answers_chunks""].create({
    ""id"": int,
    ""content"": str,
    ""embedding_type_id"": int,
    ""embedding"": bytes,
    ""embedding_content_md5"": str,
    ""source"": str,
}, pk=""id"", transform=True).schema)
```

<img width=""472"" alt=""image"" src=""https://github.com/simonw/datasette-answers/assets/9599/30237f7c-4964-4190-984c-d824c57939a9"">

Using `replace=True` to drop and then recreate the table would be neat here, and would be consistent with other places that use `replace=True`.",140912432
issues,1816830546,I_kwDODEm0Qs5sSqJS,Twitter v1 API shutdown,6341745,2023-07-22T16:57:41Z,"I've been using this project reliably over the past two years to periodically download my liked tweets, but unfortunately since 19th July I get:

```
[2023-07-19 21:00:04.937536]   File ""/home/pi/code/liked-tweets/lib/python3.7/site-packages/twitter_to_sqlite/utils.py"", line 202, in fetch_timeline
[2023-07-19 21:00:04.937606]     raise Exception(str(tweets[""errors""]))
[2023-07-19 21:00:04.937678] Exception: [{'message': 'You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: https://developer.twitter.com/en/portal/product', 'code': 453}]
```

It appears like Twitter has now shut down their v1 endpoints, which is rather gracious of them, considering they [announced they'd be deprecated on 29th April](https://twittercommunity.com/t/reminder-to-migrate-to-the-new-free-basic-or-enterprise-plans-of-the-twitter-api/189737).

Unfortunately [retrieving likes using the v2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/likes/introduction) is not part of their [free plan](https://developer.twitter.com/en/portal/products). In fact, with the free plan one can only post and delete tweets and retrieve information about oneself.

So I'm afraid this is the end of this very nice project. It was very useful, thank you!
",206156866
issue_comments,1646250291,IC_kwDOBm6k_c5iH8kz,,22429695,2023-07-21T21:21:00Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`3a51ca9`)](https://app.codecov.io/gh/simonw/datasette/commit/3a51ca901427ac93b0ed0638fd1f82fb647a299a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39% compared to head [(`3d45262`)](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2098   +/-   ##
=======================================
  Coverage   92.39%   92.39%           
=======================================
  Files          39       39           
  Lines        5802     5802           
=======================================
  Hits         5361     5361           
  Misses        441      441           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1646247246,IC_kwDOBm6k_c5iH71O,,2235371,2023-07-21T21:16:37Z,"I must be doing something wrong. On page https://192.168.1.3:5432/datasette-llm/logs/_llm_migrations there's the option to filter the results, there's an Apply button. It also tries to load an URL with an extra URL element in it:

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs/_llm_migrations?_sort=name&name__contains=initial",107914493
issue_comments,1643873232,IC_kwDOBm6k_c5h-4PQ,,2235371,2023-07-20T12:53:00Z,"I forgot to add that I followed these instructions to set up the python llm:

https://simonwillison.net/2023/Jul/18/accessing-llama-2/",107914493
issue_comments,1642808866,IC_kwDOCGYnMM5h60Yi,,25778,2023-07-19T21:54:27Z,Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?,140912432
issue_comments,1642013043,IC_kwDOBm6k_c5h3yFz,,2235371,2023-07-19T12:41:36Z,"The same problem can be seen in the links:

Advanced export

JSON shape: default, array, newline-delimited, object",107914493
issues,1811824307,I_kwDOBm6k_c5r_j6z,When reverse proxying datasette with nginx an URL element gets erronously added,2235371,2023-07-19T12:16:53Z,"I use this nginx config:
```
    location /datasette-llm {
      return 302 /datasette-llm/;
    }

    location /datasette-llm/ {
      proxy_set_header Upgrade           $http_upgrade;
      proxy_set_header Connection        ""Upgrade"";
      proxy_http_version 1.1;
      proxy_set_header X-Real-IP         $remote_addr;
      proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto https;
      proxy_set_header X-Forwarded-Host  $http_host;
      proxy_set_header Host              $host;
      proxy_max_temp_file_size           0;
      proxy_pass                         http://127.0.0.1:8001/datasette-llm/;
      proxy_redirect                     http:// https://;
      proxy_buffering off;
      proxy_request_buffering off;
      proxy_set_header Origin            '';
      client_max_body_size 0;
      auth_basic                         ""datasette-llm"";
      auth_basic_user_file               /etc/nginx/custom-userdb;
    }
```
Then I start datasette with this command:
```
datasette serve --setting base_url /datasette-llm/ $(llm logs path)
```
Everything else works right, except the links in ""This data as json, CSV"".
They get an extra URL element ""datasette-llm"" like this:

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs.json?sql=select+*+from+_llm_migrations

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs.csv?sql=select+*+from+_llm_migrations&_size=max

When I remove that extra ""datasette-llm"" from the URL, those links work too.",107914493
issue_comments,1641082395,IC_kwDOBm6k_c5h0O4b,,15178711,2023-07-18T22:41:37Z,"For filtering virtual table's ""shadow tables"" (ex the FTS5 _content and most the spatialite tables),  you can use `pragma_table_list` (first appeared in SQLite 3.37 (2021-11-27), which has a `type` column that calls out `type=""shadow""` tables https://www.sqlite.org/pragma.html#pragma_table_list",107914493
issue_comments,1640826795,IC_kwDOCGYnMM5hzQer,,76528036,2023-07-18T19:08:50Z,"Came here to report this, but instead I'll confirm the issue across two terminal emulators (Gnome Terminal and Alacritty) on Pop_OS! 22.04 (currently based on Ubuntu/Gnome). Also messes up the formatting of the terminal. Can also confirm that reset fixes it until the next sqlite-utils command. ",140912432
issue_comments,1640064620,IC_kwDOBm6k_c5hwWZs,,9599,2023-07-18T11:47:21Z,"I think I've figured out the problem here.

The question being asked is ""can this actor access this resource, which is within this database within this instance"".

The answer to this question needs to consider the full set of questions at once - yes they can access within this instance IF they have access to the specified table and that's the table being asked about.

But the questions are currently being asked independently, which means the plugin hook acting on `view-instance` can't see that the answer here should be yes because it's actually about a table that the actor has explicit permission to view.

So I think I may need to redesign the plugin hook to always see the full hierarchy of checks, not just a single check at a time.
",107914493
issue_comments,1638926655,IC_kwDOCGYnMM5hsAk_,,9599,2023-07-17T21:42:37Z,"I really like this. I'm also interested in:
- Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette
- Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes

The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ",140912432
issue_comments,1638910473,IC_kwDOCGYnMM5hr8oJ,,15178711,2023-07-17T21:27:41Z,"Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. 

```bash
sqlite-utils insert-datasette \
  --token $DATASETTE_API_KEY \
  https://latest.datasette.io/fixtures/my-table \
  'select ...'
```


This could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)",140912432
issue_comments,1638567228,IC_kwDOBm6k_c5hqo08,,9599,2023-07-17T17:24:19Z,"Confirmed that this is an issue with regular Datasette signed tokens as well. I created one on https://latest.datasette.io/-/create-token with these details:
```json
{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""sortable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""root"",
    ""d"": 3600,
    ""t"": 1689614483
}
```
Run like this:
```
curl -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsInQiOjE2ODk2MTQ0ODMsImQiOjM2MDAsIl9yIjp7InIiOnsiZml4dHVyZXMiOnsic29ydGFibGUiOlsidnQiXX19fX0.n-VGxxawz1Q0WK7sqLfhXUgcvY0' \
  https://latest.datasette.io/fixtures/sortable.json
```
Returned an HTML Forbidden page:
```html
<!DOCTYPE html>
<html>
<head>
    <title>Forbidden</title>
    <link rel=""stylesheet"" href=""/-/static/app.css?d59929"">
...
```
Same token againts `/-/actor.json` returns:
```json
{
  ""actor"": {
    ""id"": ""root"",
    ""token"": ""dstok"",
    ""_r"": {
      ""r"": {
        ""fixtures"": {
          ""sortable"": [
            ""vt""
          ]
        }
      }
    },
    ""token_expires"": 1689618083
  }
}
```
Reminder - `""_r""` means restrict, `""r""` means resource.",107914493
issue_comments,1638552567,IC_kwDOBm6k_c5hqlP3,,9599,2023-07-17T17:14:20Z,Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/database.py#L391-L451,107914493
issues,1808215339,I_kwDOBm6k_c5rxy0r,Tables starting with an underscore should be treated as hidden,9599,2023-07-17T17:13:53Z,"Plugins can then take advantage of this pattern, for example:
- https://github.com/simonw/datasette-auth-tokens/pull/8",107914493
issues,1808116827,I_kwDOBm6k_c5rxaxb,data attribute on Datasette tables exposing the primary key of the row,9599,2023-07-17T16:18:25Z,Maybe put it on the `<tr>` but probably better to go on the `td.type-pk`.,107914493
issue_comments,1637293044,IC_kwDOBm6k_c5hlxv0,,4863782,2023-07-17T02:23:32Z,Is there any working version of datasette/postgresql ?,107914493
issue_comments,1636134091,IC_kwDOBm6k_c5hhWzL,,653549,2023-07-14T17:02:03Z,"@asg017  - the docs say that the autodetection only occurs in configuration directory mode. I for one would also be interested in the `--settings settings.json` feature. 

For context, I am developing a large database for use with Datasette, but the database lives in a different network volume than my source code, since the volume in which my source code lives is aggressively backed up, while the location where the database lives is meant for temporary files and is not as aggressively backed up (since the backups would get unreasonably large).",107914493
issue_comments,1636093730,IC_kwDOBm6k_c5hhM8i,,9599,2023-07-14T16:26:27Z,"Here's that crucial comment:

>  If _r is defined then we use those to further restrict the actor.
>
>Crucially, we only use this to say NO (return False) - we never use it to return YES (True) because that might over-ride other restrictions placed on this actor

So that's why I implemented it like this.

The goal here is to be able to issue a token which can't do anything _more_ than the actor it is associated with, but CAN be configured to do less.

So I think the solution here is for the `_r` checking code to perhaps implement its own view cascade logic - it notices if you have `view-table` and consequently fails to block `view-table` and `view-instance`.

I'm not sure that's going to work though - would that mean that granting `view-table` grants `view-database` in a surprising and harmful way?

Maybe that's OK: if you have `view-database` but permission checks fail for individual tables and queries you shouldn't be able to see a thing that you shouldn't. Need to verify that though.

Also, do `Permission` instances have enough information to implement this kind of cascade without hard-coding anything? 
",107914493
issue_comments,1636053060,IC_kwDOBm6k_c5hhDBE,,9599,2023-07-14T15:51:36Z,"This might only be an issue with the code that checks `_r` on actors.

https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/default_permissions.py#L185-L222

Added in https://github.com/simonw/datasette/commit/bcc781f4c50a8870e3389c4e60acb625c34b0317 - refs:

- #1855 ",107914493
issue_comments,1636042066,IC_kwDOBm6k_c5hhAVS,,9599,2023-07-14T15:41:54Z,"I tried some code spelunking and came across https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7 which says:

> - If you have table permission but not database permission you can now view the table page

Refs:
- #832 

Which suggests that my initial design decision wasn't what appears to be implemented today.

Needs more investigation.",107914493
issue_comments,1636040164,IC_kwDOBm6k_c5hg_3k,,9599,2023-07-14T15:40:21Z,"Relevant code: 
https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/app.py#L822-L855",107914493
issue_comments,1636036312,IC_kwDOBm6k_c5hg-7Y,,9599,2023-07-14T15:37:14Z,"I think I made this decision because I was thinking about default deny: obviously if a user has been denied access to a database. It doesn't make sense that they could access tables within it.

But now that I am spending more time with authentication tokens, which default to denying everything, except for the things that you have explicitly listed, this policy, no longer makes as much sense.

",107914493
issues,1805076818,I_kwDOBm6k_c5rl0lS,API tokens with view-table but not view-database/view-instance cannot access the table,9599,2023-07-14T15:34:27Z,"> Spotted a problem while working on this: if you grant a token access to view table for a specific table but don't also grant view database and view instance permissions, that token is useless.
>
> This was a deliberate design decision in Datasette - it's documented on https://docs.datasette.io/en/1.0a2/authentication.html#access-permissions-in-metadata
>
>> If a user cannot access a specific database, they will not be able to access tables, views or queries within that database. If a user cannot access the instance they will not be able to access any of the databases, tables, views or queries.
>
> I'm now second-guessing if this was a good decision.

_Originally posted by @simonw in https://github.com/simonw/datasette-auth-tokens/issues/7#issuecomment-1636031702_
            ",107914493
issue_comments,1634443907,IC_kwDOBm6k_c5ha6KD,,9599,2023-07-13T15:24:17Z,https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/views/table.py#L486-L506,107914493
issues,1803264272,I_kwDOBm6k_c5re6EQ,alter: true support for JSON write API,9599,2023-07-13T15:24:11Z,"Requested here: https://discord.com/channels/823971286308356157/823971286941302908/1129034187073134642

> The former datasette-insert plugin had an option `?alter=1` to auto-add new columns. Does the JSON write API also have this?",107914493
issues,1802613340,PR_kwDOBm6k_c5VZhfw,Make primary key view accessible to render_cell hook,1563881,2023-07-13T09:30:36Z,"

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2100.org.readthedocs.build/en/2100/

<!-- readthedocs-preview datasette end -->",107914493
issues,1801394744,I_kwDOCGYnMM5rXxo4,Plugin system,15178711,2023-07-12T17:02:14Z,"I'd like there to be a plugin system for sqlite-utils, similar to the datasette/llm plugins. I'd like to make plugins that would do things like:

- Register SQLite extensions for more SQL functions + virtual tables
- Register new subcommands
- Different input file formats for `sqlite-utils memory`
- Different output file formats (in addition to `--csv` `--tsv` `--nl` etc.

A few real-world use-cases of plugins I'd like to see in sqlite-utils:

- Register many of my sqlite extensions in sqlite-utils (`sqlite-http`, `sqlite-lines`, `sqlite-regex`, etc.)
- New subcommands to work with `sqlite-vss` vector tables
- Input/ouput Parquet/Avro/Arrow IPC files with `sqlite-arrow`",140912432
issue_comments,1632867333,IC_kwDOBm6k_c5hU5QF,,22429695,2023-07-12T16:38:27Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified lines are covered by tests :white_check_mark:
> Comparison is base [(`3feed1f`)](https://app.codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`8ae479c`)](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.
> Report is 112 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2052      +/-   ##
==========================================
+ Coverage   92.46%   92.69%   +0.22%     
==========================================
  Files          38       40       +2     
  Lines        5750     6047     +297     
==========================================
+ Hits         5317     5605     +288     
- Misses        433      442       +9     
```


[see 19 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2052/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1798901709,PR_kwDOBm6k_c5VM2MK,Bump black from 23.3.0 to 23.7.0,49699333,2023-07-11T13:05:53Z,"Bumps [black](https://github.com/psf/black) from 23.3.0 to 23.7.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/releases"">black's releases</a>.</em></p>
<blockquote>
<h2>23.7.0</h2>
<h3>Highlights</h3>
<ul>
<li>Runtime support for Python 3.7 has been removed. Formatting 3.7 code will still be
supported until further notice (<a href=""https://redirect.github.com/psf/black/issues/3765"">#3765</a>)</li>
</ul>
<h3>Stable style</h3>
<ul>
<li>Fix a bug where an illegal trailing comma was added to return type annotations using
PEP 604 unions (<a href=""https://redirect.github.com/psf/black/issues/3735"">#3735</a>)</li>
<li>Fix several bugs and crashes where comments in stub files were removed or mishandled
under some circumstances (<a href=""https://redirect.github.com/psf/black/issues/3745"">#3745</a>)</li>
<li>Fix a crash with multi-line magic comments like <code>type: ignore</code> within parentheses
(<a href=""https://redirect.github.com/psf/black/issues/3740"">#3740</a>)</li>
<li>Fix error in AST validation when <em>Black</em> removes trailing whitespace in a type comment
(<a href=""https://redirect.github.com/psf/black/issues/3773"">#3773</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Implicitly concatenated strings used as function args are no longer wrapped inside
parentheses (<a href=""https://redirect.github.com/psf/black/issues/3640"">#3640</a>)</li>
<li>Remove blank lines between a class definition and its docstring (<a href=""https://redirect.github.com/psf/black/issues/3692"">#3692</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>The <code>--workers</code> argument to <em>Black</em> can now be specified via the <code>BLACK_NUM_WORKERS</code>
environment variable (<a href=""https://redirect.github.com/psf/black/issues/3743"">#3743</a>)</li>
<li><code>.pytest_cache</code>, <code>.ruff_cache</code> and <code>.vscode</code> are now excluded by default (<a href=""https://redirect.github.com/psf/black/issues/3691"">#3691</a>)</li>
<li>Fix <em>Black</em> not honouring <code>pyproject.toml</code> settings when running <code>--stdin-filename</code>
and the <code>pyproject.toml</code> found isn't in the current working directory (<a href=""https://redirect.github.com/psf/black/issues/3719"">#3719</a>)</li>
<li><em>Black</em> will now error if <code>exclude</code> and <code>extend-exclude</code> have invalid data types in
<code>pyproject.toml</code>, instead of silently doing the wrong thing (<a href=""https://redirect.github.com/psf/black/issues/3764"">#3764</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Upgrade mypyc from 0.991 to 1.3 (<a href=""https://redirect.github.com/psf/black/issues/3697"">#3697</a>)</li>
<li>Remove patching of Click that mitigated errors on Python 3.6 with <code>LANG=C</code> (<a href=""https://redirect.github.com/psf/black/issues/3768"">#3768</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Add support for the new PEP 695 syntax in Python 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3703"">#3703</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Speed up <em>Black</em> significantly when the cache is full (<a href=""https://redirect.github.com/psf/black/issues/3751"">#3751</a>)</li>
<li>Avoid importing <code>IPython</code> in a case where we wouldn't need it (<a href=""https://redirect.github.com/psf/black/issues/3748"">#3748</a>)</li>
</ul>
<h3>Output</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/psf/black/blob/main/CHANGES.md"">black's changelog</a>.</em></p>
<blockquote>
<h2>23.7.0</h2>
<h3>Highlights</h3>
<ul>
<li>Runtime support for Python 3.7 has been removed. Formatting 3.7 code will still be
supported until further notice (<a href=""https://redirect.github.com/psf/black/issues/3765"">#3765</a>)</li>
</ul>
<h3>Stable style</h3>
<ul>
<li>Fix a bug where an illegal trailing comma was added to return type annotations using
PEP 604 unions (<a href=""https://redirect.github.com/psf/black/issues/3735"">#3735</a>)</li>
<li>Fix several bugs and crashes where comments in stub files were removed or mishandled
under some circumstances (<a href=""https://redirect.github.com/psf/black/issues/3745"">#3745</a>)</li>
<li>Fix a crash with multi-line magic comments like <code>type: ignore</code> within parentheses
(<a href=""https://redirect.github.com/psf/black/issues/3740"">#3740</a>)</li>
<li>Fix error in AST validation when <em>Black</em> removes trailing whitespace in a type comment
(<a href=""https://redirect.github.com/psf/black/issues/3773"">#3773</a>)</li>
</ul>
<h3>Preview style</h3>
<ul>
<li>Implicitly concatenated strings used as function args are no longer wrapped inside
parentheses (<a href=""https://redirect.github.com/psf/black/issues/3640"">#3640</a>)</li>
<li>Remove blank lines between a class definition and its docstring (<a href=""https://redirect.github.com/psf/black/issues/3692"">#3692</a>)</li>
</ul>
<h3>Configuration</h3>
<ul>
<li>The <code>--workers</code> argument to <em>Black</em> can now be specified via the <code>BLACK_NUM_WORKERS</code>
environment variable (<a href=""https://redirect.github.com/psf/black/issues/3743"">#3743</a>)</li>
<li><code>.pytest_cache</code>, <code>.ruff_cache</code> and <code>.vscode</code> are now excluded by default (<a href=""https://redirect.github.com/psf/black/issues/3691"">#3691</a>)</li>
<li>Fix <em>Black</em> not honouring <code>pyproject.toml</code> settings when running <code>--stdin-filename</code>
and the <code>pyproject.toml</code> found isn't in the current working directory (<a href=""https://redirect.github.com/psf/black/issues/3719"">#3719</a>)</li>
<li><em>Black</em> will now error if <code>exclude</code> and <code>extend-exclude</code> have invalid data types in
<code>pyproject.toml</code>, instead of silently doing the wrong thing (<a href=""https://redirect.github.com/psf/black/issues/3764"">#3764</a>)</li>
</ul>
<h3>Packaging</h3>
<ul>
<li>Upgrade mypyc from 0.991 to 1.3 (<a href=""https://redirect.github.com/psf/black/issues/3697"">#3697</a>)</li>
<li>Remove patching of Click that mitigated errors on Python 3.6 with <code>LANG=C</code> (<a href=""https://redirect.github.com/psf/black/issues/3768"">#3768</a>)</li>
</ul>
<h3>Parser</h3>
<ul>
<li>Add support for the new PEP 695 syntax in Python 3.12 (<a href=""https://redirect.github.com/psf/black/issues/3703"">#3703</a>)</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Speed up <em>Black</em> significantly when the cache is full (<a href=""https://redirect.github.com/psf/black/issues/3751"">#3751</a>)</li>
<li>Avoid importing <code>IPython</code> in a case where we wouldn't need it (<a href=""https://redirect.github.com/psf/black/issues/3748"">#3748</a>)</li>
</ul>
<h3>Output</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/psf/black/commit/193ee766ca496871f93621d6b58d57a6564ff81b""><code>193ee76</code></a> Prepare release 23.7.0 (<a href=""https://redirect.github.com/psf/black/issues/3776"">#3776</a>)</li>
<li><a href=""https://github.com/psf/black/commit/38723bb7787d50f8751fad2eaa48b52c9e94c18d""><code>38723bb</code></a> Unpin pytest-xdist (<a href=""https://redirect.github.com/psf/black/issues/3772"">#3772</a>)</li>
<li><a href=""https://github.com/psf/black/commit/138769aa27d6bd86507a0cd98d9a5bf8f63a8e99""><code>138769a</code></a> Disable coverage on pypy tests (<a href=""https://redirect.github.com/psf/black/issues/3777"">#3777</a>)</li>
<li><a href=""https://github.com/psf/black/commit/ad3724b7ffc01d8152c97fe7f4dcf35220f21a8e""><code>ad3724b</code></a> Upgrade to latest mypy (<a href=""https://redirect.github.com/psf/black/issues/3775"">#3775</a>)</li>
<li><a href=""https://github.com/psf/black/commit/b8e2ec728cc09d0f00829a9cffcb54da3efa5760""><code>b8e2ec7</code></a> Fix crash on type comment with trailing space (<a href=""https://redirect.github.com/psf/black/issues/3773"">#3773</a>)</li>
<li><a href=""https://github.com/psf/black/commit/257d392217974a76231e306133288748c7b70786""><code>257d392</code></a> Fix removed comments in stub files (<a href=""https://redirect.github.com/psf/black/issues/3745"">#3745</a>)</li>
<li><a href=""https://github.com/psf/black/commit/2593af2c5d211b58e28f7c1472f1f67e6783216a""><code>2593af2</code></a> Improve performance by skipping unnecessary normalisation (<a href=""https://redirect.github.com/psf/black/issues/3751"">#3751</a>)</li>
<li><a href=""https://github.com/psf/black/commit/f3b50e466969f9142393ec32a4b2a383ffbe5f23""><code>f3b50e4</code></a> Add CITATION.cff file (<a href=""https://redirect.github.com/psf/black/issues/3723"">#3723</a>)</li>
<li><a href=""https://github.com/psf/black/commit/0b4d7d55f78913be9e0a3738681ef3aafd5d9a5a""><code>0b4d7d5</code></a> Run pyupgrade on blib2to3 and src (<a href=""https://redirect.github.com/psf/black/issues/3771"">#3771</a>)</li>
<li><a href=""https://github.com/psf/black/commit/114e8357e65384e17baaa3c31aa528371e15679b""><code>114e835</code></a> Remove click patch (<a href=""https://redirect.github.com/psf/black/issues/3768"">#3768</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/psf/black/compare/23.3.0...23.7.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=black&package-manager=pip&previous-version=23.3.0&new-version=23.7.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2099.org.readthedocs.build/en/2099/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1630776144,IC_kwDOBm6k_c5hM6tQ,,9020979,2023-07-11T12:54:03Z,"Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. ",107914493
issue_comments,1629337927,IC_kwDOBm6k_c5hHblH,,9599,2023-07-10T16:43:38Z,"I tried running this locally just now. I made one edit:
```diff
diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py
index 7bdb9f3f..f9dfa8e6 100644
--- a/demos/plugins/example_js_manager_plugins.py
+++ b/demos/plugins/example_js_manager_plugins.py
@@ -15,6 +15,6 @@ def extra_js_urls(view_name):
     if view_name in PERMITTED_VIEWS:
         return [
             {
-                ""url"": f""/-/demos/plugins/static/table-example-plugins.js"",
+                ""url"": f""/static/table-example-plugins.js"",
             }
         ]
```
And then started it running like this:
```bash
wget https://datasette.io/content.db
```
```bash
datasette content.db \
  --plugins-dir demos/plugins \
  --static static:datasette/demos/plugins/static
```

It didn't quite work for me - I got this error on a table page:

<img width=""681"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/02f5c16d-ea43-4c36-920d-4b14406645b5"">

And this error on a query page:

<img width=""689"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/994db3a0-502a-4f36-8581-c47556600d63"">
",107914493
issue_comments,1629123734,IC_kwDOC8tyDs5hGnSW,,44622670,2023-07-10T14:46:52Z,@simonw any chance to get this fixed soon? ,197882382
issues,1796830110,PR_kwDOBm6k_c5VFw3j,Bump blacken-docs from 1.14.0 to 1.15.0,49699333,2023-07-10T13:49:12Z,"Bumps [blacken-docs](https://github.com/asottile/blacken-docs) from 1.14.0 to 1.15.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.15.0 (2023-07-09)</h2>
<ul>
<li>Drop Python 3.7 support.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/2e74dc4c30c336176e76e0426c66f09a4e79df9c""><code>2e74dc4</code></a> Version 1.15.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/8ddb2f1269b7652a9b668faa92c9d63d764696c0""><code>8ddb2f1</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/257"">#257</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/64fbca6cad36db1822528ece7f97a8152d29e05f""><code>64fbca6</code></a> Drop Python 3.7 support (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/256"">#256</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/f6a0817dbb7957f96ea6cc251edf3a50bf1c3c08""><code>f6a0817</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/255"">#255</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/cbdcb50b125b4f2fdb470122b2ba5e3c15de241a""><code>cbdcb50</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/254"">#254</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/37fc70ee00c7063777ed7343c1bd26fceb5422a6""><code>37fc70e</code></a> Upgrade requirements (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/253"">#253</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7bbe434415ce9f590cb057010590a648e7bea072""><code>7bbe434</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/250"">#250</a>)</li>
<li>See full diff in <a href=""https://github.com/asottile/blacken-docs/compare/1.14.0...1.15.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=blacken-docs&package-manager=pip&previous-version=1.14.0&new-version=1.15.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2098.org.readthedocs.build/en/2098/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1627598570,IC_kwDOCGYnMM5hAy7q,,9599,2023-07-09T04:13:34Z,On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.,140912432
issue_comments,1627597872,IC_kwDOCGYnMM5hAyww,,9599,2023-07-09T04:09:56Z,"Thanks, looks like a bug.",140912432
issues,1795219865,I_kwDOCGYnMM5rAOGZ,`--no-headers` doesn't work on most formats,33625,2023-07-09T03:43:36Z,"Version 3.33

```
sqlite-utils query library.db 'select asin from audible' --fmt plain --no-headers | head -3
asin
0062804006
0062891421
```",140912432
issue_comments,1627564127,IC_kwDODLZ_YM5hAqhf,,9599,2023-07-09T01:19:42Z,https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/,213286752
issue_comments,1627563202,IC_kwDODLZ_YM5hAqTC,,9599,2023-07-09T01:14:27Z,I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.,213286752
issues,1795187493,I_kwDODLZ_YM5rAGMl,Switch to pyproject.toml,9599,2023-07-09T01:06:56Z,First of my CLI tools to use https://til.simonwillison.net/python/pyproject,213286752
issue_comments,1627480353,IC_kwDOBm6k_c5hAWEh,,9599,2023-07-08T20:09:48Z,https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration is fixed now.,107914493
issue_comments,1627478910,IC_kwDOBm6k_c5hAVt-,,9599,2023-07-08T20:01:19Z,"Some examples:

-  https://docs.datasette.io/en/latest/sql_queries.html#canned-queries
- https://docs.datasette.io/en/latest/sql_queries.html#canned-query-parameters
- https://docs.datasette.io/en/latest/authentication.html#access-to-an-instance
- https://docs.datasette.io/en/latest/facets.html#facets-in-metadata
- https://docs.datasette.io/en/latest/full_text_search.html#configuring-full-text-search-for-a-table-or-view
- https://docs.datasette.io/en/latest/metadata.html
- https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript
- https://docs.datasette.io/en/latest/plugins.html#plugin-configuration

I need to fix this section: https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration",107914493
issues,1795051447,I_kwDOBm6k_c5q_k-3,Drop Python 3.7,9599,2023-07-08T18:39:44Z,">  I'm going to drop Python 3.7.

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1153#issuecomment-1627455892_

It's not supported any more: https://devguide.python.org/versions/",107914493
issue_comments,1627455892,IC_kwDOBm6k_c5hAQGU,,9599,2023-07-08T18:39:19Z,"```
ERROR: Could not find a version that satisfies the requirement Sphinx==6.1.3; extra == ""docs"" (from datasette[docs,test]) (from versions: 0.1.61611, 0.1.61798, 0.1.61843, 0.1.61945, 0.1.61950, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.5, 0.5.1, 0.5.2b1, 0.5.2, 0.6b1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 1.0b1, 1.0b2, 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.2b1, 1.2b2, 1.2b3, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3b1, 1.3b2, 1.3b3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.4a1, 1.4b1, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5a1, 1.5a2, 1.5b1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.6b1, 1.6b2, 1.6b3, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.7.0b1, 1.7.0b2, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.8.0b1, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 2.0.0b1, 2.0.0b2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 3.0.0b1, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.2.1, 3.3.0, 3.3.1, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 4.0.0b1, 4.0.0b2, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.3.0, 4.3.1, 4.3.2, 4.4.0, 4.5.0, 5.0.0b1, 5.0.0, 5.0.1, 5.0.2, 5.1.0, 5.1.1, 5.2.0, 5.2.0.post0, 5.2.1, 5.2.2, 5.2.3, 5.3.0)
ERROR: No matching distribution found for Sphinx==6.1.3; extra == ""docs""
```
I'm going to drop Python 3.7.",107914493
issue_comments,1627451646,IC_kwDOBm6k_c5hAPD-,,9599,2023-07-08T18:21:24Z,"This one was tricky:

<img width=""764"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/0fe2886b-4e07-4539-a254-4c47f02c3f65"">

I wanted complete control over the YAML example here, so I could ensure it used multi-line strings correctly.

I ended up changing my cog helper function to this:

```python
import json
import textwrap
from yaml import safe_dump
from ruamel.yaml import round_trip_load


def metadata_example(cog, data=None, yaml=None):
    assert data or yaml, ""Must provide data= or yaml=""
    assert not (data and yaml), ""Cannot use data= and yaml=""
    output_yaml = None
    if yaml:
        # dedent it first
        yaml = textwrap.dedent(yaml).strip()
        # round_trip_load to preserve key order:
        data = round_trip_load(yaml)
        output_yaml = yaml
    else:
        output_yaml = safe_dump(data, sort_keys=False)
    cog.out(""\n.. tab:: YAML\n\n"")
    cog.out(""    .. code-block:: yaml\n\n"")
    cog.out(textwrap.indent(output_yaml, ""        ""))
    cog.out(""\n\n.. tab:: JSON\n\n"")
    cog.out(""    .. code-block:: json\n\n"")
    cog.out(textwrap.indent(json.dumps(data, indent=2), ""        ""))
    cog.out(""\n"")
```
This allows me to call it ith YAML in some places:
```
.. [[[cog
    metadata_example(cog, yaml=""""""
    databases:
      fixtures:
        queries:
          neighborhood_search:
            fragment: fragment-goes-here
            hide_sql: true
            sql: |-
              select neighborhood, facet_cities.name, state
              from facetable join facet_cities on facetable.city_id = facet_cities.id
              where neighborhood like '%' || :text || '%' order by neighborhood;
    """""")
.. ]]]
```
I had to introduce https://pypi.org/project/ruamel.yaml/ as a dependency here in order to load YAML from disk while maintaining key order.

I'm still using `safe_dump(data, sort_keys=False)` from PyYAML as I couldn't get the result I wanted for outputting YAML from an input of JSON using PyYAML.",107914493
issue_comments,1627450852,IC_kwDOBm6k_c5hAO3k,,9599,2023-07-08T18:17:35Z,"I figured out a workaround:
```python
extensions = [
    ""sphinx.ext.extlinks"",
    ""sphinx.ext.autodoc"",
    ""sphinx_copybutton"",
]
if not os.environ.get(""DISABLE_SPHINX_INLINE_TABS""):
    extensions += [""sphinx_inline_tabs""]
```
That way I can run `sphinx-build -b xml . _build` successfully if I set that environment variable.

I get some noisy warnings, but it runs OK. And the resulting `docs.db` file has rows like this, which I think are fine:

<img width=""1533"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/0a289f53-2c6c-43e9-bda1-d4380383a02f"">
",107914493
issue_comments,1627448542,IC_kwDOBm6k_c5hAOTe,,9599,2023-07-08T18:05:44Z,"Running with `-P` opens a debugger when it hits the error:
```bash
sphinx-build -P -b xml . _build
```
```
(Pdb) list
2023 	
2024 	        Raise an exception unless overridden.
2025 	        """"""
2026 	        if (self.document.settings.strict_visitor
2027 	            or node.__class__.__name__ not in self.optional):
2028 ->	            raise NotImplementedError(
2029 	                '%s visiting unknown node type: %s'
2030 	                % (self.__class__, node.__class__.__name__))
2031 	
2032 	    def unknown_departure(self, node):
2033 	        """"""
(Pdb) self.optional
('meta',)
(Pdb) node.__class__.__name__
'TabContainer'
(Pdb) self.document.settings.strict_visitor
(Pdb) type(self.document.settings.strict_visitor)
<class 'NoneType'>
```
So if I can get `TabContainer` into that `self.optional` list I'll have fixed this problem.",107914493
issue_comments,1627448180,IC_kwDOBm6k_c5hAON0,,9599,2023-07-08T18:03:31Z,"Relevant code: https://github.com/docutils/docutils/blob/3b53ded52bc439d8068b6ecb20ea0a761247e479/docutils/docutils/nodes.py#L2021-L2031

```python
    def unknown_visit(self, node):
        """"""
        Called when entering unknown `Node` types.

        Raise an exception unless overridden.
        """"""
        if (self.document.settings.strict_visitor
            or node.__class__.__name__ not in self.optional):
            raise NotImplementedError(
                '%s visiting unknown node type: %s'
                % (self.__class__, node.__class__.__name__))
```",107914493
issue_comments,1627447750,IC_kwDOBm6k_c5hAOHG,,9599,2023-07-08T18:00:56Z,"Actually no it's in `sphinx-build`:
```
% sphinx-build -b xml . _build
Running Sphinx v6.1.3
building [mo]: targets for 0 po files that are out of date
writing output... 
building [xml]: targets for 28 source files that are out of date
updating environment: [new config] 28 added, 0 changed, 0 removed
reading sources... [100%] writing_plugins                                                                                                                          
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [  3%] authentication                                                                                                                            
Exception occurred:
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/docutils/nodes.py"", line 2028, in unknown_visit
    raise NotImplementedError(
NotImplementedError: <class 'docutils.writers.docutils_xml.XMLTranslator'> visiting unknown node type: TabContainer
The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1wkxmkji.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
```",107914493
issue_comments,1627447478,IC_kwDOBm6k_c5hAOC2,,9599,2023-07-08T17:59:25Z,"Hit a problem:
```
Exception occurred:
  File ""/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/docutils/nodes.py"", line 2028, in unknown_visit
    raise NotImplementedError(
NotImplementedError: <class 'docutils.writers.docutils_xml.XMLTranslator'> visiting unknown node type: TabContainer
The full traceback has been saved in /tmp/sphinx-err-tfujyw1h.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
```
That's happening here: https://github.com/simonw/datasette/blob/0183e1a72d4d93b1d9a9363f4d47fcc0b5d5849c/.github/workflows/deploy-latest.yml#L42-L48

My https://github.com/simonw/sphinx-to-sqlite tool can't handle the new `TabContainer` elements introduced by `sphinx-inline-tabs`.",107914493
issue_comments,1627396658,IC_kwDOBm6k_c5hABoy,,9599,2023-07-08T16:40:07Z,"https://docs.datasette.io/en/latest/metadata.html

![inline-tabs](https://github.com/simonw/datasette/assets/9599/975bdff5-74ac-451e-92c3-a7dd05d4b862)
",107914493
issue_comments,1627396450,IC_kwDOBm6k_c5hABli,,9599,2023-07-08T16:38:58Z,"I'm using `cog` and this utility function to generate the YAML/JSON tabs:

https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata_doc.py#L1-L13

Example usage:

https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata.rst?plain=1#L17-L53",107914493
issue_comments,1627395947,IC_kwDOBm6k_c5hABdr,,9599,2023-07-08T16:35:45Z,I was inspired to finally address this after seeing `sphinx-inline-tabs` at work in https://webcolors.readthedocs.io/en/latest/install.html,107914493
issues,1794604602,PR_kwDOBm6k_c5U-akg,Clarify docs for descriptions in metadata,15906,2023-07-08T01:57:58Z,"G'day! I got confused while debugging, earlier today. That's on me, but it does strike me a little repetition in the metadata documentation might help those flicking around it rather than reading it from top to bottom. No worries if you think otherwise.

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2096.org.readthedocs.build/en/2096/

<!-- readthedocs-preview datasette end -->",107914493
issues,1794097871,I_kwDOBm6k_c5q78LP,"Introduce ""dark mode"" CSS",3315059,2023-07-07T19:15:58Z,Using [the CSS media query `prefers-color-scheme`](https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme) we can provide a dark-mode version of Datasette,107914493
issue_comments,1618380888,IC_kwDOCGYnMM5gdohY,,9599,2023-07-03T14:09:11Z,"For the CLI:

```bash
sqlite-utils rename-table data.db old_table_name new_table_name
```

For the Python code, should it go on Table or on Database?
```python
db[""foo""].rename_table(""bar"")

db.rename_table(""foo"", ""bar"")
```
I think I like the second better, it's slightly more clear.

Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.",140912432
issues,1786258502,I_kwDOCGYnMM5qeCRG,Table renaming: db.rename_table() and sqlite-utils rename-table,9599,2023-07-03T14:07:42Z,"> I find myself wanting two new features in `sqlite-utils`:
> - The ability to have the new transformed table set to a specific name, while keeping the old table around
> - The ability to rename a table (`sqlite-utils` doesn't have a table rename function at all right now)

_Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618375042_
            ",140912432
issues,1786243905,I_kwDOCGYnMM5qd-tB,Document that running `db.transform()` tidies up the schema indentation,9599,2023-07-03T13:59:28Z,"> ... and it turns out running `.transform()` with no arguments still fixes the format of the schema!

```pycon
>>> db[""log""].add_column(""foo"", str)
<Table log (id, name2, age, weight, foo)>
>>> db[""log""].add_column(""bar"", str)
<Table log (id, name2, age, weight, foo, bar)>
>>> db[""log""].add_column(""baz"", str)
<Table log (id, name2, age, weight, foo, bar, baz)>
>>> print(db[""log""].schema)
CREATE TABLE ""log"" (
   [id] INTEGER PRIMARY KEY,
   [name2] TEXT,
   [age] INTEGER,
   [weight] FLOAT
, [foo] TEXT, [bar] TEXT, [baz] TEXT)
>>> db[""log""].transform()
<Table log (id, name2, age, weight, foo, bar, baz)>
>>> print(db[""log""].schema)
CREATE TABLE ""log"" (
   [id] INTEGER PRIMARY KEY,
   [name2] TEXT,
   [age] INTEGER,
   [weight] FLOAT,
   [foo] TEXT,
   [bar] TEXT,
   [baz] TEXT
)
```

_Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1618347727_
            ",140912432
issue_comments,1617395444,IC_kwDOCGYnMM5gZ370,,9599,2023-07-03T05:44:43Z,Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data,140912432
issues,1785360409,I_kwDOCGYnMM5qanAZ,`--empty-null` option when importing CSV,9599,2023-07-03T05:23:36Z,"CSV files with empty cells in (which come through as the empty string) are common and a bit gross.

Having an option that means ""and if it's an empty string store `null` instead) would be cool.

I brainstormed name options here https://chat.openai.com/share/c947b738-ee7d-419c-af90-bc84e90987da",140912432
issue_comments,1616853644,IC_kwDOBm6k_c5gXzqM,,15178711,2023-07-02T22:00:48Z,"I just saw in the docs that Dasette auto-detects `settings.json`:

> settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs
> [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)",107914493
issue_comments,1616782404,IC_kwDOCGYnMM5gXiRE,,9599,2023-07-02T19:24:14Z,"[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now.

The thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns.

Example for https://datasette.io/content/users

```sql
CREATE TABLE [users] (
   [login] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [avatar_url] TEXT,
   [gravatar_id] TEXT,
   [html_url] TEXT,
   [type] TEXT,
   [site_admin] INTEGER,
   [name] TEXT
);
```
And the dataclass:
```python
from dataclasses import dataclass

@dataclass
class User:
    id: int
    login: str
    node_id: str
    avatar_url: str
    gravatar_id: str
    html_url: str
    type: str
    site_admin: int
    name: str
```
",140912432
issues,1784794489,I_kwDOCGYnMM5qYc15,Explore the intersection between sqlite-utils and dataclasses,9599,2023-07-02T19:23:08Z,"> Aside: this makes me think it might be cool if `sqlite-utils` had a way of working with dataclasses rather than just dicts, and knew how to create a SQLite table to match a dataclass and maybe how to code-generate dataclasses for a specific table schema (dynamically or even using code-generation that can be written to disk, for better editor integrations).

_Originally posted by @simonw in https://github.com/simonw/llm/issues/65#issuecomment-1616742529_
            ",140912432
issue_comments,1616286848,IC_kwDOBm6k_c5gVpSA,,15178711,2023-07-02T02:17:46Z,"Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way. 

The next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs",107914493
issue_comments,1616195496,IC_kwDOBm6k_c5gVS-o,,273509,2023-07-02T00:06:54Z,"I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.",107914493
issue_comments,1616095810,IC_kwDOBm6k_c5gU6pC,,15178711,2023-07-01T20:31:31Z,"> Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

I just did a github search for `user:simonw ""def extra_js_urls(""` ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code",107914493
issue_comments,1615997736,IC_kwDOBm6k_c5gUiso,,9020979,2023-07-01T16:55:24Z,"> Ok @hydrosquall a couple things before this PR should be good to go:

Thank you @asg017 ! I've pushed both suggested changes onto this branch.

> Not sure how difficult it'll be to inject it server-side

If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a ""template"" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.

In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a ""manual sync"" might be good enough. 

> In terms of how to integrate this into Datasette, a few options I can see working:

This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.

> I'm also curious to see how ""plugins for a plugin' would work

I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a ""public API of other plugins"", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that. 

> here's a list of Simon's Datasette plugins that use ""extra_js_urls()""

Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

> I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)

I look forward to trying this out 👍 

",107914493
issues,1783304750,I_kwDOBm6k_c5qSxIu,JS Plugin Hooks for the Code Editor,15178711,2023-07-01T00:51:57Z,"When #2052 merges, I'd like to add support to add extensions/functions to the Datasette code editor. 

I'd eventually like to build a JS plugin for [`sqlite-docs`](https://github.com/asg017/sqlite-docs), to add things like:

- Inline documentation for tables/columns on hover
- Inline docs for custom functions that are loaded in
- More detailed autocomplete for tables/columns/functions

I did some hacking to see what this would look like, see here:

<img width=""1223"" alt=""image"" src=""https://github.com/simonw/datasette/assets/15178711/64f95cbc-1492-4365-896f-b88c6d08a649"">
<img width=""1223"" alt=""image"" src=""https://github.com/simonw/datasette/assets/15178711/73e602ba-5f45-417a-997e-5aea1738527a"">

There can be a new hook that allows JS plugins to add new ""extension"" in the CodeMirror editorview here:

https://github.com/simonw/datasette/blob/8cd60fd1d899952f1153460469b3175465f33f80/datasette/static/cm-editor-6.0.1.js#L25

Will need some more planning. For example, the Codemirror bundle in Datasette has functions that we could re-export for plugins to use (so we don't load 2 version of `""@codemirror/autocomplete""`, for example. ",107914493
issue_comments,1614652001,IC_kwDOBm6k_c5gPaJh,,9599,2023-06-30T13:27:13Z,"I agree, settings in the DB doesn't make sense but metadata does.

On the JSON v YAML v TOML issue I just spotted Caddy has a concept of config adapters which they use to resolve exactly that problem: https://caddyserver.com/docs/config-adapters",107914493
issue_comments,1613896210,IC_kwDOBm6k_c5gMhoS,,15178711,2023-06-29T22:53:33Z,"Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent",107914493
issue_comments,1613895188,IC_kwDOBm6k_c5gMhYU,,15178711,2023-06-29T22:51:53Z,"I agree with not liking `metadata.json` stuff in a `datasette.*` config file. Editing description of a table/column in a file like `datasette.*` seems odd to me. 

Though since plugin configuration currently lives in `metadata.json`, I think it should be removed from there and placed in `datasette.*`, at least for top-level config like `datasette-auth-github`'s config. Keeping `metadata.json` strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like `datasette.*`.

And ya, supporting both `datasette.*` and CLI flags makes a lot of sense to me. Any `--setting` flag should override anything in `datasette.*` for easier debugging, with possibly a warning message so people don't get confused. Same with `--port` and a port defined in `datasette.*`",107914493
issue_comments,1613889979,IC_kwDOBm6k_c5gMgG7,,9599,2023-06-29T22:44:08Z,"I do like also being able to set options using command line options though - for things like SQL time limits I'd much rather be able to throw on `--setting sql_time_limit_ms 10000` than have to save a config file to disk.

So I'd want to support both. Which maybe means also having a way to set plugin options with CLI options. `datasette publish` kind of has that ability already:

```
datasette publish heroku my_database.db \
    --name my-heroku-app-demo \
    --install=datasette-auth-github \
    --plugin-secret datasette-auth-github client_id your_client_id \
    --plugin-secret datasette-auth-github client_secret your_client_secret
```",107914493
issue_comments,1613887492,IC_kwDOBm6k_c5gMfgE,,9599,2023-06-29T22:40:25Z,"I'm strongly in favour of combining settings, configuration and plugin configuration.

I'm not keen on mixing in metadata as well - that feels like a different concept to me, and I'm unhappy with how that's already had things like plugin settings leak into it.

I'm not yet sold on TOML - I actually find it less intuitive than YAML, surprisingly. They all have their warts I guess.

Datasette already has the ability to consume JSON or YAML for metadata - maybe it could grow TOML support too? That way users could have a `datasette.json` or `datasette.yaml` or `datasette.toml` file depending on their preference.

In terms of metadata: since that's means to be driven by a plugin hook anyway, maybe one of the potential sources of metadata is a `metadata` nested object in that `datasette.*` configuration file. Or you can have it in a separate `metadata.json` or bundled into the SQLite database or some other plugin-driven mechanism.",107914493
issues,1781530343,I_kwDOBm6k_c5qL_7n,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",15178711,2023-06-29T21:18:23Z,"Very often I get tripped up when trying to configure my Datasette instances. For example: if I want to change the port my app listen too, do I do that with a CLI flag, a `--setting` flag, inside `metadata.json`, or an env var? If I want to up the time limit of SQL statements, is that under `metadata.json` or a setting? Where does my plugin configuration go?

Normally I need to look it up in Datasette docs, and I quickly find my answer, but the number of places where ""config"" goes it overwhelming.

- Flat CLI flags like `--port`, `--host`, `--cors`, etc.
- `--setting`, like `default_page_size`, `sql_time_limit_ms` etc
- Inside `metadata.json`, including plugin configuration

Typically my Datasette deploys are extremely long shell commands, with multiple `--setting` and other CLI flags.

## Proposal: Consolidate all ""config"" into `datasette.toml`

I propose that we add a new `datasette.toml` that combines ""settings"", ""metadata"", and other common CLI flags like `--port` and `--cors` into a single file. It would be similar to ""Cargo.toml"" in Rust projects, ""package.json"" in Node projects, and ""pyproject.toml"" in Python, etc.

A sample of what it could look like:

```toml
# ""top level"" configuration that are currently CLI flags on `datasette serve`
[config]
port = 8020
host = ""0.0.0.0""
cors = true

# replaces multiple `--setting` flags
[settings]
base_url = ""/app/datasette/""
default_allow_sql = true
sql_time_limit_ms = 3500

# replaces `metadata.json`.
# The contents of datasette-metadata.json could be defined in this file instead, but supporting separate files is nice (since those are easy to machine-generate)
[metadata]
include=""./datasette-metadata.json""

# plugin-specific 
[plugins]
[plugins.datasette-auth-github]
client_id = {env = ""DATASETTE_AUTH_GITHUB_CLIENT_ID""}
client_secret = {env = ""GITHUB_CLIENT_SECRET""}

[plugins.datasette-cluster-map]

latitude_column = ""lat""
longitude_column = ""lon""
```

## Pros
- Instead of multiple files and CLI flags, everything could be in one tidy file
- Editing config in a separate file is easier than editing CLI flags, since you don't have to kill a process + edit a command every time
- New users will know ""just edit my `datasette.toml` instead of needing to learn metadata + settings + CLI flags
- Better dev experience for multiple environment. For example, could have `datasette -c datasette-dev.toml` for local dev environments (enables SQL, debug plugins, long timeouts, etc.), and a `datasette -c datasette-prod.toml` for ""production"" (lower timeouts, less plugins, monitoring plugins, etc.)

## Cons
- Yet another config-management system. Now Datasette users will need to know about metadata, settings, CLI flags, _and_ `datasette.toml`. However with enough documentation + announcements + examples, I think we can get ahead of it.
- If toml is chosen, would need to add a toml parser for Python version <3.11
- Multiple sources of config require priority. For example: Would `--setting default_allow_sql off` override the value inside `[settings]`? What about `--port`? 

## Other Notes

### Toml

I chose toml over json because toml supports comments. I chose toml over yaml because Python 3.11 has builtin support for it. I also find toml easier to work with since it doesn't have the odd ""gotchas"" that YAML has (""ex `3.10` resolving to `3.1`, Norway `NO` resolving to `false`, etc.). It also mimics `pyproject.toml` which is nice. Happy to change my mind about this however


### Plugin config will be difficult

Plugin config is currently in `metadata.json` in two places:

1. Top level, under `""plugins.[plugin-name]""`. This fits well into `datasette.toml` as `[plugins.plugin-name]`
2. Table level, under `""databases.[db-name].tables.[table-name].plugins.[plugin-name]`. This doesn't fit that well into `datasette.toml`, unless it's nested under `[metadata]`?

### Extensions, static, one-off plugins?

We could also include equivalents of `--plugins-dir`, `--static`, and `--load-extension` into `datasette.toml`, but I'd imagine there's a few security concerns there to think through. 


### Explicitly list with plugins to use?

I believe Datasette by default will load all install plugins on startup, but maybe `datasette.toml` can specify a list of plugins to use? For example, a dev version of `datasette.toml` can specify `datasette-pretty-traces`, but the prod version can leave it out",107914493
issue_comments,1613778296,IC_kwDOBm6k_c5gME14,,15178711,2023-06-29T20:36:09Z,"Ok @hydrosquall a couple things before this PR should be good to go:

- Can we move `datasette/static/table-example-plugins.js` into `demos/plugins/static`?
- For `datasetteManager.VERSION`, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into `manager.js` directly, so we don't have to worry about server-rendering (but that can be a future PR)

In terms of how to integrate this into Datasette, a few options I can see working:

- Push this as-is and figure it out before the next release
- Hide this feature behind a settings flag (`--setting unstable-js-plugins on`) and use that setting to hide/show `<script src=""{{ urls.static('datasette-manager.js') }}"" defer></script>` in `base.html`

I'll let @simonw decide which one to work with. I kindof like the idea of having an ""unstable"" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready.

I'm also curious to see how ""plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling ""plugins of plugins"" at the top with Datasette seems really convoluted. 

Also for posterity, here's a list of Simon's Datasette plugins that use ""extra_js_urls()"", which probably means they can be ported/re-written to use this new plugin system:

- [`datasette-vega`](https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/__init__.py#L25)
- [`datasette-cluster-map`](https://github.com/simonw/datasette-cluster-map/blob/795d25ad9ff6cba0307191f44fecc8f8070bef5c/datasette_cluster_map/__init__.py#L14)
- [`datasette-leaflet-geojson`](https://github.com/simonw/datasette-leaflet-geojson/blob/64713aa497750400b9ac2c12e8bb6ffab8eb77f3/datasette_leaflet_geojson/__init__.py#L47)
- [`datasette-pretty-traces`](https://github.com/simonw/datasette-pretty-traces/blob/5219d65eca3d7d7a73bb9d3120df42fe046a1315/datasette_pretty_traces/__init__.py#L5)
- [`datasette-youtube-embed`](https://github.com/simonw/datasette-youtube-embed/blob/4b4a0d7e58ebe15f47e9baf68beb9908c1d899da/datasette_youtube_embed/__init__.py#L55)
- [`datasette-leaflet-freedraw`](https://github.com/simonw/datasette-leaflet-freedraw/blob/8f28c2c2080ec9d29f18386cc6a2573a1c8fbde7/datasette_leaflet_freedraw/__init__.py#L66)
- [`datasette-hovercards`](https://github.com/simonw/datasette-hovercards/blob/9439ba46b7140fb03223faff0d21aeba5615a287/datasette_hovercards/__init__.py#L5)
- [`datasette-mp3-audio`](https://github.com/simonw/datasette-mp3-audio/blob/4402168792f452a46ab7b488e40ec49cd4b12185/datasette_mp3_audio/__init__.py#L6)
- [`datasette-geojson-map`](https://github.com/simonw/datasette-geojson-map/blob/32af5f1fd1a07278bbf8071fbb20a61e0f613246/datasette_geojson_map/__init__.py#L30)",107914493
issue_comments,1613381990,IC_kwDOBm6k_c5gKkFm,,9599,2023-06-29T15:23:04Z,"Felt lazy:
```bash
symbex test_homepage -f tests/test_api.py | \
  llm -m 4 --system 'Change all of the == some integer tests in this code to isinstance(thing, int) instead'
```
Output:
```python
# File: tests/test_api.py Line: 26
@pytest.mark.asyncio
async def test_homepage(ds_client):
    response = await ds_client.get(""/.json"")
    assert response.status_code == 200
    assert ""application/json; charset=utf-8"" == response.headers[""content-type""]
    data = response.json()
    assert isinstance(data.keys(), int)
    d = data[""fixtures""]
    assert d[""name""] == ""fixtures""
    assert isinstance(d[""tables_count""], int)
    assert isinstance(len(d[""tables_and_views_truncated""]), int)
    assert d[""tables_and_views_more""] is True
    # 4 hidden FTS tables + no_primary_key (hidden in metadata)
    assert isinstance(d[""hidden_tables_count""], int)
    # 201 in no_primary_key, plus 6 in other hidden tables:
    assert isinstance(d[""hidden_table_rows_sum""], int), data
    assert isinstance(d[""views_count""], int)
```
I'll use most of that and delete the obsoleted comments.",107914493
issue_comments,1613375407,IC_kwDOBm6k_c5gKiev,,9599,2023-06-29T15:20:52Z,I'm going to remove this assertion entirely. The homepage JSON needs a refactor anyway.,107914493
issues,1781047747,I_kwDOBm6k_c5qKKHD,test_homepage intermittent failure,9599,2023-06-29T15:20:37Z,"e.g. in https://github.com/simonw/datasette/actions/runs/5413590227/jobs/9839373852

```
=================================== FAILURES ===================================
________________________________ test_homepage _________________________________
[gw0] linux -- Python 3.7.17 /opt/hostedtoolcache/Python/3.7.17/x64/bin/python

ds_client = <datasette.app.DatasetteClient object at 0x7f85d271ef50>

    @pytest.mark.asyncio
    async def test_homepage(ds_client):
        response = await ds_client.get(""/.json"")
        assert response.status_code == 200
        assert ""application/json; charset=utf-8"" == response.headers[""content-type""]
        data = response.json()
        assert data.keys() == {""fixtures"": 0}.keys()
        d = data[""fixtures""]
        assert d[""name""] == ""fixtures""
        assert d[""tables_count""] == 24
        assert len(d[""tables_and_views_truncated""]) == 5
        assert d[""tables_and_views_more""] is True
        # 4 hidden FTS tables + no_primary_key (hidden in metadata)
        assert d[""hidden_tables_count""] == 6
        # 201 in no_primary_key, plus 6 in other hidden tables:
>       assert d[""hidden_table_rows_sum""] == 207, data
E       AssertionError: {'fixtures': {'color': '9403e5', 'hash': None, 'hidden_table_rows_sum': 0, 'hidden_tables_count': 6, ...}}
E       assert 0 == 207
```
My guess is that this is a timing error, where very occasionally the ""count rows but stop counting if it exceeds a time limit"" thing fails.",107914493
issue_comments,1613369355,IC_kwDOBm6k_c5gKhAL,,9599,2023-06-29T15:18:34Z,Posted on the Glitch feedback forum about this here: https://support.glitch.com/t/upgrade-python-version-from-3-7-which-is-now-eol-to-something-more-recent/63011,107914493
issue_comments,1613360413,IC_kwDOBm6k_c5gKe0d,,9599,2023-06-29T15:13:04Z,"One problem: https://glitch.com/ still provides 3.7:
```
$ python3 --version
Python 3.7.10
```
<img width=""956"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/e1fa44c0-4362-40e6-a2c7-d5b09b784c66"">
",107914493
issues,1781022369,I_kwDOBm6k_c5qKD6h,Drop support for Python 3.7,9599,2023-06-29T15:06:38Z,"It's EOL now, as of 2023-06-27 (two days ago): https://devguide.python.org/versions/

",107914493
issue_comments,1613346412,IC_kwDOBm6k_c5gKbZs,,9599,2023-06-29T15:05:04Z,"Decided to fix just those ""Ambiguous variable name"" ones:

```bash
ruff check . | grep E741
```
Then iterated through and fixed them all.",107914493
issue_comments,1613339404,IC_kwDOBm6k_c5gKZsM,,9599,2023-06-29T15:01:01Z,"I tried it just now and got some interesting results.

I dropped in a `ruff.toml` file:
```toml
line-length = 160
```
Because the default line length limit of 88 was causing a lot of noisy errors.

Then run:

```bash
pip install ruff
ruff check .
```
Plenty of warnings about unused imports - running `ruff check . --fix` fixed those automatically, but I think I still need to manually review them as some might be imports which are deliberate and should be in `__all__` to ensure they are visible from that module as well.

Some lines in tests are longer than even 160 chars, e.g.:

https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L673-L681

These can have `  # noqa: E501` added to the end of those lines to skip the check for them.

That got it down to:

```
% ruff check . 
datasette/views/table.py:23:5: F811 Redefinition of unused `format_bytes` from line 19
run_tests.py:2:5: E401 Multiple imports on one line
tests/test_api.py:591:40: F811 Redefinition of unused `app_client_no_files` from line 7
tests/test_api.py:629:35: F811 Redefinition of unused `app_client_no_files` from line 7
tests/test_api.py:635:54: F811 Redefinition of unused `app_client_with_dot` from line 8
tests/test_api.py:661:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 9
tests/test_api.py:759:25: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10
tests/test_api.py:892:28: F811 Redefinition of unused `app_client_larger_cache_size` from line 11
tests/test_api.py:928:5: F811 Redefinition of unused `app_client_with_cors` from line 12
tests/test_api.py:929:5: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10
tests/test_api.py:969:38: F811 Redefinition of unused `app_client_two_attached_databases` from line 13
tests/test_api.py:976:39: F811 Redefinition of unused `app_client_conflicting_database_names` from line 14
tests/test_api.py:987:38: F811 Redefinition of unused `app_client_immutable_and_inspect_file` from line 15
tests/test_api.py:1002:24: F811 Redefinition of unused `app_client` from line 6
tests/test_csv.py:67:33: F811 Redefinition of unused `app_client_with_cors` from line 6
tests/test_csv.py:157:21: F811 Redefinition of unused `app_client_csv_max_mb_one` from line 5
tests/test_csv.py:198:20: F811 Redefinition of unused `app_client_with_trace` from line 7
tests/test_csv.py:209:53: F811 Redefinition of unused `app_client_with_trace` from line 7
tests/test_csv.py:215:53: F811 Redefinition of unused `app_client_with_trace` from line 7
tests/test_filters.py:102:11: F811 Redefinition of unused `test_through_filters_from_request` from line 81
tests/test_html.py:19:19: F811 Redefinition of unused `app_client_two_attached_databases` from line 7
tests/test_html.py:175:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 6
tests/test_html.py:469:51: F811 Redefinition of unused `app_client` from line 4
tests/test_html.py:797:26: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
tests/test_html.py:840:44: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
tests/test_html.py:850:51: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
tests/test_pagination.py:50:43: F821 Undefined name `parse_next`
tests/test_pagination.py:82:7: F811 Redefinition of unused `KeysetPaginator` from line 36
tests/test_plugins.py:115:15: E741 Ambiguous variable name: `l`
tests/test_plugins.py:482:161: E501 Line too long (170 > 160 characters)
tests/test_plugins.py:543:29: E741 Ambiguous variable name: `l`
tests/test_plugins.py:563:161: E501 Line too long (170 > 160 characters)
tests/test_plugins.py:940:62: E741 Ambiguous variable name: `l`
tests/test_table_api.py:739:5: F811 Redefinition of unused `app_client_returned_rows_matches_page_size` from line 6
tests/test_table_api.py:1066:45: F811 Redefinition of unused `app_client_with_trace` from line 5
tests/test_table_html.py:484:29: E741 Ambiguous variable name: `l`
tests/test_table_html.py:524:29: E741 Ambiguous variable name: `l`
tests/test_table_html.py:675:161: E501 Line too long (165 > 160 characters)
tests/test_table_html.py:897:161: E501 Line too long (164 > 160 characters)
tests/test_table_html.py:902:161: E501 Line too long (164 > 160 characters)
tests/test_utils.py:141:161: E501 Line too long (176 > 160 characters)
Found 41 errors.
```
Those ""Redefinition of unused `app_client_two_attached_databases`"" lines are caused because of the fixtures pattern I'm using here:

https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L3-L20

I could fix that by getting rid of `fixtures.py` and moving those into `conftest.py`.",107914493
issues,1781005740,I_kwDOBm6k_c5qJ_2s,Adopt ruff for linting,9599,2023-06-29T14:56:43Z,https://beta.ruff.rs/docs/,107914493
issue_comments,1613316722,IC_kwDOBm6k_c5gKUJy,,9599,2023-06-29T14:48:10Z,Spell check is passing now.,107914493
issue_comments,1613315851,IC_kwDOBm6k_c5gKT8L,,9599,2023-06-29T14:47:38Z,"Confirmed, this was a 2.2.5 change: https://github.com/codespell-project/codespell/releases/tag/v2.2.5

> -   Add displaing->displaying by [@peternewman](https://github.com/peternewman) in [#2808](https://github.com/codespell-project/codespell/pull/2808)",107914493
issue_comments,1613307716,IC_kwDOBm6k_c5gKR9E,,9599,2023-06-29T14:42:23Z,"Yes, upgrading locally got me the correct version and the test failure:
```
% pip install -U codespell
Requirement already satisfied: codespell in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages (2.2.2)
Collecting codespell
  Downloading codespell-2.2.5-py3-none-any.whl (242 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 4.9 MB/s eta 0:00:00
Installing collected packages: codespell
  Attempting uninstall: codespell
    Found existing installation: codespell 2.2.2
    Uninstalling codespell-2.2.2:
      Successfully uninstalled codespell-2.2.2
Successfully installed codespell-2.2.5
% codespell docs/metadata.rst
docs/metadata.rst:192: displaing ==> displaying
```",107914493
issue_comments,1613306787,IC_kwDOBm6k_c5gKRuj,,9599,2023-06-29T14:41:47Z,"Looks like in CI it's running 2.2.5:
```
Collecting codespell (from datasette==1.0a2)
  Downloading codespell-2.2.5-py3-none-any.whl (242 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 31.1 MB/s eta 0:00:00
```
But on my laptop it's 2.2.2:
```
% codespell --version
2.2.2
```",107914493
issue_comments,1613305070,IC_kwDOBm6k_c5gKRTu,,9599,2023-06-29T14:40:44Z,"I'm not sure why I can't duplicate this failure in my local development environment:
```
% codespell docs/metadata.rst
```
It finds no errors.",107914493
issues,1780973290,I_kwDOBm6k_c5qJ37q,codespell test failure,9599,2023-06-29T14:40:10Z,"https://github.com/simonw/datasette/actions/runs/5413443676/jobs/9838999356
```
  codespell docs/*.rst --ignore-words docs/codespell-ignore-words.txt
  codespell datasette -S datasette/static --ignore-words docs/codespell-ignore-words.txt
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.17/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.17/x64/lib
docs/metadata.rst:192: displaing ==> displaying
```
This failure is legit, it found a spelling mistake: https://github.com/simonw/datasette/blob/ede62036180993dbd9d4e5d280fc21c183cda1c3/docs/metadata.rst#L192",107914493
issue_comments,1613290899,IC_kwDOBm6k_c5gKN2T,,9599,2023-06-29T14:32:16Z,@dependabot recreate,107914493
issue_comments,1610512875,IC_kwDOBm6k_c5f_nnr,,9599,2023-06-28T02:02:10Z,"I prototyped an approach to this using dataclasses and a `cog` mechanism for turning those into rendered tables in Sphinx. Here's what that prototype looks like:

<img width=""1069"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/79210a43-716c-4470-a62d-01aa871973ec"">

See https://github.com/simonw/datasette/commit/68223784167fdec4e7ebfca56002a6548ba7b423 for how it works.

Here's the class that documented:

https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L54-L68

And the code that generates the rST: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L19-L45

And the bit that cog executes: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/docs/template_context.rst?plain=1#L9-L12",107914493
issue_comments,1610040517,IC_kwDOCGYnMM5f90TF,,9599,2023-06-27T18:44:31Z,"Got this working:
```bash
sqlite-utils insert /tmp/playground.db Playground_Submission_Data \
  ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000
```
",140912432
issues,1777548699,I_kwDOCGYnMM5p8z2b,`--stop-after` option for `insert` and `upsert` commands,9599,2023-06-27T18:44:15Z,I found myself wanting to insert rows from a 849MB CSV file without processing the whole thing: https://huggingface.co/datasets/jerpint-org/HackAPrompt-Playground-Submissions/tree/main,140912432
issue_comments,1606415188,IC_kwDOCGYnMM5fv_NU,,9599,2023-06-26T01:46:47Z,"I just tested this in a brand new virtual environment using the macOS Python 3:
```bash
pipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3
```
Then in that virtual environment I ran:
```bash
pip install sqlite-utils
# Confirm the right one is on the path:
which sqlite-utils

curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \
    sqlite-utils insert meteorites.db meteorites - --pk=id
sqlite-utils extract meteorites.db meteorites  recclass
```
This threw the same error reported above. Then I did this:
```bash
rm meteorites.db
pip install sqlean.py

curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \
    sqlite-utils insert meteorites.db meteorites - --pk=id
sqlite-utils extract meteorites.db meteorites  recclass
```
And that second time it worked correctly.",140912432
issue_comments,1606411508,IC_kwDOCGYnMM5fv-T0,,9599,2023-06-26T01:42:10Z,https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.,140912432
issue_comments,1606352600,IC_kwDOBm6k_c5fvv7Y,,15178711,2023-06-26T00:17:04Z,":wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch",107914493
issue_comments,1606315321,IC_kwDOCGYnMM5fvm05,,9599,2023-06-25T23:18:33Z,Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3,140912432
issue_comments,1606310630,IC_kwDOCGYnMM5fvlrm,,9599,2023-06-25T23:06:07Z,"Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here:
- https://github.com/coleifer/pysqlite3/issues/58",140912432
issue_comments,1606297356,IC_kwDOCGYnMM5fvicM,,9599,2023-06-25T22:42:41Z,"Yes that does seem to do the trick:
```pycon
>>> import sqlean
>>> db = sqlean.connect(""/tmp/4.db"")
>>> db.execute('PRAGMA journal_mode;').fetchall()
[('delete',)]
>>> db.isolation_level
''
>>> db.execute('PRAGMA journal_mode=wal;')
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction
>>> db.isolation_level = None
>>> db.isolation_level
>>> db.execute('PRAGMA journal_mode=wal;')
<sqlean.dbapi2.Cursor object at 0x104437c20>
```
Weird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.",140912432
issue_comments,1606294627,IC_kwDOCGYnMM5fvhxj,,9599,2023-06-25T22:40:10Z,I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.,140912432
issue_comments,1606293382,IC_kwDOCGYnMM5fvheG,,9599,2023-06-25T22:34:47Z,"```pycon
>>> import sqlite3
>>> db = sqlite3.connect(""/tmp/1.db"")
>>> db.execute('PRAGMA journal_mode=wal;')
<sqlite3.Cursor object at 0x102475ec0>
>>> import sqlean
>>> db2 = sqlean.connect(""/tmp/2.db"")
>>> db2.execute('PRAGMA journal_mode=wal;')
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction
```",140912432
issue_comments,1606290917,IC_kwDOCGYnMM5fvg3l,,9599,2023-06-25T22:32:28Z,"I've fixed most of the test failures, but I still need to fix this one:

> cannot change into wal mode from within a transaction",140912432
issue_comments,1606273005,IC_kwDOCGYnMM5fvcft,,9599,2023-06-25T21:47:47Z,I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.,140912432
issue_comments,1606270887,IC_kwDOCGYnMM5fvb-n,,9599,2023-06-25T21:37:12Z,"On my own laptop I got a crash running the tests - details here:

- https://github.com/nalgeon/sqlean.py/issues/3",140912432
issue_comments,1606270055,IC_kwDOCGYnMM5fvbxn,,9599,2023-06-25T21:31:56Z,"Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953

A bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`:
```
    @pytest.fixture
    def db_to_analyze_path(db_to_analyze, tmpdir):
        path = str(tmpdir / ""test.db"")
        db = sqlite3.connect(path)
>       db.executescript(""\n"".join(db_to_analyze.conn.iterdump()))
E       AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump'
```
Also some of these:
```
    def test_analyze_whole_database(db):
        assert set(db.table_names()) == {""one_index"", ""two_indexes""}
        db.analyze()
>       assert set(db.table_names()) == {""one_index"", ""two_indexes"", ""sqlite_stat1""}
E       AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'}
E         Extra items in the left set:
E         'sqlite_stat4'
E         Full diff:
E         - {'two_indexes', 'sqlite_stat1', 'one_index'}
E         + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'}
E         ?                                 ++++++++++++++++
```
Apparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting.

Plus some errors that look like this:
```
    def test_enable_wal():
        runner = CliRunner()
        dbs = [""test.db"", ""test2.db""]
        with runner.isolated_filesystem():
            for dbname in dbs:
                db = Database(dbname)
                db[""t""].create({""pk"": int}, pk=""pk"")
                assert db.journal_mode == ""delete""
            result = runner.invoke(cli.cli, [""enable-wal""] + dbs)
>           assert 0 == result.exit_code
E           AssertionError: assert 0 == 1
E            +  where 1 = <Result OperationalError('cannot change into wal mode from within a transaction')>.exit_code
```
Test summary:
```
============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s =============
```",140912432
issue_comments,1606237836,IC_kwDOCGYnMM5fvT6M,,22429695,2023-06-25T19:49:45Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`80.55`**% and project coverage change: **`-0.15`** :warning:
> Comparison is base [(`2747257`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/2747257a3334d55e890b40ec58fada57ae8cfbfd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`5e7d27e`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.22%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #560      +/-   ##
==========================================
- Coverage   96.36%   96.22%   -0.15%     
==========================================
  Files           6        6              
  Lines        2726     2752      +26     
==========================================
+ Hits         2627     2648      +21     
- Misses         99      104       +5     
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <63.63%> (-0.62%)` | :arrow_down: |
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <86.36%> (-0.20%)` | :arrow_down: |
| [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issues,1773458985,PR_kwDOCGYnMM5T2mMb,Use sqlean if available in environment,9599,2023-06-25T19:48:48Z,"Refs:
- #559 

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--560.org.readthedocs.build/en/560/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issues,1773450152,I_kwDOCGYnMM5ptLOo,sqlean support,9599,2023-06-25T19:27:26Z,"If sqlean is available, use that.

Refs:
- https://github.com/nalgeon/sqlean.py/issues/1#issuecomment-1605707788

This will provide a good workaround for:
- #235 ",140912432
issue_comments,1604379952,IC_kwDOCGYnMM5foOUw,,9599,2023-06-23T14:39:55Z,"Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment:

    sqlite-utils install pysqlite3-binary

But `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people.

The ""easiest"" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!",140912432
issue_comments,1600778057,IC_kwDOBm6k_c5fae9J,,9599,2023-06-21T12:51:22Z,"Another example of confusion from this today: https://discord.com/channels/823971286308356157/823971286941302908/1121042411238457374

See also https://gist.github.com/BinomeDeNewton/651ac8b50dd5420f8e54d1682eee5fed?permalink_comment_id=4605982#gistcomment-4605982",107914493
issues,1765870617,I_kwDOBm6k_c5pQQwZ,`--settings settings.json` option,9599,2023-06-20T17:48:45Z,"https://discord.com/channels/823971286308356157/823971286941302908/1120705940728066080

> May I add a request to the whole metadata / settings ? Allow to pass `--settings path/to/settings.json` instead of having to rely exclusively on directory mode to centralize settings (this would reflect the behavior of providing metadata)",107914493
issues,1764792125,I_kwDOBm6k_c5pMJc9,Show information on startup in directory configuration mode,9599,2023-06-20T07:13:33Z,"https://discord.com/channels/823971286308356157/823971286941302908/1120516587036889098

> One thing that would be helpful would be message at launch indicating a metadata.json is getting picked up. I'm using directory mode and was editing the wrong file for awhile before I realize nothing I was doing was having any effect.",107914493
issues,1762180409,I_kwDOBm6k_c5pCL05,Interactive row selection in Datasette ,24938923,2023-06-18T08:29:45Z,"Simon did a excellent [prototype](https://til.simonwillison.net/datasette/row-selection-prototype) of an interactive row selection in Datasette.

I hope this [functionality](https://camo.githubusercontent.com/3d4a0f31fb6a27fd279f809af5b53dc3b76faa63c7721e228951c5252b645a77/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032332f6461746173657474652d7069636b65722e676966) can be turned into a Datasette plugin.
",107914493
issues,1761613778,I_kwDOBm6k_c5pABfS,Support facets for columns that contain timestamps,19492893,2023-06-17T03:33:54Z,"
Django has this very nice filter for datetime fields -

<img width=""176"" alt=""image"" src=""https://github.com/simonw/datasette/assets/19492893/3c66d7c4-1579-4d30-8f08-89d111f4566e"">

It would be nice to have something similar to facet by a field that contains a timestamp in datasette too - Which doesn't seem to do anything with timestamps right now...

<img width=""283"" alt=""image"" src=""https://github.com/simonw/datasette/assets/19492893/069083e4-13f5-4b28-9473-a7b9d48839ea"">
",107914493
issue_comments,1592617271,IC_kwDOCGYnMM5e7Wk3,,13780613,2023-06-15T08:39:49Z,"> piping to `jq` is good enough usually

... or `python -m json.tool`[^1], if you don't have `jq` installed.

[^1]: no fancy colouring, like `jq`; only pretty-printing",140912432
issue_comments,1592110694,IC_kwDOCGYnMM5e5a5m,,7908073,2023-06-14T23:11:47Z,"sorry i was wrong. `sqlite-utils --raw-lines` works correctly

```
sqlite-utils --raw-lines :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$

sqlite-utils --csv --no-headers :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$
```

I think this was fixed somewhat recently",140912432
issue_comments,1592052320,IC_kwDOCGYnMM5e5Mpg,,7908073,2023-06-14T22:05:28Z,piping to `jq` is good enough usually,140912432
issue_comments,1592047502,IC_kwDOCGYnMM5e5LeO,,7908073,2023-06-14T22:00:10Z,"You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.

That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...

You may also try chunking like this:

```py
def chunks(lst, n) -> Generator:
    for i in range(0, len(lst), n):
        yield lst[i : i + n]

SQLITE_PARAM_LIMIT = 32765

data = []
chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)
for ids in chunked:
    data.expand(
        list(
            db.query(
                f""""""SELECT * from videos
                WHERE id in (""""""
                + "","".join([""?""] * len(ids))
                + "")"",
                (*ids,),
            )
        )
    )
```

but that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.

Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library",140912432
issues,1756975532,PR_kwDOBm6k_c5S_5Jl,Bump blacken-docs from 1.13.0 to 1.14.0,49699333,2023-06-14T13:57:52Z,"Bumps [blacken-docs](https://github.com/asottile/blacken-docs) from 1.13.0 to 1.14.0.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/adamchainz/blacken-docs/blob/main/CHANGELOG.rst"">blacken-docs's changelog</a>.</em></p>
<blockquote>
<h2>1.14.0 (2023-06-13)</h2>
<ul>
<li>Support Python 3.12.</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/4b94add9bc83d772ae5e31a679a18609c1750513""><code>4b94add</code></a> Version 1.14.0</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/292cb18828c73dfa5273c88ce261b453fb9fdf43""><code>292cb18</code></a> Support Python 3.12 (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/251"">#251</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/72ecf2c719027fd464f5561d4ab119f736e8a4ad""><code>72ecf2c</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/249"">#249</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/ea8f1ca88f2f0a58371bb07dab9c647d56d5b3ed""><code>ea8f1ca</code></a> Upgrade requirements (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/248"">#248</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/9979d7a78be5b7749ebb000f10f4515e5dc9a574""><code>9979d7a</code></a> Upgrade requirements (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/247"">#247</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7b71075ceb458be255e24da587c0275818b51faa""><code>7b71075</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/245"">#245</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/6280868a8b944193e72436bb467749a21b790d7c""><code>6280868</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/244"">#244</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/83efc37a18b2aea999985439841d644833cb9b90""><code>83efc37</code></a> Upgrade requirements (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/243"">#243</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/7aedc23ae3fa040c30e122d8beb66928a05d5320""><code>7aedc23</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/242"">#242</a>)</li>
<li><a href=""https://github.com/adamchainz/blacken-docs/commit/d78d593d9a523c8d82d234b94d8ced690ee55fe6""><code>d78d593</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/asottile/blacken-docs/issues/241"">#241</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/asottile/blacken-docs/compare/1.13.0...1.14.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=blacken-docs&package-manager=pip&previous-version=1.13.0&new-version=1.14.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2083.org.readthedocs.build/en/2083/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1590531892,IC_kwDOCGYnMM5ezZc0,,7908073,2023-06-14T06:09:21Z,"I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help

```
CREATE TABLE my_table (
    id INTEGER PRIMARY KEY,
    column1 TEXT,
    column2 TEXT,
    column3 TEXT
);

INSERT INTO my_table (column1, column2, column3)
VALUES
    ('Value 1', 'Duplicate 1', 'Duplicate A'),
    ('Value 2', 'Duplicate 2', 'Duplicate B'),
    ('Value 3', 'Duplicate 2', 'Duplicate C'),
    ('Value 4', 'Duplicate 3', 'Duplicate D'),
    ('Value 5', 'Duplicate 3', 'Duplicate E'),
    ('Value 6', 'Duplicate 3', 'Duplicate F');
```

```
library dedupe-db test.db my_table --bk column2
```",140912432
issues,1754174496,I_kwDOCGYnMM5ojpQg,Ability to define unique columns when creating a table,1910303,2023-06-13T06:56:19Z,"When creating a new table, it would be good to have an option to set unique columns similar to how not_null is set.

```python
from sqlite_utils import Database

columns = {""mRID"": str, ""name"": str}
db = Database(""example.db"")
db[""ExampleTable""].create(columns, pk=""mRID"", not_null=[""mRID""], if_not_exists=True)
db[""ExampleTable""].create_index([""mRID""], unique=True, if_not_exists=True)
```

So something like this would add the UNIQUE flag to the table definition. 

```python
db[""ExampleTable""].create(columns, pk=""mRID"", not_null=[""mRID""], unique=[""mRID""], if_not_exists=True)
```

```sql
CREATE TABLE ExampleTable (
    mRID TEXT PRIMARY KEY
              NOT NULL
              UNIQUE,
    name TEXT
);
```",140912432
issues,1751214236,I_kwDOC8SPRc5oYWic,Getting sqlite_master may not be modified when creating dogsheep index,8711912,2023-06-11T03:21:53Z,"When creating a `dogsheep` index from `config.yml` file on pocket.db (created using pocket-to-sqlite), I am getting this error

```
Traceback (most recent call last):
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/bin/dogsheep-beta"", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/cli.py"", line 36, in index
    run_indexer(
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py"", line 32, in run_indexer
    ensure_table_and_indexes(db, tokenize)
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/dogsheep_beta/utils.py"", line 91, in ensure_table_and_indexes
    table.add_foreign_key(*fk)
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py"", line 2155, in add_foreign_key
    self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
  File ""/Users/khushmeeet/.pyenv/versions/3.11.2/lib/python3.11/site-packages/sqlite_utils/db.py"", line 1116, in add_foreign_keys
    cursor.execute(
sqlite3.OperationalError: table sqlite_master may not be modified
```

Command I ran to get this error
```
dogsheep-beta index pocket.db config.yml
```

Dogsheep version
```
dogsheep-beta, version 0.10.2
```

Python version 
```
Python 3.11.2
```",197431109
issue_comments,1585149909,IC_kwDOBm6k_c5ee3fV,,9020979,2023-06-09T21:35:00Z,"Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.

> Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

Ah, I understand now! Thanks for explaining. ",107914493
issue_comments,1578840450,IC_kwDOCGYnMM5eGzGC,,392720,2023-06-06T14:09:04Z,"I also ran into this recently. See below for a patch for one possible solution (tested via ""it works on my machine"", but I don't expect that this behavior would vary a whole lot across terminal emulators and shells). Another possible solution might be to subclass click's `ProgressBar` to keep the logic within the original context manager. Happy to send a PR or for this patch to serve as the basis for a fix that someone else authors.

```patch
diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
index 06c1a4c..530a3a3 100644
--- a/sqlite_utils/utils.py
+++ b/sqlite_utils/utils.py
@@ -147,14 +147,23 @@ def decode_base64_values(doc):
 
 
 class UpdateWrapper:
-    def __init__(self, wrapped, update):
+    def __init__(self, wrapped, update, render_finish):
         self._wrapped = wrapped
         self._update = update
+        self._render_finish = render_finish
 
     def __iter__(self):
-        for line in self._wrapped:
-            self._update(len(line))
-            yield line
+        return self
+
+    def __next__(self):
+        try:
+            line = next(self._wrapped)
+        except StopIteration as e:
+            self._render_finish()
+            raise
+
+        self._update(len(line))
+        return line
 
     def read(self, size=-1):
         data = self._wrapped.read(size)
@@ -178,7 +187,7 @@ def file_progress(file, silent=False, **kwargs):
     else:
         file_length = os.path.getsize(file.name)
         with click.progressbar(length=file_length, **kwargs) as bar:
-            yield UpdateWrapper(file, bar.update)
+            yield UpdateWrapper(file, bar.update, bar.render_finish)
 
 
 class Format(enum.Enum):

```",140912432
issue_comments,1577355134,IC_kwDOCGYnMM5eBId-,,7908073,2023-06-05T19:26:26Z,"this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts",140912432
issues,1740150327,I_kwDOCGYnMM5nuJY3,Aliased ROWID option for tables created from alter=True commands,7908073,2023-06-04T05:29:28Z,"> If you use INTEGER PRIMARY KEY column, the VACUUM does not change the values of that column. However, if you use unaliased rowid, the VACUUM command will reset the rowid values.

ROWID should never be used with foreign keys but the simple act of aliasing rowid to id (which is what happens when one does `id integer primary key` DDL) makes it OK.

It would be convenient if there were more options to use a string column (eg. filepath) as the PK, and be able to use it during upserts, but when creating a foreign key, to create an integer column which aliases rowid

I made an attempt to switch to integer primary keys here but it is not going well... In my usecase the path column is a business key. Yes, it should be as simple as including the `id` column in any select statement where I plan on using `upsert` but it would be nice if this could be abstracted away somehow  https://github.com/chapmanjacobd/library/commit/788cd125be01d76f0fe2153335d9f6b21db1343c

https://github.com/chapmanjacobd/library/actions/runs/5173602136/jobs/9319024777",140912432
issue_comments,1575310378,IC_kwDOCGYnMM5d5VQq,,601708,2023-06-04T01:21:15Z,"I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).",140912432
issues,1740026046,I_kwDOCGYnMM5ntrC-,Support storing incrementally piped values,601708,2023-06-04T00:45:23Z,"I'm trying to use sqlite-utils to data generated incrementally. There are a few
aspects of this that I don't currently know how to handle. I would like an option
to apply writes incrementally, line-by-line as they are received. I would like an
option to echo incremental progress. And, it would be nice to have 

In particular, I'm using CoreLocationCLI -w -j to generate, newline-delimited JSON.

One variant of the command 

`stdbuf -oL CoreLocationCLI -w -j | pee 'sqlite-utils insert loc.db loc -' nl`

`pee`, from `moreutils`, is like `tee` but spawns and pipes to the processes
created by invoking each of its arguments, so, for gratuitous demonstration,
`pee 'sponge out.log' cat` would behave like `tee`.

It looks like I can get what I want with:
`stdbuf -oL CoreLocationCLI -w -j | while read line; do <<<""$line"" sqlite-utils insert loc.db loc -; echo ""$line""; done | nl`




",140912432
issues,1734786661,PR_kwDOBm6k_c5R0fcK,Catch query interrupted on facet suggest row count,10843208,2023-05-31T18:42:46Z,"Just like facet's `suggest()` is trapping `QueryInterrupted` for facet columns, we also need to trap `get_row_count()`, which can reach timeout if database tables are big enough. 

I've included `get_columns()` inside the block as that's just another query, despite it's a really cheap one and might never raise the exception.

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2082.org.readthedocs.build/en/2082/

<!-- readthedocs-preview datasette end -->",107914493
issues,1733198948,I_kwDOCGYnMM5nToRk,Filter table by a large bunch of ids,10843208,2023-05-31T00:29:51Z,"Hi! this might be a question related to both SQLite & sqlite-utils, and you might be more experienced with them.

I have a large bunch of ids, and I'm wondering which is the best way to query them in terms of performance, and simplicity if possible.

The naive approach would be something like `select * from table where rowid in (?, ?, ?...)` but that wouldn't scale if ids are >1k.

Another approach might be creating a temp table, or in-memory db table, insert all ids in that table and then join with the target one.

I failed to attach an in-memory db both using sqlite-utils, and plain sql's execute(), so my closest approach is something like,

```python
def filter_existing_video_ids(video_ids):
    db = get_db()  # contains a ""videos"" table
    db.execute(""CREATE TEMPORARY TABLE IF NOT EXISTS tmp (video_id TEXT NOT NULL PRIMARY KEY)"")
    db[""tmp""].insert_all([{""video_id"": video_id} for video_id in video_ids])
    for row in db[""tmp""].rows_where(""video_id not in (select video_id from videos)""):
        yield row[""video_id""]
    db[""tmp""].drop()
```

That kinda worked, I couldn't find an option in sqlite-utils's `create_table()` to tell it's a temporary table. Also, `tmp` table is not dropped finally, neither using `.drop()` despite being created with the keyword `TEMPORARY`. I believe it should be automatically dropped after connection/session ends though I read.",140912432
issue_comments,1565058994,IC_kwDOBm6k_c5dSOey,,9599,2023-05-26T23:13:02Z,"I should have an extra called `extra_html_context` which bundles together all of the weird extra stuff needed by the HTML template, and is then passed as the root context when the template is rendered (with the other stuff from extras patched into it).",107914493
issues,1727478903,I_kwDOBm6k_c5m9zx3,Update Endpoints defined in metadata throws 403 Forbidden after a while,15085007,2023-05-26T11:52:30Z,"Hello. I expose an endpoint to update `tasks`:
```
{
    ""title"": ""My Datasette Instance"",
    ""databases"": {
        ""tasks"": {
            ""queries"": {
                ""update_task"": {
                    ""sql"": ""UPDATE tasks SET status = :status, result = :result, systemMessage = :systemMessage WHERE queueID = :queueID"",
                    ""write"": true,
                    ""on_success_message"": ""Task updated"",
                    ""on_success_redirect"": ""/tasks/tasks.json"",
                    ""on_error_message"": ""Task update failed"",
                    ""on_error_redirect"": ""/tasks.json"",
                    ""params"": [""queueID"", ""taskData"", ""status"", ""result"", ""systemMessage""]
                }
            }
        }
    }
}
```

This works really well! But after a while, the Datasette Instanz answers with **403 Forbidden**.
I have to delete the database and recreate it in order to work again.

Any help here? (´。＿。｀)",107914493
issue_comments,1563793781,IC_kwDOBm6k_c5dNZl1,,9599,2023-05-26T04:27:55Z,"I should split out a `canned_query.html` template too, as something that extends the `query.html` template.",107914493
issue_comments,1563667574,IC_kwDOBm6k_c5dM6x2,,9599,2023-05-26T00:40:22Z,"Or maybe...

- `BaseQueryView(View)` - knows how to render the results of a SQL query
- `QueryView(BaseQueryView)` - renders from `?sql=`
- `CannedQueryView(BaseQueryView)` - renders for a named canned query

And then later perhaps:

- `RowQueryView(BaseQueryView)` - renders the `select * from t where pk = ?`
- `TableQueryView(BaseQueryView)` - replaces the super complex existing `TableView`",107914493
issue_comments,1563663925,IC_kwDOBm6k_c5dM541,,9599,2023-05-26T00:32:47Z,"I'm going to entirely split canned queries off from `?sql=` queries - they share a bunch of code right now which is just making everything much harder to follow.

I'll refactor their shared bits into functions that they both call.

Or _maybe_ I'll try having `CannedQueryView` as a subclass of `QueryView`.",107914493
issue_comments,1563663616,IC_kwDOBm6k_c5dM50A,,9599,2023-05-26T00:32:08Z,"Now that I have the new `View` subclass from #2078 I want to use it to simplify this code.

Challenge: there are several things to consider here:

- The `/db` page without `?sql=` displays a list of tables in that database
- With `?sql=` it shows the query results for that query (or an error)
- If it's a `/db/name-of-canned-query` it works a bit like the query page, but executes a canned query instead of the `?sql=` query
- POST `/db/name-of-canned-query` is support for writable canned queries",107914493
issue_comments,1563650990,IC_kwDOBm6k_c5dM2uu,,9599,2023-05-26T00:08:59Z,"I'm not going to document this yet, I want to let it bake for a bit longer first.",107914493
issue_comments,1563629348,IC_kwDOBm6k_c5dMxck,,22429695,2023-05-25T23:31:10Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`95.45`**% and project coverage change: **`+0.01`** :tada:
> Comparison is base [(`b49fa44`)](https://app.codecov.io/gh/simonw/datasette/commit/b49fa446d683ddcaf6faf2944dacc0d866bf2d70?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`df5fd73`)](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41%.

> :exclamation: Current head df5fd73 differs from pull request most recent head e990fbc. Consider uploading reports for the commit e990fbc to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2080      +/-   ##
==========================================
+ Coverage   92.40%   92.41%   +0.01%     
==========================================
  Files          39       39              
  Lines        5768     5790      +22     
==========================================
+ Hits         5330     5351      +21     
- Misses        438      439       +1     
```


| [Impacted Files](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/base.py](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.97% <95.45%> (+0.18%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1563626231,IC_kwDOBm6k_c5dMwr3,,9599,2023-05-25T23:25:17Z,I'm going to try using this for the `/-/patterns` page.,107914493
issue_comments,1563625093,IC_kwDOBm6k_c5dMwaF,,9599,2023-05-25T23:23:15Z,"Rest of the work on this will happen in the PR:
- #2080",107914493
issues,1726603778,PR_kwDOBm6k_c5RYvTU,New View base class,9599,2023-05-25T23:22:55Z,"Refs:

- #2078

TODO:

- [x] Teach router layer how to handle this
- [x] Use it for something

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2080.org.readthedocs.build/en/2080/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1563607291,IC_kwDOBm6k_c5dMsD7,,9599,2023-05-25T22:56:28Z,Wrote this up as a TIL: https://til.simonwillison.net/http/testing-cors-max-age,107914493
issue_comments,1563597589,IC_kwDOBm6k_c5dMpsV,,9599,2023-05-25T22:42:07Z,"Mystery solved as to why I wasn't seeing this work:

<img width=""951"" alt=""CleanShot 2023-05-25 at 15 38 32@2x"" src=""https://github.com/simonw/datasette/assets/9599/140ee046-69c2-4a68-b7cb-a4603552477c"">

I had ""Disable Cache"" checked!

I ran this experiment after un-checking that box:

```javascript
fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// And run it again
fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// Now try a thing that doesn't serve that max-age header yet:
fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// And a second time but within 5s
fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// Third time after waiting longer than 5s
fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// Try that original one again - still within the 1hr cache time
fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
```
The results show that the cache of 1hr was being obeyed for `latest.datasette.io` while the `latest-with-plugins.datasette.io` default cache of 5s was being obeyed too.

<img width=""904"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/39609d24-ef68-4db0-905a-451ed2462859"">
",107914493
issue_comments,1563588199,IC_kwDOBm6k_c5dMnZn,,9599,2023-05-25T22:29:47Z,"https://fetch.spec.whatwg.org/#http-access-control-max-age says:

> Indicates the number of seconds (5 by default) the information provided by the [Access-Control-Allow-Methods](https://fetch.spec.whatwg.org/#http-access-control-allow-methods) and [Access-Control-Allow-Headers](https://fetch.spec.whatwg.org/#http-access-control-allow-headers) [headers](https://fetch.spec.whatwg.org/#concept-header) can be cached.

So there was already a 5s cache anyway.",107914493
issue_comments,1563587230,IC_kwDOBm6k_c5dMnKe,,9599,2023-05-25T22:28:20Z,"Weird... after the deploy went out:

<img width=""945"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/a8abb12a-446b-4970-8e76-3b218b6d02c7"">

But the request did indeed get the new header:

<img width=""951"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/7c72578c-9250-47d3-a4d2-1ece8c07ef84"">

So I'm not sure why it's making multiple `POST` requests like that.

Maybe it's because the attempted `POST` failed with a 404?",107914493
issue_comments,1563565407,IC_kwDOBm6k_c5dMh1f,,9599,2023-05-25T22:09:53Z,Updated docs: https://docs.datasette.io/en/latest/json_api.html#enabling-cors,107914493
issue_comments,1563563438,IC_kwDOBm6k_c5dMhWu,,9599,2023-05-25T22:08:28Z,"I ran this on https://www.example.com/ twice using the console:
```javascript
fetch(
  `https://latest.datasette.io/ephemeral/foo/1/-/update`,
  {
    method: ""POST"",
    mode: ""cors"",
    headers: {
      Authorization: `Bearer tok`,
      ""Content-Type"": ""application/json"",
    },
    body: JSON.stringify({update: {blah: 1}}),
  }
)
  .then((r) => r.json())
  .then((data) => {
    console.log(data);
  });
```
And got this in the network pane:

<img width=""945"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/cc767917-388b-4616-b35d-efaa1591948a"">
",107914493
issue_comments,1563558915,IC_kwDOBm6k_c5dMgQD,,9599,2023-05-25T22:04:41Z,I'm going with 3600 for 1 hour instead of 600 for 10 minutes.,107914493
issue_comments,1563547097,IC_kwDOBm6k_c5dMdXZ,,9599,2023-05-25T21:51:38Z,"Also need to update this documentation:

https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/docs/json_api.rst?plain=1#L453-L465

Or maybe make that automated via `cog`.",107914493
issues,1726531350,I_kwDOBm6k_c5m6McW,Datasette should serve Access-Control-Max-Age,9599,2023-05-25T21:50:50Z,"Currently the CORS headers served are:

https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/datasette/utils/__init__.py#L1139-L1143

Serving `Access-Control-Max-Age: 600` would allow browsers to cache that for 10 minutes, avoiding additional CORS pre-flight OPTIONS requests during that time.",107914493
issue_comments,1563522011,IC_kwDOBm6k_c5dMXPb,,9599,2023-05-25T21:22:30Z,"This is bad:
```python
    async def __call__(self, request, datasette):
        try:
            handler = getattr(self, request.method.lower())
            return await handler(request, datasette)
        except AttributeError:
            return await self.method_not_allowed(request)
```
Because it hides any `AttributeError` exceptions that might occur in the view code.",107914493
issue_comments,1563511171,IC_kwDOBm6k_c5dMUmD,,9599,2023-05-25T21:11:20Z,I'm going to call this `VerbView` for the moment. Might even rename it to `View` later.,107914493
issue_comments,1563498048,IC_kwDOBm6k_c5dMRZA,,9599,2023-05-25T20:57:52Z,"Here's a new `BaseView` class that automatically populates `OPTIONS` based on available methods:
```python
class BaseView:
    async def head(self, *args, **kwargs):
        try:
            response = await self.get(*args, **kwargs)
            response.body = b""""
            return response
        except AttributeError:
            raise

    async def method_not_allowed(self, request):
        if (
            request.path.endswith("".json"")
            or request.headers.get(""content-type"") == ""application/json""
        ):
            response = Response.json(
                {""ok"": False, ""error"": ""Method not allowed""}, status=405
            )
        else:
            response = Response.text(""Method not allowed"", status=405)
        return response

    async def options(self, request, *args, **kwargs):
        response = Response.text(""ok"")
        response.headers[""allow""] = "", "".join(
            method.upper()
            for method in (""head"", ""get"", ""post"", ""put"", ""patch"", ""delete"")
            if hasattr(self, method)
        )
        return response

    async def __call__(self, request, datasette):
        try:
            handler = getattr(self, request.method.lower())
            return await handler(request, datasette)
        except AttributeError:
            return await self.method_not_allowed(request)


class DemoView(BaseView):
    async def get(self, datasette, request):
        return Response.text(""Hello there! {} - {}"".format(datasette, request))

    post = get
```",107914493
issue_comments,1563488929,IC_kwDOBm6k_c5dMPKh,,9599,2023-05-25T20:48:12Z,"Actually no need for that extra level of parameter detection: `BaseView.__call__` should _always_ take `datasette, request` - `scope` and `receive` are both available on `request`, and `send` is only needed if you're not planning on returning a `Response` object.

So the `get` and `post` and suchlike methods should take `datasette` and `request` too.",107914493
issue_comments,1563444296,IC_kwDOBm6k_c5dMERI,,9599,2023-05-25T20:06:08Z,"This prototype seems to work well:

```diff
diff --git a/datasette/app.py b/datasette/app.py
index d7dace67..ed0edf28 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -17,6 +17,7 @@ import secrets
 import sys
 import threading
 import time
+import types
 import urllib.parse
 from concurrent import futures
 from pathlib import Path
@@ -1266,6 +1267,8 @@ class Datasette:
         # TODO: /favicon.ico and /-/static/ deserve far-future cache expires
         add_route(favicon, ""/favicon.ico"")
 
+        add_route(wrap_view(DemoView, self), '/demo')
+
         add_route(
             asgi_static(app_root / ""datasette"" / ""static""), r""/-/static/(?P<path>.*)$""
         )
@@ -1673,8 +1676,46 @@ def _cleaner_task_str(task):
     return _cleaner_task_str_re.sub("""", s)
 
 
-def wrap_view(view_fn, datasette):
-    @functools.wraps(view_fn)
+class DemoView:
+    async def __call__(self, datasette, request):
+        return Response.text(""Hello there! {} - {}"".format(datasette, request))
+
+def wrap_view(view_fn_or_class, datasette):
+    is_function = isinstance(view_fn_or_class, types.FunctionType)
+    if is_function:
+        return wrap_view_function(view_fn_or_class, datasette)
+    else:
+        if not isinstance(view_fn_or_class, type):
+            raise ValueError(""view_fn_or_class must be a function or a class"")
+        return wrap_view_class(view_fn_or_class, datasette)
+
+
+def wrap_view_class(view_class, datasette):
+    async def async_view_for_class(request, send):
+        instance = view_class()
+        if inspect.iscoroutinefunction(instance.__call__):
+            return await async_call_with_supported_arguments(
+                instance.__call__,
+                scope=request.scope,
+                receive=request.receive,
+                send=send,
+                request=request,
+                datasette=datasette,
+            )
+        else:
+            return call_with_supported_arguments(
+                instance.__call__,
+                scope=request.scope,
+                receive=request.receive,
+                send=send,
+                request=request,
+                datasette=datasette,
+            )
+
+    return async_view_for_class
+
+
+def wrap_view_function(view_fn, datasette):
     async def async_view_fn(request, send):
         if inspect.iscoroutinefunction(view_fn):
             response = await async_call_with_supported_arguments(
```",107914493
issue_comments,1563419066,IC_kwDOBm6k_c5dL-G6,,9599,2023-05-25T19:42:16Z,"Maybe what I want here is the ability to register classes with the router - and have the router know that if it's a class it should instantiate it via its constructor and then await `__call__` it.

The neat thing about it is that it can reduce the risk of having a class instance that accidentally shares state between requests.

It also encourages that each class only responds based on the `datasette, request, ...` objects that are passed to its methods.",107914493
issue_comments,1563359114,IC_kwDOBm6k_c5dLveK,,9599,2023-05-25T18:47:57Z,"Oops, that broke everything:
```
    @documented
    async def await_me_maybe(value: typing.Any) -> typing.Any:
        ""If value is callable, call it. If awaitable, await it. Otherwise return it.""
>       if callable(value):
E       TypeError: 'module' object is not callable
```",107914493
issue_comments,1563329245,IC_kwDOBm6k_c5dLoLd,,9599,2023-05-25T18:26:47Z,"With type hints and a namedtuple:
```python
import asyncio
import types
from typing import NamedTuple, Any


class CallableStatus(NamedTuple):
    is_callable: bool
    is_async_callable: bool


def check_callable(obj: Any) -> CallableStatus:
    if not callable(obj):
        return CallableStatus(False, False)

    if isinstance(obj, type):
        # It's a class
        return CallableStatus(True, False)

    if isinstance(obj, types.FunctionType):
        return CallableStatus(True, asyncio.iscoroutinefunction(obj))

    if hasattr(obj, ""__call__""):
        return CallableStatus(True, asyncio.iscoroutinefunction(obj.__call__))

    assert False, ""obj {} is somehow callable with no __call__ method"".format(repr(obj))
```
```python
for thing in (
    async_func,
    non_async_func,
    AsyncClass(),
    NotAsyncClass(),
    ClassNoCall(),
    AsyncClass,
    NotAsyncClass,
    ClassNoCall,
):
    print(thing, check_callable(thing))
```
```
<function async_func at 0x1073d5120> CallableStatus(is_callable=True, is_async_callable=True)
<function non_async_func at 0x1073d5080> CallableStatus(is_callable=True, is_async_callable=False)
<__main__.AsyncClass object at 0x106ba7490> CallableStatus(is_callable=True, is_async_callable=True)
<__main__.NotAsyncClass object at 0x106740150> CallableStatus(is_callable=True, is_async_callable=False)
<__main__.ClassNoCall object at 0x10676d910> CallableStatus(is_callable=False, is_async_callable=False)
<class '__main__.AsyncClass'> CallableStatus(is_callable=True, is_async_callable=False)
<class '__main__.NotAsyncClass'> CallableStatus(is_callable=True, is_async_callable=False)
<class '__main__.ClassNoCall'> CallableStatus(is_callable=True, is_async_callable=False)
```",107914493
issue_comments,1563326000,IC_kwDOBm6k_c5dLnYw,,9599,2023-05-25T18:23:38Z,I don't like that `is_callable()` implies a single boolean result but actually returns a pair. I'll call it `check_callable(obj)` instead.,107914493
issue_comments,1563318598,IC_kwDOBm6k_c5dLllG,,9599,2023-05-25T18:17:03Z,"I think I want that to return `(is_callable, is_async)` - so I can both test if the thing can be called AND if it should be awaited in the same operation (without any exceptions).

I tried this:
```python
def is_callable(obj):
    ""Returns (is_callable, is_async_callable)""
    if not callable(obj):
        return False, False

    if isinstance(obj, types.FunctionType):
        return True, asyncio.iscoroutinefunction(obj)
        
    if hasattr(obj, '__call__'):
        return True, asyncio.iscoroutinefunction(obj.__call__)

    return False, False
```
```python
for thing in (
    async_func, non_async_func, AsyncClass(), NotAsyncClass(), ClassNoCall(),
    AsyncClass, NotAsyncClass, ClassNoCall
):
    print(thing, is_callable(thing))
```
And got:
```
<function async_func at 0x1073d5120> (True, True)
<function non_async_func at 0x1073d5080> (True, False)
<__main__.AsyncClass object at 0x106cce490> (True, True)
<__main__.NotAsyncClass object at 0x106ccf710> (True, False)
<__main__.ClassNoCall object at 0x106ccc810> (False, False)
<class '__main__.AsyncClass'> (True, True)
<class '__main__.NotAsyncClass'> (True, False)
<class '__main__.ClassNoCall'> (True, False)
```
Which is almost right, but I don't like that `AsyncClass` is shown as callable (which it is, since it's a class) and awaitable (which it is not - the `__call__` method may be async but calling the class constructor is not).

So I'm going to detect classes using `isinstance(obj, type)`.

```python
def is_callable(obj):
    ""Returns (is_callable, is_async_callable)""
    if not callable(obj):
        return False, False

    if isinstance(obj, type):
        # It's a class
        return True, False
    
    if isinstance(obj, types.FunctionType):
        return True, asyncio.iscoroutinefunction(obj)
        
    if hasattr(obj, '__call__'):
        return True, asyncio.iscoroutinefunction(obj.__call__)

    assert False, ""obj {} somehow is callable with no __call__ method"".format(obj)
```
I am reasonably confident the `AssertionError` can never be raised.

And now:
```
<function async_func at 0x1073d5120> (True, True)
<function non_async_func at 0x1073d5080> (True, False)
<__main__.AsyncClass object at 0x106ccfa50> (True, True)
<__main__.NotAsyncClass object at 0x106ccc8d0> (True, False)
<__main__.ClassNoCall object at 0x106cd7690> (False, False)
<class '__main__.AsyncClass'> (True, False)
<class '__main__.NotAsyncClass'> (True, False)
<class '__main__.ClassNoCall'> (True, False)
```
Which is what I wanted.",107914493
issue_comments,1563308919,IC_kwDOBm6k_c5dLjN3,,9599,2023-05-25T18:08:34Z,"After much fiddling this seems to work:
```python
import asyncio, types

def is_async_callable(obj):
    if not callable(obj):
        raise ValueError(""Object is not callable"")

    if isinstance(obj, types.FunctionType):
        return asyncio.iscoroutinefunction(obj)
        
    if hasattr(obj, '__call__'):
        return asyncio.iscoroutinefunction(obj.__call__)

    raise ValueError(""Not a function and has no __call__ attribute"")
```
Tested like so:
```python
class AsyncClass:
    async def __call__(self):
        pass

class NotAsyncClass:
    def __call__(self):
        pass

class ClassNoCall:
    pass
    
async def async_func():
    pass

def non_async_func():
    pass

for thing in (AsyncClass(), NotAsyncClass(), ClassNoCall(), async_func, non_async_func):
    try:
        print(thing, is_async_callable(thing))
    except Exception as ex:
        print(thing, ex)
```
```
<__main__.AsyncClass object at 0x106c32150> True
<__main__.NotAsyncClass object at 0x106c32390> False
<__main__.ClassNoCall object at 0x106c32750> Object is not callable
<function async_func at 0x1073d5120> True
<function non_async_func at 0x1073d5080> False
```",107914493
issue_comments,1563294669,IC_kwDOBm6k_c5dLfvN,,9599,2023-05-25T17:57:06Z,"I may need to be able to detect if a class instance has an `async def __call__` method - I think I can do that like so:

```python
def iscoroutinefunction(obj):
    if inspect.iscoroutinefunction(obj):
        return True
    if hasattr(obj, '__call__') and inspect.iscoroutinefunction(obj.__call__):
        return True
    return False
```
From https://github.com/encode/starlette/issues/886#issuecomment-606585152",107914493
issue_comments,1563292373,IC_kwDOBm6k_c5dLfLV,,9599,2023-05-25T17:55:12Z,"So I think subclasses of `BaseView` need to offer a callable which accepts all five of the DI arguments - `datasette`, `request`, `scope`, `send`, `receive` - and then makes a decision based on the HTTP verb as to which method of the class to call. Those methods themselves can accept a subset of those parameters and will only be sent on to them.

Having two layers of parameter detection feels a little bit untidy, but I think it will work.",107914493
issue_comments,1563285150,IC_kwDOBm6k_c5dLdae,,9599,2023-05-25T17:48:50Z,"Uncommitted experimental code:
```diff
diff --git a/datasette/views/database.py b/datasette/views/database.py
index 455ebd1f..85775433 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -909,12 +909,13 @@ async def query_view(
     elif format_ in datasette.renderers.keys():
         # Dispatch request to the correct output format renderer
         # (CSV is not handled here due to streaming)
+        print(data)
         result = call_with_supported_arguments(
             datasette.renderers[format_][0],
             datasette=datasette,
-            columns=columns,
-            rows=rows,
-            sql=sql,
+            columns=data[""rows""][0].keys(),
+            rows=data[""rows""],
+            sql='',
             query_name=None,
             database=db.name,
             table=None,
@@ -923,7 +924,7 @@ async def query_view(
             # These will be deprecated in Datasette 1.0:
             args=request.args,
             data={
-                ""rows"": rows,
+                ""rows"": data[""rows""],
             },  # TODO what should this be?
         )
         result = await await_me_maybe(result)
diff --git a/docs/index.rst b/docs/index.rst
index 5a9cc7ed..254ed3da 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -57,6 +57,7 @@ Contents
    settings
    introspection
    custom_templates
+   template_context
    plugins
    writing_plugins
    plugin_hooks
```
Where `docs/template_context.rst` looked like this:
```rst
.. _template_context:

Template context
================

.. currentmodule:: datasette.context

This page describes the variables made available to templates used by Datasette to render different pages of the application.

.. autoclass:: QueryContext
    :members:
```
And `datasette/context.py` had this:
```python
from dataclasses import dataclass

@dataclass
class QueryContext:
    """"""
    Used by the ``/database`` page when showing the results of a SQL query
    """"""
    id: int
    ""Id is a thing""
    rows: list[dict]
    ""Name is another thing""
```",107914493
issue_comments,1563283939,IC_kwDOBm6k_c5dLdHj,,9599,2023-05-25T17:47:38Z,"The idea behind `wrap_view()` is dependency injection - it's mainly used by plugins:

https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-routes-datasette

But I like the pattern so I started using it for some of Datasette's own features.

I should use it for _all_ of Datasette's own features.

But I still like the way `BaseView` helps with running different code for GET/POST/etc verbs.",107914493
issue_comments,1563282327,IC_kwDOBm6k_c5dLcuX,,9599,2023-05-25T17:46:05Z,"Here's what `wrap_view()` does:

https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1676-L1700

It's used e.g. here:

https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1371-L1375

The `BaseView` thing meanwhile works like this:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/views/base.py#L56-L157",107914493
issues,1726236847,I_kwDOBm6k_c5m5Eiv,Resolve the difference between `wrap_view()` and `BaseView`,9599,2023-05-25T17:44:32Z,"There are two patterns for implementing views in Datasette at the moment. I want to combine those.

Part of:
- #2053",107914493
issue_comments,1557607516,IC_kwDOCGYnMM5c1zRc,,1231935,2023-05-22T17:18:33Z,"Oh and for context - this goes away if I use `.upsert` instead of `insert(..., ignore=True)`, but I don't want to update the value if it's written, just do an insert if it's new. The code is basically:

```py
def save_items(table, items):
    db[""users""].insert(build_user(items[0]), pk=""id"",ignore=True)
    db[table].insert_all(items)

if comments := fetch_comments():
    save_items('comments', comments)

if posts := fetch_posts():
    save_items('posts', posts)
```

So either `comments` or `post` could create the relevant user if those items exist. In cases where they _both_ exist, I get this error. I need the `pk` because either call could create the table.",140912432
issues,1720096994,I_kwDOCGYnMM5mhpji,"`IndexError` when doing `.insert(..., pk='id')` after `insert_all`",1231935,2023-05-22T17:13:02Z,"I believe this is related to https://github.com/simonw/sqlite-utils/issues/98.

When `pk` is specified by table A's `insert` call, it throws an index error if a different table has written a row with a higher rowid than exists in the first table. Here's a basic example:

```py
from sqlite_utils import Database


def test_pk_for_insert(fresh_db):
    user = {""id"": ""abc"", ""name"": ""david""}

    fresh_db[""users""].insert(user, pk=""id"")

    fresh_db[""comments""].insert_all(
        [
            {""id"": ""def"", ""text"": ""ok""},
            {""id"": ""ghi"", ""text"": ""great""},
        ],
    )

    fresh_db[""users""].insert(
        user,
        ignore=True,
        # BUG: when specifying pk on the second insert call 
        # db.py goes into a block it doesn't expect and we get the error
        pk=""id"",
    )


if __name__ == ""__main__"":
    db = Database(""bug.db"")
    if db[""users""].exists():
        raise ValueError(
            ""bug only shows on a new database - remove bug.db before running the script""
        )
    test_pk_for_insert(db)
```

The error is:

```py
  File ""/Users/david/projects/reddit-to-sqlite/.venv/lib/python3.11/site-packages/sqlite_utils/db.py"", line 2960, in insert_chunk
    row = list(self.rows_where(""rowid = ?"", [self.last_rowid]))[0]
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
```

The issue is in this block: 

https://github.com/simonw/sqlite-utils/blob/2747257a3334d55e890b40ec58fada57ae8cfbfd/sqlite_utils/db.py#L2954-L2958

relevant locals are:

- `pk`: `'id'`
- `result.lastrowid`: `2`

What's most interesting is the comment `# self.last_rowid will be 0 if a ""INSERT OR IGNORE"" happened`, which doesn't seem to be the case here. ",140912432
issue_comments,1557289070,IC_kwDOBm6k_c5c0lhu,,22429695,2023-05-22T14:08:33Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`ede6203`)](https://app.codecov.io/gh/simonw/datasette/commit/ede62036180993dbd9d4e5d280fc21c183cda1c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`9785c4f`)](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2077   +/-   ##
=======================================
  Coverage   92.40%   92.40%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5362     5362           
  Misses        441      441           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issues,1719759468,PR_kwDOBm6k_c5RBXH_,Bump furo from 2023.3.27 to 2023.5.20,49699333,2023-05-22T13:58:16Z,"Bumps [furo](https://github.com/pradyunsg/furo) from 2023.3.27 to 2023.5.20.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/pradyunsg/furo/blob/main/docs/changelog.md"">furo's changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<!-- raw HTML omitted -->
<h2>2023.05.20 -- Unassuming Ultramarine</h2>
<ul>
<li>✨ Add support for Sphinx 7.</li>
<li>Drop support for Sphinx 5.</li>
<li>Improve the screen-reader label for sidebar collapse.</li>
<li>Make it easier to create derived themes from Furo.</li>
<li>Bump all JS dependencies (NodeJS and npm packages).</li>
</ul>
<h2>2023.03.27 -- Tasty Tangerine</h2>
<ul>
<li>Regenerate with newer version of sphinx-theme-builder, to fix RECORD hashes.</li>
<li>Add missing class to Font Awesome examples</li>
</ul>
<h2>2023.03.23 -- Sassy Saffron</h2>
<ul>
<li>Update Python version classifiers.</li>
<li>Increase the icon size in mobile header.</li>
<li>Increase admonition title bg opacity.</li>
<li>Change the default API background to transparent.</li>
<li>Transition the API background change.</li>
<li>Remove the &quot;indent&quot; of API entries which have a background.</li>
<li>Break long inline code literals.</li>
</ul>
<h2>2022.12.07 -- Reverent Raspberry</h2>
<ul>
<li>✨ Add support for Sphinx 6.</li>
<li>✨ Improve footnote presentation with docutils 0.18+.</li>
<li>Drop support for Sphinx 4.</li>
<li>Improve documentation about what the edit button does.</li>
<li>Improve handling of empty-flexboxes for better print experience on Chrome.</li>
<li>Improve styling for inline signatures.</li>
<li>Replace the <code>meta</code> generator tag with a comment.</li>
<li>Tweak labels with icons to prevent users selecting icons as text on touch.</li>
</ul>
<h2>2022.09.29 -- Quaint Quartz</h2>
<ul>
<li>Add ability to set arbitrary URLs for edit button.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/pradyunsg/furo/commit/d2c9ca852542820632361f05521b5c94824d1fdc""><code>d2c9ca8</code></a> Prepare release: 2023.05.20</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/662d21b66c0422693e3bcdac87e1c605ed95b32f""><code>662d21b</code></a> Update changelog</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/591780b0f01378aef769da40e5d7a917304bd1a5""><code>591780b</code></a> Bump compatible Sphinx version</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/c2e7837352d290f75579b1859233e9c77afc8911""><code>c2e7837</code></a> Bump NodeJS and package versions</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/dd85574e803bb5e9416be66696acc1a6f91f3470""><code>dd85574</code></a> Use the reference HtmlFormatter class defined on PygmentsBridge. (<a href=""https://redirect.github.com/pradyunsg/furo/issues/657"">#657</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/6bff41949f18dc204cfaeae82c42a7197ff3a200""><code>6bff419</code></a> Fix broken link (<a href=""https://redirect.github.com/pradyunsg/furo/issues/654"">#654</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/e7f732ea5133d29fc6ca62af0455b16ffaab00dc""><code>e7f732e</code></a> Improve the screen-reader label for sidebar collapse</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/48c0bf2aa983fb1ccd79b4167fe85387249afedf""><code>48c0bf2</code></a> Drop the check for the theme name</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/1b17d8102079c3ec2a49f57ebc1931f6869cd61d""><code>1b17d81</code></a> [pre-commit.ci] pre-commit autoupdate (<a href=""https://redirect.github.com/pradyunsg/furo/issues/646"">#646</a>)</li>
<li><a href=""https://github.com/pradyunsg/furo/commit/4904fd5ce0c74386eaee3bd07940aa46254a11ce""><code>4904fd5</code></a> Remove Python 3.8 constraint from Black pre-commit config (<a href=""https://redirect.github.com/pradyunsg/furo/issues/647"">#647</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/pradyunsg/furo/compare/2023.03.27...2023.05.20"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=furo&package-manager=pip&previous-version=2023.3.27&new-version=2023.5.20)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2077.org.readthedocs.build/en/2077/

<!-- readthedocs-preview datasette end -->
> **Note**
> Automatic rebases have been disabled on this pull request as it has been open for over 30 days.",107914493
issue_comments,1556292204,IC_kwDOCGYnMM5cwyJs,,9599,2023-05-21T21:05:15Z,Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion,140912432
issue_comments,1556291915,IC_kwDOCGYnMM5cwyFL,,9599,2023-05-21T21:04:03Z,Now live at https://sqlite-utils.datasette.io/en/latest/cli.html,140912432
issue_comments,1556288300,IC_kwDOCGYnMM5cwxMs,,9599,2023-05-21T20:48:01Z,If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.,140912432
issue_comments,1556288270,IC_kwDOCGYnMM5cwxMO,,9599,2023-05-21T20:47:51Z,This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries,140912432
issue_comments,1556287870,IC_kwDOCGYnMM5cwxF-,,22429695,2023-05-21T20:45:58Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`e240133`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/e240133b11588d31dc22c632f7a7ca636c72978d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`0b81794`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.

> :exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #553   +/-   ##
=======================================
  Coverage   96.36%   96.36%           
=======================================
  Files           6        6           
  Lines        2726     2726           
=======================================
  Hits         2627     2627           
  Misses         99       99           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issue_comments,1556287599,IC_kwDOCGYnMM5cwxBv,,9599,2023-05-21T20:44:55Z,"Put this in a PR so I can preview it: 
- #553
",140912432
issues,1718635018,PR_kwDOCGYnMM5Q9lY4,Reformatted CLI examples in docs,9599,2023-05-21T20:44:34Z,"Refs:
- #551

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--553.org.readthedocs.build/en/553/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1556269616,IC_kwDOCGYnMM5cwsow,,9599,2023-05-21T19:33:13Z,Now released: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-32,140912432
issues,1718612569,I_kwDOCGYnMM5mb_JZ,Document how to setup shell auto-completion,9599,2023-05-21T19:20:41Z,"https://click.palletsprojects.com/en/8.1.x/shell-completion/

This works for `zsh`:

    eval ""$(_SQLITE_UTILS_COMPLETE=zsh_source sqlite-utils)""

This will probably work for `bash`:

    eval ""$(_SQLITE_UTILS_COMPLETE=bash_source sqlite-utils)""

Need to add this to the installation docs here: https://sqlite-utils.datasette.io/en/stable/installation.html - along with the pattern for adding that to `.zshrc` or whatever.",140912432
issue_comments,1556265772,IC_kwDOCGYnMM5cwrss,,9599,2023-05-21T19:16:15Z,"Another option:

<img width=""593"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/a9d819a0-89e3-4b67-b2f0-1052b0a7a7c0"">

That's using this markup:

```
Newline-delimited JSON
~~~~~~~~~~~~~~~~~~~~~~

Use ``--nl`` to get back newline-delimited JSON objects:

.. code-block:: bash

    sqlite-utils dogs.db ""select * from dogs"" --nl

.. code-block:: output

    {""id"": 1, ""age"": 4, ""name"": ""Cleo""}
    {""id"": 2, ""age"": 2, ""name"": ""Pancakes""}
```
And this extra CSS:

```css
.highlight-output .highlight {
  border-left: 9px solid #30c94f;
}
```",140912432
issue_comments,1556263182,IC_kwDOCGYnMM5cwrEO,,9599,2023-05-21T19:06:48Z,"I could split them up into two blocks like this:

<img width=""771"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/9b042ac8-db30-49d5-8882-1ea7be272a49"">

I do miss the visual indication that one of these is the command and one is the output though.",140912432
issue_comments,1556262574,IC_kwDOCGYnMM5cwq6u,,9599,2023-05-21T19:04:59Z,"I wrote the docs like this because early examples include both the command and its output:

https://sqlite-utils.datasette.io/en/stable/cli.html#returning-json

<img width=""766"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/a5d05f24-ca0f-4718-a0a8-d85adc736141"">",140912432
issues,1718607907,I_kwDOCGYnMM5mb-Aj,Make as many examples in the CLI docs as possible copy-and-pastable,9599,2023-05-21T19:04:10Z,"e.g. in this section:

https://sqlite-utils.datasette.io/en/stable/cli.html#running-queries-directly-against-csv-or-json

<img width=""771"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/3935111d-441e-4edb-9100-ce210117dbe5"">

The little copy button will also copy the `$ ` which breaks the examples when copied.",140912432
issue_comments,1556255309,IC_kwDOCGYnMM5cwpJN,,9599,2023-05-21T18:42:25Z,Tests passed here: https://github.com/simonw/sqlite-utils/actions/runs/5039119716,140912432
issue_comments,1556250236,IC_kwDOCGYnMM5cwn58,,9599,2023-05-21T18:25:26Z,"Relevant issues:
- https://github.com/python/importlib_metadata/issues/406
- https://github.com/PyCQA/flake8/issues/1701

It looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.",140912432
issue_comments,1556249984,IC_kwDOCGYnMM5cwn2A,,9599,2023-05-21T18:24:48Z,"This is blocking:
- #549",140912432
issues,1718595700,I_kwDOCGYnMM5mb7B0,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7,9599,2023-05-21T18:24:39Z,"https://github.com/simonw/sqlite-utils/actions/runs/5039064797/jobs/9036965488

```
Traceback (most recent call last):
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/bin/flake8"", line 8, in <module>
    sys.exit(main())
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/cli.py"", line 22, in main
    app.run(argv)
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 363, in run
    self._run(argv)
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 350, in _run
    self.initialize(argv)
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 330, in initialize
    self.find_plugins(config_finder)
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/main/application.py"", line 153, in find_plugins
    self.check_plugins = plugin_manager.Checkers(local_plugins.extension)
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py"", line 357, in __init__
    self.namespace, local_plugins=local_plugins
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py"", line 238, in __init__
    self._load_entrypoint_plugins()
  File ""/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/flake8/plugins/manager.py"", line 254, in _load_entrypoint_plugins
    eps = importlib_metadata.entry_points().get(self.namespace, ())
AttributeError: 'EntryPoints' object has no attribute 'get'
Error: Process completed with exit code 1.
```",140912432
issue_comments,1556247818,IC_kwDOCGYnMM5cwnUK,,9599,2023-05-21T18:17:46Z,Draft documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,140912432
issue_comments,1556242262,IC_kwDOCGYnMM5cwl9W,,9599,2023-05-21T18:00:05Z,"Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465
```
sqlite_utils/cli.py:37: error: Skipping analyzing ""trogon"": module is installed, but missing library stubs or py.typed marker  [import]
sqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 52 source files)
```",140912432
issue_comments,1556241812,IC_kwDOCGYnMM5cwl2U,,9599,2023-05-21T17:58:25Z,Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,140912432
issue_comments,1556241555,IC_kwDOCGYnMM5cwlyT,,22429695,2023-05-21T17:57:24Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`83.33`**% and project coverage change: **`+0.06`** :tada:
> Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`948692a`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #549      +/-   ##
==========================================
+ Coverage   96.30%   96.36%   +0.06%     
==========================================
  Files           6        6              
  Lines        2707     2726      +19     
==========================================
+ Hits         2607     2627      +20     
+ Misses        100       99       -1     
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <83.33%> (-0.03%)` | :arrow_down: |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issues,1718586377,PR_kwDOCGYnMM5Q9cAv,TUI powered by Trogon,9599,2023-05-21T17:55:42Z,"Refs:
- #545

<!-- readthedocs-preview sqlite-utils start -->
----
:books: Documentation preview :books:: https://sqlite-utils--549.org.readthedocs.build/en/549/

<!-- readthedocs-preview sqlite-utils end -->",140912432
issue_comments,1556231832,IC_kwDOCGYnMM5cwjaY,,9599,2023-05-21T17:24:13Z,"Oh, I see why that is now:

https://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679

This is because of the following command:

    sqlite-utils analyze-tables table1 table2 --column x

Since you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables.

I'm going to change this so if the column is not on ANY of those tables you get an error.",140912432
issues,1718576761,I_kwDOCGYnMM5mb2Z5,analyze-tables should validate provide --column names,9599,2023-05-21T17:20:24Z,"Noticed this while testing:
- #547 

If you pass a non-existent column to `-c/--column` you don't get an error message.",140912432
issue_comments,1556228395,IC_kwDOCGYnMM5cwikr,,9599,2023-05-21T17:11:15Z,"This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`.

I can add an optimization though to avoid running the SQL count query if we know that it's all `null`.",140912432
issues,1718572201,I_kwDOCGYnMM5mb1Sp,No need to show common values if everything is null,9599,2023-05-21T17:05:07Z,"Noticed this:

```
% sqlite-utils analyze-tables content.db repos -c delete_branch_on_merge --common-limit 20 --no-least
repos.delete_branch_on_merge: (1/1)

  Total rows: 158
  Null rows: 158
  Blank rows: 0

  Distinct values: 0

  Most common:
    158: None
```

The `158: None` there is duplicate information considering we already know there are 158/158 null rows.",140912432
issue_comments,1556225788,IC_kwDOCGYnMM5cwh78,,9599,2023-05-21T17:02:05Z,"New docs:

- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables
- https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column
- https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column

New help output:

```
 % sqlite-utils analyze-tables --help
Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]...

  Analyze the columns in one or more tables

  Example:

      sqlite-utils analyze-tables data.db trees

Options:
  -c, --column TEXT       Specific columns to analyze
  --save                  Save results to _analyze_tables table
  --common-limit INTEGER  How many common values
  --no-most               Skip most common values
  --no-least              Skip least common values
  --load-extension TEXT   Path to SQLite extension, with optional :entrypoint
  -h, --help              Show this message and exit.
```",140912432
issue_comments,1556213396,IC_kwDOCGYnMM5cwe6U,,9599,2023-05-21T15:58:12Z,"Documentation preview:
- https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables
- https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables
- https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column
- https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column",140912432
issue_comments,1556213031,IC_kwDOCGYnMM5cwe0n,,22429695,2023-05-21T15:56:05Z,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`93.75`**% and no project coverage change.
> Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`9f23e68`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.31%.

> :exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #546   +/-   ##
=======================================
  Coverage   96.30%   96.31%           
=======================================
  Files           6        6           
  Lines        2707     2712    +5     
=======================================
+ Hits         2607     2612    +5     
  Misses        100      100           
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.37% <90.90%> (+<0.01%)` | :arrow_up: |
| [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.01%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",140912432
issues,1718550688,PR_kwDOCGYnMM5Q9VH0,"Analyze tables options: --common-limit, --no-most, --no-least",9599,2023-05-21T15:54:39Z,"Refs #544

- [x] Documentation for CLI options
- [x] Documentation for new Python API parameters:  `most_common: bool` and `least_common: bool`
- [x] Tests for CLI
- [x] Tests for Python API",140912432
issue_comments,1556211643,IC_kwDOCGYnMM5cwee7,,9599,2023-05-21T15:48:17Z,I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.,140912432
issue_comments,1556210844,IC_kwDOCGYnMM5cweSc,,9599,2023-05-21T15:44:10Z,"It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here:
- https://github.com/Textualize/trogon/issues/4",140912432
issue_comments,1556191894,IC_kwDOCGYnMM5cwZqW,,9599,2023-05-21T14:20:14Z,"Opened a feature request for customizing the help and command name:

- https://github.com/Textualize/trogon/issues/2",140912432
issue_comments,1556190531,IC_kwDOCGYnMM5cwZVD,,9599,2023-05-21T14:13:43Z,"OK, this works!

![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3)

To try it out, install that branch from GitHub:

    pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip

Then run this:

    sqlite-utils install trogon

And this:

    sqlite-utils tui
",140912432
issue_comments,1556189823,IC_kwDOCGYnMM5cwZJ_,,9599,2023-05-21T14:09:59Z,"I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only.

But I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install

So I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.",140912432
issues,1718517882,I_kwDOCGYnMM5mboB6,Try out Trogon for a tui interface,9599,2023-05-21T14:08:25Z,https://github.com/Textualize/trogon,140912432
issues,1718515590,I_kwDOCGYnMM5mbneG,New options for analyze-tables --common-limit --no-most and --no-least,9599,2023-05-21T14:03:19Z,"The ""least common"" section is frequently uninteresting, especially for huge tables with a large number of repeated-once values.

    sqlite-utils analyze-tables content.db repos --common-limit 20 --no-least",140912432
issues,1715468032,PR_kwDOBm6k_c5QzEAM,Datsette gpt plugin,130708713,2023-05-18T11:22:30Z,"

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2076.org.readthedocs.build/en/2076/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1548913065,IC_kwDOCGYnMM5cUomp,,433780,2023-05-16T03:11:03Z,"Using this thread and some [other resources](https://sqlite-utils.datasette.io/en/stable/cli.html#spatialite-helpers) I managed to cobble together a couple of sqlite-utils lines to add a geometry column for a table that already has a lat/lng column.

```
# add a geometry column
sqlite-utils add-geometry-column [db name] [table name] geometry --type POINT --srid 4326

# add a point for each row to geometry column
sqlite-utils --load-extension=spatialite [db name] 'update [table name] SET Geometry=MakePoint(longitude, latitude, 4326);'
```",140912432
issue_comments,1548617257,IC_kwDOBm6k_c5cTgYp,,193185,2023-05-15T21:32:20Z,"> Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

The latter - that you could return a promise of arrays, so it parallels the [""await me maybe"" pattern in Datasette](https://simonwillison.net/2020/Sep/2/await-me-maybe/), where you can return either a value, a callable or an awaitable.

> I have a hunch that what you're describing might be achievable without adding Promises to the API with something

Oops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

This isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.

This could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.

> especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag

Ah, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.

My experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.

Again, a fairly minor thing, though.",107914493
issue_comments,1547944971,IC_kwDOBm6k_c5cQ8QL,,22429695,2023-05-15T14:12:20Z,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`49184c5`)](https://app.codecov.io/gh/simonw/datasette/commit/49184c569cd70efbda4f3f062afef3a34401d8d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`b99e1d3`)](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2075   +/-   ##
=======================================
  Coverage   92.40%   92.40%           
=======================================
  Files          38       38           
  Lines        5751     5751           
=======================================
  Hits         5314     5314           
  Misses        437      437           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
",107914493
issue_comments,1547911570,IC_kwDOBm6k_c5cQ0GS,,49699333,2023-05-15T13:59:35Z,Superseded by #2075.,107914493
issues,1710164693,PR_kwDOBm6k_c5QhIL2,Bump sphinx from 6.1.3 to 7.0.1,49699333,2023-05-15T13:59:31Z,"Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 7.0.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>v7.0.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0rc1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.0.1 (released May 12, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11411"">#11411</a>: Support <code>Docutils 0.20</code>_. Patch by Adam Turner.</li>
</ul>
<p>.. _Docutils 0.20: <a href=""https://docutils.sourceforge.io/RELEASE-NOTES.html#release-0-20-2023-05-04"">https://docutils.sourceforge.io/RELEASE-NOTES.html#release-0-20-2023-05-04</a></p>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11418"">#11418</a>: Clean up remaining references to <code>sphinx.setup_command</code>
following the removal of support for setuptools.
Patch by Willem Mulder.</li>
</ul>
<h1>Release 7.0.0 (released Apr 29, 2023)</h1>
<h2>Incompatible changes</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11359"">#11359</a>: Remove long-deprecated aliases for <code>MecabSplitter</code> and
<code>DefaultSplitter</code> in <code>sphinx.search.ja</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11360"">#11360</a>: Remove deprecated <code>make_old_id</code> functions in domain object
description classes.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11363"">#11363</a>: Remove the Setuptools integration (<code>build_sphinx</code> hook in
<code>setup.py</code>).</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11364"">#11364</a>: Remove deprecated <code>sphinx.ext.napoleon.iterators</code> module.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11365"">#11365</a>: Remove support for the <code>jsdump</code> format in <code>sphinx.search</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11366"">#11366</a>: Make <code>locale</code> a required argument to
<code>sphinx.util.i18n.format_date()</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11370"">#11370</a>: Remove deprecated <code>sphinx.util.stemmer</code> module.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11371"">#11371</a>: Remove deprecated <code>sphinx.pycode.ast.parse()</code> function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11372"">#11372</a>: Remove deprecated <code>sphinx.io.read_doc()</code> function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11373"">#11373</a>: Removed deprecated <code>sphinx.util.get_matching_files()</code> function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11378"">#11378</a>: Remove deprecated <code>sphinx.util.docutils.is_html5_writer_available()</code>
function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11379"">#11379</a>: Make the <code>env</code> argument to <code>Builder</code> subclasses required.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11380"">#11380</a>: autosummary: Always emit grouped import exceptions.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11381"">#11381</a>: Remove deprecated <code>style</code> key for HTML templates.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11382"">#11382</a>: Remove deprecated <code>sphinx.writers.latex.LaTeXTranslator.docclasses</code>
attribute.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11383"">#11383</a>: Remove deprecated <code>sphinx.builders.html.html5_ready</code> and
<code>sphinx.builders.html.HTMLTranslator</code> attributes.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11385"">#11385</a>: Remove support for HTML 4 output.</li>
</ul>
<h1>Release 6.2.1 (released Apr 25, 2023)</h1>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d48cc783b6e48b99cbbcf5462f44c59a4ed78e6a""><code>d48cc78</code></a> Bump to 7.0.1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/de6b18ac7de317030be4344ea7a9d553c423ffcf""><code>de6b18a</code></a> Remove remaining references to the setuptools integration (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11418"">#11418</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/db546189ce1d2a345f4399367ced6ecdd538be5d""><code>db54618</code></a> Support Docutils 0.20 (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11411"">#11411</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/8942a1dddf2355928f088d6b631db8658034eaae""><code>8942a1d</code></a> Test with Docutils 0.20</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/c9d0933e5d8e34aa9d2c1d88c5a80b46b575730e""><code>c9d0933</code></a> linkcheck: Use context managers for HTTP requests (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11318"">#11318</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/2b1c106bbff5265e8a6076318db5d083c329d575""><code>2b1c106</code></a> Update documentation workflow</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ba7408209e84ee413f240afc20f3c6b484a81f8f""><code>ba74082</code></a> Change concurrency groups for GitHub workflows</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/b546879539200ec4128bcc6d0ed911ebf28bb3cb""><code>b546879</code></a> Bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d568b2f4f7cca743fcbf70814d15602d8129b790""><code>d568b2f</code></a> Bump to 7.0.0 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ff79edf353f5cc6e02036f58e0295dc704c5e681""><code>ff79edf</code></a> Remove <code>jsdump</code> references post removal</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v7.0.1"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=6.1.3&new-version=7.0.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

You can trigger a rebase of this PR by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2075.org.readthedocs.build/en/2075/

<!-- readthedocs-preview datasette end -->
> **Note**
> Automatic rebases have been disabled on this pull request as it has been open for over 30 days.
",107914493
issues,1708981860,PR_kwDOBm6k_c5QdMea,sort files by mtime,3919561,2023-05-14T15:25:15Z,"serving multiple database files and getting tired by the default sort, changes so the sort order puts the latest changed databases to be on top of the list so don't have to scroll down, lazy as i am ;)

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2074.org.readthedocs.build/en/2074/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1546362374,IC_kwDOBm6k_c5cK54G,,9020979,2023-05-12T22:09:03Z,"Hey @cldellow , thanks for the thoughtful feedback and describing the ""lazy facets"" feature! 

It sounds like the [postTask](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/postTask) API might be relevant for the types of network request scheduling you have in mind. 

Addressing your points inline below:

> It might also be nice if the plugins could return Promises.

Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like `makeColumnActions` or `makeAboveTablePanelConfigs` supported returning a promise of arrays instead only returning plain arrays?

I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable . 

I have a hunch that what you're describing might be achievable without adding Promises to the API with something like

```
fetch('/api/with-custom-facets').then(myFacets => {
    // reusing the go() idiom
   go(manager, myFacets);
})
```

but I'd like to confirm if that's the case before investigating adding support.

> bulletproof plugin registration code that is robust against the order in which the script tags load

Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.

I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:

I can see that this approach brings advantages

- Plugin developers don't need to know the name of the datasette initialization event to start their plugin
- Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
- One less event listener sitting in memory

It also has some minor costs

- A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
- Some risk in encouraging people to mutate global state
- (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the `window` provided by Datasette as an inline script tag). 


",107914493
issue_comments,1546119773,IC_kwDOBm6k_c5cJ-pd,,9599,2023-05-12T18:24:07Z,"Here's a demo of this breaking in Datasette Lite:

https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data/baseline?_filter_column_1=is_baseline&_filter_op_1=exact&_filter_value_1=1&_filter_column_2=&_filter_op_2=notnull__1&_filter_value_2=1&_filter_column=&_filter_op=exact&_filter_value=&_sort=&_facet=is_baseline

<img width=""609"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/3c23ae6e-7fb0-41d4-92b8-421c400a731a"">

Here's a SQL query that demonstrates the underlying issue:

```sql
select 'working', count(*) from baseline where is_baseline = 1
union all
select 'broken', count(*) from baseline where is_baseline = '1'
```
https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data?sql=select+%27working%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+1%0Aunion+all%0Aselect+%27broken%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+%271%27

<img width=""673"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/867914f8-8cef-432c-a518-2739d1caf964"">",107914493
issue_comments,1546117538,IC_kwDOBm6k_c5cJ-Gi,,9599,2023-05-12T18:21:38Z,"https://latest.datasette.io/fixtures doesn't currently have a view with any integer columns in it, making this bug harder to demonstrate there.

I can't replicate the bug using https://datasette.io/content/plugins?_facet=stargazers_count&stargazers_count=3 - I would expect that not to work correctly.",107914493
issues,1708030220,I_kwDOBm6k_c5lznkM,Faceting doesn't work against integer columns in views,9599,2023-05-12T18:20:10Z,"Spotted this issue here: https://til.simonwillison.net/datasette/baseline

I had to do this workaround:
```sql
create view baseline as select
  _key,
  spec,
  '' || json_extract(status, '$.is_baseline') as is_baseline,
  json_extract(status, '$.since') as baseline_since,
  json_extract(status, '$.support.chrome') as baseline_chrome,
  json_extract(status, '$.support.edge') as baseline_edge,
  json_extract(status, '$.support.firefox') as baseline_firefox,
  json_extract(status, '$.support.safari') as baseline_safari,
  compat_features,
  caniuse,
  usage_stats,
  status
from
  [index]
```
I think the core issue here is that, against a table, `select * from x where integer_column = '1'` works correctly, due to some kind of column type conversion mechanism... but this mechanism doesn't work against views.",107914493
issue_comments,1540900733,IC_kwDOCGYnMM5b2Ed9,,167893,2023-05-09T21:15:05Z,"Sorry, I completely missed your first comment whilst on Easter break.

This looks like a good practical compromise before v4. Thanks!",140912432
issue_comments,1540494121,IC_kwDOBm6k_c5b0hMp,,9599,2023-05-09T16:25:00Z,Can now be used here: https://github.com/simonw/datasette/actions/workflows/deploy-branch-preview.yml,107914493
issue_comments,1540491751,IC_kwDOBm6k_c5b0gnn,,9599,2023-05-09T16:23:12Z,Added a actions `BRANCH_PREVIEW_VERCEL_TOKEN` secret to this repository.,107914493
issues,1702354223,I_kwDOBm6k_c5ld90v,Mechanism for deploying a preview of a branch using Vercel,9599,2023-05-09T16:21:45Z,"I prototyped that here: https://github.com/simonw/one-off-actions/blob/main/.github/workflows/deploy-datasette-branch-preview.yml

It deployed the `json-extras-query` branch here: https://datasette-preview-json-extras-query.vercel.app/",107914493
issue_comments,1539157643,IC_kwDOCGYnMM5bva6L,,9599,2023-05-08T22:45:09Z,"Here's an example from the new tests:

https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307",140912432
issues,1701018909,I_kwDOCGYnMM5lY30d,Tests broken on Windows due to new convert() lambda names,9599,2023-05-08T22:11:29Z,"https://github.com/simonw/sqlite-utils/actions/runs/4920084038/jobs/8788501314
```python
sql = 'update [example] set [dt] = lambda_-9223371942137158589([dt]);'
```
From:
- #526",140912432
issue_comments,1539109816,IC_kwDOCGYnMM5bvPO4,,9599,2023-05-08T22:01:00Z,"This is being handled in:
- #520",140912432
issue_comments,1539109587,IC_kwDOCGYnMM5bvPLT,,9599,2023-05-08T22:00:46Z,"> Hey, isn't this essentially the same issue as #448 ?

Yes it is, good catch!",140912432
issue_comments,1539108140,IC_kwDOCGYnMM5bvO0s,,9599,2023-05-08T21:59:41Z,That original example passes against `main` now.,140912432
issue_comments,1539101853,IC_kwDOCGYnMM5bvNSd,,9599,2023-05-08T21:52:44Z,I like the `lambda-{uuid}` idea.,140912432
issue_comments,1539100300,IC_kwDOCGYnMM5bvM6M,,9599,2023-05-08T21:50:51Z,Seeing as `sqlite-utils` doesn't currently provide mechanisms for adding `check` constraints like this I'm going to leave this - I'm happy with the fix I put in for the `not null` constraints.,140912432
issue_comments,1539099703,IC_kwDOCGYnMM5bvMw3,,9599,2023-05-08T21:50:06Z,"Applying the fix from the PR here doesn't fix the above problem either:
- https://github.com/simonw/sqlite-utils/pull/515

So it looks like these kinds of `check` constraints currently aren't compatible with how `upsert()` works.",140912432
issue_comments,1539094287,IC_kwDOCGYnMM5bvLcP,,9599,2023-05-08T21:44:11Z,"OK, this fails silently:
```python
import sqlite_utils
db = sqlite_utils.Database(memory=True)
db.execute('''CREATE TABLE employees (
    id INTEGER PRIMARY KEY,
    name TEXT,
    age INTEGER,
    salary REAL,
    CHECK (salary is not null and salary > 0)
);''')
db[""employees""].upsert({""id"": 1, ""name"": ""Bob""}, pk=""id"")
list(db[""employees""].rows)
````
It outputs:
```python
[]
```",140912432
issue_comments,1539079507,IC_kwDOCGYnMM5bvH1T,,9599,2023-05-08T21:28:37Z,"> This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted.

Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.",140912432
issue_comments,1539078429,IC_kwDOCGYnMM5bvHkd,,9599,2023-05-08T21:27:40Z,"Dupe of:
- #538",140912432
issue_comments,1539077777,IC_kwDOCGYnMM5bvHaR,,9599,2023-05-08T21:27:10Z,I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304,140912432
issue_comments,1539058795,IC_kwDOCGYnMM5bvCxr,,9599,2023-05-08T21:12:52Z,"This is a really neat fix, thank you.",140912432
issue_comments,1539055393,IC_kwDOCGYnMM5bvB8h,,9599,2023-05-08T21:10:06Z,Thanks!,140912432
issue_comments,1539053230,IC_kwDOCGYnMM5bvBau,,9599,2023-05-08T21:08:23Z,"I fixed this in:
- #527

Will fully remove this misfeature in:
- #542",140912432
issue_comments,1539052467,IC_kwDOCGYnMM5bvBOz,,9599,2023-05-08T21:07:41Z,"Relevant commits (will mostly revert these):
- https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd
- https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2",140912432
issue_comments,1539051724,IC_kwDOCGYnMM5bvBDM,,9599,2023-05-08T21:07:04Z,"Updated documentation:
- https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns
- https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#convert",140912432
issues,1700936245,I_kwDOCGYnMM5lYjo1,Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0,9599,2023-05-08T21:04:28Z,"Following:
- #527

The only reason I didn't remove fix this mis-feature entirely is that it represents a backwards incompatible change. I'll make that change in 4.0.",140912432
issue_comments,1539035838,IC_kwDOCGYnMM5bu9K-,,9599,2023-05-08T20:55:00Z,"I'm going to go with `--no-skip-false` as the CLI option. It's ugly, but this whole thing is ugly. I'm going to make a note to remove this misfeature in `sqlite-utils` 4.",140912432
issue_comments,1539033736,IC_kwDOCGYnMM5bu8qI,,9599,2023-05-08T20:52:51Z,"OK, I implemented that at the Python API level. I need to decide how it should work for the `sqlite-utils convert` command too: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#convert",140912432
issue_comments,1539018912,IC_kwDOCGYnMM5bu5Cg,,9599,2023-05-08T20:39:00Z,"I think the natural place to add these in the Python library API would be https://sqlite-utils.datasette.io/en/stable/python-api.html#adding-foreign-key-constraints - maybe something like this:

```python
db[""books""].add_foreign_key(""author_id"", ""authors"", ""id"", on_delete=RESTRICT)
```

Then for the CLI tool could be added to https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key

```
sqlite-utils add-foreign-key my.db books author_id authors id --on-update SET_NULL
```
I wouldn't support these for the other methods of adding foreign keys - `foreign_keys(...)` for the various `.insert()` etc methods and the `add_foreign_keys(...)` bulk menthod.",140912432
issue_comments,1539015064,IC_kwDOCGYnMM5bu4GY,,9599,2023-05-08T20:35:07Z,"Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options:

- NO ACTION (default)
- RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it
- SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values
- SET DEFAULT: set a specific default
- CASCADE: propagates the delete or update operation on the parent key to each dependent child key",140912432
issue_comments,1539009453,IC_kwDOCGYnMM5bu2ut,,9599,2023-05-08T20:30:29Z,"Here's an improvement:
```
% sqlite-utils insert /tmp/b.db blah /tmp/blah.txt
  [####################################]  100%
Error: Invalid JSON - use --csv for CSV or --tsv for TSV files

JSON error: Expecting value: line 1 column 1 (char 0)
```",140912432
issue_comments,1539006509,IC_kwDOCGYnMM5bu2At,,9599,2023-05-08T20:28:56Z,Was this a newline-delimited JSON file perhaps?,140912432
issue_comments,1538975545,IC_kwDOCGYnMM5buuc5,,1231935,2023-05-08T20:06:35Z,"perfect, thank you!",140912432
issue_comments,1538963959,IC_kwDOCGYnMM5burn3,,9599,2023-05-08T19:59:34Z,"There are 8 failing tests left:

```
==== short test summary info ====
FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bcaca0>
FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bc9620>
ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ====
```
Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e",140912432
issues,1700840265,I_kwDOCGYnMM5lYMNJ,Get tests to pass with `pytest -Werror`,9599,2023-05-08T19:57:23Z,"Inspired by:
- #534",140912432
issue_comments,1538933540,IC_kwDOCGYnMM5bukMk,,9599,2023-05-08T19:34:37Z,"On macOS this shows the same warning:
```
% python -Wdefault $(which sqlite-utils) insert dogs.db dogs dogs.csv --csv
  [############------------------------]   35%
  [####################################]  100%/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs.csv' encoding='utf-8-sig'>
  insert_upsert_implementation(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
```
The file itself is a `click.File` which is automatically closed - https://click.palletsprojects.com/en/8.1.x/api/#click.File - but it looks like it's the `_io.TextIOWrapper` which is not being closed:

https://github.com/simonw/sqlite-utils/blob/2376c452a56b0c3e75e7ca698273434e32945304/sqlite_utils/cli.py#L949-L956",140912432
issue_comments,1538921774,IC_kwDOCGYnMM5buhUu,,9599,2023-05-08T19:24:41Z,That fix seems to work!,140912432
issue_comments,1538910894,IC_kwDOCGYnMM5buequ,,9599,2023-05-08T19:16:52Z,"How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`?

Something like this:

```python
[
  ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']),
  ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
]
```",140912432
issue_comments,1538903556,IC_kwDOCGYnMM5buc4E,,9599,2023-05-08T19:11:24Z,"I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though.

This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`:

https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878
",140912432
issue_comments,1538893329,IC_kwDOCGYnMM5buaYR,,9599,2023-05-08T19:04:47Z,"This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`.

One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html

But...

> UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).

I still want to support SQLite versions earlier than that.",140912432
issue_comments,1538889482,IC_kwDOCGYnMM5buZcK,,9599,2023-05-08T19:02:38Z,"Here's the code at fault:
https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788",140912432
issue_comments,1538887361,IC_kwDOCGYnMM5buY7B,,9599,2023-05-08T19:01:20Z,"Here's the problem:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
db.execute('create table foo (id integer primary key, name not null)')
db.execute('insert into foo (id) values (1)')
```
Produces:
```
IntegrityError: NOT NULL constraint failed: foo.name
```
But this:
```python
db.execute('insert or ignore into foo (id) values (1)')
```
Completes without an exception.",140912432
issue_comments,1538801855,IC_kwDOCGYnMM5buEC_,,9599,2023-05-08T18:00:17Z,"From time in the debugger, after creating the table it ends up doing this:
```
(Pdb) queries_and_params
[
  ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]),
  ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
]
```",140912432
issue_comments,1538793817,IC_kwDOCGYnMM5buCFZ,,9599,2023-05-08T17:55:10Z,"Confirmed - I added this test and it fails:
```python
def test_upsert_all_not_null(fresh_db):
    # https://github.com/simonw/sqlite-utils/issues/538
    fresh_db[""comments""].upsert_all(
        [{""id"": 1, ""name"": ""Cleo""}],
        pk=""id"",
        not_null=[""name""],
    )
    assert list(fresh_db[""comments""].rows) == [{""id"": 1, ""name"": ""Cleo""}]
```",140912432
issue_comments,1537744000,IC_kwDOCGYnMM5bqByA,,42327,2023-05-08T04:56:12Z,"Hey @simonw, urllib3 maintainer here :wave:

Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just `python: ""3""` will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml

I can open PRs to sqlite-utils / datasette if you're interested",140912432
issue_comments,1537514610,IC_kwDOCGYnMM5bpJxy,,9599,2023-05-07T18:43:24Z,"Documentation:
- https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory",140912432
issue_comments,1537514069,IC_kwDOCGYnMM5bpJpV,,9599,2023-05-07T18:40:18Z,"https://docs.readthedocs.io/en/stable/config-file/v2.html suggests:

```yaml
build:
  os: ubuntu-22.04
  tools:
    python: ""3.11""
```",140912432
issue_comments,1537513912,IC_kwDOCGYnMM5bpJm4,,9599,2023-05-07T18:39:29Z,"https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said:

> Problem in your project's configuration. Invalid ""python.version"": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11",140912432
issue_comments,1537513653,IC_kwDOCGYnMM5bpJi1,,9599,2023-05-07T18:37:59Z,"Useful comment here:

- https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928

> I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.",140912432
issues,1699184583,I_kwDOCGYnMM5lR3_H,sphinx.builders.linkcheck build error,9599,2023-05-07T18:37:09Z,"https://readthedocs.org/projects/sqlite-utils/builds/20512693/
```
 Running Sphinx v6.2.1

Traceback (most recent call last):
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py"", line 442, in load_extension
    mod = import_module(extname)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/importlib/__init__.py"", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File ""<frozen importlib._bootstrap>"", line 1014, in _gcd_import
  File ""<frozen importlib._bootstrap>"", line 991, in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 975, in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 671, in _load_unlocked
  File ""<frozen importlib._bootstrap_external>"", line 783, in exec_module
  File ""<frozen importlib._bootstrap>"", line 219, in _call_with_frames_removed
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/builders/linkcheck.py"", line 20, in <module>
    from requests import Response
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/requests/__init__.py"", line 43, in <module>
    import urllib3
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/urllib3/__init__.py"", line 38, in <module>
    raise ImportError(
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n  7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/cmd/build.py"", line 280, in build_main
    app = Sphinx(args.sourcedir, args.confdir, args.outputdir,
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py"", line 225, in __init__
    self.setup_extension(extension)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/application.py"", line 404, in setup_extension
    self.registry.load_extension(self, extname)
  File ""/home/docs/checkouts/readthedocs.org/user_builds/sqlite-utils/envs/latest/lib/python3.8/site-packages/sphinx/registry.py"", line 445, in load_extension
    raise ExtensionError(__('Could not import extension %s') % extname,
sphinx.errors.ExtensionError: Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n  7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)

Extension error:
Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n  7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)
```",140912432
issue_comments,1537507676,IC_kwDOCGYnMM5bpIFc,,9599,2023-05-07T18:09:43Z,"This worked:

```bash
sqlite-utils memory /tmp/books3.json:nl \
  'select name from books3' --raw-lines > titles.txt
```",140912432
issue_comments,1537507525,IC_kwDOCGYnMM5bpIDF,,9599,2023-05-07T18:09:09Z,"I'm tempted to upgrade `--raw` to do this instead, but that would be a breaking change.",140912432
issue_comments,1537507394,IC_kwDOCGYnMM5bpIBC,,9599,2023-05-07T18:08:44Z,"Prototype:
```diff
diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst
index 153e5f9..c830518 100644
--- a/docs/cli-reference.rst
+++ b/docs/cli-reference.rst
@@ -124,6 +124,7 @@ See :ref:`cli_query`.
       --json-cols                 Detect JSON cols and output them as JSON, not
                                   escaped strings
       -r, --raw                   Raw output, first column of first row
+      --raw-lines                 Raw output, first column of each row
       -p, --param <TEXT TEXT>...  Named :parameters for SQL query
       --functions TEXT            Python code defining one or more custom SQL
                                   functions
@@ -192,6 +193,7 @@ See :ref:`cli_memory`.
       --json-cols                 Detect JSON cols and output them as JSON, not
                                   escaped strings
       -r, --raw                   Raw output, first column of first row
+      --raw-lines                 Raw output, first column of each row
       -p, --param <TEXT TEXT>...  Named :parameters for SQL query
       --encoding TEXT             Character encoding for CSV input, defaults to
                                   utf-8
diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index d25b1df..da0e4b6 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension):
 )
 @output_options
 @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"")
+@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"")
 @click.option(
     ""-p"",
     ""--param"",
@@ -1677,6 +1678,7 @@ def query(
     fmt,
     json_cols,
     raw,
+    raw_lines,
     param,
     load_extension,
     functions,
@@ -1700,7 +1702,19 @@ def query(
         _register_functions(db, functions)
 
     _execute_query(
-        db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
+        db,
+        sql,
+        param,
+        raw,
+        raw_lines,
+        table,
+        csv,
+        tsv,
+        no_headers,
+        fmt,
+        nl,
+        arrays,
+        json_cols,
     )
 
 
@@ -1728,6 +1742,7 @@ def query(
 )
 @output_options
 @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"")
+@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"")
 @click.option(
     ""-p"",
     ""--param"",
@@ -1773,6 +1788,7 @@ def memory(
     fmt,
     json_cols,
     raw,
+    raw_lines,
     param,
     encoding,
     no_detect_types,
@@ -1879,12 +1895,36 @@ def memory(
         _register_functions(db, functions)
 
     _execute_query(
-        db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
+        db,
+        sql,
+        param,
+        raw,
+        raw_lines,
+        table,
+        csv,
+        tsv,
+        no_headers,
+        fmt,
+        nl,
+        arrays,
+        json_cols,
     )
 
 
 def _execute_query(
-    db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
+    db,
+    sql,
+    param,
+    raw,
+    raw_lines,
+    table,
+    csv,
+    tsv,
+    no_headers,
+    fmt,
+    nl,
+    arrays,
+    json_cols,
 ):
     with db.conn:
         try:
@@ -1903,6 +1943,13 @@ def _execute_query(
                 sys.stdout.buffer.write(data)
             else:
                 sys.stdout.write(str(data))
+        elif raw_lines:
+            for row in cursor:
+                data = row[0]
+                if isinstance(data, bytes):
+                    sys.stdout.buffer.write(data + b""\n"")
+                else:
+                    sys.stdout.write(str(data) + ""\n"")
         elif fmt or table:
             print(
                 tabulate.tabulate(
```
Needs tests and more documentation.",140912432
issues,1699174055,I_kwDOCGYnMM5lR1an,"`--raw-lines` option, like `--raw` for multiple lines",9599,2023-05-07T18:07:46Z,I wanted to output newline-separated output of the first column of every row in the results - like `--row` but for more than one line.,140912432
issue_comments,1537277919,IC_kwDOBm6k_c5boP_f,,31861128,2023-05-07T03:17:35Z,"Some updates:

I notice that there is an option in the CLI where we can explicitly set `immutable` mode when spinning up the server

```console
Options:
  -i, --immutable PATH            Database files to open in immutable mode
```

Then, the question is - how can I disable immutable mode in the deployed instance on Vercel?",107914493
issues,1698865182,I_kwDOBm6k_c5lQqAe,[BUG] Cannot insert new data to deployed instance,31861128,2023-05-07T02:59:42Z,"## Summary

Recently, I deployed an instance of datasette to Vercel with the following plugins:

- datasette-auth-tokens
- datasette-insert

With the above plugins, I was able to insert new data to local sqlite db. However, when it comes to the deployment on Vercel, things behave differently. I observed some errors from the logs console on Vercel:

```console
File ""/var/task/datasette/database.py"", line 179, in _execute_writes
conn = self.connect(write=True)
File ""/var/task/datasette/database.py"", line 93, in connect
assert not (write and not self.is_mutable)
AssertionError
``` 

<img width=""1380"" alt=""image"" src=""https://user-images.githubusercontent.com/31861128/236655274-3fd5e16f-4e06-4ba8-87ff-41130b8ebe90.png"">

I think it is a potential bug.

## Reproduce

<details><summary>metadata.json</summary>
</br>

```json
{
  ""plugins"": {
    ""datasette-insert"": {
      ""allow"": {
        ""id"": ""*""
      }
    },
    ""datasette-auth-tokens"": {
      ""tokens"": [
        {
          ""token"": {
            ""$env"": ""INSERT_TOKEN""
          },
          ""actor"": {
            ""id"": ""repeater""
          }
        }
      ],
      ""param"": ""_auth_token""
    }
  }
}
```

</details>

<details><summary>commands</summary>
</br>

```bash
# deploy
datasette publish vercel remote.db \
  --project=repeater-bot-sqlite \
  --metadata metadata.json \
  --install datasette-auth-tokens \
  --install datasette-insert \
  --vercel-json=vercel.json

# test insert
cat fixtures/dogs.json | curl --request POST -d @- -H ""Authorization: Bearer <token>"" \
  'https://repeater-bot-sqlite.vercel.app/-/insert/remote/dogs?pk=id'
```

</details>

<details><summary>logs</summary>
</br>

```console
Traceback (most recent call last):
File ""/var/task/datasette/app.py"", line 1354, in route_path
response = await view(request, send)
File ""/var/task/datasette/app.py"", line 1500, in async_view_fn
response = await async_call_with_supported_arguments(
File ""/var/task/datasette/utils/__init__.py"", line 1005, in async_call_with_supported_arguments
return await fn(*call_with)
File ""/var/task/datasette_insert/__init__.py"", line 14, in insert_or_upsert
response = await insert_or_upsert_implementation(request, datasette)
File ""/var/task/datasette_insert/__init__.py"", line 91, in insert_or_upsert_implementation
table_count = await db.execute_write_fn(write_in_thread, block=True)
File ""/var/task/datasette/database.py"", line 167, in execute_write_fn
raise result
File ""/var/task/datasette/database.py"", line 179, in _execute_writes
conn = self.connect(write=True)
File ""/var/task/datasette/database.py"", line 93, in connect
assert not (write and not self.is_mutable)
AssertionError
```

</details>",107914493
issues,1695428235,I_kwDOCGYnMM5lDi6L,`table.upsert_all` fails to write rows when `not_null` is present,1231935,2023-05-04T07:30:38Z,"I found an odd bug today, where calls to `table.upsert_all` don't write rows if you include the `not_null` kwarg.

## Repro Example

```py
from sqlite_utils import Database

db = Database(""upsert-test.db"")

db[""comments""].upsert_all(
    [{""id"": 1, ""name"": ""david""}],
    pk=""id"",
    not_null=[""name""],
)

assert list(db[""comments""].rows) # err!
```

The schema is correctly created:

```sql
CREATE TABLE [comments] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT NOT NULL
)
```

But no rows are created. Removing either the `not_null` kwargs works as expected, as does an `insert_all` call.

## Version Info

- Python: `3.11.0`
- sqlite-utils: `3.30`
- sqlite: `3.39.5 2022-10-14`",140912432
issue_comments,1532481862,IC_kwDOCGYnMM5bV9FG,,1231935,2023-05-03T05:53:26Z,"Would love to put our heads together for improvements here. 

I _think_ anything that is `argname=DEFAULT` needs to be typed as `argname: str | Default = DEFAULT` (replacing `str` with the appropriate type(s)). We may be able to get clever and tie the types to that key in the `_defaults` dict (definitely possible in Typescript, but I'm less familiar with advanced python types). 

Right now, all args are typed as `Default`, which means all callers get type errors. 

As for table/view, given that they don't have the same methods, it would be nice to be able to get one or the other specifically.
",140912432
issue_comments,1532304714,IC_kwDOBm6k_c5bVR1K,,39538958,2023-05-03T00:16:03Z,"Curiously, after running commands on the database that was litestream-restored, datasette starts to work again, e.g.

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
datasette data/db.sqlite
# fails (OperationalError described above)
```

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite-utils enable-wal data/db.sqlite
datasette data/db.sqlite
# works
```

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite-utils optimize data/db.sqlite
datasette data/db.sqlite
# works
```

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite3 data/db.sqlite "".clone test.db""
datasette test.db
# works
```",107914493
issue_comments,1530822437,IC_kwDOBm6k_c5bPn8l,,193185,2023-05-02T03:35:30Z,"Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a `<script async src='...'/>` tag)?

```js
if (window.__DATASETTE__)
  go(window.__DATASETTE__);
else
  document.addEventListener(""datasette_init"", (evt) => go(evt.detail));

function go(manager) {
  manager.registerPlugin(...)
}
```

I don't know if it'd make sense, but you could also consider the asynchronous queuing pattern that Google Analytics uses (see [this Stack Overflow post](https://stackoverflow.com/questions/6963779/whats-the-name-of-google-analytics-async-design-pattern-and-where-is-it-used) for more details):

```js
__DATASETTE__ = __DATASETTE__ || [];
__DATASETTE__.push(go);

function go(manager) {
  manager.registerPlugin(...);
}
```",107914493
issue_comments,1530817667,IC_kwDOBm6k_c5bPmyD,,193185,2023-05-02T03:24:53Z,"Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(

tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return `Promise`s.

The long version: I read the design notes and example plugin. I think I'd be able to use this in [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras) for my lazy-facets feature.

The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.

First, at page render time:
(A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.
(B) there's a python plugin that implements the [extra_body_script](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette) hook, to write out the list of column names for future use by JavaScript

Second, at page load time: there is some JavaScript that:
(C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)
(D) wires up the column menus to add Facet-by-this options for each facet

With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.

If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.

Does that make sense? Sorry for the wall of text!",107914493
issue_comments,1529737426,IC_kwDOBm6k_c5bLfDS,,49699333,2023-05-01T13:58:50Z,Superseded by #2068.,107914493
issues,1690842199,PR_kwDOBm6k_c5PgNaA,Bump sphinx from 6.1.3 to 7.0.0,49699333,2023-05-01T13:58:46Z,"Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 7.0.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>v7.0.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v7.0.0rc1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 7.0.0 (released Apr 29, 2023)</h1>
<h2>Incompatible changes</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11359"">#11359</a>: Remove long-deprecated aliases for <code>MecabSplitter</code> and
<code>DefaultSplitter</code> in <code>sphinx.search.ja</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11360"">#11360</a>: Remove deprecated <code>make_old_id</code> functions in domain object
description classes.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11363"">#11363</a>: Remove the Setuptools integration (<code>build_sphinx</code> hook in
<code>setup.py</code>).</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11364"">#11364</a>: Remove deprecated <code>sphinx.ext.napoleon.iterators</code> module.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11365"">#11365</a>: Remove support for the <code>jsdump</code> format in <code>sphinx.search</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11366"">#11366</a>: Make <code>locale</code> a required argument to
<code>sphinx.util.i18n.format_date()</code>.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11370"">#11370</a>: Remove deprecated <code>sphinx.util.stemmer</code> module.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11371"">#11371</a>: Remove deprecated <code>sphinx.pycode.ast.parse()</code> function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11372"">#11372</a>: Remove deprecated <code>sphinx.io.read_doc()</code> function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11373"">#11373</a>: Removed deprecated <code>sphinx.util.get_matching_files()</code> function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11378"">#11378</a>: Remove deprecated <code>sphinx.util.docutils.is_html5_writer_available()</code>
function.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11379"">#11379</a>: Make the <code>env</code> argument to <code>Builder</code> subclasses required.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11380"">#11380</a>: autosummary: Always emit grouped import exceptions.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11381"">#11381</a>: Remove deprecated <code>style</code> key for HTML templates.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11382"">#11382</a>: Remove deprecated <code>sphinx.writers.latex.LaTeXTranslator.docclasses</code>
attribute.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11383"">#11383</a>: Remove deprecated <code>sphinx.builders.html.html5_ready</code> and
<code>sphinx.builders.html.HTMLTranslator</code> attributes.</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11385"">#11385</a>: Remove support for HTML 4 output.</li>
</ul>
<h1>Release 6.2.1 (released Apr 25, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11355"">#11355</a>: Revert the default type of :confval:<code>nitpick_ignore</code> and
:confval:<code>nitpick_ignore_regex</code> to <code>list</code>.</li>
</ul>
<h1>Release 6.2.0 (released Apr 23, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li>Require Docutils 0.18.1 or greater.</li>
</ul>
<h2>Incompatible changes</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d568b2f4f7cca743fcbf70814d15602d8129b790""><code>d568b2f</code></a> Bump to 7.0.0 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ff79edf353f5cc6e02036f58e0295dc704c5e681""><code>ff79edf</code></a> Remove <code>jsdump</code> references post removal</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/1a5133ac391931d8e0e2a271fe8ea9c505176908""><code>1a5133a</code></a> Bump to 7.0.0rc1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/5795fc7f9f7a5fe968cd11b961fc33fbd952a152""><code>5795fc7</code></a> Update <code>sphinx.deprecation</code> for Sphinx 7.0 (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11386"">#11386</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/6202087f912a2ae32808865072f884ea44cd31cb""><code>6202087</code></a> Add a note to CHANGES for PR 11385</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ad473730a3763f241e9aaea1a87d1893f01b86fd""><code>ad47373</code></a> Remove HTML 4 support (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11385"">#11385</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/3e3251d3ba4f1e5fa6d34d995505f4d3ace5acad""><code>3e3251d</code></a> Remove <code>HTMLTranslator</code> and <code>html5_ready</code> from <code>sphinx.builders.html</code> (...</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/77fd819a09881102bdd48593cc69ec164b2495c8""><code>77fd819</code></a> Remove deprecated <code>LaTeXTranslator.docclasses</code> attribute (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11382"">#11382</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4be56f3b7ddf44265d2e7a8f26a33ba46be52ab9""><code>4be56f3</code></a> Remove deprecated <code>style</code> key for HTML templates (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11381"">#11381</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/49027a9ae61d0d00016182025f68239a50ba4c09""><code>49027a9</code></a> Autosummary: Always emit grouped <code>ImportError</code> exceptions (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11380"">#11380</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v7.0.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=6.1.3&new-version=7.0.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2068.org.readthedocs.build/en/2068/

<!-- readthedocs-preview datasette end -->",107914493
issues,1690765434,I_kwDOBm6k_c5kxwh6,Litestream-restored db: errors on 3.11 and 3.10.8; but works on py3.10.7 and 3.10.6,39538958,2023-05-01T12:42:28Z,"Hi! Wondering if this issue is limited to my local system or if it affects others as well. 

It seems like 3.11 errors out on a ""litestream-restored"" database. On further investigation, it also appears to conk out on 3.10.8 but works on 3.10.7 and 3.10.6.

To demo issue I created a test database, replicated it to an aws s3 bucket, then restored the same under various .pyenv-versioned shells where I test whether I can read the database via the sqlite3 cli.

```sh
# create new shell with 3.11.3
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite3 data/db.sqlite  
# SQLite version 3.41.2 2023-03-22 11:56:21
# Enter "".help"" for usage hints.
# sqlite> .tables
# _litestream_lock  _litestream_seq   movie           
# sqlite> 
```

However this get me an `OperationalError` when reading via datasette:

<details>
<summary>Error on 3.11.3 and 3.10.8</summary>

```sh
datasette data/db.sqlite
```

```console
/tester/.venv/lib/python3.11/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
  warnings.warn(""pkg_resources is deprecated as an API"", DeprecationWarning)
Traceback (most recent call last):
  File ""/tester/.venv/bin/datasette"", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/datasette/cli.py"", line 143, in wrapped
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/datasette/cli.py"", line 615, in serve
    asyncio.get_event_loop().run_until_complete(check_databases(ds))
  File ""/Users/mv/.pyenv/versions/3.11.3/lib/python3.11/asyncio/base_events.py"", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/datasette/cli.py"", line 660, in check_databases
    await database.execute_fn(check_connection)
  File ""/tester/.venv/lib/python3.11/site-packages/datasette/database.py"", line 213, in execute_fn
    return await asyncio.get_event_loop().run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/mv/.pyenv/versions/3.11.3/lib/python3.11/concurrent/futures/thread.py"", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/datasette/database.py"", line 211, in in_thread
    return fn(conn)
           ^^^^^^^^
  File ""/tester/.venv/lib/python3.11/site-packages/datasette/utils/__init__.py"", line 951, in check_connection
    for r in conn.execute(
             ^^^^^^^^^^^^^
sqlite3.OperationalError: unable to open database file
```

</details>


<details>
<summary>Works on 3.10.7, 3.10.6</summary>

```sh
# create new shell with 3.10.7 / 3.10.6
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
datasette data/db.sqlite
# ...
# INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
```

</details>

In both scenarios, the only dependencies were the pinned python version and the latest Datasette version 0.64.",107914493
issue_comments,1524709988,IC_kwDOBm6k_c5a4Tpk,,9599,2023-04-27T05:09:36Z,"That fixed it - after installing `main.zip` again I ran this and it worked:

    ~/.rye/tools/main-zip/bin/datasette install datasette-graphql

```
% ~/.rye/tools/main-zip/bin/datasette plugins                  
[
    {
        ""name"": ""datasette-graphql"",
        ""static"": true,
        ""templates"": true,
        ""version"": ""2.2"",
        ""hooks"": [
            ""database_actions"",
            ""extra_template_vars"",
            ""menu_links"",
            ""register_routes"",
            ""startup"",
            ""table_actions""
        ]
    }
]
```",107914493
issue_comments,1524707628,IC_kwDOBm6k_c5a4TEs,,9599,2023-04-27T05:06:44Z,"I need `pip` as a dependency too:

```
% ~/.rye/tools/main-zip/bin/datasette install datasette-graphql
Traceback (most recent call last):
  File ""/Users/simon/.rye/tools/main-zip/bin/datasette"", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/datasette/cli.py"", line 365, in install
    run_module(""pip"", run_name=""__main__"")
  File ""<frozen runpy>"", line 222, in run_module
  File ""<frozen runpy>"", line 142, in _get_module_details
ImportError: No module named pip
```",107914493
issue_comments,1524699863,IC_kwDOBm6k_c5a4RLX,,9599,2023-04-27T04:56:22Z,Turned this into a TIL: https://til.simonwillison.net/python/rye,107914493
issue_comments,1524680160,IC_kwDOBm6k_c5a4MXg,,9599,2023-04-27T04:27:50Z,That fixed it.,107914493
issue_comments,1524675817,IC_kwDOBm6k_c5a4LTp,,9599,2023-04-27T04:21:03Z,I went with this to generate the long string: https://github.com/simonw/datasette/blob/0b0c5cd7a94fe3f151a3e10261b5c84ee64f2f18/tests/test_csv.py#L157-L176,107914493
issue_comments,1524669124,IC_kwDOBm6k_c5a4JrE,,9599,2023-04-27T04:10:44Z,"I need an alternative way of generating a long string with a shorter URL.

```sql
select group_concat('abcabcabc', '') from json_each(json_array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
```
https://latest.datasette.io/_memory?sql=select+group_concat%28%27abcabcabc%27%2C+%27%27%29+from+json_each%28json_array%281%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%29%29",107914493
issue_comments,1524666049,IC_kwDOBm6k_c5a4I7B,,9599,2023-04-27T04:06:18Z,"Most recent `httpx` release is 0.24 a couple of weeks ago. Here's what changed in that version: https://github.com/encode/httpx/compare/0.23.3...0.24.0

It looks like that URL limit is new: https://github.com/encode/httpx/commit/57daabf673705954afa94686c0002801c93d31f3#diff-78d8d93b5dd4c77d99c3e2b46b7286ba71a8fd60e92d8bd4eee5fb200b4f87bfR149-R155

```python
def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult:
    # Initial basic checks on allowable URLs.
    # ---------------------------------------

    # Hard limit the maximum allowable URL length.
    if len(url) > MAX_URL_LENGTH:
        raise InvalidURL(""URL too long"")
```
https://github.com/encode/httpx/blob/32e25497a36e6222cc64a758c98275b450dac28d/httpx/_urlparse.py#L153-L155",107914493
issue_comments,1524660603,IC_kwDOBm6k_c5a4Hl7,,9599,2023-04-27T04:02:55Z,"In the debugger:
```
(Pdb) MAX_URL_LENGTH
65536
```
Weird this only seems to be a problem with `httpx` on Python 3.7 though.",107914493
issue_comments,1524659084,IC_kwDOBm6k_c5a4HOM,,9599,2023-04-27T04:02:07Z,"This is the failing test:

https://github.com/simonw/datasette/blob/249fcf8e3e2a90e763f41b080c1b9ec8017f5005/tests/test_csv.py#L156-L167",107914493
issue_comments,1524655203,IC_kwDOBm6k_c5a4GRj,,9599,2023-04-27T03:59:56Z,"In a fresh Datasette checkout I ran:

    pipenv shell --python /Users/simon/.pyenv/versions/3.7.16/bin/python

That gave me a virtual environment with 3.7.16 Python.

Then I ran:

    pip install -e '.[test]'

Weirdly that gave me a `which pytest` of `/opt/homebrew/bin/pytest` which ran the tests on 3.11.

I figured out the location of the virtual environment with `which python` and then ran this:

```
% /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/bin/pytest tests/test_csv.py 
============================================================================================== test session starts ===============================================================================================
platform darwin -- Python 3.7.16, pytest-7.3.1, pluggy-1.0.0
SQLite: 3.39.5
rootdir: /private/tmp/datasette
configfile: pytest.ini
plugins: asyncio-0.21.0, timeout-2.1.0, xdist-3.2.1, anyio-3.6.2
asyncio: mode=strict
collected 15 items                                                                                                                                                                                               

tests/test_csv.py ..........F....                                                                                                                                                                          [100%]

==================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________________________________ test_max_csv_mb _________________________________________________________________________________________________

app_client_csv_max_mb_one = <datasette.utils.testing.TestClient object at 0x107ab5c08>

    def test_max_csv_mb(app_client_csv_max_mb_one):
        response = app_client_csv_max_mb_one.get(
            (
                ""/fixtures.csv?sql=select+'{}'+""
                ""from+compound_three_primary_keys&_stream=1&_size=max""
>           ).format(""abcdefg"" * 10000)
        )

/private/tmp/datasette/tests/test_csv.py:161: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:240: in __call__
    return call_result.result()
/Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:428: in result
    return self.__get_result()
/Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:384: in __get_result
    raise self._exception
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:306: in main_wrap
    result = await self.awaitable(*args, **kwargs)
/private/tmp/datasette/datasette/utils/testing.py:76: in get
    headers=headers,
/private/tmp/datasette/datasette/utils/testing.py:167: in _request
    content=post_body,
/private/tmp/datasette/datasette/app.py:1787: in request
    method, self._fix(path, avoid_path_rewrites), **kwargs
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:1528: in request
    extensions=extensions,
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:346: in build_request
    url = self._merge_url(url)
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:376: in _merge_url
    merge_url = URL(url)
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urls.py:113: in __init__
    self._uri_reference = urlparse(url, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

url = ""http://localhost/fixtures.csv?sql=select+'abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcde...gabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg'+from+compound_three_primary_keys&_stream=1&_size=max""
kwargs = {}

    def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult:
        # Initial basic checks on allowable URLs.
        # ---------------------------------------
    
        # Hard limit the maximum allowable URL length.
        if len(url) > MAX_URL_LENGTH:
>           raise InvalidURL(""URL too long"")
E           httpx.InvalidURL: URL too long

/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL
============================================================================================ short test summary info =============================================================================================
FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long
========================================================================================== 1 failed, 14 passed in 1.83s ==========================================================================================
(datasette) simon@Simons-MacBook-Pro datasette % 
```",107914493
issue_comments,1524648995,IC_kwDOBm6k_c5a4Ewj,,9599,2023-04-27T03:56:42Z,"I don't have 3.7 locally. Trying to install it with `pyenv`.

    brew install pyenv

Then:

    pyenv install --list | grep 3.7

Installing:

    pyenv install 3.7.16

Output:

    Installed Python-3.7.16 to /Users/simon/.pyenv/versions/3.7.16

So the executable is `/Users/simon/.pyenv/versions/3.7.16/bin/python`.",107914493
issue_comments,1524638233,IC_kwDOBm6k_c5a4CIZ,,9599,2023-04-27T03:50:51Z,Failure was on 3.7. I'll try that.,107914493
issue_comments,1524637376,IC_kwDOBm6k_c5a4B7A,,9599,2023-04-27T03:50:19Z,"Having trouble replicating this on my laptop. I tried a fresh virtual environment with fresh packages (in case this is a `httpx` change) but this passed:

    pytest tests/test_csv.py",107914493
issues,1686042269,I_kwDOBm6k_c5kfvad,Failing test: httpx.InvalidURL: URL too long,9599,2023-04-27T03:48:47Z,"https://github.com/simonw/datasette/actions/runs/4815723640/jobs/8574667731
```
    def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult:
        # Initial basic checks on allowable URLs.
        # ---------------------------------------
    
        # Hard limit the maximum allowable URL length.
        if len(url) > MAX_URL_LENGTH:
>           raise InvalidURL(""URL too long"")
E           httpx.InvalidURL: URL too long

/opt/hostedtoolcache/Python/3.7.16/x64/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL
=========================== short test summary info ============================
FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long
```",107914493
issue_comments,1524616740,IC_kwDOBm6k_c5a384k,,9599,2023-04-27T03:38:21Z,"Tried this:

    rye install https://github.com/simonw/datasette/archive/refs/heads/main.zip

But got this error:

    Error: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `:`

I instead downloaded that file and ran:

    rye install main.zip

This worked! And now:
```
~/.rye/tools/main-zip/bin/datasette --version
```
```
datasette, version 1.0a2
```",107914493
issues,1686033652,I_kwDOBm6k_c5kftT0,Datasette cannot be installed with Rye,9599,2023-04-27T03:35:42Z,"https://github.com/mitsuhiko/rye

I tried this:

    rye install datasette

But now:

```
% ~/.rye/shims/datasette
Traceback (most recent call last):
  File ""/Users/simon/.rye/shims/datasette"", line 5, in <module>
    from datasette.cli import cli
  File ""/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/cli.py"", line 17, in <module>
    from .app import (
  File ""/Users/simon/.rye/tools/datasette/lib/python3.11/site-packages/datasette/app.py"", line 14, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
```
I think that's because `setuptools` is not included in Rye.",107914493
issue_comments,1521837780,IC_kwDOBm6k_c5atWbU,,49699333,2023-04-25T13:57:52Z,Superseded by #2064.,107914493
issues,1683229834,PR_kwDOBm6k_c5PG0wF,Bump sphinx from 6.1.3 to 6.2.1,49699333,2023-04-25T13:57:49Z,"Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 6.2.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>v6.2.1</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 6.2.1 (released Apr 25, 2023)</h1>
<h2>Bugs fixed</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11355"">#11355</a>: Revert the default type of :confval:<code>nitpick_ignore</code> and
:confval:<code>nitpick_ignore_regex</code> to <code>list</code>.</li>
</ul>
<h1>Release 6.2.0 (released Apr 23, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li>Require Docutils 0.18.1 or greater.</li>
</ul>
<h2>Incompatible changes</h2>
<ul>
<li>LaTeX: removal of some internal TeX <code>\dimen</code> registers (not previously
publicly documented) as per 5.1.0 code comments in <code>sphinx.sty</code>:
<code>\sphinxverbatimsep</code>, <code>\sphinxverbatimborder</code>, <code>\sphinxshadowsep</code>,
<code>\sphinxshadowsize</code>, and <code>\sphinxshadowrule</code>. (refs: <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11105"">#11105</a>)</li>
<li>Remove <code>.egg</code> support from pycode <code>ModuleAnalyser</code>; Python eggs are a
now-obsolete binary distribution format</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11089"">#11089</a>: Remove deprecated code in <code>sphinx.builders.linkcheck</code>.
Patch by Daniel Eades</li>
<li>Remove internal-only <code>sphinx.locale.setlocale</code></li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11247"">#11247</a>: Deprecate the legacy <code>intersphinx_mapping</code> format</li>
<li><code>sphinx.util.osutil.cd</code> is deprecated in favour of <code>contextlib.chdir</code>.</li>
</ul>
<h2>Features added</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11277"">#11277</a>: :rst:dir:<code>autoproperty</code> allows the return type to be specified as
a type comment (e.g., <code># type: () -&gt; int</code>). Patch by Bénédikt Tran</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10811"">#10811</a>: Autosummary: extend <code>__all__</code> to imported members for template rendering
when option <code>autosummary_ignore_module_all</code> is set to <code>False</code>. Patch by
Clement Pinard</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11147"">#11147</a>: Add a <code>content_offset</code> parameter to <code>nested_parse_with_titles()</code>,
allowing for correct line numbers during nested parsing.
Patch by Jeremy Maitin-Shepard</li>
<li>Update to Unicode CLDR 42</li>
<li>Add a <code>--jobs</code> synonym for <code>-j</code>. Patch by Hugo van Kemenade</li>
<li>LaTeX: a command <code>\sphinxbox</code> for styling text elements with a (possibly</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/ec993dda3690f260345133c47a4a0f6ef0b18493""><code>ec993dd</code></a> Bump to 6.2.1 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d2aa91f63ce2fa749e4d8835ea7b771ead29e54a""><code>d2aa91f</code></a> Revert the default type of <code>nitpick_ignore[_regex]</code> to <code>list</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/60d8fa10b881308c97012d478fb13770a41b469f""><code>60d8fa1</code></a> Bump version</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/70102ac2ee4d89e1293d8efa7763dcdd104ad548""><code>70102ac</code></a> Bump to 6.2.0 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4e27a5fba146bf85b6e2e0ba8b9f5ac30b1c1e29""><code>4e27a5f</code></a> Remove unneeded JavaScript from <code>sphinx13</code> theme</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/bffb54734f64e88c95b1d5558ec3fb7cb44ceda7""><code>bffb547</code></a> Note correct deprecation version for <code>sphinx.util.osutil.cd</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/59de8d52029deca803d38b5d084a3a32c7b7f994""><code>59de8d5</code></a> Revert &quot;Support and prefer <code>.jinja</code> to <code>_t</code> for static templates (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11165"">#11165</a>)...</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/aee3c0ab75974790adf359a9c5089d1d781a6b21""><code>aee3c0a</code></a> Partially revert &quot;Disable localisation when SOURCE_DATE_EPOCH is set (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10949"">#10949</a>)...</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/186d596f3323f3141c34618e2e788e5cdff8fec3""><code>186d596</code></a> Use <code>overwrite_file</code> context manager in <code>test_ext_autodoc_configs</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11320"">#11320</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/77483f28244dac1e7459d09397180812cfb36946""><code>77483f2</code></a> Add missing test decorator for <code>test_util_inspect</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11321"">#11321</a>)</li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v6.2.1"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=6.1.3&new-version=6.2.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2064.org.readthedocs.build/en/2064/

<!-- readthedocs-preview datasette end -->",107914493
issues,1681339696,PR_kwDOBm6k_c5PAcGt,Bump sphinx from 6.1.3 to 6.2.0,49699333,2023-04-24T13:58:21Z,"Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.1.3 to 6.2.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/releases"">sphinx's releases</a>.</em></p>
<blockquote>
<h2>v6.2.0</h2>
<p>Changelog: <a href=""https://www.sphinx-doc.org/en/master/changes.html"">https://www.sphinx-doc.org/en/master/changes.html</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href=""https://github.com/sphinx-doc/sphinx/blob/master/CHANGES"">sphinx's changelog</a>.</em></p>
<blockquote>
<h1>Release 6.2.0 (released Apr 23, 2023)</h1>
<h2>Dependencies</h2>
<ul>
<li>Require Docutils 0.18.1 or greater.</li>
</ul>
<h2>Incompatible changes</h2>
<ul>
<li>LaTeX: removal of some internal TeX <code>\dimen</code> registers (not previously
publicly documented) as per 5.1.0 code comments in <code>sphinx.sty</code>:
<code>\sphinxverbatimsep</code>, <code>\sphinxverbatimborder</code>, <code>\sphinxshadowsep</code>,
<code>\sphinxshadowsize</code>, and <code>\sphinxshadowrule</code>. (refs: <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11105"">#11105</a>)</li>
<li>Remove <code>.egg</code> support from pycode <code>ModuleAnalyser</code>; Python eggs are a
now-obsolete binary distribution format</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11089"">#11089</a>: Remove deprecated code in <code>sphinx.builders.linkcheck</code>.
Patch by Daniel Eades</li>
<li>Remove internal-only <code>sphinx.locale.setlocale</code></li>
</ul>
<h2>Deprecated</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11247"">#11247</a>: Deprecate the legacy <code>intersphinx_mapping</code> format</li>
<li><code>sphinx.util.osutil.cd</code> is deprecated in favour of <code>contextlib.chdir</code>.</li>
</ul>
<h2>Features added</h2>
<ul>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11277"">#11277</a>: :rst:dir:<code>autoproperty</code> allows the return type to be specified as
a type comment (e.g., <code># type: () -&gt; int</code>). Patch by Bénédikt Tran</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10811"">#10811</a>: Autosummary: extend <code>__all__</code> to imported members for template rendering
when option <code>autosummary_ignore_module_all</code> is set to <code>False</code>. Patch by
Clement Pinard</li>
<li><a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11147"">#11147</a>: Add a <code>content_offset</code> parameter to <code>nested_parse_with_titles()</code>,
allowing for correct line numbers during nested parsing.
Patch by Jeremy Maitin-Shepard</li>
<li>Update to Unicode CLDR 42</li>
<li>Add a <code>--jobs</code> synonym for <code>-j</code>. Patch by Hugo van Kemenade</li>
<li>LaTeX: a command <code>\sphinxbox</code> for styling text elements with a (possibly
rounded) box, optional background color and shadow, has been added.
See :ref:<code>sphinxbox</code>. (refs: <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11224"">#11224</a>)</li>
<li>LaTeX: add <code>\sphinxstylenotetitle</code>, ..., <code>\sphinxstylewarningtitle</code>, ...,
for an extra layer of mark-up freeing up <code>\sphinxstrong</code> for other uses.
See :ref:<code>latex-macros</code>. (refs: <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11267"">#11267</a>)</li>
<li>LaTeX: :dudir:<code>note</code>, :dudir:<code>hint</code>, :dudir:<code>important</code> and :dudir:<code>tip</code> can
now each be styled as the other admonitions, i.e. possibly with a background
color, individual border widths and paddings, possibly rounded corners, and
optional shadow.  See :ref:<code>additionalcss</code>. (refs: <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11234"">#11234</a>)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e7d4c3669cec32721ee7fc25d9cb8dc3497095ed""><code>e7d4c36</code></a> Bump to 6.2.0 final</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/4e27a5fba146bf85b6e2e0ba8b9f5ac30b1c1e29""><code>4e27a5f</code></a> Remove unneeded JavaScript from <code>sphinx13</code> theme</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/bffb54734f64e88c95b1d5558ec3fb7cb44ceda7""><code>bffb547</code></a> Note correct deprecation version for <code>sphinx.util.osutil.cd</code></li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/59de8d52029deca803d38b5d084a3a32c7b7f994""><code>59de8d5</code></a> Revert &quot;Support and prefer <code>.jinja</code> to <code>_t</code> for static templates (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11165"">#11165</a>)...</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/aee3c0ab75974790adf359a9c5089d1d781a6b21""><code>aee3c0a</code></a> Partially revert &quot;Disable localisation when SOURCE_DATE_EPOCH is set (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/10949"">#10949</a>)...</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/186d596f3323f3141c34618e2e788e5cdff8fec3""><code>186d596</code></a> Use <code>overwrite_file</code> context manager in <code>test_ext_autodoc_configs</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11320"">#11320</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/77483f28244dac1e7459d09397180812cfb36946""><code>77483f2</code></a> Add missing test decorator for <code>test_util_inspect</code> (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11321"">#11321</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/d8f15c7361c4766667db65252b685f19d4a08b56""><code>d8f15c7</code></a> Increase timeout threshold for <code>linkcheck</code> tests (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11326"">#11326</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/b430e052858c490a4e90f3143f159ee6240eb948""><code>b430e05</code></a> Create a 'search field' component for themes (<a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11045"">#11045</a>)</li>
<li><a href=""https://github.com/sphinx-doc/sphinx/commit/e2f66cea4997b6d8c588d3509adb68d4e9108ee6""><code>e2f66ce</code></a> Update CHANGES for PR <a href=""https://redirect.github.com/sphinx-doc/sphinx/issues/11333"">#11333</a></li>
<li>Additional commits viewable in <a href=""https://github.com/sphinx-doc/sphinx/compare/v6.1.3...v6.2.0"">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=sphinx&package-manager=pip&previous-version=6.1.3&new-version=6.2.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2063.org.readthedocs.build/en/2063/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1515694393,IC_kwDOBm6k_c5aV6k5,,9020979,2023-04-20T04:25:55Z,"Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.

> comments

These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week. 

> I know TypeScript can be a little controversial

FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files. 

I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.

> Maybe they should have more ""action-y"" names

This is a great observation. I'm inclined towards something like `make*` or `build*` since to me `add*` make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current `get*` setup. I'll go through and update these. 

> Maybe we can make it easier to do pure-js datasette plugins?

I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at _all_. 

> And then do the PERMITTED_VIEWS filtering in JS rather than Python.

One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal. ",107914493
issues,1674322631,PR_kwDOBm6k_c5OpEz_,"Add ""Packaging a plugin using Poetry"" section in docs",1238873,2023-04-19T07:23:28Z,"This PR adds a new section about packaging a plugin using `poetry` within the ""Writing plugins"" page of the documentation.

<!-- readthedocs-preview datasette start -->
----
:books: Documentation preview :books:: https://datasette--2061.org.readthedocs.build/en/2061/

<!-- readthedocs-preview datasette end -->",107914493
issue_comments,1510423215,IC_kwDOBm6k_c5aBzqv,,9020979,2023-04-16T16:12:59Z,"## Research notes

- I stuck to the ""minimal dependencies"" ethos of datasette (no React, Typescript, JS linting, etc).
- Main threads on JS plugin development
  - Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
  - Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to ""search boxes""): https://github.com/simonw/datasette/issues/1191
  - Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
- Prior threads on JS plugins in Datasette for future design directions
  - Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
  - Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done.  Other plugins can listen for it, or ask the manager to inform them when the dependency is available). 
  - Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
- Articles on plugins reviewed
  - https://css-tricks.com/designing-a-javascript-plugin-system/
- Plugin/Extension systems reviewed (mostly JS).
  - Yarn: https://yarnpkg.com/advanced/plugin-tutorial
  - Tappable https://github.com/webpack/tapable (used by Auto, webpack)
  - Pluggy: https://pluggy.readthedocs.io/en/stable/
  - VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
  - Chrome: https://developer.chrome.com/docs/extensions/reference/
  - Figma/Figjam Widget: https://www.figma.com/widget-docs/
  - Datadog Apps: [Programming Model](https://github.com/DataDog/apps/blob/master/docs/en/programming-model.md)
  - Storybook: https://storybook.js.org/docs/react/addons/addons-api",107914493
issue_comments,1510423051,IC_kwDOBm6k_c5aBzoL,,9020979,2023-04-16T16:12:14Z,"# Javascript Plugin Docs (alpha)

## Motivation

The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code. 

## Setup

No external/NPM dependencies are needed.

Plugin behavior is coordinated by the Datasette `manager`. Every page has 1 `manager`.

There are 2 ways to add your plugin to the `manager`.

1. Read `window.__DATASETTE__` if the manager was already loaded.

```js
const manager = window.__DATASETTE__;
```

2. Wait for the `datasette_init` event to fire if your code was loaded before the manager is ready. 

```js
document.addEventListener(""datasette_init"", function (evt) {
  const { detail: manager } = evt;
  
  // register plugin here
});
```

3. Add plugin to the manager by calling `manager.registerPlugin` in a JS file. Each plugin will supply 1 or more hooks with

- unique name (`YOUR_PLUGIN_NAME`)
- a numeric version (starting at `0.1`), 
- configuration value, the details vary by hook. (In this example, `getColumnActions` takes a function)

```js
manager.registerPlugin(""YOUR_PLUGIN_NAME"", {
    version: 0.1,
    makeColumnActions: (columnMeta) => {
      return [
        {
          label: ""Copy name to clipboard"",
          // evt = native click event
          onClick: (evt) => copyToClipboard(columnMeta.column),
        }
      ];
    },
  });
```

There are 2 plugin hooks available to `manager.registerPlugin`:

- `makeColumnActions` - Add items to the cog menu for headers on datasette table pages
- `makeAboveTablePanelConfigs` - Add items to ""tabbed"" panel above the `<table/>` on pages that use the Datasette table template.

While there are additional properties on the `manager`, but it's not advised to depend on them directly as the shape is subject to change.

4. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling [this](https://github.com/simonw/datasette/pull/2052/files#diff-c5ecf3d22075a60d04a4e95da2e15c612cf1bc84e38d777b67ba60dbd156e293) to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.

I welcome ideas for more hooks, or feedback on the current design!

## Examples

See the [example plugins file](https://github.com/simonw/datasette/blob/2d92b9328022d86505261bcdac419b6ed9cb2236/datasette/static/table-example-plugins.js) for additional examples.

## Hooks API Guide

### `makeAboveTablePanelConfigs`

Provide a function with a list of panel objects. Each panel object should contain

1. A unique string `id`
2. A string `label` for the tab
3. A `render` function. The first argument is reference to an HTML [Element](https://developer.mozilla.org/en-US/docs/Web/API/Element). 

Example:

```js
 manager.registerPlugin(""panel-plugin-graphs"", {
    version: 0.1,
    makeAboveTablePanelConfigs: () => {
      return [
        {
          id: 'first-panel',
          label: ""My new panel"",
          render: node => {
            const description = document.createElement('p');
            description.innerText = 'Hello world';
            node.appendChild(description);
          }
        }
      ];
    },
  });
```

### `makeColumnActions`

Provide a function that returns a list of action objects. Each action object has

1. A string `label` for the menu dropdown label
2. An onClick `render` function.

Example:

```js
  manager.registerPlugin(""column-name-plugin"", {
    version: 0.1,
    getColumnActions: (columnMeta) => {
      
      // Info about selected column. 
      const { columnName, columnNotNull, columnType, isPk } = columnMeta;

      return [
        {
          label: ""Copy name to clipboard"",
          onClick: (evt) => copyToClipboard(column),
        }
      ];
    },
  });
```

The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:

- columnName: string (name of the column)
- columnNotNull: boolean
- columnType: sqlite datatype enum (text, number, etc)
- isPk: Whether this is the primary key: boolean

You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).


",107914493
issue_comments,1509951952,IC_kwDOCGYnMM5aAAnQ,,89400147,2023-04-15T20:14:58Z,is this change released ? Because when we run docker containers issue still persists for production deployments.,140912432
issue_comments,1509852821,IC_kwDOBm6k_c5Z_oaV,,3709715,2023-04-15T14:24:45Z,Status?,107914493
issue_comments,1509461324,IC_kwDOBm6k_c5Z-I1M,,9020979,2023-04-15T01:57:06Z,Notes from 1:1 - it _is_ possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters,107914493
issue_comments,1508784533,IC_kwDOJHON9s5Z7jmV,,579727,2023-04-14T15:22:09Z,"Just changing the encoding in `extract_notes` to `utf8` seems to fix it for my titles that were messed up.

![Screen Shot 2023-04-14 at 5 14 18 PM](https://user-images.githubusercontent.com/579727/232086062-e7edc4d1-0880-417a-925b-fd6c65b05155.png)
",611552758