html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1847317568,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79,1847317568,IC_kwDODFdgUs5uG9RA,23789,nedbat,2023-12-08T14:50:13Z,2023-12-08T14:50:13Z,NONE,Adding `&per_page=100` would reduce the number of API requests by 3x.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1570375808,Deploy demo job is failing due to rate limit,
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1846560096,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1846560096,IC_kwDOCGYnMM5uEEVg,9599,simonw,2023-12-08T05:16:44Z,2023-12-08T05:17:20Z,OWNER,"Also tested this manually like so:

```bash
sqlite-utils create-table strict.db strictint id integer size integer --strict
sqlite-utils create-table strict.db notstrictint id integer size integer         
sqlite-utils install sqlite-utils-shell
sqlite-utils shell strict.db                                             
```
```
Attached to strict.db
Type 'exit' to exit.
sqlite-utils> insert into strictint (size) values (4);
1 row affected
sqlite-utils> insert into strictint (size) values ('four');
An error occurred: cannot store TEXT value in INTEGER column strictint.size
sqlite-utils> insert into notstrictint (size) values ('four');
1 row affected
sqlite-utils> commit;
Done
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support,
https://github.com/simonw/sqlite-utils/issues/603#issuecomment-1846555822,https://api.github.com/repos/simonw/sqlite-utils/issues/603,1846555822,IC_kwDOCGYnMM5uEDSu,9599,simonw,2023-12-08T05:09:55Z,2023-12-08T05:10:31Z,OWNER,"I'm unable to replicate this issue. This is with a fresh install of `sqlite-utils==3.35.2`:
```
(base) ~ python3.12
Python 3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> db[""foo""].insert({""bar"": 1})
<Table foo (bar)>
>>> import sys
>>> sys.version
'3.12.0 (v3.12.0:0fb18b02c8, Oct  2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1988525411,Pyhton 3.12 Bug report,
https://github.com/simonw/sqlite-utils/issues/605#issuecomment-1846554637,https://api.github.com/repos/simonw/sqlite-utils/issues/605,1846554637,IC_kwDOCGYnMM5uEDAN,9599,simonw,2023-12-08T05:07:54Z,2023-12-08T05:07:54Z,OWNER,"Thanks for opening an issue - this should help future Google searchers figure out what's going on here.

Another approach here could be to store large integers as `TEXT` in SQLite (or even as `BLOB`).

Both storing as `REAL` and storing as `TEXT/BLOB` feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2007893839,Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?,
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843585454,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843585454,IC_kwDOCGYnMM5t4uGu,22429695,codecov[bot],2023-12-06T19:48:26Z,2023-12-08T05:05:03Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`9286c1b`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/9286c1ba432e890b1bb4b2a1f847b15364c1fa18?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`1698a9d`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.72%.
> Report is 1 commits behind head on main.

> :exclamation: Current head 1698a9d differs from pull request most recent head 61c6e26. Consider uploading reports for the commit 61c6e26 to get more accurate results


<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #604      +/-   ##
==========================================
- Coverage   95.77%   95.72%   -0.06%     
==========================================
  Files           8        8              
  Lines        2842     2852      +10     
==========================================
+ Hits         2722     2730       +8     
- Misses        120      122       +2     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}"
https://github.com/simonw/datasette/issues/2214#issuecomment-1844819002,https://api.github.com/repos/simonw/datasette/issues/2214,1844819002,IC_kwDOBm6k_c5t9bQ6,2874,precipice,2023-12-07T07:36:33Z,2023-12-07T07:36:33Z,NONE,"If I uncheck `expand labels` in the Advanced CSV export dialog, the error does not occur. Re-checking that box and re-running the export does cause the error to occur.

![CleanShot 2023-12-06 at 23 34 58@2x](https://github.com/simonw/datasette/assets/2874/12c6c241-35ce-4ded-8dc7-fc250d809ed9)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029908157,CSV export fails for some `text` foreign key references,
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843975536,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843975536,IC_kwDOCGYnMM5t6NVw,16437338,tkhattra,2023-12-07T01:17:05Z,2023-12-07T01:17:05Z,CONTRIBUTOR,Apologies - I pushed a fix that addresses the mypy failures.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support,
https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843586503,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843586503,IC_kwDOCGYnMM5t4uXH,9599,simonw,2023-12-06T19:49:10Z,2023-12-06T19:49:29Z,OWNER,"This looks really great on first glance - design is good, implementation is solid, tests and documentation look great.

Looks like a couple of `mypy` failures in the tests at the moment:

```
  mypy sqlite_utils tests

sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type ""type[Table]"", variable has type ""type[View]"")  [assignment]
tests/test_lookup.py:156: error: Name ""test_lookup_new_table"" already defined on line 5  [no-redef]
Found 2 errors in 2 files (checked 54 source files)
Error: Process completed with exit code 1.
```

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support,
https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843579184,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843579184,IC_kwDOCGYnMM5t4skw,9599,simonw,2023-12-06T19:43:55Z,2023-12-06T19:43:55Z,OWNER,"Updated documentation:
- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,str and int as aliases for text and integer,
https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843465748,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843465748,IC_kwDOCGYnMM5t4Q4U,9599,simonw,2023-12-06T18:36:51Z,2023-12-06T18:36:51Z,OWNER,I'll add `bytes` too - `float` already works. This makes sense because when you are working with the Python API you use `str` and `float` and `bytes` and `int` to specify column types.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,str and int as aliases for text and integer,
https://github.com/simonw/datasette/issues/2213#issuecomment-1843072926,https://api.github.com/repos/simonw/datasette/issues/2213,1843072926,IC_kwDOBm6k_c5t2w-e,536941,fgregg,2023-12-06T15:05:44Z,2023-12-06T15:05:44Z,CONTRIBUTOR,"it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like.

i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2028698018,feature request: gzip compression of database downloads,
https://github.com/simonw/datasette/issues/670#issuecomment-1816642044,https://api.github.com/repos/simonw/datasette/issues/670,1816642044,IC_kwDOBm6k_c5sR8H8,16142258,tf13,2023-11-17T15:32:20Z,2023-11-17T15:32:20Z,NONE,Any progress on this? It would be very helpful on my end as well. Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL,
https://github.com/simonw/sqlite-utils/issues/344#issuecomment-1815825863,https://api.github.com/repos/simonw/sqlite-utils/issues/344,1815825863,IC_kwDOCGYnMM5sO03H,16437338,tkhattra,2023-11-17T06:44:49Z,2023-11-17T06:44:49Z,CONTRIBUTOR,"hello Simon,

I've added more STRICT table support per https://github.com/simonw/sqlite-utils/issues/344#issuecomment-982014776 in changeset https://github.com/simonw/sqlite-utils/commit/e4b9b582cdb4e48430865f8739f341bc8017c1e4.
It also fixes table.transform() to preserve STRICT mode.
Please pull if you deem appropriate. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1066474200,Support STRICT tables,
https://github.com/simonw/datasette/pull/2209#issuecomment-1812753347,https://api.github.com/repos/simonw/datasette/issues/2209,1812753347,IC_kwDOBm6k_c5sDGvD,22429695,codecov[bot],2023-11-15T15:31:12Z,2023-11-15T15:31:12Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`c88414b`)](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.


<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2209   +/-   ##
=======================================
  Coverage   92.69%   92.69%           
=======================================
  Files          40       40           
  Lines        6047     6047           
=======================================
  Hits         5605     5605           
  Misses        442      442           
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}"
https://github.com/simonw/datasette/pull/2209#issuecomment-1812750369,https://api.github.com/repos/simonw/datasette/issues/2209,1812750369,IC_kwDOBm6k_c5sDGAh,198537,rgieseke,2023-11-15T15:29:37Z,2023-11-15T15:29:37Z,CONTRIBUTOR,"Looks like tests are passing now but there is an issue with yaml loading and/or cog.

https://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value,
https://github.com/simonw/datasette/pull/2209#issuecomment-1812623778,https://api.github.com/repos/simonw/datasette/issues/2209,1812623778,IC_kwDOBm6k_c5sCnGi,198537,rgieseke,2023-11-15T14:22:42Z,2023-11-15T15:24:09Z,CONTRIBUTOR,"Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value,
https://github.com/simonw/datasette/issues/2208#issuecomment-1812617851,https://api.github.com/repos/simonw/datasette/issues/2208,1812617851,IC_kwDOBm6k_c5sClp7,198537,rgieseke,2023-11-15T14:18:58Z,2023-11-15T14:18:58Z,CONTRIBUTOR,"Without aliases:

![image](https://github.com/simonw/datasette/assets/198537/d9703d3b-9733-4e87-9954-4fc60a07784a)

The proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column).

![image](https://github.com/simonw/datasette/assets/198537/a37a0a1a-c36a-4c78-bdce-01b582637cc6)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994857251,No suggested facets when a column named 'value' is included,
https://github.com/simonw/datasette/pull/2206#issuecomment-1801888957,https://api.github.com/repos/simonw/datasette/issues/2206,1801888957,IC_kwDOBm6k_c5rZqS9,22429695,codecov[bot],2023-11-08T13:26:13Z,2023-11-08T13:26:13Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`eec10df`)](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2206   +/-   ##
=======================================
  Coverage   92.69%   92.69%           
=======================================
  Files          40       40           
  Lines        6047     6047           
=======================================
  Hits         5605     5605           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1983600865,Bump the python-packages group with 1 update,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}"
https://github.com/simonw/datasette/pull/2202#issuecomment-1801876943,https://api.github.com/repos/simonw/datasette/issues/2202,1801876943,IC_kwDOBm6k_c5rZnXP,49699333,dependabot[bot],2023-11-08T13:19:00Z,2023-11-08T13:19:00Z,CONTRIBUTOR,Superseded by #2206.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1959278971,Bump the python-packages group with 1 update,"{""id"": 29110, ""slug"": ""dependabot"", ""node_id"": ""MDM6QXBwMjkxMTA="", ""owner"": {""login"": ""github"", ""id"": 9919, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjk5MTk="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/9919?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/github"", ""html_url"": ""https://github.com/github"", ""followers_url"": ""https://api.github.com/users/github/followers"", ""following_url"": ""https://api.github.com/users/github/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/github/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/github/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/github/subscriptions"", ""organizations_url"": ""https://api.github.com/users/github/orgs"", ""repos_url"": ""https://api.github.com/users/github/repos"", ""events_url"": ""https://api.github.com/users/github/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/github/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Dependabot"", ""description"": """", ""external_url"": ""https://dependabot-api.githubapp.com"", ""html_url"": ""https://github.com/apps/dependabot"", ""created_at"": ""2019-04-16T22:34:25Z"", ""updated_at"": ""2023-10-12T13:35:09Z"", ""permissions"": {""checks"": ""write"", ""contents"": ""write"", ""issues"": ""write"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""read"", ""vulnerability_alerts"": ""read"", ""workflows"": ""write""}, ""events"": [""check_suite"", ""issues"", ""issue_comment"", ""label"", ""pull_request"", ""pull_request_review"", ""pull_request_review_comment"", ""repository""]}"
https://github.com/simonw/datasette/issues/2205#issuecomment-1794054390,https://api.github.com/repos/simonw/datasette/issues/2205,1794054390,IC_kwDOBm6k_c5q7xj2,9599,simonw,2023-11-06T04:09:43Z,2023-11-06T04:10:34Z,OWNER,"That `keep_blank_values=True` is from https://github.com/simonw/datasette/commit/0934844c0b6d124163d0185fb6a41ba5a71433da

Commit message:
>  request.post_vars() no longer discards empty values 

Relevant test:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/tests/test_internals_request.py#L19-L27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1978023780,request.post_vars() method obliterates form keys with multiple values,
https://github.com/simonw/datasette/issues/2205#issuecomment-1794052079,https://api.github.com/repos/simonw/datasette/issues/2205,1794052079,IC_kwDOBm6k_c5q7w_v,9599,simonw,2023-11-06T04:06:05Z,2023-11-06T04:08:50Z,OWNER,"It should return a `MultiParams`:

https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/__init__.py#L900-L917

Change needs to be made before 1.0.

```python
return MultiParams(urllib.parse.parse_qs(body.decode(""utf-8"")))
```
Need to remember why I was using `keep_blank_values=True` there and check that using `MultiParams` doesn't conflict with that reason.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1978023780,request.post_vars() method obliterates form keys with multiple values,
https://github.com/simonw/datasette/issues/2205#issuecomment-1793880738,https://api.github.com/repos/simonw/datasette/issues/2205,1793880738,IC_kwDOBm6k_c5q7HKi,9599,simonw,2023-11-05T23:26:14Z,2023-11-05T23:26:14Z,OWNER,"I found this problem while trying to use WTForms with this pattern:

```python
choices = [(col, col) for col in await db.table_columns(table)]

class ConfigForm(Form):
    template = TextAreaField(""Template"")
    api_token = PasswordField(""OpenAI API token"")
    columns = MultiCheckboxField('Columns', choices=choices)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1978023780,request.post_vars() method obliterates form keys with multiple values,
https://github.com/simonw/datasette/issues/1415#issuecomment-1793787454,https://api.github.com/repos/simonw/datasette/issues/1415,1793787454,IC_kwDOBm6k_c5q6wY-,45269373,jimmybutton,2023-11-05T16:44:49Z,2023-11-05T16:46:59Z,NONE,"thanks for documenting this @bendnorman! got stuck at exactly the same point `gcloud builds submit ... returned non-zero exit status 1`, without a clue why this was happening. i now managed to get the github action to deploy datasette by assigning the following roles to the service account: `roles/run.admin`, `roles/storage.admin`, `roles/cloudbuild.builds.builder`, `roles/viewer`, `roles/iam.serviceAccountUser`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",959137143,feature request: document minimum permissions for service account for cloudrun,
https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1793278279,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1793278279,IC_kwDOCGYnMM5q40FH,9599,simonw,2023-11-04T00:58:03Z,2023-11-04T00:58:03Z,OWNER,I'm going to abandon this PR and ship the 3.12 testing change directly to `main`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview,
https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708693020,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708693020,IC_kwDOCGYnMM5l2JYc,22429695,codecov[bot],2023-09-06T16:14:03Z,2023-11-04T00:54:25Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`347fdc8`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/347fdc865e91b8d3410f49a5c9d5b499fbb594c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74% compared to head [(`1f14df1`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #591   +/-   ##
=======================================
  Coverage   95.74%   95.74%           
=======================================
  Files           8        8           
  Lines        2842     2842           
=======================================
  Hits         2721     2721           
  Misses        121      121           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview,
https://github.com/simonw/sqlite-utils/pull/596#issuecomment-1793274869,https://api.github.com/repos/simonw/sqlite-utils/issues/596,1793274869,IC_kwDOCGYnMM5q4zP1,9599,simonw,2023-11-04T00:47:55Z,2023-11-04T00:47:55Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1919296686,"Fixes mapping for time fields related to mysql, closes #522",
https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793274485,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793274485,IC_kwDOCGYnMM5q4zJ1,9599,simonw,2023-11-04T00:46:55Z,2023-11-04T00:46:55Z,OWNER,"Manually tested. Before:

![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/3bdd30ea-1a54-4fec-b42d-793130a17bc1)

After:

![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/015d4e4e-b40c-4a93-81f5-1a4adef69b11)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,Fixed issue #433 - CLI eats cursor,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1793274350,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1793274350,IC_kwDOCGYnMM5q4zHu,9599,simonw,2023-11-04T00:46:30Z,2023-11-04T00:46:30Z,OWNER,"And a GIF of the fix after applying:
- #598

![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/35829aec-c9ac-4925-a8e6-ffe7c2ab0d96)
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1793273968,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1793273968,IC_kwDOCGYnMM5q4zBw,9599,simonw,2023-11-04T00:45:19Z,2023-11-04T00:45:19Z,OWNER,"Here's an animated GIF that demonstrates the bug:

![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/18e54b38-a466-4384-9162-114eb01526a2)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793272429,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793272429,IC_kwDOCGYnMM5q4ypt,9599,simonw,2023-11-04T00:40:34Z,2023-11-04T00:40:34Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,Fixed issue #433 - CLI eats cursor,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793269219,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793269219,IC_kwDOCGYnMM5q4x3j,9599,simonw,2023-11-04T00:34:33Z,2023-11-04T00:34:33Z,OWNER,"The GIS tests now pass in that container too:
```bash
pytest tests/test_gis.py
```
```
======================== test session starts =========================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/sqlite-utils
plugins: hypothesis-6.88.1
collected 12 items                                                   

tests/test_gis.py ............                                 [100%]

========================= 12 passed in 0.48s =========================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path,
https://github.com/simonw/sqlite-utils/issues/599#issuecomment-1793268750,https://api.github.com/repos/simonw/sqlite-utils/issues/599,1793268750,IC_kwDOCGYnMM5q4xwO,9599,simonw,2023-11-04T00:33:25Z,2023-11-04T00:33:25Z,OWNER,"See details of how I tested this here:
- https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126

Short version: having applied this fix, the following command (on simulated `aarch64`):
```bash
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Outputs:
```json
[{""spatialite_version()"": ""5.0.1""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1976986318,Cannot find spatialite on arm64 linux,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793268126,IC_kwDOCGYnMM5q4xme,9599,simonw,2023-11-04T00:31:34Z,2023-11-04T00:31:34Z,OWNER,"Testing this manually on macOS using Docker Desk top like this:
```bash
docker run -it --rm arm64v8/ubuntu /bin/bash
```
Then inside the container:
```bash
uname -m
```
Outputs: `aarch64`

Then:
```bash
apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y
cd /tmp
git clone https://github.com/simonw/sqlite-utils
cd sqlite-utils
python3 -m venv venv
source venv/bin/activate
pip install -e '.[test]'
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Which output:
```
Traceback (most recent call last):
  File ""/tmp/sqlite-utils/venv/bin/sqlite-utils"", line 33, in <module>
    sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')())
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1157, in __call__
    return self.main(*args, **kwargs)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1078, in main
    rv = self.invoke(ctx)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 783, in invoke
    return __callback(*args, **kwargs)
  File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 1959, in memory
    _load_extensions(db, load_extension)
  File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 3232, in _load_extensions
    if "":"" in ext:
TypeError: argument of type 'NoneType' is not iterable
```
Then I ran this:
```bash
git checkout -b MikeCoats-spatialite-paths-linux-arm main
git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm
```
And now:
```bash
sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite
```
Outputs:
```json
[{""spatialite_version()"": ""5.0.1""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793264654,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793264654,IC_kwDOCGYnMM5q4wwO,22429695,codecov[bot],2023-11-04T00:22:07Z,2023-11-04T00:27:29Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`622c3a5`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/622c3a5a7dd53a09c029e2af40c2643fe7579340?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b1a6076`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #600   +/-   ##
=======================================
  Coverage   95.77%   95.77%           
=======================================
  Files           8        8           
  Lines        2840     2840           
=======================================
  Hits         2720     2720           
  Misses        120      120           
```


| [Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <ø> (ø)` | |
| [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <ø> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}"
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793265952,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793265952,IC_kwDOCGYnMM5q4xEg,9599,simonw,2023-11-04T00:25:34Z,2023-11-04T00:25:34Z,OWNER,The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path,
https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793263638,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793263638,IC_kwDOCGYnMM5q4wgW,9599,simonw,2023-11-04T00:19:58Z,2023-11-04T00:19:58Z,OWNER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path,
https://github.com/simonw/datasette/issues/949#issuecomment-1791911093,https://api.github.com/repos/simonw/datasette/issues/949,1791911093,IC_kwDOBm6k_c5qzmS1,9599,simonw,2023-11-03T05:28:09Z,2023-11-03T05:28:58Z,OWNER,"Datasette is using that now, see:
- #1893","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/949#issuecomment-1791571572,https://api.github.com/repos/simonw/datasette/issues/949,1791571572,IC_kwDOBm6k_c5qyTZ0,498744,mhkeller,2023-11-02T21:36:24Z,2023-11-02T21:36:24Z,NONE,"FWIW, code mirror 6 now has this standard although if you want table-specific suggestions, you'd have to handle parsing out which table the user is querying yourself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints,
https://github.com/simonw/datasette/issues/411#issuecomment-1779267468,https://api.github.com/repos/simonw/datasette/issues/411,1779267468,IC_kwDOBm6k_c5qDXeM,363004,hcarter333,2023-10-25T13:23:04Z,2023-10-25T13:23:04Z,NONE,"Using the [Counties example](https://us-counties.datasette.io/counties/county_for_latitude_longitude?longitude=-122&latitude=37), I was able to pull out the MakePoint method as 

MakePoint(cast(rm_rnb_history_pres.rx_lng as float), cast(rm_rnb_history_pres.rx_lat as float)) as geometry

which worked, giving me a geometry column.

![image](https://github.com/simonw/datasette/assets/363004/6393b712-9e3d-416d-ba37-202934d5f604)


gave

![image](https://github.com/simonw/datasette/assets/363004/219db7b2-8107-41b3-a049-ef4d6bd7ac7a)

I believe it's the cast to float that does the trick. Prior to using the cast, I also received a 'wrong number of arguments' eror.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",410384988,How to pass named parameter into spatialite MakePoint() function,
https://github.com/simonw/datasette/pull/2202#issuecomment-1777247375,https://api.github.com/repos/simonw/datasette/issues/2202,1777247375,IC_kwDOBm6k_c5p7qSP,22429695,codecov[bot],2023-10-24T13:49:27Z,2023-10-24T13:49:27Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`be4d0f0`)](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2202   +/-   ##
=======================================
  Coverage   92.69%   92.69%           
=======================================
  Files          40       40           
  Lines        6047     6047           
=======================================
  Hits         5605     5605           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1959278971,Bump the python-packages group with 1 update,
https://github.com/simonw/datasette/pull/2200#issuecomment-1777228352,https://api.github.com/repos/simonw/datasette/issues/2200,1777228352,IC_kwDOBm6k_c5p7lpA,49699333,dependabot[bot],2023-10-24T13:40:25Z,2023-10-24T13:40:25Z,CONTRIBUTOR,Superseded by #2202.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1949756141,Bump the python-packages group with 1 update,
https://github.com/simonw/datasette/issues/1655#issuecomment-1767248394,https://api.github.com/repos/simonw/datasette/issues/1655,1767248394,IC_kwDOBm6k_c5pVhIK,6262071,yejiyang,2023-10-17T21:53:17Z,2023-10-17T21:53:17Z,NONE,"@fgregg, I am happy to do that and just could not find a way to create issues at your fork repo. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/issues/1655#issuecomment-1767219901,https://api.github.com/repos/simonw/datasette/issues/1655,1767219901,IC_kwDOBm6k_c5pVaK9,536941,fgregg,2023-10-17T21:29:03Z,2023-10-17T21:29:03Z,CONTRIBUTOR,@yejiyang why don’t you move this discussion to my fork to spare simon’s notifications ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/issues/1655#issuecomment-1767133832,https://api.github.com/repos/simonw/datasette/issues/1655,1767133832,IC_kwDOBm6k_c5pVFKI,6262071,yejiyang,2023-10-17T20:37:18Z,2023-10-17T21:12:48Z,NONE,"@fgregg Thanks for your reply. I tried to use your fork branch `datasette = {url = ""https://github.com/fgregg/datasette/archive/refs/heads/no_limit_csv_publish.zip""}` and got error - TypeError: 'str' object is not callable. I used the same templates as in your branch [here ](https://github.com/labordata/warehouse/tree/main/templates).

```
INFO:     127.0.0.1:47232 - ""GET /-/static/sql-formatter-2.3.3.min.js HTTP/1.1"" 200 OK
Traceback (most recent call last):
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py"", line 1632, in route_path
    response = await view(request, send)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py"", line 1814, in async_view_fn
    response = await async_call_with_supported_arguments(
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/utils/__init__.py"", line 1016, in async_call_with_supported_arguments
    return await fn(*call_with)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py"", line 673, in table_view
    response = await table_view_traced(datasette, request)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/views/table.py"", line 822, in table_view_traced
    await datasette.render_template(
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/app.py"", line 1307, in render_template
    return await template.render_async(template_context)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py"", line 1324, in render_async
    return self.environment.handle_exception()
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py"", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/jinja2/environment.py"", line 1321, in <listcomp>
    [n async for n in self.root_render_func(ctx)]  # type: ignore
  File ""templates/table.html"", line 1, in top-level template code
    {% extends ""base.html"" %}
  File ""/home/jiyang/github/global-chemical-inventory-database/.venv/lib/python3.10/site-packages/datasette/templates/base.html"", line 62, in top-level template code
    {% block content %}
  File ""templates/table.html"", line 24, in block 'content'
    <div class=""page-header"" style=""border-color: #{{ database_color(database) }}"">
TypeError: 'str' object is not callable
INFO:     127.0.0.1:47228 - ""GET /zeropm-v0-0-3/api_ready_query HTTP/1.1"" 500 Internal Server Error
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/issues/1655#issuecomment-1766994810,https://api.github.com/repos/simonw/datasette/issues/1655,1766994810,IC_kwDOBm6k_c5pUjN6,536941,fgregg,2023-10-17T19:01:59Z,2023-10-17T19:01:59Z,CONTRIBUTOR,"hi @yejiyang, have your tried using my fork of datasette: https://github.com/fgregg/datasette/tree/no_limit_csv_publish
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/issues/1655#issuecomment-1761630595,https://api.github.com/repos/simonw/datasette/issues/1655,1761630595,IC_kwDOBm6k_c5pAFmD,6262071,yejiyang,2023-10-13T14:37:48Z,2023-10-13T14:37:48Z,NONE,"Hi @fgregg, I came across this issue and found your setup at labordata.bunkum.us can help me with a research project at https://database.zeropm.eu/. 
I really like the approach [here](https://labordata.bunkum.us/f7-06c761c?sql=select+*+from+f7) when dealing with a custom SQL query returning more than 1000 rows:
1) At the table in HTML page, only first 1000 rows displayed;
2) When click the ""Download this data as a CSV Spreadsheet(All Rows)"" button, a csv with ALL ROWS (could be > 100 Mb) get downloaded.

I am trying to repeat the setup but have yet to be successful so far. 
What I tried:
1) copy the query.html & table.html templates from this [github repo](https://github.com/labordata/warehouse/tree/main/templates) and use it my project
2) use the same datasette version 1.0a2.

Do you know what else I should try to set it correctly? I appreciate your help.

@simonw I would like to use this opportunity to thank you for developing & maintaining such an amazing project. 
I introduce your datasette to several projects in my institute. I am also interested in your cloud version.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1163369515,query result page is using 400mb of browser memory 40x size of html page and 400x size of csv data,
https://github.com/simonw/datasette/issues/2196#issuecomment-1760560526,https://api.github.com/repos/simonw/datasette/issues/2196,1760560526,IC_kwDOBm6k_c5o8AWO,1892194,Olshansk,2023-10-13T00:07:07Z,2023-10-13T00:07:07Z,NONE,That worked!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1910269679,Discord invite link returns 401,
https://github.com/simonw/datasette/pull/2052#issuecomment-1760552652,https://api.github.com/repos/simonw/datasette/issues/2052,1760552652,IC_kwDOBm6k_c5o7-bM,9599,simonw,2023-10-12T23:59:21Z,2023-10-12T23:59:21Z,OWNER,I'm landing this despite the cog failures. I'll fix them on main if I have to.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1632867333,https://api.github.com/repos/simonw/datasette/issues/2052,1632867333,IC_kwDOBm6k_c5hU5QF,22429695,codecov[bot],2023-07-12T16:38:27Z,2023-10-12T23:52:24Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified lines are covered by tests :white_check_mark:
> Comparison is base [(`3feed1f`)](https://app.codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`8ae479c`)](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.
> Report is 112 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2052      +/-   ##
==========================================
+ Coverage   92.46%   92.69%   +0.22%     
==========================================
  Files          38       40       +2     
  Lines        5750     6047     +297     
==========================================
+ Hits         5317     5605     +288     
- Misses        433      442       +9     
```


[see 19 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2052/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1760545012,https://api.github.com/repos/simonw/datasette/issues/2052,1760545012,IC_kwDOBm6k_c5o78j0,9599,simonw,2023-10-12T23:48:16Z,2023-10-12T23:48:16Z,OWNER,Oh! I think I broke Cog on `main` and these tests are running against this branch rebased against main.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1760542865,https://api.github.com/repos/simonw/datasette/issues/2052,1760542865,IC_kwDOBm6k_c5o78CR,9599,simonw,2023-10-12T23:44:53Z,2023-10-12T23:45:15Z,OWNER,"Weird, the `cog` check is failing in CI.

```
Run cog --check docs/*.rst
  cog --check docs/*.rst
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib
Check failed
Checking docs/authentication.rst
Checking docs/binary_data.rst
Checking docs/changelog.rst
Checking docs/cli-reference.rst
Checking docs/configuration.rst  (changed)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/issues/2199#issuecomment-1760441535,https://api.github.com/repos/simonw/datasette/issues/2199,1760441535,IC_kwDOBm6k_c5o7jS_,9599,simonw,2023-10-12T22:08:42Z,2023-10-12T22:08:42Z,OWNER,Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/issues/2196#issuecomment-1760417555,https://api.github.com/repos/simonw/datasette/issues/2196,1760417555,IC_kwDOBm6k_c5o7dcT,9599,simonw,2023-10-12T21:54:52Z,2023-10-12T21:54:52Z,OWNER,"I can't replicate this bug. Closing, but please re-open if it's still happening.

As an aside, the link I promote is https://datasette.io/discord which redirects:

```
curl -i 'https://datasette.io/discord'
HTTP/2 301 
location: https://discord.gg/ktd74dm5mw
content-type: text/plain
x-cloud-trace-context: 8dcfd08d3d1fa44f7ee78568e0f5305e;o=1
date: Thu, 12 Oct 2023 21:54:17 GMT
server: Google Frontend
content-length: 0
```
```
curl -i 'https://discord.gg/ktd74dm5mw'
HTTP/2 301 
date: Thu, 12 Oct 2023 21:54:28 GMT
content-type: text/plain;charset=UTF-8
content-length: 0
location: https://discord.com/invite/ktd74dm5mw
strict-transport-security: max-age=31536000; includeSubDomains; preload
permissions-policy: interest-cohort=()
x-content-type-options: nosniff
x-frame-options: DENY
x-robots-tag: noindex, nofollow, noarchive, nocache, noimageindex, noodp
x-xss-protection: 1; mode=block
report-to: {""endpoints"":[{""url"":""https:\/\/a.nel.cloudflare.com\/report\/v3?s=Dzzrf%2FgGkfFxtzSAQ46slMVDLcFjsH9fsvVkzHtgUUiZ891rXAa6LvTRpHK%2BdSMSQ54F57hS9z1mZXXklIbONZW1bfBuFjSK9J4XmjjLjsFUulMXvpjfCLkB6PI%3D""}],""group"":""cf-nel"",""max_age"":604800}
nel: {""success_fraction"":0,""report_to"":""cf-nel"",""max_age"":604800}
server: cloudflare
cf-ray: 815294ddff282511-SJC
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1910269679,Discord invite link returns 401,
https://github.com/simonw/datasette/issues/2199#issuecomment-1760413191,https://api.github.com/repos/simonw/datasette/issues/2199,1760413191,IC_kwDOBm6k_c5o7cYH,9599,simonw,2023-10-12T21:52:25Z,2023-10-12T21:52:25Z,OWNER,"Demo of that logic:
```
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-1.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch metadata-new-1.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette.yaml
$ touch datasette.yaml
$ datasette upgrade metadata-to-config ../datasette/metadata.json
Upgrading ../datasette/metadata.json to new metadata.yaml format
New metadata.yaml file will be written to metadata-new-2.yaml
New datasette.yaml file will be written to datasette-new.yaml
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/issues/2199#issuecomment-1760412424,https://api.github.com/repos/simonw/datasette/issues/2199,1760412424,IC_kwDOBm6k_c5o7cMI,9599,simonw,2023-10-12T21:51:44Z,2023-10-12T21:51:44Z,OWNER,"Started playing with this plugin idea, now tearing myself away to work on something more important:
```python
from datasette import hookimpl
import click
import pathlib


@hookimpl
def register_commands(cli):
    @cli.group()
    def upgrade():
        """"""
        Apply configuration upgrades to an existing Datasette instance
        """"""
        pass


    @upgrade.command()
    @click.argument(
        ""metadata"", type=click.Path(exists=True)
    )
    @click.option(
        ""new_metadata"", ""-m"", ""--new-metadata"", help=""Path to new metadata.yaml file"", type=click.Path(exists=False)
    )
    @click.option(
        ""new_datasette"", ""-c"", ""--new-datasette"", help=""Path to new datasette.yaml file"", type=click.Path(exists=False)
    )
    @click.option(
        ""output_dir"", ""-e"", ""--output-dir"", help=""Directory to write new files to"", type=click.Path(), default="".""
    )
    def metadata_to_config(metadata, new_metadata, new_datasette, output_dir):
        """"""
        Upgrade an existing metadata.json/yaml file to the new metadata.yaml and
        datasette.yaml split introduced prior to Datasette 1.0.
        """"""
        print(""Upgrading {} to new metadata.yaml format"".format(metadata))
        output_dir = pathlib.Path(output_dir)
        if not new_metadata:
            # Pick a filename for the new metadata.yaml file that does not yet exist
            new_metadata = pick_filename(""metadata"", output_dir)
        if not new_datasette:
            new_datasette = pick_filename(""datasette"", output_dir)
        print(""New metadata.yaml file will be written to {}"".format(new_metadata))
        print(""New datasette.yaml file will be written to {}"".format(new_datasette))


def pick_filename(base, output_dir):
    options = [""{}.yaml"".format(base), ""{}-new.yaml"".format(base)]
    i = 0
    while True:
        option = options.pop(0)
        option_path = output_dir / option
        if not option_path.exists():
            return option_path
        # If we ran out
        if not options:
            i += 1
            options = [""{}-new-{}.yaml"".format(base, i)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/issues/2199#issuecomment-1760411937,https://api.github.com/repos/simonw/datasette/issues/2199,1760411937,IC_kwDOBm6k_c5o7cEh,9599,simonw,2023-10-12T21:51:16Z,2023-10-12T21:51:16Z,OWNER,"I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides.

If people want to keep their comments I'm happy to leave them to copy those over by hand.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731,https://api.github.com/repos/simonw/datasette/issues/2199,1760401731,IC_kwDOBm6k_c5o7ZlD,15178711,asg017,2023-10-12T21:41:42Z,2023-10-12T21:41:42Z,CONTRIBUTOR,"I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins.

One thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/issues/2199#issuecomment-1760396195,https://api.github.com/repos/simonw/datasette/issues/2199,1760396195,IC_kwDOBm6k_c5o7YOj,9599,simonw,2023-10-12T21:36:25Z,2023-10-12T21:36:25Z,OWNER,"Related idea: how about a `datasette-upgrade` plugin which adds a `datasette upgrade` command that can be used to automate this process?

Maybe something like this:
```bash
datasette install datasette-upgrade
datasette upgrade metadata-to-config metadata.json
```
This would output two new files: `metadata.yaml` and `datasette.yaml`. If files with those names existed already in the current directory they would be called `metadata-new.yaml` and `datasette-new.yaml`.

The command would tell you what it did:
```
Your metadata.json file has been rewritten as two files:

    metadata-new.yaml
    datasette.yaml

Start Datasette like this to try them out:

    datasette -m metadata-new.yaml -c datasette.yaml
```
The command is `datasette upgrade metadata-to-config` because `metadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/issues/2199#issuecomment-1759952247,https://api.github.com/repos/simonw/datasette/issues/2199,1759952247,IC_kwDOBm6k_c5o5r13,9599,simonw,2023-10-12T16:23:10Z,2023-10-12T16:23:10Z,OWNER,"Some options for where this could go:
- Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides.
- On a new page? We could have a ""upgrade instructions"" page in the documentation.
- At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page

I'm leaning towards the third option at the moment.

But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml,
https://github.com/simonw/datasette/pull/2190#issuecomment-1759948683,https://api.github.com/repos/simonw/datasette/issues/2190,1759948683,IC_kwDOBm6k_c5o5q-L,9599,simonw,2023-10-12T16:20:41Z,2023-10-12T16:20:41Z,OWNER,I'm going to land this and open a new issue for the upgrade instructions.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json",
https://github.com/simonw/datasette/pull/2190#issuecomment-1759947534,https://api.github.com/repos/simonw/datasette/issues/2190,1759947534,IC_kwDOBm6k_c5o5qsO,9599,simonw,2023-10-12T16:19:59Z,2023-10-12T16:19:59Z,OWNER,It would be nice if we could catch that and turn that into a less intimidating Click exception too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json",
https://github.com/simonw/datasette/pull/2190#issuecomment-1759947021,https://api.github.com/repos/simonw/datasette/issues/2190,1759947021,IC_kwDOBm6k_c5o5qkN,9599,simonw,2023-10-12T16:19:38Z,2023-10-12T16:19:38Z,OWNER,"

This looks good and works well. The error from this currently looks like:

```
datasette -m metadata.json -p 8844
Traceback (most recent call last):
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 33, in <module>
    sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')())
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 98, in wrapped
    return fn(*args, **kwargs)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 546, in serve
    metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read()))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 1282, in fail_if_plugins_in_metadata
    raise Exception(
Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.
```
With wrapping:

`Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.`

I think we should link directly to documentation that tells people how to perform this upgrade.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json",
https://github.com/simonw/datasette/pull/2191#issuecomment-1724480716,https://api.github.com/repos/simonw/datasette/issues/2191,1724480716,IC_kwDOBm6k_c5myXzM,22429695,codecov[bot],2023-09-18T21:28:36Z,2023-10-12T16:15:40Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
All modified lines are covered by tests :white_check_mark:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`0135e7c`)](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
> Report is 14 commits behind head on main.

> :exclamation: Current head 0135e7c differs from pull request most recent head 18b48f8. Consider uploading reports for the commit 18b48f8 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2191      +/-   ##
==========================================
- Coverage   92.69%   92.68%   -0.02%     
==========================================
  Files          40       40              
  Lines        6039     6042       +3     
==========================================
+ Hits         5598     5600       +2     
- Misses        441      442       +1     
```


| [Files](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.09% <100.00%> (-0.11%)` | :arrow_down: |
| [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.36% <100.00%> (+0.01%)` | :arrow_up: |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2191/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`,  `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`",
https://github.com/simonw/datasette/issues/2197#issuecomment-1752096661,https://api.github.com/repos/simonw/datasette/issues/2197,1752096661,IC_kwDOBm6k_c5obt-V,9599,simonw,2023-10-08T16:17:04Z,2023-10-08T16:17:29Z,OWNER,"https://lite.datasette.io/?install=datasette-packages#/-/packages confirms that Datasette Lite still works (`click-default-group-wheel` was originally built to allow it to run) and that it's now using these packages:
```
    ""click-default-group"": ""1.2.4"",
    ""datasette"": ""0.64.5"",
```
Full list:
```json
{
    ""aiofiles"": ""23.2.1"",
    ""anyio"": ""3.7.1"",
    ""asgi-csrf"": ""0.9"",
    ""asgiref"": ""3.7.2"",
    ""certifi"": ""2022.12.7"",
    ""click"": ""8.1.3"",
    ""click-default-group"": ""1.2.4"",
    ""datasette"": ""0.64.5"",
    ""datasette-packages"": ""0.2"",
    ""h11"": ""0.12.0"",
    ""httpcore"": ""0.15.0"",
    ""httpx"": ""0.23.0"",
    ""hupper"": ""1.12"",
    ""idna"": ""3.4"",
    ""itsdangerous"": ""2.1.2"",
    ""janus"": ""1.0.0"",
    ""Jinja2"": ""3.1.2"",
    ""MarkupSafe"": ""2.1.2"",
    ""mergedeep"": ""1.3.4"",
    ""micropip"": ""0.3.0"",
    ""packaging"": ""23.0"",
    ""Pint"": ""0.22"",
    ""pip"": ""23.2.1"",
    ""pluggy"": ""1.0.0"",
    ""pyparsing"": ""3.0.9"",
    ""python-multipart"": ""0.0.6"",
    ""PyYAML"": ""6.0"",
    ""rfc3986"": ""1.5.0"",
    ""setuptools"": ""67.6.1"",
    ""sniffio"": ""1.3.0"",
    ""typing-extensions"": ""4.5.0"",
    ""uvicorn"": ""0.23.2""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1930008379,click-default-group-wheel dependency conflict,
https://github.com/simonw/datasette/issues/2197#issuecomment-1752095961,https://api.github.com/repos/simonw/datasette/issues/2197,1752095961,IC_kwDOBm6k_c5obtzZ,9599,simonw,2023-10-08T16:13:42Z,2023-10-08T16:14:39Z,OWNER,"Confirmed - I ran this in a fresh virtual environment:
```bash
pip install --no-cache datasette
```
And now:
```bash
pip freeze | grep click
```
```
click==8.1.7
click-default-group==1.2.4
```
```bash
datasette --version
```
```
datasette, version 0.64.5
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1930008379,click-default-group-wheel dependency conflict,
https://github.com/simonw/datasette/issues/2197#issuecomment-1752093039,https://api.github.com/repos/simonw/datasette/issues/2197,1752093039,IC_kwDOBm6k_c5obtFv,9599,simonw,2023-10-08T15:59:53Z,2023-10-08T15:59:53Z,OWNER,"Replicated this myself:
```bash
cd /tmp
mkdir dddd
cd dddd
pipenv shell
pip install datasette
pip freeze | grep click
```
```
click==8.1.7
click-default-group==1.2.4
click-default-group-wheel==1.2.3
```
Yeah this is bad, I'll ship a `0.64.5` release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1930008379,click-default-group-wheel dependency conflict,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1747231893,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1747231893,IC_kwDOCGYnMM5oJKSV,62745,spookylukey,2023-10-04T16:15:09Z,2023-10-04T16:28:21Z,CONTRIBUTOR,"I confirm the bug, as above, and that @jonafato 's patch fixes it for me. However, it's not the right fix. The problem is that ProgressBar is being used in the wrong way. This also results in two lines being printed instead of one, like this:

```
  [#######-----------------------------]   20%
  [####################################]  100%%  
```
The bug is reproducible for me in any terminal, including Gnome Terminal and Guake, and VSCode. With VSCode I can use this launch.json to reproduce it:

```json

{
    ""version"": ""0.2.0"",
    ""configurations"": [
        {
            ""name"": ""Python: Module"",
            ""type"": ""python"",
            ""request"": ""launch"",
            ""module"": ""sqlite_utils"",
            ""justMyCode"": false,
            ""args"": [""insert"", ""test.db"", ""test"", ""--csv"", ""tests/sniff/example1.csv""]
        }
    ]
}
```

[edit - deleted my analysis of why the current code is wrong, which was confused and confusing]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/datasette/pull/1870#issuecomment-1745568725,https://api.github.com/repos/simonw/datasette/issues/1870,1745568725,IC_kwDOBm6k_c5oC0PV,2495794,jdangerx,2023-10-03T19:12:37Z,2023-10-03T19:12:37Z,NONE,"Hello! Resurrecting this issue since we're running into something similar with data.catalyst.coop as our database files have ballooned up to several GB. Our Cloud Run revisions now require huge amounts of RAM to start up without receiving a SIGBUS.

I'd love to see this fix merged in. It sounds like we want to make the immutable/read-only mode decision more flexible before doing so, so that we can use `ro` in Docker and `immutable` outside. If that sounds right, I'm happy to take a crack at adding that as a command-line flag or something that gets set automatically based on the expected execution environment.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1426379903,"don't use immutable=1, only mode=ro",
https://github.com/simonw/datasette/issues/1168#issuecomment-1739816358,https://api.github.com/repos/simonw/datasette/issues/1168,1739816358,IC_kwDOBm6k_c5ns32m,9599,simonw,2023-09-28T18:29:05Z,2023-09-28T18:29:05Z,OWNER,Datasette Cloud really wants this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",777333388,Mechanism for storing metadata in _metadata tables,
https://github.com/simonw/datasette/pull/2155#issuecomment-1737906995,https://api.github.com/repos/simonw/datasette/issues/2155,1737906995,IC_kwDOBm6k_c5nllsz,79087,cadeef,2023-09-27T18:44:02Z,2023-09-27T18:44:02Z,NONE,@simonw Any chance we can get this tiny patch merged for an upcoming release?,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865572575,Fix hupper.start_reloader entry point,
https://github.com/simonw/datasette/pull/2155#issuecomment-1737363182,https://api.github.com/repos/simonw/datasette/issues/2155,1737363182,IC_kwDOBm6k_c5njg7u,418191,jaywgraves,2023-09-27T13:05:41Z,2023-09-27T13:05:41Z,CONTRIBUTOR,I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865572575,Fix hupper.start_reloader entry point,
https://github.com/simonw/sqlite-utils/issues/595#issuecomment-1733312349,https://api.github.com/repos/simonw/sqlite-utils/issues/595,1733312349,IC_kwDOCGYnMM5nUD9d,123451970,cycle-data,2023-09-25T09:38:13Z,2023-09-25T09:38:57Z,NONE,"Never mind

When I created the connection using 
`sqlite_utils.Database(path)`

I just needed to add the following statement right after and it did the trick

`self.db.conn.execute(""PRAGMA foreign_keys = ON"")`

Hope this helps people in the future 👍 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907281675,Cascading DELETE not working with Table.delete(pk),
https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1732018273,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1732018273,IC_kwDOCGYnMM5nPIBh,1108600,radusuciu,2023-09-22T20:49:51Z,2023-09-22T20:49:51Z,NONE,This would be awesome to have for multi-gig tsv and csv files! I'm currently looking at a 10 hour countdown for one such important. Not a problem because I'm lazy and happy to let it run and check on it tomorrow..,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730458954,https://api.github.com/repos/simonw/datasette/issues/2195,1730458954,IC_kwDOBm6k_c5nJLVK,9599,simonw,2023-09-21T22:57:39Z,2023-09-21T22:57:48Z,OWNER,"Worth noting that it already sets `--cors` automatically without you needing to specify it:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L374-L374

I wonder if that's actually surprising behaviour that we should change before 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730457374,https://api.github.com/repos/simonw/datasette/issues/2195,1730457374,IC_kwDOBm6k_c5nJK8e,9599,simonw,2023-09-21T22:56:18Z,2023-09-21T22:56:18Z,OWNER,Maybe I should add `--cors` and `--crossdb` to `datasette publish cloudrun` as well?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730446937,https://api.github.com/repos/simonw/datasette/issues/2195,1730446937,IC_kwDOBm6k_c5nJIZZ,9599,simonw,2023-09-21T22:46:42Z,2023-09-21T22:46:52Z,OWNER,"Found more when I [searched for YAML](https://github.com/search?q=datasette+publish+extra-options++language%3AYAML&type=code).

Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59

`--extra-options=""--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1""`

Uses both `--cors` and `--crossdb`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730441613,https://api.github.com/repos/simonw/datasette/issues/2195,1730441613,IC_kwDOBm6k_c5nJHGN,9599,simonw,2023-09-21T22:42:12Z,2023-09-21T22:42:12Z,OWNER,"https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here:

```
--extra-options=""--setting sql_time_limit_ms 5000""
--extra-options=""--config default_cache_ttl:3600 --config hash_urls:1""
--extra-options ""--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1""
--extra-options=""--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000""
--extra-options=""--setting sql_time_limit_ms 5000""
--extra-options ""--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730438503,https://api.github.com/repos/simonw/datasette/issues/2195,1730438503,IC_kwDOBm6k_c5nJGVn,9599,simonw,2023-09-21T22:38:10Z,2023-09-21T22:38:10Z,OWNER,"I'd really like to remove `--extra-options`. I think the new design makes that completely obsolete?

Maybe it doesn't. You still need `--extra-options` for the `--crossdb` option for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730437934,https://api.github.com/repos/simonw/datasette/issues/2195,1730437934,IC_kwDOBm6k_c5nJGMu,9599,simonw,2023-09-21T22:37:22Z,2023-09-21T22:37:22Z,OWNER,"Here's the full help for Cloud Run at the moment:
```bash
datasette publish cloudrun --help
```
```
Usage: datasette publish cloudrun [OPTIONS] [FILES]...

  Publish databases to Datasette running on Cloud Run

Options:
  -m, --metadata FILENAME         Path to JSON/YAML file containing metadata
                                  to publish
  --extra-options TEXT            Extra options to pass to datasette serve
  --branch TEXT                   Install datasette from a GitHub branch e.g.
                                  main
  --template-dir DIRECTORY        Path to directory containing custom
                                  templates
  --plugins-dir DIRECTORY         Path to directory containing custom plugins
  --static MOUNT:DIRECTORY        Serve static files from this directory at
                                  /MOUNT/...
  --install TEXT                  Additional packages (e.g. plugins) to
                                  install
  --plugin-secret <TEXT TEXT TEXT>...
                                  Secrets to pass to plugins, e.g. --plugin-
                                  secret datasette-auth-github client_id xxx
  --version-note TEXT             Additional note to show on /-/versions
  --secret TEXT                   Secret used for signing secure values, such
                                  as signed cookies
  --title TEXT                    Title for metadata
  --license TEXT                  License label for metadata
  --license_url TEXT              License URL for metadata
  --source TEXT                   Source label for metadata
  --source_url TEXT               Source URL for metadata
  --about TEXT                    About label for metadata
  --about_url TEXT                About URL for metadata
  -n, --name TEXT                 Application name to use when building
  --service TEXT                  Cloud Run service to deploy (or over-write)
  --spatialite                    Enable SpatialLite extension
  --show-files                    Output the generated Dockerfile and
                                  metadata.json
  --memory TEXT                   Memory to allocate in Cloud Run, e.g. 1Gi
  --cpu [1|2|4]                   Number of vCPUs to allocate in Cloud Run
  --timeout INTEGER               Build timeout in seconds
  --apt-get-install TEXT          Additional packages to apt-get install
  --max-instances INTEGER         Maximum Cloud Run instances
  --min-instances INTEGER         Minimum Cloud Run instances
  --help                          Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730437237,https://api.github.com/repos/simonw/datasette/issues/2195,1730437237,IC_kwDOBm6k_c5nJGB1,9599,simonw,2023-09-21T22:36:22Z,2023-09-21T22:36:22Z,OWNER,"I think the actual design of this is pretty simple. Current help starts like this:

```
Usage: datasette publish cloudrun [OPTIONS] [FILES]...

  Publish databases to Datasette running on Cloud Run

Options:
  -m, --metadata FILENAME         Path to JSON/YAML file containing metadata
                                  to publish
  --extra-options TEXT            Extra options to pass to datasette serve
```
The `-s` and `-c` short options are not being used.

So I think `-c/--config` can point to a JSON or YAML `datasette.yaml` file, and `-s/--setting key value` can mirror the new `-s/--setting` option in `datasette serve` itself (a shortcut for populating the config file directly from the CLI).

Here's the relevant help section from `datasette serve`:
```
  -m, --metadata FILENAME         Path to JSON/YAML file containing
                                  license/source metadata
  -c, --config FILENAME           Path to JSON/YAML Datasette configuration
                                  file
  -s, --setting SETTING...        nested.key, value setting to use in
                                  Datasette configuration
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2189#issuecomment-1730388418,https://api.github.com/repos/simonw/datasette/issues/2189,1730388418,IC_kwDOBm6k_c5nI6HC,9599,simonw,2023-09-21T22:26:19Z,2023-09-21T22:26:19Z,OWNER,1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730363182,https://api.github.com/repos/simonw/datasette/issues/2057,1730363182,IC_kwDOBm6k_c5nIz8u,9599,simonw,2023-09-21T22:09:10Z,2023-09-21T22:09:10Z,OWNER,Tests all pass now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2194#issuecomment-1730362441,https://api.github.com/repos/simonw/datasette/issues/2194,1730362441,IC_kwDOBm6k_c5nIzxJ,9599,simonw,2023-09-21T22:08:19Z,2023-09-21T22:08:19Z,OWNER,"That worked

https://github.com/simonw/datasette/commit/e4f868801a6633400045f59584cfe650961c3fa6 is the latest commit right now and https://latest.datasette.io/-/versions shows that as the deployed version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2057#issuecomment-1730356422,https://api.github.com/repos/simonw/datasette/issues/2057,1730356422,IC_kwDOBm6k_c5nIyTG,9599,simonw,2023-09-21T22:01:00Z,2023-09-21T22:01:00Z,OWNER,Tested that locally with Python 3.9 from `pyenv` and it worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730353462,https://api.github.com/repos/simonw/datasette/issues/2057,1730353462,IC_kwDOBm6k_c5nIxk2,9599,simonw,2023-09-21T21:57:17Z,2023-09-21T21:57:17Z,OWNER,"Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302
```
    plugin_info[""name""] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
Test failed: datasette-json-html should not have been loaded
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2193#issuecomment-1730353006,https://api.github.com/repos/simonw/datasette/issues/2193,1730353006,IC_kwDOBm6k_c5nIxdu,9599,simonw,2023-09-21T21:56:43Z,2023-09-21T21:56:43Z,OWNER,"The test fails as expected now. Closing this issue, will solve the remaining problems in:
- #2057","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run",
https://github.com/simonw/datasette/issues/2193#issuecomment-1730352111,https://api.github.com/repos/simonw/datasette/issues/2193,1730352111,IC_kwDOBm6k_c5nIxPv,9599,simonw,2023-09-21T21:55:41Z,2023-09-21T21:55:41Z,OWNER,"https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time.

```
    plugin_info[""name""] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
Test failed: datasette-json-html should not have been loaded
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run",
https://github.com/simonw/datasette/issues/2195#issuecomment-1730313565,https://api.github.com/repos/simonw/datasette/issues/2195,1730313565,IC_kwDOBm6k_c5nIn1d,9599,simonw,2023-09-21T21:16:31Z,2023-09-21T21:16:31Z,OWNER,"The `@add_common_publish_arguments_and_options` decorator described here is bad. If I update it to support a new `config` option all plugins that use it will break.

https://github.com/simonw/datasette/blob/f130c7c0a88e50cea4121ea18d1f6db2431b6fab/docs/plugin_hooks.rst#L347-L355

I want to deprecate it and switch to a different, better design to address the same problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2195#issuecomment-1730312128,https://api.github.com/repos/simonw/datasette/issues/2195,1730312128,IC_kwDOBm6k_c5nInfA,9599,simonw,2023-09-21T21:15:11Z,2023-09-21T21:15:11Z,OWNER,"As soon as `datasette publish cloudrun` has this I can re-enable this bit of the demo deploy:

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97

Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://todomvc.datasette.io/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split,
https://github.com/simonw/datasette/issues/2194#issuecomment-1730305920,https://api.github.com/repos/simonw/datasette/issues/2194,1730305920,IC_kwDOBm6k_c5nIl-A,9599,simonw,2023-09-21T21:09:21Z,2023-09-21T21:09:21Z,OWNER,"I'm going to disable this bit of the deploy for the moment, which will break the demo linked to from https://simonwillison.net/2022/Dec/2/datasette-write-api/

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2194#issuecomment-1730259871,https://api.github.com/repos/simonw/datasette/issues/2194,1730259871,IC_kwDOBm6k_c5nIauf,9599,simonw,2023-09-21T20:34:09Z,2023-09-21T20:34:09Z,OWNER,"... which raises the challenge that `datasette publish` doesn't yet know what to do with a config file!

https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L114-L122","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2194#issuecomment-1730258302,https://api.github.com/repos/simonw/datasette/issues/2194,1730258302,IC_kwDOBm6k_c5nIaV-,9599,simonw,2023-09-21T20:32:53Z,2023-09-21T20:33:02Z,OWNER,"Correct usage is now:
```bash
python tests/fixtures.py fixtures.db fixtures-config.json fixtures-metadata.json \
  plugins --extra-db-filename extra_database.db
```
```
Test tables written to fixtures.db
- metadata written to fixtures-metadata.json
- config written to fixtures-config.json
  Wrote plugin: plugins/register_output_renderer.py
  Wrote plugin: plugins/view_name.py
  Wrote plugin: plugins/my_plugin.py
  Wrote plugin: plugins/messages_output_renderer.py
  Wrote plugin: plugins/sleep_sql_function.py
  Wrote plugin: plugins/my_plugin_2.py
Test tables written to extra_database.db
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2194#issuecomment-1730256435,https://api.github.com/repos/simonw/datasette/issues/2194,1730256435,IC_kwDOBm6k_c5nIZ4z,9599,simonw,2023-09-21T20:31:22Z,2023-09-21T20:31:31Z,OWNER,"New error: ""Error: Metadata should end with .json""

https://github.com/simonw/datasette/actions/runs/6266720924/job/17018265851

<img width=""678"" alt=""CleanShot 2023-09-21 at 13 31 08@2x"" src=""https://github.com/simonw/datasette/assets/9599/720726be-fcfc-49a4-8316-ee49d96ac86b"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2057#issuecomment-1730250337,https://api.github.com/repos/simonw/datasette/issues/2057,1730250337,IC_kwDOBm6k_c5nIYZh,9599,simonw,2023-09-21T20:26:12Z,2023-09-21T20:26:12Z,OWNER,That does seem to fix the problem! ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730247545,https://api.github.com/repos/simonw/datasette/issues/2057,1730247545,IC_kwDOBm6k_c5nIXt5,9599,simonw,2023-09-21T20:23:47Z,2023-09-21T20:23:47Z,OWNER,Hunch: https://pypi.org/project/importlib-metadata/ may help here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2194#issuecomment-1730245204,https://api.github.com/repos/simonw/datasette/issues/2194,1730245204,IC_kwDOBm6k_c5nIXJU,9599,simonw,2023-09-21T20:21:42Z,2023-09-21T20:21:42Z,OWNER,"I think I see the problem - it's from here: https://github.com/simonw/datasette/commit/b2ec8717c3619260a1b535eea20e618bf95aa30b#diff-5dbc88d6e5c3615caf10e32a9d6fc6ff683f5b5814948928cb84c3ab91c038b6L770

The `config` and `metadata` Click options are the wrong way round:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L785-L786

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L801","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2194#issuecomment-1730242734,https://api.github.com/repos/simonw/datasette/issues/2194,1730242734,IC_kwDOBm6k_c5nIWiu,9599,simonw,2023-09-21T20:19:29Z,2023-09-21T20:19:29Z,OWNER,"Maybe `plugins/` does not exist? It should have been created by this line:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L41-L42","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2194#issuecomment-1730241813,https://api.github.com/repos/simonw/datasette/issues/2194,1730241813,IC_kwDOBm6k_c5nIWUV,9599,simonw,2023-09-21T20:18:40Z,2023-09-21T20:18:40Z,OWNER,"This looks to be the step that is failing:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L50-L60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""",
https://github.com/simonw/datasette/issues/2189#issuecomment-1730232308,https://api.github.com/repos/simonw/datasette/issues/2189,1730232308,IC_kwDOBm6k_c5nIT_0,9599,simonw,2023-09-21T20:11:16Z,2023-09-21T20:11:16Z,OWNER,"We're planning a breaking change in `1.0a7`:
- #2191 

Since that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as `1.0a8` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1730231404,https://api.github.com/repos/simonw/datasette/issues/2189,1730231404,IC_kwDOBm6k_c5nITxs,9599,simonw,2023-09-21T20:10:28Z,2023-09-21T20:10:28Z,OWNER,Release 0.64.4: https://docs.datasette.io/en/stable/changelog.html#v0-64-4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730226107,https://api.github.com/repos/simonw/datasette/issues/2057,1730226107,IC_kwDOBm6k_c5nISe7,9599,simonw,2023-09-21T20:06:19Z,2023-09-21T20:06:19Z,OWNER,"No that's not it actually, it's something else.

Got to this point:
```bash
DATASETTE_LOAD_PLUGINS=datasette-init python -i $(which datasette) plugins
```
That fails and drops me into a debugger:

```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 186, in plugins
    app = Datasette([], plugins_dir=plugins_dir)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 405, in __init__
    for plugin in get_plugins()
  File ""/Users/simon/Dropbox/Development/datasette/datasette/plugins.py"", line 89, in get_plugins
    plugin_info[""name""] = distinfo.name or distinfo.project_name
AttributeError: 'PathDistribution' object has no attribute 'name'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730219703,https://api.github.com/repos/simonw/datasette/issues/2057,1730219703,IC_kwDOBm6k_c5nIQ63,9599,simonw,2023-09-21T20:01:54Z,2023-09-21T20:01:54Z,OWNER,"The problem is here:
```
 86  	        distinfo = plugin_to_distinfo.get(plugin)
 87  	        if distinfo is None:
 88  	            breakpoint()
 89  ->	            assert False
 90  	        if distinfo.name is None:
 91  	            breakpoint()
 92  	            assert False
 93  	        if distinfo:
 94  	            plugin_info[""version""] = distinfo.version
(Pdb) distinfo
(Pdb) plugin
<module 'datasette.sql_functions' from '/Users/simon/Dropbox/Development/datasette/datasette/sql_functions.py'>
```
That `plugin_to_distinfo` is missing some stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730214654,https://api.github.com/repos/simonw/datasette/issues/2057,1730214654,IC_kwDOBm6k_c5nIPr-,9599,simonw,2023-09-21T19:59:51Z,2023-09-21T19:59:51Z,OWNER,"So the problem is the `get_plugins()` function returning plugins with `None` for their name:

https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/plugins.py#L61-L91","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730212597,https://api.github.com/repos/simonw/datasette/issues/2057,1730212597,IC_kwDOBm6k_c5nIPL1,9599,simonw,2023-09-21T19:58:38Z,2023-09-21T19:58:38Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/app.py#L1127-L1146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730211445,https://api.github.com/repos/simonw/datasette/issues/2057,1730211445,IC_kwDOBm6k_c5nIO51,9599,simonw,2023-09-21T19:57:44Z,2023-09-21T19:57:44Z,OWNER,"In the debugger:
```
>>> import pdb
>>> pdb.pm()
> /Users/simon/Dropbox/Development/datasette/datasette/app.py(1136)_plugins()
-> ps.sort(key=lambda p: p[""name""])
(Pdb) ps
[{'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['prepare_connection', 'render_cell'], 'version': '1.0.1'}, {'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['startup'], 'version': '0.2'}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730210728,https://api.github.com/repos/simonw/datasette/issues/2057,1730210728,IC_kwDOBm6k_c5nIOuo,9599,simonw,2023-09-21T19:57:08Z,2023-09-21T19:57:08Z,OWNER,"In my Python 3.8 environment I ran:
```bash
datasette install datasette-init datasette-json-html
```
And now `datasette plugins` produces this error:
```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 192, in plugins
    click.echo(json.dumps(app._plugins(all=all), indent=4))
  File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 1136, in _plugins
    ps.sort(key=lambda p: p[""name""])
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2193#issuecomment-1730208566,https://api.github.com/repos/simonw/datasette/issues/2193,1730208566,IC_kwDOBm6k_c5nIOM2,9599,simonw,2023-09-21T19:55:19Z,2023-09-21T19:55:19Z,OWNER,"Yes, the new script seems to work. On Python 3.11:

```
tests/test-datasette-load-plugins.sh 
echo $?
0
```
On Python 3.8:
```
tests/test-datasette-load-plugins.sh
Test failed: datasette-json-html not found
echo $?
1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run",
https://github.com/simonw/datasette/issues/2193#issuecomment-1730206629,https://api.github.com/repos/simonw/datasette/issues/2193,1730206629,IC_kwDOBm6k_c5nINul,9599,simonw,2023-09-21T19:53:39Z,2023-09-21T19:53:39Z,OWNER,"[GPT-4 says](https://chat.openai.com/share/65a95561-8f3c-4f9b-a4ab-c2508e87d2a9):

> In the script, you're using a subshell `( ... )` to group commands. If you `exit 1` within the subshell, it will only exit the subshell and not the main script. This is why GitHub Actions does not see it as a failure.

It suggested doing this instead:

```bash
#!/bin/bash

PLUGINS=$(datasette plugins)
if ! echo ""$PLUGINS"" | jq 'any(.[]; .name == ""datasette-json-html"")' | grep -q true; then
  echo ""Test failed: datasette-json-html not found""
  exit 1
fi

PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins)
if ! echo ""$PLUGINS2"" | jq 'any(.[]; .name == ""datasette-json-html"")' | grep -q false; then
  echo ""Test failed: datasette-json-html should not have been loaded""
  exit 1
fi

if ! echo ""$PLUGINS2"" | jq 'any(.[]; .name == ""datasette-init"")' | grep -q true; then
  echo ""Test failed: datasette-init should have been loaded""
  exit 1
fi

PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins)
if ! echo ""$PLUGINS3"" | grep -q '\[\]'; then
  echo ""Test failed: datasette plugins should have returned []""
  exit 1
fi
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run",
https://github.com/simonw/datasette/issues/2193#issuecomment-1730203356,https://api.github.com/repos/simonw/datasette/issues/2193,1730203356,IC_kwDOBm6k_c5nIM7c,9599,simonw,2023-09-21T19:51:04Z,2023-09-21T19:51:04Z,OWNER,"The script:

https://github.com/simonw/datasette/blob/b0d0a0e5de8bb5b9b6c253e8af451a532266bcf1/tests/test-datasette-load-plugins.sh#L1-L29

I'm not sure why those `exit 1` lines did not cause a CI failure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run",
https://github.com/simonw/datasette/issues/2193#issuecomment-1730202533,https://api.github.com/repos/simonw/datasette/issues/2193,1730202533,IC_kwDOBm6k_c5nIMul,9599,simonw,2023-09-21T19:50:22Z,2023-09-21T19:50:22Z,OWNER,"Here's the failure in CI, which did not cause the workflow to fail even though it should have:

<img width=""733"" alt=""CleanShot 2023-09-21 at 12 50 01@2x"" src=""https://github.com/simonw/datasette/assets/9599/4c1d61be-b6f6-4699-a885-ed2d4edd85ab"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run",
https://github.com/simonw/datasette/issues/2057#issuecomment-1730201226,https://api.github.com/repos/simonw/datasette/issues/2057,1730201226,IC_kwDOBm6k_c5nIMaK,9599,simonw,2023-09-21T19:49:20Z,2023-09-21T19:49:20Z,OWNER,"That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the ""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730188367,https://api.github.com/repos/simonw/datasette/issues/2057,1730188367,IC_kwDOBm6k_c5nIJRP,9599,simonw,2023-09-21T19:38:28Z,2023-09-21T19:40:38Z,OWNER,"I'll imitate `certbot`:

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot-ci/setup.py#L9

```python
    'importlib_resources>=1.3.1; python_version < ""3.9""',
```
Looks like `1.3` is the minimum version needed for compatibility with the 3.9 standard library, according to https://github.com/python/importlib_resources/blob/main/README.rst#compatibility

https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot/certbot/_internal/constants.py#L13C29-L16

```python
if sys.version_info >= (3, 9):  # pragma: no cover
    import importlib.resources as importlib_resources
else:  # pragma: no cover
    import importlib_resources
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730185322,https://api.github.com/repos/simonw/datasette/issues/2057,1730185322,IC_kwDOBm6k_c5nIIhq,9599,simonw,2023-09-21T19:35:49Z,2023-09-21T19:35:49Z,OWNER,I think I can fix this using https://importlib-resources.readthedocs.io/en/latest/using.html - maybe as a dependency only installed if the Python version is less than 3.9.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730183405,https://api.github.com/repos/simonw/datasette/issues/2057,1730183405,IC_kwDOBm6k_c5nIIDt,9599,simonw,2023-09-21T19:34:09Z,2023-09-21T19:34:09Z,OWNER,"Confirmed: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files

> `importlib.resources.files(package)`
> [...]
> New in version 3.9.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1730171241,https://api.github.com/repos/simonw/datasette/issues/2057,1730171241,IC_kwDOBm6k_c5nIFFp,9599,simonw,2023-09-21T19:27:25Z,2023-09-21T19:27:25Z,OWNER,"This broke in Python 3.8:

```
if plugin.__name__ not in DEFAULT_PLUGINS:
    try:
        if (importlib.resources.files(plugin.__name__) / ""static"").is_dir():
E                   AttributeError: module 'importlib.resources' has no attribute 'files'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2189#issuecomment-1730162283,https://api.github.com/repos/simonw/datasette/issues/2189,1730162283,IC_kwDOBm6k_c5nIC5r,9599,simonw,2023-09-21T19:19:47Z,2023-09-21T19:19:47Z,OWNER,"I'm going to release this in `1.0a7`, and I'll backport it to a `0.64.4` release too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/pull/2190#issuecomment-1729961503,https://api.github.com/repos/simonw/datasette/issues/2190,1729961503,IC_kwDOBm6k_c5nHR4f,15178711,asg017,2023-09-21T16:56:57Z,2023-09-21T16:56:57Z,CONTRIBUTOR,TODO: add similar checks for permissions/allow/canned queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json",
https://github.com/simonw/datasette/pull/2191#issuecomment-1728504633,https://api.github.com/repos/simonw/datasette/issues/2191,1728504633,IC_kwDOBm6k_c5nBuM5,9599,simonw,2023-09-20T22:24:51Z,2023-09-20T22:25:16Z,OWNER,"The `{""units"": {""distance"": ""m"", ""frequency"": ""Hz""}}` bit is for the [units feature](https://docs.datasette.io/en/1.0a6/metadata.html#specifying-units-for-a-column) which I've half-disabled already and would like to remove before 1.0, though ideally turning that functionality into a plugin instead (if I can figure out how to do that).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`,  `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`",
https://github.com/simonw/datasette/pull/2191#issuecomment-1728503623,https://api.github.com/repos/simonw/datasette/issues/2191,1728503623,IC_kwDOBm6k_c5nBt9H,9599,simonw,2023-09-20T22:23:33Z,2023-09-20T22:24:10Z,OWNER,"This is one of the most interesting illustrative examples in the new code:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L301-L349

Interesting to note that it now has canned queries in it, which include this bit:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L341-L342

It looks like metadata, but in this case it's configuration. That blur between metadata and configuration at the canned query level still feels a little bit odd to me, but I still think we're going in the right direction with it.

Also interesting, from that same file:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L351-L399

There are a few things in that metadata block that are arguably configuration, not metadata - for example:

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L360

I think `extra_css_urls` is definitely configuration, not metadata.

https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L369-L395

Most of that stuff is arguably configuration too, with the exception of the `roadside_attractions.columns` bit which is metadata about those columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`,  `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`",
https://github.com/simonw/datasette/pull/2191#issuecomment-1728498221,https://api.github.com/repos/simonw/datasette/issues/2191,1728498221,IC_kwDOBm6k_c5nBsot,9599,simonw,2023-09-20T22:17:26Z,2023-09-20T22:17:26Z,OWNER,"I tested this locally for permissions like this. `datasette.yml`:

```yaml
databases:
  content:
    allow:
      id: root
```
Started Datasette like this:
```bash
datasette --root content.db pottery2.db -c datasette.yml
```
As root I could see this (note the padlock):

<img width=""630"" alt=""CleanShot 2023-09-20 at 15 17 02@2x"" src=""https://github.com/simonw/datasette/assets/9599/3e0c3abd-9a90-43f4-ad68-bb490d19eef8"">

http://127.0.0.1:8001/-/metadata returned `{}` showing that the permissions must have come from the config file instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`,  `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`",
https://github.com/simonw/datasette/issues/2189#issuecomment-1728192688,https://api.github.com/repos/simonw/datasette/issues/2189,1728192688,IC_kwDOBm6k_c5nAiCw,173848,yozlet,2023-09-20T17:53:31Z,2023-09-20T17:53:31Z,NONE,"`/me munches popcorn at a furious rate, utterly entralled`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/pull/2192#issuecomment-1726754119,https://api.github.com/repos/simonw/datasette/issues/2192,1726754119,IC_kwDOBm6k_c5m7C1H,22429695,codecov[bot],2023-09-20T01:35:45Z,2023-09-20T01:35:45Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`+0.02%`** :tada:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`4e6a341`)](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.72%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2192      +/-   ##
==========================================
+ Coverage   92.69%   92.72%   +0.02%     
==========================================
  Files          40       40              
  Lines        6039     6036       -3     
==========================================
- Hits         5598     5597       -1     
+ Misses        441      439       -2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/table.py](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.31% <100.00%> (+0.51%)` | :arrow_up: |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2192/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1903932086,Stop using parallel SQL queries for tables,
https://github.com/simonw/datasette/issues/2189#issuecomment-1726749355,https://api.github.com/repos/simonw/datasette/issues/2189,1726749355,IC_kwDOBm6k_c5m7Bqr,9599,simonw,2023-09-20T01:28:16Z,2023-09-20T01:28:16Z,OWNER,Added a note to that example in the documentation: https://github.com/simonw/datasette/blob/4e6a34179eaedec44c1263275d7592fd83d7e2ac/docs/internals.rst?plain=1#L1320,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/pull/2185#issuecomment-1719468727,https://api.github.com/repos/simonw/datasette/issues/2185,1719468727,IC_kwDOBm6k_c5mfQK3,22429695,codecov[bot],2023-09-14T13:36:07Z,2023-09-19T13:42:35Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage has no change and project coverage change: **`-0.04%`** :warning:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fe5f881`)](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2185      +/-   ##
==========================================
- Coverage   92.69%   92.66%   -0.04%     
==========================================
  Files          40       40              
  Lines        6039     6039              
==========================================
- Hits         5598     5596       -2     
- Misses        441      443       +2     
```


[see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2185/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1896578249,Bump the python-packages group with 3 updates,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724325068,https://api.github.com/repos/simonw/datasette/issues/2189,1724325068,IC_kwDOBm6k_c5mxxzM,9599,simonw,2023-09-18T20:29:41Z,2023-09-18T20:29:41Z,OWNER,"The one other thing affected by this change is this documentation, which suggests a not-actually-safe pattern: https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/docs/internals.rst#L1292-L1321","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724317367,https://api.github.com/repos/simonw/datasette/issues/2189,1724317367,IC_kwDOBm6k_c5mxv63,9599,simonw,2023-09-18T20:25:44Z,2023-09-18T20:25:44Z,OWNER,"My current hunch is that SQLite gets unhappy if multiple threads access the same underlying C object - which sometimes happens with in-memory connections and Datasette presumably because they are faster than file-backed databases.

I'm going to remove the `asyncio.gather()` code from the table view. I'll ship a 0.x release with that fix too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724315591,https://api.github.com/repos/simonw/datasette/issues/2189,1724315591,IC_kwDOBm6k_c5mxvfH,9599,simonw,2023-09-18T20:24:30Z,2023-09-18T20:24:30Z,OWNER,"[Using SQLite In Multi-Threaded Applications](https://www.sqlite.org/threadsafe.html)

That indicates that there's a SQLite option for ""Serialized"" mode where it's safe to access anything SQLite provides from multiple threads, but as far as I can tell Python doesn't give you an option to turn that mode on or off for a connection - you can read `sqlite3.threadsafet`y to see if that mode was compiled in or not, but not actually change it.

On my Mac `sqlite3.threadsafety` returns 1 which means https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety ""Multi-thread: In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads."" - it would need to return 3 for that serialized mode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724305169,https://api.github.com/repos/simonw/datasette/issues/2189,1724305169,IC_kwDOBm6k_c5mxs8R,9599,simonw,2023-09-18T20:16:22Z,2023-09-18T20:16:36Z,OWNER,"Looking again at this code:

https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/datasette/database.py#L87-L117

`check_same_thread=False` really stands out here.

Python docs at https://docs.python.org/3/library/sqlite3.html

> **check_same_thread** ([*bool*](https://docs.python.org/3/library/functions.html#bool ""bool"")) -- If `True` (default), [`ProgrammingError`](https://docs.python.org/3/library/sqlite3.html#sqlite3.ProgrammingError ""sqlite3.ProgrammingError"") will be raised if the database connection is used by a thread other than the one that created it. If `False`, the connection may be accessed in multiple threads; write operations may need to be serialized by the user to avoid data corruption. See [`threadsafety`](https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety ""sqlite3.threadsafety"") for more information.

I think I'm playing with fire by allowing multiple threads to access the same connection without doing my own serialization of those requests.

I _do_ do that using the write connection - and in this particular case the bug isn't coming from write queries, it's coming from read queries - but perhaps SQLite has issues with threading for reads, too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724298817,https://api.github.com/repos/simonw/datasette/issues/2189,1724298817,IC_kwDOBm6k_c5mxrZB,9599,simonw,2023-09-18T20:11:26Z,2023-09-18T20:11:26Z,OWNER,"Now that I've confirmed that parallel query execution of the kind introduced in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 can cause hangs (presumably some kind of locking issue) against in-memory databases, some options:

1. Disable parallel execution entirely and rip out related code.
2. Disable parallel execution entirely by leaving that code but having it always behave as if `_noparallel=1`
3. Continue digging and try and find some way to avoid this problem

The parallel execution work is something I was playing with last year in the hope of speeding up Datasette pages like the table page which need to execute a bunch of queries - one for each facet, plus one for each column to see if it should be suggested as a facet.

I wrote about this at the time here: https://simonwillison.net/2022/May/6/weeknotes/

My hope was that despite Python's GIL this optimization would still help, because the SQLite C module releases the GIL once it gets to SQLite.

But... that didn't hold up. It looked like enough work was happening in Python land with the GIL that the optimization didn't improve things.

Running the `nogil` fork of Python DID improve things though! I left the code in partly on the hope that the `nogil` fork would be accepted into Python core.

... which it now has! But it will still be a year or two before it fully lands: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474

So I'm not particularly concerned about dropping the parallel execution. If I do drop it though do I leave the potentially complex code in that relates to it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724281824,https://api.github.com/repos/simonw/datasette/issues/2189,1724281824,IC_kwDOBm6k_c5mxnPg,9599,simonw,2023-09-18T19:58:06Z,2023-09-18T19:58:06Z,OWNER,I also confirmed that `http://127.0.0.1:8064/airtable_refs/airtable_refs?_noparallel=1` does not trigger the bug but `http://127.0.0.1:8064/airtable_refs/airtable_refs` does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724278386,https://api.github.com/repos/simonw/datasette/issues/2189,1724278386,IC_kwDOBm6k_c5mxmZy,9599,simonw,2023-09-18T19:55:32Z,2023-09-18T19:55:32Z,OWNER,"OK it looks like it found it!

```
942411ef946e9a34a2094944d3423cddad27efd3 is the first bad commit
commit 

Author: Simon Willison <swillison@gmail.com>
Date:   Tue Apr 26 15:48:56 2022 -0700

    Execute some TableView queries in parallel
    
    Use ?_noparallel=1 to opt out (undocumented, useful for benchmark comparisons)
    
    Refs #1723, #1715

 datasette/views/table.py | 93 ++++++++++++++++++++++++++++++++++--------------
 1 file changed, 67 insertions(+), 26 deletions(-)
bisect found first bad commit
```
https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 does look like the cause of this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724276917,https://api.github.com/repos/simonw/datasette/issues/2189,1724276917,IC_kwDOBm6k_c5mxmC1,9599,simonw,2023-09-18T19:54:23Z,2023-09-18T19:54:23Z,OWNER,"Turned out I wasn't running the `datasette` from the current directory, so it was not testing what I intended.

FIxed that with `pip install -e .` in the `datasette/` directory.

Now I'm seeing some passes, which look like this:
```
running  '../airtable-export/testit.sh'
INFO:     Started server process [77810]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
INFO:     127.0.0.1:59439 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK
INFO:     127.0.0.1:59440 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK
INFO:     127.0.0.1:59441 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK
  All curl succeeded
Killing datasette server with PID 77810
../airtable-export/testit.sh: line 54: 77810 Killed: 9               datasette pottery2.db -p $port
All three curls succeeded.
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[7463b051cf8d7f856df5eba9f7aa944183ebabe5] Cosmetic tweaks after blacken-docs, refs #1718
running  '../airtable-export/testit.sh'
INFO:     Started server process [77826]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724257290,https://api.github.com/repos/simonw/datasette/issues/2189,1724257290,IC_kwDOBm6k_c5mxhQK,9599,simonw,2023-09-18T19:39:27Z,2023-09-18T19:44:26Z,OWNER,"I'm now trying this test script:
```bash
#!/bin/bash

port=8064
# Start datasette server in the background and get its PID
datasette pottery2.db -p $port &
server_pid=$!

# Wait for a moment to ensure the server has time to start up
sleep 2

# Initialize counters and parameters
retry_count=0
max_retries=3
success_count=0
path=""/airtable_refs/airtable_refs""

# Function to run curl with a timeout
function test_curl {
    # Run the curl command with a timeout of 3 seconds
    timeout 3s curl -s ""http://localhost:${port}${path}"" > /dev/null
    if [ $? -eq 0 ]; then
        # Curl was successful
        ((success_count++))
    fi
}

# Try three parallel curl requests
while [[ $retry_count -lt $max_retries ]]; do
    # Reset the success counter
    success_count=0

    # Run the curls in parallel
    echo ""  Running curls""
    test_curl
    test_curl
    test_curl #  & test_curl & test_curl &

    # Wait for all curls to finish
    #wait

    # Check the success count
    if [[ $success_count -eq 3 ]]; then
        # All curls succeeded, break out of the loop
        echo ""  All curl succeeded""
        break
    fi

    ((retry_count++))
done

# Kill the datasette server
echo ""Killing datasette server with PID $server_pid""
kill -9 $server_pid
sleep 2

# Print result
if [[ $success_count -eq 3 ]]; then
    echo ""All three curls succeeded.""
    exit 0
else
    echo ""Error: Not all curls succeeded after $retry_count attempts.""
    exit 1
fi
```
I run it like this:
```bash
git bisect reset
git bisect start
git bisect good 0.59.4
git bisect bad 1.0a6
git bisect run ../airtable-export/testit.sh
```
But... it's not having the desired result, I think because the bug is intermittent so each time I run it the bisect spits out a different commit as the one that is to blame.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724263390,https://api.github.com/repos/simonw/datasette/issues/2189,1724263390,IC_kwDOBm6k_c5mxive,9599,simonw,2023-09-18T19:44:03Z,2023-09-18T19:44:03Z,OWNER,I knocked it down to 1 retry just to see what happened.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724259229,https://api.github.com/repos/simonw/datasette/issues/2189,1724259229,IC_kwDOBm6k_c5mxhud,9599,simonw,2023-09-18T19:40:56Z,2023-09-18T19:40:56Z,OWNER,I tried it with a path of `/` and everything passed - so it's definitely the path of `/airtable_refs/airtable_refs` (an in-memory database created by an experimental branch of https://github.com/simonw/airtable-export) that triggers the problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724258279,https://api.github.com/repos/simonw/datasette/issues/2189,1724258279,IC_kwDOBm6k_c5mxhfn,9599,simonw,2023-09-18T19:40:13Z,2023-09-18T19:40:13Z,OWNER,"Output while it is running looks like this:
```
running  '../airtable-export/testit.sh'
INFO:     Started server process [75649]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
  Running curls
  Running curls
Killing datasette server with PID 75649
../airtable-export/testit.sh: line 54: 75649 Killed: 9               datasette pottery2.db -p $port
Error: Not all curls succeeded after 3 attempts.
Bisecting: 155 revisions left to test after this (roughly 7 steps)
[247e460e08bf823142f7b84058fe44e43626787f] Update beautifulsoup4 requirement (#1703)
running  '../airtable-export/testit.sh'
INFO:     Started server process [75722]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
  Running curls
  Running curls
Killing datasette server with PID 75722
../airtable-export/testit.sh: line 54: 75722 Killed: 9               datasette pottery2.db -p $port
Error: Not all curls succeeded after 3 attempts.
Bisecting: 77 revisions left to test after this (roughly 6 steps)
[3ef47a0896c7e63404a34e465b7160c80eaa571d] Link rel=alternate header for tables and rows
running  '../airtable-export/testit.sh'
INFO:     Started server process [75818]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit)
  Running curls
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/pull/2190#issuecomment-1724169693,https://api.github.com/repos/simonw/datasette/issues/2190,1724169693,IC_kwDOBm6k_c5mxL3d,22429695,codecov[bot],2023-09-18T18:39:19Z,2023-09-18T18:39:19Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning:
> Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fc7dbe0`)](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.67%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2190      +/-   ##
==========================================
- Coverage   92.69%   92.67%   -0.03%     
==========================================
  Files          40       40              
  Lines        6039     6044       +5     
==========================================
+ Hits         5598     5601       +3     
- Misses        441      443       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <100.00%> (ø)` | |
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.83% <100.00%> (+0.03%)` | :arrow_up: |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2190/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json",
https://github.com/simonw/datasette/issues/2189#issuecomment-1724159882,https://api.github.com/repos/simonw/datasette/issues/2189,1724159882,IC_kwDOBm6k_c5mxJeK,9599,simonw,2023-09-18T18:32:29Z,2023-09-18T18:32:29Z,OWNER,"This worked, including on macOS even though GPT-4 thought `timeout` would not work there: https://chat.openai.com/share/cc4628e9-5240-4f35-b640-16a9c178b315
```bash
#!/bin/bash

# Run the command with a timeout of 5 seconds
timeout 5s datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs

# Check the exit code from timeout
if [ $? -eq 124 ]; then
    echo ""Error: Command timed out after 5 seconds.""
    exit 1
fi
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724157182,https://api.github.com/repos/simonw/datasette/issues/2189,1724157182,IC_kwDOBm6k_c5mxIz-,9599,simonw,2023-09-18T18:30:30Z,2023-09-18T18:30:30Z,OWNER,"OK, I can trigger the bug like this:

```bash
datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs
```
Can I write a bash script that fails (and terminates the process) if it takes longer than X seconds?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724089666,https://api.github.com/repos/simonw/datasette/issues/2189,1724089666,IC_kwDOBm6k_c5mw4VC,9599,simonw,2023-09-18T17:49:24Z,2023-09-18T17:49:24Z,OWNER,I switched that particular implementation to using an on-disk database instead of an in-memory database and could no longer recreate the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724084199,https://api.github.com/repos/simonw/datasette/issues/2189,1724084199,IC_kwDOBm6k_c5mw2_n,9599,simonw,2023-09-18T17:47:01Z,2023-09-18T17:47:01Z,OWNER,I managed to trigger it by loading `http://127.0.0.1:8045/airtable_refs/airtable_refs` - which worked - and then hitting refresh on that page a bunch of times until it hung.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724083324,https://api.github.com/repos/simonw/datasette/issues/2189,1724083324,IC_kwDOBm6k_c5mw2x8,9599,simonw,2023-09-18T17:46:21Z,2023-09-18T17:46:21Z,OWNER,"Sometimes it takes a few clicks for the bug to occur, but it does seem to always be within the in-memory database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724081909,https://api.github.com/repos/simonw/datasette/issues/2189,1724081909,IC_kwDOBm6k_c5mw2b1,9599,simonw,2023-09-18T17:45:27Z,2023-09-18T17:45:27Z,OWNER,Maybe it's not related to faceting - I just got it on a hit to `http://127.0.0.1:8045/airtable_refs/airtable_refs` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724072390,https://api.github.com/repos/simonw/datasette/issues/2189,1724072390,IC_kwDOBm6k_c5mw0HG,9599,simonw,2023-09-18T17:39:06Z,2023-09-18T17:39:06Z,OWNER,Landing a version of that test anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724064440,https://api.github.com/repos/simonw/datasette/issues/2189,1724064440,IC_kwDOBm6k_c5mwyK4,9599,simonw,2023-09-18T17:36:00Z,2023-09-18T17:36:00Z,OWNER,"I wrote this test, but it passes:
```python
@pytest.mark.asyncio
async def test_facet_against_in_memory_database():
    ds = Datasette()
    db = ds.add_memory_database(""mem"")
    await db.execute_write(""create table t (id integer primary key, name text)"")
    await db.execute_write_many(
        ""insert into t (name) values (?)"", [[""one""], [""one""], [""two""]]
    )
    response1 = await ds.client.get(""/mem/t.json"")
    assert response1.status_code == 200
    response2 = await ds.client.get(""/mem/t.json?_facet=name"")
    assert response2.status_code == 200
    assert response2.json() == {
        ""ok"": True,
        ""next"": None,
        ""facet_results"": {
            ""results"": {
                ""name"": {
                    ""name"": ""name"",
                    ""type"": ""column"",
                    ""hideable"": True,
                    ""toggle_url"": ""/mem/t.json"",
                    ""results"": [
                        {
                            ""value"": ""one"",
                            ""label"": ""one"",
                            ""count"": 2,
                            ""toggle_url"": ""http://localhost/mem/t.json?_facet=name&name=one"",
                            ""selected"": False,
                        },
                        {
                            ""value"": ""two"",
                            ""label"": ""two"",
                            ""count"": 1,
                            ""toggle_url"": ""http://localhost/mem/t.json?_facet=name&name=two"",
                            ""selected"": False,
                        },
                    ],
                    ""truncated"": False,
                }
            },
            ""timed_out"": [],
        },
        ""rows"": [
            {""id"": 1, ""name"": ""one""},
            {""id"": 2, ""name"": ""one""},
            {""id"": 3, ""name"": ""two""},
        ],
        ""truncated"": False,
    }
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724055823,https://api.github.com/repos/simonw/datasette/issues/2189,1724055823,IC_kwDOBm6k_c5mwwEP,9599,simonw,2023-09-18T17:31:10Z,2023-09-18T17:31:10Z,OWNER,That line was added in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 which first shipped in 0.62a0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724051886,https://api.github.com/repos/simonw/datasette/issues/2189,1724051886,IC_kwDOBm6k_c5mwvGu,9599,simonw,2023-09-18T17:28:20Z,2023-09-18T17:30:30Z,OWNER,"The bug exhibits when I try to add a facet. I think it's caused by the parallel query execution I added to facets at some point.

http://127.0.0.1:8045/airtable_refs/airtable_refs - no error
http://127.0.0.1:8045/airtable_refs/airtable_refs?_facet=table_name#facet-table_name - hangs the server

Crucial line in the traceback:
```
    await gather(execute_facets(), execute_suggested_facets())
```
From here: https://github.com/simonw/datasette/blob/917272c864ad7b8a00c48c77f5c2944093babb4e/datasette/views/table.py#L568","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724049538,https://api.github.com/repos/simonw/datasette/issues/2189,1724049538,IC_kwDOBm6k_c5mwuiC,9599,simonw,2023-09-18T17:26:44Z,2023-09-18T17:26:44Z,OWNER,"Just managed to get this exception trace:
```
    return await self.route_path(scope, receive, send, path)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/app.py"", line 1354, in route_path
    response = await view(request, send)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 134, in view
    return await self.dispatch_request(request)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 91, in dispatch_request
    return await handler(request)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 361, in get
    response_or_template_contexts = await self.data(request, **data_kwargs)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 158, in data
    return await self._data_traced(request, default_labels, _next, _size)
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 568, in _data_traced
    await gather(execute_facets(), execute_suggested_facets())
  File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 177, in _gather_parallel
    return await asyncio.gather(*args)
asyncio.exceptions.CancelledError
INFO:     127.0.0.1:64109 - ""GET /airtable_refs/airtable_refs?_facet=table_name&table_name=Sessions HTTP/1.1"" 500 Internal Server Error
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/concurrent/futures/thread.py"", line 40, in _python_exit
    t.join()
  File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py"", line 1011, in join
    self._wait_for_tstate_lock()
  File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py"", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724048314,https://api.github.com/repos/simonw/datasette/issues/2189,1724048314,IC_kwDOBm6k_c5mwuO6,9599,simonw,2023-09-18T17:25:55Z,2023-09-18T17:25:55Z,OWNER,"The good news is that this bug is currently unlikely to affect most users since named in-memory databases (created using `datasette.add_memory_database(""airtable_refs"")` ([docs](https://docs.datasette.io/en/stable/internals.html#add-memory-database-name)) are a pretty obscure feature, only available to plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2189#issuecomment-1724045748,https://api.github.com/repos/simonw/datasette/issues/2189,1724045748,IC_kwDOBm6k_c5mwtm0,9599,simonw,2023-09-18T17:24:07Z,2023-09-18T17:24:07Z,OWNER,"I need reliable steps to reproduce, then I can bisect and figure out which exact version of Datasette introduced the problem.

I have a hunch that it relates to changes made to the `datasette/database.py` module, maybe one of these changes here: https://github.com/simonw/datasette/compare/0.61...0.63.1#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases,
https://github.com/simonw/datasette/issues/2123#issuecomment-1723362847,https://api.github.com/repos/simonw/datasette/issues/2123,1723362847,IC_kwDOBm6k_c5muG4f,6523121,hueyy,2023-09-18T13:02:46Z,2023-09-18T13:02:46Z,NONE,"Can confirm that this bug can be reproduced as follows:

```
docker run datasetteproject/datasette datasette serve --reload 
```

which produces the following output:

> Starting monitor for PID 10.
> Error: Invalid value for '[FILES]...': Path 'serve' does not exist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1825007061,datasette serve when invoked with --reload interprets the serve command as a file,
https://github.com/simonw/datasette/pull/2052#issuecomment-1722943484,https://api.github.com/repos/simonw/datasette/issues/2052,1722943484,IC_kwDOBm6k_c5msgf8,30934,20after4,2023-09-18T08:14:47Z,2023-09-18T08:14:47Z,NONE,This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/issues/2188#issuecomment-1722848454,https://api.github.com/repos/simonw/datasette/issues/2188,1722848454,IC_kwDOBm6k_c5msJTG,15178711,asg017,2023-09-18T06:58:53Z,2023-09-18T06:58:53Z,CONTRIBUTOR,"Thinking about this more, here a list of things I imagine a ""compile-to-sql"" plugin would want to do:

1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting)
2. Add ""Query using PRQL"" buttons in various parts of Datasette's UI, like `/dbname` page
3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages
4. Have their own dedicated code editor page


1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of ""slots"" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks. 

So it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here. 

If I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1900026059,"Plugin Hooks for ""compile to SQL"" languages",
https://github.com/simonw/datasette/issues/1191#issuecomment-1722845490,https://api.github.com/repos/simonw/datasette/issues/1191,1722845490,IC_kwDOBm6k_c5msIky,15178711,asg017,2023-09-18T06:55:52Z,2023-09-18T06:55:52Z,CONTRIBUTOR,"One note here: this feature could be called ""slots"", similar to [Layout Slots](https://vitepress.dev/guide/extending-default-theme#layout-slots) in Vitepress.

In Vitepress, you can add custom components/widget/gadgets into determined named ""slots"", like so:

```
doc-top
doc-bottom
doc-footer-before
doc-before
doc-after
...
```

Would be great to do in both Python and Javascript, with the upcoming JavaScript API #2052. In `datasette-write-ui`, all we do is add a few ""Insert row"" and ""edit this row"" buttons and that required completely capturing the `table.html` template, which isn't great for other plugins. But having ""slots"" like `table-footer-before` or `table-row-id` or something would be great to work with.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098345,Ability for plugins to collaborate when adding extra HTML to blocks in default templates,
https://github.com/simonw/datasette/issues/2188#issuecomment-1722662413,https://api.github.com/repos/simonw/datasette/issues/2188,1722662413,IC_kwDOBm6k_c5mrb4N,9599,simonw,2023-09-18T02:01:34Z,2023-09-18T02:01:34Z,OWNER,"I'm not interested in these in Datasette core itself, but I think they have a ton of potential for plugins.

I wonder what the best way to handle that would be?

Right now it's possible to write a plugin that [adds extra routes](https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes-datasette), so someone could build a `/dbname/-/prql?query=xxx` endpoint.

If this could return JSON, they could add JavaScript to the `/dbname` page that provided a UI for kicking off one of those queries.

Something that could make that more ergonomic might be the plugin hook that allows plugins to add extra HTML to different core database pages - e.g. adding a ""Query this database using PRQL"" button or link or even a full form at the top of that database page. That's this issue here:

- #1191","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1900026059,"Plugin Hooks for ""compile to SQL"" languages",
https://github.com/simonw/datasette/issues/2057#issuecomment-1722323967,https://api.github.com/repos/simonw/datasette/issues/2057,1722323967,IC_kwDOBm6k_c5mqJP_,9599,simonw,2023-09-16T21:54:33Z,2023-09-16T21:54:33Z,OWNER,Just found this migration guide: https://importlib-metadata.readthedocs.io/en/latest/migration.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1722266942,https://api.github.com/repos/simonw/datasette/issues/2057,1722266942,IC_kwDOBm6k_c5mp7U-,9599,simonw,2023-09-16T16:38:27Z,2023-09-16T16:38:27Z,OWNER,"The `importlib.metadata.entry_points()` function is pretty interesting:
```pycon
>>> import importlib.metadata
>>> from pprint import pprint
>>> pprint(importlib.metadata.entry_points())
{'babel.checkers': [EntryPoint(name='num_plurals', value='babel.messages.checkers:num_plurals', group='babel.checkers'),
                    EntryPoint(name='python_format', value='babel.messages.checkers:python_format', group='babel.checkers')],
 'babel.extractors': [EntryPoint(name='jinja2', value='jinja2.ext:babel_extract[i18n]', group='babel.extractors'),
                      EntryPoint(name='ignore', value='babel.messages.extract:extract_nothing', group='babel.extractors'),
                      EntryPoint(name='javascript', value='babel.messages.extract:extract_javascript', group='babel.extractors'),
                      EntryPoint(name='python', value='babel.messages.extract:extract_python', group='babel.extractors')],
 'console_scripts': [EntryPoint(name='datasette', value='datasette.cli:cli', group='console_scripts'),
                     EntryPoint(name='normalizer', value='charset_normalizer.cli.normalizer:cli_detect', group='console_scripts'),
                     EntryPoint(name='pypprint', value='pprintpp:console', group='console_scripts'),
                     EntryPoint(name='cog', value='cogapp:main', group='console_scripts'),
                     EntryPoint(name='icdiff', value='icdiff:start', group='console_scripts'),
                     EntryPoint(name='pycodestyle', value='pycodestyle:_main', group='console_scripts'),
                     EntryPoint(name='sphinx-autobuild', value='sphinx_autobuild.__main__:main', group='console_scripts'),
                     EntryPoint(name='sphinx-apidoc', value='sphinx.ext.apidoc:main', group='console_scripts'),
                     EntryPoint(name='sphinx-autogen', value='sphinx.ext.autosummary.generate:main', group='console_scripts'),
                     EntryPoint(name='sphinx-build', value='sphinx.cmd.build:main', group='console_scripts'),
                     EntryPoint(name='sphinx-quickstart', value='sphinx.cmd.quickstart:main', group='console_scripts'),
                     EntryPoint(name='sphinx-to-sqlite', value='sphinx_to_sqlite.cli:cli', group='console_scripts'),
                     EntryPoint(name='pybabel', value='babel.messages.frontend:main', group='console_scripts'),
                     EntryPoint(name='docutils', value='docutils.__main__:main', group='console_scripts'),
                     EntryPoint(name='isort', value='isort.main:main', group='console_scripts'),
                     EntryPoint(name='isort-identify-imports', value='isort.main:identify_imports_main', group='console_scripts'),
                     EntryPoint(name='hupper', value='hupper.cli:main', group='console_scripts'),
                     EntryPoint(name='sqlite-utils', value='sqlite_utils.cli:cli', group='console_scripts'),
                     EntryPoint(name='py.test', value='pytest:console_main', group='console_scripts'),
                     EntryPoint(name='pytest', value='pytest:console_main', group='console_scripts'),
                     EntryPoint(name='pyflakes', value='pyflakes.api:main', group='console_scripts'),
                     EntryPoint(name='livereload', value='livereload.cli:main', group='console_scripts'),
                     EntryPoint(name='uvicorn', value='uvicorn.main:main', group='console_scripts'),
                     EntryPoint(name='httpx', value='httpx:main', group='console_scripts'),
                     EntryPoint(name='flake8', value='flake8.main.cli:main', group='console_scripts'),
                     EntryPoint(name='blacken-docs', value='blacken_docs:main', group='console_scripts'),
                     EntryPoint(name='pip', value='pip._internal.cli.main:main', group='console_scripts'),
                     EntryPoint(name='pip3', value='pip._internal.cli.main:main', group='console_scripts'),
                     EntryPoint(name='pip3.10', value='pip._internal.cli.main:main', group='console_scripts'),
                     EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts'),
                     EntryPoint(name='pygmentize', value='pygments.cmdline:main', group='console_scripts'),
                     EntryPoint(name='black', value='black:patched_main', group='console_scripts'),
                     EntryPoint(name='blackd', value='blackd:patched_main [d]', group='console_scripts'),
                     EntryPoint(name='codespell', value='codespell_lib:_script_main', group='console_scripts'),
                     EntryPoint(name='tabulate', value='tabulate:_main', group='console_scripts')],
 'datasette': [EntryPoint(name='debug_permissions', value='datasette_debug_permissions', group='datasette'),
               EntryPoint(name='codespaces', value='datasette_codespaces', group='datasette'),
               EntryPoint(name='vega', value='datasette_vega', group='datasette'),
               EntryPoint(name='x_forwarded_host', value='datasette_x_forwarded_host', group='datasette'),
               EntryPoint(name='json_html', value='datasette_json_html', group='datasette'),
               EntryPoint(name='datasette_write_ui', value='datasette_write_ui', group='datasette'),
               EntryPoint(name='pretty_json', value='datasette_pretty_json', group='datasette'),
               EntryPoint(name='graphql', value='datasette_graphql', group='datasette')],
 'distutils.commands': [EntryPoint(name='compile_catalog', value='babel.messages.frontend:compile_catalog', group='distutils.commands'),
                        EntryPoint(name='extract_messages', value='babel.messages.frontend:extract_messages', group='distutils.commands'),
                        EntryPoint(name='init_catalog', value='babel.messages.frontend:init_catalog', group='distutils.commands'),
                        EntryPoint(name='update_catalog', value='babel.messages.frontend:update_catalog', group='distutils.commands'),
                        EntryPoint(name='isort', value='isort.setuptools_commands:ISortCommand', group='distutils.commands'),
                        EntryPoint(name='alias', value='setuptools.command.alias:alias', group='distutils.commands'),
                        EntryPoint(name='bdist_egg', value='setuptools.command.bdist_egg:bdist_egg', group='distutils.commands'),
                        EntryPoint(name='bdist_rpm', value='setuptools.command.bdist_rpm:bdist_rpm', group='distutils.commands'),
                        EntryPoint(name='build', value='setuptools.command.build:build', group='distutils.commands'),
                        EntryPoint(name='build_clib', value='setuptools.command.build_clib:build_clib', group='distutils.commands'),
                        EntryPoint(name='build_ext', value='setuptools.command.build_ext:build_ext', group='distutils.commands'),
                        EntryPoint(name='build_py', value='setuptools.command.build_py:build_py', group='distutils.commands'),
                        EntryPoint(name='develop', value='setuptools.command.develop:develop', group='distutils.commands'),
                        EntryPoint(name='dist_info', value='setuptools.command.dist_info:dist_info', group='distutils.commands'),
                        EntryPoint(name='easy_install', value='setuptools.command.easy_install:easy_install', group='distutils.commands'),
                        EntryPoint(name='editable_wheel', value='setuptools.command.editable_wheel:editable_wheel', group='distutils.commands'),
                        EntryPoint(name='egg_info', value='setuptools.command.egg_info:egg_info', group='distutils.commands'),
                        EntryPoint(name='install', value='setuptools.command.install:install', group='distutils.commands'),
                        EntryPoint(name='install_egg_info', value='setuptools.command.install_egg_info:install_egg_info', group='distutils.commands'),
                        EntryPoint(name='install_lib', value='setuptools.command.install_lib:install_lib', group='distutils.commands'),
                        EntryPoint(name='install_scripts', value='setuptools.command.install_scripts:install_scripts', group='distutils.commands'),
                        EntryPoint(name='rotate', value='setuptools.command.rotate:rotate', group='distutils.commands'),
                        EntryPoint(name='saveopts', value='setuptools.command.saveopts:saveopts', group='distutils.commands'),
                        EntryPoint(name='sdist', value='setuptools.command.sdist:sdist', group='distutils.commands'),
                        EntryPoint(name='setopt', value='setuptools.command.setopt:setopt', group='distutils.commands'),
                        EntryPoint(name='test', value='setuptools.command.test:test', group='distutils.commands'),
                        EntryPoint(name='upload_docs', value='setuptools.command.upload_docs:upload_docs', group='distutils.commands'),
                        EntryPoint(name='bdist_wheel', value='wheel.bdist_wheel:bdist_wheel', group='distutils.commands')],
 'distutils.setup_keywords': [EntryPoint(name='message_extractors', value='babel.messages.frontend:check_message_extractors', group='distutils.setup_keywords'),
                              EntryPoint(name='cffi_modules', value='cffi.setuptools_ext:cffi_modules', group='distutils.setup_keywords'),
                              EntryPoint(name='dependency_links', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
                              EntryPoint(name='eager_resources', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'),
                              EntryPoint(name='entry_points', value='setuptools.dist:check_entry_points', group='distutils.setup_keywords'),
                              EntryPoint(name='exclude_package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
                              EntryPoint(name='extras_require', value='setuptools.dist:check_extras', group='distutils.setup_keywords'),
                              EntryPoint(name='include_package_data', value='setuptools.dist:assert_bool', group='distutils.setup_keywords'),
                              EntryPoint(name='install_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                              EntryPoint(name='namespace_packages', value='setuptools.dist:check_nsp', group='distutils.setup_keywords'),
                              EntryPoint(name='package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'),
                              EntryPoint(name='packages', value='setuptools.dist:check_packages', group='distutils.setup_keywords'),
                              EntryPoint(name='python_requires', value='setuptools.dist:check_specifier', group='distutils.setup_keywords'),
                              EntryPoint(name='setup_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                              EntryPoint(name='test_loader', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
                              EntryPoint(name='test_runner', value='setuptools.dist:check_importable', group='distutils.setup_keywords'),
                              EntryPoint(name='test_suite', value='setuptools.dist:check_test_suite', group='distutils.setup_keywords'),
                              EntryPoint(name='tests_require', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'),
                              EntryPoint(name='use_2to3', value='setuptools.dist:invalid_unless_false', group='distutils.setup_keywords'),
                              EntryPoint(name='zip_safe', value='setuptools.dist:assert_bool', group='distutils.setup_keywords')],
 'egg_info.writers': [EntryPoint(name='PKG-INFO', value='setuptools.command.egg_info:write_pkg_info', group='egg_info.writers'),
                      EntryPoint(name='dependency_links.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                      EntryPoint(name='depends.txt', value='setuptools.command.egg_info:warn_depends_obsolete', group='egg_info.writers'),
                      EntryPoint(name='eager_resources.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                      EntryPoint(name='entry_points.txt', value='setuptools.command.egg_info:write_entries', group='egg_info.writers'),
                      EntryPoint(name='namespace_packages.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'),
                      EntryPoint(name='requires.txt', value='setuptools.command.egg_info:write_requirements', group='egg_info.writers'),
                      EntryPoint(name='top_level.txt', value='setuptools.command.egg_info:write_toplevel_names', group='egg_info.writers')],
 'flake8.extension': [EntryPoint(name='C90', value='mccabe:McCabeChecker', group='flake8.extension'),
                      EntryPoint(name='E', value='flake8.plugins.pycodestyle:pycodestyle_logical', group='flake8.extension'),
                      EntryPoint(name='F', value='flake8.plugins.pyflakes:FlakesChecker', group='flake8.extension'),
                      EntryPoint(name='W', value='flake8.plugins.pycodestyle:pycodestyle_physical', group='flake8.extension')],
 'flake8.report': [EntryPoint(name='default', value='flake8.formatting.default:Default', group='flake8.report'),
                   EntryPoint(name='pylint', value='flake8.formatting.default:Pylint', group='flake8.report'),
                   EntryPoint(name='quiet-filename', value='flake8.formatting.default:FilenameOnly', group='flake8.report'),
                   EntryPoint(name='quiet-nothing', value='flake8.formatting.default:Nothing', group='flake8.report')],
 'pylama.linter': [EntryPoint(name='isort', value='isort.pylama_isort:Linter', group='pylama.linter')],
 'pytest11': [EntryPoint(name='icdiff', value='pytest_icdiff', group='pytest11'),
              EntryPoint(name='asyncio', value='pytest_asyncio.plugin', group='pytest11'),
              EntryPoint(name='xdist', value='xdist.plugin', group='pytest11'),
              EntryPoint(name='xdist.looponfail', value='xdist.looponfail', group='pytest11'),
              EntryPoint(name='timeout', value='pytest_timeout', group='pytest11'),
              EntryPoint(name='anyio', value='anyio.pytest_plugin', group='pytest11')],
 'setuptools.finalize_distribution_options': [EntryPoint(name='keywords', value='setuptools.dist:Distribution._finalize_setup_keywords', group='setuptools.finalize_distribution_options'),
                                              EntryPoint(name='parent_finalize', value='setuptools.dist:_Distribution.finalize_options', group='setuptools.finalize_distribution_options')],
 'sphinx.html_themes': [EntryPoint(name='alabaster', value='alabaster', group='sphinx.html_themes'),
                        EntryPoint(name='basic-ng', value='sphinx_basic_ng', group='sphinx.html_themes'),
                        EntryPoint(name='furo', value='furo', group='sphinx.html_themes')],
 'sqlite_utils': [EntryPoint(name='hello_world', value='sqlite_utils_hello_world', group='sqlite_utils')]}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1722266513,https://api.github.com/repos/simonw/datasette/issues/2057,1722266513,IC_kwDOBm6k_c5mp7OR,9599,simonw,2023-09-16T16:36:09Z,2023-09-16T16:36:09Z,OWNER,"Now I need to switch out `pkg_resources` in `plugins.py`:

https://github.com/simonw/datasette/blob/852f5014853943fa27f43ddaa2d442545b3259fb/datasette/plugins.py#L33-L74","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1722265848,https://api.github.com/repos/simonw/datasette/issues/2057,1722265848,IC_kwDOBm6k_c5mp7D4,9599,simonw,2023-09-16T16:32:42Z,2023-09-16T16:32:42Z,OWNER,"Here's the exception it uses:
```pycon
>>> importlib.metadata.version(""datasette"")
'1.0a6'
>>> importlib.metadata.version(""datasette2"")
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 996, in version
    return distribution(distribution_name).version
  File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 969, in distribution
    return Distribution.from_name(distribution_name)
  File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 548, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for datasette2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1722258980,https://api.github.com/repos/simonw/datasette/issues/2057,1722258980,IC_kwDOBm6k_c5mp5Yk,9599,simonw,2023-09-16T15:56:45Z,2023-09-16T15:56:45Z,OWNER,"Weird, I still can't get the warning to show even with this:
```python
@pytest.mark.asyncio
async def test_plugin_is_installed():
    datasette = Datasette(memory=True)

    class DummyPlugin:
        __name__ = ""DummyPlugin""

        @hookimpl
        def actors_from_ids(self, datasette, actor_ids):
            return {}

    try:
        pm.register(DummyPlugin(), name=""DummyPlugin"")
        response = await datasette.client.get(""/-/plugins.json"")
        assert response.status_code == 200
        installed_plugins = {p[""name""] for p in response.json()}
        assert ""DummyPlugin"" in installed_plugins

    finally:
        pm.unregister(name=""ReturnNothingPlugin"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1722257328,https://api.github.com/repos/simonw/datasette/issues/2057,1722257328,IC_kwDOBm6k_c5mp4-w,9599,simonw,2023-09-16T15:47:32Z,2023-09-16T15:47:32Z,OWNER,"Frustrating that this warning doesn't show up in the Datasette test suite itself. It shows up in plugin test suites that run this test:
```python
@pytest.mark.asyncio
async def test_plugin_is_installed():
    datasette = Datasette(memory=True)
    response = await datasette.client.get(""/-/plugins.json"")
    assert response.status_code == 200
    installed_plugins = {p[""name""] for p in response.json()}
    assert ""datasette-chronicle"" in installed_plugins
```
If you run that test inside Datasette core `installed_plugins` is an empty set, which presumably is why the warning doesn't get triggered there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2186#issuecomment-1721742055,https://api.github.com/repos/simonw/datasette/issues/2186,1721742055,IC_kwDOBm6k_c5mn7Ln,9599,simonw,2023-09-15T19:27:59Z,2023-09-15T19:27:59Z,OWNER,This feels like it might be quite a nice pattern generally - providing optional arguments to plugins and views that can be `await get_x()` called to run an extra calculation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1898927976,Mechanism for register_output_renderer hooks to access full count,
https://github.com/simonw/datasette/issues/2186#issuecomment-1721740872,https://api.github.com/repos/simonw/datasette/issues/2186,1721740872,IC_kwDOBm6k_c5mn65I,9599,simonw,2023-09-15T19:26:51Z,2023-09-15T19:27:19Z,OWNER,"Here's where it's called at the moment: https://github.com/simonw/datasette/blob/16f0b6d8222d06682a31b904d0a402c391ae1c1c/datasette/views/base.py#L297-L313

And the docs: https://github.com/simonw/datasette/blob/1.0a6/docs/plugin_hooks.rst#register-output-renderer-datasette

I'm tempted to add a `get_count` argument which, when awaited, returns the full count. Then plugins could do this:

```python
async def render_notebook(datasette, request, get_count, rows):
    count = await get_count()
    # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1898927976,Mechanism for register_output_renderer hooks to access full count,
https://github.com/simonw/datasette/pull/2182#issuecomment-1719451803,https://api.github.com/repos/simonw/datasette/issues/2182,1719451803,IC_kwDOBm6k_c5mfMCb,49699333,dependabot[bot],2023-09-14T13:27:26Z,2023-09-14T13:27:26Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1890593563,Bump the python-packages group with 2 updates,
https://github.com/simonw/datasette/pull/2183#issuecomment-1718316733,https://api.github.com/repos/simonw/datasette/issues/2183,1718316733,IC_kwDOBm6k_c5ma269,9599,simonw,2023-09-13T21:05:36Z,2023-09-13T21:05:36Z,OWNER,I'm going to land this and make any further documentation tweaks on `main`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support,
https://github.com/simonw/datasette/pull/2183#issuecomment-1714544153,https://api.github.com/repos/simonw/datasette/issues/2183,1714544153,IC_kwDOBm6k_c5mMd4Z,22429695,codecov[bot],2023-09-11T20:37:52Z,2023-09-13T20:58:51Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`95.00%`** and project coverage change: **`-0.04%`** :warning:
> Comparison is base [(`a4c96d0`)](https://app.codecov.io/gh/simonw/datasette/commit/a4c96d01b27ce7cd06662a024da3547132a7c412?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`659dcbd`)](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.

> :exclamation: Current head 659dcbd differs from pull request most recent head acca338. Consider uploading reports for the commit acca338 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2183      +/-   ##
==========================================
- Coverage   92.69%   92.66%   -0.04%     
==========================================
  Files          40       40              
  Lines        6025     6039      +14     
==========================================
+ Hits         5585     5596      +11     
- Misses        440      443       +3     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <95.00%> (-0.24%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support,
https://github.com/simonw/datasette/pull/2183#issuecomment-1716801971,https://api.github.com/repos/simonw/datasette/issues/2183,1716801971,IC_kwDOBm6k_c5mVFGz,15178711,asg017,2023-09-13T01:34:01Z,2023-09-13T01:34:01Z,CONTRIBUTOR,"@simonw docs are finished, this is ready for review!

One thing: I added ""Configuration"" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named ""datasette.yaml Configuration"" or something similar?

Mostly because ""Configuration"" by itself can mean many things, but adding ""datasette.yaml"" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using ""datasette.yaml"" instead of ""datasette.json"", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported)

Also, maybe this is a chance to consolidate the docs a bit? I think ""Settings"", ""Configuration"", ""Metadata"", and ""Authentication and permissions"" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support,
https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714920708,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714920708,IC_kwDOCGYnMM5mN50E,9599,simonw,2023-09-12T03:51:13Z,2023-09-12T03:51:13Z,OWNER,"Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`:

https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150

I could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works:
```python
for table, column, other_table, other_column in table.foreign_keys:
   # ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,Represent compound foreign keys in table.foreign_keys output,
https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714919806,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714919806,IC_kwDOCGYnMM5mN5l-,9599,simonw,2023-09-12T03:49:41Z,2023-09-12T03:49:41Z,OWNER,"Digging in a bit more:
```pycon
>>> pprint(list(db.query('PRAGMA foreign_key_list(courses)')))
[{'from': 'campus_name',
  'id': 0,
  'match': 'NONE',
  'on_delete': 'NO ACTION',
  'on_update': 'NO ACTION',
  'seq': 0,
  'table': 'departments',
  'to': 'campus_name'},
 {'from': 'dept_code',
  'id': 0,
  'match': 'NONE',
  'on_delete': 'NO ACTION',
  'on_update': 'NO ACTION',
  'seq': 1,
  'table': 'departments',
  'to': 'dept_code'}]
```
I think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`.

Right now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,Represent compound foreign keys in table.foreign_keys output,
https://github.com/simonw/datasette/pull/2183#issuecomment-1714699724,https://api.github.com/repos/simonw/datasette/issues/2183,1714699724,IC_kwDOBm6k_c5mND3M,9599,simonw,2023-09-11T23:01:36Z,2023-09-11T23:02:30Z,OWNER,"On thinking about this further, I'm fine releasing it as another alpha provided it causes Datasette to error loudly with an explanatory message if you attempt to load `-m metadata.json` at a metadata file that includes `""plugins""` configuration.

Something like this:
```bash
datasette -m metadata.json
```
Outputs:

> Datasette no longer accepts plugin configuration in --metadata. Move your `""plugins""` configuration blocks to a separate file - we suggest calling that `datasette.yaml` - and start Datasette with `datasette -c datasette.yaml`.

For added usability points, let's have it suggest `datasette.json` if they used `metadata.json` and `datasette.yaml` if they tried to use `metadata.yaml`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support,
https://github.com/simonw/datasette/issues/2168#issuecomment-1712897194,https://api.github.com/repos/simonw/datasette/issues/2168,1712897194,IC_kwDOBm6k_c5mGLyq,9599,simonw,2023-09-10T17:54:07Z,2023-09-10T17:54:07Z,OWNER,"This looks relevant:

https://pluggy.readthedocs.io/en/stable/#wrappers

> A *hookimpl* can be marked with the `""wrapper""` option, which indicates that the function will be called to *wrap* (or surround) all other normal *hookimpl* calls. A *hook wrapper* can thus execute some code ahead and after the execution of all corresponding non-wrappper *hookimpls*.

This could be the perfect mechanism for implementing this hook, although I still need to figure out how it interacts with streaming.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1712895580,IC_kwDOCGYnMM5mGLZc,9599,simonw,2023-09-10T17:46:41Z,2023-09-10T17:46:41Z,OWNER,"In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`:

https://www.sqlite.org/lang_vacuum.html#how_vacuum_works

> The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid).

So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values,
https://github.com/simonw/datasette/issues/2180#issuecomment-1712895084,https://api.github.com/repos/simonw/datasette/issues/2180,1712895084,IC_kwDOBm6k_c5mGLRs,9599,simonw,2023-09-10T17:44:14Z,2023-09-10T17:44:14Z,OWNER,Used by `datasette-short-links` here: https://github.com/datasette/datasette-short-links/blob/468c3e25dbe06a8dcba8edda59bc16a18e126a51/datasette_short_links/__init__.py#L108-L115,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`,
https://github.com/simonw/sqlite-utils/pull/593#issuecomment-1710939868,https://api.github.com/repos/simonw/sqlite-utils/issues/593,1710939868,IC_kwDOCGYnMM5l-t7c,22429695,codecov[bot],2023-09-08T01:03:40Z,2023-09-09T00:44:52Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and no project coverage change.
> Comparison is base [(`5d123f0`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/5d123f031fc4fadc98f508e0ef6b7b6671e86155?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b86374f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #593   +/-   ##
=======================================
  Coverage   95.77%   95.77%           
=======================================
  Files           8        8           
  Lines        2837     2840    +3     
=======================================
+ Hits         2717     2720    +3     
  Misses        120      120           
```


| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <100.00%> (+<0.01%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886783150,".transform() now preserves rowid values, refs #592",
https://github.com/simonw/datasette/issues/2180#issuecomment-1711057080,https://api.github.com/repos/simonw/datasette/issues/2180,1711057080,IC_kwDOBm6k_c5l_Ki4,9599,simonw,2023-09-08T04:28:03Z,2023-09-08T04:28:03Z,OWNER,"Landed:

- https://docs.datasette.io/en/latest/plugin_hooks.html#actors-from-ids-datasette-actor-ids
- https://docs.datasette.io/en/latest/internals.html#await-actors-from-ids-actor-ids","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`,
https://github.com/simonw/datasette/pull/2181#issuecomment-1711054840,https://api.github.com/repos/simonw/datasette/issues/2181,1711054840,IC_kwDOBm6k_c5l_J_4,9599,simonw,2023-09-08T04:23:47Z,2023-09-08T04:23:47Z,OWNER,"I've implemented this hook once now in:
- https://github.com/datasette/datasette-remote-actors

And built and tested a debug tool for it in:
- https://github.com/datasette/datasette-debug-actors-from-ids

I'm now confident in the design of this plugin hook, I'm going to land it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886812002,actors_from_ids plugin hook and datasette.actors_from_ids() method,
https://github.com/simonw/datasette/issues/2180#issuecomment-1711054624,https://api.github.com/repos/simonw/datasette/issues/2180,1711054624,IC_kwDOBm6k_c5l_J8g,9599,simonw,2023-09-08T04:23:23Z,2023-09-08T04:23:23Z,OWNER,"I've implemented this hook once now in:
- https://github.com/datasette/datasette-remote-actors

And built and tested a debug tool for it in:
- https://github.com/datasette/datasette-debug-actors-from-ids

I'm now confident in the design of this plugin hook, I'm going to land it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`,
https://github.com/simonw/datasette/pull/2181#issuecomment-1710972324,https://api.github.com/repos/simonw/datasette/issues/2181,1710972324,IC_kwDOBm6k_c5l-12k,22429695,codecov[bot],2023-09-08T01:58:44Z,2023-09-08T03:43:43Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and no project coverage change.
> Comparison is base [(`c263704`)](https://app.codecov.io/gh/simonw/datasette/commit/c26370485a4fd4bf130da051be9163d92c57f24f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`e1c5a9d`)](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2181   +/-   ##
=======================================
  Coverage   92.68%   92.69%           
=======================================
  Files          40       40           
  Lines        6017     6025    +8     
=======================================
+ Hits         5577     5585    +8     
  Misses        440      440           
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.43% <100.00%> (+0.03%)` | :arrow_up: |
| [datasette/hookspecs.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   

:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886812002,actors_from_ids plugin hook and datasette.actors_from_ids() method,
https://github.com/simonw/datasette/issues/2180#issuecomment-1711028355,https://api.github.com/repos/simonw/datasette/issues/2180,1711028355,IC_kwDOBm6k_c5l_DiD,9599,simonw,2023-09-08T03:36:02Z,2023-09-08T03:36:02Z,OWNER,"I shipped the first version of 

- https://github.com/datasette/datasette-remote-actors/issues/1

When I land this plugin in a Datasette release I need to update that repo to depend on the new alpha.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`,
https://github.com/simonw/datasette/pull/2181#issuecomment-1710969448,https://api.github.com/repos/simonw/datasette/issues/2181,1710969448,IC_kwDOBm6k_c5l-1Jo,9599,simonw,2023-09-08T01:53:27Z,2023-09-08T01:53:27Z,OWNER,"Documentation preview:
- https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids
- https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886812002,actors_from_ids plugin hook and datasette.actors_from_ids() method,
https://github.com/simonw/datasette/issues/2180#issuecomment-1710969339,https://api.github.com/repos/simonw/datasette/issues/2180,1710969339,IC_kwDOBm6k_c5l-1H7,9599,simonw,2023-09-08T01:53:16Z,2023-09-08T01:53:16Z,OWNER,"Documentation preview:
- https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids
- https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1710950671,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1710950671,IC_kwDODFE5qs5l-wkP,150855,iloveitaly,2023-09-08T01:22:49Z,2023-09-08T01:22:49Z,NONE,"Makes sense, thanks for explaining!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/simonw/datasette/issues/2180#issuecomment-1710947637,https://api.github.com/repos/simonw/datasette/issues/2180,1710947637,IC_kwDOBm6k_c5l-v01,9599,simonw,2023-09-08T01:17:28Z,2023-09-08T01:17:28Z,OWNER,I think this is both a plugin hook and a `await datasette.actors_from_ids(actor_ids)` internal API function that calls it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710935270,IC_kwDOCGYnMM5l-szm,9599,simonw,2023-09-08T00:55:30Z,2023-09-08T00:55:30Z,OWNER,"Yes! That recreated the bug:
```
>       assert previous_rows == next_rows
E       AssertionError: assert equals failed
E         [                                                                [                                                               
E           (1, '1', 'Paris'),                                               (1, '1', 'Paris'),                                            
E           (3, '3', 'New York'),                                            (2, '3', 'New York'),                                         
E           (4, '4', 'London'),                                              (3, '4', 'London'),                                           
E         ]                                                       ...
E         
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710934448,IC_kwDOCGYnMM5l-smw,9599,simonw,2023-09-08T00:54:03Z,2023-09-08T00:54:03Z,OWNER,"Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3.

If I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710933716,IC_kwDOCGYnMM5l-sbU,9599,simonw,2023-09-08T00:52:42Z,2023-09-08T00:52:42Z,OWNER,"I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key.

The reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows.

But I tried this test and it passed, too:
```python
def test_transform_preserves_rowids(fresh_db):
    fresh_db[""places""].insert_all(
        [
            {""id"": ""1"", ""name"": ""Paris"", ""country"": ""France""},
            {""id"": ""2"", ""name"": ""London"", ""country"": ""UK""},
            {""id"": ""3"", ""name"": ""New York"", ""country"": ""USA""},
        ],
        pk=""id"",
    )
    previous_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"")
    )
    # Transform it
    fresh_db[""places""].transform(column_order=(""country"", ""name""))
    # Should be the same
    next_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"")
    )
    assert previous_rows == next_rows
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710931605,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710931605,IC_kwDOCGYnMM5l-r6V,9599,simonw,2023-09-08T00:49:02Z,2023-09-08T00:49:02Z,OWNER,"I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values,
https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710930934,IC_kwDOCGYnMM5l-rv2,9599,simonw,2023-09-08T00:47:57Z,2023-09-08T00:47:57Z,OWNER,"That's odd, I wrote a test for this just now and it passes already:
```python
def test_transform_preserves_rowids(fresh_db):
    # Create a rowid table
    fresh_db[""places""].insert_all(
        (
            {""name"": ""Paris"", ""country"": ""France""},
            {""name"": ""London"", ""country"": ""UK""},
            {""name"": ""New York"", ""country"": ""USA""},
        ),
    )
    assert fresh_db[""places""].use_rowid
    previous_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, name from places"")
    )
    # Transform it
    fresh_db[""places""].transform(column_order=(""country"", ""name""))
    # Should be the same
    next_rows = list(
        tuple(row) for row in fresh_db.execute(""select rowid, name from places"")
    )
    assert previous_rows == next_rows
```
So maybe I'm wrong about the cause of that bug?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values,
https://github.com/simonw/datasette/pull/2174#issuecomment-1708699926,https://api.github.com/repos/simonw/datasette/issues/2174,1708699926,IC_kwDOBm6k_c5l2LEW,22429695,codecov[bot],2023-09-06T16:18:13Z,2023-09-08T00:46:13Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`d75b519`)](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
> Report is 7 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2174   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          40       40           
  Lines        6012     6017    +5     
=======================================
+ Hits         5572     5577    +5     
  Misses        440      440           
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <ø> (ø)` | |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2174/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   

:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884330740,Use $DATASETTE_INTERNAL in absence of --internal,
https://github.com/simonw/datasette/pull/2174#issuecomment-1708727204,https://api.github.com/repos/simonw/datasette/issues/2174,1708727204,IC_kwDOBm6k_c5l2Ruk,9599,simonw,2023-09-06T16:35:39Z,2023-09-07T23:45:42Z,OWNER,"We can use this here: https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables
```python
@click.option(..., envvar=""DATASETTE_INTERNAL"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884330740,Use $DATASETTE_INTERNAL in absence of --internal,
https://github.com/simonw/datasette/issues/2178#issuecomment-1710879239,https://api.github.com/repos/simonw/datasette/issues/2178,1710879239,IC_kwDOBm6k_c5l-fIH,9599,simonw,2023-09-07T23:20:32Z,2023-09-07T23:20:32Z,OWNER,"To test that locally, use this YAML instead:
```yaml
databases:
  content:
    allow:
      id: root
    tables:
      releases:
        allow: true
```
And:
```yaml
allow:
  id: root
databases:
  content:
    tables:
      releases:
        allow: true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access,
https://github.com/simonw/datasette/issues/2178#issuecomment-1710878391,https://api.github.com/repos/simonw/datasette/issues/2178,1710878391,IC_kwDOBm6k_c5l-e63,9599,simonw,2023-09-07T23:19:05Z,2023-09-07T23:19:05Z,OWNER,"This fix didn't work on Datasette Cloud. I used `/-/permissions` to debug it and saw this:

![image](https://github.com/simonw/datasette/assets/9599/61d2bc5f-1f96-41ea-8658-91dfbcb6610c)

Only checking `view-table` is not enough: for my instance on Datasette Cloud the view permission check that should have failed was for the database or instance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access,
https://github.com/simonw/datasette/issues/2178#issuecomment-1710871095,https://api.github.com/repos/simonw/datasette/issues/2178,1710871095,IC_kwDOBm6k_c5l-dI3,9599,simonw,2023-09-07T23:07:16Z,2023-09-07T23:07:16Z,OWNER,"I ran this:

    datasette content.db -p 8043 -m fk-auth.yml --root

Against this YAML:
```yaml
databases:
  content:
    tables:
      users:
        allow:
          id: root
```
And it worked as it should - here's a screenshot of an anonymous user and a root user viewing the same page:

![CleanShot 2023-09-07 at 16 05 34@2x](https://github.com/simonw/datasette/assets/9599/3e91da08-107c-421c-8a00-aa650b960c58)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access,
https://github.com/simonw/datasette/issues/2178#issuecomment-1710567329,https://api.github.com/repos/simonw/datasette/issues/2178,1710567329,IC_kwDOBm6k_c5l9S-h,9599,simonw,2023-09-07T17:59:59Z,2023-09-07T17:59:59Z,OWNER,Should I put the permission check in that undocumented `datasette.expand_foreign_keys()` method? I think so - it should accept `request.actor` as one of its arguments.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access,
https://github.com/simonw/datasette/issues/2178#issuecomment-1710565268,https://api.github.com/repos/simonw/datasette/issues/2178,1710565268,IC_kwDOBm6k_c5l9SeU,9599,simonw,2023-09-07T17:58:04Z,2023-09-07T17:59:06Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/views/table.py#L1132-L1149

Which calls this undocumented method:

https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/app.py#L938-L973","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access,
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1710380941,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1710380941,IC_kwDODFE5qs5l8leN,28565,maxhawkins,2023-09-07T15:39:59Z,2023-09-07T15:39:59Z,NONE,"> @maxhawkins curious why you didn't use the stdlib `mailbox` to parse the `mbox` files?

Mailbox parses the entire mbox into memory. Using the lower level library lets us stream the emails in one at a time to support larger archives. Both libraries are in the stdlib.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/simonw/datasette/issues/1479#issuecomment-1709373304,https://api.github.com/repos/simonw/datasette/issues/1479,1709373304,IC_kwDOBm6k_c5l4vd4,363004,hcarter333,2023-09-07T02:14:15Z,2023-09-07T02:14:15Z,NONE,"I ran into the same issue on Windows using 

`datasette publish cloudrun mydatabase.db --service=my-database`
do do a [google cloud publish](https://docs.datasette.io/en/stable/publish.html).

@Rik-de-Kort your fix worked perfectly! Thanks!

I can always go back and delete the temp directories myself :)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1010112818,"Win32 ""used by another process"" error with datasette publish",
https://github.com/dogsheep/google-takeout-to-sqlite/pull/8#issuecomment-1708945716,https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/8,1708945716,IC_kwDODFE5qs5l3HE0,150855,iloveitaly,2023-09-06T19:12:33Z,2023-09-06T19:12:33Z,NONE,@maxhawkins curious why you didn't use the stdlib `mailbox` to parse the `mbox` files?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",954546309,Add Gmail takeout mbox import (v2),
https://github.com/simonw/datasette/issues/2177#issuecomment-1708777964,https://api.github.com/repos/simonw/datasette/issues/2177,1708777964,IC_kwDOBm6k_c5l2eHs,9599,simonw,2023-09-06T17:04:30Z,2023-09-06T17:04:30Z,OWNER,"Here's the thinking: setting the`DATASETTE_INTERNAL` environment variable for your entire machine will cause any time you run `datasette x.db` to result in a process that shares the same `internal.db` database as other instances.

You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables:

https://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62

The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad.

This convinced us that actually we should move those `catalog_` tables OUT of `internal.db`. The `_internal` database will be reserved for plugins that want to use it for caching, storing progress, etc.

I think we move them to an in-memory `_catalog` database which is excluded from `ds.databases` (like `_internal` is ) but can be accessed using `datasette.get_catalog_database()` - similar to `datasette.get_internal_database()`.

So each instance of Datasette gets its own truly private `_catalog`, which is in-memory and so gets cleared at the end of each process.

An interesting thing that came up about a shared `_internal` database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884408624,Move schema tables from _internal to _catalog,
https://github.com/simonw/datasette/pull/2174#issuecomment-1708728926,https://api.github.com/repos/simonw/datasette/issues/2174,1708728926,IC_kwDOBm6k_c5l2SJe,9599,simonw,2023-09-06T16:36:56Z,2023-09-06T16:36:56Z,OWNER,`DATASETTE_INTERNAL` would be more consistent with `DATASETTE_SECRET`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884330740,Use $DATASETTE_INTERNAL in absence of --internal,
https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708695907,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708695907,IC_kwDOCGYnMM5l2KFj,9599,simonw,2023-09-06T16:15:59Z,2023-09-06T16:19:14Z,OWNER,"The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant:
- https://github.com/pypa/setuptools/issues/3661

```
25h  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
...
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py"", line 697, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py"", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py"", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""<frozen importlib._bootstrap>"", line 1381, in _gcd_import
  File ""<frozen importlib._bootstrap>"", line 1354, in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1304, in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 488, in _call_with_frames_removed
  File ""<frozen importlib._bootstrap>"", line 1381, in _gcd_import
  File ""<frozen importlib._bootstrap>"", line 1354, in _find_and_load
  File ""<frozen importlib._bootstrap>"", line 1325, in _find_and_load_unlocked
  File ""<frozen importlib._bootstrap>"", line 929, in _load_unlocked
  File ""<frozen importlib._bootstrap_external>"", line 994, in exec_module
  File ""<frozen importlib._bootstrap>"", line 488, in _call_with_frames_removed
  File ""/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py"", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview,
https://github.com/simonw/datasette/pull/2166#issuecomment-1701045404,https://api.github.com/repos/simonw/datasette/issues/2166,1701045404,IC_kwDOBm6k_c5lY-Sc,22429695,codecov[bot],2023-08-31T13:31:15Z,2023-09-06T13:25:17Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`7c9df6e`)](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2166   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          40       40           
  Lines        6012     6012           
=======================================
  Hits         5572     5572           
  Misses        440      440           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1875519316,Bump the python-packages group with 1 update,
https://github.com/simonw/datasette/pull/2173#issuecomment-1707570044,https://api.github.com/repos/simonw/datasette/issues/2173,1707570044,IC_kwDOBm6k_c5lx3N8,22429695,codecov[bot],2023-09-06T02:39:59Z,2023-09-06T02:49:01Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`fd083e3`)](https://app.codecov.io/gh/simonw/datasette/commit/fd083e37ec53e7e625111168d324a572344a3b19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`a481ebd`)](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2173   +/-   ##
=======================================
  Coverage   92.68%   92.68%           
=======================================
  Files          40       40           
  Lines        6012     6012           
=======================================
  Hits         5572     5572           
  Misses        440      440           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1883055640,click-default-group>=1.2.3,
https://github.com/simonw/datasette/pull/2173#issuecomment-1707570378,https://api.github.com/repos/simonw/datasette/issues/2173,1707570378,IC_kwDOBm6k_c5lx3TK,9599,simonw,2023-09-06T02:40:31Z,2023-09-06T02:40:31Z,OWNER,"Surprising error:
```
  blacken-docs -l 60 docs/*.rst
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib
docs/writing_plugins.rst:365: code block parse error Cannot parse: 8:0: <line number missing in source>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1883055640,click-default-group>=1.2.3,
https://github.com/simonw/datasette/pull/2173#issuecomment-1707565495,https://api.github.com/repos/simonw/datasette/issues/2173,1707565495,IC_kwDOBm6k_c5lx2G3,9599,simonw,2023-09-06T02:33:56Z,2023-09-06T02:33:56Z,OWNER,Running tests to see if Pyodide works - that was the reason I switched to my `click-default-group-wheel` package originally.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1883055640,click-default-group>=1.2.3,
https://github.com/simonw/datasette/issues/328#issuecomment-1706701195,https://api.github.com/repos/simonw/datasette/issues/328,1706701195,IC_kwDOBm6k_c5lujGL,7983005,eric-burel,2023-09-05T14:10:39Z,2023-09-05T14:10:39Z,NONE,"Hey @simonw I hit the same issue as mentionned by @chmaynard on a fresh install, ""/mnt/fixtures.db"" doesn't seem to exist in the docker image","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",336464733,"Installation instructions, including how to use the docker image",
https://github.com/simonw/sqlite-utils/issues/590#issuecomment-1704387161,https://api.github.com/repos/simonw/sqlite-utils/issues/590,1704387161,IC_kwDOCGYnMM5lluJZ,9599,simonw,2023-09-03T19:50:36Z,2023-09-03T19:50:36Z,OWNER,"Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879214365,Ability to tell if a Database is an in-memory one,
https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384393,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384393,IC_kwDOCGYnMM5llteJ,9599,simonw,2023-09-03T19:36:34Z,2023-09-03T19:36:34Z,OWNER,"Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b

Still needs tests and documentation (and some more thought to make sure it's doing the right thing).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions,
https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384111,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384111,IC_kwDOCGYnMM5lltZv,9599,simonw,2023-09-03T19:35:03Z,2023-09-03T19:35:03Z,OWNER,"Normally in Python/`sqlite3` you de-register a function by passing `None` to it.

You can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else:

https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions,
https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704383901,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704383901,IC_kwDOCGYnMM5lltWd,9599,simonw,2023-09-03T19:34:05Z,2023-09-03T19:34:05Z,OWNER,"For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager.

I played with this idea a bit:

```python
with db.register_functions(md5, md5_random):
    db.query(...)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions,
https://github.com/simonw/datasette/issues/1765#issuecomment-1701894468,https://api.github.com/repos/simonw/datasette/issues/1765,1701894468,IC_kwDOBm6k_c5lcNlE,9599,simonw,2023-08-31T23:08:24Z,2023-08-31T23:08:24Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#plugins-that-define-new-plugin-hooks,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1292370469,Document plugins providing new plugin hook-,
https://github.com/simonw/datasette/issues/2168#issuecomment-1701831013,https://api.github.com/repos/simonw/datasette/issues/2168,1701831013,IC_kwDOBm6k_c5lb-Fl,9599,simonw,2023-08-31T21:51:12Z,2023-08-31T21:52:15Z,OWNER,"Need to make sure the design of this takes streaming responses into account. Those could be pretty tricky here.

I nice thing about `asgi_wrapper()` is that it handles those already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),
https://github.com/simonw/datasette/issues/2168#issuecomment-1701830241,https://api.github.com/repos/simonw/datasette/issues/2168,1701830241,IC_kwDOBm6k_c5lb95h,9599,simonw,2023-08-31T21:50:18Z,2023-08-31T21:50:18Z,OWNER,"The hook could be called `register_middleware()` and could work like `register_routes()` and `register_commands()`:

```python
@hookspec
def register_middleware(datasette):
    """"""Register middleware: returns a list of async def middleware functions""""""

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),
https://github.com/simonw/datasette/issues/2168#issuecomment-1701828197,https://api.github.com/repos/simonw/datasette/issues/2168,1701828197,IC_kwDOBm6k_c5lb9Zl,9599,simonw,2023-08-31T21:48:00Z,2023-08-31T21:48:57Z,OWNER,"A pattern like this could be interesting:
```python
async def my_middleware(datasette, request, get_response):
    # Mess with request here if neccessary
    response = await get_response(request)
    # mess with response
    return response
```
The Django pattern is more complicated but does have that mechanism for running one-time configuration prior to defining the `middleware()` function, which is neat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),
https://github.com/simonw/datasette/issues/2168#issuecomment-1701826521,https://api.github.com/repos/simonw/datasette/issues/2168,1701826521,IC_kwDOBm6k_c5lb8_Z,9599,simonw,2023-08-31T21:46:13Z,2023-08-31T21:46:13Z,OWNER,"This could even be a pair of hooks - `process_request()` and `process_response()`.

Or could take a leaf from Django, which redesigned middleware to use this pattern instead:

```python
def simple_middleware(get_response):
    # One-time configuration and initialization.
    def middleware(request):
        # Code to be executed for each request before
        # the view (and later middleware) are called.
        response = get_response(request)
        # Code to be executed for each request/response after
        # the view is called.
        return response
    return middleware
```
Or even borrow an idea from `pytest` where fixtures can `yield` in the middle, like this:
```python
@pytest.fixture
def sending_user(mail_admin):
    user = mail_admin.create_user()
    yield user
    mail_admin.delete_user(user)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),
https://github.com/simonw/datasette/issues/2168#issuecomment-1701823609,https://api.github.com/repos/simonw/datasette/issues/2168,1701823609,IC_kwDOBm6k_c5lb8R5,9599,simonw,2023-08-31T21:43:06Z,2023-08-31T21:44:13Z,OWNER,"Not sure what to call this. Maybe `app_wrapper()`?

Or perhaps it's simpler than that, something like this:

```python
@hookspec
def process_response(datasette, request, response):
    """"""Last chance to modify the response before it is returned to the client""""""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(),
https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967,https://api.github.com/repos/simonw/datasette/issues/2157,1700291967,IC_kwDOBm6k_c5lWGV_,15178711,asg017,2023-08-31T02:45:56Z,2023-08-31T02:45:56Z,CONTRIBUTOR,"@simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database)

My rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH=""/path/to/internal.db""` to their bashrc/zshrc/whereever to not have to worry about `--internal`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,"Proposal: Make the `_internal` database persistent, customizable, and hidden",
https://github.com/simonw/datasette/issues/1092#issuecomment-1699926384,https://api.github.com/repos/simonw/datasette/issues/1092,1699926384,IC_kwDOBm6k_c5lUtFw,9599,simonw,2023-08-30T22:17:42Z,2023-08-30T22:17:42Z,OWNER,"This is implemented now:
- https://docs.datasette.io/en/stable/internals.html#await-ensure-permissions-actor-permissions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",742041667,Make cascading permission checks available to plugins,
https://github.com/simonw/datasette/issues/1190#issuecomment-1699925224,https://api.github.com/repos/simonw/datasette/issues/1190,1699925224,IC_kwDOBm6k_c5lUszo,9599,simonw,2023-08-30T22:16:38Z,2023-08-30T22:16:38Z,OWNER,"This is going to happen in this tool instead:
- https://github.com/simonw/dclient","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",787098146,`datasette publish upload` mechanism for uploading databases to an existing Datasette instance,
https://github.com/simonw/datasette/pull/2165#issuecomment-1699910555,https://api.github.com/repos/simonw/datasette/issues/2165,1699910555,IC_kwDOBm6k_c5lUpOb,9599,simonw,2023-08-30T22:05:14Z,2023-08-30T22:05:14Z,OWNER,Documentation preview: https://github.com/simonw/datasette/blob/6321c9c055a640ed6ea98e231dc5813dcde1f773/docs/plugins.rst#controlling-which-plugins-are-loaded,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,
https://github.com/simonw/datasette/pull/2165#issuecomment-1699812599,https://api.github.com/repos/simonw/datasette/issues/2165,1699812599,IC_kwDOBm6k_c5lURT3,22429695,codecov[bot],2023-08-30T20:43:19Z,2023-08-30T22:05:05Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`26.66%`** and project coverage change: **`-0.17%`** :warning:
> Comparison is base [(`30b28c8`)](https://app.codecov.io/gh/simonw/datasette/commit/30b28c8367a9c6870386ea10a202705b40862457?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.84% compared to head [(`5eddf34`)](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.

> :exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2165      +/-   ##
==========================================
- Coverage   92.84%   92.68%   -0.17%     
==========================================
  Files          40       40              
  Lines        5998     6012      +14     
==========================================
+ Hits         5569     5572       +3     
- Misses        429      440      +11     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/plugins.py](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `64.58% <26.66%> (-17.77%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,
https://github.com/simonw/datasette/pull/2165#issuecomment-1699884314,https://api.github.com/repos/simonw/datasette/issues/2165,1699884314,IC_kwDOBm6k_c5lUi0a,9599,simonw,2023-08-30T21:38:29Z,2023-08-30T21:39:51Z,OWNER,"Here's the reason for that name disparity:

https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L54-L65

Note how the `distinfo.project_name` name is used when available. That seems to work for regularly installed plugins but not for plugins loaded via `DATASETTE_LOAD_PLUGINS`.

And that's looking things up in `plugin_to_distinfo` which is populated here:

https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L37","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,
https://github.com/simonw/datasette/pull/2165#issuecomment-1699811810,https://api.github.com/repos/simonw/datasette/issues/2165,1699811810,IC_kwDOBm6k_c5lURHi,9599,simonw,2023-08-30T20:42:41Z,2023-08-30T20:42:41Z,OWNER,"The `load_setuptools_entrypoints()` function in Pluggy [does this](https://github.com/pytest-dev/pluggy/blob/0b41c9766508a46ae666cf281684df3164b3e2a9/src/pluggy/_manager.py#L376):

```python
            for ep in dist.entry_points:
                if (
                    ep.group != group
                    or (name is not None and ep.name != name)
                    # already registered
                    or self.get_plugin(ep.name)
                    or self.is_blocked(ep.name)
                ):
                    continue
                plugin = ep.load()
                self.register(plugin, name=ep.name)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,
https://github.com/simonw/datasette/pull/2165#issuecomment-1699809688,https://api.github.com/repos/simonw/datasette/issues/2165,1699809688,IC_kwDOBm6k_c5lUQmY,9599,simonw,2023-08-30T20:41:06Z,2023-08-30T20:41:06Z,OWNER,"Slight weirdness: I noticed that the output from the `datasette plugins` command looks like this for plugins loaded with the new environment variable:
```json
    {
        ""name"": ""datasette_pretty_json"",
        ""static"": false,
        ""templates"": false,
        ""version"": null,
        ""hooks"": [
            ""render_cell""
        ]
    },
```
That should ideally be `datasette-pretty-json`, not `datasette_pretty_json`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,
https://github.com/simonw/datasette/pull/2165#issuecomment-1699802028,https://api.github.com/repos/simonw/datasette/issues/2165,1699802028,IC_kwDOBm6k_c5lUOus,9599,simonw,2023-08-30T20:35:02Z,2023-08-30T20:35:02Z,OWNER,"Testing this is going to be a bit of a pain.

I think I'll add a whole separate test block to CI which installs a couple of plugins and then exercises this feature using `datasette plugins`.

I'll use `datasette-init` and `datasette-json-html` just because they are small and simple.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins,
https://github.com/simonw/datasette/issues/2164#issuecomment-1699728102,https://api.github.com/repos/simonw/datasette/issues/2164,1699728102,IC_kwDOBm6k_c5lT8rm,9599,simonw,2023-08-30T19:35:32Z,2023-08-30T19:35:47Z,OWNER,"Figured this out so far with the help of GPT-4: https://chat.openai.com/share/0e785865-621b-4fb3-ba05-7449e57c8496

Now this works:
```bash
DATASETTE_LOAD_PLUGINS=datasette-write-ui datasette plugins
```
```json
[
    {
        ""name"": ""datasette_write_ui"",
        ""static"": true,
        ""templates"": true,
        ""version"": null,
        ""hooks"": [
            ""extra_template_vars"",
            ""register_routes""
        ]
    }
]
```
Or multiple plugins:
```bash
DATASETTE_LOAD_PLUGINS=datasette-write-ui,datasette-pretty-json datasette plugins
```
Outputs:
```json
[
    {
        ""name"": ""datasette_pretty_json"",
        ""static"": false,
        ""templates"": false,
        ""version"": null,
        ""hooks"": [
            ""render_cell""
        ]
    },
    {
        ""name"": ""datasette_write_ui"",
        ""static"": true,
        ""templates"": true,
        ""version"": null,
        ""hooks"": [
            ""extra_template_vars"",
            ""register_routes""
        ]
    }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874255116,Ability to only load a specific list of plugins,
https://github.com/simonw/datasette/issues/2163#issuecomment-1697818917,https://api.github.com/repos/simonw/datasette/issues/2163,1697818917,IC_kwDOBm6k_c5lMqkl,9599,simonw,2023-08-29T16:56:20Z,2023-08-29T16:56:20Z,OWNER,https://github.com/simonw/datasette/blob/50da908213a0fc405ecd7a40090dfea7a2e7395c/datasette/utils/internal_db.py#L8-L62,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1872043170,Rename core_X to catalog_X in the internals,
https://github.com/simonw/datasette/pull/2154#issuecomment-1691798722,https://api.github.com/repos/simonw/datasette/issues/2154,1691798722,IC_kwDOBm6k_c5k1szC,22429695,codecov[bot],2023-08-24T14:31:42Z,2023-08-29T16:15:12Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`+0.03%`** :tada:
> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`3e49fd3`)](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.85%.
> Report is 3 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2154      +/-   ##
==========================================
+ Coverage   92.82%   92.85%   +0.03%     
==========================================
  Files          40       40              
  Lines        5948     6008      +60     
==========================================
+ Hits         5521     5579      +58     
- Misses        427      429       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <ø> (-0.85%)` | :arrow_down: |
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.37% <100.00%> (+0.07%)` | :arrow_up: |
| [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.48% <100.00%> (+0.57%)` | :arrow_up: |
| [datasette/permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3Blcm1pc3Npb25zLnB5) | `100.00% <100.00%> (ø)` | |

... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2154/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/pull/2154#issuecomment-1697725150,https://api.github.com/repos/simonw/datasette/issues/2154,1697725150,IC_kwDOBm6k_c5lMTre,9599,simonw,2023-08-29T15:54:12Z,2023-08-29T15:54:12Z,OWNER,"In that last commit I also upgraded `Permission` from a named tuple to a dataclass, and added a `implies_can_view=True` private (not documented) option to it.

Last step is to refactor the code to use that new property.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/pull/2154#issuecomment-1697612168,https://api.github.com/repos/simonw/datasette/issues/2154,1697612168,IC_kwDOBm6k_c5lL4GI,9599,simonw,2023-08-29T14:57:54Z,2023-08-29T14:57:54Z,OWNER,"The code to refactor is this: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L181-L280

I'm going to turn that into a more general `restrictions_allow_action` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/pull/2162#issuecomment-1696710911,https://api.github.com/repos/simonw/datasette/issues/2162,1696710911,IC_kwDOBm6k_c5lIcD_,9599,simonw,2023-08-29T03:23:44Z,2023-08-29T03:23:44Z,OWNER,I'm going to merge this so we can see how it feels.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database",
https://github.com/simonw/datasette/pull/2162#issuecomment-1696709110,https://api.github.com/repos/simonw/datasette/issues/2162,1696709110,IC_kwDOBm6k_c5lIbn2,9599,simonw,2023-08-29T03:20:40Z,2023-08-29T03:22:47Z,OWNER,"> However, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch.

I'm completely happy for the `core_*` tables (or `datasette_*` or some other name) to live in the persisted-to-disk `internal.db` database, even though they're effectively meant to be an in-memory cache.

I don't think it causes any harm, and it could even be quite useful to have them visible on disk - other applications could read the `internal.db` database while Datasette itself is running, should they have some weird reason to want to do that!

Having those tables stick around in `internal.db` after Datasette shuts down could be useful for other debugging activities as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database",
https://github.com/simonw/datasette/pull/2162#issuecomment-1696594855,https://api.github.com/repos/simonw/datasette/issues/2162,1696594855,IC_kwDOBm6k_c5lH_un,22429695,codecov[bot],2023-08-29T00:19:29Z,2023-08-29T03:22:21Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning:
> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`73489ca`)](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.79%.
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2162      +/-   ##
==========================================
- Coverage   92.82%   92.79%   -0.03%     
==========================================
  Files          40       40              
  Lines        5948     5953       +5     
==========================================
+ Hits         5521     5524       +3     
- Misses        427      429       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `96.87% <ø> (-0.04%)` | :arrow_down: |
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.32% <100.00%> (+0.03%)` | :arrow_up: |
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (-0.06%)` | :arrow_down: |
| [datasette/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.72% <100.00%> (+0.04%)` | :arrow_up: |
| [datasette/utils/internal\_db.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2ludGVybmFsX2RiLnB5) | `100.00% <100.00%> (ø)` | |
| [datasette/views/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.45% <100.00%> (ø)` | |
| [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <100.00%> (-0.85%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database",
https://github.com/simonw/datasette/pull/2162#issuecomment-1696707458,https://api.github.com/repos/simonw/datasette/issues/2162,1696707458,IC_kwDOBm6k_c5lIbOC,9599,simonw,2023-08-29T03:17:54Z,2023-08-29T03:17:54Z,OWNER,Documentation preview: https://datasette--2162.org.readthedocs.build/en/2162/internals.html#datasette-s-internal-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database",
https://github.com/simonw/datasette/pull/2154#issuecomment-1696644066,https://api.github.com/repos/simonw/datasette/issues/2154,1696644066,IC_kwDOBm6k_c5lILvi,9599,simonw,2023-08-29T01:37:47Z,2023-08-29T03:00:25Z,OWNER,"Code for this might be cleaner with a `Restrictions()` class that takes a `""_r""` dictionary to the constructor and can then answer questions like `.any_resource_has_permission(""view-table"")` - where it can resolve aliases etc as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/pull/2154#issuecomment-1696642671,https://api.github.com/repos/simonw/datasette/issues/2154,1696642671,IC_kwDOBm6k_c5lILZv,9599,simonw,2023-08-29T01:35:37Z,2023-08-29T01:35:37Z,OWNER,"Reminder that I also need to confirm that `insert-row` works if you have it at the instance level, the database level or the resource level in `_r`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/pull/2154#issuecomment-1696618784,https://api.github.com/repos/simonw/datasette/issues/2154,1696618784,IC_kwDOBm6k_c5lIFkg,9599,simonw,2023-08-29T00:58:34Z,2023-08-29T00:58:34Z,OWNER,Should this have `implies_can_view=True` too? Probably: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L20-L22,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/pull/2160#issuecomment-1695754277,https://api.github.com/repos/simonw/datasette/issues/2160,1695754277,IC_kwDOBm6k_c5lEygl,22429695,codecov[bot],2023-08-28T14:00:17Z,2023-08-29T00:32:52Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`c81b024`)](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82%.

> :exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2160   +/-   ##
=======================================
  Coverage   92.82%   92.82%           
=======================================
  Files          40       40           
  Lines        5948     5948           
=======================================
  Hits         5521     5521           
  Misses        427      427           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2160#issuecomment-1696595326,https://api.github.com/repos/simonw/datasette/issues/2160,1696595326,IC_kwDOBm6k_c5lH_1-,9599,simonw,2023-08-29T00:20:08Z,2023-08-29T00:20:08Z,OWNER,Cog failed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2160#issuecomment-1696592763,https://api.github.com/repos/simonw/datasette/issues/2160,1696592763,IC_kwDOBm6k_c5lH_N7,9599,simonw,2023-08-29T00:16:38Z,2023-08-29T00:16:38Z,OWNER,"Since this bumps Sphinx I'm manually reviewing the ReadTheDocs preview a bit, looks good to me: https://datasette--2160.org.readthedocs.build/en/2160/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2148#issuecomment-1696591957,https://api.github.com/repos/simonw/datasette/issues/2148,1696591957,IC_kwDOBm6k_c5lH_BV,49699333,dependabot[bot],2023-08-29T00:15:29Z,2023-08-29T00:15:29Z,CONTRIBUTOR,This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2148#issuecomment-1696591943,https://api.github.com/repos/simonw/datasette/issues/2148,1696591943,IC_kwDOBm6k_c5lH_BH,9599,simonw,2023-08-29T00:15:27Z,2023-08-29T00:15:27Z,OWNER,"Now solving this here:
- #2160 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2160#issuecomment-1696586767,https://api.github.com/repos/simonw/datasette/issues/2160,1696586767,IC_kwDOBm6k_c5lH9wP,9599,simonw,2023-08-29T00:08:26Z,2023-08-29T00:08:35Z,OWNER,"https://github.com/simonw/datasette/issues/2148#issuecomment-1689177556

> Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2160#issuecomment-1696586213,https://api.github.com/repos/simonw/datasette/issues/2160,1696586213,IC_kwDOBm6k_c5lH9nl,9599,simonw,2023-08-29T00:07:46Z,2023-08-29T00:07:46Z,OWNER,"I figured out why this was failing in:
- https://github.com/simonw/datasette/pull/2148

It's because Sphinx dropped support for Python 3.8.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/issues/2102#issuecomment-1696378239,https://api.github.com/repos/simonw/datasette/issues/2102,1696378239,IC_kwDOBm6k_c5lHK1_,9599,simonw,2023-08-28T20:38:01Z,2023-08-28T20:38:01Z,OWNER,"I want to test ""for this set of restrictions, does a GET/POST to this path return 200 or 403""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1696361304,https://api.github.com/repos/simonw/datasette/issues/2102,1696361304,IC_kwDOBm6k_c5lHGtY,9599,simonw,2023-08-28T20:23:47Z,2023-08-28T20:24:35Z,OWNER,"Here's an existing relevant test:

https://github.com/simonw/datasette/blob/2e2825869fc2655b5fcadc743f6f9dec7a49bc65/tests/test_permissions.py#L616-L666

It's not quite right for this new set of tests though, since they need to be exercising actual endpoints (`/.json` etc) in order to check that this works correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2153#issuecomment-1696355634,https://api.github.com/repos/simonw/datasette/issues/2153,1696355634,IC_kwDOBm6k_c5lHFUy,9599,simonw,2023-08-28T20:19:15Z,2023-08-28T20:19:15Z,OWNER,"Documentation:
- https://docs.datasette.io/en/latest/cli-reference.html#datasette-get
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option,
https://github.com/simonw/datasette/pull/2161#issuecomment-1696267473,https://api.github.com/repos/simonw/datasette/issues/2161,1696267473,IC_kwDOBm6k_c5lGvzR,22429695,codecov[bot],2023-08-28T19:38:28Z,2023-08-28T19:39:33Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`75.00%`** and project coverage change: **`+0.05%`** :tada:
> Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`a5cbf80`)](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.81%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2161      +/-   ##
==========================================
+ Coverage   92.76%   92.81%   +0.05%     
==========================================
  Files          40       40              
  Lines        5943     5944       +1     
==========================================
+ Hits         5513     5517       +4     
+ Misses        430      427       -3     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.37% <75.00%> (+1.02%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870345352,"-s/--setting x y gets merged into datasette.yml, refs #2143, #2156",
https://github.com/simonw/datasette/issues/2156#issuecomment-1696260446,https://api.github.com/repos/simonw/datasette/issues/2156,1696260446,IC_kwDOBm6k_c5lGuFe,9599,simonw,2023-08-28T19:33:05Z,2023-08-28T19:33:05Z,OWNER,"Now a PR:
- #2161 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options,
https://github.com/simonw/datasette/pull/2152#issuecomment-1695736691,https://api.github.com/repos/simonw/datasette/issues/2152,1695736691,IC_kwDOBm6k_c5lEuNz,49699333,dependabot[bot],2023-08-28T13:49:35Z,2023-08-28T13:49:35Z,CONTRIBUTOR,Superseded by #2160.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,Bump the python-packages group with 3 updates,
https://github.com/simonw/sqlite-utils/issues/588#issuecomment-1694823972,https://api.github.com/repos/simonw/sqlite-utils/issues/588,1694823972,IC_kwDOCGYnMM5lBPYk,9599,simonw,2023-08-28T00:41:54Z,2023-08-28T00:41:54Z,OWNER,Tips on typing `**kwargs`: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1868713944,`table.get(column=value)` option for retrieving things not by their primary key,
https://github.com/simonw/datasette/pull/2152#issuecomment-1691710474,https://api.github.com/repos/simonw/datasette/issues/2152,1691710474,IC_kwDOBm6k_c5k1XQK,22429695,codecov[bot],2023-08-24T13:45:21Z,2023-08-25T13:18:34Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`2e45686`)](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76%.

> :exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2152   +/-   ##
=======================================
  Coverage   92.76%   92.76%           
=======================================
  Files          40       40           
  Lines        5943     5919   -24     
=======================================
- Hits         5513     5491   -22     
+ Misses        430      428    -2     
```


[see 3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2152/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,Bump the python-packages group with 3 updates,
https://github.com/simonw/datasette/issues/950#issuecomment-1692494455,https://api.github.com/repos/simonw/datasette/issues/950,1692494455,IC_kwDOBm6k_c5k4Wp3,9599,simonw,2023-08-24T22:26:08Z,2023-08-24T22:26:08Z,OWNER,"Closing this issue in favour of this one:
- #2157 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/2157#issuecomment-1692465763,https://api.github.com/repos/simonw/datasette/issues/2157,1692465763,IC_kwDOBm6k_c5k4Ppj,9599,simonw,2023-08-24T21:54:29Z,2023-08-24T21:54:29Z,OWNER,"But yes, I'm a big +1 on this whole plan.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,"Proposal: Make the `_internal` database persistent, customizable, and hidden",
https://github.com/simonw/datasette/issues/2157#issuecomment-1692465334,https://api.github.com/repos/simonw/datasette/issues/2157,1692465334,IC_kwDOBm6k_c5k4Pi2,9599,simonw,2023-08-24T21:54:09Z,2023-08-24T21:54:09Z,OWNER,"We discussed this in-person this morning and these notes reflect what we talked about perfectly.

I've had so many bugs with plugins that I've written myself that have forgotten to special-case the `_internal` database when looping through `datasette.databases.keys()` - removing it from there entirely would help a lot.

Just one tiny disagreement: for `datasette-comments` I think having it store things in `_internal` could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful.

Show me all rows in `foia_requests` with at least one associated comment in `datasette_comments.comments` kind of tihng.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,"Proposal: Make the `_internal` database persistent, customizable, and hidden",
https://github.com/simonw/datasette/issues/1241#issuecomment-1692322342,https://api.github.com/repos/simonw/datasette/issues/1241,1692322342,IC_kwDOBm6k_c5k3som,52261150,publicmatt,2023-08-24T19:56:15Z,2023-08-24T20:09:52Z,NONE,"Something to think about, but I hate how long the url is when sharing a custom SQL query. Would it be possible to hash the query and state of a page instead so the url is more manageable? The mapping from hash to query would have to be stored in order to recover/lookup the page after sharing.

It's not uncommon to have things like this currently:

```https://global-power-plants.datasettes.com/global-power-plants?sql=select+rowid%2C+country%2C+country_long%2C+name%2C+gppd_idnr%2C+capacity_mw%2C+latitude%2C+longitude%2C+primary_fuel%2C+other_fuel1%2C+other_fuel2%2C+other_fuel3%2C+commissioning_year%2C+owner%2C+source%2C+url%2C+geolocation_source%2C+wepp_id%2C+year_of_capacity_data%2C+generation_gwh_2013%2C+generation_gwh_2014%2C+generation_gwh_2015%2C+generation_gwh_2016%2C+generation_gwh_2017%2C+generation_gwh_2018%2C+generation_gwh_2019%2C+generation_data_source%2C+estimated_generation_gwh_2013%2C+estimated_generation_gwh_2014%2C+estimated_generation_gwh_2015%2C+estimated_generation_gwh_2016%2C+estimated_generation_gwh_2017%2C+estimated_generation_note_2013%2C+estimated_generation_note_2014%2C+estimated_generation_note_2015%2C+estimated_generation_note_2016%2C+estimated_generation_note_2017+from+%5Bglobal-power-plants%5D+order+by+rowid+limit+101```

I'm thinking a plugin like [https://datasette.io/plugins/datasette-query-files](https://datasette.io/plugins/datasette-query-files), but could be created and managed from the UI (with the right permissions).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",814595021,Share button for copying current URL,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692210044,https://api.github.com/repos/simonw/datasette/issues/2143,1692210044,IC_kwDOBm6k_c5k3RN8,9599,simonw,2023-08-24T18:28:27Z,2023-08-24T18:28:27Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360

Looks to me like that second bit of code doesn't yet handle `datasette.yml`

This code does though:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335

`parse_metadata()` is clearly a bad name for this function:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L980-L990

That ` @documented` decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content

So we should rename it to something better like `parse_json_or_yaml()` but keep `parse_metadata` as an undocumented alias for that to avoid any unnecessary plugin breaks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2156#issuecomment-1692206200,https://api.github.com/repos/simonw/datasette/issues/2156,1692206200,IC_kwDOBm6k_c5k3QR4,9599,simonw,2023-08-24T18:25:23Z,2023-08-24T18:25:23Z,OWNER,"Ran out of time for this, I'll look at the next step next week.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options,
https://github.com/simonw/datasette/issues/2156#issuecomment-1692201647,https://api.github.com/repos/simonw/datasette/issues/2156,1692201647,IC_kwDOBm6k_c5k3PKv,9599,simonw,2023-08-24T18:21:53Z,2023-08-24T18:21:53Z,OWNER,"Oops, that was meant to be a PR. It's just a utility function though so it's safe to land already. I'll do a PR for the actual integration of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options,
https://github.com/simonw/datasette/issues/2156#issuecomment-1692186522,https://api.github.com/repos/simonw/datasette/issues/2156,1692186522,IC_kwDOBm6k_c5k3Lea,9599,simonw,2023-08-24T18:10:04Z,2023-08-24T18:10:04Z,OWNER,I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692182910,https://api.github.com/repos/simonw/datasette/issues/2143,1692182910,IC_kwDOBm6k_c5k3Kl-,9599,simonw,2023-08-24T18:06:57Z,2023-08-24T18:08:17Z,OWNER,"The other thing that could work is something like this:
```bash
export AUTH_TOKENS_DB=""tokens""
datasette \
  -s settings.sql_time_limit_ms 1000 \
  -s plugins.datasette-auth-tokens.manage_tokens true \
  -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB
```
So `-e` is an alternative version of `-s` which reads from the named environment variable instead of having the value provided directly as the second value in the pair.

I quite like this, because it could replace the really ugly `$ENV` pattern we have in plugin configuration at the moment:  https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values
```yaml
plugins:
  datasette-auth-github:
    client_secret:
      $env: GITHUB_CLIENT_SECRET
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1692180683,https://api.github.com/repos/simonw/datasette/issues/2143,1692180683,IC_kwDOBm6k_c5k3KDL,9599,simonw,2023-08-24T18:05:17Z,2023-08-24T18:05:17Z,OWNER,"That's a really good call, thanks @rclement - environment variable configuration totally makes sense here.

Need to figure out the right syntax for that. Something like this perhaps:

```bash
DATASETTE_CONFIG_PLUGINS='{""datasette-ripgrep"": ...}'
```
Hard to know how to make this nestable though. I considered this:
```bash
DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/'
```
But that doesn't work, because how does the processing code know that it should split on `_` for most of the tokens but NOT split `DATASETTE_RIPGREP`, instead treating that as `datasette-ripgrep`?

I checked and `-` is not a valid character in an environment variable, at least in zsh on macOS:
```
% export FOO_BAR-BAZ=1
export: not valid in this context: FOO_BAR-BAZ
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/pull/2154#issuecomment-1691845306,https://api.github.com/repos/simonw/datasette/issues/2154,1691845306,IC_kwDOBm6k_c5k14K6,9599,simonw,2023-08-24T14:57:39Z,2023-08-24T14:57:39Z,OWNER,"Notes on manual testing so far - it looks like this might be working!
- https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691842259,https://api.github.com/repos/simonw/datasette/issues/2102,1691842259,IC_kwDOBm6k_c5k13bT,9599,simonw,2023-08-24T14:55:54Z,2023-08-24T14:55:54Z,OWNER,"So what's needed to finish this is:
- Tests that demonstrate that nothing is revealed that shouldn't be by tokens restricted in this way
- Similar tests for other permissions like `create-table` that check that they work (and don't also need `view-instance` etc).
- Documentation","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713,https://api.github.com/repos/simonw/datasette/issues/2102,1691824713,IC_kwDOBm6k_c5k1zJJ,9599,simonw,2023-08-24T14:45:49Z,2023-08-24T14:45:49Z,OWNER,"I tested this out against a Datasette Cloud instance. I created a restricted token and tested it like this:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/-/actor.json' | jq
```
```json
{
  ""actor"": {
    ""id"": ""245"",
    ""token"": ""dsatok"",
    ""token_id"": 2,
    ""_r"": {
      ""r"": {
        ""data"": {
          ""all_stocks"": [
            ""vt""
          ]
        }
      }
    }
  }
}
```
It can access the `all_stocks` demo table:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/data/all_stocks.json?_size=1' | jq
```
```json
{
  ""ok"": true,
  ""next"": ""1"",
  ""rows"": [
    {
      ""rowid"": 1,
      ""Date"": ""2013-01-02"",
      ""Open"": 79.12,
      ""High"": 79.29,
      ""Low"": 77.38,
      ""Close"": 78.43,
      ""Volume"": 140124866,
      ""Name"": ""AAPL""
    }
  ],
  ""truncated"": false
}
```
Accessing the database returns just information about that table, even though other tables exist:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/data.json?_size=1'
```
```json
{
  ""database"": ""data"",
  ""private"": true,
  ""path"": ""/data"",
  ""size"": 3796992,
  ""tables"": [
    {
      ""name"": ""all_stocks"",
      ""columns"": [
        ""Date"",
        ""Open"",
        ""High"",
        ""Low"",
        ""Close"",
        ""Volume"",
        ""Name""
      ],
      ""primary_keys"": [],
      ""count"": 8813,
      ""hidden"": false,
      ""fts_table"": null,
      ""foreign_keys"": {
        ""incoming"": [],
        ""outgoing"": []
      },
      ""private"": true
    }
  ],
  ""hidden_count"": 0,
  ""views"": [],
  ""queries"": [],
  ""allow_execute_sql"": false,
  ""table_columns"": {}
}
```
And hitting the top-level `/.json` thing does the same - it reveals that table but not any of the other tables or databases:
```bash
curl -H ""Authorization: Bearer $TOKEN"" \
  'https://$INSTANCE/.json?_size=1'
```
```json
{
  ""data"": {
    ""name"": ""data"",
    ""hash"": null,
    ""color"": ""8d777f"",
    ""path"": ""/data"",
    ""tables_and_views_truncated"": [
      {
        ""name"": ""all_stocks"",
        ""columns"": [
          ""Date"",
          ""Open"",
          ""High"",
          ""Low"",
          ""Close"",
          ""Volume"",
          ""Name""
        ],
        ""primary_keys"": [],
        ""count"": null,
        ""hidden"": false,
        ""fts_table"": null,
        ""num_relationships_for_sorting"": 0,
        ""private"": false
      }
    ],
    ""tables_and_views_more"": false,
    ""tables_count"": 1,
    ""table_rows_sum"": 0,
    ""show_table_row_counts"": false,
    ""hidden_table_rows_sum"": 0,
    ""hidden_tables_count"": 0,
    ""views_count"": 0,
    ""private"": false
  }
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/pull/2154#issuecomment-1691788400,https://api.github.com/repos/simonw/datasette/issues/2154,1691788400,IC_kwDOBm6k_c5k1qRw,9599,simonw,2023-08-24T14:25:56Z,2023-08-24T14:25:56Z,OWNER,"Can be tested with:
```bash
pip install https://github.com/simonw/datasette/archive/6d57a8c23043e99b27f7a2afbe58f4d58815fd51.zip
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691779180,https://api.github.com/repos/simonw/datasette/issues/2153,1691779180,IC_kwDOBm6k_c5k1oBs,9599,simonw,2023-08-24T14:21:03Z,2023-08-24T14:21:03Z,OWNER,"`datasette serve` currently only has a `--get` - for this to be really useful it needs to grow `--post` and maybe other verbs too.

Which is a good argument for moving this functionality to `datasette client get ...` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option,
https://github.com/simonw/datasette/pull/2152#issuecomment-1691767797,https://api.github.com/repos/simonw/datasette/issues/2152,1691767797,IC_kwDOBm6k_c5k1lP1,9599,simonw,2023-08-24T14:15:10Z,2023-08-24T14:15:10Z,OWNER,This is broken because Sphinx no longer supports Python 3.8.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,Bump the python-packages group with 3 updates,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691763427,https://api.github.com/repos/simonw/datasette/issues/2153,1691763427,IC_kwDOBm6k_c5k1kLj,9599,simonw,2023-08-24T14:12:43Z,2023-08-24T14:12:43Z,OWNER,Annoying that `datasette client ...` makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing `datasette.client` Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691761685,https://api.github.com/repos/simonw/datasette/issues/2153,1691761685,IC_kwDOBm6k_c5k1jwV,9599,simonw,2023-08-24T14:11:41Z,2023-08-24T14:11:41Z,OWNER,"Another option: implement this as a plugin, providing a new command like `datasette get ...`

Or implement `datasette client get ...` as core commands or a plugin - except that clashes with the `datasette client` command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691758168,https://api.github.com/repos/simonw/datasette/issues/2102,1691758168,IC_kwDOBm6k_c5k1i5Y,9599,simonw,2023-08-24T14:09:45Z,2023-08-24T14:09:45Z,OWNER,I'm going to implement this in a branch to make it easier to test out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2153#issuecomment-1691753489,https://api.github.com/repos/simonw/datasette/issues/2153,1691753489,IC_kwDOBm6k_c5k1hwR,9599,simonw,2023-08-24T14:07:25Z,2023-08-24T14:09:16Z,OWNER,"Building that `""_r""` array is the main reason this would be useful, but it's also fiddly to get right.

`datasette create-token` has a design for that already: https://docs.datasette.io/en/1.0a4/authentication.html#datasette-create-token
```
datasette create-token root \
    --secret mysecret \
    --all view-instance \
    --all view-table \
    --database docs view-query \
    --resource docs documents insert-row \
    --resource docs documents update-row
```
Adding imitations of those options (excluding `--secret`, not needed here) to `datasette serve` would add a LOT of extra options, but it would also make it really convenient to attempt a request with a specific set of restrictions. Not sure if that would be worth the extra `--help` output or not.

I feel like the names would have to have a common prefix though. Maybe something like this:

```bash
datasette serve data.db --get `/data/mytable.json' \
  --actor-id root \
  --r-all view-instance \
  --r-database data view-query \
  --r-resource data documents update-row
```
Other options could be the longer `--restrict-all/--restrict-database/--restrict-resource`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option,
https://github.com/simonw/datasette/issues/2143#issuecomment-1691094870,https://api.github.com/repos/simonw/datasette/issues/2143,1691094870,IC_kwDOBm6k_c5kzA9W,1238873,rclement,2023-08-24T06:43:40Z,2023-08-24T06:43:40Z,NONE,"If I may, the ""path-like"" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables.

For instance:

```
datasette -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}'
```

could also be provided using:

```
export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{""foo"": [1,2,3], ""bar"": ""baz""}'
datasette
```

(I do not like mixing `-` and `_` in env vars but I do not have a best easily reversible example at the moment)

FYI, you could take some inspiration from another great open source data project, Metabase:
https://www.metabase.com/docs/latest/configuring-metabase/config-file
https://www.metabase.com/docs/latest/configuring-metabase/environment-variables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691045051,https://api.github.com/repos/simonw/datasette/issues/2102,1691045051,IC_kwDOBm6k_c5ky0y7,9599,simonw,2023-08-24T05:51:59Z,2023-08-24T05:51:59Z,OWNER,"With that fix in place, this works:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```
But this fails, because it's for a table not explicitly listed:
```bash
datasette fixtures.db --get '/fixtures/searchable.json' --actor '{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691044283,https://api.github.com/repos/simonw/datasette/issues/2102,1691044283,IC_kwDOBm6k_c5ky0m7,9599,simonw,2023-08-24T05:51:02Z,2023-08-24T05:51:02Z,OWNER,"Also need to confirm that permissions like `insert-row`, `delete-row`, `create-table` etc don't also need special cases to ensure they get through the `view-instance` etc checks, if those exist for those actions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691043475,https://api.github.com/repos/simonw/datasette/issues/2102,1691043475,IC_kwDOBm6k_c5ky0aT,9599,simonw,2023-08-24T05:50:04Z,2023-08-24T05:50:04Z,OWNER,"On first test this seems to work!

```diff
diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py
index 63a66c3c..9303dac8 100644
--- a/datasette/default_permissions.py
+++ b/datasette/default_permissions.py
@@ -187,6 +187,30 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource):
         return None
     _r = actor.get(""_r"")
 
+    # Special case for view-instance: it's allowed if there are any view-database
+    # or view-table permissions defined
+    if action == ""view-instance"":
+        database_rules = _r.get(""d"") or {}
+        for rules in database_rules.values():
+            if ""vd"" in rules or ""view-database"" in rules:
+                return None
+        # Now check resources
+        resource_rules = _r.get(""r"") or {}
+        for _database, resources in resource_rules.items():
+            for rules in resources.values():
+                if ""vt"" in rules or ""view-table"" in rules:
+                    return None
+
+    # Special case for view-database: it's allowed if there are any view-table permissions
+    # defined within that database
+    if action == ""view-database"":
+        database_name = resource
+        resource_rules = _r.get(""r"") or {}
+        resources_in_database = resource_rules.get(database_name) or {}
+        for rules in resources_in_database.values():
+            if ""vt"" in rules or ""view-table"" in rules:
+                return None
+
     # Does this action have an abbreviation?
     to_check = {action}
     permission = datasette.permissions.get(action)
```
Needs a LOT of testing to make sure what it's doing is sensible though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971,https://api.github.com/repos/simonw/datasette/issues/2102,1691037971,IC_kwDOBm6k_c5kyzET,9599,simonw,2023-08-24T05:42:47Z,2023-08-24T05:42:47Z,OWNER,"I applied a fun trick to help test this out:
```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 58f89c1c..830f47ef 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -445,6 +445,10 @@ def uninstall(packages, yes):
     ""--token"",
     help=""API token to send with --get requests"",
 )
+@click.option(
+    ""--actor"",
+    help=""Actor to use for --get requests"",
+)
 @click.option(""--version-note"", help=""Additional note to show on /-/versions"")
 @click.option(""--help-settings"", is_flag=True, help=""Show available settings"")
 @click.option(""--pdb"", is_flag=True, help=""Launch debugger on any errors"")
@@ -499,6 +503,7 @@ def serve(
     root,
     get,
     token,
+    actor,
     version_note,
     help_settings,
     pdb,
@@ -611,7 +616,10 @@ def serve(
         headers = {}
         if token:
             headers[""Authorization""] = ""Bearer {}"".format(token)
-        response = client.get(get, headers=headers)
+        cookies = {}
+        if actor:
+            cookies[""ds_actor""] = client.actor_cookie(json.loads(actor))
+        response = client.get(get, headers=headers, cookies=cookies)
         click.echo(response.text)
         exit_code = 0 if response.status == 200 else 1
         sys.exit(exit_code)
```
This adds a `--actor` option to `datasette ... --get /path` which makes it easy to test an API endpoint using a fake actor with a set of `_r` restrictions.

With that in place I can try this, with a token that has view-instance and view-database and view-table:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
    ""_r"": {
        ""a"": [
            ""vi""
        ],
        ""d"": {
            ""fixtures"": [
                ""vd""
            ]
        },
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```
Or this, with a token that just has view-table but is missing the view-database and view-instance:
```bash
datasette fixtures.db --get '/fixtures/facetable.json' --actor '{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""facetable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""user""
}'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1691036559,https://api.github.com/repos/simonw/datasette/issues/2102,1691036559,IC_kwDOBm6k_c5kyyuP,9599,simonw,2023-08-24T05:40:53Z,2023-08-24T05:40:53Z,OWNER,"There might be an easier way to solve this. Here's some permission checks that run when hitting `/fixtures/facetable.json`:

```
permission_allowed: action=view-table, resource=('fixtures', 'facetable'), actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

  File ""/datasette/views/table.py"", line 727, in table_view_traced
    view_data = await table_view_data(

  File ""/datasette/views/table.py"", line 875, in table_view_data
    visible, private = await datasette.check_visibility(

  File ""/datasette/app.py"", line 890, in check_visibility
    await self.ensure_permissions(actor, permissions)

permission_allowed: action=view-database, resource=fixtures, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

  File ""/datasette/views/table.py"", line 727, in table_view_traced
    view_data = await table_view_data(

  File ""/datasette/views/table.py"", line 875, in table_view_data
    visible, private = await datasette.check_visibility(

  File ""/datasette/app.py"", line 890, in check_visibility
    await self.ensure_permissions(actor, permissions)

permission_allowed: action=view-instance, resource=<None>, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'}

  File ""/datasette/views/table.py"", line 727, in table_view_traced
    view_data = await table_view_data(

  File ""/datasette/views/table.py"", line 875, in table_view_data
    visible, private = await datasette.check_visibility(

  File ""/datasette/app.py"", line 890, in check_visibility
    await self.ensure_permissions(actor, permissions)
```
That's with a token that has the view instance, view database and view table permissions required.

But... what if the restrictions logic said that if you have view-table you automatically also get view-database and view-instance?

Would that actually let people do anything they shouldn't be able to do? I don't think it would even let them see a list of tables that they weren't allowed to visit, so it might be OK.

I'll try that and see how it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2147#issuecomment-1690955706,https://api.github.com/repos/simonw/datasette/issues/2147,1690955706,IC_kwDOBm6k_c5kye-6,18899,jackowayed,2023-08-24T03:54:35Z,2023-08-24T03:54:35Z,NONE,"That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries.

But without a real motivating example, I see why you don't want to add that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119,https://api.github.com/repos/simonw/datasette/issues/2143,1690800119,IC_kwDOBm6k_c5kx4_3,9599,simonw,2023-08-24T00:10:32Z,2023-08-24T00:39:00Z,OWNER,"Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin):

```bash
datasette data.db \
  -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}'
```
Which would be equivalent to:
```yaml
plugins:
  datasette-complex-plugin:
    configs:
      foo:
        - 1
        - 2
        - 3
      bar: baz
```
This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex.

That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again.

For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641,https://api.github.com/repos/simonw/datasette/issues/2143,1690800641,IC_kwDOBm6k_c5kx5IB,9599,simonw,2023-08-24T00:11:16Z,2023-08-24T00:11:16Z,OWNER,"> @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.

That's a neat example thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690799608,https://api.github.com/repos/simonw/datasette/issues/2143,1690799608,IC_kwDOBm6k_c5kx434,77071,pkulchenko,2023-08-24T00:09:47Z,2023-08-24T00:10:41Z,NONE,"@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514,https://api.github.com/repos/simonw/datasette/issues/2143,1690792514,IC_kwDOBm6k_c5kx3JC,9599,simonw,2023-08-24T00:00:16Z,2023-08-24T00:02:55Z,OWNER,"I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax.

Here's what I've come up with:
```
datasette \
  -s settings.sql_time_limit_ms 1000 \
  -s plugins.datasette-auth-tokens.manage_tokens true \
  -s plugins.datasette-auth-tokens.manage_tokens_database tokens \
  -s plugins.datasette-ripgrep.path ""/home/simon/code-to-search"" \
  -s databases.mydatabase.tables.example_table.sort created \
  mydatabase.db tokens.db
```
Which would be equivalent to `datasette.yml` containing this:
```yaml
plugins:
  datasette-auth-tokens:
    manage_tokens: true
    manage_tokens_database: tokens
  datasette-ripgrep:
    path: /home/simon/code-to-search
databases:
  mydatabase:
    tables:
      example_table:
        sort: created
settings:
  sql_time_limit_ms: 1000
```
Here's a prototype implementation of this:
```python
import json
from typing import Any, List, Tuple

def _handle_pair(key: str, value: str) -> dict:
    """"""
    Turn a key-value pair into a nested dictionary.
    foo, bar => {'foo': 'bar'}
    foo.bar, baz => {'foo': {'bar': 'baz'}}
    foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}}
    foo.bar, ""baz"" => {'foo': {'bar': 'baz'}}
    foo.bar, '{""baz"": ""qux""}' => {'foo': {'bar': ""{'baz': 'qux'}""}}
    """"""
    try:
        value = json.loads(value)
    except json.JSONDecodeError:
        # If it doesn't parse as JSON, treat it as a string
        pass

    keys = key.split('.')
    result = current_dict = {}
    
    for k in keys[:-1]:
        current_dict[k] = {}
        current_dict = current_dict[k]
    
    current_dict[keys[-1]] = value
    return result


def _combine(base: dict, update: dict) -> dict:
    """"""
    Recursively merge two dictionaries.
    """"""
    for key, value in update.items():
        if isinstance(value, dict) and key in base and isinstance(base[key], dict):
            base[key] = _combine(base[key], value)
        else:
            base[key] = value
    return base

def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict:
    """"""
    Parse a list of key-value pairs into a nested dictionary.
    """"""
    result = {}
    for key, value in pairs:
        parsed_pair = _handle_pair(key, value)
        result = _combine(result, parsed_pair)
    return result
```
Exercised like this:
```python
print(json.dumps(handle_pairs([
    (""settings.sql_time_limit_ms"", ""1000""),
    (""plugins.datasette-auth-tokens.manage_tokens"", ""true""),
    (""plugins.datasette-auth-tokens.manage_tokens_database"", ""tokens""),
    (""plugins.datasette-ripgrep.path"", ""/home/simon/code-to-search""),
    (""databases.mydatabase.tables.example_table.sort"", ""created""),
]), indent=4))
```
Output:
```json
{
    ""settings"": {
        ""sql_time_limit_ms"": 1000
    },
    ""plugins"": {
        ""datasette-auth-tokens"": {
            ""manage_tokens"": true,
            ""manage_tokens_database"": ""tokens""
        },
        ""datasette-ripgrep"": {
            ""path"": ""/home/simon/code-to-search""
        }
    },
    ""databases"": {
        ""mydatabase"": {
            ""tables"": {
                ""example_table"": {
                    ""sort"": ""created""
                }
            }
        }
    }
}
```
Note that `-s` isn't currently an option for `datasette serve`.

`--setting key value` IS an existing option, but it isn't completely compatible with this because it maps directly just to settings.

Although... we could keep compatibility by saying that if you call `--setting known_setting value` and that `known_setting` is in this list then we treat it as if you said `-s settings.known_setting value` instead:

https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1690787394,https://api.github.com/repos/simonw/datasette/issues/2143,1690787394,IC_kwDOBm6k_c5kx15C,9599,simonw,2023-08-23T23:52:02Z,2023-08-23T23:52:02Z,OWNER,"> This also makes it simple to separate out secrets.
> 
> `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml`

Having multiple configs that combine in that way is a really interesting direction.

> To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight.

I'm very keen on separating out the ""metadata"" - where metadata is the slimmest possible set of things, effectively the data license and the source and the column and table descriptions - from everything else, mainly because I want metadata to be able to travel with the data.

One idea that's been discussed before is having an optional mechanism for storing metadata in the SQLite database file itself - potentially in a `_datasette_metadata` table. That way you could distribute a DB file and anyone who opened it in Datasette would also see the correct metadata about it.

That's why I'm so keen on splitting out metadata from all of the other stuff - settings and plugin configuration and authentication rules.

So really it becomes ""true metadata"" v.s. ""all of the other junk that's accumulated in metadata and `settings.json`"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/pull/2151#issuecomment-1690746371,https://api.github.com/repos/simonw/datasette/issues/2151,1690746371,IC_kwDOBm6k_c5kxr4D,22429695,codecov[bot],2023-08-23T22:51:57Z,2023-08-23T22:58:13Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`bdf59eb`)](https://app.codecov.io/gh/simonw/datasette/commit/bdf59eb7db42559e538a637bacfe86d39e5d17ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73% compared to head [(`b895cd2`)](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2151   +/-   ##
=======================================
  Coverage   92.73%   92.73%           
=======================================
  Files          40       40           
  Lines        5919     5919           
=======================================
  Hits         5489     5489           
  Misses        430      430           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1864112887,Test Datasette on multiple SQLite versions,
https://github.com/simonw/datasette/issues/2102#issuecomment-1690705243,https://api.github.com/repos/simonw/datasette/issues/2102,1690705243,IC_kwDOBm6k_c5kxh1b,9599,simonw,2023-08-23T22:03:54Z,2023-08-23T22:03:54Z,OWNER,Idea: `datasette-permissions-debug` plugin which simply prints out a stacktrace for every permission check so you can see where in the code they are.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1690703764,https://api.github.com/repos/simonw/datasette/issues/2102,1690703764,IC_kwDOBm6k_c5kxheU,9599,simonw,2023-08-23T22:02:14Z,2023-08-23T22:02:14Z,OWNER,"Built this new test:
```python
@pytest.mark.asyncio
async def test_view_table_token_can_access_table(perms_ds):
    actor = {
        ""id"": ""restricted-token"",
        ""token"": ""dstok"",
        # Restricted to just view-table on perms_ds_two/t1
        ""_r"": {""r"": {""perms_ds_two"": {""t1"": [""vt""]}}},
    }
    cookies = {""ds_actor"": perms_ds.client.actor_cookie(actor)}
    response = await perms_ds.client.get(""/perms_ds_two/t1.json"", cookies=cookies)
    assert response.status_code == 200
```
The test fails. Running it with `pytest --pdb` let me do this:
```
(Pdb) from pprint import pprint
(Pdb) pprint(perms_ds._permission_checks)
deque([{'action': 'view-table',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': ('perms_ds_two', 't1'),
        'result': None,
        'used_default': True,
        'when': '2023-08-23T21:59:45.117155'},
       {'action': 'view-database',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': 'perms_ds_two',
        'result': False,
        'used_default': False,
        'when': '2023-08-23T21:59:45.117189'},
       {'action': 'view-instance',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': None,
        'result': False,
        'used_default': False,
        'when': '2023-08-23T21:59:45.126751'},
       {'action': 'debug-menu',
        'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}},
                  'id': 'restricted-token',
                  'token': 'dstok'},
        'resource': None,
        'result': False,
        'used_default': False,
        'when': '2023-08-23T21:59:45.126777'}],
      maxlen=200)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1690693830,https://api.github.com/repos/simonw/datasette/issues/2102,1690693830,IC_kwDOBm6k_c5kxfDG,9599,simonw,2023-08-23T21:51:52Z,2023-08-23T21:52:58Z,OWNER,"This is the hook in question: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/hookspecs.py#L108-L110

- `True` means they are allowed to access it. You only need a single`True` from a plugin to allow it.
- `False` means they are not, and just one `False` from a plugin will deny it (even if another one returned `True` I think)
- `None` means that the plugin has no opinion on this question.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/950#issuecomment-680262437,https://api.github.com/repos/simonw/datasette/issues/950,680262437,MDEyOklzc3VlQ29tbWVudDY4MDI2MjQzNw==,9599,simonw,2020-08-25T20:49:24Z,2023-08-23T21:34:09Z,OWNER,"The alternative to this would be to use regular databases and control access to them using [Authentication and permissions](https://docs.datasette.io/en/stable/authentication.html). My concern there is that it's just too easy for someone to mess up their configuration, which would be really bad. I like the idea of a much stronger defense mechanism specifically designed for secrets that should not be exposed.

Outside of secrets, passwords and tokens this mechanism could also be useful for the use-case of using Datasette to power websites - as seen on https://www.niche-museums.com/ and https://www.rockybeaches.com/ - maybe those sites don't want to expose their data through their API but still want to use `datasette-template-sql` and the `graphql()` template tag in `datasette-graphql` to render data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511,Private/secret databases: database files that are only visible to plugins,
https://github.com/simonw/datasette/issues/2150#issuecomment-1690438270,https://api.github.com/repos/simonw/datasette/issues/2150,1690438270,IC_kwDOBm6k_c5kwgp-,9599,simonw,2023-08-23T18:27:47Z,2023-08-23T18:27:47Z,OWNER,"I added `outline: 3px dotted pink` to that `form label` block to help spot where else it's being used.

Oh interesting, looks like it's over-ridden here too:

<img width=""950"" alt=""CleanShot 2023-08-23 at 11 27 15@2x"" src=""https://github.com/simonw/datasette/assets/9599/f3f3e995-2d6c-4afe-8370-01e0b5141a64"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default,
https://github.com/simonw/datasette/issues/2150#issuecomment-1690435866,https://api.github.com/repos/simonw/datasette/issues/2150,1690435866,IC_kwDOBm6k_c5kwgEa,9599,simonw,2023-08-23T18:25:51Z,2023-08-23T18:25:51Z,OWNER,"Looks like that affects a few forms:

<img width=""701"" alt=""CleanShot 2023-08-23 at 11 24 46@2x"" src=""https://github.com/simonw/datasette/assets/9599/7145aec1-832e-4eea-84a6-a735df4dfd6e"">

The search form on the table page over-rides it already:

<img width=""1243"" alt=""CleanShot 2023-08-23 at 11 25 26@2x"" src=""https://github.com/simonw/datasette/assets/9599/e002089e-e992-4cc5-bb6b-c54cf614ec95"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default,
https://github.com/simonw/datasette/issues/2150#issuecomment-1690432928,https://api.github.com/repos/simonw/datasette/issues/2150,1690432928,IC_kwDOBm6k_c5kwfWg,9599,simonw,2023-08-23T18:23:26Z,2023-08-23T18:23:26Z,OWNER,"That should be scoped just to the query filters form on the table page.

I'll fix it in `main` but I'm still going to ship a fix for those plugins separately so they work well before people upgrade to the next Datasette release with this change in it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default,
https://github.com/simonw/datasette/issues/2150#issuecomment-1690431509,https://api.github.com/repos/simonw/datasette/issues/2150,1690431509,IC_kwDOBm6k_c5kwfAV,9599,simonw,2023-08-23T18:22:47Z,2023-08-23T18:22:47Z,OWNER,https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/static/app.css#L485-L489,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default,
https://github.com/simonw/datasette/issues/2091#issuecomment-1690423878,https://api.github.com/repos/simonw/datasette/issues/2091,1690423878,IC_kwDOBm6k_c5kwdJG,9599,simonw,2023-08-23T18:18:18Z,2023-08-23T18:18:18Z,OWNER,"Dupe of:
- #2097 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7,
https://github.com/simonw/datasette/pull/2148#issuecomment-1689133247,https://api.github.com/repos/simonw/datasette/issues/2148,1689133247,IC_kwDOBm6k_c5kriC_,22429695,codecov[bot],2023-08-23T01:36:42Z,2023-08-23T03:09:43Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage has no change and project coverage change: **`+0.01%`** :tada:
> Comparison is base [(`17ec309`)](https://app.codecov.io/gh/simonw/datasette/commit/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.71% compared to head [(`a651d3c`)](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%.
> Report is 2 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2148      +/-   ##
==========================================
+ Coverage   92.71%   92.73%   +0.01%     
==========================================
  Files          40       40              
  Lines        5919     5919              
==========================================
+ Hits         5488     5489       +1     
+ Misses        431      430       -1     
```


[see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2148/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/issues/2123#issuecomment-1689207309,https://api.github.com/repos/simonw/datasette/issues/2123,1689207309,IC_kwDOBm6k_c5kr0IN,9599,simonw,2023-08-23T03:07:27Z,2023-08-23T03:07:27Z,OWNER,"> I'm happy to debug and land a patch if it's welcome.

Yes please! What an odd bug.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1825007061,datasette serve when invoked with --reload interprets the serve command as a file,
https://github.com/simonw/datasette/issues/2147#issuecomment-1689206768,https://api.github.com/repos/simonw/datasette/issues/2147,1689206768,IC_kwDOBm6k_c5krz_w,9599,simonw,2023-08-23T03:06:32Z,2023-08-23T03:06:32Z,OWNER,"I'm less convinced by the ""rewrite the query in some way"" optional idea. What kind of use-cases can you imagine for that?

My hunch is that it's much more likely to cause weird breakages than it is to allow for useful plugin extensions, but I'm willing to be convinced otherwise.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run,
https://github.com/simonw/datasette/issues/2147#issuecomment-1689206170,https://api.github.com/repos/simonw/datasette/issues/2147,1689206170,IC_kwDOBm6k_c5krz2a,9599,simonw,2023-08-23T03:05:32Z,2023-08-23T03:05:32Z,OWNER,"Interestingly enough there's actually a mechanism that looks like that a bit already: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/views/database.py#L496-L508

That `validate_sql_select()` function is defined here: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L256-L265

Against these constants:

https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L223-L253

Which isn't a million miles away from your suggestion to have a hook that can say if the query should be executed or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run,
https://github.com/simonw/datasette/pull/2148#issuecomment-1689198413,https://api.github.com/repos/simonw/datasette/issues/2148,1689198413,IC_kwDOBm6k_c5krx9N,49699333,dependabot[bot],2023-08-23T02:57:55Z,2023-08-23T02:57:55Z,CONTRIBUTOR,"Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry!

If you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2148#issuecomment-1689198368,https://api.github.com/repos/simonw/datasette/issues/2148,1689198368,IC_kwDOBm6k_c5krx8g,9599,simonw,2023-08-23T02:57:53Z,2023-08-23T02:57:53Z,OWNER,@dependabot rebase,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2148#issuecomment-1689177556,https://api.github.com/repos/simonw/datasette/issues/2148,1689177556,IC_kwDOBm6k_c5krs3U,9599,simonw,2023-08-23T02:44:29Z,2023-08-23T02:44:29Z,OWNER,"Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2148#issuecomment-1689175062,https://api.github.com/repos/simonw/datasette/issues/2148,1689175062,IC_kwDOBm6k_c5krsQW,9599,simonw,2023-08-23T02:40:46Z,2023-08-23T02:40:46Z,OWNER,"Here's why the tests are failing:

https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/.github/workflows/test.yml#L30-L46

It looks like those tests don't actually need Sphinx installed - they install `pip install -e '.[test,docs]'` to get the other docs dependencies:

https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/setup.py#L70-L80","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2148#issuecomment-1689173748,https://api.github.com/repos/simonw/datasette/issues/2148,1689173748,IC_kwDOBm6k_c5krr70,9599,simonw,2023-08-23T02:38:31Z,2023-08-23T02:38:31Z,OWNER,"Sphinx dropped support for Python 3.8 in their version 7.2.0:

- https://www.sphinx-doc.org/en/master/changes.html#release-7-2-0-released-aug-17-2023
- https://www.sphinx-doc.org/en/master/internals/release-process.html#python-version-support-policy
- https://github.com/sphinx-doc/sphinx/pull/11511
- ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/issues/516#issuecomment-1689159200,https://api.github.com/repos/simonw/datasette/issues/516,1689159200,IC_kwDOBm6k_c5kroYg,9599,simonw,2023-08-23T02:15:36Z,2023-08-23T02:15:36Z,OWNER,This could play havoc with unmerged PRs. I should merge any big ones (like the JavaScript plugins work) first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459509126,Enforce import sort order with isort,
https://github.com/simonw/datasette/issues/516#issuecomment-1689158712,https://api.github.com/repos/simonw/datasette/issues/516,1689158712,IC_kwDOBm6k_c5kroQ4,9599,simonw,2023-08-23T02:14:45Z,2023-08-23T02:14:45Z,OWNER,"Thinking about this again today. Posted about it on Discord: https://discord.com/channels/823971286308356157/823971286941302908/1143729300349132933

I won't enforce it in a `pytest` test, I'll enforce it with a CI lint check that's also in the `Justfile` here instead:

https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/Justfile#L19-L23","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459509126,Enforce import sort order with isort,
https://github.com/simonw/datasette/issues/516#issuecomment-1689154837,https://api.github.com/repos/simonw/datasette/issues/516,1689154837,IC_kwDOBm6k_c5krnUV,9599,simonw,2023-08-23T02:08:33Z,2023-08-23T02:08:50Z,OWNER,Browse this commit to see the result: https://github.com/simonw/datasette/tree/59a5d336bd4336bc53103922ada4bf726f4336c9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459509126,Enforce import sort order with isort,
https://github.com/simonw/datasette/issues/516#issuecomment-1689153446,https://api.github.com/repos/simonw/datasette/issues/516,1689153446,IC_kwDOBm6k_c5krm-m,9599,simonw,2023-08-23T02:06:35Z,2023-08-23T02:06:35Z,OWNER,"I just tried this again today by dropping this into `.isort.cfg`:
```ini
[settings]
multi_line_output=3
include_trailing_comma=True
force_grid_wrap=0
use_parentheses=True
line_length=88
known_first_party=datasette
```
And running this in the root of the project:
```
isort .
```
It produced a huge diff, but when I ran the tests:
```bash
pytest -n auto
```
The tests all passed. I'll push a PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459509126,Enforce import sort order with isort,
https://github.com/simonw/datasette/pull/524#issuecomment-1689130061,https://api.github.com/repos/simonw/datasette/issues/524,1689130061,IC_kwDOBm6k_c5krhRN,9599,simonw,2023-08-23T01:31:08Z,2023-08-23T01:31:08Z,OWNER,This branch is WAY out of date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459689615,"Sort commits using isort, refs #516",
https://github.com/simonw/datasette/issues/493#issuecomment-1689128911,https://api.github.com/repos/simonw/datasette/issues/493,1689128911,IC_kwDOBm6k_c5krg_P,9599,simonw,2023-08-23T01:29:20Z,2023-08-23T01:29:20Z,OWNER,"It's going to be called `datasette.json` and the concept of metadata will be split out separately. See:

- #2149 ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",449886319,Rename metadata.json to config.json,
https://github.com/simonw/datasette/issues/275#issuecomment-1689128553,https://api.github.com/repos/simonw/datasette/issues/275,1689128553,IC_kwDOBm6k_c5krg5p,9599,simonw,2023-08-23T01:28:37Z,2023-08-23T01:28:37Z,OWNER,"This is obsoleted by the work happening here:
- #2093 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",324720095,"""config"" section in metadata.json (root, database and table level)",
https://github.com/simonw/datasette/pull/2148#issuecomment-1689127479,https://api.github.com/repos/simonw/datasette/issues/2148,1689127479,IC_kwDOBm6k_c5krgo3,9599,simonw,2023-08-23T01:26:53Z,2023-08-23T01:26:53Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies",
https://github.com/simonw/datasette/pull/2149#issuecomment-1689125244,https://api.github.com/repos/simonw/datasette/issues/2149,1689125244,IC_kwDOBm6k_c5krgF8,9599,simonw,2023-08-23T01:23:27Z,2023-08-23T01:23:27Z,OWNER,"This is a really great start - tests pass, implementation looks clean, the new stubbed documentation page makes sense. Let's land it in `main` and iterate on it further in future PRs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1861812208,"Start a new `datasette.yaml` configuration file, with settings support",
https://github.com/simonw/datasette/pull/2149#issuecomment-1688547401,https://api.github.com/repos/simonw/datasette/issues/2149,1688547401,IC_kwDOBm6k_c5kpTBJ,22429695,codecov[bot],2023-08-22T16:30:41Z,2023-08-22T16:30:41Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`76.47%`** and project coverage change: **`+0.10%`** :tada:
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`db720cd`)](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.17%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2149      +/-   ##
==========================================
+ Coverage   92.06%   92.17%   +0.10%     
==========================================
  Files          40       40              
  Lines        5937     5916      -21     
==========================================
- Hits         5466     5453      -13     
+ Misses        471      463       -8     
```


| [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `81.02% <25.00%> (+1.20%)` | :arrow_up: |
| [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.29% <92.30%> (-0.09%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1861812208,"Start a new `datasette.yaml` configuration file, with settings support",
https://github.com/simonw/datasette/issues/2093#issuecomment-1688532012,https://api.github.com/repos/simonw/datasette/issues/2093,1688532012,IC_kwDOBm6k_c5kpPQs,15178711,asg017,2023-08-22T16:21:40Z,2023-08-22T16:21:40Z,CONTRIBUTOR,"OK Here's the gameplan for this, which is closely tied to #2143 :

- We will add a new `datasette.json`/`datasette.yaml` configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format
- Metadata will NOT be a part of this file
- TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML
- The `settings.json` file will be deprecated, and the `--config` arg will be brought back.
- Command line arguments can still be used to overwrite values (ex `--setting` will overwrite settings in `datasette.yaml`

The format of `datasette.json` will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426

Here's the current implementation plan:

1. Add a new `--config` flag and port over `""settings""` into a new datasette.json config file, remove settings.json
2. Add top-level plugin config support to `datasette.json`
3. Figure out database/table structure of config `datasette.json`
4. Port over database/table level plugin config support `datasette.json`
5. Port over permissions/auth settings to `datasette.json`
6. Deprecate non-metadata values in `metadata.json`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2147#issuecomment-1687433388,https://api.github.com/repos/simonw/datasette/issues/2147,1687433388,IC_kwDOBm6k_c5klDCs,18899,jackowayed,2023-08-22T05:05:33Z,2023-08-22T05:05:33Z,NONE,"Thanks for all this! You're totally right that the ASGI option is doable, if a bit low level and coupled to the current URI design. I'm totally fine with that being the final answer.

process_view is interesting and in the general direction of what I had in mind.

A somewhat less powerful idea: Is there value in giving a hook for just the query that's about to be run? Maybe I'm thinking a little narrowly about this problem I decided I wanted to solve, but I could see other uses for a hook of the sketch below:

```
def prepare_query(database, table, query):
   """"""Modify query that is about to be run in some way. Return the (possibly rewritten) query to run, or None to disallow running the query""""""
```
(Maybe you actually want to return a tuple so there can be an error message when you disallow, or something.)

Maybe it's too narrowly useful and some of the other pieces of datasette obviate some of these ideas, but off the cuff I could imagine using it to:
* Require a LIMIT. Either fail the query or add the limit if it's not there.
* Do logging, like my usecase.
* Do other analysis on whether you want to allow the query to run; a linter? query complexity? 

Definitely feel free to say no, or not now. This is all me just playing around with what datasette and its plugin architecture can do with toy ideas, so don't let me push you to commit to a hook you don't feel confident fits well in the design.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run,
https://github.com/simonw/datasette/issues/2147#issuecomment-1686747420,https://api.github.com/repos/simonw/datasette/issues/2147,1686747420,IC_kwDOBm6k_c5kibkc,9599,simonw,2023-08-21T17:31:42Z,2023-08-21T17:34:19Z,OWNER,"Are you talking just about queries submitted to `/database?sql=` using the interface on https://latest.datasette.io/fixtures?sql=select+*+from+facetable or are you interested in queries that are run to power other pages like https://latest.datasette.io/fixtures/facetable as well? I'll assume the former.

There are a few ways you could solve this at the moment.

The easiest would be with a piece of ASGI middleware that looks for URLs matching `/dbname?sql=...` and logs those. I played with a version of that a few years ago: https://simonwillison.net/2019/Dec/16/logging-sqlite-asgi-middleware/ - see also https://github.com/simonw/asgi-log-to-sqlite

Then you can load that middleware from a plugin using https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette

That feels a bit delicate because it's relying on the URL design not changing, but I'm happy to confirm that URL is going to stay the same for Datasette 1.0 and I have no plans to change it ever.

There's also a tracing mechanism built into Datasette itself that you could hook into. The internals of that are documented here: https://docs.datasette.io/en/stable/internals.html#datasette-tracer - but I don't yet consider it a 100% stable API. I don't plan to change it but I won't promise not to either.

I used that mechanism in this plugin: https://datasette.io/plugins/datasette-pretty-traces - demonstrated here: https://latest-with-plugins.datasette.io/github/commits?_trace=1

The hackiest way to do this would be to patch Datasette itself and try to replace the `query_view`. This definitely isn't a documented, stable API though and would be very likely to break at arbitrary points in the future.

So my recommendation for the moment is the ASGI middleware option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run,
https://github.com/simonw/datasette/issues/2147#issuecomment-1686749342,https://api.github.com/repos/simonw/datasette/issues/2147,1686749342,IC_kwDOBm6k_c5kicCe,9599,simonw,2023-08-21T17:33:11Z,2023-08-21T17:33:11Z,OWNER,"I'm definitely open to suggestions for plugin hooks that might make this kind of thing easier.

One idea I've been mulling is whether there should be a plugin hook that files on arbitrary views - similar to Django's `process_view` mechanism: https://docs.djangoproject.com/en/4.2/topics/http/middleware/#process-view

That would allow people to setup code that runs before or after any of the default views in Datasette.

I'm not yet 100% sold on the idea, because I worry about implementing it in a way that guarantees plugins won't break on future releases. But I'm open to considering it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run,
https://github.com/simonw/datasette/issues/2145#issuecomment-1686745094,https://api.github.com/repos/simonw/datasette/issues/2145,1686745094,IC_kwDOBm6k_c5kibAG,15178711,asg017,2023-08-21T17:30:01Z,2023-08-21T17:30:01Z,CONTRIBUTOR,"Another point: The new Datasette write API should refuse to insert a row with a NULL primary key. That will likely decrease the likelihood someone find themselves with NULLs in their primary keys, at least with Datasette users. Especially buggy code that uses the write API, like our `datasette-write-ui` bug that led to this issue.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1686683596,https://api.github.com/repos/simonw/datasette/issues/2145,1686683596,IC_kwDOBm6k_c5kiL_M,9599,simonw,2023-08-21T16:49:12Z,2023-08-21T16:49:12Z,OWNER,"Suggestion from @asg017 is that we say that if your row has a null primary key you don't get a link to a row page for that row.

Which has some precedent, because our SQL view display doesn't link to row pages at all (since they don't make sense for views): https://latest.datasette.io/fixtures/simple_view","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/pull/2144#issuecomment-1686366557,https://api.github.com/repos/simonw/datasette/issues/2144,1686366557,IC_kwDOBm6k_c5kg-ld,49699333,dependabot[bot],2023-08-21T13:48:15Z,2023-08-21T13:48:15Z,CONTRIBUTOR,Superseded by #2148.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856760386,Bump the python-packages group with 3 updates,
https://github.com/simonw/datasette/issues/2145#issuecomment-1685471752,https://api.github.com/repos/simonw/datasette/issues/2145,1685471752,IC_kwDOBm6k_c5kdkII,77071,pkulchenko,2023-08-21T01:07:23Z,2023-08-21T01:07:23Z,NONE,"@simonw, since you're referencing ""rowid"" column by name, I just want to note that there may be an existing rowid column with completely different semantics (https://www.sqlite.org/lang_createtable.html#rowid), which is likely to break this logic. I don't see a good way to detect a proper ""rowid"" name short of checking if there is a field with that name and using the alternative (`_rowid_` or `oid`), which is not ideal, but may work.

In terms of the original issue, maybe a way to deal with it is to use rowid by default and then use primary key for WITHOUT ROWID tables (as they are guaranteed to be not null), but I suspect it may require significant changes to the API (and doesn't fully address the issue of what value to pass to indicate NULL when editing records). Would it make sense to generate a random string to indicate NULL values when editing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2143#issuecomment-1685263948,https://api.github.com/repos/simonw/datasette/issues/2143,1685263948,IC_kwDOBm6k_c5kcxZM,11784304,dvizard,2023-08-20T11:50:10Z,2023-08-20T11:50:10Z,NONE,"This also makes it simple to separate out secrets.

`datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml`

settings.yaml
```
settings:
  default_page_size: 10
  max_returned_rows: 3000
  sql_time_limit_ms"": 8000
plugins:
  datasette-ripgrep:
    path: /usr/local/lib/python3.11/site-packages
```

secrets.yaml
```
plugins:
  datasette-auth-github:
            client_secret: SUCH_SECRET 
```


db-docs.yaml
```
databases:
  docs:
    permissions:
      create-table:
        id: editor
```

db-fixtures.yaml
```
databases:
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1685260624,https://api.github.com/repos/simonw/datasette/issues/2143,1685260624,IC_kwDOBm6k_c5kcwlQ,11784304,dvizard,2023-08-20T11:31:16Z,2023-08-20T11:31:16Z,NONE,https://pypi.org/project/deep-chainmap/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1685260244,https://api.github.com/repos/simonw/datasette/issues/2143,1685260244,IC_kwDOBm6k_c5kcwfU,11784304,dvizard,2023-08-20T11:29:00Z,2023-08-20T11:29:00Z,NONE,https://docs.python.org/3/library/collections.html#collections.ChainMap,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1685259985,https://api.github.com/repos/simonw/datasette/issues/2143,1685259985,IC_kwDOBm6k_c5kcwbR,11784304,dvizard,2023-08-20T11:27:21Z,2023-08-20T11:27:21Z,NONE,"To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight. Even separating db/table-scope settings from pure descriptive metadata is not always easy. Like, do canned queries belong to database metadata or to settings? Do I need two separate files for this?

One pragmatic solution I used in a project is stacking yaml configuration files. Basically, have an arbitrary number of yaml or json settings files that you load in a specified order. Every file adds to the corresponding settings in the earlier-loaded file (if it already existed). I implemented this myself but found later that there is an existing Python ""cascading dict"" type of thing, I forget what it's called. There is a bit of a challenge deciding whether there is ""replacement"" or ""addition"" (I think I pragmatically ran `update` on the second level of the dict but better solutions are certainly possible). 

This way, one allows separation of settings into different blocks, while not imposing a specific idea of what belongs where that might not apply equally to all cases.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096381,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096381,IC_kwDOCGYnMM5kcIe9,9599,simonw,2023-08-19T20:04:32Z,2023-08-19T20:04:32Z,OWNER,I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference,
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096284,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096284,IC_kwDOCGYnMM5kcIdc,9599,simonw,2023-08-19T20:03:59Z,2023-08-19T20:03:59Z,OWNER,"Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in:
- #577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference,
https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096129,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096129,IC_kwDOCGYnMM5kcIbB,9599,simonw,2023-08-19T20:03:00Z,2023-08-19T20:03:00Z,OWNER,"Simplest possible recreation of the bug:
```bash
python -c '
import sqlite_utils
db = sqlite_utils.Database(memory=True)
db.execute(""""""
CREATE TABLE ""logs"" (
   [id] INTEGER PRIMARY KEY,
   [chat_id] INTEGER REFERENCES [log]([id]),
   [reply_to_id] INTEGER
);
"""""")
db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"")
'
```
That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684530060,https://api.github.com/repos/simonw/datasette/issues/2145,1684530060,IC_kwDOBm6k_c5kZ-OM,9599,simonw,2023-08-18T23:09:03Z,2023-08-18T23:09:14Z,OWNER,"Ran a quick benchmark on ChatGPT Code Interpreter: https://chat.openai.com/share/8357dc01-a97e-48ae-b35a-f06249935124

Conclusion from there is that this query returns fast no matter how much the table grows:

```sql
SELECT EXISTS(SELECT 1 FROM ""nasty"" WHERE ""id"" IS NULL)
```
So detecting if a table contains any null primary keys is definitely feasible without a performance hit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684526447,https://api.github.com/repos/simonw/datasette/issues/2145,1684526447,IC_kwDOBm6k_c5kZ9Vv,9599,simonw,2023-08-18T23:05:02Z,2023-08-18T23:05:02Z,OWNER,How expensive is it to detect if a SQLite table contains at least one `null` primary key?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684525943,https://api.github.com/repos/simonw/datasette/issues/2145,1684525943,IC_kwDOBm6k_c5kZ9N3,9599,simonw,2023-08-18T23:04:14Z,2023-08-18T23:04:14Z,OWNER,"This is hard. I tried this:
```python
def path_from_row_pks(row, pks, use_rowid, quote=True):
    """"""Generate an optionally tilde-encoded unique identifier
    for a row from its primary keys.""""""
    if use_rowid or any(row[pk] is None for pk in pks):
        bits = [row[""rowid""]]
    else:
        bits = [
            row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks
        ]
    if quote:
        bits = [tilde_encode(str(bit)) for bit in bits]
    else:
        bits = [str(bit) for bit in bits]

    return "","".join(bits)
```
The `    if use_rowid or any(row[pk] is None for pk in pks)` bit is new.

But I got this error on http://127.0.0.1:8003/nulls/nasty :

```
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 1364, in run_display_columns_and_rows
    display_columns, display_rows = await display_columns_and_rows(
  File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 186, in display_columns_and_rows
    pk_path = path_from_row_pks(row, pks, not pks, False)
  File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 124, in path_from_row_pks
    bits = [row[""rowid""]]
IndexError: No item with that key
```
Because the SQL query I ran to populate the page didn't know that it would need to select `rowid` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684525054,https://api.github.com/repos/simonw/datasette/issues/2145,1684525054,IC_kwDOBm6k_c5kZ8_-,9599,simonw,2023-08-18T23:02:26Z,2023-08-18T23:02:26Z,OWNER,"Creating a quick test database:
```bash
sqlite-utils create-table nulls.db nasty id text --pk id
sqlite-utils nulls.db 'insert into nasty (id) values (null)'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684523322,https://api.github.com/repos/simonw/datasette/issues/2145,1684523322,IC_kwDOBm6k_c5kZ8k6,9599,simonw,2023-08-18T22:59:14Z,2023-08-18T22:59:14Z,OWNER,"Except it looks like the Links from other tables section is broken:

<img width=""649"" alt=""CleanShot 2023-08-18 at 15 58 54@2x"" src=""https://github.com/simonw/datasette/assets/9599/ddebf0a7-9855-4cf0-b283-f06f8c25414d"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684522567,https://api.github.com/repos/simonw/datasette/issues/2145,1684522567,IC_kwDOBm6k_c5kZ8ZH,9599,simonw,2023-08-18T22:58:07Z,2023-08-18T22:58:07Z,OWNER,"Here's a prototype of that:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index b2644ace..acc55249 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -1386,7 +1386,7 @@ class Datasette:
         )
         add_route(
             RowView.as_view(self),
-            r""/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[^/]+?)(\.(?P<format>\w+))?$"",
+            r""/(?P<database>[^\/\.]+)/(?P<table>[^/]+?)/(?P<pks>[A-Za-z0-9\_\-\~]+|\.\d+)(\.(?P<format>\w+))?$"",
         )
         add_route(
             TableInsertView.as_view(self),
@@ -1440,7 +1440,15 @@ class Datasette:
     async def resolve_row(self, request):
         db, table_name, _ = await self.resolve_table(request)
         pk_values = urlsafe_components(request.url_vars[""pks""])
-        sql, params, pks = await row_sql_params_pks(db, table_name, pk_values)
+
+        if len(pk_values) == 1 and pk_values[0].startswith("".""):
+            # It's a special .rowid value
+            pk_values = (pk_values[0][1:],)
+            sql, params, pks = await row_sql_params_pks(
+                db, table_name, pk_values, rowid=True
+            )
+        else:
+            sql, params, pks = await row_sql_params_pks(db, table_name, pk_values)
         results = await db.execute(sql, params, truncate=True)
         row = results.first()
         if row is None:
diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py
index c388673d..96669281 100644
--- a/datasette/utils/__init__.py
+++ b/datasette/utils/__init__.py
@@ -1206,9 +1206,12 @@ def truncate_url(url, length):
     return url[: length - 1] + ""…""
 
 
-async def row_sql_params_pks(db, table, pk_values):
+async def row_sql_params_pks(db, table, pk_values, rowid=False):
     pks = await db.primary_keys(table)
-    use_rowid = not pks
+    if rowid:
+        use_rowid = True
+    else:
+        use_rowid = not pks
     select = ""*""
     if use_rowid:
         select = ""rowid, *""
```
It works:

<img width=""649"" alt=""CleanShot 2023-08-18 at 15 57 36@2x"" src=""https://github.com/simonw/datasette/assets/9599/93803f41-d194-4193-8320-7c47d0373372"">

<img width=""649"" alt=""CleanShot 2023-08-18 at 15 57 53@2x"" src=""https://github.com/simonw/datasette/assets/9599/28d28efa-5ee0-469b-a31d-ce00f2d1f77a"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684505071,https://api.github.com/repos/simonw/datasette/issues/2145,1684505071,IC_kwDOBm6k_c5kZ4Hv,9599,simonw,2023-08-18T22:44:35Z,2023-08-18T22:44:35Z,OWNER,"Also relevant:
https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L1147-L1153","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684504398,https://api.github.com/repos/simonw/datasette/issues/2145,1684504398,IC_kwDOBm6k_c5kZ39O,9599,simonw,2023-08-18T22:43:31Z,2023-08-18T22:43:46Z,OWNER,"`(?P<pks>[^/]+?)` could instead be a regex that is restricted to the tilde-encoded set of characters, or `\.\d+`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684504051,https://api.github.com/repos/simonw/datasette/issues/2145,1684504051,IC_kwDOBm6k_c5kZ33z,9599,simonw,2023-08-18T22:43:06Z,2023-08-18T22:43:06Z,OWNER,Here's the regex in question at the moment: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/app.py#L1387-L1390,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684503587,https://api.github.com/repos/simonw/datasette/issues/2145,1684503587,IC_kwDOBm6k_c5kZ3wj,9599,simonw,2023-08-18T22:42:28Z,2023-08-18T22:42:39Z,OWNER,"I could set a rule that extensions (including custom render extensions set by plugins) must not be valid integers, and teach Datasette that `/\.\d+` is the indication of a `rowid`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684503189,https://api.github.com/repos/simonw/datasette/issues/2145,1684503189,IC_kwDOBm6k_c5kZ3qV,9599,simonw,2023-08-18T22:41:51Z,2023-08-18T22:41:51Z,OWNER,"```pycon
>>> tilde_encode(""~"")
'~7E'
>>> tilde_encode(""."")
'~2E'
>>> tilde_encode(""-"")
'-'
```
I think `.` might be the way to do this:

     /database/table/.4

But... I worry about that colliding with my URL routing code that spots the difference between these:

     /database/table/.4
     /database/table/.4.json
     /database/table/.4.csv

etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684502278,https://api.github.com/repos/simonw/datasette/issues/2145,1684502278,IC_kwDOBm6k_c5kZ3cG,9599,simonw,2023-08-18T22:40:20Z,2023-08-18T22:40:20Z,OWNER,"From reviewing https://simonwillison.net/2022/Mar/19/weeknotes/

    unreserved  = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~""

That's how I chose the tilde character - but it also suggests that I could use `-` or `.` or `_` for my new `rowid` encoding.

So maybe `/database/table/_4` could indicate ""the row with `rowid` of 4"".

No, that doesn't work:
```pycon
>>> from datasette.utils import tilde_encode
>>> tilde_encode(""_"")
'_'
```
I need a character which tilde-encoding does indeed encode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684500540,https://api.github.com/repos/simonw/datasette/issues/2145,1684500540,IC_kwDOBm6k_c5kZ3A8,9599,simonw,2023-08-18T22:37:37Z,2023-08-18T22:37:37Z,OWNER,"I just found this and panicked, thinking maybe tilde encoding is a bad idea after all! https://jkorpela.fi/tilde.html

But... ""Date of last update: 1999-08-27"" - I think I'm OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684500172,https://api.github.com/repos/simonw/datasette/issues/2145,1684500172,IC_kwDOBm6k_c5kZ27M,9599,simonw,2023-08-18T22:37:04Z,2023-08-18T22:37:04Z,OWNER,"Looking at the way these URLs work: because the components themselves in `a~2Fb,~2Ec-d` are tilde-encoded, any character that's ""safe"" in tilde-encoding could be used to indicate ""this is actually a rowid"".

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684498947,https://api.github.com/repos/simonw/datasette/issues/2145,1684498947,IC_kwDOBm6k_c5kZ2oD,9599,simonw,2023-08-18T22:35:04Z,2023-08-18T22:35:04Z,OWNER,"The most interesting row URL in the fixtures database right now is this one:

https://latest.datasette.io/fixtures/compound_primary_key/a~2Fb,~2Ec-d

<img width=""163"" alt=""CleanShot 2023-08-18 at 15 34 52@2x"" src=""https://github.com/simonw/datasette/assets/9599/dd79b0b6-e1c3-4e30-b64b-973b062bfe70"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684497642,https://api.github.com/repos/simonw/datasette/issues/2145,1684497642,IC_kwDOBm6k_c5kZ2Tq,9599,simonw,2023-08-18T22:32:53Z,2023-08-18T22:32:53Z,OWNER,"Here's a potential solution: make it so ALL `rowid` tables in SQLite can be optionally addressed by their `rowid` instead of by their primary key.

Then teach the code that outputs the URL to a row page to spot if there are `null` primary keys and switch to that alternative addressing mechanism instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684497000,https://api.github.com/repos/simonw/datasette/issues/2145,1684497000,IC_kwDOBm6k_c5kZ2Jo,9599,simonw,2023-08-18T22:31:53Z,2023-08-18T22:31:53Z,OWNER,"So it sounds like SQLite does ensure that a `rowid` before it allows a primary key to be null.

So one solution here would be to detect a null primary key and switch that table over to using `rowid` URLs instead. The key problem we're trying to solve here after all is how to link to a row:

https://latest.datasette.io/fixtures/infinity/1

But when would we run that check? And does every row in the table get a new `/rowid/` URL just because someone messed up and inserted a `null` by mistake?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684496274,https://api.github.com/repos/simonw/datasette/issues/2143,1684496274,IC_kwDOBm6k_c5kZ1-S,15178711,asg017,2023-08-18T22:30:45Z,2023-08-18T22:30:45Z,CONTRIBUTOR,"> That said, I do really like a bias towards settings that can be changed at runtime

Does this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff. 

Well it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684495674,https://api.github.com/repos/simonw/datasette/issues/2145,1684495674,IC_kwDOBm6k_c5kZ106,9599,simonw,2023-08-18T22:29:47Z,2023-08-18T22:29:47Z,OWNER,"https://www.sqlite.org/lang_createtable.html#the_primary_key says:

>According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid) or the table is a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table or a [STRICT](https://www.sqlite.org/stricttables.html) table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684494464,https://api.github.com/repos/simonw/datasette/issues/2145,1684494464,IC_kwDOBm6k_c5kZ1iA,9599,simonw,2023-08-18T22:27:51Z,2023-08-18T22:28:40Z,OWNER,"Oh wow, null primary keys are bad news... SQLite lets you insert multiple rows with the same `null` value!
```pycon
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
<Table foo (id, name)>
>>> db.schema
'CREATE TABLE [foo] (\n   [id] TEXT PRIMARY KEY,\n   [name] TEXT\n);'
>>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
<Table foo (id, name)>
>>> db.schema
'CREATE TABLE [foo] (\n   [id] TEXT PRIMARY KEY,\n   [name] TEXT\n);'
>>> list(db[""foo""].rows)
[{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}]
>>> list(db.query('select * from foo where id = null'))
[]
>>> list(db.query('select * from foo where id is null'))
[{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684488526,https://api.github.com/repos/simonw/datasette/issues/2143,1684488526,IC_kwDOBm6k_c5kZ0FO,9599,simonw,2023-08-18T22:18:39Z,2023-08-18T22:18:39Z,OWNER,"> Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

> Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

I'm not a fan of that. I feel like software history is full of examples of projects that implemented configuration-as-code and then later regretted it - the most recent example is `setup.py` in Python turning into `pyproject.yaml`, but I feel like I've seen that pattern play out elsewhere too.

I don't think having people dynamically generate JSON/YAML for their configuration is a big burden. I'd have to see some very compelling use-cases to convince me otherwise.

That said, I do really like a bias towards settings that can be changed at runtime. Datasette has suffered a bit from some settings that can't be easily changed at runtime already - hence my gnarly https://github.com/simonw/datasette-remote-metadata plugin.

For things like Datasette Cloud for example the more people can configure without rebooting their container the better!

I don't think live reconfiguration at runtime is incompatible with JSON/YAML configuration though. Caddy is one of my favourite examples of software that can be entirely re-configured at runtime by POSTING a big blob of JSON to it: https://caddyserver.com/docs/quick-starts/api
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684485591,https://api.github.com/repos/simonw/datasette/issues/2143,1684485591,IC_kwDOBm6k_c5kZzXX,9599,simonw,2023-08-18T22:14:35Z,2023-08-18T22:14:35Z,OWNER,"Actually there is one thing that I'm not comfortable about with respect to the existing design: the way the database / tables stuff is nested.

They assume that the user will attach the database to Datasette using a fixed name - `docs.db` or whatever.

But what if we want to support users downloading databases from each other and attaching them to Datasette where those DBs might carry some of their own configuration?

Moving metadata into the databases makes sense there, but what about database-specific settings like the default sort order for a table, or configured canned queries?

Having those tied to the filename of the database itself feels unpleasant to me. But how else could we handle this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426,https://api.github.com/repos/simonw/datasette/issues/2143,1684484426,IC_kwDOBm6k_c5kZzFK,9599,simonw,2023-08-18T22:12:52Z,2023-08-18T22:12:52Z,OWNER,"Yeah, I'm convinced by that. There's not point in having both `settings.json` and `datasette.json`.

I like `datasette.json` ( / `datasette.yml`) as a name. That can be the file that lives in your config directory too, so if you run `datasette .` in a folder containing `datasette.yml` all of those settings get picked up.

Here's a thought for how it could look - I'll go with the YAML format because I expect that to be the default most people use, just because it supports multi-line strings better.

I based this on the big example at https://docs.datasette.io/en/1.0a3/metadata.html#using-yaml-for-metadata - and combined some bits from https://docs.datasette.io/en/1.0a3/authentication.html as well.

```yaml
title: Demonstrating Metadata from YAML
description_html: |-
  <p>This description includes a long HTML string</p>
  <ul>
    <li>YAML is better for embedding HTML strings than JSON!</li>
  </ul>

settings:
  default_page_size: 10
  max_returned_rows: 3000
  sql_time_limit_ms"": 8000

databases:
  docs:
    permissions:
      create-table:
        id: editor
  fixtures:
    tables:
      no_primary_key:
        hidden: true
    queries:
      neighborhood_search:
        sql: |-
          select neighborhood, facet_cities.name, state
          from facetable join facet_cities on facetable.city_id = facet_cities.id
          where neighborhood like '%' || :text || '%' order by neighborhood;
        title: Search neighborhoods
        description_html: |-
          <p>This demonstrates <em>basic</em> LIKE search

permissions:
  debug-menu:
    id: '*'

plugins:
  datasette-ripgrep:
    path: /usr/local/lib/python3.11/site-packages
```
I'm inclined to say we try to be a super-set of the existing `metadata.yml` format, at least where it makes sense to do so. That way the upgrade path is smooth for people. Also, I don't think the format itself is terrible - it's the name that's the big problem.

In this example I've mixed in one extra concept: that `settings:` block with a bunch of settings in it.

There are some things in there that look a little bit like metadata - the `title` and `description_html` fields.

But _are they_ metadata? The title and description of the overall instance feels like it could be described as general configuration. The stuff for the `query` should live where the query itself is defined.

Note that queries can be defined by a plugin hook too: https://docs.datasette.io/en/1.0a3/plugin_hooks.html#canned-queries-datasette-database-actor

What do you think? Is this the right direction, or are you thinking there's a more radical redesign that would make sense here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2145#issuecomment-1684384750,https://api.github.com/repos/simonw/datasette/issues/2145,1684384750,IC_kwDOBm6k_c5kZavu,9599,simonw,2023-08-18T20:07:18Z,2023-08-18T20:07:18Z,OWNER,The big challenge here is what the URL to that row page should look like. How can I encode a `None` in a form that can be encoded and decoded without clashing with primary keys that are the string `None` or `null`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1684235760,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1684235760,IC_kwDOCGYnMM5kY2Xw,9599,simonw,2023-08-18T17:43:11Z,2023-08-18T17:43:11Z,OWNER,"Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it:

https://github.com/simonw/sqlite-utils-fast-fks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684205563,https://api.github.com/repos/simonw/datasette/issues/2143,1684205563,IC_kwDOBm6k_c5kYu_7,15178711,asg017,2023-08-18T17:12:54Z,2023-08-18T17:12:54Z,CONTRIBUTOR,"Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings.

Though I imagine Python imports might make this complex to do, and json/yaml is already supported and pretty easy to write
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1684202932,https://api.github.com/repos/simonw/datasette/issues/2143,1684202932,IC_kwDOBm6k_c5kYuW0,15178711,asg017,2023-08-18T17:10:21Z,2023-08-18T17:10:21Z,CONTRIBUTOR,"I agree with all your points!

I think the best solution would be having a `datasette.json` config file, where you ""configure"" your datasette instances, with settings, permissions/auth, plugin configuration, and table settings (sortable column, label columns, etc.). Which #2093 would do.

Then optionally, you have a `metadata.json`, or use `datasette_metadata`, or some other plugin to define metadata (ex the future [sqlite-docs](https://github.com/asg017/sqlite-docs) plugin).

Everything in `datasette.json` could also be overwritten by CLI flags, like `--setting key value`, `--plugin xxxx key value`.

We could even completely remove `settings.json` in favor or just `datasette.json`. Mostly because I think the less files the better, especially if they have generic names like `settings.json` or `config.json`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1683429959,https://api.github.com/repos/simonw/datasette/issues/2143,1683429959,IC_kwDOBm6k_c5kVxpH,9599,simonw,2023-08-18T06:43:33Z,2023-08-18T15:19:07Z,OWNER,"The single biggest design challenge I've had with metadata relates to how it should or should not be inherited.

If you apply a license to a Datasette instance, it feels like that should flow down to cover all of the databases and all of the tables within those databases.

If the license is at the database level, it should cover all tables.

But... should source do the same thing? I made it behave the same way as license, but it's presumably common for one database to have a single license but multiple different sources of data.

Then there's title - should that inherit? It feels like title should apply to only one level - you may want a title that applies to the instance, then a different custom title for databases and tables.

Here's the current state of play for metadata: https://docs.datasette.io/en/1.0a3/metadata.html

So there's `title` and `description` - and I'll be honest, I'm not 100% sure even I understand how those should be inherited down by tables/etc.

There's `description_html` which over-rides the `description` if it is set. It's a useful customization hack, but a bit surprising.

Then there are these six:

- `license`
- `license_url`
- `source`
- `source_url`
- `about`
- `about_url`

I added `about` later than the others, because I realized that plenty of my own projects needed a link to an article explaining them somewhere - e.g. https://scotrail.datasette.io/

Tables can also have column descriptions - just a string for each column. There's a demo of those here: https://latest.datasette.io/fixtures/roadside_attractions

And then there's all of the other stuff, most of which feels much more like ""settings"" than ""metadata"":

- `sort: created` - the custom sort order
- `size: 10` for a custom page size for a specific table
- `sortable_columns` to set which columns can be used to sort
- `hidden: true` to hide a table
- `label_column: title` is an interesting one - it lets you hint to Datasette which column should be displayed when there is a foreign key relationship. It's sort-of-metadata and sort-of-a-setting.
- `facets` sets default facets, see https://docs.datasette.io/en/1.0a3/facets.html#facets-in-metadata
- `facet_size` sets the number of facets to display
- `fts_table` and `fts_pk` can be used to configure FTS, especially for views: https://docs.datasette.io/en/1.0a3/full_text_search.html

And the authentication stuff! `allow` and `allow_sql` blocks: https://docs.datasette.io/en/1.0a3/authentication.html#defining-permissions-with-allow-blocks

And the new `permissions` key in the 1.0 alphas: https://docs.datasette.io/en/1.0a3/authentication.html#other-permissions-in-metadata

I think that might be everything (excluding the `plugins` settings stuff, which is also a bad fit for metadata.)

And to make things even more confusing... I believe you can add arbitrary key/value pairs to your metadata and then use them in your templates! I think I've heard from at least one person who uses that ability.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1683420879,https://api.github.com/repos/simonw/datasette/issues/2143,1683420879,IC_kwDOBm6k_c5kVvbP,9599,simonw,2023-08-18T06:33:24Z,2023-08-18T15:15:34Z,OWNER,"I completely agree: metadata is a mess, and it deserves our attention.

> 1. Metadata cannot be updated without re-starting the entire Datasette instance.

That's not completely true - there are hacks around that. I have a plugin that applies one set of gnarly hacks for that here: https://github.com/simonw/datasette-remote-metadata - it's pretty grim though!

> 2. The `metadata.json`/`metadata.yaml` has become a kitchen sink of unrelated (imo) features like plugin config, authentication config, canned queries

100% this: it's a complete mess.

Datasette used to have a `datasette --config foo:bar` mechanism, which I deprecated in favour of `datasette --setting foo bar` partly because I wanted to free up `--config` for pointing at a real config file, so we could stop dropping everything in `--metadata metadata.yml`.

> 3. The Python APIs for defining extra metadata are a bit awkward (the `datasette.metadata()` class, `get_metadata()` hook, etc.)

Yes, they're not pretty at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/pull/2144#issuecomment-1683963463,https://api.github.com/repos/simonw/datasette/issues/2144,1683963463,IC_kwDOBm6k_c5kXz5H,22429695,codecov[bot],2023-08-18T13:58:39Z,2023-08-18T13:58:39Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`3a97755`)](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2144   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          40       40           
  Lines        5937     5937           
=======================================
  Hits         5466     5466           
  Misses        471      471           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856760386,Bump the python-packages group with 3 updates,
https://github.com/simonw/datasette/pull/2142#issuecomment-1683950031,https://api.github.com/repos/simonw/datasette/issues/2142,1683950031,IC_kwDOBm6k_c5kXwnP,49699333,dependabot[bot],2023-08-18T13:49:24Z,2023-08-18T13:49:24Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1854970601,Bump the python-packages group with 2 updates,
https://github.com/simonw/datasette/issues/2143#issuecomment-1683443891,https://api.github.com/repos/simonw/datasette/issues/2143,1683443891,IC_kwDOBm6k_c5kV1Cz,9599,simonw,2023-08-18T06:58:15Z,2023-08-18T06:58:15Z,OWNER,"Hah, that `--plugin-secret` thing was a messy solution I came up with to the problem that all metadata is visible at `/-/metadata` - so if you need to stash a secret you need a way to keep it not-visible in there!

Hence the whole `$env` mess: https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values

```json
{
    ""plugins"": {
        ""datasette-auth-github"": {
            ""client_secret"": {
                ""$env"": ""GITHUB_CLIENT_SECRET""
            }
        }
    }
}
```

If configuration and metadata were separate we could ditch that whole messy situation - configuration can stay hidden, metadata can stay public.

Though I have been thinking that Datasette might benefit from a ""secrets"" mechanism that's separate from configuration and metadata... kind of like what LLM has: https://llm.datasette.io/en/stable/help.html#llm-keys-help","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1683440597,https://api.github.com/repos/simonw/datasette/issues/2143,1683440597,IC_kwDOBm6k_c5kV0PV,9599,simonw,2023-08-18T06:54:49Z,2023-08-18T06:54:49Z,OWNER,"A related point that I've been considering a lot recently: it turns out that sometimes I really want to define settings on the CLI instead of in a file, purely for convenience.

It's pretty annoying when I want to try out a new plugin but I have to create a dedicated `metadata.yml` file for it just to setup a single option - I'd love to have the option to be able to run this instead:

```bash
datasette data.db --plugin-setting datasette-upload-csvs default-database data
```

So maybe there's a world in which all of the settings can be applied in a `datasette.yml` file OR with command-line options.

That gets trickier when you need to pass a nested structure or similar, but we could always support those as JSON:

```bash
datasette data.db --plugin-setting datasette-emoji-reactions emoji '[""😼"", ""🐺""]'
```
Note that we kind of have precedent for this in `datasette publish`: https://docs.datasette.io/en/stable/publish.html#custom-metadata-and-plugins

```bash
datasette publish heroku my_database.db \
    --name my-heroku-app-demo \
    --install=datasette-auth-github \
    --plugin-secret datasette-auth-github client_id your_client_id \
    --plugin-secret datasette-auth-github client_secret your_client_secret
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/datasette/issues/2143#issuecomment-1683435579,https://api.github.com/repos/simonw/datasette/issues/2143,1683435579,IC_kwDOBm6k_c5kVzA7,9599,simonw,2023-08-18T06:49:39Z,2023-08-18T06:49:39Z,OWNER,"My ideal situation then would be something like this:

- Metadata itself is VERY clearly described, including sensible rules for metadata inheritance where it makes sense. There is a `datasette.X` method for accessing it which is much more intuitive than `datasette.metadata()`.
- It's possible that method should be an `async` method, because that would support things like plugins that lookup metadata in database tables better.
- All templates etc switch to the new, clean, intuitive metadata mechanism before 1.0.
- I'm interested in the option of metadata being able to live in a `_datasette_metadata` table in the databases themselves - either as a plugin or as a core feature. I think it makes a lot of sense for metadata to optionally live with the data that it describes.
- Configuration gets split from metadata. The stuff that configures Datasette no longer lives in the `metadata.yml` file - it lives in `config.yml` (or even `datasette.yml`).

Currently we have three types of things:

- Metadata - information about the data
- Configuration - stuff like ""these columns should be sortable"" and ""this is configured as `fts_table`"" and suchlike
- Settings - the stuff that you pass to `datasette --setting x y` on server start.

Should settings and configuration be separate? I'm not 100% sure that they should - maybe those two concepts should be combined somehow.

Configuration directory mode needs to be considered too: https://docs.datasette.io/en/stable/settings.html#configuration-directory-mode - interestingly it already has a thing where it can pick up settings from a `settings.json` file - where settings are things like `datasette --setting sql_time_limit_ms 4000`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0,
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683404978,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683404978,IC_kwDOCGYnMM5kVriy,9599,simonw,2023-08-18T06:13:46Z,2023-08-18T06:13:46Z,OWNER,"I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view,
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683398866,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683398866,IC_kwDOCGYnMM5kVqDS,9599,simonw,2023-08-18T06:05:50Z,2023-08-18T06:06:42Z,OWNER,"Options:
- Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default?
- Read that `PRAGMA` and automatically do that view workaround if it's turned on
- Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection?
- Try the `transform()` in a transaction, detect the `""error in view""`, `""no such table""`error, if spotted then do the VIEW workaround and try again

I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view,
https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683396150,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683396150,IC_kwDOCGYnMM5kVpY2,9599,simonw,2023-08-18T06:02:18Z,2023-08-18T06:06:31Z,OWNER,"More notes in here:
- https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873

Not all Python/SQLite installations exhibit this problem by default!

It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table

If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view.

Here's a one-liner to test if it is on or not:

```bash
python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""PRAGMA legacy_alter_table"").fetchall())'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683217284,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683217284,IC_kwDOCGYnMM5kU9uE,9599,simonw,2023-08-18T01:50:21Z,2023-08-18T01:50:21Z,OWNER,"And a test of the `--sql` option:
```bash
sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
sqlite-utils create-table /tmp/t.db country id integer name text
sqlite-utils create-table /tmp/t.db city id integer name text
sqlite-utils create-table /tmp/t.db continent id integer name text
sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql
```
Outputs:
```sql
CREATE TABLE [places_new_6a705d2f5a13] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER REFERENCES [country]([id]),
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
);
INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent])
   SELECT [id], [name], [country], [city], [continent] FROM [places];
DROP TABLE [places];
ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places];
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683212074,IC_kwDOCGYnMM5kU8cq,9599,simonw,2023-08-18T01:43:54Z,2023-08-18T01:43:54Z,OWNER,"Some manual testing:
```bash
sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER
);
```
```bash
sqlite-utils create-table /tmp/t.db country id integer name text
sqlite-utils create-table /tmp/t.db city id integer name text
sqlite-utils create-table /tmp/t.db continent id integer name text
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [places] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER
);
CREATE TABLE [country] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [city] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [continent] (
   [id] INTEGER,
   [name] TEXT
);
```
```bash
sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id
sqlite-utils schema /tmp/t.db
```
```sql
CREATE TABLE [country] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [city] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE [continent] (
   [id] INTEGER,
   [name] TEXT
);
CREATE TABLE ""places"" (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER REFERENCES [country]([id]),
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
);
```
```bash
sqlite-utils transform /tmp/t.db places --drop-foreign-key country
sqlite-utils schema /tmp/t.db places
```
```sql
CREATE TABLE ""places"" (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT,
   [country] INTEGER,
   [city] INTEGER,
   [continent] INTEGER REFERENCES [continent]([id])
)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683201239,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683201239,IC_kwDOCGYnMM5kU5zX,9599,simonw,2023-08-18T01:30:46Z,2023-08-18T01:30:46Z,OWNER,"Help can now look like this:
```
  --drop-foreign-key TEXT         Drop foreign key constraint for this column
  --add-foreign-key <TEXT TEXT TEXT>...
                                  Add a foreign key constraint from a column
                                  to another table with another column
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683200128,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683200128,IC_kwDOCGYnMM5kU5iA,9599,simonw,2023-08-18T01:29:00Z,2023-08-18T01:29:00Z,OWNER,I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683198740,IC_kwDOCGYnMM5kU5MU,9599,simonw,2023-08-18T01:26:47Z,2023-08-18T01:26:47Z,OWNER,"The only CLI feature that supports providing just the column name appears to be this:
```bash
sqlite-utils add-foreign-key --help
```
```
Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE]
                                    [OTHER_COLUMN]

  Add a new foreign key constraint to an existing table

  Example:

      sqlite-utils add-foreign-key my.db books author_id authors id

  WARNING: Could corrupt your database! Back up your database file first.
```
I can drop that WARNING now since I'm not writing to `sqlite_master` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683197882,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683197882,IC_kwDOCGYnMM5kU4-6,9599,simonw,2023-08-18T01:25:53Z,2023-08-18T01:25:53Z,OWNER,"Probably most relevant here is this snippet from:
```bash
sqlite-utils create-table --help
```
```
  --default <TEXT TEXT>...  Default value that should be set for a column
  --fk <TEXT TEXT TEXT>...  Column, other table, other column to set as a
                            foreign key
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`,
https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683195669,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683195669,IC_kwDOCGYnMM5kU4cV,9599,simonw,2023-08-18T01:24:57Z,2023-08-18T01:24:57Z,OWNER,"Currently:
```bash
sqlite-utils transform --help
```
```
Usage: sqlite-utils transform [OPTIONS] PATH TABLE

  Transform a table beyond the capabilities of ALTER TABLE

  Example:

      sqlite-utils transform mydb.db mytable \
          --drop column1 \
          --rename column2 column_renamed

Options:
  --type <TEXT CHOICE>...   Change column type to INTEGER, TEXT, FLOAT or BLOB
  --drop TEXT               Drop this column
  --rename <TEXT TEXT>...   Rename this column to X
  -o, --column-order TEXT   Reorder columns
  --not-null TEXT           Set this column to NOT NULL
  --not-null-false TEXT     Remove NOT NULL from this column
  --pk TEXT                 Make this column the primary key
  --pk-none                 Remove primary key (convert to rowid table)
  --default <TEXT TEXT>...  Set default value for this column
  --default-none TEXT       Remove default from this column
  --drop-foreign-key TEXT   Drop foreign key constraint for this column
  --sql                     Output SQL without executing it
  --load-extension TEXT     Path to SQLite extension, with optional
                            :entrypoint
  -h, --help                Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122490,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122490,IC_kwDOCGYnMM5kUmk6,22429695,codecov[bot],2023-08-17T23:45:44Z,2023-08-18T00:46:07Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`92.85%`** and project coverage change: **`-0.07%`** :warning:
> Comparison is base [(`1dc6b5a`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/1dc6b5aa644a92d3654f7068110ed7930989ce71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.82% compared to head [(`2915050`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.76%.
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #584      +/-   ##
==========================================
- Coverage   95.82%   95.76%   -0.07%     
==========================================
  Files           8        8              
  Lines        2829     2834       +5     
==========================================
+ Hits         2711     2714       +3     
- Misses        118      120       +2     
```


| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <92.85%> (-0.15%)` | :arrow_down: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683164661,IC_kwDOCGYnMM5kUw31,9599,simonw,2023-08-18T00:45:53Z,2023-08-18T00:45:53Z,OWNER,"More updated documentation:
- https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform
- https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145819,IC_kwDOCGYnMM5kUsRb,9599,simonw,2023-08-18T00:17:26Z,2023-08-18T00:17:26Z,OWNER,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145110,IC_kwDOCGYnMM5kUsGW,9599,simonw,2023-08-18T00:16:28Z,2023-08-18T00:16:48Z,OWNER,"One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`:

https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708

I should write tests for them too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683143723,IC_kwDOCGYnMM5kUrwr,9599,simonw,2023-08-18T00:14:52Z,2023-08-18T00:14:52Z,OWNER,"Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times:

> The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries.
> 
> `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up.
> 
> Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`.

That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683139304,IC_kwDOCGYnMM5kUqro,9599,simonw,2023-08-18T00:09:56Z,2023-08-18T00:09:56Z,OWNER,"Upgrading `flake8` locally replicated the error:
```
pip install -U flake8
flake8
```
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683138953,IC_kwDOCGYnMM5kUqmJ,9599,simonw,2023-08-18T00:09:20Z,2023-08-18T00:09:20Z,OWNER,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop:
```
./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96
./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683137259,IC_kwDOCGYnMM5kUqLr,9599,simonw,2023-08-18T00:06:59Z,2023-08-18T00:06:59Z,OWNER,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,simonw,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures:

```
sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"")  [assignment]
sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[<type>, <type>] = ..."")  [var-annotated]
sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform""  [union-attr]
sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected <nothing>  [arg-type]
sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected <nothing>  [arg-type]
Found 5 errors in 1 file (checked 56 source files)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,simonw,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line:
```python
fresh_db.create_table(""breeds"", {""name"": str})
```
Has this schema:
```sql
CREATE TABLE [breeds] (
   [name] TEXT
);
```
SQLite creates an invisible `rowid` column for it automatically.

On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added:
```sql
(Pdb) print(fresh_db.schema)
CREATE TABLE [dogs] (
   [name] TEXT
, [breed_id] INTEGER,
   FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid])
);
CREATE TABLE [breeds] (
   [name] TEXT
);
```
But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884

Here's what the debugger reveals about this code:
```python
        for fk in foreign_keys:
            if fk.other_table == name and columns.get(fk.other_column):
                continue
            if not any(
                c for c in self[fk.other_table].columns if c.name == fk.other_column
            ):
                raise AlterError(
                    ""No such column: {}.{}"".format(fk.other_table, fk.other_column)
                )
```
```
(Pdb) fk
ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid')
(Pdb) self[fk.other_table].columns
[Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,simonw,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem:
```python
from sqlite_utils import Database

fresh_db = Database(memory=True)

fresh_db.create_table(""dogs"", {""name"": str})
fresh_db.create_table(""breeds"", {""name"": str})
fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"")
```
Traceback:
```
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column
    self.add_foreign_key(col_name, fk, fk_col)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key
    self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys
    self[table].transform(add_foreign_keys=fks)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform
    sqls = self.transform_sql(
           ^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql
    self.db.create_table_sql(
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql
    raise AlterError(
sqlite_utils.db.AlterError: No such column: breeds.rowid
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,simonw,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test:

https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,simonw,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left:

```
        # Soundness check foreign_keys point to existing tables
        for fk in foreign_keys:
            if fk.other_table == name and columns.get(fk.other_column):
                continue
            if not any(
                c for c in self[fk.other_table].columns if c.name == fk.other_column
            ):
>               raise AlterError(
                    ""No such column: {}.{}"".format(fk.other_table, fk.other_column)
                )
E               sqlite_utils.db.AlterError: No such column: breeds.rowid

sqlite_utils/db.py:882: AlterError
==== short test summary info ====
FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid
==== 1 failed, 378 deselected in 0.49s ====
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,simonw,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on:
- #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914,Get rid of test.utils.collapse_whitespace,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,simonw,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133

Needs updating like so:
```diff
diff --git a/tests/test_extract.py b/tests/test_extract.py
index 70ad0cf..fd52534 100644
--- a/tests/test_extract.py
+++ b/tests/test_extract.py
@@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db):
     assert fresh_db[""tree""].schema == (
         'CREATE TABLE ""tree"" (\n'
         ""   [name] TEXT,\n""
-        ""   [common_name_latin_name_id] INTEGER,\n""
-        ""   FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n""
+        ""   [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n""
         "")""
     )
     assert (
```
Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in.

I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683076325,IC_kwDOCGYnMM5kUbTl,9599,simonw,2023-08-17T22:48:36Z,2023-08-17T22:48:36Z,OWNER,"I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism.

It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,simonw,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments:
```python
    def transform(
        self,
        *,
        # ...
        # This one exists already:
        drop_foreign_keys: Optional[Iterable] = None,
        # These two are new. This one specifies keys to add:
        add_foreign_keys: Optional[ForeignKeysType] = None,
        # Or this one causes them all to be replaced with the new definitions:
        foreign_keys: Optional[ForeignKeysType] = None,
```
There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,simonw,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder:
https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,simonw,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this:
```python
        drop_foreign_keys: Optional[Iterable] = None,
```
Should be this:
```python
        drop_foreign_keys: Optional[Iterable[str]] = None,
```
Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,simonw,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation:

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149

At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`.

Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`:

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,simonw,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others:
- https://github.com/simonw/llm/issues/60
- https://github.com/simonw/llm/issues/116
- https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,simonw,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work.

It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter:

https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872

Improving this code to support adding foreign keys as well would be pretty simple.

And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/datasette/pull/2142#issuecomment-1682272492,https://api.github.com/repos/simonw/datasette/issues/2142,1682272492,IC_kwDOBm6k_c5kRXDs,22429695,codecov[bot],2023-08-17T13:18:08Z,2023-08-17T13:18:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`6b617f7`)](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2142   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          40       40           
  Lines        5937     5937           
=======================================
  Hits         5466     5466           
  Misses        471      471           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1854970601,Bump the python-packages group with 2 updates,
https://github.com/simonw/datasette/pull/2141#issuecomment-1682256251,https://api.github.com/repos/simonw/datasette/issues/2141,1682256251,IC_kwDOBm6k_c5kRTF7,49699333,dependabot[bot],2023-08-17T13:07:43Z,2023-08-17T13:07:43Z,CONTRIBUTOR,"Looks like blacken-docs is updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1853289039,Bump the python-packages group with 1 update,
https://github.com/simonw/datasette/pull/2141#issuecomment-1680660343,https://api.github.com/repos/simonw/datasette/issues/2141,1680660343,IC_kwDOBm6k_c5kLNd3,22429695,codecov[bot],2023-08-16T13:55:22Z,2023-08-16T13:55:22Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`e2be94a`)](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2141   +/-   ##
=======================================
  Coverage   92.06%   92.06%           
=======================================
  Files          40       40           
  Lines        5937     5937           
=======================================
  Hits         5466     5466           
  Misses        471      471           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1853289039,Bump the python-packages group with 1 update,
https://github.com/simonw/datasette/issues/2140#issuecomment-1675157445,https://api.github.com/repos/simonw/datasette/issues/2140,1675157445,IC_kwDOBm6k_c5j2N_F,9599,simonw,2023-08-11T17:45:54Z,2023-08-11T17:51:59Z,OWNER,"Affected pages:

- https://docs.datasette.io/en/latest/authentication.html
- https://docs.datasette.io/en/latest/changelog.html
- https://docs.datasette.io/en/latest/cli-reference.html
- https://docs.datasette.io/en/latest/contributing.html
- https://docs.datasette.io/en/latest/custom_templates.html
- https://docs.datasette.io/en/latest/deploying.html
- https://docs.datasette.io/en/latest/facets.html
- https://docs.datasette.io/en/latest/full_text_search.html
- https://docs.datasette.io/en/latest/installation.html
- https://docs.datasette.io/en/latest/plugin_hooks.html
- https://docs.datasette.io/en/latest/publish.html
- https://docs.datasette.io/en/latest/settings.html
- https://docs.datasette.io/en/latest/spatialite.html
- https://docs.datasette.io/en/latest/sql_queries.html

e.g. on https://docs.datasette.io/en/latest/authentication.html#using-the-root-actor

<img width=""759"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/aa3ea5bd-1b4b-4ecf-a5ed-a62508dcde31"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1847201263,Remove all remaining documentation instances of '$ ',
https://github.com/simonw/datasette/issues/2126#issuecomment-1674242356,https://api.github.com/repos/simonw/datasette/issues/2126,1674242356,IC_kwDOBm6k_c5jyuk0,36199671,ctsrc,2023-08-11T05:52:29Z,2023-08-11T05:52:29Z,NONE,"I see :) yeah, I’m on the stable version installed from homebrew on macOS","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838266862,Permissions in metadata.yml / metadata.json,
https://github.com/simonw/datasette/issues/2119#issuecomment-1674227353,https://api.github.com/repos/simonw/datasette/issues/2119,1674227353,IC_kwDOBm6k_c5jyq6Z,9599,simonw,2023-08-11T05:25:45Z,2023-08-11T05:25:45Z,OWNER,"Demo:

- https://latest.datasette.io/
- https://latest.datasette.io/fixtures
- https://latest.datasette.io/fixtures/facetable
- https://latest.datasette.io/fixtures/pragma_cache_size
- https://latest.datasette.io/fixtures/attraction_characteristic/2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823393475,"database color shows only on index page, not other pages",
https://github.com/simonw/datasette/issues/568#issuecomment-1674223722,https://api.github.com/repos/simonw/datasette/issues/568,1674223722,IC_kwDOBm6k_c5jyqBq,9599,simonw,2023-08-11T05:19:42Z,2023-08-11T05:19:42Z,OWNER,"I decided not to implement this yet, because making `database.color` dependent on metadata interferes with my goal to eventually make metadata lookups `async` in order to serve them better from plugins.

I'll leave this issue open though as it may get solved in future redesigns of the metadata system, refs:

- #1384","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",476852861,Add database_color as a configurable option,
https://github.com/simonw/datasette/issues/2119#issuecomment-1674211804,https://api.github.com/repos/simonw/datasette/issues/2119,1674211804,IC_kwDOBm6k_c5jynHc,9599,simonw,2023-08-11T04:56:28Z,2023-08-11T04:56:28Z,OWNER,"That `database.color` is only available in `index.html` at the moment thanks to this:

https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/views/index.py#L108-L110","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823393475,"database color shows only on index page, not other pages",
https://github.com/simonw/datasette/issues/2119#issuecomment-1674210215,https://api.github.com/repos/simonw/datasette/issues/2119,1674210215,IC_kwDOBm6k_c5jymun,9599,simonw,2023-08-11T04:53:11Z,2023-08-11T04:53:11Z,OWNER,"Relevant template snippet:

https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823393475,"database color shows only on index page, not other pages",
https://github.com/simonw/datasette/issues/2139#issuecomment-1674210009,https://api.github.com/repos/simonw/datasette/issues/2139,1674210009,IC_kwDOBm6k_c5jymrZ,9599,simonw,2023-08-11T04:52:44Z,2023-08-11T04:52:44Z,OWNER,"I'll fix these too:
- https://github.com/simonw/datasette/issues/2119
- https://github.com/simonw/datasette/issues/568","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1846076261,border-color: ##ff0000 bug - two hashes,
https://github.com/simonw/datasette/issues/2139#issuecomment-1674115279,https://api.github.com/repos/simonw/datasette/issues/2139,1674115279,IC_kwDOBm6k_c5jyPjP,9599,simonw,2023-08-11T01:27:30Z,2023-08-11T01:27:30Z,OWNER,"https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/database.html#L13

Looks like I should be doing this instead:

https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1846076261,border-color: ##ff0000 bug - two hashes,
https://github.com/simonw/datasette/issues/2127#issuecomment-1672404276,https://api.github.com/repos/simonw/datasette/issues/2127,1672404276,IC_kwDOBm6k_c5jrt00,9599,simonw,2023-08-10T01:30:25Z,2023-08-10T01:30:25Z,OWNER,"Here's the prototype of adding documentation for these using Cog: https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875

![](https://user-images.githubusercontent.com/9599/249324964-79210a43-716c-4470-a62d-01aa871973ec.png)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838469176,Context base class to support documenting the context,
https://github.com/simonw/datasette/issues/2134#issuecomment-1672384439,https://api.github.com/repos/simonw/datasette/issues/2134,1672384439,IC_kwDOBm6k_c5jro-3,9599,simonw,2023-08-10T01:05:56Z,2023-08-10T01:22:46Z,OWNER,"Now live at https://latest.datasette.io/counters

![increment](https://github.com/simonw/datasette/assets/9599/ffb3625b-b61b-4189-80e3-7b4e8b411f14)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io,
https://github.com/simonw/datasette/issues/2126#issuecomment-1672385674,https://api.github.com/repos/simonw/datasette/issues/2126,1672385674,IC_kwDOBm6k_c5jrpSK,9599,simonw,2023-08-10T01:07:43Z,2023-08-10T01:07:43Z,OWNER,"What version of Datasette are you running?

That feature was added in Datasette 1.0a2, so if you're on the current stable release you won't have it yet.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838266862,Permissions in metadata.yml / metadata.json,
https://github.com/simonw/datasette/issues/2138#issuecomment-1672366410,https://api.github.com/repos/simonw/datasette/issues/2138,1672366410,IC_kwDOBm6k_c5jrklK,9599,simonw,2023-08-10T00:39:16Z,2023-08-10T00:39:16Z,OWNER,"Here's what it looks like when it works:

<img width=""715"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/a53efc81-8ceb-44a2-b083-2efb27d20cf7"">

And here's what you get if there's a SQL error in the `on_success_message_sql` query itself:

<img width=""715"" alt=""CleanShot 2023-08-09 at 17 38 12@2x"" src=""https://github.com/simonw/datasette/assets/9599/e211cee2-3edb-475a-906d-b0bb5a118d77"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1844213115,on_success_message_sql option for writable canned queries,
https://github.com/simonw/datasette/issues/2133#issuecomment-1672360472,https://api.github.com/repos/simonw/datasette/issues/2133,1672360472,IC_kwDOBm6k_c5jrjIY,54462,HaveF,2023-08-10T00:31:24Z,2023-08-10T00:31:24Z,NONE,"It looks very nice now.
Finally, no more manual installation of plugins one by one. Thank you, Simon!  ❤️ 
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2138#issuecomment-1672353428,https://api.github.com/repos/simonw/datasette/issues/2138,1672353428,IC_kwDOBm6k_c5jrhaU,9599,simonw,2023-08-10T00:20:56Z,2023-08-10T00:20:56Z,OWNER,"Goal here is to customize the message displayed after a query has succeeded. e.g.

```yaml
databases:
  mydatabase:
    queries:
      add_name:
        sql: INSERT INTO names (name) VALUES (:name)
        write: true
        on_success_message_sql: 'select ''Name inserted: '' || :name'
        on_success_redirect: /mydatabase/names
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1844213115,on_success_message_sql option for writable canned queries,
https://github.com/simonw/datasette/issues/2134#issuecomment-1671581610,https://api.github.com/repos/simonw/datasette/issues/2134,1671581610,IC_kwDOBm6k_c5jok-q,9599,simonw,2023-08-09T15:01:14Z,2023-08-10T00:19:56Z,OWNER,Or... how about if the `on_success_message` option could define a SQL query to be executed to generate that message? Maybe `on_success_message_sql`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io,
https://github.com/simonw/datasette/issues/782#issuecomment-1672340556,https://api.github.com/repos/simonw/datasette/issues/782,1672340556,IC_kwDOBm6k_c5jreRM,9599,simonw,2023-08-10T00:07:17Z,2023-08-10T00:07:17Z,OWNER,Most of this is shipped in https://docs.datasette.io/en/1.0a3/changelog.html#a3-2023-08-09,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format,
https://github.com/simonw/datasette/issues/2133#issuecomment-1672224611,https://api.github.com/repos/simonw/datasette/issues/2133,1672224611,IC_kwDOBm6k_c5jrB9j,9599,simonw,2023-08-09T22:07:43Z,2023-08-09T22:07:43Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugins.html#seeing-what-plugins-are-installed,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2133#issuecomment-1672076671,https://api.github.com/repos/simonw/datasette/issues/2133,1672076671,IC_kwDOBm6k_c5jqd1_,9599,simonw,2023-08-09T20:09:31Z,2023-08-09T20:09:31Z,OWNER,I'm going to have it only return plugins with a version other than `None`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2133#issuecomment-1672076284,https://api.github.com/repos/simonw/datasette/issues/2133,1672076284,IC_kwDOBm6k_c5jqdv8,9599,simonw,2023-08-09T20:09:12Z,2023-08-09T20:09:12Z,OWNER,"Problem - if there are plugins in the plugins directory it was returning this:
```
messages_output_renderer.py==None
my_plugin.py==None
my_plugin_2.py==None
register_output_renderer.py==None
sleep_sql_function.py==None
view_name.py==None
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2137#issuecomment-1671983228,https://api.github.com/repos/simonw/datasette/issues/2137,1671983228,IC_kwDOBm6k_c5jqHB8,9599,simonw,2023-08-09T19:02:47Z,2023-08-09T19:02:47Z,OWNER,"As part of this I'd like to refactor the JSON renderer code - I started attempting that here: https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2

In particular I want to get rid of that `data` argument.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843821954,Redesign row default JSON,
https://github.com/simonw/datasette/issues/2136#issuecomment-1671982215,https://api.github.com/repos/simonw/datasette/issues/2136,1671982215,IC_kwDOBm6k_c5jqGyH,9599,simonw,2023-08-09T19:02:19Z,2023-08-09T19:02:19Z,OWNER,"I'm going with a bit of a hacky workaround for the moment, until I get a chance to refactor this:
- #2137 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`,
https://github.com/simonw/datasette/issues/2136#issuecomment-1671964003,https://api.github.com/repos/simonw/datasette/issues/2136,1671964003,IC_kwDOBm6k_c5jqCVj,9599,simonw,2023-08-09T18:53:10Z,2023-08-09T18:53:18Z,OWNER,"I made an initial attempt at this in a branch, but it broke too much other stuff.  https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`,
https://github.com/simonw/datasette/issues/2113#issuecomment-1671862711,https://api.github.com/repos/simonw/datasette/issues/2113,1671862711,IC_kwDOBm6k_c5jppm3,9599,simonw,2023-08-09T17:35:22Z,2023-08-09T17:35:22Z,OWNER,"Need `?_extra=columns` too since I'm removing `""columns""` from the default query JSON in https://github.com/simonw/datasette/issues/2136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822939274,Implement and document extras for the new query view page,
https://github.com/simonw/datasette/issues/2136#issuecomment-1671861623,https://api.github.com/repos/simonw/datasette/issues/2136,1671861623,IC_kwDOBm6k_c5jppV3,9599,simonw,2023-08-09T17:34:35Z,2023-08-09T17:34:35Z,OWNER,"I'll make `?_extra=columns` work in:
- #2113","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`,
https://github.com/simonw/datasette/issues/2136#issuecomment-1671848664,https://api.github.com/repos/simonw/datasette/issues/2136,1671848664,IC_kwDOBm6k_c5jpmLY,9599,simonw,2023-08-09T17:24:37Z,2023-08-09T17:24:37Z,OWNER,"It also bothers me that `ok` comes first in one but not first in the other - I'd like the order of keys in the JSON to be consistent, even though order isn't meant to mean anything.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`,
https://github.com/simonw/datasette/issues/2135#issuecomment-1671844664,https://api.github.com/repos/simonw/datasette/issues/2135,1671844664,IC_kwDOBm6k_c5jplM4,9599,simonw,2023-08-09T17:21:22Z,2023-08-09T17:21:22Z,OWNER,"OK, I've done the small changes - still need to document the big JSON changes though.

Those aren't covered by the regular documentation either so I need to update that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843600087,Release notes for 1.0a3,
https://github.com/simonw/datasette/issues/2007#issuecomment-1671836347,https://api.github.com/repos/simonw/datasette/issues/2007,1671836347,IC_kwDOBm6k_c5jpjK7,9599,simonw,2023-08-09T17:15:03Z,2023-08-09T17:15:03Z,OWNER,Updated documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#render-cell-row-value-column-table-database-datasette-request,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1560662739,`render_cell()` hook should take an optional `request` argument,
https://github.com/simonw/datasette/issues/2135#issuecomment-1671753753,https://api.github.com/repos/simonw/datasette/issues/2135,1671753753,IC_kwDOBm6k_c5jpPAZ,9599,simonw,2023-08-09T16:26:17Z,2023-08-09T16:28:19Z,OWNER,"- Dropped support for Python 3.7
- New `Justfile`
- #1970
- `datasette.render_template()` now accepts a `Context` subclass as an alternative to a dictionary
- #2106
- #2007
- https://github.com/simonw/datasette/issues/2130
- https://github.com/simonw/datasette/issues/2079
- https://github.com/simonw/datasette/issues/1153","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843600087,Release notes for 1.0a3,
https://github.com/simonw/datasette/issues/2135#issuecomment-1671750913,https://api.github.com/repos/simonw/datasette/issues/2135,1671750913,IC_kwDOBm6k_c5jpOUB,9599,simonw,2023-08-09T16:24:26Z,2023-08-09T16:24:26Z,OWNER,"Remember not to include changes from this:

https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/docs/changelog.rst?plain=1#L9-L31
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843600087,Release notes for 1.0a3,
https://github.com/simonw/datasette/issues/2113#issuecomment-1671725144,https://api.github.com/repos/simonw/datasette/issues/2113,1671725144,IC_kwDOBm6k_c5jpIBY,9599,simonw,2023-08-09T16:07:58Z,2023-08-09T16:07:58Z,OWNER,Punting this out of the 1.0a3 alpha for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822939274,Implement and document extras for the new query view page,
https://github.com/simonw/datasette/issues/2133#issuecomment-1671722482,https://api.github.com/repos/simonw/datasette/issues/2133,1671722482,IC_kwDOBm6k_c5jpHXy,9599,simonw,2023-08-09T16:06:28Z,2023-08-09T16:06:28Z,OWNER,"Plugin configuration itself currently lives in `metadata.json` which I don't like - that file was supposed to be for metadata about your databases, it's a bit of an accident that it grew configuration options as well.

I'm hoping to fix that for Datasette 1.0, but I'm still figuring out how to do that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2133#issuecomment-1671720761,https://api.github.com/repos/simonw/datasette/issues/2133,1671720761,IC_kwDOBm6k_c5jpG85,9599,simonw,2023-08-09T16:05:33Z,2023-08-09T16:05:33Z,OWNER,"Honestly the `--all` option isn't particularly useful - it's really more of a debugging tool for myself than something that I expect anyone else to use.

The purpose of `datasette plugins` generally is to help people confirm that plugins installed correctly. It's effectively the CLI alternative to hitting https://datasette.io/-/plugins","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2133#issuecomment-1671649530,https://api.github.com/repos/simonw/datasette/issues/2133,1671649530,IC_kwDOBm6k_c5jo1j6,54462,HaveF,2023-08-09T15:41:14Z,2023-08-09T15:41:14Z,NONE,"Yes,  using this approach(`datasette install -r requirements.txt`) will result in more consistency. 

I'm curious about the results of the `datasette plugins --all` command. Where will we use the output of this command? Will it include configuration information for these plugins in the future? If so, will we need to consider the configuration of these plugins in addition to installing them on different computers?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2133#issuecomment-1671634654,https://api.github.com/repos/simonw/datasette/issues/2133,1671634654,IC_kwDOBm6k_c5jox7e,9599,simonw,2023-08-09T15:32:14Z,2023-08-09T15:32:14Z,OWNER,"```diff
diff --git a/datasette/cli.py b/datasette/cli.py
index 32266888..370b722c 100644
--- a/datasette/cli.py
+++ b/datasette/cli.py
@@ -223,15 +223,22 @@ pm.hook.publish_subcommand(publish=publish)
 
 @cli.command()
 @click.option(""--all"", help=""Include built-in default plugins"", is_flag=True)
+@click.option(
+    ""--requirements"", help=""Output requirements.txt of installed plugins"", is_flag=True
+)
 @click.option(
     ""--plugins-dir"",
     type=click.Path(exists=True, file_okay=False, dir_okay=True),
     help=""Path to directory containing custom plugins"",
 )
-def plugins(all, plugins_dir):
+def plugins(all, requirements, plugins_dir):
     """"""List currently installed plugins""""""
     app = Datasette([], plugins_dir=plugins_dir)
-    click.echo(json.dumps(app._plugins(all=all), indent=4))
+    if requirements:
+        for plugin in app._plugins():
+            click.echo(""{}=={}"".format(plugin[""name""], plugin[""version""]))
+    else:
+        click.echo(json.dumps(app._plugins(all=all), indent=4))
 
 
 @cli.command()
```
Output:
```
% datasette plugins --requirements  
datasette-codespaces==0.1.1
datasette-graphql==2.2
datasette-json-html==1.0.1
datasette-pretty-json==0.2.2
datasette-x-forwarded-host==0.1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2133#issuecomment-1671628602,https://api.github.com/repos/simonw/datasette/issues/2133,1671628602,IC_kwDOBm6k_c5jowc6,9599,simonw,2023-08-09T15:28:42Z,2023-08-09T15:28:42Z,OWNER,"Neat idea! We already have this command:

    datasette install -r requirements.txt

So maybe `datasetet plugins --requirements` could generate a `requirements.txt` file which could be used there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options,
https://github.com/simonw/datasette/issues/2131#issuecomment-1671626876,https://api.github.com/repos/simonw/datasette/issues/2131,1671626876,IC_kwDOBm6k_c5jowB8,9599,simonw,2023-08-09T15:27:40Z,2023-08-09T15:27:40Z,OWNER,Another place that needs fixing: https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/datasette/views/database.py#L244-L249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840417903,Refactor code that supports templates_considered comment,
https://github.com/simonw/datasette/issues/2134#issuecomment-1671526372,https://api.github.com/repos/simonw/datasette/issues/2134,1671526372,IC_kwDOBm6k_c5joXfk,9599,simonw,2023-08-09T14:39:54Z,2023-08-09T14:39:54Z,OWNER,"Alternatively, what about if there was a custom SQL function available during canned write queries for setting the output message? Then the fact that canned queries don't return a table view wouldn't be a problem. Bit weird though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io,
https://github.com/simonw/datasette/issues/2134#issuecomment-1671503163,https://api.github.com/repos/simonw/datasette/issues/2134,1671503163,IC_kwDOBm6k_c5joR07,9599,simonw,2023-08-09T14:32:30Z,2023-08-09T14:32:30Z,OWNER,"Annoying thing about this plugin is that you don't see the new counter value when you submit the increment or decrement query.

Maybe canned queries should support SQL multiple statements? Could return the result of the last one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io,
https://github.com/simonw/datasette/issues/2134#issuecomment-1671500965,https://api.github.com/repos/simonw/datasette/issues/2134,1671500965,IC_kwDOBm6k_c5joRSl,9599,simonw,2023-08-09T14:31:43Z,2023-08-09T14:31:43Z,OWNER,"Here's a simple plugin that can do this:
```python
from datasette import hookimpl

@hookimpl
def startup(datasette):
    db = datasette.add_memory_database(""counters"")

    async def inner():
        await db.execute_write(""create table if not exists counters (name text primary key, value integer)"")
        await db.execute_write(""insert or ignore into counters (name, value) values ('counter', 0)"")

    return inner


@hookimpl
def canned_queries(database):
    if database == ""counters"":
        return {
            ""increment"": {
                ""sql"": ""update counters set value = value + 1 where name = 'counter'"",
                ""write"": True,
            },
            ""decrement"": {
                ""sql"": ""update counters set value = value - 1 where name = 'counter'"",
                ""write"": True,
            },
        }
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io,
https://github.com/simonw/datasette/issues/2114#issuecomment-1670470516,https://api.github.com/repos/simonw/datasette/issues/2114,1670470516,IC_kwDOBm6k_c5jkVt0,9599,simonw,2023-08-09T00:09:55Z,2023-08-09T00:10:32Z,OWNER,"I decided to figure out where the `_size=` parameter here comes from: https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/database.py#L576-L588

It took some spelunking, but it looks like I added that five years ago as part of CSV streaming in https://github.com/simonw/datasette/commit/fc3660cfad7668dbce6ead12766e048fc1f78b11

It's there purely to allow CSV streaming to set `_size=""max""`.

And since CSV streaming is still just a table thing, not a query thing, I can drop it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822940263,Implement canned queries against new query JSON work,
https://github.com/simonw/datasette/issues/2114#issuecomment-1670454222,https://api.github.com/repos/simonw/datasette/issues/2114,1670454222,IC_kwDOBm6k_c5jkRvO,9599,simonw,2023-08-08T23:44:42Z,2023-08-08T23:44:42Z,OWNER,"The tests for this are still passing:

```bash
pytest tests/test_canned_queries.py
```
Because I left in the old code:

https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L752-L767

https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L701-L727","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822940263,Implement canned queries against new query JSON work,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1669877769,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,1669877769,IC_kwDOD079W85jiFAJ,22996,chrismytton,2023-08-08T15:52:52Z,2023-08-08T15:52:52Z,NONE,"You can also install this with pip using this oneliner:

```
pip install git+https://github.com/RhetTbull/dogsheep-photos.git@update_for_bigsur
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/simonw/datasette/issues/2132#issuecomment-1669640626,https://api.github.com/repos/simonw/datasette/issues/2132,1669640626,IC_kwDOBm6k_c5jhLGy,9599,simonw,2023-08-08T13:39:54Z,2023-08-08T13:39:54Z,OWNER,https://github.com/simonw/datasette/blob/1377a290cd85ba8d3338b1da47c4665ed4c6c625/datasette/views/database.py#L509-L517,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841343173,Get form fields on query page working again ,
https://github.com/simonw/datasette/issues/2113#issuecomment-1668983779,https://api.github.com/repos/simonw/datasette/issues/2113,1668983779,IC_kwDOBm6k_c5jeqvj,9599,simonw,2023-08-08T06:27:42Z,2023-08-08T06:27:42Z,OWNER,"I've now shipped the new query JSON without any extras: https://latest.datasette.io/fixtures.json?sql=select+*+from+facetable

There aren't actually that many extras that make sense for the query page. These are the most obvious:

- `query` - adding a `{""sql"": ""..."", ""params"": ""..""}` block, as seen on https://latest.datasette.io/fixtures/facetable.json?_extra=query&_size=0
- `duration_ms` to show the duration - which isn't an extra on table right now but should be.
- `metadata` - https://latest.datasette.io/fixtures/facetable.json?_extra=metadata&_size=0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822939274,Implement and document extras for the new query view page,
https://github.com/simonw/datasette/issues/2117#issuecomment-1668799233,https://api.github.com/repos/simonw/datasette/issues/2117,1668799233,IC_kwDOBm6k_c5jd9sB,9599,simonw,2023-08-08T02:02:07Z,2023-08-08T02:02:07Z,OWNER,I just need to pass `view_name` to `datasette.render_template()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822982933,Figure out what to do about `DatabaseView.name`,
https://github.com/simonw/datasette/issues/2112#issuecomment-1668798642,https://api.github.com/repos/simonw/datasette/issues/2112,1668798642,IC_kwDOBm6k_c5jd9iy,9599,simonw,2023-08-08T02:01:01Z,2023-08-08T02:01:01Z,OWNER,Shipped: https://latest.datasette.io/fixtures?sql=select+*+from+facetable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=...,
https://github.com/simonw/datasette/issues/2111#issuecomment-1668798261,https://api.github.com/repos/simonw/datasette/issues/2111,1668798261,IC_kwDOBm6k_c5jd9c1,9599,simonw,2023-08-08T02:00:22Z,2023-08-08T02:00:37Z,OWNER,"Shipped: https://latest.datasette.io/_memory.json?sql=select+sqlite_version()
```json
{
  ""rows"": [
    {
      ""sqlite_version()"": ""3.34.1""
    }
  ],
  ""columns"": [
    ""sqlite_version()""
  ],
  ""ok"": true,
  ""truncated"": false
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=...,
https://github.com/simonw/datasette/issues/2130#issuecomment-1668793100,https://api.github.com/repos/simonw/datasette/issues/2130,1668793100,IC_kwDOBm6k_c5jd8MM,9599,simonw,2023-08-08T01:51:54Z,2023-08-08T01:51:54Z,OWNER,Updated documentation is here: https://docs.datasette.io/en/latest/plugin_hooks.html#register-output-renderer-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840329615,Render plugin mechanism needs `error` and `truncated` fields,
https://github.com/simonw/datasette/issues/2127#issuecomment-1668792576,https://api.github.com/repos/simonw/datasette/issues/2127,1668792576,IC_kwDOBm6k_c5jd8EA,9599,simonw,2023-08-08T01:51:14Z,2023-08-08T01:51:14Z,OWNER,"OK, I've landed the first example of this now. I'm cautiously optimistic about it, but I think I need to use it in a few more places AND add the actual code that uses it for documentation before I'm completely happy and close this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838469176,Context base class to support documenting the context,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668789134,https://api.github.com/repos/simonw/datasette/issues/2118,1668789134,IC_kwDOBm6k_c5jd7OO,9599,simonw,2023-08-08T01:46:12Z,2023-08-08T01:46:12Z,OWNER,"I'm going to squash merge this even though it's a bit big, because I want a clean passing commit in the `main` branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668786770,https://api.github.com/repos/simonw/datasette/issues/2118,1668786770,IC_kwDOBm6k_c5jd6pS,9599,simonw,2023-08-08T01:42:55Z,2023-08-08T01:42:55Z,OWNER,"> UPDATE: Fixed the `_size=max` bit, but I've not replicated the behaviour where it adds `?_labels=on` if there are expandable columns yet.

It looks like that behaviour is only relevant to table views, and it's already implemented - https://latest.datasette.io/fixtures/roadside_attraction_characteristics links to https://latest.datasette.io/fixtures/roadside_attraction_characteristics.csv?_labels=on&_size=max","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668715803,https://api.github.com/repos/simonw/datasette/issues/2118,1668715803,IC_kwDOBm6k_c5jdpUb,9599,simonw,2023-08-07T23:50:26Z,2023-08-08T01:41:53Z,OWNER,"This bit needs some more work: https://github.com/simonw/datasette/blob/a791115e31e64e9bbd1d36f4d46887b3e42e752a/datasette/views/database.py#L514-L518

The original logic for that is from here, I need to figure out how to replicate it:

https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L368

https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L342-L344

UPDATE: Fixed the `_size=max` bit, but I've not replicated the behaviour where it adds `?_labels=on` if there are expandable columns yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668781558,https://api.github.com/repos/simonw/datasette/issues/2118,1668781558,IC_kwDOBm6k_c5jd5X2,9599,simonw,2023-08-08T01:34:42Z,2023-08-08T01:34:42Z,OWNER,"3 failures left:
```
=================================================================== short test summary info ====================================================================
FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href=""/fixtures.csv?sql=select+1&amp;_size=max"">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link r...
FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - KeyError: 'Access-Control-Allow-Origin'
FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
=============================================== 3 failed, 1309 passed, 2 skipped, 1 xfailed in 61.34s (0:01:01) ================================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668720671,https://api.github.com/repos/simonw/datasette/issues/2118,1668720671,IC_kwDOBm6k_c5jdqgf,9599,simonw,2023-08-07T23:58:21Z,2023-08-07T23:58:21Z,OWNER,"11 tests left:
```
FAILED tests/test_html.py::test_alternate_url_json[/fixtures-http://localhost/fixtures.json] - KeyError: 'link'
FAILED tests/test_html.py::test_alternate_url_json[/fixtures?sql=select+*+from+facetable-http://localhost/fixtures.json?sql=select+*+from+facetable] - assert '<link rel=""alternate"" type=""application/json+datasette"" href=""http://localhost/fixtures.json?sql=select+*+from+facetable"">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select * from...
FAILED tests/test_html.py::test_query_page_truncates - assert equals failed
FAILED tests/test_html.py::test_templates_considered[/fixtures-database-fixtures.html, *database.html] - assert '<!-- Templates considered: database-fixtures.html, *database.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d599......
FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href=""/fixtures.csv?sql=select+1&amp;_size=max"">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link rel=""stylesheet"" href=""/-/static/ap...se""}, ""view...
FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures-expected_decoded_object1] - AssertionError: assert equals failed
FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - KeyError: 'Access-Control-Allow-Origin'
FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures-expected_extra_body_script1] - AssertionError: assert equals failed
FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
FAILED tests/test_plugins.py::test_view_names[/fixtures-database] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_view_names[/fixtures?sql=select+1-database] - AssertionError: assert equals failed
======================================================================= 11 failed, 1301 passed, 2 skipped, 1 xfailed in 64.10s (0:01:04) =======================================================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/issues/2130#issuecomment-1668695958,https://api.github.com/repos/simonw/datasette/issues/2130,1668695958,IC_kwDOBm6k_c5jdkeW,9599,simonw,2023-08-07T23:21:07Z,2023-08-07T23:21:16Z,OWNER,"Main place the code needs updating is here: https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L301-L310

Plus the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840329615,Render plugin mechanism needs `error` and `truncated` fields,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668691898,https://api.github.com/repos/simonw/datasette/issues/2118,1668691898,IC_kwDOBm6k_c5jdje6,9599,simonw,2023-08-07T23:15:14Z,2023-08-07T23:16:28Z,OWNER,"New error design:
```json
{""ok"": false, ""error"": ""Error message here"", ""rows"": [], ""columns"": [], ""truncated"": false}
```
I guess that means I need to add `""ok"": true` to the default format as well - which makes sense because table JSON has that already:

https://latest.datasette.io/fixtures/facetable.json

```json
{
  ""ok"": true,
  ""next"": null,
  ""rows"": [...]
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668687891,https://api.github.com/repos/simonw/datasette/issues/2118,1668687891,IC_kwDOBm6k_c5jdigT,9599,simonw,2023-08-07T23:11:04Z,2023-08-07T23:14:34Z,OWNER,"One of the failing tests is because this URL on my local environment returns this:

http://127.0.0.1:8001/_memory.json?sql=select+blah
```json
{""rows"": [], ""columns"": [], ""truncated"": false}
```
With a 200 status code. It should be an error.

https://latest.datasette.io/_memory.json?sql=select+blah returns a 400 status code with the following JSON:

```json
{
  ""ok"": false,
  ""database"": ""_memory"",
  ""query_name"": null,
  ""rows"": [],
  ""truncated"": false,
  ""columns"": [],
  ""query"": {
    ""sql"": ""select blah"",
    ""params"": {}
  },
  ""error"": ""no such column: blah"",
  ""private"": false,
  ""allow_execute_sql"": true,
  ""query_ms"": 2.68310698447749,
  ""source"": ""tests/fixtures.py"",
  ""source_url"": ""https://github.com/simonw/datasette/blob/main/tests/fixtures.py"",
  ""license"": ""Apache License 2.0"",
  ""license_url"": ""https://github.com/simonw/datasette/blob/main/LICENSE""
}
```
So I need a design for what this should look like for the error case.

I should consider what errors look like for alternative formats too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/issues/2129#issuecomment-1668690232,https://api.github.com/repos/simonw/datasette/issues/2129,1668690232,IC_kwDOBm6k_c5jdjE4,9599,simonw,2023-08-07T23:13:24Z,2023-08-07T23:13:24Z,OWNER,This may end up being a renderer plugin hook design question too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840324765,CSV ?sql= should indicate errors,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668688947,https://api.github.com/repos/simonw/datasette/issues/2118,1668688947,IC_kwDOBm6k_c5jdiwz,9599,simonw,2023-08-07T23:12:37Z,2023-08-07T23:12:37Z,OWNER,"https://latest.datasette.io/_memory.csv?sql=select+blah is a blank page right now:

```bash
curl -I 'https://latest.datasette.io/_memory.csv?sql=select+blah'
```
```
HTTP/2 200 
access-control-allow-origin: *
access-control-allow-headers: Authorization, Content-Type
access-control-expose-headers: Link
access-control-allow-methods: GET, POST, HEAD, OPTIONS
access-control-max-age: 3600
content-type: text/plain; charset=utf-8
x-databases: _memory, _internal, fixtures, fixtures2, extra_database, ephemeral
date: Mon, 07 Aug 2023 23:12:15 GMT
server: Google Frontend
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668578107,https://api.github.com/repos/simonw/datasette/issues/2118,1668578107,IC_kwDOBm6k_c5jdHs7,9599,simonw,2023-08-07T21:06:50Z,2023-08-07T21:06:50Z,OWNER,"15:
```
=========================================================================================== short test summary info ============================================================================================
FAILED tests/test_cli.py::test_sql_errors_logged_to_stderr - assert 0 == 1
FAILED tests/test_html.py::test_show_hide_sql_query - AssertionError: assert False
FAILED tests/test_canned_queries.py::test_magic_parameters_cannot_be_used_in_arbitrary_queries - assert 400 == 200
FAILED tests/test_html.py::test_templates_considered[/fixtures-database-fixtures.html, *database.html] - assert '<!-- Templates considered: database-fixtures.html, *database.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d599......
FAILED tests/test_html.py::test_query_json_csv_export_links - assert '<a href=""/fixtures.csv?sql=select+1&amp;_size=max"">CSV</a>' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select 1</title>\n    <link rel=""stylesheet"" href=""/-/static/ap...se""}, ""view...
FAILED tests/test_html.py::test_base_url_config[True-/fixtures?sql=select+1] - AssertionError: {
FAILED tests/test_api.py::test_cors[/fixtures.json?sql=select+blah-400] - assert 200 == 400
FAILED tests/test_html.py::test_alternate_url_json[/fixtures?sql=select+*+from+facetable-http://localhost/fixtures.json?sql=select+*+from+facetable] - assert '<link rel=""alternate"" type=""application/json+datasette"" href=""http://localhost/fixtures.json?sql=select+*+from+facetable"">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: select * from...
FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures-expected_decoded_object1] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_view_names[/fixtures-database] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_view_names[/fixtures?sql=select+1-database] - AssertionError: assert equals failed
FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures-expected_extra_body_script1] - AssertionError: assert equals failed
FAILED tests/test_html.py::test_base_url_config[False-/fixtures?sql=select+1] - AssertionError: {
FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query'
FAILED tests/test_html.py::test_alternate_url_json[/fixtures-http://localhost/fixtures.json] - KeyError: 'link'
============================================================================ 15 failed, 1297 passed, 2 skipped, 1 xfailed in 58.15s ============================================================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2118#issuecomment-1668568445,https://api.github.com/repos/simonw/datasette/issues/2118,1668568445,IC_kwDOBm6k_c5jdFV9,9599,simonw,2023-08-07T20:57:46Z,2023-08-07T20:57:46Z,OWNER,17 failing tests now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/datasette/pull/2125#issuecomment-1668187546,https://api.github.com/repos/simonw/datasette/issues/2125,1668187546,IC_kwDOBm6k_c5jboWa,49699333,dependabot[bot],2023-08-07T16:20:26Z,2023-08-07T16:20:26Z,CONTRIBUTOR,"Looks like sphinx is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1833193570,Bump sphinx from 6.1.3 to 7.1.2,
https://github.com/simonw/datasette/pull/2121#issuecomment-1668186872,https://api.github.com/repos/simonw/datasette/issues/2121,1668186872,IC_kwDOBm6k_c5jboL4,49699333,dependabot[bot],2023-08-07T16:20:19Z,2023-08-07T16:20:19Z,CONTRIBUTOR,"Looks like furo is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1824399610,Bump furo from 2023.3.27 to 2023.7.26,
https://github.com/simonw/datasette/pull/2098#issuecomment-1668186815,https://api.github.com/repos/simonw/datasette/issues/2098,1668186815,IC_kwDOBm6k_c5jboK_,49699333,dependabot[bot],2023-08-07T16:20:18Z,2023-08-07T16:20:18Z,CONTRIBUTOR,"Looks like blacken-docs is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1796830110,Bump blacken-docs from 1.14.0 to 1.15.0,
https://github.com/simonw/datasette/pull/2128#issuecomment-1668143992,https://api.github.com/repos/simonw/datasette/issues/2128,1668143992,IC_kwDOBm6k_c5jbdt4,22429695,codecov[bot],2023-08-07T15:58:08Z,2023-08-07T15:58:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`adf54f5`)](https://app.codecov.io/gh/simonw/datasette/commit/adf54f5c808c16e9ad812c3a4376f35cf829a93c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`935d336`)](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2128   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1839766197,"Bump blacken-docs, furo, blacken-docs",
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1668113177,IC_kwDOCGYnMM5jbWMZ,25778,eyeseast,2023-08-07T15:41:49Z,2023-08-07T15:41:49Z,CONTRIBUTOR,I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats,
https://github.com/simonw/datasette/issues/2127#issuecomment-1667011607,https://api.github.com/repos/simonw/datasette/issues/2127,1667011607,IC_kwDOBm6k_c5jXJQX,9599,simonw,2023-08-07T00:02:49Z,2023-08-07T00:02:49Z,OWNER,"This is pretty neat:

https://github.com/simonw/datasette/blob/2f9038a831a3510d4c9ab39a12d96259b3a55bc7/datasette/views/database.py#L157-L206","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838469176,Context base class to support documenting the context,
https://github.com/simonw/datasette/issues/2126#issuecomment-1666912107,https://api.github.com/repos/simonw/datasette/issues/2126,1666912107,IC_kwDOBm6k_c5jWw9r,36199671,ctsrc,2023-08-06T16:27:34Z,2023-08-06T16:27:34Z,NONE,"And in similar fashion, how can I assign the `edit-tiddlywiki` permission to my user `myuser` in `metadata.yml` / `metadata.json`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838266862,Permissions in metadata.yml / metadata.json,
https://github.com/simonw/datasette/pull/2125#issuecomment-1662228666,https://api.github.com/repos/simonw/datasette/issues/2125,1662228666,IC_kwDOBm6k_c5jE5i6,22429695,codecov[bot],2023-08-02T13:36:22Z,2023-08-02T13:36:22Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`730f554`)](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2125   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1833193570,Bump sphinx from 6.1.3 to 7.1.2,
https://github.com/simonw/datasette/pull/2124#issuecomment-1662215579,https://api.github.com/repos/simonw/datasette/issues/2124,1662215579,IC_kwDOBm6k_c5jE2Wb,49699333,dependabot[bot],2023-08-02T13:28:43Z,2023-08-02T13:28:43Z,CONTRIBUTOR,Superseded by #2125.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1826424151,Bump sphinx from 6.1.3 to 7.1.1,
https://github.com/simonw/datasette/pull/1159#issuecomment-1658968449,https://api.github.com/repos/simonw/datasette/issues/1159,1658968449,IC_kwDOBm6k_c5i4dmB,9599,simonw,2023-07-31T18:57:58Z,2023-07-31T18:57:58Z,OWNER,"Added to the 1.0 milestone because: https://news.ycombinator.com/item?id=36932876#36939734

>  Hah, wow, it looks like I've been procrastinating on making a decision if I like that or not for three years!
>
> I'll add it to the Datasette 1.0 milestone so it definitely gets my attention before shipping that release. ","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",774332247,Improve the display of facets information,
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1656696679,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,1656696679,IC_kwDOD079W85ivy9n,319473,coldclimate,2023-07-29T10:10:29Z,2023-07-29T10:10:29Z,NONE,"+1 to getting this merged down.

For future googlers, I installed by...
```
git clone git@github.com:RhetTbull/dogsheep-photos.git
cd dogsheep-photos
git checkout update_for_bigsur
python setup.py install
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/dogsheep-photos/pull/38#issuecomment-1656694944,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/38,1656694944,IC_kwDOD079W85ivyig,319473,coldclimate,2023-07-29T10:01:19Z,2023-07-29T10:01:19Z,NONE,Duplicate of https://github.com/dogsheep/dogsheep-photos/pull/36 - closing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1827427757,photos-to-sql not found?,
https://github.com/dogsheep/dogsheep-photos/pull/38#issuecomment-1656694854,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/38,1656694854,IC_kwDOD079W85ivyhG,319473,coldclimate,2023-07-29T10:00:45Z,2023-07-29T10:00:45Z,NONE,Ran across https://github.com/dogsheep/dogsheep-photos/issues/33 which is the same subject.  My PR just fixes docs,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1827427757,photos-to-sql not found?,
https://github.com/simonw/datasette/pull/2124#issuecomment-1655691977,https://api.github.com/repos/simonw/datasette/issues/2124,1655691977,IC_kwDOBm6k_c5ir9rJ,22429695,codecov[bot],2023-07-28T13:33:54Z,2023-07-28T13:33:54Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`9388102`)](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2124   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1826424151,Bump sphinx from 6.1.3 to 7.1.1,
https://github.com/simonw/datasette/pull/2107#issuecomment-1655678215,https://api.github.com/repos/simonw/datasette/issues/2107,1655678215,IC_kwDOBm6k_c5ir6UH,49699333,dependabot[bot],2023-07-28T13:23:16Z,2023-07-28T13:23:16Z,CONTRIBUTOR,Superseded by #2124.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1820346348,Bump sphinx from 6.1.3 to 7.1.0,
https://github.com/simonw/datasette/pull/2121#issuecomment-1653666188,https://api.github.com/repos/simonw/datasette/issues/2121,1653666188,IC_kwDOBm6k_c5ikPGM,22429695,codecov[bot],2023-07-27T13:48:43Z,2023-07-27T13:48:43Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`3aca01f`)](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2121   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5361     5361           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1824399610,Bump furo from 2023.3.27 to 2023.7.26,
https://github.com/simonw/datasette/pull/2077#issuecomment-1653652665,https://api.github.com/repos/simonw/datasette/issues/2077,1653652665,IC_kwDOBm6k_c5ikLy5,49699333,dependabot[bot],2023-07-27T13:40:52Z,2023-07-27T13:40:52Z,CONTRIBUTOR,Superseded by #2121.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20,
https://github.com/simonw/datasette/issues/2112#issuecomment-1652751140,https://api.github.com/repos/simonw/datasette/issues/2112,1652751140,IC_kwDOBm6k_c5igvsk,9599,simonw,2023-07-27T00:52:53Z,2023-07-27T00:52:53Z,OWNER,"Maybe I teach `datasette.render_template()` to take a dataclass instance as an optional alternative to a dictionary, and if it gets one it turns it into something that works well in the template context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=...,
https://github.com/simonw/datasette/issues/2112#issuecomment-1652746185,https://api.github.com/repos/simonw/datasette/issues/2112,1652746185,IC_kwDOBm6k_c5igufJ,9599,simonw,2023-07-27T00:46:37Z,2023-07-27T00:46:37Z,OWNER,"Got Claude to generate this code:
```python
from dataclasses import dataclass, field

@dataclass
class QueryContext:
    database: str = field(metadata={""help"": ""The name of the database being queried""})
    query: dict = field(metadata={""help"": ""The SQL query object containing the `sql` string""})
    canned_query: str = field(metadata={""help"": ""The name of the canned query if this is a canned query""})
    private: bool = field(metadata={""help"": ""Boolean indicating if this is a private database""})
    urls: dict = field(metadata={""help"": ""Object containing URL helpers like `database()`""})
    canned_write: bool = field(metadata={""help"": ""Boolean indicating if this canned query allows writes""})
    db_is_immutable: bool = field(metadata={""help"": ""Boolean indicating if this database is immutable""})
    error: str = field(metadata={""help"": ""Any query error message""})
    hide_sql: bool = field(metadata={""help"": ""Boolean indicating if the SQL should be hidden""})
    show_hide_link: str = field(metadata={""help"": ""The URL to toggle showing/hiding the SQL""})
    show_hide_text: str = field(metadata={""help"": ""The text for the show/hide SQL link""})
    editable: bool = field(metadata={""help"": ""Boolean indicating if the SQL can be edited""})
    allow_execute_sql: bool = field(metadata={""help"": ""Boolean indicating if custom SQL can be executed""})
    tables: list = field(metadata={""help"": ""List of table objects in the database""})
    named_parameter_values: dict = field(metadata={""help"": ""Dictionary of parameter names/values""})
    csrftoken: callable = field(metadata={""help"": ""Function to generate a CSRF token""})
    edit_sql_url: str = field(metadata={""help"": ""URL to edit the SQL for a canned query""})
    display_rows: list = field(metadata={""help"": ""List of result rows to display""})
    columns: list = field(metadata={""help"": ""List of column names""})
    renderers: dict = field(metadata={""help"": ""Dictionary of renderer name to URL""})
    url_csv: str = field(metadata={""help"": ""URL for CSV export""})
    metadata: dict = field(metadata={""help"": ""Metadata about the query/database""})
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=...,
https://github.com/simonw/datasette/issues/2112#issuecomment-1652723029,https://api.github.com/repos/simonw/datasette/issues/2112,1652723029,IC_kwDOBm6k_c5igo1V,9599,simonw,2023-07-27T00:25:49Z,2023-07-27T00:25:49Z,OWNER,"Getting the query page working with all of the necessary HTML context variables is a slog.

I uploaded `query.html` to Claude and asked it ""List of every template context variable with a detailed description of each one"":

<img width=""518"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/9c7f3292-d3f3-49bc-9263-dc39bed5aa58"">

> Based on the template code provided, here is a list of the template context variables and a description of each one:
> 
> - `database` - The name of the database being queried.
> - `query` - The SQL query object containing the `sql` string.
> - `canned_query` - The name of the canned query if this is a canned query. 
> - `private` - Boolean indicating if this is a private database.
> - `urls` - Object containing URL helpers like `database()`.
> - `canned_write` - Boolean indicating if this canned query allows writes.
> - `db_is_immutable` - Boolean indicating if this database is immutable. 
> - `error` - Any query error message.
> - `hide_sql` - Boolean indicating if the SQL should be hidden. 
> - `show_hide_link` - The URL to toggle showing/hiding the SQL.
> - `show_hide_text` - The text for the show/hide SQL link. 
> - `editable` - Boolean indicating if the SQL can be edited.
> - `allow_execute_sql` - Boolean indicating if custom SQL can be executed.
> - `tables` - List of table objects in the database.
> - `named_parameter_values` - Dictionary of parameter names/values.
> - `csrftoken()` - Function to generate a CSRF token.
> - `edit_sql_url` - URL to edit the SQL for a canned query.
> - `display_rows` - List of result rows to display.
> - `columns` - List of column names. 
> - `renderers` - Dictionary of renderer name to URL.
> - `url_csv` - URL for CSV export.
> - `metadata` - Metadata about the query/database.
> - `crumbs` - Breadcrumb navigation object.
> 
> So in summary, it includes variables for the database, query, results, URLs, permissions, and metadata to power the template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=...,
https://github.com/simonw/datasette/issues/2112#issuecomment-1652722208,https://api.github.com/repos/simonw/datasette/issues/2112,1652722208,IC_kwDOBm6k_c5igoog,9599,simonw,2023-07-27T00:24:50Z,2023-07-27T00:25:16Z,OWNER,"I changed my mind, I'm not going to do this with a format hook.

Also: https://github.com/simonw/datasette/issues/2109#issuecomment-1652715823

I've decided to let HTML have its own special context. I'm going to try formalizing that as a `@dataclass` with fields with `metadata={""help"": ...}` to see what that looks like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=...,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652721111,https://api.github.com/repos/simonw/datasette/issues/2109,1652721111,IC_kwDOBm6k_c5igoXX,9599,simonw,2023-07-27T00:23:28Z,2023-07-27T00:23:37Z,OWNER,"Once again I'm tempted to formalize the extra HTML context as a `dataclass` so I can ensure it is documented correctly.

Here's an example of Hugging Face doing that: https://github.com/huggingface/transformers/blob/1689aea73346816b936b84932e12b774974e61a6/src/transformers/training_args.py#L622C1-L624

```python
from dataclasses import dataclass, field

@dataclass
class TrainingArguments:
    output_dir: str = field(
        metadata={""help"": ""The output directory where the model predictions and checkpoints will be written.""},
    )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652715823,https://api.github.com/repos/simonw/datasette/issues/2109,1652715823,IC_kwDOBm6k_c5ignEv,9599,simonw,2023-07-27T00:17:04Z,2023-07-27T00:18:07Z,OWNER,"New decision: I had originally decided that the HTML view would just use data that could otherwise be extracted from the JSON view if you fed in enough extras.

I've changed my mind. I'm OK with the HTML view getting a few bonus things available in its context, provided those are clearly documented for template authors.

This should help avoid me having to make many changes to the templates themselves.

HTML-specific stuff will be things like `csrftoken()` and `edit_sql_url` and `show_hide_link` and `database_color` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/pull/2118#issuecomment-1652681136,https://api.github.com/repos/simonw/datasette/issues/2118,1652681136,IC_kwDOBm6k_c5igemw,9599,simonw,2023-07-26T23:30:44Z,2023-07-26T23:30:44Z,OWNER,"The `_shape=` stuff should use `json_renderer` instead - that's how the table view did it:
https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d#diff-5c9ef29c33ed0fde413565b23fa258d60fc3a2bb205b016db9e915c9bd5ecfb3

https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/table.py#L822-L850

Instead of:

https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/database.py#L239-L288","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views,
https://github.com/simonw/sqlite-utils/issues/581#issuecomment-1652496702,https://api.github.com/repos/simonw/sqlite-utils/issues/581,1652496702,IC_kwDOCGYnMM5ifxk-,9599,simonw,2023-07-26T21:07:45Z,2023-07-26T21:07:45Z,OWNER,Docs: https://sqlite-utils.datasette.io/en/latest/cli.html#using-the-debugger,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823160748,`sqlite-utils convert --pdb` option,
https://github.com/simonw/datasette/issues/2111#issuecomment-1652407208,https://api.github.com/repos/simonw/datasette/issues/2111,1652407208,IC_kwDOBm6k_c5ifbuo,9599,simonw,2023-07-26T19:54:20Z,2023-07-26T19:54:20Z,OWNER,"I implemented `/content` and `/content.json` but I left `/content?sql=...` as this:

https://github.com/simonw/datasette/blob/2e40a506ad45b44fd7642474f630a31ef18b5911/datasette/views/database.py#L220-L221","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=...,
https://github.com/simonw/datasette/issues/2111#issuecomment-1652354606,https://api.github.com/repos/simonw/datasette/issues/2111,1652354606,IC_kwDOBm6k_c5ifO4u,9599,simonw,2023-07-26T19:16:35Z,2023-07-26T19:17:45Z,OWNER,"I just noticed that this URL: https://latest.datasette.io/fixtures.csv

Returns a 500 error right now!

<img width=""275"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/bd560821-07ba-4e8f-8472-9e677294687d"">

It's fine with a `?sql=` query: https://latest.datasette.io/fixtures.csv?sql=select+*+from+facetable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=...,
https://github.com/simonw/datasette/issues/2110#issuecomment-1652330111,https://api.github.com/repos/simonw/datasette/issues/2110,1652330111,IC_kwDOBm6k_c5ifI5_,9599,simonw,2023-07-26T18:55:31Z,2023-07-26T18:55:31Z,OWNER,Changed my mind on this - I'm going to have the `query_view` mapped to `/db` but have the first code on there notice if `?sql=` is missing and return a `database_view()` function instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822936521,Merge database index page and query view,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652325193,https://api.github.com/repos/simonw/datasette/issues/2109,1652325193,IC_kwDOBm6k_c5ifHtJ,9599,simonw,2023-07-26T18:51:15Z,2023-07-26T18:51:15Z,OWNER,"Here's the code I'm going to be entirely replacing:

https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L213-L530

Plus this weird class in `views/table.py`:

https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/table.py#L701-L749","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/issues/2114#issuecomment-1652321419,https://api.github.com/repos/simonw/datasette/issues/2114,1652321419,IC_kwDOBm6k_c5ifGyL,9599,simonw,2023-07-26T18:48:03Z,2023-07-26T18:48:03Z,OWNER,"This is also where I'll bring back writable canned queries:

https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L281-L334","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822940263,Implement canned queries against new query JSON work,
https://github.com/simonw/datasette/issues/2111#issuecomment-1652318269,https://api.github.com/repos/simonw/datasette/issues/2111,1652318269,IC_kwDOBm6k_c5ifGA9,9599,simonw,2023-07-26T18:45:23Z,2023-07-26T18:45:23Z,OWNER,"To avoid confusion I'll start by having `/content` return a HTML ""TODO: implement this"" message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=...,
https://github.com/simonw/datasette/issues/2116#issuecomment-1652315926,https://api.github.com/repos/simonw/datasette/issues/2116,1652315926,IC_kwDOBm6k_c5ifFcW,9599,simonw,2023-07-26T18:43:17Z,2023-07-26T18:43:17Z,OWNER,"Tests pass, and manually tested like this too:
```bash
datasette -i pelicans.db
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function,
https://github.com/simonw/datasette/issues/2116#issuecomment-1652304289,https://api.github.com/repos/simonw/datasette/issues/2116,1652304289,IC_kwDOBm6k_c5ifCmh,9599,simonw,2023-07-26T18:33:07Z,2023-07-26T18:33:07Z,OWNER,"This runs three tests:
```bash
pytest -k test_database_download
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function,
https://github.com/simonw/datasette/issues/2116#issuecomment-1652303471,https://api.github.com/repos/simonw/datasette/issues/2116,1652303471,IC_kwDOBm6k_c5ifCZv,9599,simonw,2023-07-26T18:32:24Z,2023-07-26T18:32:24Z,OWNER,https://github.com/simonw/datasette/blob/dc5171eb1b1d9f1d55e367f8a4d93edb55a43351/datasette/views/database.py#L172-L212,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function,
https://github.com/simonw/datasette/issues/2106#issuecomment-1652298879,https://api.github.com/repos/simonw/datasette/issues/2106,1652298879,IC_kwDOBm6k_c5ifBR_,9599,simonw,2023-07-26T18:28:33Z,2023-07-26T18:28:33Z,OWNER,"Applied the same fix as here:
- https://github.com/simonw/llm/issues/136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816857442,`datasette install -e` option,
https://github.com/simonw/datasette/pull/2053#issuecomment-1652296467,https://api.github.com/repos/simonw/datasette/issues/2053,1652296467,IC_kwDOBm6k_c5ifAsT,9599,simonw,2023-07-26T18:26:44Z,2023-07-26T18:26:44Z,OWNER,"I'm abandoning this branch in favour of a fresh attempt, described here:
- https://github.com/simonw/datasette/issues/2109

I'll copy bits and pieces of this branch across as-needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652295866,https://api.github.com/repos/simonw/datasette/issues/2109,1652295866,IC_kwDOBm6k_c5ifAi6,9599,simonw,2023-07-26T18:26:18Z,2023-07-26T18:26:18Z,OWNER,"I'm going to do this work in a fresh branch, copying bits over from here as needed:
- https://github.com/simonw/datasette/pull/2053","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/issues/2109#issuecomment-1652294920,https://api.github.com/repos/simonw/datasette/issues/2109,1652294920,IC_kwDOBm6k_c5ifAUI,9599,simonw,2023-07-26T18:25:34Z,2023-07-26T18:25:34Z,OWNER,"OK, these issues will do for the plan.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651904060,https://api.github.com/repos/simonw/datasette/issues/2053,1651904060,IC_kwDOBm6k_c5idg48,9599,simonw,2023-07-26T14:19:00Z,2023-07-26T15:25:15Z,OWNER,"Worth noting that the `register_output_renderer()` is actually pretty easy to extend, because it returns a dictionary which could have more keys (like the required set of extras) added to it:
```python
@hookimpl
def register_output_renderer(datasette):
    return {
        ""extension"": ""test"",
        ""render"": render_demo,
        ""can_render"": can_render_demo,  # Optional
    }
```
https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-output-renderer-datasette

One slight hiccup with that plugin hook is this:

> rows - list of `sqlite3.Row` objects

I could turn that into a Datasette defined object that behaves like a [sqlite3.Row](https://docs.python.org/3/library/sqlite3.html#sqlite3.Row) though, which would give me extra flexibility in the future.

A bit tricky though since it's implemented in C for performance: https://github.com/python/cpython/blob/b0202a4e5d6b629ba5acbc703e950f08ebaf07df/Modules/_sqlite/row.c

Pasted that into Claude for the following explanation:

> -   pysqlite_Row is the structure defining the Row object. It contains the tuple of data (self->data) and description of columns (self->description).
> -   pysqlite_row_new() is the constructor which creates a new Row object given a cursor and tuple of data.
> -   pysqlite_row_dealloc() frees the memory when Row object is deleted.
> -   pysqlite_row_keys() returns the column names of the row.
> -   pysqlite_row_length() and pysqlite_row_subscript() implement sequence like behavior to access row elements by index.
> -   pysqlite_row_subscript() also allows accessing by column name by doing a lookup in description.
> -   pysqlite_row_hash() and pysqlite_row_richcompare() implement equality checks and hash function.

I could use protocols in Python to make my own `DatasetteRow` which can be used interchangeably with `sqlite3.Row` - https://docs.python.org/3/library/typing.html#typing.Protocol

Turned this into a TIL: https://til.simonwillison.net/python/protocols","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651894668,https://api.github.com/repos/simonw/datasette/issues/2053,1651894668,IC_kwDOBm6k_c5idemM,9599,simonw,2023-07-26T14:14:34Z,2023-07-26T14:14:34Z,OWNER,"Another point of confusion is how `/content` sometimes serves the database index page (with a list of tables) and sometimes solves the results of a query.

I could resolve this by turning the information on the index page into extras, which can optionally be requested any time a query is run but default to being shown if there is no query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651883505,https://api.github.com/repos/simonw/datasette/issues/2053,1651883505,IC_kwDOBm6k_c5idb3x,9599,simonw,2023-07-26T14:08:20Z,2023-07-26T14:08:20Z,OWNER,"I think the hardest part of getting this working is dealing with the different formats.

Idea: refactor `.html` as a format (since it's by far the most complex) and tweak the plugin hook a bit as part of that, then use what I learn from that to get the other formats working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1651874649,https://api.github.com/repos/simonw/datasette/issues/2053,1651874649,IC_kwDOBm6k_c5idZtZ,9599,simonw,2023-07-26T14:03:37Z,2023-07-26T14:03:37Z,OWNER,"Big chunk of commented-out code I just removed:
```python

    import pdb

    pdb.set_trace()

    if isinstance(output, dict) and output.get(""ok"") is False:
        # TODO: Other error codes?

        response.status_code = 400

    if datasette.cors:
        add_cors_headers(response.headers)

    return response

    # registry = Registry(
    #     extra_count,
    #     extra_facet_results,
    #     extra_facets_timed_out,
    #     extra_suggested_facets,
    #     facet_instances,
    #     extra_human_description_en,
    #     extra_next_url,
    #     extra_columns,
    #     extra_primary_keys,
    #     run_display_columns_and_rows,
    #     extra_display_columns,
    #     extra_display_rows,
    #     extra_debug,
    #     extra_request,
    #     extra_query,
    #     extra_metadata,
    #     extra_extras,
    #     extra_database,
    #     extra_table,
    #     extra_database_color,
    #     extra_table_actions,
    #     extra_filters,
    #     extra_renderers,
    #     extra_custom_table_templates,
    #     extra_sorted_facet_results,
    #     extra_table_definition,
    #     extra_view_definition,
    #     extra_is_view,
    #     extra_private,
    #     extra_expandable_columns,
    #     extra_form_hidden_args,
    # )

    results = await registry.resolve_multi(
        [""extra_{}"".format(extra) for extra in extras]
    )
    data = {
        ""ok"": True,
        ""next"": next_value and str(next_value) or None,
    }
    data.update(
        {
            key.replace(""extra_"", """"): value
            for key, value in results.items()
            if key.startswith(""extra_"") and key.replace(""extra_"", """") in extras
        }
    )
    raw_sqlite_rows = rows[:page_size]
    data[""rows""] = [dict(r) for r in raw_sqlite_rows]

    private = False
    if canned_query:
        # Respect canned query permissions
        visible, private = await datasette.check_visibility(
            request.actor,
            permissions=[
                (""view-query"", (database, canned_query)),
                (""view-database"", database),
                ""view-instance"",
            ],
        )
        if not visible:
            raise Forbidden(""You do not have permission to view this query"")

    else:
        await datasette.ensure_permissions(request.actor, [(""execute-sql"", database)])

    # If there's no sql, show the database index page
    if not sql:
        return await database_index_view(request, datasette, db)

    validate_sql_select(sql)

    # Extract any :named parameters
    named_parameters = named_parameters or await derive_named_parameters(db, sql)
    named_parameter_values = {
        named_parameter: params.get(named_parameter) or """"
        for named_parameter in named_parameters
        if not named_parameter.startswith(""_"")
    }

    # Set to blank string if missing from params
    for named_parameter in named_parameters:
        if named_parameter not in params and not named_parameter.startswith(""_""):
            params[named_parameter] = """"

    extra_args = {}
    if params.get(""_timelimit""):
        extra_args[""custom_time_limit""] = int(params[""_timelimit""])
    if _size:
        extra_args[""page_size""] = _size

    templates = [f""query-{to_css_class(database)}.html"", ""query.html""]
    if canned_query:
        templates.insert(
            0,
            f""query-{to_css_class(database)}-{to_css_class(canned_query)}.html"",
        )

    query_error = None

    # Execute query - as write or as read
    if write:
        raise NotImplementedError(""Write queries not yet implemented"")
        # if request.method == ""POST"":
        #     # If database is immutable, return an error
        #     if not db.is_mutable:
        #         raise Forbidden(""Database is immutable"")
        #     body = await request.post_body()
        #     body = body.decode(""utf-8"").strip()
        #     if body.startswith(""{"") and body.endswith(""}""):
        #         params = json.loads(body)
        #         # But we want key=value strings
        #         for key, value in params.items():
        #             params[key] = str(value)
        #     else:
        #         params = dict(parse_qsl(body, keep_blank_values=True))
        #     # Should we return JSON?
        #     should_return_json = (
        #         request.headers.get(""accept"") == ""application/json""
        #         or request.args.get(""_json"")
        #         or params.get(""_json"")
        #     )
        #     if canned_query:
        #         params_for_query = MagicParameters(params, request, self.ds)
        #     else:
        #         params_for_query = params
        #     ok = None
        #     try:
        #         cursor = await self.ds.databases[database].execute_write(
        #             sql, params_for_query
        #         )
        #         message = metadata.get(
        #             ""on_success_message""
        #         ) or ""Query executed, {} row{} affected"".format(
        #             cursor.rowcount, """" if cursor.rowcount == 1 else ""s""
        #         )
        #         message_type = self.ds.INFO
        #         redirect_url = metadata.get(""on_success_redirect"")
        #         ok = True
        #     except Exception as e:
        #         message = metadata.get(""on_error_message"") or str(e)
        #         message_type = self.ds.ERROR
        #         redirect_url = metadata.get(""on_error_redirect"")
        #         ok = False
        #     if should_return_json:
        #         return Response.json(
        #             {
        #                 ""ok"": ok,
        #                 ""message"": message,
        #                 ""redirect"": redirect_url,
        #             }
        #         )
        #     else:
        #         self.ds.add_message(request, message, message_type)
        #         return self.redirect(request, redirect_url or request.path)
        # else:

        #     async def extra_template():
        #         return {
        #             ""request"": request,
        #             ""db_is_immutable"": not db.is_mutable,
        #             ""path_with_added_args"": path_with_added_args,
        #             ""path_with_removed_args"": path_with_removed_args,
        #             ""named_parameter_values"": named_parameter_values,
        #             ""canned_query"": canned_query,
        #             ""success_message"": request.args.get(""_success"") or """",
        #             ""canned_write"": True,
        #         }

        #     return (
        #         {
        #             ""database"": database,
        #             ""rows"": [],
        #             ""truncated"": False,
        #             ""columns"": [],
        #             ""query"": {""sql"": sql, ""params"": params},
        #             ""private"": private,
        #         },
        #         extra_template,
        #         templates,
        #     )

    # Not a write
    rows = []
    if canned_query:
        params_for_query = MagicParameters(params, request, datasette)
    else:
        params_for_query = params
    try:
        results = await datasette.execute(
            database, sql, params_for_query, truncate=True, **extra_args
        )
        columns = [r[0] for r in results.description]
        rows = list(results.rows)
    except sqlite3.DatabaseError as e:
        query_error = e
        results = None
        columns = []

    allow_execute_sql = await datasette.permission_allowed(
        request.actor, ""execute-sql"", database
    )

    format_ = request.url_vars.get(""format"") or ""html""

    if format_ == ""csv"":
        raise NotImplementedError(""CSV format not yet implemented"")
    elif format_ in datasette.renderers.keys():
        # Dispatch request to the correct output format renderer
        # (CSV is not handled here due to streaming)
        result = call_with_supported_arguments(
            datasette.renderers[format_][0],
            datasette=datasette,
            columns=columns,
            rows=rows,
            sql=sql,
            query_name=None,
            database=db.name,
            table=None,
            request=request,
            view_name=""table"",  # TODO: should this be ""query""?
            # These will be deprecated in Datasette 1.0:
            args=request.args,
            data={
                ""rows"": rows,
            },  # TODO what should this be?
        )
        result = await await_me_maybe(result)
        if result is None:
            raise NotFound(""No data"")
        if isinstance(result, dict):
            r = Response(
                body=result.get(""body""),
                status=result.get(""status_code"") or 200,
                content_type=result.get(""content_type"", ""text/plain""),
                headers=result.get(""headers""),
            )
        elif isinstance(result, Response):
            r = result
            # if status_code is not None:
            #     # Over-ride the status code
            #     r.status = status_code
        else:
            assert False, f""{result} should be dict or Response""
    elif format_ == ""html"":
        headers = {}
        templates = [f""query-{to_css_class(database)}.html"", ""query.html""]
        template = datasette.jinja_env.select_template(templates)
        alternate_url_json = datasette.absolute_url(
            request,
            datasette.urls.path(path_with_format(request=request, format=""json"")),
        )
        headers.update(
            {
                ""Link"": '{}; rel=""alternate""; type=""application/json+datasette""'.format(
                    alternate_url_json
                )
            }
        )
        r = Response.html(
            await datasette.render_template(
                template,
                dict(
                    data,
                    append_querystring=append_querystring,
                    path_with_replaced_args=path_with_replaced_args,
                    fix_path=datasette.urls.path,
                    settings=datasette.settings_dict(),
                    # TODO: review up all of these hacks:
                    alternate_url_json=alternate_url_json,
                    datasette_allow_facet=(
                        ""true"" if datasette.setting(""allow_facet"") else ""false""
                    ),
                    is_sortable=any(c[""sortable""] for c in data[""display_columns""]),
                    allow_execute_sql=await datasette.permission_allowed(
                        request.actor, ""execute-sql"", resolved.db.name
                    ),
                    query_ms=1.2,
                    select_templates=[
                        f""{'*' if template_name == template.name else ''}{template_name}""
                        for template_name in templates
                    ],
                ),
                request=request,
                view_name=""table"",
            ),
            headers=headers,
        )
    else:
        assert False, ""Invalid format: {}"".format(format_)
    # if next_url:
    #     r.headers[""link""] = f'<{next_url}>; rel=""next""'
    return r
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2107#issuecomment-1649863533,https://api.github.com/repos/simonw/datasette/issues/2107,1649863533,IC_kwDOBm6k_c5iVutt,22429695,codecov[bot],2023-07-25T13:37:07Z,2023-07-25T13:37:07Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`278ac91`)](https://app.codecov.io/gh/simonw/datasette/commit/278ac91a4d68da333a6ed74a2c75b3ec9db8bd19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`7fc79ba`)](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2107   +/-   ##
=======================================
  Coverage   92.38%   92.38%           
=======================================
  Files          39       39           
  Lines        5805     5805           
=======================================
  Hits         5363     5363           
  Misses        442      442           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1820346348,Bump sphinx from 6.1.3 to 7.1.0,
https://github.com/simonw/datasette/pull/2075#issuecomment-1649849249,https://api.github.com/repos/simonw/datasette/issues/2075,1649849249,IC_kwDOBm6k_c5iVrOh,49699333,dependabot[bot],2023-07-25T13:28:35Z,2023-07-25T13:28:35Z,CONTRIBUTOR,Superseded by #2107.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1710164693,Bump sphinx from 6.1.3 to 7.0.1,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648339661,IC_kwDOCGYnMM5iP6rN,25778,eyeseast,2023-07-24T17:44:30Z,2023-07-24T17:44:30Z,CONTRIBUTOR,"> A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV.

This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this:

```sh
sqlite-utils insert data.db table file.geojson --format geojson
# ... explore and manipulate in Datasette
sqlite-utils query data.db ... --format geojson > output.geojson
```

This would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648325682,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648325682,IC_kwDOCGYnMM5iP3Qy,9599,simonw,2023-07-24T17:33:10Z,2023-07-24T17:33:10Z,OWNER,"A related feature would be support for plugins to add new ways of _ingesting_ data - currently `sqlite-utils insert` works against JSON, newline-JSON, CSV and TSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648324312,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648324312,IC_kwDOCGYnMM5iP27Y,9599,simonw,2023-07-24T17:32:21Z,2023-07-24T17:32:21Z,OWNER,I'm currently leaning very slightly towards `--format geojson` - and explaining in the docs that `--format` is for formats added by plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats,
https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648323482,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648323482,IC_kwDOCGYnMM5iP2ua,9599,simonw,2023-07-24T17:31:56Z,2023-07-24T17:31:56Z,OWNER,"The main blocker here is coming up with a design. The challenge is cleanly integrating it with the existing format options:

https://github.com/simonw/sqlite-utils/blob/8bee14588687b66c54c7a3dfae5de2f9cc7cac3f/docs/cli-reference.rst#L114-L132

I'd like to avoid breaking backwards compatibility.

The most obvious solution is to add a `--format x` option. This is slightly confusing as `--fmt` already exists.

Or... `--fmt` could be enhanced to handle plugins too - and the existing set of formats (from the `tabulate` integration) could be refactored to use the new hook.

If I ever do release `sqlite-utils 4` I'm tempted to move that tabulate stuff into a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats,
https://github.com/dogsheep/dogsheep-beta/issues/37#issuecomment-1646950438,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/37,1646950438,IC_kwDOC8SPRc5iKngm,10352819,rprimet,2023-07-23T20:18:26Z,2023-07-23T20:18:26Z,NONE,"My bad, although I could not find how to use `url`s, I was using a function call syntax for the filters instead of jinja's pipe syntax; closing this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817281557,cannot use jinja filters in display?,
https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646753678,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646753678,IC_kwDOCGYnMM5iJ3eO,9599,simonw,2023-07-23T05:49:51Z,2023-07-23T05:49:51Z,OWNER,Done - bottom of https://sqlite-utils.datasette.io/en/latest/changelog.html#id116,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,Backfill the release notes prior to 0.4,
https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646752814,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646752814,IC_kwDOCGYnMM5iJ3Qu,9599,simonw,2023-07-23T05:43:13Z,2023-07-23T05:43:13Z,OWNER,"Using `llm` in my `sqlite-utils` local checkout:
```bash
git diff 0.2 0.3 | llm -s 'release notes as markdown bullet points'
```
> - Added a new `Table` class in the `db` module which represents a table in a SQLite database
> - The `Table` class has properties that provide information about the table, such as column names, foreign keys, and schema
> - The `Table` class also has a `count` property that returns the number of rows in the table
> - The `Table` class has a `create` method that can be used to create a new table with specified columns, primary key, and foreign keys
> - Updated the `Database` class in the `__init__` module to include the `Table` class and expose it via the `__all__` variable
> - Added new tests for creating tables and introspecting tables

```bash
git diff 0.3 0.3.1 | llm -s 'release notes as markdown bullet points'
```

> - Updated PyPI version badge to display the latest version
> - Updated Travis CI badge to link to the master branch
> - Added a badge for documentation status
> - Added a license badge
> - Added a description of the package
> - Added installation instructions
> - Added a link to the documentation
> - Added a ""Related projects"" section and linked to Datasette and csvs-to-sqlite
> - Updated the version in setup.py to 0.3.1

```bash
git diff 0.3.1 0.4 | llm -s 'release notes as markdown bullet points'
```

> - Added an `enable_fts` method to the `Table` class that enables full-text search on specified columns
> - Added a `populate_fts` method to the `Table` class that refreshes the search index for full-text search
> - Added a `search` method to the `Table` class that allows searching the table using full-text search
> - Updated the `setup.py` script to version 0.4
> - Updated the `db.py` file to include the `enable_fts`, `populate_fts`, and `search` methods
> - Added test cases for the `enable_fts`, `populate_fts`, and `search` methods in the `test_enable_fts.py` file
> - Added test cases for the `count`, `columns`, and `schema` properties in the `test_introspect.py` file
> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,Backfill the release notes prior to 0.4,
https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688748,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688748,IC_kwDOCGYnMM5iJnns,9599,simonw,2023-07-22T23:12:03Z,2023-07-22T23:13:14Z,OWNER,"Just tried this out by adding the example from the docs to my `sqlite-utils-hello-world` plugin and running:
```bash
sqlite-utils install -e ../sqlite-utils-hello-world 
```
```
Obtaining file:///Users/simon/Dropbox/Development/sqlite-utils-hello-world
...
```
Then:
```bash
sqlite-utils memory ""select hello('simon')""
```
```json
[{""hello('simon')"": ""Hello, simon!""}]
```
Also:
```pycon
>>> import sqlite_utils
>>> db = sqlite_utils.Database(memory=True)
>>> list(db.query(""select hello('simon')""))
[{""hello('simon')"": 'Hello, simon!'}]
>>> db2 = sqlite_utils.Database(memory=True, execute_plugins=False)
>>> list(db2.query(""select hello('simon')""))
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 494, in query
    cursor = self.execute(sql, params or tuple())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 512, in execute
    return self.conn.execute(sql, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlean.dbapi2.OperationalError: no such function: hello
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook,
https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688339,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688339,IC_kwDOCGYnMM5iJnhT,9599,simonw,2023-07-22T23:08:49Z,2023-07-22T23:08:49Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#prepare-connection-conn,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook,
https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646688288,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646688288,IC_kwDOCGYnMM5iJngg,9599,simonw,2023-07-22T23:08:22Z,2023-07-22T23:08:22Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,Python API ability to opt-out of connection plugins,
https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646687461,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646687461,IC_kwDOCGYnMM5iJnTl,9599,simonw,2023-07-22T23:01:44Z,2023-07-22T23:01:44Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,Python API ability to opt-out of connection plugins,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646687219,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646687219,IC_kwDOCGYnMM5iJnPz,9599,simonw,2023-07-22T22:59:36Z,2023-07-22T22:59:36Z,OWNER,Now that we have two plugin hooks I'm closing this issue (we can open other issues for further hooks).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646687103,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646687103,IC_kwDOCGYnMM5iJnN_,9599,simonw,2023-07-22T22:58:35Z,2023-07-22T22:58:35Z,OWNER,"https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn

<img width=""773"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/3211a37a-c30d-47f9-bc10-c2c3240b6f6e"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook,
https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686382,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686382,IC_kwDOCGYnMM5iJnCu,22429695,codecov[bot],2023-07-22T22:52:22Z,2023-07-22T22:56:49Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00`**% and project coverage change: **`+0.03`** :tada:
> Comparison is base [(`86a352f`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/86a352f8b713ca30a65a2048170bd510d529d8c4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.09% compared to head [(`faf398f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.12%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #573      +/-   ##
==========================================
+ Coverage   96.09%   96.12%   +0.03%     
==========================================
  Files           8        8              
  Lines        2794     2816      +22     
==========================================
+ Hits         2685     2707      +22     
  Misses        109      109              
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | |
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.36% <100.00%> (+0.01%)` | :arrow_up: |
| [sqlite\_utils/hookspecs.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook,
https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686675,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686675,IC_kwDOCGYnMM5iJnHT,9599,simonw,2023-07-22T22:54:38Z,2023-07-22T22:54:38Z,OWNER,"<img width=""779"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/c4e5769d-f1a3-427b-b260-882fce5c7205"">

Glitch in the rendered documentation from https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook,
https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646686477,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646686477,IC_kwDOCGYnMM5iJnEN,9599,simonw,2023-07-22T22:52:56Z,2023-07-22T22:52:56Z,OWNER,"Alex built this in:
- #573","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646686424,IC_kwDOCGYnMM5iJnDY,9599,simonw,2023-07-22T22:52:34Z,2023-07-22T22:52:34Z,OWNER,Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686332,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686332,IC_kwDOCGYnMM5iJnB8,9599,simonw,2023-07-22T22:52:01Z,2023-07-22T22:52:01Z,OWNER,I was literally seconds away from shipping version 3.34 but I this looks good so I'm going to try and get it in there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook,
https://github.com/simonw/sqlite-utils/issues/571#issuecomment-1646682686,https://api.github.com/repos/simonw/sqlite-utils/issues/571,1646682686,IC_kwDOCGYnMM5iJmI-,9599,simonw,2023-07-22T22:23:47Z,2023-07-22T22:23:47Z,OWNER,"Demo:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True)
>>> db[""foo""].insert({""name"": ""Cleo""})
<Table foo (name)>
>>> print(db[""foo""].schema)
CREATE TABLE [foo] (
   [name] TEXT
)
>>> db[""foo""].transform(not_null={""name""})
<Table foo (name)>
>>> print(db[""foo""].schema)
CREATE TABLE ""foo"" (
   [name] TEXT NOT NULL
)
>>> db[""foo""].transform(types={""name"": int}, keep_table=""kept"")
<Table foo (name)>
>>> print(db.schema)
CREATE TABLE ""kept"" (
   [name] TEXT NOT NULL
);
CREATE TABLE ""foo"" (
   [name] INTEGER NOT NULL
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816876211,`.transform(keep_table=...)` option,
https://github.com/simonw/sqlite-utils/issues/426#issuecomment-1646681386,https://api.github.com/repos/simonw/sqlite-utils/issues/426,1646681386,IC_kwDOCGYnMM5iJl0q,9599,simonw,2023-07-22T22:13:22Z,2023-07-22T22:13:22Z,OWNER,I'm happy with how this works on https://sqlite-utils.datasette.io/en/stable/cli-reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1205687423,CLI docs should link to Python docs and vice versa,
https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646681192,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646681192,IC_kwDOCGYnMM5iJlxo,9599,simonw,2023-07-22T22:12:08Z,2023-07-22T22:12:08Z,OWNER,Found it: https://github.com/simonw/sqlite-utils/blob/18f190e28334d821be78a1dbbf31d7610fc1f9c1/.github/workflows/test.yml#L29-L31,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910,Don't test Python 3.7 against textual,
https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646660777,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646660777,IC_kwDOCGYnMM5iJgyp,9599,simonw,2023-07-22T19:59:13Z,2023-07-22T19:59:13Z,OWNER,I don't know where that is coming from! I can't see `textual` as a dependency we are pulling in anywhere.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910,Don't test Python 3.7 against textual,
https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646659809,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646659809,IC_kwDOCGYnMM5iJgjh,9599,simonw,2023-07-22T19:53:56Z,2023-07-22T19:53:56Z,OWNER,"CLI documentation:

- https://sqlite-utils.datasette.io/en/latest/cli.html#renaming-a-table
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#rename-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table,
https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646658978,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646658978,IC_kwDOCGYnMM5iJgWi,9599,simonw,2023-07-22T19:48:54Z,2023-07-22T19:48:54Z,OWNER,"Python method documentation:
- https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.rename_table
- https://sqlite-utils.datasette.io/en/latest/python-api.html#renaming-a-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table,
https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657849,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646657849,IC_kwDOCGYnMM5iJgE5,9599,simonw,2023-07-22T19:42:02Z,2023-07-22T19:42:02Z,OWNER,"Manually testing new `rename-table` command:
```
$ sqlite-utils schema /tmp/f.db 
CREATE TABLE [foo] (
   [id] INTEGER,
   [name] TEXT
);
$ sqlite-utils rename-table /tmp/f.db bad-table hi
Error: Table ""bad-table"" could not be renamed. no such table: bad-table
$ sqlite-utils rename-table /tmp/f.db foo foo     
Error: Table ""foo"" could not be renamed. there is already another table or index with this name: foo
$ sqlite-utils rename-table /tmp/f.db foo bar
$ sqlite-utils schema /tmp/f.db                   
CREATE TABLE ""bar"" (
   [id] INTEGER,
   [name] TEXT
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table,
https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646657324,IC_kwDOCGYnMM5iJf8s,9599,simonw,2023-07-22T19:39:06Z,2023-07-22T19:39:06Z,OWNER,"> Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.

I think `keep_table=""name_of_table""` is good for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646656283,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646656283,IC_kwDOCGYnMM5iJfsb,25778,eyeseast,2023-07-22T19:32:24Z,2023-07-22T19:32:24Z,CONTRIBUTOR,Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646656246,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646656246,IC_kwDOCGYnMM5iJfr2,9599,simonw,2023-07-22T19:32:07Z,2023-07-22T19:32:07Z,OWNER,"Trying out a simple first implementation:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True, tracer=print)
PRAGMA recursive_triggers=on; None
>>> db[""foo""].insert({""id"": 1})
select name from sqlite_master where type = 'view' None
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'view' None
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'view' None
CREATE TABLE [foo] (
   [id] INTEGER
);
         None
select name from sqlite_master where type = 'view' None
INSERT INTO [foo] ([id]) VALUES (?); [1]
select name from sqlite_master where type = 'table' None
select name from sqlite_master where type = 'table' None
PRAGMA table_info([foo]) None
<Table foo (id)>
>>> db.rename_table(""foo"", ""baz"")
ALTER TABLE [foo] RENAME TO [baz] None
>>> print(db.schema)
select sql from sqlite_master where sql is not null None
CREATE TABLE ""baz"" (
   [id] INTEGER
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table,
https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646655272,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646655272,IC_kwDOCGYnMM5iJfco,9599,simonw,2023-07-22T19:25:35Z,2023-07-22T19:25:35Z,OWNER,"Here's why that test broke: https://github.com/simonw/sqlite-utils/blob/58b577279fcd5ef6ce88f88b28668dffebfe7f44/sqlite_utils/db.py#L960-L964

I added an extra `if self[name].exists()` check to the `db.create_table()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)",
https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646654818,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646654818,IC_kwDOCGYnMM5iJfVi,9599,simonw,2023-07-22T19:22:40Z,2023-07-22T19:22:40Z,OWNER,"I think this broke a test:
```
$ pytest tests/test_tracer.py                           
=============================================== test session starts ================================================
platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/simon/Dropbox/Development/sqlite-utils
plugins: icdiff-0.6, hypothesis-6.68.2
collected 2 items                                                                                                  

tests/test_tracer.py F.                                                                                      [100%]

===================================================== FAILURES =====================================================
___________________________________________________ test_tracer ____________________________________________________

    def test_tracer():
        collected = []
        db = Database(
            memory=True, tracer=lambda sql, params: collected.append((sql, params))
        )
        db[""dogs""].insert({""name"": ""Cleopaws""})
        db[""dogs""].enable_fts([""name""])
        db[""dogs""].search(""Cleopaws"")
>       assert collected == [
            (""PRAGMA recursive_triggers=on;"", None),
            (""select name from sqlite_master where type = 'view'"", None),
            (""select name from sqlite_master where type = 'table'"", None),
            (""select name from sqlite_master where type = 'view'"", None),
            (""CREATE TABLE [dogs] (\n   [name] TEXT\n);\n        "", None),
            (""select name from sqlite_master where type = 'view'"", None),
            (""INSERT INTO [dogs] ([name]) VALUES (?);"", [""Cleopaws""]),
            (""select name from sqlite_master where type = 'view'"", None),
            (
                ""CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n    [name],\n    content=[dogs]\n)"",
                None,
            ),
            (
                ""INSERT INTO [dogs_fts] (rowid, [name])\n    SELECT rowid, [name] FROM [dogs];"",
                None,
            ),
            (""select name from sqlite_master where type = 'view'"", None),
        ]
E       assert equals failed
E         [                                                  [                                                 
E           ('PRAGMA recursive_triggers=on;', None),           ('PRAGMA recursive_triggers=on;', None),        
E           (                                                                                                  
E             ""select name from sqlite_master where type =                                                     
E         'view'"",                                                                                             
E             None,                                                       ...
E         
E         ...Full output truncated (13 lines hidden), use '-vv' to show

tests/test_tracer.py:12: AssertionError
============================================= short test summary info ==============================================
FAILED tests/test_tracer.py::test_tracer - assert equals failed
=========================================== 1 failed, 1 passed in 0.05s ============================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)",
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646654383,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646654383,IC_kwDOCGYnMM5iJfOv,9599,simonw,2023-07-22T19:20:16Z,2023-07-22T19:20:16Z,OWNER,"Here's documentation for the new plugins mechanism, including a very short tutorial on writing a new plugin (inspired by https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html):

https://sqlite-utils.datasette.io/en/latest/plugins.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646654275,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646654275,IC_kwDOCGYnMM5iJfND,9599,simonw,2023-07-22T19:19:35Z,2023-07-22T19:19:35Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook,
https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653610,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653610,IC_kwDOCGYnMM5iJfCq,9599,simonw,2023-07-22T19:14:56Z,2023-07-22T19:14:56Z,OWNER,"Manual testing of CLI command as well:
```
$ sqlite-utils create-table /tmp/f.db foo id integer                         
$ sqlite-utils create-table /tmp/f.db foo id integer
Error: Table ""foo"" already exists. Use --replace to delete and replace it.
$ sqlite-utils create-table /tmp/f.db foo id integer --replace
$ sqlite-utils create-table /tmp/f.db foo id                  
$ sqlite-utils schema /tmp/f.db
CREATE TABLE [foo] (
   [id] INTEGER
);
$ sqlite-utils create-table /tmp/f.db foo id integer name str --transform
Error: column types must be one of ('INTEGER', 'TEXT', 'FLOAT', 'BLOB')
$ sqlite-utils create-table /tmp/f.db foo id integer name text --transform
$ sqlite-utils schema /tmp/f.db                                          
CREATE TABLE ""foo"" (
   [id] INTEGER,
   [name] TEXT
);
$ sqlite-utils create-table /tmp/f.db foo id integer name text --ignore   
$ sqlite-utils create-table /tmp/f.db foo id integer name text --replace
$ sqlite-utils schema /tmp/f.db                                        
CREATE TABLE [foo] (
   [id] INTEGER,
   [name] TEXT
);                                                                    
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)",
https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653382,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653382,IC_kwDOCGYnMM5iJe_G,9599,simonw,2023-07-22T19:13:20Z,2023-07-22T19:13:20Z,OWNER,"Demo:
```pycon
>>> from sqlite_utils import Database
>>> db = Database(memory=True)
>>> db[""foo""].create({""id"": int})
<Table foo (id)>
>>> db[""foo""].create({""id"": int})
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1647, in create
    self.db.create_table(
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1030, in create_table
    self.execute(sql)
  File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 510, in execute
    return self.conn.execute(sql)
           ^^^^^^^^^^^^^^^^^^^^^^
sqlean.dbapi2.OperationalError: table [foo] already exists
>>> db[""foo""].create({""id"": int}, ignore=True)
<Table foo (id)>
>>> db[""foo""].create({""id"": int, ""name"": str}, replace=True)
<Table foo (id, name)>
>>> db[""foo""].create({""id"": int, ""name"": str, ""age"": int}, transform=True)
<Table foo (id, name, age)>
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)",
https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646652105,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646652105,IC_kwDOCGYnMM5iJerJ,9599,simonw,2023-07-22T19:05:13Z,2023-07-22T19:05:13Z,OWNER,I think this is `replace=True` and `ignore=True` to match the CLI. And refactoring the CLI to use them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)",
https://github.com/simonw/datasette/issues/2106#issuecomment-1646648262,https://api.github.com/repos/simonw/datasette/issues/2106,1646648262,IC_kwDOBm6k_c5iJdvG,9599,simonw,2023-07-22T18:42:20Z,2023-07-22T18:42:20Z,OWNER,"Tested this locally with:
```bash
datasette install -e ../datasette-graphql
```
Running `datasette plugins` confirmed that the plugin had been installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816857442,`datasette install -e` option,
https://github.com/simonw/datasette/issues/2106#issuecomment-1646646931,https://api.github.com/repos/simonw/datasette/issues/2106,1646646931,IC_kwDOBm6k_c5iJdaT,9599,simonw,2023-07-22T18:34:04Z,2023-07-22T18:34:04Z,OWNER,Here's the diff for adding it to `sqlite-utils`: https://github.com/simonw/sqlite-utils/commit/ef31210bf06f920e0890e171c3198f0b0dc8d72d,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816857442,`datasette install -e` option,
https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646645990,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646645990,IC_kwDOCGYnMM5iJdLm,9599,simonw,2023-07-22T18:29:40Z,2023-07-22T18:30:16Z,OWNER,"Example plugin: https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf

```bash
sqlite-utils install https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf/archive/cd7960e476c441a3c5f619e2a44a641a39b91467.zip
```
`pyproject.toml`:
```toml
[project]
name = ""sqlite-utils-hello-world""
version = ""0.1""

[project.entry-points.sqlite_utils]
hello_world = ""sqlite_utils_hello_world""
```
`sqlite_utils_hello_world.py`:
```python
import click
import sqlite_utils


@sqlite_utils.hookimpl
def register_commands(cli):
    @cli.command()
    def hello_world():
        ""Say hello world""
        click.echo(""Hello world!"")
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook,
https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646643676,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646643676,IC_kwDOCGYnMM5iJcnc,9599,simonw,2023-07-22T18:18:24Z,2023-07-22T18:18:24Z,OWNER,"Here's where I added that to LLM: https://github.com/simonw/llm/commit/a396950f7934e82a9968703bb3ce9ab7ab62f7f8

- https://github.com/simonw/llm/issues/49","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643450,IC_kwDOCGYnMM5iJcj6,9599,simonw,2023-07-22T18:17:18Z,2023-07-22T18:17:18Z,OWNER,I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643379,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643379,IC_kwDOCGYnMM5iJciz,9599,simonw,2023-07-22T18:16:54Z,2023-07-22T18:16:54Z,OWNER,"> Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?

Yeah, it could do! That's not a terrible idea to be honest, those things have really been proliferating.

Alternatively, they could each register themselves as plugins in addition - so if you install e.g. `pocket-to-sqlite` you could then optionally also run it as `sqlite-utils pocket-to-sqlite ...`

The benefit there is for people who install `sqlite-utils` from Homebrew, where it gets its own virtual environment. They could run:

```bash
brew install sqlite-utils
sqlite-utils install pocket-to-sqlite

sqlite-utils pocket-to-sqlite ...
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642959,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642959,IC_kwDOCGYnMM5iJccP,9599,simonw,2023-07-22T18:14:49Z,2023-07-22T18:14:49Z,OWNER,Here's where those are implemented for the `create-table` CLI command: https://github.com/simonw/sqlite-utils/blob/f7af23837deab5c98dae9441d1f68318065d7d8c/sqlite_utils/cli.py#L1543-L1564,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)",
https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642666,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642666,IC_kwDOCGYnMM5iJcXq,9599,simonw,2023-07-22T18:13:19Z,2023-07-22T18:13:19Z,OWNER,"https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table

```bash
sqlite-utils create-table ... --replace
```
That also has `--ignore`:

```
  --ignore                  If table already exists, do nothing
  --replace                 If table already exists, replace it
  --transform               If table already exists, try to transform the schema
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)",
https://github.com/simonw/datasette/pull/2098#issuecomment-1646250291,https://api.github.com/repos/simonw/datasette/issues/2098,1646250291,IC_kwDOBm6k_c5iH8kz,22429695,codecov[bot],2023-07-21T21:21:00Z,2023-07-21T21:27:25Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`3a51ca9`)](https://app.codecov.io/gh/simonw/datasette/commit/3a51ca901427ac93b0ed0638fd1f82fb647a299a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39% compared to head [(`3d45262`)](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2098   +/-   ##
=======================================
  Coverage   92.39%   92.39%           
=======================================
  Files          39       39           
  Lines        5802     5802           
=======================================
  Hits         5361     5361           
  Misses        441      441           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1796830110,Bump blacken-docs from 1.14.0 to 1.15.0,
https://github.com/simonw/datasette/issues/2105#issuecomment-1646247246,https://api.github.com/repos/simonw/datasette/issues/2105,1646247246,IC_kwDOBm6k_c5iH71O,2235371,aki-k,2023-07-21T21:16:37Z,2023-07-21T21:17:09Z,NONE,"I must be doing something wrong. On page https://192.168.1.3:5432/datasette-llm/logs/_llm_migrations there's the option to filter the results, there's an Apply button. It also tries to load an URL with an extra URL element in it:

https://192.168.1.3:5432/datasette-llm/datasette-llm/logs/_llm_migrations?_sort=name&name__contains=initial","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1811824307,When reverse proxying datasette with nginx an URL element gets erronously added,
https://github.com/simonw/datasette/issues/2105#issuecomment-1643873232,https://api.github.com/repos/simonw/datasette/issues/2105,1643873232,IC_kwDOBm6k_c5h-4PQ,2235371,aki-k,2023-07-20T12:53:00Z,2023-07-20T12:53:34Z,NONE,"I forgot to add that I followed these instructions to set up the python llm:

https://simonwillison.net/2023/Jul/18/accessing-llama-2/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1811824307,When reverse proxying datasette with nginx an URL element gets erronously added,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1642808866,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1642808866,IC_kwDOCGYnMM5h60Yi,25778,eyeseast,2023-07-19T21:54:27Z,2023-07-19T21:54:27Z,CONTRIBUTOR,Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/datasette/issues/2105#issuecomment-1642013043,https://api.github.com/repos/simonw/datasette/issues/2105,1642013043,IC_kwDOBm6k_c5h3yFz,2235371,aki-k,2023-07-19T12:41:36Z,2023-07-19T12:41:36Z,NONE,"The same problem can be seen in the links:

Advanced export

JSON shape: default, array, newline-delimited, object","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1811824307,When reverse proxying datasette with nginx an URL element gets erronously added,
https://github.com/simonw/datasette/issues/2104#issuecomment-1641082395,https://api.github.com/repos/simonw/datasette/issues/2104,1641082395,IC_kwDOBm6k_c5h0O4b,15178711,asg017,2023-07-18T22:41:37Z,2023-07-18T22:41:37Z,CONTRIBUTOR,"For filtering virtual table's ""shadow tables"" (ex the FTS5 _content and most the spatialite tables),  you can use `pragma_table_list` (first appeared in SQLite 3.37 (2021-11-27), which has a `type` column that calls out `type=""shadow""` tables https://www.sqlite.org/pragma.html#pragma_table_list","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1808215339,Tables starting with an underscore should be treated as hidden,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1640826795,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1640826795,IC_kwDOCGYnMM5hzQer,76528036,J450n-4-W,2023-07-18T19:08:50Z,2023-07-18T19:08:50Z,NONE,"Came here to report this, but instead I'll confirm the issue across two terminal emulators (Gnome Terminal and Alacritty) on Pop_OS! 22.04 (currently based on Ubuntu/Gnome). Also messes up the formatting of the terminal. Can also confirm that reset fixes it until the next sqlite-utils command. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/datasette/issues/2102#issuecomment-1640064620,https://api.github.com/repos/simonw/datasette/issues/2102,1640064620,IC_kwDOBm6k_c5hwWZs,9599,simonw,2023-07-18T11:47:21Z,2023-07-18T11:47:21Z,OWNER,"I think I've figured out the problem here.

The question being asked is ""can this actor access this resource, which is within this database within this instance"".

The answer to this question needs to consider the full set of questions at once - yes they can access within this instance IF they have access to the specified table and that's the table being asked about.

But the questions are currently being asked independently, which means the plugin hook acting on `view-instance` can't see that the answer here should be yes because it's actually about a table that the actor has explicit permission to view.

So I think I may need to redesign the plugin hook to always see the full hierarchy of checks, not just a single check at a time.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638926655,IC_kwDOCGYnMM5hsAk_,9599,simonw,2023-07-17T21:42:37Z,2023-07-17T21:42:37Z,OWNER,"I really like this. I'm also interested in:
- Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette
- Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes

The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638910473,IC_kwDOCGYnMM5hr8oJ,15178711,asg017,2023-07-17T21:27:41Z,2023-07-17T21:27:41Z,CONTRIBUTOR,"Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. 

```bash
sqlite-utils insert-datasette \
  --token $DATASETTE_API_KEY \
  https://latest.datasette.io/fixtures/my-table \
  'select ...'
```


This could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system,
https://github.com/simonw/datasette/issues/2102#issuecomment-1638567228,https://api.github.com/repos/simonw/datasette/issues/2102,1638567228,IC_kwDOBm6k_c5hqo08,9599,simonw,2023-07-17T17:24:19Z,2023-07-17T17:25:12Z,OWNER,"Confirmed that this is an issue with regular Datasette signed tokens as well. I created one on https://latest.datasette.io/-/create-token with these details:
```json
{
    ""_r"": {
        ""r"": {
            ""fixtures"": {
                ""sortable"": [
                    ""vt""
                ]
            }
        }
    },
    ""a"": ""root"",
    ""d"": 3600,
    ""t"": 1689614483
}
```
Run like this:
```
curl -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsInQiOjE2ODk2MTQ0ODMsImQiOjM2MDAsIl9yIjp7InIiOnsiZml4dHVyZXMiOnsic29ydGFibGUiOlsidnQiXX19fX0.n-VGxxawz1Q0WK7sqLfhXUgcvY0' \
  https://latest.datasette.io/fixtures/sortable.json
```
Returned an HTML Forbidden page:
```html
<!DOCTYPE html>
<html>
<head>
    <title>Forbidden</title>
    <link rel=""stylesheet"" href=""/-/static/app.css?d59929"">
...
```
Same token againts `/-/actor.json` returns:
```json
{
  ""actor"": {
    ""id"": ""root"",
    ""token"": ""dstok"",
    ""_r"": {
      ""r"": {
        ""fixtures"": {
          ""sortable"": [
            ""vt""
          ]
        }
      }
    },
    ""token_expires"": 1689618083
  }
}
```
Reminder - `""_r""` means restrict, `""r""` means resource.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2104#issuecomment-1638552567,https://api.github.com/repos/simonw/datasette/issues/2104,1638552567,IC_kwDOBm6k_c5hqlP3,9599,simonw,2023-07-17T17:14:20Z,2023-07-17T17:14:20Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/database.py#L391-L451,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1808215339,Tables starting with an underscore should be treated as hidden,
https://github.com/simonw/datasette/issues/670#issuecomment-1637293044,https://api.github.com/repos/simonw/datasette/issues/670,1637293044,IC_kwDOBm6k_c5hlxv0,4863782,yairlenga,2023-07-17T02:23:32Z,2023-07-17T02:23:32Z,NONE,Is there any working version of datasette/postgresql ?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL,
https://github.com/simonw/datasette/issues/2087#issuecomment-1636134091,https://api.github.com/repos/simonw/datasette/issues/2087,1636134091,IC_kwDOBm6k_c5hhWzL,653549,adarshp,2023-07-14T17:02:03Z,2023-07-14T17:02:03Z,NONE,"@asg017  - the docs say that the autodetection only occurs in configuration directory mode. I for one would also be interested in the `--settings settings.json` feature. 

For context, I am developing a large database for use with Datasette, but the database lives in a different network volume than my source code, since the volume in which my source code lives is aggressively backed up, while the location where the database lives is meant for temporary files and is not as aggressively backed up (since the backups would get unreasonably large).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1765870617,`--settings settings.json` option,
https://github.com/simonw/datasette/issues/2102#issuecomment-1636093730,https://api.github.com/repos/simonw/datasette/issues/2102,1636093730,IC_kwDOBm6k_c5hhM8i,9599,simonw,2023-07-14T16:26:27Z,2023-07-14T16:32:49Z,OWNER,"Here's that crucial comment:

>  If _r is defined then we use those to further restrict the actor.
>
>Crucially, we only use this to say NO (return False) - we never use it to return YES (True) because that might over-ride other restrictions placed on this actor

So that's why I implemented it like this.

The goal here is to be able to issue a token which can't do anything _more_ than the actor it is associated with, but CAN be configured to do less.

So I think the solution here is for the `_r` checking code to perhaps implement its own view cascade logic - it notices if you have `view-table` and consequently fails to block `view-table` and `view-instance`.

I'm not sure that's going to work though - would that mean that granting `view-table` grants `view-database` in a surprising and harmful way?

Maybe that's OK: if you have `view-database` but permission checks fail for individual tables and queries you shouldn't be able to see a thing that you shouldn't. Need to verify that though.

Also, do `Permission` instances have enough information to implement this kind of cascade without hard-coding anything? 
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1636053060,https://api.github.com/repos/simonw/datasette/issues/2102,1636053060,IC_kwDOBm6k_c5hhDBE,9599,simonw,2023-07-14T15:51:36Z,2023-07-14T16:14:05Z,OWNER,"This might only be an issue with the code that checks `_r` on actors.

https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/default_permissions.py#L185-L222

Added in https://github.com/simonw/datasette/commit/bcc781f4c50a8870e3389c4e60acb625c34b0317 - refs:

- #1855 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1636042066,https://api.github.com/repos/simonw/datasette/issues/2102,1636042066,IC_kwDOBm6k_c5hhAVS,9599,simonw,2023-07-14T15:41:54Z,2023-07-14T15:42:32Z,OWNER,"I tried some code spelunking and came across https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7 which says:

> - If you have table permission but not database permission you can now view the table page

Refs:
- #832 

Which suggests that my initial design decision wasn't what appears to be implemented today.

Needs more investigation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1636040164,https://api.github.com/repos/simonw/datasette/issues/2102,1636040164,IC_kwDOBm6k_c5hg_3k,9599,simonw,2023-07-14T15:40:21Z,2023-07-14T15:40:21Z,OWNER,"Relevant code: 
https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/app.py#L822-L855","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2102#issuecomment-1636036312,https://api.github.com/repos/simonw/datasette/issues/2102,1636036312,IC_kwDOBm6k_c5hg-7Y,9599,simonw,2023-07-14T15:37:14Z,2023-07-14T15:37:14Z,OWNER,"I think I made this decision because I was thinking about default deny: obviously if a user has been denied access to a database. It doesn't make sense that they could access tables within it.

But now that I am spending more time with authentication tokens, which default to denying everything, except for the things that you have explicitly listed, this policy, no longer makes as much sense.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table,
https://github.com/simonw/datasette/issues/2101#issuecomment-1634443907,https://api.github.com/repos/simonw/datasette/issues/2101,1634443907,IC_kwDOBm6k_c5ha6KD,9599,simonw,2023-07-13T15:24:17Z,2023-07-13T15:24:17Z,OWNER,https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/views/table.py#L486-L506,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1803264272,alter: true support for JSON write API,
https://github.com/simonw/datasette/pull/2052#issuecomment-1630776144,https://api.github.com/repos/simonw/datasette/issues/2052,1630776144,IC_kwDOBm6k_c5hM6tQ,9020979,hydrosquall,2023-07-11T12:54:03Z,2023-07-11T12:54:03Z,CONTRIBUTOR,"Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1629337927,https://api.github.com/repos/simonw/datasette/issues/2052,1629337927,IC_kwDOBm6k_c5hHblH,9599,simonw,2023-07-10T16:43:38Z,2023-07-10T16:44:23Z,OWNER,"I tried running this locally just now. I made one edit:
```diff
diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py
index 7bdb9f3f..f9dfa8e6 100644
--- a/demos/plugins/example_js_manager_plugins.py
+++ b/demos/plugins/example_js_manager_plugins.py
@@ -15,6 +15,6 @@ def extra_js_urls(view_name):
     if view_name in PERMITTED_VIEWS:
         return [
             {
-                ""url"": f""/-/demos/plugins/static/table-example-plugins.js"",
+                ""url"": f""/static/table-example-plugins.js"",
             }
         ]
```
And then started it running like this:
```bash
wget https://datasette.io/content.db
```
```bash
datasette content.db \
  --plugins-dir demos/plugins \
  --static static:datasette/demos/plugins/static
```

It didn't quite work for me - I got this error on a table page:

<img width=""681"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/02f5c16d-ea43-4c36-920d-4b14406645b5"">

And this error on a query page:

<img width=""689"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/994db3a0-502a-4f36-8581-c47556600d63"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-1629123734,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14,1629123734,IC_kwDOC8tyDs5hGnSW,44622670,philipp-heinrich,2023-07-10T14:46:52Z,2023-07-10T14:46:52Z,NONE,@simonw any chance to get this fixed soon? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771608692,UNIQUE constraint failed: workouts.id,
https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627598570,IC_kwDOCGYnMM5hAy7q,9599,simonw,2023-07-09T04:13:34Z,2023-07-09T04:13:34Z,OWNER,On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats,
https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627597872,IC_kwDOCGYnMM5hAyww,9599,simonw,2023-07-09T04:09:56Z,2023-07-09T04:09:56Z,OWNER,"Thanks, looks like a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats,
https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627564127,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627564127,IC_kwDODLZ_YM5hAqhf,9599,simonw,2023-07-09T01:19:42Z,2023-07-09T01:19:42Z,MEMBER,https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml,
https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627563202,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627563202,IC_kwDODLZ_YM5hAqTC,9599,simonw,2023-07-09T01:14:27Z,2023-07-09T01:14:27Z,MEMBER,I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml,
https://github.com/simonw/datasette/issues/1153#issuecomment-1627480353,https://api.github.com/repos/simonw/datasette/issues/1153,1627480353,IC_kwDOBm6k_c5hAWEh,9599,simonw,2023-07-08T20:09:48Z,2023-07-08T20:09:48Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration is fixed now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627478910,https://api.github.com/repos/simonw/datasette/issues/1153,1627478910,IC_kwDOBm6k_c5hAVt-,9599,simonw,2023-07-08T20:01:19Z,2023-07-08T20:01:19Z,OWNER,"Some examples:

-  https://docs.datasette.io/en/latest/sql_queries.html#canned-queries
- https://docs.datasette.io/en/latest/sql_queries.html#canned-query-parameters
- https://docs.datasette.io/en/latest/authentication.html#access-to-an-instance
- https://docs.datasette.io/en/latest/facets.html#facets-in-metadata
- https://docs.datasette.io/en/latest/full_text_search.html#configuring-full-text-search-for-a-table-or-view
- https://docs.datasette.io/en/latest/metadata.html
- https://docs.datasette.io/en/latest/custom_templates.html#custom-css-and-javascript
- https://docs.datasette.io/en/latest/plugins.html#plugin-configuration

I need to fix this section: https://docs.datasette.io/en/latest/writing_plugins.html#writing-plugins-that-accept-configuration","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627455892,https://api.github.com/repos/simonw/datasette/issues/1153,1627455892,IC_kwDOBm6k_c5hAQGU,9599,simonw,2023-07-08T18:39:19Z,2023-07-08T18:39:19Z,OWNER,"```
ERROR: Could not find a version that satisfies the requirement Sphinx==6.1.3; extra == ""docs"" (from datasette[docs,test]) (from versions: 0.1.61611, 0.1.61798, 0.1.61843, 0.1.61945, 0.1.61950, 0.2, 0.3, 0.4, 0.4.1, 0.4.2, 0.4.3, 0.5, 0.5.1, 0.5.2b1, 0.5.2, 0.6b1, 0.6, 0.6.1, 0.6.2, 0.6.3, 0.6.4, 0.6.5, 0.6.6, 0.6.7, 1.0b1, 1.0b2, 1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.2b1, 1.2b2, 1.2b3, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.3b1, 1.3b2, 1.3b3, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.4a1, 1.4b1, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.5a1, 1.5a2, 1.5b1, 1.5, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.6b1, 1.6b2, 1.6b3, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.7.0b1, 1.7.0b2, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.8.0b1, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 2.0.0b1, 2.0.0b2, 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 3.0.0b1, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.2.1, 3.3.0, 3.3.1, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.5.0, 3.5.1, 3.5.2, 3.5.3, 3.5.4, 4.0.0b1, 4.0.0b2, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.3.0, 4.3.1, 4.3.2, 4.4.0, 4.5.0, 5.0.0b1, 5.0.0, 5.0.1, 5.0.2, 5.1.0, 5.1.1, 5.2.0, 5.2.0.post0, 5.2.1, 5.2.2, 5.2.3, 5.3.0)
ERROR: No matching distribution found for Sphinx==6.1.3; extra == ""docs""
```
I'm going to drop Python 3.7.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627451646,https://api.github.com/repos/simonw/datasette/issues/1153,1627451646,IC_kwDOBm6k_c5hAPD-,9599,simonw,2023-07-08T18:21:24Z,2023-07-08T18:21:24Z,OWNER,"This one was tricky:

<img width=""764"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/0fe2886b-4e07-4539-a254-4c47f02c3f65"">

I wanted complete control over the YAML example here, so I could ensure it used multi-line strings correctly.

I ended up changing my cog helper function to this:

```python
import json
import textwrap
from yaml import safe_dump
from ruamel.yaml import round_trip_load


def metadata_example(cog, data=None, yaml=None):
    assert data or yaml, ""Must provide data= or yaml=""
    assert not (data and yaml), ""Cannot use data= and yaml=""
    output_yaml = None
    if yaml:
        # dedent it first
        yaml = textwrap.dedent(yaml).strip()
        # round_trip_load to preserve key order:
        data = round_trip_load(yaml)
        output_yaml = yaml
    else:
        output_yaml = safe_dump(data, sort_keys=False)
    cog.out(""\n.. tab:: YAML\n\n"")
    cog.out(""    .. code-block:: yaml\n\n"")
    cog.out(textwrap.indent(output_yaml, ""        ""))
    cog.out(""\n\n.. tab:: JSON\n\n"")
    cog.out(""    .. code-block:: json\n\n"")
    cog.out(textwrap.indent(json.dumps(data, indent=2), ""        ""))
    cog.out(""\n"")
```
This allows me to call it ith YAML in some places:
```
.. [[[cog
    metadata_example(cog, yaml=""""""
    databases:
      fixtures:
        queries:
          neighborhood_search:
            fragment: fragment-goes-here
            hide_sql: true
            sql: |-
              select neighborhood, facet_cities.name, state
              from facetable join facet_cities on facetable.city_id = facet_cities.id
              where neighborhood like '%' || :text || '%' order by neighborhood;
    """""")
.. ]]]
```
I had to introduce https://pypi.org/project/ruamel.yaml/ as a dependency here in order to load YAML from disk while maintaining key order.

I'm still using `safe_dump(data, sort_keys=False)` from PyYAML as I couldn't get the result I wanted for outputting YAML from an input of JSON using PyYAML.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627450852,https://api.github.com/repos/simonw/datasette/issues/1153,1627450852,IC_kwDOBm6k_c5hAO3k,9599,simonw,2023-07-08T18:17:35Z,2023-07-08T18:17:35Z,OWNER,"I figured out a workaround:
```python
extensions = [
    ""sphinx.ext.extlinks"",
    ""sphinx.ext.autodoc"",
    ""sphinx_copybutton"",
]
if not os.environ.get(""DISABLE_SPHINX_INLINE_TABS""):
    extensions += [""sphinx_inline_tabs""]
```
That way I can run `sphinx-build -b xml . _build` successfully if I set that environment variable.

I get some noisy warnings, but it runs OK. And the resulting `docs.db` file has rows like this, which I think are fine:

<img width=""1533"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/0a289f53-2c6c-43e9-bda1-d4380383a02f"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627448542,https://api.github.com/repos/simonw/datasette/issues/1153,1627448542,IC_kwDOBm6k_c5hAOTe,9599,simonw,2023-07-08T18:05:44Z,2023-07-08T18:05:44Z,OWNER,"Running with `-P` opens a debugger when it hits the error:
```bash
sphinx-build -P -b xml . _build
```
```
(Pdb) list
2023 	
2024 	        Raise an exception unless overridden.
2025 	        """"""
2026 	        if (self.document.settings.strict_visitor
2027 	            or node.__class__.__name__ not in self.optional):
2028 ->	            raise NotImplementedError(
2029 	                '%s visiting unknown node type: %s'
2030 	                % (self.__class__, node.__class__.__name__))
2031 	
2032 	    def unknown_departure(self, node):
2033 	        """"""
(Pdb) self.optional
('meta',)
(Pdb) node.__class__.__name__
'TabContainer'
(Pdb) self.document.settings.strict_visitor
(Pdb) type(self.document.settings.strict_visitor)
<class 'NoneType'>
```
So if I can get `TabContainer` into that `self.optional` list I'll have fixed this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627448180,https://api.github.com/repos/simonw/datasette/issues/1153,1627448180,IC_kwDOBm6k_c5hAON0,9599,simonw,2023-07-08T18:03:31Z,2023-07-08T18:03:31Z,OWNER,"Relevant code: https://github.com/docutils/docutils/blob/3b53ded52bc439d8068b6ecb20ea0a761247e479/docutils/docutils/nodes.py#L2021-L2031

```python
    def unknown_visit(self, node):
        """"""
        Called when entering unknown `Node` types.

        Raise an exception unless overridden.
        """"""
        if (self.document.settings.strict_visitor
            or node.__class__.__name__ not in self.optional):
            raise NotImplementedError(
                '%s visiting unknown node type: %s'
                % (self.__class__, node.__class__.__name__))
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627447750,https://api.github.com/repos/simonw/datasette/issues/1153,1627447750,IC_kwDOBm6k_c5hAOHG,9599,simonw,2023-07-08T18:00:56Z,2023-07-08T18:00:56Z,OWNER,"Actually no it's in `sphinx-build`:
```
% sphinx-build -b xml . _build
Running Sphinx v6.1.3
building [mo]: targets for 0 po files that are out of date
writing output... 
building [xml]: targets for 28 source files that are out of date
updating environment: [new config] 28 added, 0 changed, 0 removed
reading sources... [100%] writing_plugins                                                                                                                          
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [  3%] authentication                                                                                                                            
Exception occurred:
  File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/docutils/nodes.py"", line 2028, in unknown_visit
    raise NotImplementedError(
NotImplementedError: <class 'docutils.writers.docutils_xml.XMLTranslator'> visiting unknown node type: TabContainer
The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1wkxmkji.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627447478,https://api.github.com/repos/simonw/datasette/issues/1153,1627447478,IC_kwDOBm6k_c5hAOC2,9599,simonw,2023-07-08T17:59:25Z,2023-07-08T17:59:25Z,OWNER,"Hit a problem:
```
Exception occurred:
  File ""/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/docutils/nodes.py"", line 2028, in unknown_visit
    raise NotImplementedError(
NotImplementedError: <class 'docutils.writers.docutils_xml.XMLTranslator'> visiting unknown node type: TabContainer
The full traceback has been saved in /tmp/sphinx-err-tfujyw1h.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
```
That's happening here: https://github.com/simonw/datasette/blob/0183e1a72d4d93b1d9a9363f4d47fcc0b5d5849c/.github/workflows/deploy-latest.yml#L42-L48

My https://github.com/simonw/sphinx-to-sqlite tool can't handle the new `TabContainer` elements introduced by `sphinx-inline-tabs`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627396658,https://api.github.com/repos/simonw/datasette/issues/1153,1627396658,IC_kwDOBm6k_c5hABoy,9599,simonw,2023-07-08T16:40:07Z,2023-07-08T16:40:07Z,OWNER,"https://docs.datasette.io/en/latest/metadata.html

![inline-tabs](https://github.com/simonw/datasette/assets/9599/975bdff5-74ac-451e-92c3-a7dd05d4b862)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627396450,https://api.github.com/repos/simonw/datasette/issues/1153,1627396450,IC_kwDOBm6k_c5hABli,9599,simonw,2023-07-08T16:38:58Z,2023-07-08T16:38:58Z,OWNER,"I'm using `cog` and this utility function to generate the YAML/JSON tabs:

https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata_doc.py#L1-L13

Example usage:

https://github.com/simonw/datasette/blob/3b336d8071fb5707bd006de1d614f701d20246a3/docs/metadata.rst?plain=1#L17-L53","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/datasette/issues/1153#issuecomment-1627395947,https://api.github.com/repos/simonw/datasette/issues/1153,1627395947,IC_kwDOBm6k_c5hABdr,9599,simonw,2023-07-08T16:35:45Z,2023-07-08T16:35:45Z,OWNER,I was inspired to finally address this after seeing `sphinx-inline-tabs` at work in https://webcolors.readthedocs.io/en/latest/install.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771202454,"Use YAML examples in documentation by default, not JSON",
https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1618380888,IC_kwDOCGYnMM5gdohY,9599,simonw,2023-07-03T14:09:11Z,2023-07-03T14:09:31Z,OWNER,"For the CLI:

```bash
sqlite-utils rename-table data.db old_table_name new_table_name
```

For the Python code, should it go on Table or on Database?
```python
db[""foo""].rename_table(""bar"")

db.rename_table(""foo"", ""bar"")
```
I think I like the second better, it's slightly more clear.

Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table,
https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444,https://api.github.com/repos/simonw/sqlite-utils/issues/563,1617395444,IC_kwDOCGYnMM5gZ370,9599,simonw,2023-07-03T05:44:43Z,2023-07-03T05:44:43Z,OWNER,Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1785360409,`--empty-null` option when importing CSV,
https://github.com/simonw/datasette/issues/2087#issuecomment-1616853644,https://api.github.com/repos/simonw/datasette/issues/2087,1616853644,IC_kwDOBm6k_c5gXzqM,15178711,asg017,2023-07-02T22:00:48Z,2023-07-02T22:00:48Z,CONTRIBUTOR,"I just saw in the docs that Dasette auto-detects `settings.json`:

> settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs
> [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1765870617,`--settings settings.json` option,
https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404,https://api.github.com/repos/simonw/sqlite-utils/issues/562,1616782404,IC_kwDOCGYnMM5gXiRE,9599,simonw,2023-07-02T19:24:14Z,2023-07-02T19:26:39Z,OWNER,"[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now.

The thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns.

Example for https://datasette.io/content/users

```sql
CREATE TABLE [users] (
   [login] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [avatar_url] TEXT,
   [gravatar_id] TEXT,
   [html_url] TEXT,
   [type] TEXT,
   [site_admin] INTEGER,
   [name] TEXT
);
```
And the dataclass:
```python
from dataclasses import dataclass

@dataclass
class User:
    id: int
    login: str
    node_id: str
    avatar_url: str
    gravatar_id: str
    html_url: str
    type: str
    site_admin: int
    name: str
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1784794489,Explore the intersection between sqlite-utils and dataclasses,
https://github.com/simonw/datasette/issues/2093#issuecomment-1616286848,https://api.github.com/repos/simonw/datasette/issues/2093,1616286848,IC_kwDOBm6k_c5gVpSA,15178711,asg017,2023-07-02T02:17:46Z,2023-07-02T02:17:46Z,CONTRIBUTOR,"Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way. 

The next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2093#issuecomment-1616195496,https://api.github.com/repos/simonw/datasette/issues/2093,1616195496,IC_kwDOBm6k_c5gVS-o,273509,terinjokes,2023-07-02T00:06:54Z,2023-07-02T00:07:17Z,NONE,"I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/pull/2052#issuecomment-1616095810,https://api.github.com/repos/simonw/datasette/issues/2052,1616095810,IC_kwDOBm6k_c5gU6pC,15178711,asg017,2023-07-01T20:31:31Z,2023-07-01T20:31:31Z,CONTRIBUTOR,"> Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

I just did a github search for `user:simonw ""def extra_js_urls(""` ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1615997736,https://api.github.com/repos/simonw/datasette/issues/2052,1615997736,IC_kwDOBm6k_c5gUiso,9020979,hydrosquall,2023-07-01T16:55:24Z,2023-07-01T16:55:24Z,CONTRIBUTOR,"> Ok @hydrosquall a couple things before this PR should be good to go:

Thank you @asg017 ! I've pushed both suggested changes onto this branch.

> Not sure how difficult it'll be to inject it server-side

If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a ""template"" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file.

In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a ""manual sync"" might be good enough. 

> In terms of how to integrate this into Datasette, a few options I can see working:

This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it.

> I'm also curious to see how ""plugins for a plugin' would work

I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a ""public API of other plugins"", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that. 

> here's a list of Simon's Datasette plugins that use ""extra_js_urls()""

Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory?

> I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.)

I look forward to trying this out 👍 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/issues/2093#issuecomment-1614652001,https://api.github.com/repos/simonw/datasette/issues/2093,1614652001,IC_kwDOBm6k_c5gPaJh,9599,simonw,2023-06-30T13:27:13Z,2023-06-30T13:27:13Z,OWNER,"I agree, settings in the DB doesn't make sense but metadata does.

On the JSON v YAML v TOML issue I just spotted Caddy has a concept of config adapters which they use to resolve exactly that problem: https://caddyserver.com/docs/config-adapters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2093#issuecomment-1613889979,https://api.github.com/repos/simonw/datasette/issues/2093,1613889979,IC_kwDOBm6k_c5gMgG7,9599,simonw,2023-06-29T22:44:08Z,2023-06-30T13:25:39Z,OWNER,"I do like also being able to set options using command line options though - for things like SQL time limits I'd much rather be able to throw on `--setting sql_time_limit_ms 10000` than have to save a config file to disk.

So I'd want to support both. Which maybe means also having a way to set plugin options with CLI options. `datasette publish` kind of has that ability already:

```
datasette publish heroku my_database.db \
    --name my-heroku-app-demo \
    --install=datasette-auth-github \
    --plugin-secret datasette-auth-github client_id your_client_id \
    --plugin-secret datasette-auth-github client_secret your_client_secret
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2093#issuecomment-1613896210,https://api.github.com/repos/simonw/datasette/issues/2093,1613896210,IC_kwDOBm6k_c5gMhoS,15178711,asg017,2023-06-29T22:53:33Z,2023-06-29T22:53:33Z,CONTRIBUTOR,"Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2093#issuecomment-1613895188,https://api.github.com/repos/simonw/datasette/issues/2093,1613895188,IC_kwDOBm6k_c5gMhYU,15178711,asg017,2023-06-29T22:51:53Z,2023-06-29T22:51:53Z,CONTRIBUTOR,"I agree with not liking `metadata.json` stuff in a `datasette.*` config file. Editing description of a table/column in a file like `datasette.*` seems odd to me. 

Though since plugin configuration currently lives in `metadata.json`, I think it should be removed from there and placed in `datasette.*`, at least for top-level config like `datasette-auth-github`'s config. Keeping `metadata.json` strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like `datasette.*`.

And ya, supporting both `datasette.*` and CLI flags makes a lot of sense to me. Any `--setting` flag should override anything in `datasette.*` for easier debugging, with possibly a warning message so people don't get confused. Same with `--port` and a port defined in `datasette.*`","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/issues/2093#issuecomment-1613887492,https://api.github.com/repos/simonw/datasette/issues/2093,1613887492,IC_kwDOBm6k_c5gMfgE,9599,simonw,2023-06-29T22:40:25Z,2023-06-29T22:40:25Z,OWNER,"I'm strongly in favour of combining settings, configuration and plugin configuration.

I'm not keen on mixing in metadata as well - that feels like a different concept to me, and I'm unhappy with how that's already had things like plugin settings leak into it.

I'm not yet sold on TOML - I actually find it less intuitive than YAML, surprisingly. They all have their warts I guess.

Datasette already has the ability to consume JSON or YAML for metadata - maybe it could grow TOML support too? That way users could have a `datasette.json` or `datasette.yaml` or `datasette.toml` file depending on their preference.

In terms of metadata: since that's means to be driven by a plugin hook anyway, maybe one of the potential sources of metadata is a `metadata` nested object in that `datasette.*` configuration file. Or you can have it in a separate `metadata.json` or bundled into the SQLite database or some other plugin-driven mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File",
https://github.com/simonw/datasette/pull/2052#issuecomment-1613778296,https://api.github.com/repos/simonw/datasette/issues/2052,1613778296,IC_kwDOBm6k_c5gME14,15178711,asg017,2023-06-29T20:36:09Z,2023-06-29T20:36:09Z,CONTRIBUTOR,"Ok @hydrosquall a couple things before this PR should be good to go:

- Can we move `datasette/static/table-example-plugins.js` into `demos/plugins/static`?
- For `datasetteManager.VERSION`, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into `manager.js` directly, so we don't have to worry about server-rendering (but that can be a future PR)

In terms of how to integrate this into Datasette, a few options I can see working:

- Push this as-is and figure it out before the next release
- Hide this feature behind a settings flag (`--setting unstable-js-plugins on`) and use that setting to hide/show `<script src=""{{ urls.static('datasette-manager.js') }}"" defer></script>` in `base.html`

I'll let @simonw decide which one to work with. I kindof like the idea of having an ""unstable"" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready.

I'm also curious to see how ""plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling ""plugins of plugins"" at the top with Datasette seems really convoluted. 

Also for posterity, here's a list of Simon's Datasette plugins that use ""extra_js_urls()"", which probably means they can be ported/re-written to use this new plugin system:

- [`datasette-vega`](https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/__init__.py#L25)
- [`datasette-cluster-map`](https://github.com/simonw/datasette-cluster-map/blob/795d25ad9ff6cba0307191f44fecc8f8070bef5c/datasette_cluster_map/__init__.py#L14)
- [`datasette-leaflet-geojson`](https://github.com/simonw/datasette-leaflet-geojson/blob/64713aa497750400b9ac2c12e8bb6ffab8eb77f3/datasette_leaflet_geojson/__init__.py#L47)
- [`datasette-pretty-traces`](https://github.com/simonw/datasette-pretty-traces/blob/5219d65eca3d7d7a73bb9d3120df42fe046a1315/datasette_pretty_traces/__init__.py#L5)
- [`datasette-youtube-embed`](https://github.com/simonw/datasette-youtube-embed/blob/4b4a0d7e58ebe15f47e9baf68beb9908c1d899da/datasette_youtube_embed/__init__.py#L55)
- [`datasette-leaflet-freedraw`](https://github.com/simonw/datasette-leaflet-freedraw/blob/8f28c2c2080ec9d29f18386cc6a2573a1c8fbde7/datasette_leaflet_freedraw/__init__.py#L66)
- [`datasette-hovercards`](https://github.com/simonw/datasette-hovercards/blob/9439ba46b7140fb03223faff0d21aeba5615a287/datasette_hovercards/__init__.py#L5)
- [`datasette-mp3-audio`](https://github.com/simonw/datasette-mp3-audio/blob/4402168792f452a46ab7b488e40ec49cd4b12185/datasette_mp3_audio/__init__.py#L6)
- [`datasette-geojson-map`](https://github.com/simonw/datasette-geojson-map/blob/32af5f1fd1a07278bbf8071fbb20a61e0f613246/datasette_geojson_map/__init__.py#L30)","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/issues/2092#issuecomment-1613381990,https://api.github.com/repos/simonw/datasette/issues/2092,1613381990,IC_kwDOBm6k_c5gKkFm,9599,simonw,2023-06-29T15:23:04Z,2023-06-29T15:26:28Z,OWNER,"Felt lazy:
```bash
symbex test_homepage -f tests/test_api.py | \
  llm -m 4 --system 'Change all of the == some integer tests in this code to isinstance(thing, int) instead'
```
Output:
```python
# File: tests/test_api.py Line: 26
@pytest.mark.asyncio
async def test_homepage(ds_client):
    response = await ds_client.get(""/.json"")
    assert response.status_code == 200
    assert ""application/json; charset=utf-8"" == response.headers[""content-type""]
    data = response.json()
    assert isinstance(data.keys(), int)
    d = data[""fixtures""]
    assert d[""name""] == ""fixtures""
    assert isinstance(d[""tables_count""], int)
    assert isinstance(len(d[""tables_and_views_truncated""]), int)
    assert d[""tables_and_views_more""] is True
    # 4 hidden FTS tables + no_primary_key (hidden in metadata)
    assert isinstance(d[""hidden_tables_count""], int)
    # 201 in no_primary_key, plus 6 in other hidden tables:
    assert isinstance(d[""hidden_table_rows_sum""], int), data
    assert isinstance(d[""views_count""], int)
```
I'll use most of that and delete the obsoleted comments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781047747,test_homepage intermittent failure,
https://github.com/simonw/datasette/issues/2092#issuecomment-1613375407,https://api.github.com/repos/simonw/datasette/issues/2092,1613375407,IC_kwDOBm6k_c5gKiev,9599,simonw,2023-06-29T15:20:52Z,2023-06-29T15:21:05Z,OWNER,I'm going to remove this assertion entirely. The homepage JSON needs a refactor anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781047747,test_homepage intermittent failure,
https://github.com/simonw/datasette/issues/2091#issuecomment-1613369355,https://api.github.com/repos/simonw/datasette/issues/2091,1613369355,IC_kwDOBm6k_c5gKhAL,9599,simonw,2023-06-29T15:18:34Z,2023-06-29T15:18:34Z,OWNER,Posted on the Glitch feedback forum about this here: https://support.glitch.com/t/upgrade-python-version-from-3-7-which-is-now-eol-to-something-more-recent/63011,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7,
https://github.com/simonw/datasette/issues/2091#issuecomment-1613360413,https://api.github.com/repos/simonw/datasette/issues/2091,1613360413,IC_kwDOBm6k_c5gKe0d,9599,simonw,2023-06-29T15:13:04Z,2023-06-29T15:13:04Z,OWNER,"One problem: https://glitch.com/ still provides 3.7:
```
$ python3 --version
Python 3.7.10
```
<img width=""956"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/e1fa44c0-4362-40e6-a2c7-d5b09b784c66"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7,
https://github.com/simonw/datasette/issues/2090#issuecomment-1613346412,https://api.github.com/repos/simonw/datasette/issues/2090,1613346412,IC_kwDOBm6k_c5gKbZs,9599,simonw,2023-06-29T15:05:04Z,2023-06-29T15:05:04Z,OWNER,"Decided to fix just those ""Ambiguous variable name"" ones:

```bash
ruff check . | grep E741
```
Then iterated through and fixed them all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781005740,Adopt ruff for linting,
https://github.com/simonw/datasette/issues/2090#issuecomment-1613339404,https://api.github.com/repos/simonw/datasette/issues/2090,1613339404,IC_kwDOBm6k_c5gKZsM,9599,simonw,2023-06-29T15:01:01Z,2023-06-29T15:01:20Z,OWNER,"I tried it just now and got some interesting results.

I dropped in a `ruff.toml` file:
```toml
line-length = 160
```
Because the default line length limit of 88 was causing a lot of noisy errors.

Then run:

```bash
pip install ruff
ruff check .
```
Plenty of warnings about unused imports - running `ruff check . --fix` fixed those automatically, but I think I still need to manually review them as some might be imports which are deliberate and should be in `__all__` to ensure they are visible from that module as well.

Some lines in tests are longer than even 160 chars, e.g.:

https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L673-L681

These can have `  # noqa: E501` added to the end of those lines to skip the check for them.

That got it down to:

```
% ruff check . 
datasette/views/table.py:23:5: F811 Redefinition of unused `format_bytes` from line 19
run_tests.py:2:5: E401 Multiple imports on one line
tests/test_api.py:591:40: F811 Redefinition of unused `app_client_no_files` from line 7
tests/test_api.py:629:35: F811 Redefinition of unused `app_client_no_files` from line 7
tests/test_api.py:635:54: F811 Redefinition of unused `app_client_with_dot` from line 8
tests/test_api.py:661:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 9
tests/test_api.py:759:25: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10
tests/test_api.py:892:28: F811 Redefinition of unused `app_client_larger_cache_size` from line 11
tests/test_api.py:928:5: F811 Redefinition of unused `app_client_with_cors` from line 12
tests/test_api.py:929:5: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10
tests/test_api.py:969:38: F811 Redefinition of unused `app_client_two_attached_databases` from line 13
tests/test_api.py:976:39: F811 Redefinition of unused `app_client_conflicting_database_names` from line 14
tests/test_api.py:987:38: F811 Redefinition of unused `app_client_immutable_and_inspect_file` from line 15
tests/test_api.py:1002:24: F811 Redefinition of unused `app_client` from line 6
tests/test_csv.py:67:33: F811 Redefinition of unused `app_client_with_cors` from line 6
tests/test_csv.py:157:21: F811 Redefinition of unused `app_client_csv_max_mb_one` from line 5
tests/test_csv.py:198:20: F811 Redefinition of unused `app_client_with_trace` from line 7
tests/test_csv.py:209:53: F811 Redefinition of unused `app_client_with_trace` from line 7
tests/test_csv.py:215:53: F811 Redefinition of unused `app_client_with_trace` from line 7
tests/test_filters.py:102:11: F811 Redefinition of unused `test_through_filters_from_request` from line 81
tests/test_html.py:19:19: F811 Redefinition of unused `app_client_two_attached_databases` from line 7
tests/test_html.py:175:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 6
tests/test_html.py:469:51: F811 Redefinition of unused `app_client` from line 4
tests/test_html.py:797:26: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
tests/test_html.py:840:44: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
tests/test_html.py:850:51: F811 Redefinition of unused `app_client_base_url_prefix` from line 5
tests/test_pagination.py:50:43: F821 Undefined name `parse_next`
tests/test_pagination.py:82:7: F811 Redefinition of unused `KeysetPaginator` from line 36
tests/test_plugins.py:115:15: E741 Ambiguous variable name: `l`
tests/test_plugins.py:482:161: E501 Line too long (170 > 160 characters)
tests/test_plugins.py:543:29: E741 Ambiguous variable name: `l`
tests/test_plugins.py:563:161: E501 Line too long (170 > 160 characters)
tests/test_plugins.py:940:62: E741 Ambiguous variable name: `l`
tests/test_table_api.py:739:5: F811 Redefinition of unused `app_client_returned_rows_matches_page_size` from line 6
tests/test_table_api.py:1066:45: F811 Redefinition of unused `app_client_with_trace` from line 5
tests/test_table_html.py:484:29: E741 Ambiguous variable name: `l`
tests/test_table_html.py:524:29: E741 Ambiguous variable name: `l`
tests/test_table_html.py:675:161: E501 Line too long (165 > 160 characters)
tests/test_table_html.py:897:161: E501 Line too long (164 > 160 characters)
tests/test_table_html.py:902:161: E501 Line too long (164 > 160 characters)
tests/test_utils.py:141:161: E501 Line too long (176 > 160 characters)
Found 41 errors.
```
Those ""Redefinition of unused `app_client_two_attached_databases`"" lines are caused because of the fixtures pattern I'm using here:

https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L3-L20

I could fix that by getting rid of `fixtures.py` and moving those into `conftest.py`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781005740,Adopt ruff for linting,
https://github.com/simonw/datasette/issues/2089#issuecomment-1613316722,https://api.github.com/repos/simonw/datasette/issues/2089,1613316722,IC_kwDOBm6k_c5gKUJy,9599,simonw,2023-06-29T14:48:10Z,2023-06-29T14:48:10Z,OWNER,Spell check is passing now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure,
https://github.com/simonw/datasette/issues/2089#issuecomment-1613315851,https://api.github.com/repos/simonw/datasette/issues/2089,1613315851,IC_kwDOBm6k_c5gKT8L,9599,simonw,2023-06-29T14:47:38Z,2023-06-29T14:47:38Z,OWNER,"Confirmed, this was a 2.2.5 change: https://github.com/codespell-project/codespell/releases/tag/v2.2.5

> -   Add displaing->displaying by [@peternewman](https://github.com/peternewman) in [#2808](https://github.com/codespell-project/codespell/pull/2808)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure,
https://github.com/simonw/datasette/issues/2089#issuecomment-1613307716,https://api.github.com/repos/simonw/datasette/issues/2089,1613307716,IC_kwDOBm6k_c5gKR9E,9599,simonw,2023-06-29T14:42:23Z,2023-06-29T14:42:23Z,OWNER,"Yes, upgrading locally got me the correct version and the test failure:
```
% pip install -U codespell
Requirement already satisfied: codespell in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages (2.2.2)
Collecting codespell
  Downloading codespell-2.2.5-py3-none-any.whl (242 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 4.9 MB/s eta 0:00:00
Installing collected packages: codespell
  Attempting uninstall: codespell
    Found existing installation: codespell 2.2.2
    Uninstalling codespell-2.2.2:
      Successfully uninstalled codespell-2.2.2
Successfully installed codespell-2.2.5
% codespell docs/metadata.rst
docs/metadata.rst:192: displaing ==> displaying
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure,
https://github.com/simonw/datasette/issues/2089#issuecomment-1613306787,https://api.github.com/repos/simonw/datasette/issues/2089,1613306787,IC_kwDOBm6k_c5gKRuj,9599,simonw,2023-06-29T14:41:47Z,2023-06-29T14:41:47Z,OWNER,"Looks like in CI it's running 2.2.5:
```
Collecting codespell (from datasette==1.0a2)
  Downloading codespell-2.2.5-py3-none-any.whl (242 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 31.1 MB/s eta 0:00:00
```
But on my laptop it's 2.2.2:
```
% codespell --version
2.2.2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure,
https://github.com/simonw/datasette/issues/2089#issuecomment-1613305070,https://api.github.com/repos/simonw/datasette/issues/2089,1613305070,IC_kwDOBm6k_c5gKRTu,9599,simonw,2023-06-29T14:40:44Z,2023-06-29T14:40:44Z,OWNER,"I'm not sure why I can't duplicate this failure in my local development environment:
```
% codespell docs/metadata.rst
```
It finds no errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure,
https://github.com/simonw/datasette/pull/2077#issuecomment-1557289070,https://api.github.com/repos/simonw/datasette/issues/2077,1557289070,IC_kwDOBm6k_c5c0lhu,22429695,codecov[bot],2023-05-22T14:08:33Z,2023-06-29T14:40:35Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`ede6203`)](https://app.codecov.io/gh/simonw/datasette/commit/ede62036180993dbd9d4e5d280fc21c183cda1c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`9785c4f`)](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2077   +/-   ##
=======================================
  Coverage   92.40%   92.40%           
=======================================
  Files          39       39           
  Lines        5803     5803           
=======================================
  Hits         5362     5362           
  Misses        441      441           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20,
https://github.com/simonw/datasette/pull/2077#issuecomment-1613290899,https://api.github.com/repos/simonw/datasette/issues/2077,1613290899,IC_kwDOBm6k_c5gKN2T,9599,simonw,2023-06-29T14:32:16Z,2023-06-29T14:32:16Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20,
https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875,https://api.github.com/repos/simonw/datasette/issues/1510,1610512875,IC_kwDOBm6k_c5f_nnr,9599,simonw,2023-06-28T02:02:10Z,2023-06-28T02:05:21Z,OWNER,"I prototyped an approach to this using dataclasses and a `cog` mechanism for turning those into rendered tables in Sphinx. Here's what that prototype looks like:

<img width=""1069"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/79210a43-716c-4470-a62d-01aa871973ec"">

See https://github.com/simonw/datasette/commit/68223784167fdec4e7ebfca56002a6548ba7b423 for how it works.

Here's the class that documented:

https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L54-L68

And the code that generates the rST: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/datasette/context.py#L19-L45

And the bit that cog executes: https://github.com/simonw/datasette/blob/68223784167fdec4e7ebfca56002a6548ba7b423/docs/template_context.rst?plain=1#L9-L12","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1054244712,Datasette 1.0 documented template context (maybe via API docs),
https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517,https://api.github.com/repos/simonw/sqlite-utils/issues/561,1610040517,IC_kwDOCGYnMM5f90TF,9599,simonw,2023-06-27T18:44:31Z,2023-06-27T18:44:38Z,OWNER,"Got this working:
```bash
sqlite-utils insert /tmp/playground.db Playground_Submission_Data \
  ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1777548699,`--stop-after` option for `insert` and `upsert` commands,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270887,IC_kwDOCGYnMM5fvb-n,9599,simonw,2023-06-25T21:37:12Z,2023-06-26T08:21:00Z,OWNER,"On my own laptop I got a crash running the tests - details here:

- https://github.com/nalgeon/sqlean.py/issues/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606237836,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606237836,IC_kwDOCGYnMM5fvT6M,22429695,codecov[bot],2023-06-25T19:49:45Z,2023-06-26T08:20:59Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`80.55`**% and project coverage change: **`-0.15`** :warning:
> Comparison is base [(`2747257`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/2747257a3334d55e890b40ec58fada57ae8cfbfd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`5e7d27e`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.22%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #560      +/-   ##
==========================================
- Coverage   96.36%   96.22%   -0.15%     
==========================================
  Files           6        6              
  Lines        2726     2752      +26     
==========================================
+ Hits         2627     2648      +21     
- Misses         99      104       +5     
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <63.63%> (-0.62%)` | :arrow_down: |
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <86.36%> (-0.20%)` | :arrow_down: |
| [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606415188,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606415188,IC_kwDOCGYnMM5fv_NU,9599,simonw,2023-06-26T01:46:47Z,2023-06-26T01:47:01Z,OWNER,"I just tested this in a brand new virtual environment using the macOS Python 3:
```bash
pipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3
```
Then in that virtual environment I ran:
```bash
pip install sqlite-utils
# Confirm the right one is on the path:
which sqlite-utils

curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \
    sqlite-utils insert meteorites.db meteorites - --pk=id
sqlite-utils extract meteorites.db meteorites  recclass
```
This threw the same error reported above. Then I did this:
```bash
rm meteorites.db
pip install sqlean.py

curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \
    sqlite-utils insert meteorites.db meteorites - --pk=id
sqlite-utils extract meteorites.db meteorites  recclass
```
And that second time it worked correctly.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606411508,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606411508,IC_kwDOCGYnMM5fv-T0,9599,simonw,2023-06-26T01:42:10Z,2023-06-26T01:42:22Z,OWNER,https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.,"{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/datasette/pull/2052#issuecomment-1606352600,https://api.github.com/repos/simonw/datasette/issues/2052,1606352600,IC_kwDOBm6k_c5fvv7Y,15178711,asg017,2023-06-26T00:17:04Z,2023-06-26T00:17:04Z,CONTRIBUTOR,":wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606315321,IC_kwDOCGYnMM5fvm05,9599,simonw,2023-06-25T23:18:33Z,2023-06-25T23:18:33Z,OWNER,Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606310630,IC_kwDOCGYnMM5fvlrm,9599,simonw,2023-06-25T23:06:07Z,2023-06-25T23:06:07Z,OWNER,"Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here:
- https://github.com/coleifer/pysqlite3/issues/58","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606297356,IC_kwDOCGYnMM5fvicM,9599,simonw,2023-06-25T22:42:41Z,2023-06-25T22:42:41Z,OWNER,"Yes that does seem to do the trick:
```pycon
>>> import sqlean
>>> db = sqlean.connect(""/tmp/4.db"")
>>> db.execute('PRAGMA journal_mode;').fetchall()
[('delete',)]
>>> db.isolation_level
''
>>> db.execute('PRAGMA journal_mode=wal;')
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction
>>> db.isolation_level = None
>>> db.isolation_level
>>> db.execute('PRAGMA journal_mode=wal;')
<sqlean.dbapi2.Cursor object at 0x104437c20>
```
Weird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606294627,IC_kwDOCGYnMM5fvhxj,9599,simonw,2023-06-25T22:40:10Z,2023-06-25T22:40:10Z,OWNER,I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606293382,IC_kwDOCGYnMM5fvheG,9599,simonw,2023-06-25T22:34:47Z,2023-06-25T22:34:47Z,OWNER,"```pycon
>>> import sqlite3
>>> db = sqlite3.connect(""/tmp/1.db"")
>>> db.execute('PRAGMA journal_mode=wal;')
<sqlite3.Cursor object at 0x102475ec0>
>>> import sqlean
>>> db2 = sqlean.connect(""/tmp/2.db"")
>>> db2.execute('PRAGMA journal_mode=wal;')
Traceback (most recent call last):
  File ""<stdin>"", line 1, in <module>
sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606290917,IC_kwDOCGYnMM5fvg3l,9599,simonw,2023-06-25T22:32:28Z,2023-06-25T22:32:28Z,OWNER,"I've fixed most of the test failures, but I still need to fix this one:

> cannot change into wal mode from within a transaction","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606273005,IC_kwDOCGYnMM5fvcft,9599,simonw,2023-06-25T21:47:47Z,2023-06-25T21:47:47Z,OWNER,I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270055,IC_kwDOCGYnMM5fvbxn,9599,simonw,2023-06-25T21:31:56Z,2023-06-25T21:31:56Z,OWNER,"Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953

A bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`:
```
    @pytest.fixture
    def db_to_analyze_path(db_to_analyze, tmpdir):
        path = str(tmpdir / ""test.db"")
        db = sqlite3.connect(path)
>       db.executescript(""\n"".join(db_to_analyze.conn.iterdump()))
E       AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump'
```
Also some of these:
```
    def test_analyze_whole_database(db):
        assert set(db.table_names()) == {""one_index"", ""two_indexes""}
        db.analyze()
>       assert set(db.table_names()) == {""one_index"", ""two_indexes"", ""sqlite_stat1""}
E       AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'}
E         Extra items in the left set:
E         'sqlite_stat4'
E         Full diff:
E         - {'two_indexes', 'sqlite_stat1', 'one_index'}
E         + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'}
E         ?                                 ++++++++++++++++
```
Apparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting.

Plus some errors that look like this:
```
    def test_enable_wal():
        runner = CliRunner()
        dbs = [""test.db"", ""test2.db""]
        with runner.isolated_filesystem():
            for dbname in dbs:
                db = Database(dbname)
                db[""t""].create({""pk"": int}, pk=""pk"")
                assert db.journal_mode == ""delete""
            result = runner.invoke(cli.cli, [""enable-wal""] + dbs)
>           assert 0 == result.exit_code
E           AssertionError: assert 0 == 1
E            +  where 1 = <Result OperationalError('cannot change into wal mode from within a transaction')>.exit_code
```
Test summary:
```
============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s =============
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1604379952,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1604379952,IC_kwDOCGYnMM5foOUw,9599,simonw,2023-06-23T14:39:55Z,2023-06-23T15:39:32Z,OWNER,"Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment:

    sqlite-utils install pysqlite3-binary

But `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people.

The ""easiest"" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/datasette/issues/260#issuecomment-1600778057,https://api.github.com/repos/simonw/datasette/issues/260,1600778057,IC_kwDOBm6k_c5fae9J,9599,simonw,2023-06-21T12:51:22Z,2023-06-21T12:51:22Z,OWNER,"Another example of confusion from this today: https://discord.com/channels/823971286308356157/823971286941302908/1121042411238457374

See also https://gist.github.com/BinomeDeNewton/651ac8b50dd5420f8e54d1682eee5fed?permalink_comment_id=4605982#gistcomment-4605982","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323223872,Validate metadata.json on startup,
https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592617271,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592617271,IC_kwDOCGYnMM5e7Wk3,13780613,erlend-aasland,2023-06-15T08:39:49Z,2023-06-15T08:39:49Z,NONE,"> piping to `jq` is good enough usually

... or `python -m json.tool`[^1], if you don't have `jq` installed.

[^1]: no fancy colouring, like `jq`; only pretty-printing","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,rows: --transpose or psql extended view-like functionality,
https://github.com/simonw/sqlite-utils/issues/529#issuecomment-1592110694,https://api.github.com/repos/simonw/sqlite-utils/issues/529,1592110694,IC_kwDOCGYnMM5e5a5m,7908073,chapmanjacobd,2023-06-14T23:11:47Z,2023-06-14T23:12:12Z,CONTRIBUTOR,"sorry i was wrong. `sqlite-utils --raw-lines` works correctly

```
sqlite-utils --raw-lines :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$

sqlite-utils --csv --no-headers :memory: ""SELECT * FROM (VALUES ('test'), ('line2'))"" | cat -A
test$
line2$
```

I think this was fixed somewhat recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1581090327,Microsoft line endings,
https://github.com/simonw/sqlite-utils/issues/491#issuecomment-1264218914,https://api.github.com/repos/simonw/sqlite-utils/issues/491,1264218914,IC_kwDOCGYnMM5LWnMi,7908073,chapmanjacobd,2022-10-01T03:18:36Z,2023-06-14T22:14:24Z,CONTRIBUTOR,"> some good concrete use-cases in mind

I actually found myself wanting something like this the past couple days. The use-case was databases with slightly different schema but same table names.

here is a full script:

```
import argparse
from pathlib import Path

from sqlite_utils import Database


def connect(args, conn=None, **kwargs) -> Database:
    db = Database(conn or args.database, **kwargs)
    with db.conn:
        db.conn.execute(""PRAGMA main.cache_size = 8000"")
    return db


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser()
    parser.add_argument(""database"")
    parser.add_argument(""dbs_folder"")
    parser.add_argument(""--db"", ""-db"", help=argparse.SUPPRESS)
    parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0)
    args = parser.parse_args()

    if args.db:
        args.database = args.db
    Path(args.database).touch()
    args.db = connect(args)

    return args


def merge_db(args, source_db):
    source_db = str(Path(source_db).resolve())

    s_db = connect(argparse.Namespace(database=source_db, verbose = args.verbose))
    for table in s_db.table_names():
        data = s_db[table].rows
        args.db[table].insert_all(data, alter=True, replace=True)

    args.db.conn.commit()


def merge_directory():
    args = parse_args()
    source_dbs = list(Path(args.dbs_folder).glob('*.db'))
    for s_db in source_dbs:
        merge_db(args, s_db)


if __name__ == '__main__':
    merge_directory()
```

edit: I've made some improvements to this and put it on PyPI:

```
$ pip install xklb
$ lb merge-db -h
usage: library merge-dbs DEST_DB SOURCE_DB ... [--only-target-columns] [--only-new-rows] [--upsert] [--pk PK ...] [--table TABLE ...]

    Merge-DBs will insert new rows from source dbs to target db, table by table. If primary key(s) are provided,
    and there is an existing row with the same PK, the default action is to delete the existing row and insert the new row
    replacing all existing fields.

    Upsert mode will update matching PK rows such that if a source row has a NULL field and
    the destination row has a value then the value will be preserved instead of changed to the source row's NULL value.

    Ignore mode (--only-new-rows) will insert only rows which don't already exist in the destination db

    Test first by using temp databases as the destination db.
    Try out different modes / flags until you are satisfied with the behavior of the program

        library merge-dbs --pk path (mktemp --suffix .db) tv.db movies.db

    Merge database data and tables

        library merge-dbs --upsert --pk path video.db tv.db movies.db
        library merge-dbs --only-target-columns --only-new-rows --table media,playlists --pk path audio-fts.db audio.db

        library merge-dbs --pk id --only-tables subreddits reddit/81_New_Music.db audio.db
        library merge-dbs --only-new-rows --pk subreddit,path --only-tables reddit_posts reddit/81_New_Music.db audio.db -v

positional arguments:
  database
  source_dbs
```

Also if you want to dedupe a table based on a ""business key"" which isn't explicitly your primary key(s) you can run this:

```
$ lb dedupe-db -h
usage: library dedupe-dbs DATABASE TABLE --bk BUSINESS_KEYS [--pk PRIMARY_KEYS] [--only-columns COLUMNS]

    Dedupe your database (not to be confused with the dedupe subcommand)

    It should not need to be said but *backup* your database before trying this tool!

    Dedupe-DB will help remove duplicate rows based on non-primary-key business keys

        library dedupe-db ./video.db media --bk path

    If --primary-keys is not provided table metadata primary keys will be used
    If --only-columns is not provided all non-primary and non-business key columns will be upserted

positional arguments:
  database
  table

options:
  -h, --help            show this help message and exit
  --skip-0
  --only-columns ONLY_COLUMNS
                        Comma separated column names to upsert
  --primary-keys PRIMARY_KEYS, --pk PRIMARY_KEYS
                        Comma separated primary keys
  --business-keys BUSINESS_KEYS, --bk BUSINESS_KEYS
                        Comma separated business keys
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1383646615,Ability to merge databases and tables,
https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592052320,IC_kwDOCGYnMM5e5Mpg,7908073,chapmanjacobd,2023-06-14T22:05:28Z,2023-06-14T22:05:28Z,CONTRIBUTOR,piping to `jq` is good enough usually,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,rows: --transpose or psql extended view-like functionality,
https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502,https://api.github.com/repos/simonw/sqlite-utils/issues/555,1592047502,IC_kwDOCGYnMM5e5LeO,7908073,chapmanjacobd,2023-06-14T22:00:10Z,2023-06-14T22:01:57Z,CONTRIBUTOR,"You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python.

That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query...

You may also try chunking like this:

```py
def chunks(lst, n) -> Generator:
    for i in range(0, len(lst), n):
        yield lst[i : i + n]

SQLITE_PARAM_LIMIT = 32765

data = []
chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT)
for ids in chunked:
    data.expand(
        list(
            db.query(
                f""""""SELECT * from videos
                WHERE id in (""""""
                + "","".join([""?""] * len(ids))
                + "")"",
                (*ids,),
            )
        )
    )
```

but that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row.

Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1733198948,Filter table by a large bunch of ids,
https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1590531892,IC_kwDOCGYnMM5ezZc0,7908073,chapmanjacobd,2023-06-14T06:09:21Z,2023-06-14T06:09:21Z,CONTRIBUTOR,"I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help

```
CREATE TABLE my_table (
    id INTEGER PRIMARY KEY,
    column1 TEXT,
    column2 TEXT,
    column3 TEXT
);

INSERT INTO my_table (column1, column2, column3)
VALUES
    ('Value 1', 'Duplicate 1', 'Duplicate A'),
    ('Value 2', 'Duplicate 2', 'Duplicate B'),
    ('Value 3', 'Duplicate 2', 'Duplicate C'),
    ('Value 4', 'Duplicate 3', 'Duplicate D'),
    ('Value 5', 'Duplicate 3', 'Duplicate E'),
    ('Value 6', 'Duplicate 3', 'Duplicate F');
```

```
library dedupe-db test.db my_table --bk column2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands,
https://github.com/simonw/datasette/pull/2052#issuecomment-1585149909,https://api.github.com/repos/simonw/datasette/issues/2052,1585149909,IC_kwDOBm6k_c5ee3fV,9020979,hydrosquall,2023-06-09T21:35:00Z,2023-06-09T21:35:00Z,CONTRIBUTOR,"Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world.

> Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

Ah, I understand now! Thanks for explaining. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1578840450,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1578840450,IC_kwDOCGYnMM5eGzGC,392720,jonafato,2023-06-06T14:09:04Z,2023-06-06T14:09:04Z,NONE,"I also ran into this recently. See below for a patch for one possible solution (tested via ""it works on my machine"", but I don't expect that this behavior would vary a whole lot across terminal emulators and shells). Another possible solution might be to subclass click's `ProgressBar` to keep the logic within the original context manager. Happy to send a PR or for this patch to serve as the basis for a fix that someone else authors.

```patch
diff --git a/sqlite_utils/utils.py b/sqlite_utils/utils.py
index 06c1a4c..530a3a3 100644
--- a/sqlite_utils/utils.py
+++ b/sqlite_utils/utils.py
@@ -147,14 +147,23 @@ def decode_base64_values(doc):
 
 
 class UpdateWrapper:
-    def __init__(self, wrapped, update):
+    def __init__(self, wrapped, update, render_finish):
         self._wrapped = wrapped
         self._update = update
+        self._render_finish = render_finish
 
     def __iter__(self):
-        for line in self._wrapped:
-            self._update(len(line))
-            yield line
+        return self
+
+    def __next__(self):
+        try:
+            line = next(self._wrapped)
+        except StopIteration as e:
+            self._render_finish()
+            raise
+
+        self._update(len(line))
+        return line
 
     def read(self, size=-1):
         data = self._wrapped.read(size)
@@ -178,7 +187,7 @@ def file_progress(file, silent=False, **kwargs):
     else:
         file_length = os.path.getsize(file.name)
         with click.progressbar(length=file_length, **kwargs) as bar:
-            yield UpdateWrapper(file, bar.update)
+            yield UpdateWrapper(file, bar.update, bar.render_finish)
 
 
 class Format(enum.Enum):

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1577355134,IC_kwDOCGYnMM5eBId-,7908073,chapmanjacobd,2023-06-05T19:26:26Z,2023-06-05T19:26:26Z,CONTRIBUTOR,"this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands,
https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378,https://api.github.com/repos/simonw/sqlite-utils/issues/556,1575310378,IC_kwDOCGYnMM5d5VQq,601708,mcint,2023-06-04T01:21:15Z,2023-06-04T01:21:15Z,CONTRIBUTOR,"I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740026046,Support storing incrementally piped values,
https://github.com/simonw/datasette/pull/2053#issuecomment-1565058994,https://api.github.com/repos/simonw/datasette/issues/2053,1565058994,IC_kwDOBm6k_c5dSOey,9599,simonw,2023-05-26T23:13:02Z,2023-05-26T23:13:02Z,OWNER,"I should have an extra called `extra_html_context` which bundles together all of the weird extra stuff needed by the HTML template, and is then passed as the root context when the template is rendered (with the other stuff from extras patched into it).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1563793781,https://api.github.com/repos/simonw/datasette/issues/2053,1563793781,IC_kwDOBm6k_c5dNZl1,9599,simonw,2023-05-26T04:27:55Z,2023-05-26T04:27:55Z,OWNER,"I should split out a `canned_query.html` template too, as something that extends the `query.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1563667574,https://api.github.com/repos/simonw/datasette/issues/2053,1563667574,IC_kwDOBm6k_c5dM6x2,9599,simonw,2023-05-26T00:40:22Z,2023-05-26T00:40:22Z,OWNER,"Or maybe...

- `BaseQueryView(View)` - knows how to render the results of a SQL query
- `QueryView(BaseQueryView)` - renders from `?sql=`
- `CannedQueryView(BaseQueryView)` - renders for a named canned query

And then later perhaps:

- `RowQueryView(BaseQueryView)` - renders the `select * from t where pk = ?`
- `TableQueryView(BaseQueryView)` - replaces the super complex existing `TableView`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1563663925,https://api.github.com/repos/simonw/datasette/issues/2053,1563663925,IC_kwDOBm6k_c5dM541,9599,simonw,2023-05-26T00:32:47Z,2023-05-26T00:35:47Z,OWNER,"I'm going to entirely split canned queries off from `?sql=` queries - they share a bunch of code right now which is just making everything much harder to follow.

I'll refactor their shared bits into functions that they both call.

Or _maybe_ I'll try having `CannedQueryView` as a subclass of `QueryView`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2053#issuecomment-1563663616,https://api.github.com/repos/simonw/datasette/issues/2053,1563663616,IC_kwDOBm6k_c5dM50A,9599,simonw,2023-05-26T00:32:08Z,2023-05-26T00:32:08Z,OWNER,"Now that I have the new `View` subclass from #2078 I want to use it to simplify this code.

Challenge: there are several things to consider here:

- The `/db` page without `?sql=` displays a list of tables in that database
- With `?sql=` it shows the query results for that query (or an error)
- If it's a `/db/name-of-canned-query` it works a bit like the query page, but executes a canned query instead of the `?sql=` query
- POST `/db/name-of-canned-query` is support for writable canned queries","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/pull/2080#issuecomment-1563650990,https://api.github.com/repos/simonw/datasette/issues/2080,1563650990,IC_kwDOBm6k_c5dM2uu,9599,simonw,2023-05-26T00:08:59Z,2023-05-26T00:08:59Z,OWNER,"I'm not going to document this yet, I want to let it bake for a bit longer first.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class,
https://github.com/simonw/datasette/pull/2080#issuecomment-1563629348,https://api.github.com/repos/simonw/datasette/issues/2080,1563629348,IC_kwDOBm6k_c5dMxck,22429695,codecov[bot],2023-05-25T23:31:10Z,2023-05-26T00:07:34Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`95.45`**% and project coverage change: **`+0.01`** :tada:
> Comparison is base [(`b49fa44`)](https://app.codecov.io/gh/simonw/datasette/commit/b49fa446d683ddcaf6faf2944dacc0d866bf2d70?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`df5fd73`)](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41%.

> :exclamation: Current head df5fd73 differs from pull request most recent head e990fbc. Consider uploading reports for the commit e990fbc to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2080      +/-   ##
==========================================
+ Coverage   92.40%   92.41%   +0.01%     
==========================================
  Files          39       39              
  Lines        5768     5790      +22     
==========================================
+ Hits         5330     5351      +21     
- Misses        438      439       +1     
```


| [Impacted Files](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/base.py](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.97% <95.45%> (+0.18%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class,
https://github.com/simonw/datasette/pull/2080#issuecomment-1563626231,https://api.github.com/repos/simonw/datasette/issues/2080,1563626231,IC_kwDOBm6k_c5dMwr3,9599,simonw,2023-05-25T23:25:17Z,2023-05-25T23:25:17Z,OWNER,I'm going to try using this for the `/-/patterns` page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563625093,https://api.github.com/repos/simonw/datasette/issues/2078,1563625093,IC_kwDOBm6k_c5dMwaF,9599,simonw,2023-05-25T23:23:15Z,2023-05-25T23:23:15Z,OWNER,"Rest of the work on this will happen in the PR:
- #2080","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563607291,https://api.github.com/repos/simonw/datasette/issues/2079,1563607291,IC_kwDOBm6k_c5dMsD7,9599,simonw,2023-05-25T22:56:28Z,2023-05-25T22:56:28Z,OWNER,Wrote this up as a TIL: https://til.simonwillison.net/http/testing-cors-max-age,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563597589,https://api.github.com/repos/simonw/datasette/issues/2079,1563597589,IC_kwDOBm6k_c5dMpsV,9599,simonw,2023-05-25T22:42:07Z,2023-05-25T22:42:07Z,OWNER,"Mystery solved as to why I wasn't seeing this work:

<img width=""951"" alt=""CleanShot 2023-05-25 at 15 38 32@2x"" src=""https://github.com/simonw/datasette/assets/9599/140ee046-69c2-4a68-b7cb-a4603552477c"">

I had ""Disable Cache"" checked!

I ran this experiment after un-checking that box:

```javascript
fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// And run it again
fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// Now try a thing that doesn't serve that max-age header yet:
fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// And a second time but within 5s
fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// Third time after waiting longer than 5s
fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
// Try that original one again - still within the 1hr cache time
fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', {
    method: 'POST',
    headers: {
        'Content-Type': 'application/json'
    },
    body: JSON.stringify({ test: 'test' })
});
```
The results show that the cache of 1hr was being obeyed for `latest.datasette.io` while the `latest-with-plugins.datasette.io` default cache of 5s was being obeyed too.

<img width=""904"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/39609d24-ef68-4db0-905a-451ed2462859"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563588199,https://api.github.com/repos/simonw/datasette/issues/2079,1563588199,IC_kwDOBm6k_c5dMnZn,9599,simonw,2023-05-25T22:29:47Z,2023-05-25T22:30:12Z,OWNER,"https://fetch.spec.whatwg.org/#http-access-control-max-age says:

> Indicates the number of seconds (5 by default) the information provided by the [Access-Control-Allow-Methods](https://fetch.spec.whatwg.org/#http-access-control-allow-methods) and [Access-Control-Allow-Headers](https://fetch.spec.whatwg.org/#http-access-control-allow-headers) [headers](https://fetch.spec.whatwg.org/#concept-header) can be cached.

So there was already a 5s cache anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563587230,https://api.github.com/repos/simonw/datasette/issues/2079,1563587230,IC_kwDOBm6k_c5dMnKe,9599,simonw,2023-05-25T22:28:20Z,2023-05-25T22:28:20Z,OWNER,"Weird... after the deploy went out:

<img width=""945"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/a8abb12a-446b-4970-8e76-3b218b6d02c7"">

But the request did indeed get the new header:

<img width=""951"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/7c72578c-9250-47d3-a4d2-1ece8c07ef84"">

So I'm not sure why it's making multiple `POST` requests like that.

Maybe it's because the attempted `POST` failed with a 404?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563565407,https://api.github.com/repos/simonw/datasette/issues/2079,1563565407,IC_kwDOBm6k_c5dMh1f,9599,simonw,2023-05-25T22:09:53Z,2023-05-25T22:09:53Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/json_api.html#enabling-cors,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563563438,https://api.github.com/repos/simonw/datasette/issues/2079,1563563438,IC_kwDOBm6k_c5dMhWu,9599,simonw,2023-05-25T22:08:28Z,2023-05-25T22:08:28Z,OWNER,"I ran this on https://www.example.com/ twice using the console:
```javascript
fetch(
  `https://latest.datasette.io/ephemeral/foo/1/-/update`,
  {
    method: ""POST"",
    mode: ""cors"",
    headers: {
      Authorization: `Bearer tok`,
      ""Content-Type"": ""application/json"",
    },
    body: JSON.stringify({update: {blah: 1}}),
  }
)
  .then((r) => r.json())
  .then((data) => {
    console.log(data);
  });
```
And got this in the network pane:

<img width=""945"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/cc767917-388b-4616-b35d-efaa1591948a"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563558915,https://api.github.com/repos/simonw/datasette/issues/2079,1563558915,IC_kwDOBm6k_c5dMgQD,9599,simonw,2023-05-25T22:04:41Z,2023-05-25T22:04:41Z,OWNER,I'm going with 3600 for 1 hour instead of 600 for 10 minutes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2079#issuecomment-1563547097,https://api.github.com/repos/simonw/datasette/issues/2079,1563547097,IC_kwDOBm6k_c5dMdXZ,9599,simonw,2023-05-25T21:51:38Z,2023-05-25T21:51:38Z,OWNER,"Also need to update this documentation:

https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/docs/json_api.rst?plain=1#L453-L465

Or maybe make that automated via `cog`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563522011,https://api.github.com/repos/simonw/datasette/issues/2078,1563522011,IC_kwDOBm6k_c5dMXPb,9599,simonw,2023-05-25T21:22:30Z,2023-05-25T21:22:30Z,OWNER,"This is bad:
```python
    async def __call__(self, request, datasette):
        try:
            handler = getattr(self, request.method.lower())
            return await handler(request, datasette)
        except AttributeError:
            return await self.method_not_allowed(request)
```
Because it hides any `AttributeError` exceptions that might occur in the view code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563511171,https://api.github.com/repos/simonw/datasette/issues/2078,1563511171,IC_kwDOBm6k_c5dMUmD,9599,simonw,2023-05-25T21:11:20Z,2023-05-25T21:13:05Z,OWNER,I'm going to call this `VerbView` for the moment. Might even rename it to `View` later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563498048,https://api.github.com/repos/simonw/datasette/issues/2078,1563498048,IC_kwDOBm6k_c5dMRZA,9599,simonw,2023-05-25T20:57:52Z,2023-05-25T20:58:13Z,OWNER,"Here's a new `BaseView` class that automatically populates `OPTIONS` based on available methods:
```python
class BaseView:
    async def head(self, *args, **kwargs):
        try:
            response = await self.get(*args, **kwargs)
            response.body = b""""
            return response
        except AttributeError:
            raise

    async def method_not_allowed(self, request):
        if (
            request.path.endswith("".json"")
            or request.headers.get(""content-type"") == ""application/json""
        ):
            response = Response.json(
                {""ok"": False, ""error"": ""Method not allowed""}, status=405
            )
        else:
            response = Response.text(""Method not allowed"", status=405)
        return response

    async def options(self, request, *args, **kwargs):
        response = Response.text(""ok"")
        response.headers[""allow""] = "", "".join(
            method.upper()
            for method in (""head"", ""get"", ""post"", ""put"", ""patch"", ""delete"")
            if hasattr(self, method)
        )
        return response

    async def __call__(self, request, datasette):
        try:
            handler = getattr(self, request.method.lower())
            return await handler(request, datasette)
        except AttributeError:
            return await self.method_not_allowed(request)


class DemoView(BaseView):
    async def get(self, datasette, request):
        return Response.text(""Hello there! {} - {}"".format(datasette, request))

    post = get
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563488929,https://api.github.com/repos/simonw/datasette/issues/2078,1563488929,IC_kwDOBm6k_c5dMPKh,9599,simonw,2023-05-25T20:48:12Z,2023-05-25T20:48:39Z,OWNER,"Actually no need for that extra level of parameter detection: `BaseView.__call__` should _always_ take `datasette, request` - `scope` and `receive` are both available on `request`, and `send` is only needed if you're not planning on returning a `Response` object.

So the `get` and `post` and suchlike methods should take `datasette` and `request` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563444296,https://api.github.com/repos/simonw/datasette/issues/2078,1563444296,IC_kwDOBm6k_c5dMERI,9599,simonw,2023-05-25T20:06:08Z,2023-05-25T20:06:08Z,OWNER,"This prototype seems to work well:

```diff
diff --git a/datasette/app.py b/datasette/app.py
index d7dace67..ed0edf28 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -17,6 +17,7 @@ import secrets
 import sys
 import threading
 import time
+import types
 import urllib.parse
 from concurrent import futures
 from pathlib import Path
@@ -1266,6 +1267,8 @@ class Datasette:
         # TODO: /favicon.ico and /-/static/ deserve far-future cache expires
         add_route(favicon, ""/favicon.ico"")
 
+        add_route(wrap_view(DemoView, self), '/demo')
+
         add_route(
             asgi_static(app_root / ""datasette"" / ""static""), r""/-/static/(?P<path>.*)$""
         )
@@ -1673,8 +1676,46 @@ def _cleaner_task_str(task):
     return _cleaner_task_str_re.sub("""", s)
 
 
-def wrap_view(view_fn, datasette):
-    @functools.wraps(view_fn)
+class DemoView:
+    async def __call__(self, datasette, request):
+        return Response.text(""Hello there! {} - {}"".format(datasette, request))
+
+def wrap_view(view_fn_or_class, datasette):
+    is_function = isinstance(view_fn_or_class, types.FunctionType)
+    if is_function:
+        return wrap_view_function(view_fn_or_class, datasette)
+    else:
+        if not isinstance(view_fn_or_class, type):
+            raise ValueError(""view_fn_or_class must be a function or a class"")
+        return wrap_view_class(view_fn_or_class, datasette)
+
+
+def wrap_view_class(view_class, datasette):
+    async def async_view_for_class(request, send):
+        instance = view_class()
+        if inspect.iscoroutinefunction(instance.__call__):
+            return await async_call_with_supported_arguments(
+                instance.__call__,
+                scope=request.scope,
+                receive=request.receive,
+                send=send,
+                request=request,
+                datasette=datasette,
+            )
+        else:
+            return call_with_supported_arguments(
+                instance.__call__,
+                scope=request.scope,
+                receive=request.receive,
+                send=send,
+                request=request,
+                datasette=datasette,
+            )
+
+    return async_view_for_class
+
+
+def wrap_view_function(view_fn, datasette):
     async def async_view_fn(request, send):
         if inspect.iscoroutinefunction(view_fn):
             response = await async_call_with_supported_arguments(
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563419066,https://api.github.com/repos/simonw/datasette/issues/2078,1563419066,IC_kwDOBm6k_c5dL-G6,9599,simonw,2023-05-25T19:42:16Z,2023-05-25T19:43:08Z,OWNER,"Maybe what I want here is the ability to register classes with the router - and have the router know that if it's a class it should instantiate it via its constructor and then await `__call__` it.

The neat thing about it is that it can reduce the risk of having a class instance that accidentally shares state between requests.

It also encourages that each class only responds based on the `datasette, request, ...` objects that are passed to its methods.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563359114,https://api.github.com/repos/simonw/datasette/issues/2078,1563359114,IC_kwDOBm6k_c5dLveK,9599,simonw,2023-05-25T18:47:57Z,2023-05-25T18:47:57Z,OWNER,"Oops, that broke everything:
```
    @documented
    async def await_me_maybe(value: typing.Any) -> typing.Any:
        ""If value is callable, call it. If awaitable, await it. Otherwise return it.""
>       if callable(value):
E       TypeError: 'module' object is not callable
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563329245,https://api.github.com/repos/simonw/datasette/issues/2078,1563329245,IC_kwDOBm6k_c5dLoLd,9599,simonw,2023-05-25T18:26:47Z,2023-05-25T18:28:08Z,OWNER,"With type hints and a namedtuple:
```python
import asyncio
import types
from typing import NamedTuple, Any


class CallableStatus(NamedTuple):
    is_callable: bool
    is_async_callable: bool


def check_callable(obj: Any) -> CallableStatus:
    if not callable(obj):
        return CallableStatus(False, False)

    if isinstance(obj, type):
        # It's a class
        return CallableStatus(True, False)

    if isinstance(obj, types.FunctionType):
        return CallableStatus(True, asyncio.iscoroutinefunction(obj))

    if hasattr(obj, ""__call__""):
        return CallableStatus(True, asyncio.iscoroutinefunction(obj.__call__))

    assert False, ""obj {} is somehow callable with no __call__ method"".format(repr(obj))
```
```python
for thing in (
    async_func,
    non_async_func,
    AsyncClass(),
    NotAsyncClass(),
    ClassNoCall(),
    AsyncClass,
    NotAsyncClass,
    ClassNoCall,
):
    print(thing, check_callable(thing))
```
```
<function async_func at 0x1073d5120> CallableStatus(is_callable=True, is_async_callable=True)
<function non_async_func at 0x1073d5080> CallableStatus(is_callable=True, is_async_callable=False)
<__main__.AsyncClass object at 0x106ba7490> CallableStatus(is_callable=True, is_async_callable=True)
<__main__.NotAsyncClass object at 0x106740150> CallableStatus(is_callable=True, is_async_callable=False)
<__main__.ClassNoCall object at 0x10676d910> CallableStatus(is_callable=False, is_async_callable=False)
<class '__main__.AsyncClass'> CallableStatus(is_callable=True, is_async_callable=False)
<class '__main__.NotAsyncClass'> CallableStatus(is_callable=True, is_async_callable=False)
<class '__main__.ClassNoCall'> CallableStatus(is_callable=True, is_async_callable=False)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563326000,https://api.github.com/repos/simonw/datasette/issues/2078,1563326000,IC_kwDOBm6k_c5dLnYw,9599,simonw,2023-05-25T18:23:38Z,2023-05-25T18:23:38Z,OWNER,I don't like that `is_callable()` implies a single boolean result but actually returns a pair. I'll call it `check_callable(obj)` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563318598,https://api.github.com/repos/simonw/datasette/issues/2078,1563318598,IC_kwDOBm6k_c5dLllG,9599,simonw,2023-05-25T18:17:03Z,2023-05-25T18:21:25Z,OWNER,"I think I want that to return `(is_callable, is_async)` - so I can both test if the thing can be called AND if it should be awaited in the same operation (without any exceptions).

I tried this:
```python
def is_callable(obj):
    ""Returns (is_callable, is_async_callable)""
    if not callable(obj):
        return False, False

    if isinstance(obj, types.FunctionType):
        return True, asyncio.iscoroutinefunction(obj)
        
    if hasattr(obj, '__call__'):
        return True, asyncio.iscoroutinefunction(obj.__call__)

    return False, False
```
```python
for thing in (
    async_func, non_async_func, AsyncClass(), NotAsyncClass(), ClassNoCall(),
    AsyncClass, NotAsyncClass, ClassNoCall
):
    print(thing, is_callable(thing))
```
And got:
```
<function async_func at 0x1073d5120> (True, True)
<function non_async_func at 0x1073d5080> (True, False)
<__main__.AsyncClass object at 0x106cce490> (True, True)
<__main__.NotAsyncClass object at 0x106ccf710> (True, False)
<__main__.ClassNoCall object at 0x106ccc810> (False, False)
<class '__main__.AsyncClass'> (True, True)
<class '__main__.NotAsyncClass'> (True, False)
<class '__main__.ClassNoCall'> (True, False)
```
Which is almost right, but I don't like that `AsyncClass` is shown as callable (which it is, since it's a class) and awaitable (which it is not - the `__call__` method may be async but calling the class constructor is not).

So I'm going to detect classes using `isinstance(obj, type)`.

```python
def is_callable(obj):
    ""Returns (is_callable, is_async_callable)""
    if not callable(obj):
        return False, False

    if isinstance(obj, type):
        # It's a class
        return True, False
    
    if isinstance(obj, types.FunctionType):
        return True, asyncio.iscoroutinefunction(obj)
        
    if hasattr(obj, '__call__'):
        return True, asyncio.iscoroutinefunction(obj.__call__)

    assert False, ""obj {} somehow is callable with no __call__ method"".format(obj)
```
I am reasonably confident the `AssertionError` can never be raised.

And now:
```
<function async_func at 0x1073d5120> (True, True)
<function non_async_func at 0x1073d5080> (True, False)
<__main__.AsyncClass object at 0x106ccfa50> (True, True)
<__main__.NotAsyncClass object at 0x106ccc8d0> (True, False)
<__main__.ClassNoCall object at 0x106cd7690> (False, False)
<class '__main__.AsyncClass'> (True, False)
<class '__main__.NotAsyncClass'> (True, False)
<class '__main__.ClassNoCall'> (True, False)
```
Which is what I wanted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563308919,https://api.github.com/repos/simonw/datasette/issues/2078,1563308919,IC_kwDOBm6k_c5dLjN3,9599,simonw,2023-05-25T18:08:34Z,2023-05-25T18:08:34Z,OWNER,"After much fiddling this seems to work:
```python
import asyncio, types

def is_async_callable(obj):
    if not callable(obj):
        raise ValueError(""Object is not callable"")

    if isinstance(obj, types.FunctionType):
        return asyncio.iscoroutinefunction(obj)
        
    if hasattr(obj, '__call__'):
        return asyncio.iscoroutinefunction(obj.__call__)

    raise ValueError(""Not a function and has no __call__ attribute"")
```
Tested like so:
```python
class AsyncClass:
    async def __call__(self):
        pass

class NotAsyncClass:
    def __call__(self):
        pass

class ClassNoCall:
    pass
    
async def async_func():
    pass

def non_async_func():
    pass

for thing in (AsyncClass(), NotAsyncClass(), ClassNoCall(), async_func, non_async_func):
    try:
        print(thing, is_async_callable(thing))
    except Exception as ex:
        print(thing, ex)
```
```
<__main__.AsyncClass object at 0x106c32150> True
<__main__.NotAsyncClass object at 0x106c32390> False
<__main__.ClassNoCall object at 0x106c32750> Object is not callable
<function async_func at 0x1073d5120> True
<function non_async_func at 0x1073d5080> False
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563294669,https://api.github.com/repos/simonw/datasette/issues/2078,1563294669,IC_kwDOBm6k_c5dLfvN,9599,simonw,2023-05-25T17:57:06Z,2023-05-25T17:57:06Z,OWNER,"I may need to be able to detect if a class instance has an `async def __call__` method - I think I can do that like so:

```python
def iscoroutinefunction(obj):
    if inspect.iscoroutinefunction(obj):
        return True
    if hasattr(obj, '__call__') and inspect.iscoroutinefunction(obj.__call__):
        return True
    return False
```
From https://github.com/encode/starlette/issues/886#issuecomment-606585152","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563292373,https://api.github.com/repos/simonw/datasette/issues/2078,1563292373,IC_kwDOBm6k_c5dLfLV,9599,simonw,2023-05-25T17:55:12Z,2023-05-25T17:55:30Z,OWNER,"So I think subclasses of `BaseView` need to offer a callable which accepts all five of the DI arguments - `datasette`, `request`, `scope`, `send`, `receive` - and then makes a decision based on the HTTP verb as to which method of the class to call. Those methods themselves can accept a subset of those parameters and will only be sent on to them.

Having two layers of parameter detection feels a little bit untidy, but I think it will work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/pull/2053#issuecomment-1563285150,https://api.github.com/repos/simonw/datasette/issues/2053,1563285150,IC_kwDOBm6k_c5dLdae,9599,simonw,2023-05-25T17:48:50Z,2023-05-25T17:49:52Z,OWNER,"Uncommitted experimental code:
```diff
diff --git a/datasette/views/database.py b/datasette/views/database.py
index 455ebd1f..85775433 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -909,12 +909,13 @@ async def query_view(
     elif format_ in datasette.renderers.keys():
         # Dispatch request to the correct output format renderer
         # (CSV is not handled here due to streaming)
+        print(data)
         result = call_with_supported_arguments(
             datasette.renderers[format_][0],
             datasette=datasette,
-            columns=columns,
-            rows=rows,
-            sql=sql,
+            columns=data[""rows""][0].keys(),
+            rows=data[""rows""],
+            sql='',
             query_name=None,
             database=db.name,
             table=None,
@@ -923,7 +924,7 @@ async def query_view(
             # These will be deprecated in Datasette 1.0:
             args=request.args,
             data={
-                ""rows"": rows,
+                ""rows"": data[""rows""],
             },  # TODO what should this be?
         )
         result = await await_me_maybe(result)
diff --git a/docs/index.rst b/docs/index.rst
index 5a9cc7ed..254ed3da 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -57,6 +57,7 @@ Contents
    settings
    introspection
    custom_templates
+   template_context
    plugins
    writing_plugins
    plugin_hooks
```
Where `docs/template_context.rst` looked like this:
```rst
.. _template_context:

Template context
================

.. currentmodule:: datasette.context

This page describes the variables made available to templates used by Datasette to render different pages of the application.

.. autoclass:: QueryContext
    :members:
```
And `datasette/context.py` had this:
```python
from dataclasses import dataclass

@dataclass
class QueryContext:
    """"""
    Used by the ``/database`` page when showing the results of a SQL query
    """"""
    id: int
    ""Id is a thing""
    rows: list[dict]
    ""Name is another thing""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563283939,https://api.github.com/repos/simonw/datasette/issues/2078,1563283939,IC_kwDOBm6k_c5dLdHj,9599,simonw,2023-05-25T17:47:38Z,2023-05-25T17:47:38Z,OWNER,"The idea behind `wrap_view()` is dependency injection - it's mainly used by plugins:

https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-routes-datasette

But I like the pattern so I started using it for some of Datasette's own features.

I should use it for _all_ of Datasette's own features.

But I still like the way `BaseView` helps with running different code for GET/POST/etc verbs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/datasette/issues/2078#issuecomment-1563282327,https://api.github.com/repos/simonw/datasette/issues/2078,1563282327,IC_kwDOBm6k_c5dLcuX,9599,simonw,2023-05-25T17:46:05Z,2023-05-25T17:46:05Z,OWNER,"Here's what `wrap_view()` does:

https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1676-L1700

It's used e.g. here:

https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1371-L1375

The `BaseView` thing meanwhile works like this:

https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/views/base.py#L56-L157","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`,
https://github.com/simonw/sqlite-utils/issues/554#issuecomment-1557607516,https://api.github.com/repos/simonw/sqlite-utils/issues/554,1557607516,IC_kwDOCGYnMM5c1zRc,1231935,xavdid,2023-05-22T17:18:33Z,2023-05-22T17:18:33Z,NONE,"Oh and for context - this goes away if I use `.upsert` instead of `insert(..., ignore=True)`, but I don't want to update the value if it's written, just do an insert if it's new. The code is basically:

```py
def save_items(table, items):
    db[""users""].insert(build_user(items[0]), pk=""id"",ignore=True)
    db[table].insert_all(items)

if comments := fetch_comments():
    save_items('comments', comments)

if posts := fetch_posts():
    save_items('posts', posts)
```

So either `comments` or `post` could create the relevant user if those items exist. In cases where they _both_ exist, I get this error. I need the `pk` because either call could create the table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1720096994,"`IndexError` when doing `.insert(..., pk='id')` after `insert_all`",
https://github.com/simonw/sqlite-utils/issues/552#issuecomment-1556292204,https://api.github.com/repos/simonw/sqlite-utils/issues/552,1556292204,IC_kwDOCGYnMM5cwyJs,9599,simonw,2023-05-21T21:05:15Z,2023-05-21T21:05:15Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718612569,Document how to setup shell auto-completion,
https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556291915,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556291915,IC_kwDOCGYnMM5cwyFL,9599,simonw,2023-05-21T21:04:03Z,2023-05-21T21:04:03Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/cli.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable,
https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556287870,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556287870,IC_kwDOCGYnMM5cwxF-,22429695,codecov[bot],2023-05-21T20:45:58Z,2023-05-21T20:57:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`e240133`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/e240133b11588d31dc22c632f7a7ca636c72978d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`0b81794`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.

> :exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #553   +/-   ##
=======================================
  Coverage   96.36%   96.36%           
=======================================
  Files           6        6           
  Lines        2726     2726           
=======================================
  Hits         2627     2627           
  Misses         99       99           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs,
https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556288300,IC_kwDOCGYnMM5cwxMs,9599,simonw,2023-05-21T20:48:01Z,2023-05-21T20:48:01Z,OWNER,If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs,
https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556288270,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556288270,IC_kwDOCGYnMM5cwxMO,9599,simonw,2023-05-21T20:47:51Z,2023-05-21T20:47:51Z,OWNER,This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable,
https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556287599,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556287599,IC_kwDOCGYnMM5cwxBv,9599,simonw,2023-05-21T20:44:55Z,2023-05-21T20:44:55Z,OWNER,"Put this in a PR so I can preview it: 
- #553
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable,
https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556269616,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556269616,IC_kwDOCGYnMM5cwsow,9599,simonw,2023-05-21T19:33:13Z,2023-05-21T19:33:13Z,OWNER,Now released: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface,
https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556265772,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556265772,IC_kwDOCGYnMM5cwrss,9599,simonw,2023-05-21T19:16:15Z,2023-05-21T19:16:15Z,OWNER,"Another option:

<img width=""593"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/a9d819a0-89e3-4b67-b2f0-1052b0a7a7c0"">

That's using this markup:

```
Newline-delimited JSON
~~~~~~~~~~~~~~~~~~~~~~

Use ``--nl`` to get back newline-delimited JSON objects:

.. code-block:: bash

    sqlite-utils dogs.db ""select * from dogs"" --nl

.. code-block:: output

    {""id"": 1, ""age"": 4, ""name"": ""Cleo""}
    {""id"": 2, ""age"": 2, ""name"": ""Pancakes""}
```
And this extra CSS:

```css
.highlight-output .highlight {
  border-left: 9px solid #30c94f;
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable,
https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556263182,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556263182,IC_kwDOCGYnMM5cwrEO,9599,simonw,2023-05-21T19:06:48Z,2023-05-21T19:06:48Z,OWNER,"I could split them up into two blocks like this:

<img width=""771"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/9b042ac8-db30-49d5-8882-1ea7be272a49"">

I do miss the visual indication that one of these is the command and one is the output though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable,
https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556262574,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556262574,IC_kwDOCGYnMM5cwq6u,9599,simonw,2023-05-21T19:04:59Z,2023-05-21T19:04:59Z,OWNER,"I wrote the docs like this because early examples include both the command and its output:

https://sqlite-utils.datasette.io/en/stable/cli.html#returning-json

<img width=""766"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/a5d05f24-ca0f-4718-a0a8-d85adc736141"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable,
https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556255309,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556255309,IC_kwDOCGYnMM5cwpJN,9599,simonw,2023-05-21T18:42:25Z,2023-05-21T18:42:25Z,OWNER,Tests passed here: https://github.com/simonw/sqlite-utils/actions/runs/5039119716,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7,
https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241555,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241555,IC_kwDOCGYnMM5cwlyT,22429695,codecov[bot],2023-05-21T17:57:24Z,2023-05-21T18:28:44Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`83.33`**% and project coverage change: **`+0.06`** :tada:
> Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`948692a`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #549      +/-   ##
==========================================
+ Coverage   96.30%   96.36%   +0.06%     
==========================================
  Files           6        6              
  Lines        2707     2726      +19     
==========================================
+ Hits         2607     2627      +20     
+ Misses        100       99       -1     
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <83.33%> (-0.03%)` | :arrow_down: |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon,
https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556250236,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556250236,IC_kwDOCGYnMM5cwn58,9599,simonw,2023-05-21T18:25:26Z,2023-05-21T18:25:26Z,OWNER,"Relevant issues:
- https://github.com/python/importlib_metadata/issues/406
- https://github.com/PyCQA/flake8/issues/1701

It looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7,
https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556249984,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556249984,IC_kwDOCGYnMM5cwn2A,9599,simonw,2023-05-21T18:24:48Z,2023-05-21T18:24:48Z,OWNER,"This is blocking:
- #549","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7,
https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556247818,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556247818,IC_kwDOCGYnMM5cwnUK,9599,simonw,2023-05-21T18:17:46Z,2023-05-21T18:17:46Z,OWNER,Draft documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface,
https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556242262,IC_kwDOCGYnMM5cwl9W,9599,simonw,2023-05-21T18:00:05Z,2023-05-21T18:00:05Z,OWNER,"Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465
```
sqlite_utils/cli.py:37: error: Skipping analyzing ""trogon"": module is installed, but missing library stubs or py.typed marker  [import]
sqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 52 source files)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon,
https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241812,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241812,IC_kwDOCGYnMM5cwl2U,9599,simonw,2023-05-21T17:58:25Z,2023-05-21T17:58:25Z,OWNER,Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon,
https://github.com/simonw/sqlite-utils/issues/548#issuecomment-1556231832,https://api.github.com/repos/simonw/sqlite-utils/issues/548,1556231832,IC_kwDOCGYnMM5cwjaY,9599,simonw,2023-05-21T17:24:13Z,2023-05-21T17:24:13Z,OWNER,"Oh, I see why that is now:

https://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679

This is because of the following command:

    sqlite-utils analyze-tables table1 table2 --column x

Since you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables.

I'm going to change this so if the column is not on ANY of those tables you get an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718576761,analyze-tables should validate provide --column names,
https://github.com/simonw/sqlite-utils/issues/547#issuecomment-1556228395,https://api.github.com/repos/simonw/sqlite-utils/issues/547,1556228395,IC_kwDOCGYnMM5cwikr,9599,simonw,2023-05-21T17:11:15Z,2023-05-21T17:11:15Z,OWNER,"This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`.

I can add an optimization though to avoid running the SQL count query if we know that it's all `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718572201,No need to show common values if everything is null,
https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556225788,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556225788,IC_kwDOCGYnMM5cwh78,9599,simonw,2023-05-21T17:02:05Z,2023-05-21T17:02:05Z,OWNER,"New docs:

- https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables
- https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column
- https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column

New help output:

```
 % sqlite-utils analyze-tables --help
Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]...

  Analyze the columns in one or more tables

  Example:

      sqlite-utils analyze-tables data.db trees

Options:
  -c, --column TEXT       Specific columns to analyze
  --save                  Save results to _analyze_tables table
  --common-limit INTEGER  How many common values
  --no-most               Skip most common values
  --no-least              Skip least common values
  --load-extension TEXT   Path to SQLite extension, with optional :entrypoint
  -h, --help              Show this message and exit.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least,
https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213396,IC_kwDOCGYnMM5cwe6U,9599,simonw,2023-05-21T15:58:12Z,2023-05-21T16:18:46Z,OWNER,"Documentation preview:
- https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables
- https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables
- https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column
- https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least",
https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213031,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213031,IC_kwDOCGYnMM5cwe0n,22429695,codecov[bot],2023-05-21T15:56:05Z,2023-05-21T16:18:03Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`93.75`**% and no project coverage change.
> Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`9f23e68`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.31%.

> :exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #546   +/-   ##
=======================================
  Coverage   96.30%   96.31%           
=======================================
  Files           6        6           
  Lines        2707     2712    +5     
=======================================
+ Hits         2607     2612    +5     
  Misses        100      100           
```


| [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.37% <90.90%> (+<0.01%)` | :arrow_up: |
| [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.01%)` | :arrow_up: |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least",
https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556211643,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556211643,IC_kwDOCGYnMM5cwee7,9599,simonw,2023-05-21T15:48:17Z,2023-05-21T15:48:17Z,OWNER,I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least,
https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556210844,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556210844,IC_kwDOCGYnMM5cweSc,9599,simonw,2023-05-21T15:44:10Z,2023-05-21T15:44:10Z,OWNER,"It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here:
- https://github.com/Textualize/trogon/issues/4","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface,
https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556191894,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556191894,IC_kwDOCGYnMM5cwZqW,9599,simonw,2023-05-21T14:20:14Z,2023-05-21T14:20:14Z,OWNER,"Opened a feature request for customizing the help and command name:

- https://github.com/Textualize/trogon/issues/2","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface,
https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556190531,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556190531,IC_kwDOCGYnMM5cwZVD,9599,simonw,2023-05-21T14:13:43Z,2023-05-21T14:13:43Z,OWNER,"OK, this works!

![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3)

To try it out, install that branch from GitHub:

    pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip

Then run this:

    sqlite-utils install trogon

And this:

    sqlite-utils tui
","{""total_count"": 5, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 3, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface,
https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556189823,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556189823,IC_kwDOCGYnMM5cwZJ_,9599,simonw,2023-05-21T14:09:59Z,2023-05-21T14:09:59Z,OWNER,"I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only.

But I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install

So I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface,
https://github.com/simonw/sqlite-utils/issues/399#issuecomment-1548913065,https://api.github.com/repos/simonw/sqlite-utils/issues/399,1548913065,IC_kwDOCGYnMM5cUomp,433780,chrislkeller,2023-05-16T03:11:03Z,2023-05-16T03:11:52Z,NONE,"Using this thread and some [other resources](https://sqlite-utils.datasette.io/en/stable/cli.html#spatialite-helpers) I managed to cobble together a couple of sqlite-utils lines to add a geometry column for a table that already has a lat/lng column.

```
# add a geometry column
sqlite-utils add-geometry-column [db name] [table name] geometry --type POINT --srid 4326

# add a point for each row to geometry column
sqlite-utils --load-extension=spatialite [db name] 'update [table name] SET Geometry=MakePoint(longitude, latitude, 4326);'
```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1124731464,"Make it easier to insert geometries, with documentation and maybe code",
https://github.com/simonw/datasette/pull/2052#issuecomment-1548617257,https://api.github.com/repos/simonw/datasette/issues/2052,1548617257,IC_kwDOBm6k_c5cTgYp,193185,cldellow,2023-05-15T21:32:20Z,2023-05-15T21:32:20Z,CONTRIBUTOR,"> Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays?

The latter - that you could return a promise of arrays, so it parallels the [""await me maybe"" pattern in Datasette](https://simonwillison.net/2020/Sep/2/await-me-maybe/), where you can return either a value, a callable or an awaitable.

> I have a hunch that what you're describing might be achievable without adding Promises to the API with something

Oops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation.

This isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place.

This could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent.

> especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag

Ah, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with.

My experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin.

Again, a fairly minor thing, though.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2075#issuecomment-1547944971,https://api.github.com/repos/simonw/datasette/issues/2075,1547944971,IC_kwDOBm6k_c5cQ8QL,22429695,codecov[bot],2023-05-15T14:12:20Z,2023-05-15T14:12:20Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`49184c5`)](https://app.codecov.io/gh/simonw/datasette/commit/49184c569cd70efbda4f3f062afef3a34401d8d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`b99e1d3`)](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2075   +/-   ##
=======================================
  Coverage   92.40%   92.40%           
=======================================
  Files          38       38           
  Lines        5751     5751           
=======================================
  Hits         5314     5314           
  Misses        437      437           
```




</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1710164693,Bump sphinx from 6.1.3 to 7.0.1,
https://github.com/simonw/datasette/pull/2068#issuecomment-1547911570,https://api.github.com/repos/simonw/datasette/issues/2068,1547911570,IC_kwDOBm6k_c5cQ0GS,49699333,dependabot[bot],2023-05-15T13:59:35Z,2023-05-15T13:59:35Z,CONTRIBUTOR,Superseded by #2075.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1690842199,Bump sphinx from 6.1.3 to 7.0.0,
https://github.com/simonw/datasette/pull/2052#issuecomment-1546362374,https://api.github.com/repos/simonw/datasette/issues/2052,1546362374,IC_kwDOBm6k_c5cK54G,9020979,hydrosquall,2023-05-12T22:09:03Z,2023-05-12T22:09:03Z,CONTRIBUTOR,"Hey @cldellow , thanks for the thoughtful feedback and describing the ""lazy facets"" feature! 

It sounds like the [postTask](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/postTask) API might be relevant for the types of network request scheduling you have in mind. 

Addressing your points inline below:

> It might also be nice if the plugins could return Promises.

Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like `makeColumnActions` or `makeAboveTablePanelConfigs` supported returning a promise of arrays instead only returning plain arrays?

I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable . 

I have a hunch that what you're describing might be achievable without adding Promises to the API with something like

```
fetch('/api/with-custom-facets').then(myFacets => {
    // reusing the go() idiom
   go(manager, myFacets);
})
```

but I'd like to confirm if that's the case before investigating adding support.

> bulletproof plugin registration code that is robust against the order in which the script tags load

Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet.

I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release:

I can see that this approach brings advantages

- Plugin developers don't need to know the name of the datasette initialization event to start their plugin
- Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener
- One less event listener sitting in memory

It also has some minor costs

- A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up
- Some risk in encouraging people to mutate global state
- (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the `window` provided by Datasette as an inline script tag). 


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/issues/2073#issuecomment-1546119773,https://api.github.com/repos/simonw/datasette/issues/2073,1546119773,IC_kwDOBm6k_c5cJ-pd,9599,simonw,2023-05-12T18:24:07Z,2023-05-12T18:24:07Z,OWNER,"Here's a demo of this breaking in Datasette Lite:

https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data/baseline?_filter_column_1=is_baseline&_filter_op_1=exact&_filter_value_1=1&_filter_column_2=&_filter_op_2=notnull__1&_filter_value_2=1&_filter_column=&_filter_op=exact&_filter_value=&_sort=&_facet=is_baseline

<img width=""609"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/3c23ae6e-7fb0-41d4-92b8-421c400a731a"">

Here's a SQL query that demonstrates the underlying issue:

```sql
select 'working', count(*) from baseline where is_baseline = 1
union all
select 'broken', count(*) from baseline where is_baseline = '1'
```
https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data?sql=select+%27working%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+1%0Aunion+all%0Aselect+%27broken%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+%271%27

<img width=""673"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/867914f8-8cef-432c-a518-2739d1caf964"">","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1708030220,Faceting doesn't work against integer columns in views,
https://github.com/simonw/datasette/issues/2073#issuecomment-1546117538,https://api.github.com/repos/simonw/datasette/issues/2073,1546117538,IC_kwDOBm6k_c5cJ-Gi,9599,simonw,2023-05-12T18:21:38Z,2023-05-12T18:21:38Z,OWNER,"https://latest.datasette.io/fixtures doesn't currently have a view with any integer columns in it, making this bug harder to demonstrate there.

I can't replicate the bug using https://datasette.io/content/plugins?_facet=stargazers_count&stargazers_count=3 - I would expect that not to work correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1708030220,Faceting doesn't work against integer columns in views,
https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1540900733,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1540900733,IC_kwDOCGYnMM5b2Ed9,167893,mcarpenter,2023-05-09T21:15:05Z,2023-05-09T21:15:05Z,CONTRIBUTOR,"Sorry, I completely missed your first comment whilst on Easter break.

This looks like a good practical compromise before v4. Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values,
https://github.com/simonw/datasette/issues/2070#issuecomment-1540494121,https://api.github.com/repos/simonw/datasette/issues/2070,1540494121,IC_kwDOBm6k_c5b0hMp,9599,simonw,2023-05-09T16:25:00Z,2023-05-09T16:25:00Z,OWNER,Can now be used here: https://github.com/simonw/datasette/actions/workflows/deploy-branch-preview.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1702354223,Mechanism for deploying a preview of a branch using Vercel,
https://github.com/simonw/datasette/issues/2070#issuecomment-1540491751,https://api.github.com/repos/simonw/datasette/issues/2070,1540491751,IC_kwDOBm6k_c5b0gnn,9599,simonw,2023-05-09T16:23:12Z,2023-05-09T16:23:12Z,OWNER,Added a actions `BRANCH_PREVIEW_VERCEL_TOKEN` secret to this repository.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1702354223,Mechanism for deploying a preview of a branch using Vercel,
https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539157643,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539157643,IC_kwDOCGYnMM5bva6L,9599,simonw,2023-05-08T22:45:09Z,2023-05-08T22:45:21Z,OWNER,"Here's an example from the new tests:

https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`,
https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1539109816,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1539109816,IC_kwDOCGYnMM5bvPO4,9599,simonw,2023-05-08T22:01:00Z,2023-05-08T22:01:00Z,OWNER,"This is being handled in:
- #520","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto',
https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1539109587,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1539109587,IC_kwDOCGYnMM5bvPLT,9599,simonw,2023-05-08T22:00:46Z,2023-05-08T22:00:46Z,OWNER,"> Hey, isn't this essentially the same issue as #448 ?

Yes it is, good catch!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539108140,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539108140,IC_kwDOCGYnMM5bvO0s,9599,simonw,2023-05-08T21:59:41Z,2023-05-08T21:59:41Z,OWNER,That original example passes against `main` now.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1539101853,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1539101853,IC_kwDOCGYnMM5bvNSd,9599,simonw,2023-05-08T21:52:44Z,2023-05-08T21:52:44Z,OWNER,I like the `lambda-{uuid}` idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail,
https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539100300,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539100300,IC_kwDOCGYnMM5bvM6M,9599,simonw,2023-05-08T21:50:51Z,2023-05-08T21:50:51Z,OWNER,Seeing as `sqlite-utils` doesn't currently provide mechanisms for adding `check` constraints like this I'm going to leave this - I'm happy with the fix I put in for the `not null` constraints.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails,
https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539099703,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539099703,IC_kwDOCGYnMM5bvMw3,9599,simonw,2023-05-08T21:50:06Z,2023-05-08T21:50:06Z,OWNER,"Applying the fix from the PR here doesn't fix the above problem either:
- https://github.com/simonw/sqlite-utils/pull/515

So it looks like these kinds of `check` constraints currently aren't compatible with how `upsert()` works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails,
https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539094287,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539094287,IC_kwDOCGYnMM5bvLcP,9599,simonw,2023-05-08T21:44:11Z,2023-05-08T21:44:11Z,OWNER,"OK, this fails silently:
```python
import sqlite_utils
db = sqlite_utils.Database(memory=True)
db.execute('''CREATE TABLE employees (
    id INTEGER PRIMARY KEY,
    name TEXT,
    age INTEGER,
    salary REAL,
    CHECK (salary is not null and salary > 0)
);''')
db[""employees""].upsert({""id"": 1, ""name"": ""Bob""}, pk=""id"")
list(db[""employees""].rows)
````
It outputs:
```python
[]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails,
https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539079507,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539079507,IC_kwDOCGYnMM5bvH1T,9599,simonw,2023-05-08T21:28:37Z,2023-05-08T21:28:37Z,OWNER,"> This means that a table with NON NULL (or other constraint) columns that aren't part of the pkey can't have new rows upserted.

Huh... on that basis, it's possible my fix in https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304 is incomplete. I only covered the 'not null' case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails,
https://github.com/simonw/sqlite-utils/issues/514#issuecomment-1539078429,https://api.github.com/repos/simonw/sqlite-utils/issues/514,1539078429,IC_kwDOCGYnMM5bvHkd,9599,simonw,2023-05-08T21:27:40Z,2023-05-08T21:27:40Z,OWNER,"Dupe of:
- #538","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194249,upsert of new row with check constraints fails,
https://github.com/simonw/sqlite-utils/pull/515#issuecomment-1539077777,https://api.github.com/repos/simonw/sqlite-utils/issues/515,1539077777,IC_kwDOCGYnMM5bvHaR,9599,simonw,2023-05-08T21:27:10Z,2023-05-08T21:27:10Z,OWNER,I should have spotted this PR before I shipped my own fix! https://github.com/simonw/sqlite-utils/commit/2376c452a56b0c3e75e7ca698273434e32945304,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1465194930,"upsert new rows with constraints, fixes #514",
https://github.com/simonw/sqlite-utils/pull/519#issuecomment-1539058795,https://api.github.com/repos/simonw/sqlite-utils/issues/519,1539058795,IC_kwDOCGYnMM5bvCxr,9599,simonw,2023-05-08T21:12:52Z,2023-05-08T21:12:52Z,OWNER,"This is a really neat fix, thank you.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1505568103,Fixes breaking DEFAULT values,
https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539055393,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539055393,IC_kwDOCGYnMM5bvB8h,9599,simonw,2023-05-08T21:10:06Z,2023-05-08T21:10:06Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`,
https://github.com/simonw/sqlite-utils/pull/528#issuecomment-1539053230,https://api.github.com/repos/simonw/sqlite-utils/issues/528,1539053230,IC_kwDOCGYnMM5bvBau,9599,simonw,2023-05-08T21:08:23Z,2023-05-08T21:08:23Z,OWNER,"I fixed this in:
- #527

Will fully remove this misfeature in:
- #542","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578793661,Enable `Table.convert()` on falsey values,
https://github.com/simonw/sqlite-utils/issues/542#issuecomment-1539052467,https://api.github.com/repos/simonw/sqlite-utils/issues/542,1539052467,IC_kwDOCGYnMM5bvBOz,9599,simonw,2023-05-08T21:07:41Z,2023-05-08T21:07:41Z,OWNER,"Relevant commits (will mostly revert these):
- https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd
- https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700936245,Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0,
https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539051724,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539051724,IC_kwDOCGYnMM5bvBDM,9599,simonw,2023-05-08T21:07:04Z,2023-05-08T21:07:04Z,OWNER,"Updated documentation:
- https://sqlite-utils.datasette.io/en/latest/python-api.html#converting-data-in-columns
- https://sqlite-utils.datasette.io/en/latest/cli.html#converting-data-in-columns
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values,
https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539035838,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539035838,IC_kwDOCGYnMM5bu9K-,9599,simonw,2023-05-08T20:55:00Z,2023-05-08T20:55:00Z,OWNER,"I'm going to go with `--no-skip-false` as the CLI option. It's ugly, but this whole thing is ugly. I'm going to make a note to remove this misfeature in `sqlite-utils` 4.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values,
https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1539033736,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1539033736,IC_kwDOCGYnMM5bu8qI,9599,simonw,2023-05-08T20:52:51Z,2023-05-08T20:52:51Z,OWNER,"OK, I implemented that at the Python API level. I need to decide how it should work for the `sqlite-utils convert` command too: https://sqlite-utils.datasette.io/en/stable/cli-reference.html#convert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values,
https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539018912,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539018912,IC_kwDOCGYnMM5bu5Cg,9599,simonw,2023-05-08T20:39:00Z,2023-05-08T20:39:00Z,OWNER,"I think the natural place to add these in the Python library API would be https://sqlite-utils.datasette.io/en/stable/python-api.html#adding-foreign-key-constraints - maybe something like this:

```python
db[""books""].add_foreign_key(""author_id"", ""authors"", ""id"", on_delete=RESTRICT)
```

Then for the CLI tool could be added to https://sqlite-utils.datasette.io/en/stable/cli-reference.html#add-foreign-key

```
sqlite-utils add-foreign-key my.db books author_id authors id --on-update SET_NULL
```
I wouldn't support these for the other methods of adding foreign keys - `foreign_keys(...)` for the various `.insert()` etc methods and the `add_foreign_keys(...)` bulk menthod.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:",
https://github.com/simonw/sqlite-utils/issues/530#issuecomment-1539015064,https://api.github.com/repos/simonw/sqlite-utils/issues/530,1539015064,IC_kwDOCGYnMM5bu4GY,9599,simonw,2023-05-08T20:35:07Z,2023-05-08T20:35:07Z,OWNER,"Wow, this is a neat feature I didn't know about. Looks like there are a bunch of options:

- NO ACTION (default)
- RESTRICT: application is prohibited from deleting a parent key when there exists one or more child keys mapped to it
- SET NULL: when a parent key is deleted the child key columns of all rows in the child table that mapped to the parent key are set to contain SQL NULL values
- SET DEFAULT: set a specific default
- CASCADE: propagates the delete or update operation on the parent key to each dependent child key","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1595340692,"add ability to configure ""on delete"" and ""on update"" attributes of foreign keys:",
https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539009453,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539009453,IC_kwDOCGYnMM5bu2ut,9599,simonw,2023-05-08T20:30:29Z,2023-05-08T20:30:42Z,OWNER,"Here's an improvement:
```
% sqlite-utils insert /tmp/b.db blah /tmp/blah.txt
  [####################################]  100%
Error: Invalid JSON - use --csv for CSV or --tsv for TSV files

JSON error: Expecting value: line 1 column 1 (char 0)
```","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/532#issuecomment-1539006509,https://api.github.com/repos/simonw/sqlite-utils/issues/532,1539006509,IC_kwDOCGYnMM5bu2At,9599,simonw,2023-05-08T20:28:56Z,2023-05-08T20:28:56Z,OWNER,Was this a newline-delimited JSON file perhaps?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620254998,Show more information when JSON can't be imported with sqlite-utils insert,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538975545,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538975545,IC_kwDOCGYnMM5buuc5,1231935,xavdid,2023-05-08T20:06:35Z,2023-05-08T20:06:35Z,NONE,"perfect, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/541#issuecomment-1538963959,https://api.github.com/repos/simonw/sqlite-utils/issues/541,1538963959,IC_kwDOCGYnMM5burn3,9599,simonw,2023-05-08T19:59:34Z,2023-05-08T19:59:34Z,OWNER,"There are 8 failing tests left:

```
==== short test summary info ====
FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bcaca0>
FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function Table.convert.<locals>.convert_value at 0x106bc9620>
ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ====
```
Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700840265,Get tests to pass with `pytest -Werror`,
https://github.com/simonw/sqlite-utils/issues/534#issuecomment-1538933540,https://api.github.com/repos/simonw/sqlite-utils/issues/534,1538933540,IC_kwDOCGYnMM5bukMk,9599,simonw,2023-05-08T19:34:37Z,2023-05-08T19:34:37Z,OWNER,"On macOS this shows the same warning:
```
% python -Wdefault $(which sqlite-utils) insert dogs.db dogs dogs.csv --csv
  [############------------------------]   35%
  [####################################]  100%/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/cli.py:1187: ResourceWarning: unclosed file <_io.TextIOWrapper name='dogs.csv' encoding='utf-8-sig'>
  insert_upsert_implementation(
ResourceWarning: Enable tracemalloc to get the object allocation traceback
```
The file itself is a `click.File` which is automatically closed - https://click.palletsprojects.com/en/8.1.x/api/#click.File - but it looks like it's the `_io.TextIOWrapper` which is not being closed:

https://github.com/simonw/sqlite-utils/blob/2376c452a56b0c3e75e7ca698273434e32945304/sqlite_utils/cli.py#L949-L956","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1622640374, ResourceWarning: unclosed file,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538921774,IC_kwDOCGYnMM5buhUu,9599,simonw,2023-05-08T19:24:41Z,2023-05-08T19:24:41Z,OWNER,That fix seems to work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538910894,IC_kwDOCGYnMM5buequ,9599,simonw,2023-05-08T19:16:52Z,2023-05-08T19:17:00Z,OWNER,"How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`?

Something like this:

```python
[
  ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']),
  ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538903556,IC_kwDOCGYnMM5buc4E,9599,simonw,2023-05-08T19:11:24Z,2023-05-08T19:13:23Z,OWNER,"I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though.

This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`:

https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538893329,IC_kwDOCGYnMM5buaYR,9599,simonw,2023-05-08T19:04:47Z,2023-05-08T19:04:47Z,OWNER,"This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`.

One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html

But...

> UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).

I still want to support SQLite versions earlier than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538889482,IC_kwDOCGYnMM5buZcK,9599,simonw,2023-05-08T19:02:38Z,2023-05-08T19:02:38Z,OWNER,"Here's the code at fault:
https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538887361,IC_kwDOCGYnMM5buY7B,9599,simonw,2023-05-08T19:01:20Z,2023-05-08T19:01:20Z,OWNER,"Here's the problem:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
db.execute('create table foo (id integer primary key, name not null)')
db.execute('insert into foo (id) values (1)')
```
Produces:
```
IntegrityError: NOT NULL constraint failed: foo.name
```
But this:
```python
db.execute('insert or ignore into foo (id) values (1)')
```
Completes without an exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538801855,IC_kwDOCGYnMM5buEC_,9599,simonw,2023-05-08T18:00:17Z,2023-05-08T18:00:17Z,OWNER,"From time in the debugger, after creating the table it ends up doing this:
```
(Pdb) queries_and_params
[
  ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]),
  ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538793817,IC_kwDOCGYnMM5buCFZ,9599,simonw,2023-05-08T17:55:10Z,2023-05-08T17:55:10Z,OWNER,"Confirmed - I added this test and it fails:
```python
def test_upsert_all_not_null(fresh_db):
    # https://github.com/simonw/sqlite-utils/issues/538
    fresh_db[""comments""].upsert_all(
        [{""id"": 1, ""name"": ""Cleo""}],
        pk=""id"",
        not_null=[""name""],
    )
    assert list(fresh_db[""comments""].rows) == [{""id"": 1, ""name"": ""Cleo""}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present,
https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537744000,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537744000,IC_kwDOCGYnMM5bqByA,42327,pquentin,2023-05-08T04:56:12Z,2023-05-08T04:56:12Z,NONE,"Hey @simonw, urllib3 maintainer here :wave:

Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just `python: ""3""` will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml

I can open PRs to sqlite-utils / datasette if you're interested","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error,
https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537514610,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537514610,IC_kwDOCGYnMM5bpJxy,9599,simonw,2023-05-07T18:43:24Z,2023-05-07T18:43:24Z,OWNER,"Documentation:
- https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query
- https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines",
https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537514069,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537514069,IC_kwDOCGYnMM5bpJpV,9599,simonw,2023-05-07T18:40:18Z,2023-05-07T18:40:18Z,OWNER,"https://docs.readthedocs.io/en/stable/config-file/v2.html suggests:

```yaml
build:
  os: ubuntu-22.04
  tools:
    python: ""3.11""
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error,
https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513912,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513912,IC_kwDOCGYnMM5bpJm4,9599,simonw,2023-05-07T18:39:29Z,2023-05-07T18:39:29Z,OWNER,"https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said:

> Problem in your project's configuration. Invalid ""python.version"": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error,
https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513653,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513653,IC_kwDOCGYnMM5bpJi1,9599,simonw,2023-05-07T18:37:59Z,2023-05-07T18:38:19Z,OWNER,"Useful comment here:

- https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928

> I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error,
https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507676,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507676,IC_kwDOCGYnMM5bpIFc,9599,simonw,2023-05-07T18:09:43Z,2023-05-07T18:09:54Z,OWNER,"This worked:

```bash
sqlite-utils memory /tmp/books3.json:nl \
  'select name from books3' --raw-lines > titles.txt
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines",
https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507525,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507525,IC_kwDOCGYnMM5bpIDF,9599,simonw,2023-05-07T18:09:09Z,2023-05-07T18:09:09Z,OWNER,"I'm tempted to upgrade `--raw` to do this instead, but that would be a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines",
https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507394,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507394,IC_kwDOCGYnMM5bpIBC,9599,simonw,2023-05-07T18:08:44Z,2023-05-07T18:08:44Z,OWNER,"Prototype:
```diff
diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst
index 153e5f9..c830518 100644
--- a/docs/cli-reference.rst
+++ b/docs/cli-reference.rst
@@ -124,6 +124,7 @@ See :ref:`cli_query`.
       --json-cols                 Detect JSON cols and output them as JSON, not
                                   escaped strings
       -r, --raw                   Raw output, first column of first row
+      --raw-lines                 Raw output, first column of each row
       -p, --param <TEXT TEXT>...  Named :parameters for SQL query
       --functions TEXT            Python code defining one or more custom SQL
                                   functions
@@ -192,6 +193,7 @@ See :ref:`cli_memory`.
       --json-cols                 Detect JSON cols and output them as JSON, not
                                   escaped strings
       -r, --raw                   Raw output, first column of first row
+      --raw-lines                 Raw output, first column of each row
       -p, --param <TEXT TEXT>...  Named :parameters for SQL query
       --encoding TEXT             Character encoding for CSV input, defaults to
                                   utf-8
diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py
index d25b1df..da0e4b6 100644
--- a/sqlite_utils/cli.py
+++ b/sqlite_utils/cli.py
@@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension):
 )
 @output_options
 @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"")
+@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"")
 @click.option(
     ""-p"",
     ""--param"",
@@ -1677,6 +1678,7 @@ def query(
     fmt,
     json_cols,
     raw,
+    raw_lines,
     param,
     load_extension,
     functions,
@@ -1700,7 +1702,19 @@ def query(
         _register_functions(db, functions)
 
     _execute_query(
-        db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
+        db,
+        sql,
+        param,
+        raw,
+        raw_lines,
+        table,
+        csv,
+        tsv,
+        no_headers,
+        fmt,
+        nl,
+        arrays,
+        json_cols,
     )
 
 
@@ -1728,6 +1742,7 @@ def query(
 )
 @output_options
 @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"")
+@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"")
 @click.option(
     ""-p"",
     ""--param"",
@@ -1773,6 +1788,7 @@ def memory(
     fmt,
     json_cols,
     raw,
+    raw_lines,
     param,
     encoding,
     no_detect_types,
@@ -1879,12 +1895,36 @@ def memory(
         _register_functions(db, functions)
 
     _execute_query(
-        db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
+        db,
+        sql,
+        param,
+        raw,
+        raw_lines,
+        table,
+        csv,
+        tsv,
+        no_headers,
+        fmt,
+        nl,
+        arrays,
+        json_cols,
     )
 
 
 def _execute_query(
-    db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols
+    db,
+    sql,
+    param,
+    raw,
+    raw_lines,
+    table,
+    csv,
+    tsv,
+    no_headers,
+    fmt,
+    nl,
+    arrays,
+    json_cols,
 ):
     with db.conn:
         try:
@@ -1903,6 +1943,13 @@ def _execute_query(
                 sys.stdout.buffer.write(data)
             else:
                 sys.stdout.write(str(data))
+        elif raw_lines:
+            for row in cursor:
+                data = row[0]
+                if isinstance(data, bytes):
+                    sys.stdout.buffer.write(data + b""\n"")
+                else:
+                    sys.stdout.write(str(data) + ""\n"")
         elif fmt or table:
             print(
                 tabulate.tabulate(
```
Needs tests and more documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines",
https://github.com/simonw/datasette/issues/2069#issuecomment-1537277919,https://api.github.com/repos/simonw/datasette/issues/2069,1537277919,IC_kwDOBm6k_c5boP_f,31861128,yqlbu,2023-05-07T03:17:35Z,2023-05-07T03:17:35Z,NONE,"Some updates:

I notice that there is an option in the CLI where we can explicitly set `immutable` mode when spinning up the server

```console
Options:
  -i, --immutable PATH            Database files to open in immutable mode
```

Then, the question is - how can I disable immutable mode in the deployed instance on Vercel?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1698865182,[BUG] Cannot insert new data to deployed instance,
https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1532481862,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1532481862,IC_kwDOCGYnMM5bV9FG,1231935,xavdid,2023-05-03T05:53:26Z,2023-05-03T05:53:26Z,NONE,"Would love to put our heads together for improvements here. 

I _think_ anything that is `argname=DEFAULT` needs to be typed as `argname: str | Default = DEFAULT` (replacing `str` with the appropriate type(s)). We may be able to get clever and tie the types to that key in the `_defaults` dict (definitely possible in Typescript, but I'm less familiar with advanced python types). 

Right now, all args are typed as `Default`, which means all callers get type errors. 

As for table/view, given that they don't have the same methods, it would be nice to be able to get one or the other specifically.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting,
https://github.com/simonw/datasette/issues/2067#issuecomment-1532304714,https://api.github.com/repos/simonw/datasette/issues/2067,1532304714,IC_kwDOBm6k_c5bVR1K,39538958,justmars,2023-05-03T00:16:03Z,2023-05-03T00:16:03Z,NONE,"Curiously, after running commands on the database that was litestream-restored, datasette starts to work again, e.g.

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
datasette data/db.sqlite
# fails (OperationalError described above)
```

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite-utils enable-wal data/db.sqlite
datasette data/db.sqlite
# works
```

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite-utils optimize data/db.sqlite
datasette data/db.sqlite
# works
```

```sh
litestream restore -o data/db.sqlite s3://mytestbucketxx/db
sqlite3 data/db.sqlite "".clone test.db""
datasette test.db
# works
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1690765434,Litestream-restored db: errors on 3.11 and 3.10.8; but works on py3.10.7 and 3.10.6,
https://github.com/simonw/datasette/pull/2052#issuecomment-1530822437,https://api.github.com/repos/simonw/datasette/issues/2052,1530822437,IC_kwDOBm6k_c5bPn8l,193185,cldellow,2023-05-02T03:35:30Z,2023-05-02T16:02:38Z,CONTRIBUTOR,"Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a `<script async src='...'/>` tag)?

```js
if (window.__DATASETTE__)
  go(window.__DATASETTE__);
else
  document.addEventListener(""datasette_init"", (evt) => go(evt.detail));

function go(manager) {
  manager.registerPlugin(...)
}
```

I don't know if it'd make sense, but you could also consider the asynchronous queuing pattern that Google Analytics uses (see [this Stack Overflow post](https://stackoverflow.com/questions/6963779/whats-the-name-of-google-analytics-async-design-pattern-and-where-is-it-used) for more details):

```js
__DATASETTE__ = __DATASETTE__ || [];
__DATASETTE__.push(go);

function go(manager) {
  manager.registerPlugin(...);
}
```","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1530817667,https://api.github.com/repos/simonw/datasette/issues/2052,1530817667,IC_kwDOBm6k_c5bPmyD,193185,cldellow,2023-05-02T03:24:53Z,2023-05-02T03:24:53Z,CONTRIBUTOR,"Thanks for putting this together! I've been slammed with work/personal stuff so haven't been able to actually prototype anything with this. :(

tl;dr: I think this would be useful immediately as is. It might also be nice if the plugins could return `Promise`s.

The long version: I read the design notes and example plugin. I think I'd be able to use this in [datasette-ui-extras](https://github.com/cldellow/datasette-ui-extras) for my lazy-facets feature.

The lazy-facets feature tries to provide a snappier user experience. It does this by altering how suggested facets work.

First, at page render time:
(A) it lies to Datasette and claims that no columns support facets, this avoids the lengthy delays/timeouts that can happen if the dataset is large.
(B) there's a python plugin that implements the [extra_body_script](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette) hook, to write out the list of column names for future use by JavaScript

Second, at page load time: there is some JavaScript that:
(C) makes AJAX requests to suggest facets for each column - it makes 1 request per column, using the data from (B)
(D) wires up the column menus to add Facet-by-this options for each facet

With the currently proposed plugin scheme, I think (D) could be moved into the plugin. I'd do the ajax requests, then register the plugin.

If the plugin scheme also supported promises, I think (B) and (C) could also be moved into the plugin.

Does that make sense? Sorry for the wall of text!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2064#issuecomment-1529737426,https://api.github.com/repos/simonw/datasette/issues/2064,1529737426,IC_kwDOBm6k_c5bLfDS,49699333,dependabot[bot],2023-05-01T13:58:50Z,2023-05-01T13:58:50Z,CONTRIBUTOR,Superseded by #2068.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1683229834,Bump sphinx from 6.1.3 to 6.2.1,
https://github.com/simonw/datasette/issues/2065#issuecomment-1524709988,https://api.github.com/repos/simonw/datasette/issues/2065,1524709988,IC_kwDOBm6k_c5a4Tpk,9599,simonw,2023-04-27T05:09:36Z,2023-04-27T05:09:36Z,OWNER,"That fixed it - after installing `main.zip` again I ran this and it worked:

    ~/.rye/tools/main-zip/bin/datasette install datasette-graphql

```
% ~/.rye/tools/main-zip/bin/datasette plugins                  
[
    {
        ""name"": ""datasette-graphql"",
        ""static"": true,
        ""templates"": true,
        ""version"": ""2.2"",
        ""hooks"": [
            ""database_actions"",
            ""extra_template_vars"",
            ""menu_links"",
            ""register_routes"",
            ""startup"",
            ""table_actions""
        ]
    }
]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye,
https://github.com/simonw/datasette/issues/2065#issuecomment-1524707628,https://api.github.com/repos/simonw/datasette/issues/2065,1524707628,IC_kwDOBm6k_c5a4TEs,9599,simonw,2023-04-27T05:06:44Z,2023-04-27T05:06:44Z,OWNER,"I need `pip` as a dependency too:

```
% ~/.rye/tools/main-zip/bin/datasette install datasette-graphql
Traceback (most recent call last):
  File ""/Users/simon/.rye/tools/main-zip/bin/datasette"", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/datasette/cli.py"", line 365, in install
    run_module(""pip"", run_name=""__main__"")
  File ""<frozen runpy>"", line 222, in run_module
  File ""<frozen runpy>"", line 142, in _get_module_details
ImportError: No module named pip
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye,
https://github.com/simonw/datasette/issues/2065#issuecomment-1524699863,https://api.github.com/repos/simonw/datasette/issues/2065,1524699863,IC_kwDOBm6k_c5a4RLX,9599,simonw,2023-04-27T04:56:22Z,2023-04-27T04:56:22Z,OWNER,Turned this into a TIL: https://til.simonwillison.net/python/rye,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524680160,https://api.github.com/repos/simonw/datasette/issues/2066,1524680160,IC_kwDOBm6k_c5a4MXg,9599,simonw,2023-04-27T04:27:50Z,2023-04-27T04:27:50Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524675817,https://api.github.com/repos/simonw/datasette/issues/2066,1524675817,IC_kwDOBm6k_c5a4LTp,9599,simonw,2023-04-27T04:21:03Z,2023-04-27T04:21:03Z,OWNER,I went with this to generate the long string: https://github.com/simonw/datasette/blob/0b0c5cd7a94fe3f151a3e10261b5c84ee64f2f18/tests/test_csv.py#L157-L176,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524669124,https://api.github.com/repos/simonw/datasette/issues/2066,1524669124,IC_kwDOBm6k_c5a4JrE,9599,simonw,2023-04-27T04:10:44Z,2023-04-27T04:10:52Z,OWNER,"I need an alternative way of generating a long string with a shorter URL.

```sql
select group_concat('abcabcabc', '') from json_each(json_array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10))
```
https://latest.datasette.io/_memory?sql=select+group_concat%28%27abcabcabc%27%2C+%27%27%29+from+json_each%28json_array%281%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%29%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524666049,https://api.github.com/repos/simonw/datasette/issues/2066,1524666049,IC_kwDOBm6k_c5a4I7B,9599,simonw,2023-04-27T04:06:18Z,2023-04-27T04:06:18Z,OWNER,"Most recent `httpx` release is 0.24 a couple of weeks ago. Here's what changed in that version: https://github.com/encode/httpx/compare/0.23.3...0.24.0

It looks like that URL limit is new: https://github.com/encode/httpx/commit/57daabf673705954afa94686c0002801c93d31f3#diff-78d8d93b5dd4c77d99c3e2b46b7286ba71a8fd60e92d8bd4eee5fb200b4f87bfR149-R155

```python
def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult:
    # Initial basic checks on allowable URLs.
    # ---------------------------------------

    # Hard limit the maximum allowable URL length.
    if len(url) > MAX_URL_LENGTH:
        raise InvalidURL(""URL too long"")
```
https://github.com/encode/httpx/blob/32e25497a36e6222cc64a758c98275b450dac28d/httpx/_urlparse.py#L153-L155","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524660603,https://api.github.com/repos/simonw/datasette/issues/2066,1524660603,IC_kwDOBm6k_c5a4Hl7,9599,simonw,2023-04-27T04:02:55Z,2023-04-27T04:02:55Z,OWNER,"In the debugger:
```
(Pdb) MAX_URL_LENGTH
65536
```
Weird this only seems to be a problem with `httpx` on Python 3.7 though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524659084,https://api.github.com/repos/simonw/datasette/issues/2066,1524659084,IC_kwDOBm6k_c5a4HOM,9599,simonw,2023-04-27T04:02:07Z,2023-04-27T04:02:07Z,OWNER,"This is the failing test:

https://github.com/simonw/datasette/blob/249fcf8e3e2a90e763f41b080c1b9ec8017f5005/tests/test_csv.py#L156-L167","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524655203,https://api.github.com/repos/simonw/datasette/issues/2066,1524655203,IC_kwDOBm6k_c5a4GRj,9599,simonw,2023-04-27T03:59:56Z,2023-04-27T03:59:56Z,OWNER,"In a fresh Datasette checkout I ran:

    pipenv shell --python /Users/simon/.pyenv/versions/3.7.16/bin/python

That gave me a virtual environment with 3.7.16 Python.

Then I ran:

    pip install -e '.[test]'

Weirdly that gave me a `which pytest` of `/opt/homebrew/bin/pytest` which ran the tests on 3.11.

I figured out the location of the virtual environment with `which python` and then ran this:

```
% /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/bin/pytest tests/test_csv.py 
============================================================================================== test session starts ===============================================================================================
platform darwin -- Python 3.7.16, pytest-7.3.1, pluggy-1.0.0
SQLite: 3.39.5
rootdir: /private/tmp/datasette
configfile: pytest.ini
plugins: asyncio-0.21.0, timeout-2.1.0, xdist-3.2.1, anyio-3.6.2
asyncio: mode=strict
collected 15 items                                                                                                                                                                                               

tests/test_csv.py ..........F....                                                                                                                                                                          [100%]

==================================================================================================== FAILURES ====================================================================================================
________________________________________________________________________________________________ test_max_csv_mb _________________________________________________________________________________________________

app_client_csv_max_mb_one = <datasette.utils.testing.TestClient object at 0x107ab5c08>

    def test_max_csv_mb(app_client_csv_max_mb_one):
        response = app_client_csv_max_mb_one.get(
            (
                ""/fixtures.csv?sql=select+'{}'+""
                ""from+compound_three_primary_keys&_stream=1&_size=max""
>           ).format(""abcdefg"" * 10000)
        )

/private/tmp/datasette/tests/test_csv.py:161: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:240: in __call__
    return call_result.result()
/Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:428: in result
    return self.__get_result()
/Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:384: in __get_result
    raise self._exception
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:306: in main_wrap
    result = await self.awaitable(*args, **kwargs)
/private/tmp/datasette/datasette/utils/testing.py:76: in get
    headers=headers,
/private/tmp/datasette/datasette/utils/testing.py:167: in _request
    content=post_body,
/private/tmp/datasette/datasette/app.py:1787: in request
    method, self._fix(path, avoid_path_rewrites), **kwargs
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:1528: in request
    extensions=extensions,
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:346: in build_request
    url = self._merge_url(url)
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:376: in _merge_url
    merge_url = URL(url)
/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urls.py:113: in __init__
    self._uri_reference = urlparse(url, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

url = ""http://localhost/fixtures.csv?sql=select+'abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcde...gabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg'+from+compound_three_primary_keys&_stream=1&_size=max""
kwargs = {}

    def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult:
        # Initial basic checks on allowable URLs.
        # ---------------------------------------
    
        # Hard limit the maximum allowable URL length.
        if len(url) > MAX_URL_LENGTH:
>           raise InvalidURL(""URL too long"")
E           httpx.InvalidURL: URL too long

/Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL
============================================================================================ short test summary info =============================================================================================
FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long
========================================================================================== 1 failed, 14 passed in 1.83s ==========================================================================================
(datasette) simon@Simons-MacBook-Pro datasette % 
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524648995,https://api.github.com/repos/simonw/datasette/issues/2066,1524648995,IC_kwDOBm6k_c5a4Ewj,9599,simonw,2023-04-27T03:56:42Z,2023-04-27T03:57:11Z,OWNER,"I don't have 3.7 locally. Trying to install it with `pyenv`.

    brew install pyenv

Then:

    pyenv install --list | grep 3.7

Installing:

    pyenv install 3.7.16

Output:

    Installed Python-3.7.16 to /Users/simon/.pyenv/versions/3.7.16

So the executable is `/Users/simon/.pyenv/versions/3.7.16/bin/python`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524638233,https://api.github.com/repos/simonw/datasette/issues/2066,1524638233,IC_kwDOBm6k_c5a4CIZ,9599,simonw,2023-04-27T03:50:51Z,2023-04-27T03:50:51Z,OWNER,Failure was on 3.7. I'll try that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2066#issuecomment-1524637376,https://api.github.com/repos/simonw/datasette/issues/2066,1524637376,IC_kwDOBm6k_c5a4B7A,9599,simonw,2023-04-27T03:50:19Z,2023-04-27T03:50:19Z,OWNER,"Having trouble replicating this on my laptop. I tried a fresh virtual environment with fresh packages (in case this is a `httpx` change) but this passed:

    pytest tests/test_csv.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long,
https://github.com/simonw/datasette/issues/2065#issuecomment-1524616740,https://api.github.com/repos/simonw/datasette/issues/2065,1524616740,IC_kwDOBm6k_c5a384k,9599,simonw,2023-04-27T03:38:21Z,2023-04-27T03:38:21Z,OWNER,"Tried this:

    rye install https://github.com/simonw/datasette/archive/refs/heads/main.zip

But got this error:

    Error: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `:`

I instead downloaded that file and ran:

    rye install main.zip

This worked! And now:
```
~/.rye/tools/main-zip/bin/datasette --version
```
```
datasette, version 1.0a2
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye,
https://github.com/simonw/datasette/pull/2063#issuecomment-1521837780,https://api.github.com/repos/simonw/datasette/issues/2063,1521837780,IC_kwDOBm6k_c5atWbU,49699333,dependabot[bot],2023-04-25T13:57:52Z,2023-04-25T13:57:52Z,CONTRIBUTOR,Superseded by #2064.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1681339696,Bump sphinx from 6.1.3 to 6.2.0,
https://github.com/simonw/datasette/pull/2052#issuecomment-1510423051,https://api.github.com/repos/simonw/datasette/issues/2052,1510423051,IC_kwDOBm6k_c5aBzoL,9020979,hydrosquall,2023-04-16T16:12:14Z,2023-04-20T05:14:39Z,CONTRIBUTOR,"# Javascript Plugin Docs (alpha)

## Motivation

The Datasette JS Plugin API allows developers to add interactive features to the UI, without having to modify the Python source code. 

## Setup

No external/NPM dependencies are needed.

Plugin behavior is coordinated by the Datasette `manager`. Every page has 1 `manager`.

There are 2 ways to add your plugin to the `manager`.

1. Read `window.__DATASETTE__` if the manager was already loaded.

```js
const manager = window.__DATASETTE__;
```

2. Wait for the `datasette_init` event to fire if your code was loaded before the manager is ready. 

```js
document.addEventListener(""datasette_init"", function (evt) {
  const { detail: manager } = evt;
  
  // register plugin here
});
```

3. Add plugin to the manager by calling `manager.registerPlugin` in a JS file. Each plugin will supply 1 or more hooks with

- unique name (`YOUR_PLUGIN_NAME`)
- a numeric version (starting at `0.1`), 
- configuration value, the details vary by hook. (In this example, `getColumnActions` takes a function)

```js
manager.registerPlugin(""YOUR_PLUGIN_NAME"", {
    version: 0.1,
    makeColumnActions: (columnMeta) => {
      return [
        {
          label: ""Copy name to clipboard"",
          // evt = native click event
          onClick: (evt) => copyToClipboard(columnMeta.column),
        }
      ];
    },
  });
```

There are 2 plugin hooks available to `manager.registerPlugin`:

- `makeColumnActions` - Add items to the cog menu for headers on datasette table pages
- `makeAboveTablePanelConfigs` - Add items to ""tabbed"" panel above the `<table/>` on pages that use the Datasette table template.

While there are additional properties on the `manager`, but it's not advised to depend on them directly as the shape is subject to change.

4. To make your JS file available as a Datasette plugin from the Python side, you can add a python file resembling [this](https://github.com/simonw/datasette/pull/2052/files#diff-c5ecf3d22075a60d04a4e95da2e15c612cf1bc84e38d777b67ba60dbd156e293) to your plugins directory. Note that you could host your JS file anywhere, it doesn't have to be served from the Datasette statics folder.

I welcome ideas for more hooks, or feedback on the current design!

## Examples

See the [example plugins file](https://github.com/simonw/datasette/blob/2d92b9328022d86505261bcdac419b6ed9cb2236/datasette/static/table-example-plugins.js) for additional examples.

## Hooks API Guide

### `makeAboveTablePanelConfigs`

Provide a function with a list of panel objects. Each panel object should contain

1. A unique string `id`
2. A string `label` for the tab
3. A `render` function. The first argument is reference to an HTML [Element](https://developer.mozilla.org/en-US/docs/Web/API/Element). 

Example:

```js
 manager.registerPlugin(""panel-plugin-graphs"", {
    version: 0.1,
    makeAboveTablePanelConfigs: () => {
      return [
        {
          id: 'first-panel',
          label: ""My new panel"",
          render: node => {
            const description = document.createElement('p');
            description.innerText = 'Hello world';
            node.appendChild(description);
          }
        }
      ];
    },
  });
```

### `makeColumnActions`

Provide a function that returns a list of action objects. Each action object has

1. A string `label` for the menu dropdown label
2. An onClick `render` function.

Example:

```js
  manager.registerPlugin(""column-name-plugin"", {
    version: 0.1,
    getColumnActions: (columnMeta) => {
      
      // Info about selected column. 
      const { columnName, columnNotNull, columnType, isPk } = columnMeta;

      return [
        {
          label: ""Copy name to clipboard"",
          onClick: (evt) => copyToClipboard(column),
        }
      ];
    },
  });
```

The getColumnActions callback has access to an object with metadata about the clicked column. These fields include:

- columnName: string (name of the column)
- columnNotNull: boolean
- columnType: sqlite datatype enum (text, number, etc)
- isPk: Whether this is the primary key: boolean

You can use this column metadata to customize the action config objects (for example, handling different summaries for text vs number columns).


","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1515694393,https://api.github.com/repos/simonw/datasette/issues/2052,1515694393,IC_kwDOBm6k_c5aV6k5,9020979,hydrosquall,2023-04-20T04:25:55Z,2023-04-20T04:25:55Z,CONTRIBUTOR,"Thanks for the thoughtful review and generous examples @asg017 ! I'll make the changes you suggested soon. Bonus thoughts inlined below.

> comments

These were very much appreciated, it's important to a plugin system that details like this feel right! I'll address them in batch later in the week. 

> I know TypeScript can be a little controversial

FWIW I am in favor of doing Typescript - I just wanted to keep the initial set of files in this PR as simple as possible to review. Really appreciate you scaffolding this initial set of types + I think it would be a welcome addition to maintain a set of types.d.ts files. 

I'm entertaining the idea of writing the actual source code in Typescript as long as the compiled output is readable b/c it can be tricky to keep the types and plain JS files in sync. Curious if you have encountered projects that are good at preventing drift.

> Maybe they should have more ""action-y"" names

This is a great observation. I'm inclined towards something like `make*` or `build*` since to me `add*` make me think the thing the method is attached to is being mutated, but I agree that any of these may be clearer than the current `get*` setup. I'll go through and update these. 

> Maybe we can make it easier to do pure-js datasette plugins?

I really like this idea! It'll be easier to get contributors if they don't have to touch the python side at _all_. 

> And then do the PERMITTED_VIEWS filtering in JS rather than Python.

One cost of doing this is that pages that won't use the JS would still have to load the unused code (given that I'm not sending up anything complex like lazy loading). But hopefully the manager core size is close to negligible, and it won't be a big deal. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/datasette/pull/2052#issuecomment-1510423215,https://api.github.com/repos/simonw/datasette/issues/2052,1510423215,IC_kwDOBm6k_c5aBzqv,9020979,hydrosquall,2023-04-16T16:12:59Z,2023-04-16T16:12:59Z,CONTRIBUTOR,"## Research notes

- I stuck to the ""minimal dependencies"" ethos of datasette (no React, Typescript, JS linting, etc).
- Main threads on JS plugin development
  - Main: sketch of pluggy-inspired system: https://github.com/simonw/datasette/issues/983
  - Main: provide locations in Datasette HTML that are designed for multiple plugins to safely cooperate with each other (starting with the panel, but eventually could extend to ""search boxes""): https://github.com/simonw/datasette/issues/1191
  - Main: HTML hooks for JS plugin authors: https://github.com/simonw/datasette/issues/987
- Prior threads on JS plugins in Datasette for future design directions
  - Idea: pass useful strings to JS plugins: https://github.com/simonw/datasette/issues/1565
  - Idea: help with plugin dependency loading: https://github.com/simonw/datasette/issues/1542 . (IMO - the plugin providing the dependency can emit an event once it's done.  Other plugins can listen for it, or ask the manager to inform them when the dependency is available). 
  - Idea: help plugins to manage state in shareable URLs (plugins shouldn't have to interact with the URL directly, should have some basic insulation from clobbering each others' keys): https://github.com/simonw/datasette/issues/1144
- Articles on plugins reviewed
  - https://css-tricks.com/designing-a-javascript-plugin-system/
- Plugin/Extension systems reviewed (mostly JS).
  - Yarn: https://yarnpkg.com/advanced/plugin-tutorial
  - Tappable https://github.com/webpack/tapable (used by Auto, webpack)
  - Pluggy: https://pluggy.readthedocs.io/en/stable/
  - VSCode: https://code.visualstudio.com/api/get-started/your-first-extension
  - Chrome: https://developer.chrome.com/docs/extensions/reference/
  - Figma/Figjam Widget: https://www.figma.com/widget-docs/
  - Datadog Apps: [Programming Model](https://github.com/DataDog/apps/blob/master/docs/en/programming-model.md)
  - Storybook: https://storybook.js.org/docs/react/addons/addons-api","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/simonw/sqlite-utils/issues/425#issuecomment-1509951952,https://api.github.com/repos/simonw/sqlite-utils/issues/425,1509951952,IC_kwDOCGYnMM5aAAnQ,89400147,Dhyanesh97,2023-04-15T20:14:58Z,2023-04-15T20:14:58Z,NONE,is this change released ? Because when we run docker containers issue still persists for production deployments.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1203842656,`sqlite3.NotSupportedError`: deterministic=True requires SQLite 3.8.3 or higher,
https://github.com/simonw/datasette/pull/2056#issuecomment-1509852821,https://api.github.com/repos/simonw/datasette/issues/2056,1509852821,IC_kwDOBm6k_c5Z_oaV,3709715,cclauss,2023-04-15T14:24:45Z,2023-04-15T14:24:45Z,NONE,Status?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff,
https://github.com/simonw/datasette/pull/2052#issuecomment-1509461324,https://api.github.com/repos/simonw/datasette/issues/2052,1509461324,IC_kwDOBm6k_c5Z-I1M,9020979,hydrosquall,2023-04-15T01:57:06Z,2023-04-15T01:57:06Z,CONTRIBUTOR,Notes from 1:1 - it _is_ possible to pass in URL params into a ObservableHQ notebook: https://observablehq.com/@bherbertlc/pass-values-as-url-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)",
https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-1190995982,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31,1190995982,IC_kwDOD079W85G_SgO,19231792,jakewilkins,2022-07-21T03:26:38Z,2023-04-14T22:41:31Z,NONE,"👋 Any update on getting this merged?

Alternatively, is there a work around for this issue to unblock myself?

edit to add: huge fan of both this project and `osxphotos`, thanks so much for your work here 🙏  If I had any experience with Python I would offer to help but somehow I've managed to not write any Python in 10+ years of programming 😅 

Edit again to add:

> Alternatively, is there a work around for this issue to unblock myself?

Yes, there is. I was able to apply the patch of this PR and it applies (mostly) cleanly and works.

- verified I have a high enough version of `osxphotos`
- downloaded the .patch of this (by appending `.patch` to the URL)
- edited the patch to remove the `setup.py` changes
- `cd` to the directory containing `dogsheep-photos` and `git apply 31.patch`
","{""total_count"": 4, ""+1"": 4, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",771511344,Update for Big Sur,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1508784533,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6,1508784533,IC_kwDOJHON9s5Z7jmV,579727,sirnacnud,2023-04-14T15:22:09Z,2023-04-14T15:22:09Z,NONE,"Just changing the encoding in `extract_notes` to `utf8` seems to fix it for my titles that were messed up.

![Screen Shot 2023-04-14 at 5 14 18 PM](https://user-images.githubusercontent.com/579727/232086062-e7edc4d1-0880-417a-925b-fd6c65b05155.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617602868,Character encoding problem,
https://github.com/simonw/datasette/issues/2058#issuecomment-1507264934,https://api.github.com/repos/simonw/datasette/issues/2058,1507264934,IC_kwDOBm6k_c5Z1wmm,1138559,esagara,2023-04-13T16:35:21Z,2023-04-13T16:35:21Z,NONE,"I tried deploying the fix you submitted, but still getting the same errors. I can past the logs here if needed, but I really don't see anything new in them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/datasette/pull/2056#issuecomment-1506485287,https://api.github.com/repos/simonw/datasette/issues/2056,1506485287,IC_kwDOBm6k_c5ZyyQn,3709715,cclauss,2023-04-13T07:29:38Z,2023-04-13T07:41:55Z,NONE,"Ruff (written in Rust, not Python) is a 23MB executable so the time to download and pip install it dwarfs its runtime.

Let's run ruff with and without GitHub Actions pip cache side-by-side to see the relative performance.

Once you approve the workflows below, `ruff_with_cache` should echo `cache-hit = false` but if you rerun that job hopefully it should echo `cache-hit = true`.  That will be the execution time that we are interested to compare.

There are two great problems in computer science: ;-)
1. Naming things
2. Cache invalidation
3. Off-by-one errors

For 2., https://github.com/actions/setup-python#caching-packages-dependencies is vital reading.
Only _exactly pinned requirements_ can be cached.  Currently in `setup.py` the only pinned dependencies are:
1. Sphinx==6.1.3
2. furo==2023.3.27
3. black==23.3.0
4. blacken-docs==1.13.0  # but unpinned elsewhere in `setup.py`

This means that there will be very few cache hits in the current actions.  See the link below to print out cache hits:
https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#cache-hit

> PyPI uses Fastly's CDN to quickly serve content to end-users, allowing us to minimize our hosting infrastructure and obscure possible downtime. -- https://pypi.org/sponsors

I would be shocked if Fastly does not have beefy CDN nodes in the same datacenters where GitHub Actions run so GHA requests to download `ruff` probably never hit a PyPI server.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff,
https://github.com/simonw/sqlite-utils/issues/527#issuecomment-1506223848,https://api.github.com/repos/simonw/sqlite-utils/issues/527,1506223848,IC_kwDOCGYnMM5Zxybo,9599,simonw,2023-04-13T02:08:16Z,2023-04-13T02:08:16Z,OWNER,"I agree, this is a design flaw.

It's technically a breaking change. As such, I would need to bump to v4 to responsibly release this.

I'd rather bundle in a few more breaking changes before shipping that version.

One thing we could do here is add an extra argument to `.convert()` - something like this:

```python
table.convert(col, lambda x: x+1, skip_false=False)
```

This would trigger the new, improved behaviour without breaking existing code that depends on how it works at the moment.

Then in `sqlite-utils` 4 we can change the default of that option.

What do you think?

(I'm open to suggestions for better names for this parameter too)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578790070,`Table.convert()` skips falsey values,
https://github.com/simonw/datasette/issues/2058#issuecomment-1506203550,https://api.github.com/repos/simonw/datasette/issues/2058,1506203550,IC_kwDOBm6k_c5Zxtee,547438,cephillips,2023-04-13T01:48:21Z,2023-04-13T01:48:21Z,NONE,Really interesting how you are using ChatGPT in this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1506200813,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1506200813,IC_kwDOCGYnMM5Zxszt,22429695,codecov[bot],2023-04-13T01:45:22Z,2023-04-13T01:45:22Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00`**% and no project coverage change.
> Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`a75abeb`)](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #537   +/-   ##
=======================================
  Coverage   96.25%   96.25%           
=======================================
  Files           6        6           
  Lines        2671     2673    +2     
=======================================
+ Hits         2571     2573    +2     
  Misses        100      100           
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <100.00%> (+<0.01%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`,
https://github.com/simonw/datasette/pull/2056#issuecomment-1506179555,https://api.github.com/repos/simonw/datasette/issues/2056,1506179555,IC_kwDOBm6k_c5Zxnnj,9599,simonw,2023-04-13T01:21:05Z,2023-04-13T01:22:08Z,OWNER,"OK, I'm sold - this is a really neat improvement.

One thing to change in the PR: right now it runs `pip install --user ruff` on every commit, which hits PyPI to install the package.

I prefer to avoid hitting PyPI every time, so I like to use the GitHub Actions cache. My usual pattern for that looks like this:

https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/.github/workflows/test-pyodide.yml#L16-L20

Then a separate command that runs `pip install ...` will benefit from that cache.

Are you OK to make that change?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff,
https://github.com/simonw/datasette/pull/2056#issuecomment-1506177857,https://api.github.com/repos/simonw/datasette/issues/2056,1506177857,IC_kwDOBm6k_c5ZxnNB,9599,simonw,2023-04-13T01:18:18Z,2023-04-13T01:18:18Z,OWNER,"Cool - and now https://github.com/simonw/datasette/pull/2056/files is showing me those inline annotations:

<img width=""868"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/231620660-4843d113-b671-494e-8265-24226446abba.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff,
https://github.com/simonw/datasette/pull/2056#issuecomment-1506177115,https://api.github.com/repos/simonw/datasette/issues/2056,1506177115,IC_kwDOBm6k_c5ZxnBb,9599,simonw,2023-04-13T01:17:16Z,2023-04-13T01:17:16Z,OWNER,"Here are the failures: https://github.com/simonw/datasette/actions/runs/4684460653/jobs/8300630794?pr=2056

<img width=""902"" alt=""CleanShot 2023-04-12 at 18 16 45@2x"" src=""https://user-images.githubusercontent.com/9599/231620533-bce286b7-8173-4a61-aac1-b772832fb893.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff,
https://github.com/simonw/datasette/pull/2056#issuecomment-1506175208,https://api.github.com/repos/simonw/datasette/issues/2056,1506175208,IC_kwDOBm6k_c5Zxmjo,9599,simonw,2023-04-13T01:14:13Z,2023-04-13T01:14:13Z,OWNER,https://github.com/simonw/datasette/actions/runs/4664796647/jobs/8300596121?pr=2056 it's pretty fast - that finished in 9s.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff,
https://github.com/simonw/datasette/issues/2059#issuecomment-1506174353,https://api.github.com/repos/simonw/datasette/issues/2059,1506174353,IC_kwDOBm6k_c5ZxmWR,9599,simonw,2023-04-13T01:13:00Z,2023-04-13T01:13:00Z,OWNER,"Can you provide a URL to an example, and/or a screenshot of this? Is it a browser warning or is it a warning from Heroku itself?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665053646,"""Deceptive site ahead"" alert on Heroku deployment",
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504288134,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504288134,IC_kwDOCGYnMM5ZqZ2G,9599,simonw,2023-04-11T23:55:06Z,2023-04-12T03:34:32Z,OWNER,"Also checked the official Datasette Docker image - I had to run that in Codespaces because it doesn't currently work on my M2 Mac:

```
codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker pull datasetteproject/datasette
Using default tag: latest
...
codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker run -it datasetteproject/datasette /
bin/bash
root@75ba34f501ec:/# python
Python 3.11.0 (main, Dec  6 2022, 13:31:55) [GCC 10.2.1 20210110] on linux
Type ""help"", ""copyright"", ""credits"" or ""license"" for more information.
>>> import sqlite3
.executescript(""""""
PRAGMA writable_schema = 1;
UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
PRAGMA writable_schema = 0;
"""""")>>> db = sqlite3.connect("":memory:"")
>>> db.executescript(""""""
... PRAGMA writable_schema = 1;
... UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
... PRAGMA writable_schema = 0;
... """""")
<sqlite3.Cursor object at 0x7fd9b0561140>
>>>
```
So that confirms that the official image also has a Python with a SQLite that's not in defensive mode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/datasette/issues/2058#issuecomment-1504426792,https://api.github.com/repos/simonw/datasette/issues/2058,1504426792,IC_kwDOBm6k_c5Zq7so,9599,simonw,2023-04-12T02:02:42Z,2023-04-12T02:02:42Z,OWNER,"I tightened up the benchmark (it was measuring the time taken to create the tables too) and got this:

![image](https://user-images.githubusercontent.com/9599/231328328-85ca35ac-a11b-46f4-b132-dae367103570.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/datasette/issues/2058#issuecomment-1504328395,https://api.github.com/repos/simonw/datasette/issues/2058,1504328395,IC_kwDOBm6k_c5ZqjrL,9599,simonw,2023-04-12T00:28:38Z,2023-04-12T00:28:38Z,OWNER,"Here's a much better chart, which shows that MD5 performance unsurprisingly gets worse as the number of tables increases while `schema_version` remains constant:

![image](https://user-images.githubusercontent.com/9599/231316778-513bd99f-5ea4-495c-b86d-c572a7106369.png)
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/datasette/issues/2058#issuecomment-1504315697,https://api.github.com/repos/simonw/datasette/issues/2058,1504315697,IC_kwDOBm6k_c5Zqgkx,9599,simonw,2023-04-12T00:16:22Z,2023-04-12T00:27:12Z,OWNER,"I got ChatGPT (code execution alpha) to run a micro-benchmark for me. This was the conclusion:

> The benchmark using `PRAGMA schema_version` is approximately 1.36 times faster than the benchmark using `hashlib.md5` for the case with 100 tables. For the case with 200 tables, the benchmark using `PRAGMA schema_version` is approximately 2.33 times faster than the benchmark using `hashlib.md5`.

Here's the chart it drew me:

![image](https://user-images.githubusercontent.com/9599/231315366-3a12b6d3-08d7-419d-a1fd-36eb24da0d85.png)

(It's a pretty rubbish chart though, it only took measurements at 100 and 200 and drew a line between the two, I should have told it to measure every 10 and plot that)

And the full transcript: https://gist.github.com/simonw/2fc46effbfbe49e6de0bcfdc9e31b235

The benchmark looks good enough on first glance that I don't feel the need to be more thorough with it. `PRAGMA schema_version` is faster, but not so fast that I feel like the MD5 hack is worth worrying about too much.

I'm tempted to add something to the `/-/versions` page that tries to identify if this is a problem or not though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/datasette/issues/2058#issuecomment-1504298448,https://api.github.com/repos/simonw/datasette/issues/2058,1504298448,IC_kwDOBm6k_c5ZqcXQ,9599,simonw,2023-04-12T00:04:01Z,2023-04-12T00:04:01Z,OWNER,"Here's a potential workaround: when I store the schema versions, I could also score an MD5 hash of the full schema (`select group_concat(sql) from sqlite_master`). When I read the schema version with `PRAGMA schema_version` I could catch that exception and, if I see it, I could calculate that MD5 hash again as a fallback and use that to determine if the schema has changed instead.

The performance overhead of this needs investigating - how much more expensive is `md5(... that SQL query result)` compared to just `PRAGMA schema_version`, especially on a database with a lot of tables?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/datasette/issues/2058#issuecomment-1504295345,https://api.github.com/repos/simonw/datasette/issues/2058,1504295345,IC_kwDOBm6k_c5Zqbmx,9599,simonw,2023-04-12T00:01:42Z,2023-04-12T00:02:26Z,OWNER,"Here's the relevant code:

https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/app.py#L421-L437

This function is called on almost every request (everything that subclasses `BaseView` at least - need to remember that for the refactor in #2053 etc).

https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/base.py#L101-L103

It uses `PRAGMA schema_version` as a cheap way to determine if the schema has changed, in which case it needs to refresh the internal schema tables.

This was already the cause of a subtle bug here:
- #1231
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/datasette/issues/2058#issuecomment-1504292145,https://api.github.com/repos/simonw/datasette/issues/2058,1504292145,IC_kwDOBm6k_c5Zqa0x,9599,simonw,2023-04-11T23:58:59Z,2023-04-11T23:58:59Z,OWNER,Asked on the SQLite Forum if anyone has seen this before: https://sqlite.org/forum/forumpost/793a2ed75b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/datasette/issues/2058#issuecomment-1504291892,https://api.github.com/repos/simonw/datasette/issues/2058,1504291892,IC_kwDOBm6k_c5Zqaw0,9599,simonw,2023-04-11T23:58:45Z,2023-04-11T23:58:45Z,OWNER,"I thought it might relate to the ""defensive mode"" issue described here:

- https://github.com/simonw/sqlite-utils/issues/235

But I have since determined that the Datasette official Docker image does NOT run anything in defensive mode, so I don't think it's related to that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""",
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504245029,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504245029,IC_kwDOCGYnMM5ZqPUl,9599,simonw,2023-04-11T23:13:41Z,2023-04-11T23:14:39Z,OWNER,"I also tested this against the current `ubuntu:latest` Docker image (on an M2 Mac), in Python 3.10 and 3.11:
```
docker run -it ubuntu:latest /bin/bash
```
Then in the container:
```
apt-get update
apt-get install python3
python3
# pasted in the above recipe
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.11
python3.11 
# pasted it in again
```
In both cases the Python code did not raise an exception, which suggests that on Ubuntu those two Python versions do not have the defensive mode set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/datasette/issues/2057#issuecomment-1503838640,https://api.github.com/repos/simonw/datasette/issues/2057,1503838640,IC_kwDOBm6k_c5ZosGw,9599,simonw,2023-04-11T17:48:23Z,2023-04-11T17:48:23Z,OWNER,"> This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory.

I was right about that:

```pycon
>>> importlib.resources.files('datasette_graphql')
PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql')
>>> importlib.resources.files('datasette_graphql').joinpath(""static"")
PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static')
>>> p = importlib.resources.files('datasette_graphql').joinpath(""static"")
>>> p
PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static')
>>> p.is_
p.is_absolute()     p.is_char_device()  p.is_fifo()         p.is_mount()        p.is_reserved()     p.is_symlink()     
p.is_block_device() p.is_dir()          p.is_file()         p.is_relative_to(   p.is_socket()      
>>> p.is_dir()
True
>>> p.is_file()
False
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1503832422,https://api.github.com/repos/simonw/datasette/issues/2057,1503832422,IC_kwDOBm6k_c5Zoqlm,9599,simonw,2023-04-11T17:42:57Z,2023-04-11T17:46:42Z,OWNER,"I ran this prompt against ChatGPT with the Browsing alpha:

> ```python
>                 if pkg_resources.resource_isdir(plugin.__name__, ""static""):
>                     static_path = pkg_resources.resource_filename(
>                         plugin.__name__, ""static""
>                     )
>                 if pkg_resources.resource_isdir(plugin.__name__, ""templates""):
>                     templates_path = pkg_resources.resource_filename(
>                         plugin.__name__, ""templates""
>                     )
> ```
> This code gives a deprecation warning in Python 3.11 - fix it

It looked up the fix for me:

<img width=""632"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/231245259-af4d962c-e631-420b-8312-fd9657a23bbd.png"">

And suggested:

```python
import importlib.resources

# Replace pkg_resources.resource_isdir with importlib.resources.files().is_file()
if importlib.resources.files(plugin.__name__).joinpath(""static"").is_file():
    static_path = importlib.resources.as_file(
        importlib.resources.files(plugin.__name__).joinpath(""static"")
    )
if importlib.resources.files(plugin.__name__).joinpath(""templates"").is_file():
    templates_path = importlib.resources.as_file(
        importlib.resources.files(plugin.__name__).joinpath(""templates"")
    )
```
This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/simonw/datasette/issues/2057#issuecomment-1503833906,https://api.github.com/repos/simonw/datasette/issues/2057,1503833906,IC_kwDOBm6k_c5Zoq8y,9599,simonw,2023-04-11T17:44:16Z,2023-04-11T17:45:45Z,OWNER,"Another prompt:

> How to fix this:
>
> `pkg_resources.get_distribution(package).version`

Response:

<img width=""641"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/231245559-dd943527-818a-45aa-9833-b9db8c7ca87e.png"">

```python
import importlib.metadata

# Get the version number of the specified package
package_version = importlib.metadata.version(package)
```

That seems to work:

```pycon
>>> import importlib.metadata
>>> importlib.metadata.version(""datasette"")
'0.64.2'
>>> importlib.metadata.version(""pluggy"")
'1.0.0'
>>> importlib.metadata.version(""not-a-package"")
...
importlib.metadata.PackageNotFoundError: No package metadata was found for not-a-package
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629404,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502629404,IC_kwDODD6af85ZkE4c,9599,simonw,2023-04-11T03:15:47Z,2023-04-11T03:46:17Z,MEMBER,"I think `swarm-to-sqlite` needs to avoid this error, maybe by setting up foreign keys in another way - or even by skipping foreign keys entirely on databases that don't support this kind of operation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629219,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502629219,IC_kwDODD6af85ZkE1j,9599,simonw,2023-04-11T03:15:26Z,2023-04-11T03:15:26Z,MEMBER,"OK, I figured this out. Unfortunately it's an error that occurs on Python versions that have defensive mode turned on, and it doesn't look like there's a way to turn that mode off. See notes above.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502559442,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502559442,IC_kwDOCGYnMM5ZjzzS,9599,simonw,2023-04-11T01:32:30Z,2023-04-11T01:33:27Z,OWNER,"This seems to work:
```python
import sqlite3
db = sqlite3.connect("":memory:"")
db.executescript(""""""
PRAGMA writable_schema = 1;
UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';
PRAGMA writable_schema = 0;
"""""")
```
It succeeds on my Python 3.11 and raises the following exception on my broken Python 3.9:
```
sqlite3.OperationalError: table sqlite_master may not be modified
```
Removing the `PRAGMA writable_schema = 1;` causes the same exception to be raised on both Pythons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502557629,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502557629,IC_kwDOCGYnMM5ZjzW9,9599,simonw,2023-04-11T01:30:12Z,2023-04-11T01:30:12Z,OWNER,"I'll ask on the SQLite forum if it's possible to toggle that mode on and off using regular SQL. My hunch is that it isn't.

In which case `sqlite-utils` should at least know how to catch this error and display a much more readable error message, maybe with a link to further documentation.

A utility function that can detect this mode would be really useful too. I'd probably have to do a test that tries to modify `sqlite_master` on a new in-memory database to catch if it's possible or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502556111,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502556111,IC_kwDOCGYnMM5Zjy_P,9599,simonw,2023-04-11T01:28:41Z,2023-04-11T01:28:41Z,OWNER,"Investigating this one now.

The `sqlite-utils` test suite passes without errors on my Python 3.11.2 installation... but it fails with this error on a Python 3.9.6 installation.

In the broken version's virtual environment directory I ran this:
```
cat pyvenv.cfg 
home = /Applications/Xcode.app/Contents/Developer/usr/bin
implementation = CPython
version_info = 3.9.6.final.0
virtualenv = 20.17.1
include-system-site-packages = false
base-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9
base-exec-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9
base-executable = /Applications/Xcode.app/Contents/Developer/usr/bin/python3
```
So it looks like the Xcode `python3` has ""defensive"" mode turned on for SQLite.

As far as I can tell there's no way to turn it OFF again in Python.

My virtual environment that DOES work has this:

```
home = /opt/homebrew/opt/python@3.11/bin
implementation = CPython
version_info = 3.11.2.final.0
virtualenv = 20.17.1
include-system-site-packages = false
base-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
base-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11
base-executable = /opt/homebrew/opt/python@3.11/bin/python3.11
```
So the Python 3.11 I installed through Homebrew doesn't exhibit this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502546045,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502546045,IC_kwDODD6af85Zjwh9,9599,simonw,2023-04-11T01:14:50Z,2023-04-11T01:14:50Z,MEMBER,"Related:
- https://github.com/simonw/sqlite-utils/issues/235","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502543165,https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13,1502543165,IC_kwDODD6af85Zjv09,9599,simonw,2023-04-11T01:10:36Z,2023-04-11T01:11:47Z,MEMBER,"I just had that error myself on macOS while running the tests:
```
ERROR tests/test_save_checkin.py::test_tables - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_venue - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_event - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_sticker - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_likes - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_with_ - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_users - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_photos - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_posts - sqlite3.OperationalError: table sqlite_master may not be modified
ERROR tests/test_save_checkin.py::test_view - sqlite3.OperationalError: table sqlite_master may not be modified
```
`pytest --pdb` shows it happening in the bit that adds foreign keys:
```
> /Users/simon/.local/share/virtualenvs/swarm-to-sqlite-daPW7yIJ/lib/python3.9/site-packages/sqlite_utils/db.py(1096)add_foreign_keys()
-> cursor.execute(
(Pdb) list
1096 >>	                cursor.execute(
1097 	                    ""UPDATE sqlite_master SET sql = ? WHERE name = ?"",
1098 	                    (new_sql, table_name),
1099 	                )
1100 	            cursor.execute(""PRAGMA schema_version = %d"" % (schema_version + 1))
1101 ->	            cursor.execute(""PRAGMA writable_schema = 0"")
1102 	        # Have to VACUUM outside the transaction to ensure .foreign_keys property
1103 	        # can see the newly created foreign key.
1104 	        self.vacuum()
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1373210675,fails before generating views. ERR:  table sqlite_master may not be modified,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1501017004,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1501017004,IC_kwDOCGYnMM5Zd7Os,25778,eyeseast,2023-04-09T01:49:43Z,2023-04-09T01:49:43Z,CONTRIBUTOR,I'm going to close this in favor of #536. Will try a cleaner approach to custom paths once that one is merge.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/sqlite-utils/pull/536#issuecomment-1500893216,https://api.github.com/repos/simonw/sqlite-utils/issues/536,1500893216,IC_kwDOCGYnMM5ZddAg,22429695,codecov[bot],2023-04-08T13:35:42Z,2023-04-08T13:35:42Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`cea05dc`)](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main     #536   +/-   ##
=======================================
  Coverage   96.25%   96.25%           
=======================================
  Files           6        6           
  Lines        2671     2671           
=======================================
  Hits         2571     2571           
  Misses        100      100           
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1659525418,Add paths for homebrew on Apple silicon,
https://github.com/simonw/datasette/issues/2054#issuecomment-1500608101,https://api.github.com/repos/simonw/datasette/issues/2054,1500608101,IC_kwDOBm6k_c5ZcXZl,9599,simonw,2023-04-07T20:14:38Z,2023-04-07T20:14:38Z,OWNER,"Ooh that one's really interesting - very different from the others:
```ruby
# app.rb
require ""roda""

class App < Roda
  route do |r|
    r.root do
      ""Home page""
    end

    r.on ""pages"" do
      r.get "":slug"" do |slug|
        ""Page: #{slug}""
      end
    end

    r.on ""news"" do
      r.get "":yyyy/:mm/:dd"" do |yyyy, mm, dd|
        ""News for #{yyyy}/#{mm}/#{dd}""
      end
    end
  end
end

# config.ru
require_relative ""app""
run App.freeze.app
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465038901,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465038901,IC_kwDOCGYnMM5XUrg1,22429695,codecov[bot],2023-03-11T22:29:19Z,2023-04-07T18:19:49Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`100.00`**% and project coverage change: **`+0.03`** :tada:
> Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`afdf618`)](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.29%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #531      +/-   ##
==========================================
+ Coverage   96.25%   96.29%   +0.03%     
==========================================
  Files           6        6              
  Lines        2671     2671              
==========================================
+ Hits         2571     2572       +1     
+ Misses        100       99       -1     
```


| [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | |
| [sqlite\_utils/cli.py](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.09%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/531?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/datasette/issues/2054#issuecomment-1499797384,https://api.github.com/repos/simonw/datasette/issues/2054,1499797384,IC_kwDOBm6k_c5ZZReI,6213,dsisnero,2023-04-07T00:46:50Z,2023-04-07T00:46:50Z,NONE,you should have a look at Roda written in ruby . ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499604822,https://api.github.com/repos/simonw/datasette/issues/2054,1499604822,IC_kwDOBm6k_c5ZYidW,9599,simonw,2023-04-06T20:48:19Z,2023-04-06T20:48:55Z,OWNER,"I actually quite like that. I could even use `@classmethod` and have utility methods defined on that class that both `get()` and `post()` could call.

The crucial rule here is NO INSTANCE STATE - that's what makes routing to classes particularly damaging, and encourages code that's hard to maintain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499604066,https://api.github.com/repos/simonw/datasette/issues/2054,1499604066,IC_kwDOBm6k_c5ZYiRi,9599,simonw,2023-04-06T20:47:30Z,2023-04-06T20:47:30Z,OWNER,"I'm contemplating a new approach: using a class with static methods. Something like this:

```python
class TableView(MethodRouter):
    @staticmethod
    async def get(request):
        return Response.text(""GET"")

    @staticmethod
    async def post(request):
        return Response.text(""POST"")
```
So effectively the class is just there to bundle together verb implementations, and to provide a `route(request)` method which knows how to dispatch them to the right place.

It can offer default HEAD and OPTIONS methods too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499491258,https://api.github.com/repos/simonw/datasette/issues/2054,1499491258,IC_kwDOBm6k_c5ZYGu6,9599,simonw,2023-04-06T19:00:16Z,2023-04-06T20:40:16Z,OWNER,"I asked GPT-4 to compare URL routing based on path and verb approaches in different languages and frameworks and spit out some examples. Full transcript here: https://gist.github.com/simonw/fa2379b97420404a81b0fcdb4db79657

Initial prompt:

> Show me code examples of different web frameworks in Python and JavaScript and Go illustrating how HTTP routing works - in particular the problem of mapping an incoming HTTP request to some code based on both the URL path and the HTTP verb
>
> Don't show installation instructions, just get straight to the code snippets

Flask:
```python
from flask import Flask

app = Flask(__name__)

@app.route('/example', methods=['GET'])
def get_example():
    return ""This is a GET request.""

@app.route('/example', methods=['POST'])
def post_example():
    return ""This is a POST request.""

if __name__ == '__main__':
    app.run()
```
Express:
```javascript
const express = require('express');
const app = express();

app.get('/example', (req, res) => {
  res.send('This is a GET request.');
});

app.post('/example', (req, res) => {
  res.send('This is a POST request.');
});

app.listen(3000, () => {
  console.log('Server is running on port 3000');
});
```
Gin:
```go
package main

import (
	""github.com/gin-gonic/gin""
)

func main() {
	r := gin.Default()

	r.GET(""/example"", func(c *gin.Context) {
		c.String(200, ""This is a GET request."")
	})

	r.POST(""/example"", func(c *gin.Context) {
		c.String(200, ""This is a POST request."")
	})

	r.Run()
}
```
Actix-web:
```rust
use actix_web::{web, App, HttpResponse, HttpServer, Responder};

async fn get_example() -> impl Responder {
    HttpResponse::Ok().body(""This is a GET request."")
}

async fn post_example() -> impl Responder {
    HttpResponse::Ok().body(""This is a POST request."")
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| {
        App::new()
            .route(""/example"", web::get().to(get_example))
            .route(""/example"", web::post().to(post_example))
    })
    .bind(""127.0.0.1:8080"")?
    .run()
    .await
}
```
Koa:
```javascript
const Koa = require('koa');
const Router = require('koa-router');

const app = new Koa();
const router = new Router();

router.get('/example', (ctx) => {
  ctx.body = 'This is a GET request.';
});

router.post('/example', (ctx) => {
  ctx.body = 'This is a POST request.';
});

app.use(router.routes());
app.use(router.allowedMethods());

app.listen(3000, () => {
  console.log('Server is running on port 3000');
});
```
Django:
```python
from django.http import HttpResponse
from django.views import View

class ExampleView(View):
    def get(self, request):
        return HttpResponse(""This is a GET request."")

    def post(self, request):
        return HttpResponse(""This is a POST request."")
```
```python
from django.urls import path
from .views import ExampleView

urlpatterns = [
    path('example/', ExampleView.as_view(), name='example'),
]
```
Rocket:
```rust
#![feature(proc_macro_hygiene, decl_macro)]

#[macro_use] extern crate rocket;

use rocket::response::content;

#[get(""/example"")]
fn get_example() -> content::Plain<&'static str> {
    content::Plain(""This is a GET request."")
}

#[post(""/example"")]
fn post_example() -> content::Plain<&'static str> {
    content::Plain(""This is a POST request."")
}

#[launch]
fn rocket() -> _ {
    rocket::build().mount(""/"", routes![get_example, post_example])
}
```
Elixir/Phoenix:
```elixir
defmodule ExampleAppWeb.Router do
  use ExampleAppWeb, :router

  pipeline :api do
    plug :accepts, [""json""]
  end

  scope ""/api"", ExampleAppWeb do
    pipe_through :api

    get ""/example"", ExampleController, :get_example
    post ""/example"", ExampleController, :post_example
  end
end
```
Bottle:
```python
from bottle import Bottle, run

app = Bottle()

@app.route('/example', method='GET')
def get_example():
    return ""This is a GET request.""

@app.route('/example', method='POST')
def post_example():
    return ""This is a POST request.""

if __name__ == '__main__':
    run(app, host='localhost', port=8080)
```
FastAPI:
```python
from fastapi import FastAPI

app = FastAPI()

@app.get(""/example"")
async def get_example():
    return {""message"": ""This is a GET request.""}

@app.post(""/example"")
async def post_example():
    return {""message"": ""This is a POST request.""}

if __name__ == ""__main__"":
    import uvicorn
    uvicorn.run(app, host=""127.0.0.1"", port=8000)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499596941,https://api.github.com/repos/simonw/datasette/issues/2054,1499596941,IC_kwDOBm6k_c5ZYgiN,9599,simonw,2023-04-06T20:39:35Z,2023-04-06T20:39:35Z,OWNER,"I think there are two main approaches:

- Map path to a function, that function then behaves differently on GET/POST/etc
- Map method AND path to a function - the `app.get('/example')` v.s. `app.post('/example')` ones

The decorator approach is calling out to me a bit now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499530921,https://api.github.com/repos/simonw/datasette/issues/2054,1499530921,IC_kwDOBm6k_c5ZYQap,9599,simonw,2023-04-06T19:41:52Z,2023-04-06T19:42:09Z,OWNER,Turned this into a TIL: https://til.simonwillison.net/gpt3/gpt4-api-design,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499467703,https://api.github.com/repos/simonw/datasette/issues/2054,1499467703,IC_kwDOBm6k_c5ZYA-3,9599,simonw,2023-04-06T18:37:22Z,2023-04-06T18:57:22Z,OWNER,"I've been hoping to move entirely away from class-based views, but now I'm wondering if that's the right decision given the need to support HTTP verbs.

It is cleaner to have a class that has methods for each verb, or to have a single function that can behave differently depending on the verb?

Or should I have a mechanism for dispatching to separate functions based on the verbs that isn't implemented in a base class?

The trouble with using base classes is that I've already shown that they tempt me to put logic in weird places, which makes code harder to modify later on.

The thing here is that the ONLY thing I want to use the base class for is as an HTTP verb switcher - which makes me doubt that I should risk taking on the other temptations of having a base class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499465648,https://api.github.com/repos/simonw/datasette/issues/2054,1499465648,IC_kwDOBm6k_c5ZYAew,9599,simonw,2023-04-06T18:35:03Z,2023-04-06T18:35:03Z,OWNER,"There are actually five classes that subclass `DataView`:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/row.py#L16

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L34

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L172

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L215

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/table.py#L72

I don't think `DatabaseView` and `DatabaseDownload` should have subclassed that at all, since they don't return a table of data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499462324,https://api.github.com/repos/simonw/datasette/issues/2054,1499462324,IC_kwDOBm6k_c5ZX_q0,9599,simonw,2023-04-06T18:31:56Z,2023-04-06T18:31:56Z,OWNER,"The `DataView` class does a LOT of work - mostly involving CSV responses.

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L160-L544

It has a `redirect()` method with some complex logic and CORS handling:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L163-L172

It uses this method a lot, which has to be over-ridden in the `TableView` and `RowView` and `QueryView` subclasses:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L174-L175

This method:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L180

Is the bulk of the complexity, because it knows how to both turn a list of SQLite rows into a CSV file but also knows how to call `.data()` repeatedly with different pagination arguments in order to stream CSV back for a large table.

The `async def get()` method for GET requests is also very complicated. It mainly handles format stuff - knowing how to render HTML v.s. JSON v.s. CSV v.s. other formats specified using this plugin hook: https://docs.datasette.io/en/1.0a2/plugin_hooks.html#register-output-renderer-datasette

Plus it catches interrupted queries and returns a special error page for those (and other error messages too): https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L381-L408

It adds the time taken to execute the queries: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L410-L411","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499457291,https://api.github.com/repos/simonw/datasette/issues/2054,1499457291,IC_kwDOBm6k_c5ZX-cL,9599,simonw,2023-04-06T18:26:45Z,2023-04-06T18:26:45Z,OWNER,"
Here's `BaseView`:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L56-L145

It has methods for the `options`, `get`, `post`, `delete`, `put`, `patch` and `head` HTTP verbs, most defaulting to returinng a 405 Method not allowed message in plain text or JSON, depending on this check:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L71-L81

Also adds CORS headers to anything if CORS mode is on:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L106-L107

And adds the `database_color` (weirdly) and the `select_templates` variables to the template context:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L112-L122

And has special code for setting the `Link: ...; rel=""alternate""` HTTP header:

https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L124-L136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499457201,https://api.github.com/repos/simonw/datasette/issues/2054,1499457201,IC_kwDOBm6k_c5ZX-ax,9599,simonw,2023-04-06T18:26:39Z,2023-04-06T18:26:39Z,OWNER,These classes - `TableView` and `RowView` and `QueryView` - all subclass `DataView` which subclasses` BaseView`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/issues/2054#issuecomment-1499452122,https://api.github.com/repos/simonw/datasette/issues/2054,1499452122,IC_kwDOBm6k_c5ZX9La,9599,simonw,2023-04-06T18:21:51Z,2023-04-06T18:21:51Z,OWNER,"I'm going to make notes against the code in the most recent alpha release, ignoring the recent work I did to refactor `TableView`.

https://github.com/simonw/datasette/tree/1.0a2/datasette/views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now",
https://github.com/simonw/datasette/pull/2053#issuecomment-1498279469,https://api.github.com/repos/simonw/datasette/issues/2053,1498279469,IC_kwDOBm6k_c5ZTe4t,9599,simonw,2023-04-05T23:28:53Z,2023-04-05T23:28:53Z,OWNER,"Table errors page currently does this:
```json
{
  ""ok"": false,
  ""error"": ""no such column: blah"",
  ""status"": 400,
  ""title"": ""Invalid SQL""
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries,
https://github.com/simonw/datasette/issues/2049#issuecomment-1498275621,https://api.github.com/repos/simonw/datasette/issues/2049,1498275621,IC_kwDOBm6k_c5ZTd8l,9599,simonw,2023-04-05T23:23:01Z,2023-04-05T23:23:01Z,OWNER,"The default representation here can be even smaller.

For rows it's this:

```json
{
  ""ok"": true,
  ""next"": ""d,v"",
  ""rows"": [...]
}
```
For SQL queries I'm considering this:
```json
{
  ""ok"": true,
  ""rows"": [...]
}
```
I considered adding `""sql""` and `""params""` too, but on further thought those would be entirely a waste of bytes the majority of the time. If a user wants those they can request them with an `?_extra=query` as seen here:

http://localhost:8001/content/releases.json?_size=0&_extra=query

```json
{
  ""ok"": true,
  ""next"": null,
  ""query"": {
    ""sql"": ""select html_url, id, author, node_id, tag_name, target_commitish, name, draft, prerelease, created_at, published_at, body, repo, reactions, mentions_count from releases order by id limit 1"",
    ""params"": {}
  },
  ""rows"": []
}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format,
https://github.com/dogsheep/github-to-sqlite/issues/79#issuecomment-1498167714,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/79,1498167714,IC_kwDODFdgUs5ZTDmi,9599,simonw,2023-04-05T21:12:55Z,2023-04-05T21:12:55Z,MEMBER,"It's a rate limiting problem: https://github.com/dogsheep/github-to-sqlite/actions/runs/4622674009/jobs/8175633155

```
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/cli.py"", line 407, in commits
    utils.save_commits(db, commits, repo_full[""id""])
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 635, in save_commits
    for commit in commits:
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 417, in fetch_commits
    for commits in paginate(url, headers):
  File ""/home/runner/work/github-to-sqlite/github-to-sqlite/github_to_sqlite/utils.py"", line 478, in paginate
    raise GitHubError.from_response(response)
github_to_sqlite.utils.GitHubError: ('API rate limit exceeded for user ID 9599.', 403)
Error: Process completed with exit code 1.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1570375808,Deploy demo job is failing due to rate limit,
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1495780111,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1495780111,IC_kwDOCGYnMM5ZJ8sP,19786848,Thomascountz,2023-04-04T11:09:56Z,2023-04-04T11:13:40Z,NONE,"@wpears' workaround also worked for me, but also required me to manually set `PRAGMA writable_schema`.

```sql
PRAGMA writable_schema = 1;
UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (...)'
PRAGMA writable_schema = 0;
```

```shell
$ python --version
Python 3.11.2

$ sqlite3 --version
3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da

$ sqlite-utils --version
sqlite-utils, version 3.30
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/6#issuecomment-1493442956,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/6,1493442956,IC_kwDOJHON9s5ZBCGM,14314871,amlestin,2023-04-02T21:20:43Z,2023-04-02T21:25:37Z,NONE,"I'm experiencing something similar. My apostrophes (') turn into (’) and the output is truncated. Hoping to debug next weekend 
<img width=""814"" alt=""Screenshot 2023-04-02 at 4 24 10 PM"" src=""https://user-images.githubusercontent.com/14314871/229379762-f622205a-7bba-4598-b0da-a6b86e5c0a44.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617602868,Character encoding problem,
https://github.com/simonw/sqlite-utils/issues/265#issuecomment-1493052396,https://api.github.com/repos/simonw/sqlite-utils/issues/265,1493052396,IC_kwDOCGYnMM5Y_ivs,154364,dracos,2023-04-01T17:27:18Z,2023-04-01T17:27:18Z,NONE,"`enable_fts` is a function in datasette, not in this repo, which doesn't do any escaping of search terms. It sounds like from https://docs.datasette.io/en/stable/full_text_search.html#advanced-sqlite-search-queries you might want to enable raw searching, as otherwise it's disabled and everything is escaped by default.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",907795562,Using enable_fts before search term,
https://github.com/simonw/sqlite-utils/issues/159#issuecomment-1493051222,https://api.github.com/repos/simonw/sqlite-utils/issues/159,1493051222,IC_kwDOCGYnMM5Y_idW,154364,dracos,2023-04-01T17:21:05Z,2023-04-01T17:21:05Z,NONE,"In a related issue, nearly a year later I just stumbled across this again, as I wondered why none of my rebuild-fts were rebuilding. It looks like: disable_fts in db.py commits; enable_fts partly commits except the last step (due to executescript committing a pending transaction); rebuild_fts won't commit unless manually done as above with e.g. a context manager.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",702386948,.delete_where() does not auto-commit (unlike .insert() or .upsert()),
https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1492777509,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1492777509,IC_kwDOCGYnMM5Y-fol,9020979,hydrosquall,2023-04-01T01:31:48Z,2023-04-01T01:31:48Z,NONE,"My current workaround is to use this library from a python script instead of as a CLI tool.

This lets me set the foreign key constraint at table creation time, instead of trying to modify an existing table. [docs](https://sqlite-utils.datasette.io/en/stable/python-api.html#specifying-foreign-keys)

I found this [stackoverflow helpful](https://stackoverflow.com/a/1884841/5129731), as it explained that Sqlite doesn't support modifying existing tables directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,
https://github.com/simonw/datasette/issues/2050#issuecomment-1492387771,https://api.github.com/repos/simonw/datasette/issues/2050,1492387771,IC_kwDOBm6k_c5Y9Ae7,9599,simonw,2023-03-31T17:59:48Z,2023-03-31T17:59:48Z,OWNER,"Some of the extras from the table view make sense here. A few custom ones make sense too - including this already existing but undocumented one:

https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/row.py#L86-L89","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1649791661,Row page JSON should use new ?_extra= format,
https://github.com/simonw/datasette/issues/2035#issuecomment-1492206593,https://api.github.com/repos/simonw/datasette/issues/2035,1492206593,IC_kwDOBm6k_c5Y8UQB,9599,simonw,2023-03-31T16:09:08Z,2023-03-31T16:09:08Z,OWNER,"I could ship this as part of:
- #2049 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/1989#issuecomment-1491357104,https://api.github.com/repos/simonw/datasette/issues/1989,1491357104,IC_kwDOBm6k_c5Y5E2w,1231935,xavdid,2023-03-31T06:17:23Z,2023-03-31T06:18:05Z,NONE,"I'm running into a similar use case as pax above- I made a `nice` view that just has the data I'm interested in (which doesn't include the `id`, since it's not important in this case). But, by excluding `id` from the view, I can't do fts queries against it because the view has no `id` field to tie to `rowid`:

```
ERROR: conn=<sqlite3.Connection object at 0x106521210>, 
sql = 'select time, text, permalink, num_children from nice where id in 
(select rowid from items_fts where items_fts match :search)  limit 101', 
params = {'search': 'whatever'}: no such column: id
```

It works fine when I include `id` in my view, but now my `nice` view is cluttered up. Would be great to hide it permanently in the `config.json`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1531991339,Suggestion: Hiding columns,
https://github.com/simonw/datasette/issues/2049#issuecomment-1489530555,https://api.github.com/repos/simonw/datasette/issues/2049,1489530555,IC_kwDOBm6k_c5YyG67,9599,simonw,2023-03-30T00:51:27Z,2023-03-30T00:51:27Z,OWNER,"I'd really like to refactor all of the extras functions into a `datasette/extras.py` module. The table ones currently rely a LOT on local variables in scope though, so I would need to rewrite those such that EVERY dependency they take is passed to `asyncinject` explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format,
https://github.com/simonw/datasette/issues/2049#issuecomment-1489530037,https://api.github.com/repos/simonw/datasette/issues/2049,1489530037,IC_kwDOBm6k_c5YyGy1,9599,simonw,2023-03-30T00:50:30Z,2023-03-30T00:50:30Z,OWNER,"Two things to consider here: `_shape=` and `_extra=`.

Most of the shapes make sense, with the exception of `?_shape=object` since we don't know which column we would use as a primary key.

Looking at the (undocumented) list of extras from the table view, here are the ones I think make sense:

- `count` - YES
- `facet_results` - no
- `facets_timed_out` - no
- `suggested_facets` - no
- `human_description_en` - no
- `next_url` - MAYBE
- `columns` - YES
- `primary_keys` - no
- `display_columns` - YES
- `display_rows` - YES
- `debug` - YES?
- `request` - YES
- `query` - YES
- `metadata` - YES
- `extras` - YES
- `database` - YES
- `table` - no
- `database_color` - no?
- `table_actions` - no
- `filters` - no
- `renderers` - YES
- `custom_table_templates` - no
- `sorted_facet_results` - no
- `table_definition` - no
- `view_definition` - no
- `is_view` - no
- `private` - YES
- `expandable_columns` - no
- `form_hidden_args` - no

Just the YES ones:

- `count` - this is new
- `columns`
- `display_columns`
- `display_rows`
- `debug`
- `request`
- `query`
- `metadata`
- `extras`
- `database`
- `renderers`
- `private`

The `count` one is interesting - I think I can provide that by optionally running `select count(*) from (inner query)`. It's a new feature though and not one I want to expose on the HTML view since it could result in poor performance - but having it as an extra that API users can opt into may make sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format,
https://github.com/simonw/datasette/issues/2049#issuecomment-1489526501,https://api.github.com/repos/simonw/datasette/issues/2049,1489526501,IC_kwDOBm6k_c5YyF7l,9599,simonw,2023-03-30T00:44:05Z,2023-03-30T00:44:05Z,OWNER,"As part of this I should be able to figure out which bits of the new code I wrote for the table view should actually be shared with the query view. That stuff is mostly going to be from this commit: https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d

Here's the existing QueryView class I need to replace:

https://github.com/simonw/datasette/blob/4c1e277edbd783d06840d3f9b20bf00783478ce4/datasette/views/database.py#L215-L532","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1646734246,Custom SQL queries should use new JSON ?_extra= format,
https://github.com/simonw/datasette/pull/2034#issuecomment-1489306198,https://api.github.com/repos/simonw/datasette/issues/2034,1489306198,IC_kwDOBm6k_c5YxQJW,4370201,wenhoujx,2023-03-29T20:56:19Z,2023-03-29T20:56:19Z,NONE,@simonw any idea why the test coverage check fails? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1613974869,remove an unused `app` var in cli.py,
https://github.com/dogsheep/hacker-news-to-sqlite/pull/6#issuecomment-1489110168,https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/6,1489110168,IC_kwDODtX3eM5YwgSY,1231935,xavdid,2023-03-29T18:36:16Z,2023-03-29T18:36:16Z,NONE,@simonw can you take a look when you have a chance?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1641117021,Add permalink virtual field to items table,
https://github.com/simonw/datasette/issues/262#issuecomment-1488010837,https://api.github.com/repos/simonw/datasette/issues/262,1488010837,IC_kwDOBm6k_c5YsT5V,9599,simonw,2023-03-29T06:22:21Z,2023-03-29T06:22:21Z,OWNER,I need to get the arbitrary query page to return the same format. It likely won't have nearly as many extras.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/pull/1992#issuecomment-1384039611,https://api.github.com/repos/simonw/datasette/issues/1992,1384039611,IC_kwDOBm6k_c5SfsS7,22429695,codecov[bot],2023-01-16T13:12:32Z,2023-03-29T06:10:23Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage has no change and project coverage change: **`-0.31`** :warning:
> Comparison is base [(`c025b01`)](https://codecov.io/gh/simonw/datasette/commit/c025b0180fe344e4f28177ecd7cf388d492d3064?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41% compared to head [(`bfaa59b`)](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11%.

> :exclamation: Current head bfaa59b differs from pull request most recent head 64dd698. Consider uploading reports for the commit 64dd698 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #1992      +/-   ##
==========================================
- Coverage   92.41%   92.11%   -0.31%     
==========================================
  Files          38       38              
  Lines        5750     5555     -195     
==========================================
- Hits         5314     5117     -197     
- Misses        436      438       +2     
```


[see 9 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/1992/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1534904478,Bump blacken-docs from 1.12.1 to 1.13.0,
https://github.com/simonw/datasette/pull/2014#issuecomment-1487999503,https://api.github.com/repos/simonw/datasette/issues/2014,1487999503,IC_kwDOBm6k_c5YsRIP,49699333,dependabot[bot],2023-03-29T06:09:11Z,2023-03-29T06:09:11Z,CONTRIBUTOR,Superseded by #2047.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1566081801,Bump black from 22.12.0 to 23.1.0,
https://github.com/simonw/datasette/pull/2014#issuecomment-1487998788,https://api.github.com/repos/simonw/datasette/issues/2014,1487998788,IC_kwDOBm6k_c5YsQ9E,9599,simonw,2023-03-29T06:08:23Z,2023-03-29T06:08:23Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1566081801,Bump black from 22.12.0 to 23.1.0,
https://github.com/simonw/datasette/pull/2043#issuecomment-1486944644,https://api.github.com/repos/simonw/datasette/issues/2043,1486944644,IC_kwDOBm6k_c5YoPmE,49699333,dependabot[bot],2023-03-28T13:58:20Z,2023-03-28T13:58:20Z,CONTRIBUTOR,Superseded by #2046.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1639446870,Bump furo from 2022.12.7 to 2023.3.23,
https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484277416,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484277416,IC_kwDOCGYnMM5YeEao,9599,simonw,2023-03-26T23:43:12Z,2023-03-26T23:43:12Z,OWNER,Fixed: https://sqlite-utils.datasette.io/en/latest/reference.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions,
https://github.com/simonw/datasette/issues/1608#issuecomment-1484276946,https://api.github.com/repos/simonw/datasette/issues/1608,1484276946,IC_kwDOBm6k_c5YeETS,9599,simonw,2023-03-26T23:41:11Z,2023-03-26T23:41:11Z,OWNER,It's working again now: https://docs.datasette.io/en/latest/sql_queries.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/,
https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484276302,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484276302,IC_kwDOCGYnMM5YeEJO,9599,simonw,2023-03-26T23:39:27Z,2023-03-26T23:39:27Z,OWNER,"Actually final Datasette fix was:
```html+jinja
{% block scripts %}
{{ super() }}
<script>
document.addEventListener(""DOMContentLoaded"", function() {
  // Show banner linking to /stable/ if this is a /latest/ page
  if (!/\/latest\//.test(location.pathname)) {
    return;
  }
  var stableUrl = location.pathname.replace(""/latest/"", ""/stable/"");
  // Check it's not a 404
  fetch(stableUrl, { method: ""HEAD"" }).then((response) => {
    if (response.status === 200) {
      var warning = document.createElement(""div"");
      warning.className = ""admonition warning"";
      warning.innerHTML = `
        <p class=""first admonition-title"">Note</p>
        <p class=""last"">
          This documentation covers the <strong>development version</strong> of Datasette.
        </p>
        <p>
          See <a href=""${stableUrl}"">this page</a> for the current stable release.
        </p>
      `;
      var mainArticle = document.querySelector(""article[role=main]"");
      mainArticle.insertBefore(warning, mainArticle.firstChild);
    }
  });
});
</script>
{% endblock %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions,
https://github.com/simonw/datasette/issues/1608#issuecomment-1484274796,https://api.github.com/repos/simonw/datasette/issues/1608,1484274796,IC_kwDOBm6k_c5YeDxs,9599,simonw,2023-03-26T23:38:18Z,2023-03-26T23:38:18Z,OWNER,"ChatGPT prompt:

> ```
> jQuery(function ($) {
>   // Show banner linking to /stable/ if this is a /latest/ page
>   if (!/\/latest\//.test(location.pathname)) {
>     return;
>   }
>   var stableUrl = location.pathname.replace(""/latest/"", ""/stable/"");
>   // Check it's not a 404
>   fetch(stableUrl, { method: ""HEAD"" }).then((response) => {
>     if (response.status == 200) {
>       var warning = $(
>         `<div class=""admonition warning"">
>            <p class=""first admonition-title"">Note</p>
>            <p class=""last"">
>              This documentation covers the <strong>development version</strong> of Datasette.</p>
>              <p>See <a href=""${stableUrl}"">this page</a> for the current stable release.
>            </p>
>         </div>`
>       );
>       warning.find(""a"").attr(""href"", stableUrl);
>       $(""article[role=main]"").prepend(warning);
>     }
>   });
> });
> 
> Rewrite this to use vanilla JavaScript, no jQuery
> ```

Got back:

```javascript
document.addEventListener(""DOMContentLoaded"", function() {
  // Show banner linking to /stable/ if this is a /latest/ page
  if (!/\/latest\//.test(location.pathname)) {
    return;
  }
  var stableUrl = location.pathname.replace(""/latest/"", ""/stable/"");
  // Check it's not a 404
  fetch(stableUrl, { method: ""HEAD"" }).then((response) => {
    if (response.status === 200) {
      var warning = document.createElement(""div"");
      warning.className = ""admonition warning"";
      warning.innerHTML = `
        <p class=""first admonition-title"">Note</p>
        <p class=""last"">
          This documentation covers the <strong>development version</strong> of Datasette.
        </p>
        <p>
          See <a href=""${stableUrl}"">this page</a> for the current stable release.
        </p>
      `;
      var mainArticle = document.querySelector(""article[role=main]"");
      mainArticle.insertBefore(warning, mainArticle.firstChild);
    }
  });
});
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/,
https://github.com/simonw/datasette/issues/1608#issuecomment-1484272313,https://api.github.com/repos/simonw/datasette/issues/1608,1484272313,IC_kwDOBm6k_c5YeDK5,9599,simonw,2023-03-26T23:36:17Z,2023-03-26T23:36:17Z,OWNER,"Still getting this error:

<img width=""537"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/227811947-a35a6758-cf94-4196-89ce-85ae2f48ab06.png"">

Weird because `jQuery` is present in the devtools console.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/,
https://github.com/simonw/sqlite-utils/issues/388#issuecomment-1484269326,https://api.github.com/repos/simonw/sqlite-utils/issues/388,1484269326,IC_kwDOCGYnMM5YeCcO,9599,simonw,2023-03-26T23:24:12Z,2023-03-26T23:28:14Z,OWNER,"This broke on the upgrade to Furo.

I fixed it in Datasette like this: https://github.com/simonw/datasette/commit/5c1cfa451d78e3935193f5e10eba59bf741241a1 and https://github.com/simonw/datasette/commit/db8cf899e286fbaa0a40f3a9ae8d5aaa1478822e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1114543475,Link to stable docs from older versions,
https://github.com/simonw/datasette/issues/1608#issuecomment-1484270067,https://api.github.com/repos/simonw/datasette/issues/1608,1484270067,IC_kwDOBm6k_c5YeCnz,9599,simonw,2023-03-26T23:26:10Z,2023-03-26T23:27:42Z,OWNER,"That didn't work, because `jQuery` isn't yet defined that high up the template.

I should use `{% block scripts %}` instead: https://github.com/pradyunsg/furo/blob/05350110cf02d033ebb57147eb54e9371a63545a/src/furo/theme/furo/base.html#L89-L102","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/,
https://github.com/simonw/datasette/issues/1608#issuecomment-1484268146,https://api.github.com/repos/simonw/datasette/issues/1608,1484268146,IC_kwDOBm6k_c5YeCJy,9599,simonw,2023-03-26T23:23:22Z,2023-03-26T23:23:22Z,OWNER,This broke when I upgraded to the Furo theme.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1109808154,Documentation should clarify /stable/ vs /latest/,
https://github.com/simonw/datasette/pull/2031#issuecomment-1483248966,https://api.github.com/repos/simonw/datasette/issues/2031,1483248966,IC_kwDOBm6k_c5YaJVG,82332573,tmcl-it,2023-03-24T18:35:24Z,2023-03-24T18:35:24Z,NONE,I've rebased my patch on the latest main. It should be ready to merge.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,Expand foreign key references in row view as well,
https://github.com/simonw/datasette/pull/2031#issuecomment-1457016788,https://api.github.com/repos/simonw/datasette/issues/2031,1457016788,IC_kwDOBm6k_c5W2E_U,22429695,codecov[bot],2023-03-06T21:18:18Z,2023-03-24T18:29:49Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`97.72`**% and project coverage change: **`-0.36`** :warning:
> Comparison is base [(`3feed1f`)](https://codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`c8a2904`)](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11%.

> :exclamation: Current head c8a2904 differs from pull request most recent head ef25867. Consider uploading reports for the commit ef25867 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2031      +/-   ##
==========================================
- Coverage   92.46%   92.11%   -0.36%     
==========================================
  Files          38       38              
  Lines        5750     5556     -194     
==========================================
- Hits         5317     5118     -199     
- Misses        433      438       +5     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `92.57% <97.56%> (-3.35%)` | :arrow_down: |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.17% <100.00%> (+2.38%)` | :arrow_up: |
| [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.93% <100.00%> (+0.10%)` | :arrow_up: |

... and [8 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/2031/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/2031?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,Expand foreign key references in row view as well,
https://github.com/simonw/datasette/issues/1619#issuecomment-1483009959,https://api.github.com/repos/simonw/datasette/issues/1619,1483009959,IC_kwDOBm6k_c5YZO-n,6613091,henrikek,2023-03-24T15:38:04Z,2023-03-24T15:38:04Z,NONE,"I also have the same problem when running behind an apache proxy server with base_url. However, I have researched the problem a bit and have come to the conclusion that if you change _facet_date to _facet in the following https://github.com/simonw/datasette/blob/3feed1f66e2b746f349ee56970a62246a18bb164/datasette/facets.py#LL493C57-L493C68 , the facets work. But I'm not sure if this has other consequences?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used,
https://github.com/simonw/datasette/pull/2043#issuecomment-1482859217,https://api.github.com/repos/simonw/datasette/issues/2043,1482859217,IC_kwDOBm6k_c5YYqLR,22429695,codecov[bot],2023-03-24T14:05:54Z,2023-03-24T14:05:54Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/datasette/pull/2043?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage has no change and project coverage change: **`-0.06`** :warning:
> Comparison is base [(`3feed1f`)](https://codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`85b9e51`)](https://codecov.io/gh/simonw/datasette/pull/2043?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2043      +/-   ##
==========================================
- Coverage   92.46%   92.41%   -0.06%     
==========================================
  Files          38       38              
  Lines        5750     5750              
==========================================
- Hits         5317     5314       -3     
- Misses        433      436       +3     
```


[see 2 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/2043/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/2043?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1639446870,Bump furo from 2022.12.7 to 2023.3.23,
https://github.com/simonw/datasette/issues/262#issuecomment-1480355670,https://api.github.com/repos/simonw/datasette/issues/262,1480355670,IC_kwDOBm6k_c5YPG9W,9599,simonw,2023-03-22T22:50:30Z,2023-03-22T22:50:30Z,OWNER,"I just landed this PR so this feature is now in `main`:
- #1999

Still needs documentation and maybe some extra tests too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/datasette/pull/1999#issuecomment-1480349156,https://api.github.com/repos/simonw/datasette/issues/1999,1480349156,IC_kwDOBm6k_c5YPFXk,22429695,codecov[bot],2023-03-22T22:40:58Z,2023-03-22T22:48:57Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch coverage: **`87.89`**% and project coverage change: **`-4.43`** :warning:
> Comparison is base [(`56b0758`)](https://codecov.io/gh/simonw/datasette/commit/56b0758a5fbf85d01ff80a40c9b028469d7bb65f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.15% compared to head [(`921faae`)](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 87.73%.

> :exclamation: Current head 921faae differs from pull request most recent head 69a31cd. Consider uploading reports for the commit 69a31cd to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #1999      +/-   ##
==========================================
- Coverage   92.15%   87.73%   -4.43%     
==========================================
  Files          38       38              
  Lines        5560     6066     +506     
==========================================
+ Hits         5124     5322     +198     
- Misses        436      744     +308     
```


| [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | |
|---|---|---|
| [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `96.61% <ø> (ø)` | |
| [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `87.82% <ø> (ø)` | |
| [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `69.11% <86.76%> (-23.46%)` | :arrow_down: |
| [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `93.33% <90.90%> (-0.87%)` | :arrow_down: |
| [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.78% <91.66%> (-2.39%)` | :arrow_down: |
| [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.48% <100.00%> (-0.01%)` | :arrow_down: |
| [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `79.93% <100.00%> (ø)` | |
| [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
| [datasette/publish/cloudrun.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3B1Ymxpc2gvY2xvdWRydW4ucHk=) | `97.29% <100.00%> (ø)` | |
| [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.59% <100.00%> (-0.27%)` | :arrow_down: |

... and [2 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/1999/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/1999?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1480343044,https://api.github.com/repos/simonw/datasette/issues/1999,1480343044,IC_kwDOBm6k_c5YPD4E,9599,simonw,2023-03-22T22:33:15Z,2023-03-22T22:33:15Z,OWNER,"This still needs documentation, but now the tests are passing I'm going to merge it into `main`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1480339527,https://api.github.com/repos/simonw/datasette/issues/1999,1480339527,IC_kwDOBm6k_c5YPDBH,9599,simonw,2023-03-22T22:28:54Z,2023-03-22T22:28:54Z,OWNER,"I hacked at the CSV stuff until it worked.

I need to clean it up though, but I can do that in this separate task:
- #1101","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1480299765,https://api.github.com/repos/simonw/datasette/issues/1999,1480299765,IC_kwDOBm6k_c5YO5T1,9599,simonw,2023-03-22T21:44:03Z,2023-03-22T21:45:10Z,OWNER,"Oh this is a bit tricky.

I have a failing test because a plugin that uses the `extra_css_urls` hook can't see the columns for the page.

Turns out that bit comes from here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/app.py#L1203-L1217

Which assumes the context has `""columns""` - but that's only now available if `?_extra=columns` was passed.

Actually I think I can cheat here, since it's still getting the HTML context in order to render the template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1480179217,https://api.github.com/repos/simonw/datasette/issues/1999,1480179217,IC_kwDOBm6k_c5YOb4R,9599,simonw,2023-03-22T19:55:31Z,2023-03-22T21:34:02Z,OWNER,"I rebased from `main`. Now:

```
FAILED tests/test_csv.py::test_table_csv - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101
FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text'
FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...}
FAILED tests/test_plugins.py::test_hook_render_cell_demo - AttributeError: 'NoneType' object has no attribute 'string'
FAILED tests/test_plugins.py::test_hook_render_cell_async[/fixtures/simple_primary_key] - assert b'RENDER_CELL_ASYNC_RESULT' in b'<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d5...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\n});\n</script>\n\n\n\n\n</b...
FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...}
FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets'
FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...}
FAILED tests/test_permissions.py::test_permissions_checked[/fixtures/simple_primary_key-permissions3] - assert 500 in (200, 403)
FAILED tests/test_table_html.py::test_table_csv_json_export_interface - assert 500 == 200
FAILED tests/test_table_html.py::test_table_metadata - assert 500 == 200
FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/simple_primary_key-expected_classes3] - assert 500 == 200
FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert 500 == 200
FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - AssertionError: assert [] == [{'href': '/'...simple_view'}]
ERROR tests/test_internals_database.py::test_execute_write_fn_connection_exception - Failed: Timeout >1.0s
ERROR tests/test_internals_database.py::test_mtime_ns - Failed: Timeout >1.0s
ERROR tests/test_internals_database.py::test_mtime_ns_is_none_for_memory - Failed: Timeout >1.0s
ERROR tests/test_internals_database.py::test_is_mutable - Failed: Timeout >1.0s
ERROR tests/test_internals_database.py::test_database_memory_name - Failed: Timeout >1.0s
ERROR tests/test_internals_database.py::test_in_memory_databases_forbid_writes - Failed: Timeout >1.0s
=============== 21 failed, 1275 passed, 2 skipped, 1 xfailed, 6 errors in 59.18s ================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1480163485,https://api.github.com/repos/simonw/datasette/issues/1999,1480163485,IC_kwDOBm6k_c5YOYCd,9599,simonw,2023-03-22T19:48:00Z,2023-03-22T19:48:00Z,OWNER,"Getting close now! Only 13 failures left, mostly relating to CSV.
```
FAILED tests/test_csv.py::test_table_csv - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200
FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101
FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...}
FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets'
FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text'
FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...}
FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...}
=============== 13 failed, 1287 passed, 2 skipped, 1 xfailed in 61.57s (0:01:01) ================
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1480154453,https://api.github.com/repos/simonw/datasette/issues/1999,1480154453,IC_kwDOBm6k_c5YOV1V,9599,simonw,2023-03-22T19:41:39Z,2023-03-22T19:43:04Z,OWNER,"To replace this code: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L122

Maybe `datasette.render_template()` should optionally accept a list of templates.

https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none - turns out it does already:

> If this is a list of template file names then the first one that exists will be loaded and rendered.

It doesn't have an easy way to populate that `select_templates` debug template variable though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/issues/2041#issuecomment-1477155602,https://api.github.com/repos/simonw/datasette/issues/2041,1477155602,IC_kwDOBm6k_c5YC5sS,9599,simonw,2023-03-21T01:17:43Z,2023-03-21T01:17:43Z,OWNER,"Removed code in https://github.com/simonw/datasette/commit/538ca9d2e2d41308f5b149c934e2bc43727fd77c - it will merge when I land:
- #1999","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1633077183,Remove obsolete table POST code,
https://github.com/simonw/datasette/issues/2041#issuecomment-1477144853,https://api.github.com/repos/simonw/datasette/issues/2041,1477144853,IC_kwDOBm6k_c5YC3EV,9599,simonw,2023-03-21T01:02:13Z,2023-03-21T01:02:13Z,OWNER,"This code here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/table.py#L117-L169","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1633077183,Remove obsolete table POST code,
https://github.com/simonw/datasette/pull/1999#issuecomment-1477082852,https://api.github.com/repos/simonw/datasette/issues/1999,1477082852,IC_kwDOBm6k_c5YCn7k,9599,simonw,2023-03-20T23:27:25Z,2023-03-20T23:27:25Z,OWNER,"Urgh getting CSV to work is going to be _so hard_, because the logic for that currently lives in a huge chunk of code in `BaseView` which depends on the old design of the `data()` method:

https://github.com/simonw/datasette/blob/4bb49848697e40b8b9a1557be42b8e59eac965b3/datasette/views/base.py#L177-L343","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1476860334,https://api.github.com/repos/simonw/datasette/issues/1999,1476860334,IC_kwDOBm6k_c5YBxmu,9599,simonw,2023-03-20T20:06:56Z,2023-03-20T22:09:03Z,OWNER,"`pytest -n 8` locally says:

```
FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name_specify_id-True] - assert '<input type=""hidden"" name=""csrftoken"" value=""' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d59...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\n});\n</...
FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name-True] - assert '<input type=""hidden"" name=""csrftoken"" value=""' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d59...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\n});\n</...
FAILED tests/test_canned_queries.py::test_canned_query_permissions - AssertionError: assert 200 == 500
FAILED tests/test_canned_queries.py::test_canned_query_with_named_parameter - KeyError: 'rows'
FAILED tests/test_canned_queries.py::test_canned_write_custom_template - assert 500 == 200
FAILED tests/test_canned_queries.py::test_canned_write_query_disabled_for_immutable_database - assert 500 == 200
FAILED tests/test_canned_queries.py::test_custom_params - assert '<input type=""text"" id=""qp3"" name=""extra"" value=""foo"">' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>Error 500</title>\n    <link rel=""stylesheet"" href=""/-/static/app.css?d59...ils != detailsClickedWithin\n    ).forEach(details => details.open = false);\...
FAILED tests/test_canned_queries.py::test_insert - KeyError: 'ds_csrftoken'
FAILED tests/test_canned_queries.py::test_insert_error - KeyError: 'ds_csrftoken'
FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[False-True] - KeyError: 'ds_csrftoken'
FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[True-True] - KeyError: 'ds_csrftoken'
FAILED tests/test_canned_queries.py::test_vary_header - KeyError: 'vary'
FAILED tests/test_csv.py::test_csv_trace - AssertionError: assert 'application/...charset=utf-8' == 'text/html; charset=utf-8'
FAILED tests/test_csv.py::test_table_csv - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
FAILED tests/test_csv.py::test_table_csv_blob_columns - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
FAILED tests/test_csv.py::test_table_csv_download - AssertionError: assert 'application/...charset=utf-8' == 'text/csv; charset=utf-8'
FAILED tests/test_csv.py::test_table_csv_no_header - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101
FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_counts - AttributeError: 'NoneType' object has no attribute 'text'
FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_facets - AttributeError: 'NoneType' object has no attribute 'text'
FAILED tests/test_csv.py::test_table_csv_with_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8'
FAILED tests/test_facets.py::test_array_facet_handle_duplicate_tags - KeyError: 'facet_results'
FAILED tests/test_facets.py::test_conflicting_facet_names_json - AssertionError: assert {'results', 'timed_out'} == {'created', '...gs', 'tags_2'}
FAILED tests/test_facets.py::test_facet_size - KeyError: 'suggested_facets'
FAILED tests/test_facets.py::test_json_array_with_blanks_and_nulls - KeyError: 'suggested_facets'
FAILED tests/test_facets.py::test_other_types_of_facet_in_metadata - assert '<strong>created (date)\n' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: facetable: 15 rows</title>\n    <link rel=""stylesheet"" href=""/-...name"": ""table"", ""request_path"": ""/fixtures/facetable"", ""added"": 15, ""columns"": null};</script>\n\n\n\n</bo...
FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link'
FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link'
FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: simple_primary_key: 5 rows</title>\n    <link rel=""stylesheet"" ...able"", ""request_path"": ""/fixtures/simple_primary...
FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: table/with/slashes.csv: 1 row</title>\n    <link rel=""styleshee...quest_path"": ""/fixtures/table~2Fwith~2Fs...
FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...}
FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...}
FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets'
FAILED tests/test_plugins.py::test_hook_register_magic_parameters - KeyError: 'ds_csrftoken'
FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'next': None...', ...}, ...]} == {'1+1': 2, 'c... 0xXXX>', ...}
FAILED tests/test_plugins.py::test_hook_register_output_renderer_no_parameters - assert b'Hello' == b'{""ok"": true... ""n"": null}]}'
FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_broken_value - assert 200 == 500
FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_response - assert {'next': None...', ...}, ...]} == {'this_is': 'json'}
=== 39 failed, 1259 passed, 2 skipped, 1 xfailed in 58.07s ===
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1476898261,https://api.github.com/repos/simonw/datasette/issues/1999,1476898261,IC_kwDOBm6k_c5YB63V,9599,simonw,2023-03-20T20:37:52Z,2023-03-20T20:37:52Z,OWNER,"Manual testing spotted a bug.

`/content/repos.json?owner=9599&_facet_array=topics` - does not return a `facet_results` key.

`/content/repos.json?owner=9599&_facet_array=topics&_facet=owner` does.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1476854645,https://api.github.com/repos/simonw/datasette/issues/1999,1476854645,IC_kwDOBm6k_c5YBwN1,9599,simonw,2023-03-20T20:02:24Z,2023-03-20T20:02:24Z,OWNER,"```
    def test_routes(routes, path, expected_class, expected_matches):
        match, view = resolve_routes(routes, path)
        if expected_class is None:
            assert match is None
        else:
>           assert view.view_class.__name__ == expected_class
E           AttributeError: 'function' object has no attribute 'view_class'
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1476851525,https://api.github.com/repos/simonw/datasette/issues/1999,1476851525,IC_kwDOBm6k_c5YBvdF,9599,simonw,2023-03-20T19:59:51Z,2023-03-20T20:01:40Z,OWNER,"Three failures in `test_html.py`:
```
FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '<!-- Templates considered: table-fixtures-simple_primary_key.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: simple_primary_key: 5 rows</title>\n    <link rel=""stylesheet"" ...able"", ""request_path"": ""/fixtures/simple_primary_key"", ""ad...
FAILED tests/test_html.py::test_templates_considered[/fixtures/table~2Fwith~2Fslashes~2Ecsv-table-fixtures-tablewithslashescsv-fa7563.html, *table.html] - assert '<!-- Templates considered: table-fixtures-tablewithslashescsv-fa7563.html, *table.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n    <title>fixtures: table/with/slashes.csv: 1 row</title>\n    <link rel=""styleshee...quest_path"": ""/fixtures/table~2Fwith~2Fslashes~2Ec...
FAILED tests/test_html.py::test_alternate_url_json[/fixtures/facetable-http://localhost/fixtures/facetable.json] - KeyError: 'link'
FAILED tests/test_html.py::test_alternate_url_json[/fixtures/table~2Fwith~2Fslashes~2Ecsv-http://localhost/fixtures/table~2Fwith~2Fslashes~2Ecsv.json] - KeyError: 'link'
```

In CI I'm getting this: https://github.com/simonw/datasette/actions/runs/4472331872/jobs/7858360154

    ====== 44 failed, 1222 passed, 1 skipped, 1 xfailed in 171.24s (0:02:51) =======

Including failures in `tests/test_plugins.py` and `tests/test_routes.py`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1476781401,https://api.github.com/repos/simonw/datasette/issues/1999,1476781401,IC_kwDOBm6k_c5YBeVZ,9599,simonw,2023-03-20T19:01:35Z,2023-03-20T19:01:35Z,OWNER,"Down to just these failures:
```
FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'attrs': {'class': ['col-Link'], 'scope': 'col', 'data-column': 'Link', 'data-column-type': '', 'data-column-not-null': '0', 'data-is-pk': '0'}, 'a_href': None}, {'attrs': {'class': ['col-pk1'], 'scope': 'col', 'data-column': 'pk1', 'data-column-typ...
FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AssertionError: assert 'id\xa0▼' == 'id'
FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/simple_primary_key.json?id__gt=2', '/fixtures/simple_primary_key.testall?id__gt=2', '/fixtures/simple_primary_key.testnone?id__gt=2', '/fixtures/simple_primary_key.testresponse?id__gt=2', '/fixtures/simple_primary_key.csv?id__gt=2&_size=ma...
FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/facetable.json?_labels=on', '/fixtures/facetable.testall?_labels=on', '/fixtures/facetable.testnone?_labels=on', '/fixtures/facetable.testresponse?_labels=on', '/fixtures/facetable.csv?_labels=on&_size=max', '#export'] == ['/', '', '#export']
FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AssertionError: assert 'rowid\xa0▼' == 'rowid'
FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -', 'rowid', 'value'] == ['- column -']
FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -', 'id', 'content', 'content2'] == ['- column -']
FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -', 'pk1', 'pk2', 'content'] == ['- column -']
FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -', 'pk1', 'pk2', 'content', 'sortable_with_nulls', 'sortable_with_nulls_2', 'text', 'sortable']
FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id', 'name']
FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk', 'name']
=== 11 failed, 57 passed in 7.59s ===
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460759358,https://api.github.com/repos/simonw/datasette/issues/1999,1460759358,IC_kwDOBm6k_c5XEWs-,9599,simonw,2023-03-08T19:48:13Z,2023-03-20T18:47:12Z,OWNER,"Breaking this down into smaller steps:

- [x] Get `?_next=` working
- [x] Implement extensions - so `.json` is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML
- [ ] That HTML view should only access JSON data, which can be seen by using `.context` - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context.
- [ ] Get a bunch of the existing table HTML and JSON tests to pass
- [ ] Use those tests to refactor the nasty `_next` code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469
- [ ] Figure out how the [register_output_renderer(datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette) plugin hook should work","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1475074311,https://api.github.com/repos/simonw/datasette/issues/1999,1475074311,IC_kwDOBm6k_c5X69kH,9599,simonw,2023-03-19T02:16:08Z,2023-03-19T02:17:33Z,OWNER,"Next big challenge: fix all the tests in `test_table_html.py`:

```
FAILED tests/test_table_html.py::test_add_filter_redirects - assert 500 == 302
FAILED tests/test_table_html.py::test_existing_filter_redirects - assert 500 == 302
FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood-expected_hidden0] - AssertionError: assert {} == {'_facet': '_neighborhood'}
FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_where=1+=+1&_col=_city_id-expected_hidden1] - AssertionError: assert {} == {'_col': '_ci...ere': '1 = 1'}
FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_neighborhood__exact=Downtown-expected_hidden2] - AssertionError: assert {} == {'_facet': '_neighborhood'}
FAILED tests/test_table_html.py::test_reflected_hidden_form_fields[_facet=_neighborhood&_city_id__gt=1-expected_hidden3] - AssertionError: assert {} == {'_facet': '_neighborhood'}
FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed - assert 500 == 302
FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table - AssertionError: assert [('_fts_table...ts_pk', 'pk')] == []
FAILED tests/test_table_html.py::test_sort_by_desc_redirects - assert 500 == 302
FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...]
FAILED tests/test_table_html.py::test_facets_persist_through_filter_form - AssertionError: assert [] == [('_facet', '...ray', 'tags')]
FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field - AssertionError: assert [] == [('_size', '1')]
FAILED tests/test_table_html.py::test_table_html_simple_primary_key - AttributeError: 'NoneType' object has no attribute 'string'
FAILED tests/test_table_html.py::test_table_csv_json_export_interface - AssertionError: assert ['/fixtures/s...x', '#export'] == ['/', '', '#export']
FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys - AssertionError: assert ['/fixtures/f...x', '#export'] == ['/', '', '#export']
FAILED tests/test_table_html.py::test_table_html_no_primary_key - AssertionError: assert ['content', 'a', 'b', 'c'] == ['a', 'b', 'c']
FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key - AttributeError: 'NoneType' object has no attribute 'string'
FAILED tests/test_table_html.py::test_table_html_compound_primary_key - AssertionError: assert 'Link' == 'pk1'
FAILED tests/test_table_html.py::test_table_html_foreign_key_links - assert [['<td class=..."">\xa0</td>']] == [['<td class=..."">\xa0</td>']]
FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels - assert [['<td class=...str"">b</td>']] == [['<td class=...str"">b</td>']]
FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['<td class=...1</em></td>']] == [['<td class=...1</em></td>']]
FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -'...wid', 'value'] == ['- column -']
FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -'...', 'content2'] == ['- column -']
FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -'...2', 'content'] == ['- column -']
FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -'...nulls_2', ...]
FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references - assert [['<td class=...ne</a></td>']] == [['<td class=...ne</a></td>']]
FAILED tests/test_table_html.py::test_view_html - assert None is not None
FAILED tests/test_table_html.py::test_extra_where_clauses - assert [('_where', ""...'_city_id=1')] == []
FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10-expected_hidden0] - AssertionError: assert [] == [('_size', '10')]
FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10&_ignore=1&_ignore=2-expected_hidden1] - AssertionError: assert [] == [('_size', '1...ignore', '2')]
FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated[/fixtures/searchable?_sort=text2&_where=1-expected_hidden3] - AssertionError: assert [] == [('_where', '1')]
FAILED tests/test_table_html.py::test_binary_data_display_in_table - assert [['<td class=..."">\xa0</td>']] == [['<td class=..."">\xa0</td>']]
FAILED tests/test_table_html.py::test_metadata_sort - AttributeError: 'NoneType' object has no attribute 'string'
FAILED tests/test_table_html.py::test_metadata_sort_desc - AttributeError: 'NoneType' object has no attribute 'string'
FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood-2-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 2
FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood&_facet_size=50-5-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 5
FAILED tests/test_table_html.py::test_facet_total - assert 500 == 200
```

Deduped that's 30 tests:

```
FAILED tests/test_table_html.py::test_add_filter_redirects
FAILED tests/test_table_html.py::test_binary_data_display_in_table
FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references
FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys
FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed
FAILED tests/test_table_html.py::test_existing_filter_redirects
FAILED tests/test_table_html.py::test_extra_where_clauses
FAILED tests/test_table_html.py::test_facet_more_links
FAILED tests/test_table_html.py::test_facet_total
FAILED tests/test_table_html.py::test_facets_persist_through_filter_form
FAILED tests/test_table_html.py::test_metadata_sort
FAILED tests/test_table_html.py::test_metadata_sort_desc
FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field
FAILED tests/test_table_html.py::test_other_hidden_form_fields
FAILED tests/test_table_html.py::test_reflected_hidden_form_fields
FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key
FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated
FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table
FAILED tests/test_table_html.py::test_sort_by_desc_redirects
FAILED tests/test_table_html.py::test_sort_links
FAILED tests/test_table_html.py::test_table_csv_json_export_interface
FAILED tests/test_table_html.py::test_table_html_compound_primary_key
FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels
FAILED tests/test_table_html.py::test_table_html_filter_form_column_options
FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns
FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column
FAILED tests/test_table_html.py::test_table_html_foreign_key_links
FAILED tests/test_table_html.py::test_table_html_no_primary_key
FAILED tests/test_table_html.py::test_table_html_simple_primary_key
FAILED tests/test_table_html.py::test_view_html
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1475074025,https://api.github.com/repos/simonw/datasette/issues/1999,1475074025,IC_kwDOBm6k_c5X69fp,9599,simonw,2023-03-19T02:14:28Z,2023-03-19T02:14:51Z,OWNER,"I had to replicate quite a bit of this logic from `base.py`:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L526-L544

I should refactor this when I move the canned / arbitrary query views away from that base class too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1475016834,https://api.github.com/repos/simonw/datasette/issues/1999,1475016834,IC_kwDOBm6k_c5X6viC,9599,simonw,2023-03-18T22:30:31Z,2023-03-18T22:30:31Z,OWNER,"`test_paginate_using_link_header` will be tricky.

The reason the tests are failing is that `json_renderer()` attempts to populate the `link` header using `data[""next_url""]` - but that's not present unless `?_extra=next_url` has been passed:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/renderer.py#L101-L102

But I can only rely on `next` being present, not `next_url`.

I thought I could maybe assemble the `link` header using `next`, by turning that into a `next_url` link - but there's some custom logic which kicks in for pagination against SQL views (offset/limit pagination) to calculate the `next_url` which isn't easily replicable at the `json_renderer()` layer, in the `_next_value_and_url()` utility function:

https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/views/table.py#L2275-L2282

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1475003292,https://api.github.com/repos/simonw/datasette/issues/1999,1475003292,IC_kwDOBm6k_c5X6sOc,9599,simonw,2023-03-18T21:46:20Z,2023-03-18T21:46:20Z,OWNER,"Now 25 failures in `test_table_api.py`:
```
FAILED tests/test_table_api.py::test_expand_labels - assert {'2': {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', '_city_id': 1, '_neighborhood': 'Dogpatch', 'tags': '[""tag1"", ""tag3""]', 'complex_array'...
FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_co...
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1, 'value': inf}, {'rowid': 2, 'value': -inf}, {'rowid': 3, 'value': 1.5}] == [{'rowid': 1, 'value': None}, {'rowid': 2, 'value': None}, {'rowid': 3, 'value'...
FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_compoun...
FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'rowid': 1, 'data': ""b'\\x15\\x1c\\x02\\xc7\\xad\\x05\\xfe'""}, {'rowid': 2, 'data': ""b'\\x15\\x1c\\x03\\xc7\\xad\\x05\\xfe'""}, {'rowid': 3, 'data': None}] == [{'rowid': 1, 'data': {'...
FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}\n{""rowid"": 3, ""data"": null}] - assert '{""ok"": false, ""error"": ""Object of type bytes is not JSON serializable"", ""status"": 500, ""title"": null}' == '{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid""...
FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns'
============================================================================= 25 failed, 86 passed, 1 xfailed in 7.18s =============================================================================
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1474704790,https://api.github.com/repos/simonw/datasette/issues/1999,1474704790,IC_kwDOBm6k_c5X5jWW,9599,simonw,2023-03-18T04:52:59Z,2023-03-18T04:52:59Z,OWNER,"Here are the next set of tests to get passing:
```
% pytest tests/test_table_api.py
```
```
FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id-expected_facet_results0] - KeyError: 'name'
FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id&state=MI-expected_facet_results1] - KeyError: 'name'
FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int-expected_facet_results2] - KeyError: 'name'
FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int&planet_int=1-expected_facet_results3] - KeyError: 'name'
FAILED tests/test_table_api.py::test_suggested_facets - KeyError: 'suggested_facets'
FAILED tests/test_table_api.py::test_allow_facet_off - KeyError: 'suggested_facets'
FAILED tests/test_table_api.py::test_suggest_facets_off - KeyError: 'suggested_facets'
FAILED tests/test_table_api.py::test_nofacet[True] - KeyError: 'suggested_facets'
FAILED tests/test_table_api.py::test_nofacet[False] - KeyError: 'suggested_facets'
FAILED tests/test_table_api.py::test_nosuggest[True] - KeyError: 'suggested_facets'
FAILED tests/test_table_api.py::test_nosuggest[False] - KeyError: 'suggested_facets'
FAILED tests/test_table_api.py::test_nocount[True-None] - KeyError: 'count'
FAILED tests/test_table_api.py::test_nocount[False-15] - KeyError: 'count'
FAILED tests/test_table_api.py::test_expand_labels - AssertionError: assert {'13': {'_cit...:00:00', ...}} == {'13': {'_cit...:00:00', ...}}
FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'forei...l': '1', ...}} == {'1': {'forei...': '1'}, ...}}
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control'
FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1,...'value': 1.5}] == [{'rowid': 1,...'value': 1.5}]
FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'foreign_ke...': None, ...}] == [{'foreign_ke...': None, ...}]
FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'data': ""b'..., 'rowid': 3}] == [{'data': {'$..., 'rowid': 3}]
FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}\n{""rowid"": 3, ""data"": null}] - assert '{""ok"": false...title"": null}' == '{""rowid"": 1,...""data"": null}'
FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21
FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns'
FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns'
============================================================================= 38 failed, 73 passed, 1 xfailed in 7.25s =============================================================================
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/dogsheep/apple-notes-to-sqlite/issues/8#issuecomment-1468898285,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/8,1468898285,IC_kwDOJHON9s5XjZvt,41546558,RhetTbull,2023-03-14T22:00:21Z,2023-03-14T22:00:21Z,NONE,"Well that's embarrassing. I made a fork using macnotesapp and it's actually slower. This is because the Scripting Bridge sometimes fails to return the folder and thus macnotesapp resorts to AppleScript in this situation. The repeated AppleScript calls on a large library are slower than your ""slurp it all in"" approach.  I've got some ideas about how to improve this--will make another attempt if I can fix the issues.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617823309,Increase performance using macnotesapp,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465315726,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465315726,IC_kwDOCGYnMM5XVvGO,25778,eyeseast,2023-03-12T22:21:56Z,2023-03-12T22:21:56Z,CONTRIBUTOR,"Exactly, that's what I was running into. On my M2 MacBook, SpatiaLite ends up in what is -- for the moment -- a non-standard location, so even when I passed in the location with `--load-extension`, I still hit an error on `create-spatial-index`.

What I learned doing this originally is that SQLite needs to load the extension for each connection, even if all the SpatiaLite stuff is already in the database. So that's why `init_spatialite()` gets called again.

Here's the code where I hit the error: https://github.com/eyeseast/boston-parcels/blob/main/Makefile#L30 It works using this branch.

I'm not attached to this solution if you can think of something better. And I'm not sure, TBH, my test would actually catch what I'm after here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465303378,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465303378,IC_kwDOCGYnMM5XVsFS,9599,simonw,2023-03-12T21:24:04Z,2023-03-12T21:24:04Z,OWNER,"Upgraded to Sphinx 6 locally and got the same error:

```
% just docs
Cogging README.md
Cogging docs/changelog.rst
Cogging docs/cli-reference.rst
Cogging docs/cli.rst
Cogging docs/contributing.rst
Cogging docs/index.rst
Cogging docs/installation.rst
Cogging docs/python-api.rst
Cogging docs/reference.rst
sphinx-autobuild -a -b html ""."" ""_build""   --watch ../sqlite_utils
[sphinx-autobuild] > sphinx-build -b html -a /Users/simon/Dropbox/Development/sqlite-utils/docs /Users/simon/Dropbox/Development/sqlite-utils/docs/_build
Running Sphinx v6.1.3
loading pickled environment... failed
failed: Can't get attribute '_stable_repr_object' on <sphinx.deprecation._ModuleWrapper object at 0x103f155d0>
building [mo]: all of 0 po files
writing output... 
building [html]: all source files
updating environment: [new config] 8 added, 0 changed, 0 removed
reading sources... [ 12%] changelog                                                                                                          
Exception occurred:
  File ""/Users/simon/.local/share/virtualenvs/sqlite-utils-C4Ilevlm/lib/python3.11/site-packages/sphinx/ext/extlinks.py"", line 103, in role
    title = caption % part
            ~~~~~~~~^~~~~~
TypeError: not all arguments converted during string formatting
The full traceback has been saved in /var/folders/x6/31xf1vxj0nn9mxqq8z0mmcfw0000gn/T/sphinx-err-1ey36c1n.log, if you want to report the issue to the developers.
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting,
https://github.com/simonw/sqlite-utils/issues/533#issuecomment-1465302936,https://api.github.com/repos/simonw/sqlite-utils/issues/533,1465302936,IC_kwDOCGYnMM5XVr-Y,9599,simonw,2023-03-12T21:22:09Z,2023-03-12T21:22:09Z,OWNER,"Could be the same problem as:
- https://github.com/simonw/datasette/issues/1972

Which I fixed in https://github.com/simonw/datasette/commit/3af313e165215696af899e772f47bf7c27873ae3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620516340,ReadTheDocs error: not all arguments converted during string formatting,
https://github.com/simonw/sqlite-utils/pull/531#issuecomment-1465302343,https://api.github.com/repos/simonw/sqlite-utils/issues/531,1465302343,IC_kwDOCGYnMM5XVr1H,9599,simonw,2023-03-12T21:19:13Z,2023-03-12T21:19:13Z,OWNER,"Aah, I think I see why you wrote it like that.

The problem is that `init_spatialite()` does other stuff too:

https://github.com/simonw/sqlite-utils/blob/fc221f9b62ed8624b1d2098e564f525c84497969/sqlite_utils/db.py#L1161-L1171

So it needs to be able to load the SpatiaLite extension from the correct place, and THEN run `select InitSpatialMetadata()` to configure the database, if needed.

So the problem you're trying to solve here is to let people optionally pass in the path to SpatiaLite if it's not one of the ones that are searched by default.

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1620164673,Add paths for homebrew on Apple silicon,
https://github.com/simonw/datasette/issues/236#issuecomment-1465208436,https://api.github.com/repos/simonw/datasette/issues/236,1465208436,IC_kwDOBm6k_c5XVU50,545193,sopel,2023-03-12T14:04:15Z,2023-03-12T14:04:15Z,NONE,"I keep coming back to this in search for the related exploration, so I'll just link it now:

@simonw has meanwhile researched _how to deploy Datasette to AWS Lambda using function URLs and Mangum_ via https://github.com/simonw/public-notes/issues/6 and concluded _that's everything I need to know in order to build a datasette-publish-lambda plugin_.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",317001500,datasette publish lambda plugin,
https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464796494,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24,1464796494,IC_kwDOC8tyDs5XTwVO,956433,Mjboothaus,2023-03-11T02:23:42Z,2023-03-11T02:23:42Z,NONE,@simonw - maybe put in some error handling to trap for poorly formed XML (from Apple engineers) so that it suggests that there are problems with export.zip rather than odd looking Python errors :),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515883470,DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ,
https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464786643,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24,1464786643,IC_kwDOC8tyDs5XTt7T,956433,Mjboothaus,2023-03-11T02:01:27Z,2023-03-11T02:01:27Z,NONE,Thanks for reporting this and providing a solution -- I was puzzled by this error when I revisited my walking data and experienced this issues. I haven't tried the fix yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515883470,DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 ,
https://github.com/simonw/datasette/pull/1999#issuecomment-1463113856,https://api.github.com/repos/simonw/datasette/issues/1999,1463113856,IC_kwDOBm6k_c5XNViA,9599,simonw,2023-03-10T02:13:15Z,2023-03-10T02:13:15Z,OWNER,Idea for if this change ends up making a bunch of breaking changes to the templates (which I think it should) - I can generate a GitHub diff link between the old and new templates and include that link in the 1.0 upgrade documentation to help people who wrote custom templates see what they might need to change - with minimal effort from myself to document those changes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/issues/2038#issuecomment-1463112173,https://api.github.com/repos/simonw/datasette/issues/2038,1463112173,IC_kwDOBm6k_c5XNVHt,9599,simonw,2023-03-10T02:11:06Z,2023-03-10T02:11:06Z,OWNER,"Here's an example of something that would break:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/templates/database.html#L12-L15

Because `metadata` is an empty dictionary sometimes, so `{{ metadata.title or database }}` would raise an error and need to be replaced by `{{ metadata.get(""title"") or database }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618249044,Consider a `strict_templates` setting,
https://github.com/simonw/datasette/issues/2038#issuecomment-1463110978,https://api.github.com/repos/simonw/datasette/issues/2038,1463110978,IC_kwDOBm6k_c5XNU1C,9599,simonw,2023-03-10T02:09:41Z,2023-03-10T02:09:41Z,OWNER,"I'm torn on this. It's useful for me right now for refactoring, but I feel like it should be a permanent thing, not a setting - or it should default to on and people should turn it off, but who would ever do that?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618249044,Consider a `strict_templates` setting,
https://github.com/simonw/datasette/pull/1999#issuecomment-1463024951,https://api.github.com/repos/simonw/datasette/issues/1999,1463024951,IC_kwDOBm6k_c5XM_03,9599,simonw,2023-03-10T00:17:58Z,2023-03-10T00:17:58Z,OWNER,"Renderers have an impact on three different pages: query results, table page and row page.

The row page feature is incomplete though:

https://congress-legislators.datasettes.com/legislators/social_media/A000055

<img width=""668"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/224190630-b5e07524-94a7-4d8c-98b5-2364f3b0495d.png"">

Why is there no `.csv` link there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1463023674,https://api.github.com/repos/simonw/datasette/issues/1999,1463023674,IC_kwDOBm6k_c5XM_g6,9599,simonw,2023-03-10T00:16:03Z,2023-03-10T00:16:03Z,OWNER,"I also need to figure out the `renderers` stuff, so I can link to the right URLs for CSV and JSON and other formats:

https://github.com/simonw/datasette/blob/6d07a7da1531cd749844fc6827d9a1e57009b2ea/datasette/views/base.py#L474-L518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1463022397,https://api.github.com/repos/simonw/datasette/issues/1999,1463022397,IC_kwDOBm6k_c5XM_M9,9599,simonw,2023-03-10T00:14:21Z,2023-03-10T00:14:21Z,OWNER,"Ironically the thing I most need right now is comprehensive documentation of what variables are being passed to the templates!

One big challenge is that I need to untangle the template context that happens in `BaseView` - I'm hacking that together at the moment, but I need a real answer for how that should work in a world in which view functions aren't using that base class.

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L145","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1463021383,https://api.github.com/repos/simonw/datasette/issues/1999,1463021383,IC_kwDOBm6k_c5XM-9H,9599,simonw,2023-03-10T00:12:50Z,2023-03-10T00:12:50Z,OWNER,"Now at 34 failed, 34 passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1463005744,https://api.github.com/repos/simonw/datasette/issues/1999,1463005744,IC_kwDOBm6k_c5XM7Iw,9599,simonw,2023-03-09T23:52:15Z,2023-03-09T23:52:23Z,OWNER,"I need to figure out what to do about `extra_context_from_filters` - which was previously passed straight to the HTML context.

https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1462997800,https://api.github.com/repos/simonw/datasette/issues/1999,1462997800,IC_kwDOBm6k_c5XM5Mo,9599,simonw,2023-03-09T23:39:47Z,2023-03-09T23:39:47Z,OWNER,"Found a neat trick:
```diff
diff --git a/datasette/app.py b/datasette/app.py
index 186f192d..40416713 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -23,7 +23,13 @@ from pathlib import Path
 
 from markupsafe import Markup, escape
 from itsdangerous import URLSafeSerializer
-from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
+from jinja2 import (
+    ChoiceLoader,
+    Environment,
+    FileSystemLoader,
+    PrefixLoader,
+    StrictUndefined,
+)
 from jinja2.environment import Template
 from jinja2.exceptions import TemplateNotFound
 
@@ -394,7 +400,10 @@ class Datasette:
             ]
         )
         self.jinja_env = Environment(
-            loader=template_loader, autoescape=True, enable_async=True
+            loader=template_loader,
+            autoescape=True,
+            enable_async=True,
+            undefined=StrictUndefined,
         )
         self.jinja_env.filters[""escape_css_string""] = escape_css_string
         self.jinja_env.filters[""quote_plus""] = urllib.parse.quote_plus
```
This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context.

It's helping me spot things that are still missing, rather than just relying on failed unit tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462968053,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11,1462968053,IC_kwDOJHON9s5XMx71,9599,simonw,2023-03-09T23:24:01Z,2023-03-09T23:24:01Z,MEMBER,"I improved the readability by removing some unnecessary table aliases:
```sql
with recursive nested_folders(folder_id, descendant_folder_id) as (
  -- base case: select all immediate children of the root folder
  select id, id from folders where parent is null
  union all
  -- recursive case: select all children of the previous level of nested folders
  select nested_folders.folder_id, folders.id from nested_folders
  join folders on nested_folders.descendant_folder_id = folders.parent
)
-- Find notes within all descendants of folder 1
select *
from notes
where folder in (
  select descendant_folder_id from nested_folders where folder_id = 1
);
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618130434,Implement a SQL view to make it easier to query files in a nested folder,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462962682,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11,1462962682,IC_kwDOJHON9s5XMwn6,9599,simonw,2023-03-09T23:20:35Z,2023-03-09T23:22:41Z,MEMBER,"Here's a query that returns all notes in folder 1, including notes in descendant folders:
```sql
with recursive nested_folders(folder_id, descendant_folder_id) as (
  -- base case: select all immediate children of the root folder
  select id, id from folders where parent is null
  union all
  -- recursive case: select all children of the previous level of nested folders
  select nf.folder_id, f.id from nested_folders nf
  join folders f on nf.descendant_folder_id = f.parent
)
-- Find notes within all descendants of folder 1
select *
from notes
where folder in (
  select descendant_folder_id from nested_folders where folder_id = 1
);
```
With assistance from ChatGPT. Prompts were:

```
SQLite schema:

CREATE TABLE [folders] (
   [id] INTEGER PRIMARY KEY,
   [long_id] TEXT,
   [name] TEXT,
   [parent] INTEGER,
   FOREIGN KEY([parent]) REFERENCES [folders]([id])
);

Write a recursive CTE that returns the following:

folder_id | descendant_folder_id

With a row for every nested child of every folder - so the top level folder has lots of rows
```
Then I tweaked it a bit, then ran this:
```
WITH RECURSIVE nested_folders(folder_id, descendant_folder_id) AS (
  -- base case: select all immediate children of the root folder
  SELECT id, id FROM folders WHERE parent IS NULL
  UNION ALL
  -- recursive case: select all children of the previous level of nested folders
  SELECT nf.folder_id, f.id FROM nested_folders nf
  JOIN folders f ON nf.descendant_folder_id = f.parent
)
-- select all rows from the recursive CTE
SELECT * from notes where folder in (select descendant_folder_id FROM nested_folders where folder_id = 1)

Convert all SQL keywords to lower case, and re-indent
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618130434,Implement a SQL view to make it easier to query files in a nested folder,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/11#issuecomment-1462965256,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/11,1462965256,IC_kwDOJHON9s5XMxQI,9599,simonw,2023-03-09T23:22:12Z,2023-03-09T23:22:12Z,MEMBER,"Here's what the CTE from that looks like:

<img width=""469"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/224182888-50f315eb-417f-4fb1-b624-151e1740ffed.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1618130434,Implement a SQL view to make it easier to query files in a nested folder,
https://github.com/simonw/datasette/pull/2031#issuecomment-1462921890,https://api.github.com/repos/simonw/datasette/issues/2031,1462921890,IC_kwDOBm6k_c5XMmqi,9599,simonw,2023-03-09T22:35:30Z,2023-03-09T22:35:30Z,OWNER,"> I've implemented the test (thanks for pointing me in the right direction!).
> 
> At [tmcl-it/datasette:0.64.1+row-view-expand-labels](https://github.com/tmcl-it/datasette/tree/0.64.1%2Brow-view-expand-labels) I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.

Sure, let's merge that one too - it can go out in the next `0.64.x` series release (maybe even a 0.65).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,Expand foreign key references in row view as well,
https://github.com/simonw/datasette/pull/2034#issuecomment-1462921010,https://api.github.com/repos/simonw/datasette/issues/2034,1462921010,IC_kwDOBm6k_c5XMmcy,9599,simonw,2023-03-09T22:34:29Z,2023-03-09T22:34:29Z,OWNER,"Good catch, thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1613974869,remove an unused `app` var in cli.py,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462693867,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462693867,IC_kwDOJHON9s5XLu_r,9599,simonw,2023-03-09T20:01:39Z,2023-03-09T20:02:11Z,MEMBER,"My `folders` table will have:

- `id` - rowid
- `long_id` - that long unique string ID
- `name` - the name
- `parent` - foreign key to `id`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462691466,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462691466,IC_kwDOJHON9s5XLuaK,9599,simonw,2023-03-09T19:59:52Z,2023-03-09T19:59:52Z,MEMBER,"Improved script:
```zsh
osascript -e 'tell application ""Notes""
    set allFolders to folders
    repeat with aFolder in allFolders
        set folderId to id of aFolder
        set folderName to name of aFolder
        set folderContainer to container of aFolder
        if class of folderContainer is folder then
            set folderContainerId to id of folderContainer
        else
            set folderContainerId to """"
        end if
        log ""ID: "" & folderId
        log ""Name: "" & folderName
        log ""Container: "" & folderContainerId
        log "" ""
    end repeat
end tell
'
```
```
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
Name: Blog posts
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698
Name: JSK
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Name: Nested inside blog posts
Container: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526
Name: New Folder
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839
Name: New Folder 1
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2
Name: Notes
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059
Name: Quick Notes
Container: 
 
ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283
Name: UK Christmas 2022
Container: 
```
I filtered out things where the parent was an account and not a folder using `if class of folderContainer is folder then`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462682795,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462682795,IC_kwDOJHON9s5XLsSr,9599,simonw,2023-03-09T19:52:20Z,2023-03-09T19:52:44Z,MEMBER,"Created through several rounds with ChatGPT (including hints like ""rewrite that using setdefault()""):
```python
def topological_sort(nodes):
    children = {}
    for node in nodes:
        parent_id = node[""parent""]
        if parent_id is not None:
            children.setdefault(parent_id, []).append(node)

    def traverse(node, result):
        result.append(node)
        if node[""id""] in children:
            for child in children[node[""id""]]:
                traverse(child, result)

    sorted_data = []

    for node in nodes:
        if node[""parent""] is None:
            traverse(node, sorted_data)

    return sorted_data
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462570187,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462570187,IC_kwDOJHON9s5XLQzL,9599,simonw,2023-03-09T18:30:24Z,2023-03-09T18:30:24Z,MEMBER,"I used ChatGPT to write this:
```
osascript -e 'tell application ""Notes""
    set allFolders to folders
    repeat with aFolder in allFolders
        set folderId to id of aFolder
        set folderName to name of aFolder
        set folderContainer to container of aFolder
        set folderContainerName to name of folderContainer
        log ""Folder ID: "" & folderId
        log ""Folder Name: "" & folderName
        log ""Folder Container: "" & folderContainerName
        log "" ""
        --check for nested folders
        if count of folders of aFolder > 0 then
            set nestedFolders to folders of aFolder
            repeat with aNestedFolder in nestedFolders
                set nestedFolderId to id of aNestedFolder
                set nestedFolderName to name of aNestedFolder
                set nestedFolderContainer to container of aNestedFolder
                set nestedFolderContainerName to name of nestedFolderContainer
                log ""    Nested Folder ID: "" & nestedFolderId
                log ""    Nested Folder Name: "" & nestedFolderName
                log ""    Nested Folder Container: "" & nestedFolderContainerName
                log "" ""
            end repeat
        end if
    end repeat
end tell
'
```
Which for my account output this:
```
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6113
Folder Name: Blog posts
Folder Container: iCloud
 
    Nested Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
    Nested Folder Name: Nested inside blog posts
    Nested Folder Container: Blog posts
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p698
Folder Name: JSK
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7995
Folder Name: Nested inside blog posts
Folder Container: Blog posts
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3526
Folder Name: New Folder
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p3839
Folder Name: New Folder 1
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p2
Folder Name: Notes
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p6059
Folder Name: Quick Notes
Folder Container: iCloud
 
Folder ID: x-coredata://D2D50498-BBD1-4097-B122-D15ABD32BDEC/ICFolder/p7283
Folder Name: UK Christmas 2022
Folder Container: iCloud
```
So I think the correct approach here is to run code at the start to list all of the folders (no need to do fancy recursion though, just a flat list with the parent containers is enough) and create a model of that hierarchy in SQLite.

Then when I import notes I can foreign key reference them back to their containing folder.

I'm tempted to use `rowid` for the foreign keys because the official IDs are pretty long.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462564717,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462564717,IC_kwDOJHON9s5XLPdt,9599,simonw,2023-03-09T18:25:39Z,2023-03-09T18:25:39Z,MEMBER,So it looks like folders can be hierarchical?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/7#issuecomment-1462562735,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/7,1462562735,IC_kwDOJHON9s5XLO-v,9599,simonw,2023-03-09T18:23:56Z,2023-03-09T18:25:22Z,MEMBER,"From the Script Editor library docs:

A note has a:

> - `container` (folder), r/o) : the folder of the note

Here's what a folder looks like:

> folder n : a folder containing notes
> elements:
>
> - contains folders, notes; contained by application, accounts, folders.
>
> properties:
>
> - `name` (text) : the name of the folder
> - `id` (text, r/o) : the unique identifier of the folder
> - `shared` (boolean, r/o) : Is the folder shared?
> - `container` (account or folder, r/o) : the container of the folder
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1617769847,Folder support,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462556829,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4,1462556829,IC_kwDOJHON9s5XLNid,9599,simonw,2023-03-09T18:20:56Z,2023-03-09T18:20:56Z,MEMBER,"In terms of the UI: I'm tempted to say that the default behaviour is for it to run until it sees a note that it already knows about AND that has matching update/created dates, and then stop.

You can do a full import again ignoring that logic with `apple-notes-to-sqlite notes.db --full`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616429236,Support incremental updates,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/4#issuecomment-1462554175,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/4,1462554175,IC_kwDOJHON9s5XLM4_,9599,simonw,2023-03-09T18:19:34Z,2023-03-09T18:19:34Z,MEMBER,It looks like the iteration order is most-recently-modified-first - I tried editing a note a bit further back in my notes app and it was the first one output by `apple-notes-to-sqlite --dump`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616429236,Support incremental updates,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461285545,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461285545,IC_kwDOJHON9s5XGXKp,9599,simonw,2023-03-09T05:06:24Z,2023-03-09T05:06:24Z,MEMBER,"OK, this works!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461262577,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461262577,IC_kwDOJHON9s5XGRjx,9599,simonw,2023-03-09T04:30:00Z,2023-03-09T04:30:00Z,MEMBER,It doesn't have tests yet. I guess I'll need to mock `subprocess` to test this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461260978,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461260978,IC_kwDOJHON9s5XGRKy,9599,simonw,2023-03-09T04:27:18Z,2023-03-09T04:27:18Z,MEMBER,"Before that conversion:

    Monday, March 6, 2023 at 11:55:15 AM

After:

    2023-03-06T11:55:15","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461259490,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461259490,IC_kwDOJHON9s5XGQzi,9599,simonw,2023-03-09T04:24:27Z,2023-03-09T04:24:27Z,MEMBER,"Converting AppleScript date strings to ISO format is hard!

https://forum.latenightsw.com/t/formatting-dates/841 has a recipe I'll try:

    set todayISO to (todayDate as «class isot» as string)

Not clear to me how timezones work here. I'm going to ignore them for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234591,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461234591,IC_kwDOJHON9s5XGKuf,9599,simonw,2023-03-09T03:56:45Z,2023-03-09T03:56:45Z,MEMBER,"My prototype showed that images embedded in notes come out in the HTML export as bas64 image URLs, which is neat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461234311,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461234311,IC_kwDOJHON9s5XGKqH,9599,simonw,2023-03-09T03:56:24Z,2023-03-09T03:56:24Z,MEMBER,"I opened the ""Script Editor"" app on my computer, used Window -> Library to open the Library panel, then clicked on the Notes app there. I got this:

<img width=""1240"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223912876-09e8660e-f59a-4f04-8992-668e09c8d7d4.png"">

So the notes object has these properties:

- name (text) : the name of the note (normally the first line of the body)
- id (text, r/o) : the unique identifier of the note
- container ([folder](applewebdata://621FA8D9-C995-4081-B3B3-149B0EA04C7F#Notes-Suite.folder), r/o) : the folder of the note
- body (text) : the HTML content of the note
- plaintext (text, r/o) : the plaintext content of the note
- creation date (date, r/o) : the creation date of the note
- modification date (date, r/o) : the modification date of the note
- password protected (boolean, r/o) : Is the note password protected?
- shared (boolean, r/o) : Is the note shared?

I'm going to ignore the concept of attachments for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/2#issuecomment-1461232709,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/2,1461232709,IC_kwDOJHON9s5XGKRF,9599,simonw,2023-03-09T03:54:28Z,2023-03-09T03:54:28Z,MEMBER,"I think the AppleScript I want to pass to `osascript` looks like this:
```applescript
tell application ""Notes""
   repeat with eachNote in every note
      set noteId to the id of eachNote
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      log ""------------------------"" & ""\n""
      log noteId & ""\n""
      log noteTitle & ""\n\n""
      log noteBody & ""\n""
   end repeat
end tell
```
But there are a few more properties I'd like to get - created and updated date for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616354999,First working version,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230436,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1,1461230436,IC_kwDOJHON9s5XGJtk,9599,simonw,2023-03-09T03:51:52Z,2023-03-09T03:51:52Z,MEMBER,This did the job! Next step is to turn that into a Python script.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616347574,Initial proof of concept with ChatGPT,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461230197,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1,1461230197,IC_kwDOJHON9s5XGJp1,9599,simonw,2023-03-09T03:51:36Z,2023-03-09T03:51:36Z,MEMBER,"After a few more rounds I got to this script, which outputs them to a `/tmp/notes.txt` file:

```zsh
#!/bin/zsh

osascript -e '
set notesFile to ""/tmp/notes.txt""
set fileRef to open for access notesFile with write permission
tell application ""Notes""
   repeat with eachNote in every note
      set noteId to the id of eachNote
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      write ""------------------------"" & ""\n"" to fileRef
      write noteId & ""\n"" to fileRef
      write noteTitle & ""\n\n"" to fileRef
      write noteBody & ""\n"" to fileRef
   end repeat
end tell
close access fileRef'
```
Then I wrote this little Python script to load them into a database:
```python
import sqlite_utils
split = b""------------------------\n""
s = open(""/tmp/notes.txt"", ""rb"").read()
notes = [n.decode(""mac_roman"") for n in s.split(split) if n]

cleaned_notes = [{
    ""id"": n.split(""\n"")[0],
    ""title"": n.split(""\n"")[1],
    ""body"": ""\n"".join(n.split(""\n"")[2:]).strip()
} for n in notes]

db = sqlite_utils.Database(""/tmp/notes.db"")
db[""notes""].insert_all(cleaned_notes)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616347574,Initial proof of concept with ChatGPT,
https://github.com/dogsheep/apple-notes-to-sqlite/issues/1#issuecomment-1461226402,https://api.github.com/repos/dogsheep/apple-notes-to-sqlite/issues/1,1461226402,IC_kwDOJHON9s5XGIui,9599,simonw,2023-03-09T03:47:11Z,2023-03-09T03:47:11Z,MEMBER,"Me:
> Write applescript to loop through all of my Apple Notes and output their contents

ChatGPT:

```applescript
tell application ""Notes""
   repeat with eachNote in every note
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      display dialog noteTitle & return & return & noteBody
   end repeat
end tell
```
Me:
> How can I turn that into a zsh script which will output the results to my terminal

ChatGPT:
```zsh
#!/usr/bin/zsh

osascript -e 'tell application ""Notes""
   repeat with eachNote in every note
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      set output to noteTitle & ""\n"" & noteBody & ""\n""
      display dialog output
      log output
   end repeat
end tell'
```
I had to change that first line to `#!/bin/zsh` - then when I ran it it spammed my screen with dialog boxes.

Me:

> I want to log the notes contents to the terminal, not display it in a dialog

ChatGPT:

```zsh
#!/bin/zsh

osascript -e 'tell application ""Notes""
   repeat with eachNote in every note
      set noteTitle to the name of eachNote
      set noteBody to the body of eachNote
      set output to noteTitle & ""\n"" & noteBody & ""\n""
      log output
   end repeat
end tell'
```
This really helps!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1616347574,Initial proof of concept with ChatGPT,
https://github.com/simonw/datasette/pull/1999#issuecomment-1461161256,https://api.github.com/repos/simonw/datasette/issues/1999,1461161256,IC_kwDOBm6k_c5XF40o,9599,simonw,2023-03-09T02:10:07Z,2023-03-09T02:10:07Z,OWNER,"Just ran into a `no such table: columns` error - which I think is because my new view code sometimes bypasses calling this method (currently done in `BaseView.dispatch_request()`):

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461148579,https://api.github.com/repos/simonw/datasette/issues/1999,1461148579,IC_kwDOBm6k_c5XF1uj,9599,simonw,2023-03-09T01:54:10Z,2023-03-09T01:55:33Z,OWNER,Or... I could temporarily build a quick additional `CannedQueryView` subclass that just does the necessary bits to get the existing code to work. I'm going to try that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461148254,https://api.github.com/repos/simonw/datasette/issues/1999,1461148254,IC_kwDOBm6k_c5XF1pe,9599,simonw,2023-03-09T01:53:41Z,2023-03-09T01:53:41Z,OWNER,"Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503

The problem is that calling `.data()` on `QueryView` only works here because we expect to ourselves be inside a `.data()` method, with all of the existing magic that knows how to render things that are returned by that.

So I may need to substantially re-engineer how `QueryView` works in order to get this to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461075648,https://api.github.com/repos/simonw/datasette/issues/1999,1461075648,IC_kwDOBm6k_c5XFj7A,9599,simonw,2023-03-09T00:24:22Z,2023-03-09T00:24:22Z,OWNER,`127.0.0.1:8001/fixtures/neighborhood_search` fails because the forwarding to a canned query does not yet work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461074526,https://api.github.com/repos/simonw/datasette/issues/1999,1461074526,IC_kwDOBm6k_c5XFjpe,9599,simonw,2023-03-09T00:23:06Z,2023-03-09T00:23:06Z,OWNER,"    pytest tests/test_table_html.py

Currently 44 failed, 24 passed in 7.53s

Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461070937,https://api.github.com/repos/simonw/datasette/issues/1999,1461070937,IC_kwDOBm6k_c5XFixZ,9599,simonw,2023-03-09T00:18:52Z,2023-03-09T00:19:36Z,OWNER,"I managed to get HTML view working! I did it by continuing to add more things to the extras and the `_html` bundle until the page loaded for me:

```diff
     async def extra_extras():
         ""Available ?_extra= blocks""
         return {
@@ -1981,6 +2053,14 @@ async def extra_extras():
             ""query"",
             ""display_columns"",
             ""display_rows"",
+            ""database"",
+            ""table"",
+            ""database_color"",
+            ""table_actions"",
+            ""filters"",
+            ""renderers"",
+            ""custom_table_templates"",
+            ""sorted_facet_results"",
         ]
     }
 
@@ -2006,6 +2086,14 @@ async def extra_extras():
         extra_query,
         extra_metadata,
         extra_extras,
+        extra_database,
+        extra_table,
+        extra_database_color,
+        extra_table_actions,
+        extra_filters,
+        extra_renderers,
+        extra_custom_table_templates,
+        extra_sorted_facet_results,
     )
```
I'll probably refactor this into something cleaner, and maybe but a bunch of them in a `""html""` dictionary and update the templates to use `{{ html.filters }}` or similar. Will look at that once the tests are passing.

<img width=""1161"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223882676-942ff28b-1a4b-4ef0-adaa-ccd664877dcd.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461047607,https://api.github.com/repos/simonw/datasette/issues/1999,1461047607,IC_kwDOBm6k_c5XFdE3,9599,simonw,2023-03-08T23:51:46Z,2023-03-08T23:51:46Z,OWNER,"This feels quite nice:

<img width=""906"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223879235-b5ac71d9-213c-48ff-b17f-077db5b16e90.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461044477,https://api.github.com/repos/simonw/datasette/issues/1999,1461044477,IC_kwDOBm6k_c5XFcT9,9599,simonw,2023-03-08T23:47:26Z,2023-03-08T23:47:26Z,OWNER,"I want to package together all of the extras that are needed for the HTML format. A few options for doing that:

- Introduce `?_extra=_html` where the leading underscore indicates that this is a ""bundle"" of extras, then define a bundle that's everything needed for the HTML renderer
- Have some other mechanism whereby different renderers can request a bundle of extras.

I'm leaning towards the first option. I'll try that and see what it looks like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461023559,https://api.github.com/repos/simonw/datasette/issues/1999,1461023559,IC_kwDOBm6k_c5XFXNH,9599,simonw,2023-03-08T23:23:02Z,2023-03-08T23:23:02Z,OWNER,"To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1461002039,https://api.github.com/repos/simonw/datasette/issues/1999,1461002039,IC_kwDOBm6k_c5XFR83,9599,simonw,2023-03-08T22:58:16Z,2023-03-08T23:02:09Z,OWNER,"The reason for that `Row` thing is that it allows custom templates that do things like this:

https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows

```html+jinja
{% for row in display_rows %}
    <div>
        <h2>{{ row[""title""] }}</h2>
        <p>{{ row[""description""] }}<lp>
        <p>Category: {{ row.display(""category_id"") }}</p>
    </div>
{% endfor %}
```
Is that a good design? the `.display()` thing feels weird - I wonder if anyone has ever actually used that.

It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates

> If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use `{{ row.display(""column_name"") }}`.

I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display

It is however useful to have some kind of abstraction layer here that insulates the SQLite `Row` object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460988975,https://api.github.com/repos/simonw/datasette/issues/1999,1460988975,IC_kwDOBm6k_c5XFOwv,9599,simonw,2023-03-08T22:42:57Z,2023-03-08T22:42:57Z,OWNER,"Aside idea: it might be interesting if there were ""lazy"" template variables available in the context: things that are not actually executed unless a template author requests them.

Imagine if `metadata` was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460986533,https://api.github.com/repos/simonw/datasette/issues/1999,1460986533,IC_kwDOBm6k_c5XFOKl,9599,simonw,2023-03-08T22:40:28Z,2023-03-08T22:40:28Z,OWNER,"Figuring out what to do with `display_columns_and_rows()` is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates:

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460970807,https://api.github.com/repos/simonw/datasette/issues/1999,1460970807,IC_kwDOBm6k_c5XFKU3,9599,simonw,2023-03-08T22:31:49Z,2023-03-08T22:33:03Z,OWNER,"For the HTML version, I need to decide where all of the stuff that happens in `async def extra_template()` is going to live.

I think it's another one of those extra functions, triggered for `?_extra=context`.

https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460943097,https://api.github.com/repos/simonw/datasette/issues/1999,1460943097,IC_kwDOBm6k_c5XFDj5,9599,simonw,2023-03-08T22:09:24Z,2023-03-08T22:09:47Z,OWNER,"The ease with which I added that  `?_extra=query` feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction.

```diff
diff --git a/datasette/views/table.py b/datasette/views/table.py
index 8d3bb2c930..3e1db9c85f 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -1913,6 +1913,13 @@ async def extra_request():
             ""args"": request.args._data,
         }
 
+    async def extra_query():
+        ""Details of the underlying SQL query""
+        return {
+            ""sql"": sql,
+            ""params"": params,
+        }
+
     async def extra_extras():
         ""Available ?_extra= blocks""
         return {
@@ -1938,6 +1945,7 @@ async def extra_extras():
         extra_primary_keys,
         extra_debug,
         extra_request,
+        extra_query,
         extra_extras,
     )
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460916405,https://api.github.com/repos/simonw/datasette/issues/1999,1460916405,IC_kwDOBm6k_c5XE9C1,9599,simonw,2023-03-08T21:43:27Z,2023-03-08T21:43:27Z,OWNER,"Just noticed that `_json=colname` is not working, and that's because it's handled by the renderer here:

https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40

But that's not currently being called by my new code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460907148,https://api.github.com/repos/simonw/datasette/issues/1999,1460907148,IC_kwDOBm6k_c5XE6yM,9599,simonw,2023-03-08T21:34:30Z,2023-03-08T21:34:30Z,OWNER,"I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460906741,https://api.github.com/repos/simonw/datasette/issues/1999,1460906741,IC_kwDOBm6k_c5XE6r1,9599,simonw,2023-03-08T21:34:08Z,2023-03-08T21:34:08Z,OWNER,"So maybe I can refactor it to look a bit more like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604

One thing that's useful here is that `is_view` is handled early, like this:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472

So if I omit the `is_view` bit from the extracted function I can simplify more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469,https://api.github.com/repos/simonw/datasette/issues/1999,1460905469,IC_kwDOBm6k_c5XE6X9,9599,simonw,2023-03-08T21:32:56Z,2023-03-08T21:32:56Z,OWNER,"I'd really like to extract this ugly logic out into a helper function:

https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684

I copied it in and asked ChatGPT to ""List all of the variable that are either modified or assigned to by the above code"":

> Modified variables:
> 
>     offset
>     where_clauses
>     params
>     order_by
> 
> Assigned variables:
> 
>     sort_value
>     next_by_pk_clauses
>     order_by (when sort or sort_desc is not None)

Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this:

> My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions:
> 
>     _next
>     is_view
>     sort
>     sort_desc
>     use_rowid
>     pks
>     order_by

Note that `use_rowid` is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/issues/2036#issuecomment-1460866243,https://api.github.com/repos/simonw/datasette/issues/2036,1460866243,IC_kwDOBm6k_c5XEwzD,9599,simonw,2023-03-08T20:57:34Z,2023-03-08T20:57:34Z,OWNER,This fix is released in 0.64.2 https://docs.datasette.io/en/stable/changelog.html#v0-64-2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems",
https://github.com/simonw/datasette/issues/2036#issuecomment-1460848869,https://api.github.com/repos/simonw/datasette/issues/2036,1460848869,IC_kwDOBm6k_c5XEsjl,9599,simonw,2023-03-08T20:40:55Z,2023-03-08T20:40:55Z,OWNER,"Here's the https://latest.datasette.io/ deployment that just went out, further demonstrating that this change is working correctly:

<img width=""565"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223844709-bc01813f-b4da-4bb9-bdb8-7be839700a58.png"">

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems",
https://github.com/simonw/datasette/issues/2037#issuecomment-1460840620,https://api.github.com/repos/simonw/datasette/issues/2037,1460840620,IC_kwDOBm6k_c5XEqis,9599,simonw,2023-03-08T20:33:00Z,2023-03-08T20:33:00Z,OWNER,Got the same failure again for a recent commit: https://github.com/simonw/datasette/actions/runs/4368239376/jobs/7640567282,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError,
https://github.com/simonw/datasette/issues/2037#issuecomment-1460838797,https://api.github.com/repos/simonw/datasette/issues/2037,1460838797,IC_kwDOBm6k_c5XEqGN,9599,simonw,2023-03-08T20:31:15Z,2023-03-08T20:31:15Z,OWNER,"It's this test here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/tests/test_cli.py#L181-L189

Added in:
- #2033 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError,
https://github.com/simonw/datasette/issues/2037#issuecomment-1460838109,https://api.github.com/repos/simonw/datasette/issues/2037,1460838109,IC_kwDOBm6k_c5XEp7d,9599,simonw,2023-03-08T20:30:36Z,2023-03-08T20:30:36Z,OWNER,Instead of using `isolated_filesystem()` I could use a `tmpdir` fixture instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615891776,Test failure: FAILED tests/test_cli.py::test_install_requirements - FileNotFoundError,
https://github.com/simonw/datasette/issues/2036#issuecomment-1460827178,https://api.github.com/repos/simonw/datasette/issues/2036,1460827178,IC_kwDOBm6k_c5XEnQq,9599,simonw,2023-03-08T20:25:10Z,2023-03-08T20:25:10Z,OWNER,"https://console.cloud.google.com/run/detail/us-central1/new-service/revisions?project=datasette-222320 confirms that the image deployed is:

<img width=""556"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223841493-a104df28-5c9b-47d1-9c56-e7c7f6c61b00.png"">

Compared to https://console.cloud.google.com/run/detail/us-central1/datasette-io/revisions?project=datasette-222320 which shows that `datasette.io` is running:

<img width=""539"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223841616-67a1bb99-b704-4c2c-9dcd-247fb8592766.png"">
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems",
https://github.com/simonw/datasette/issues/2036#issuecomment-1460816528,https://api.github.com/repos/simonw/datasette/issues/2036,1460816528,IC_kwDOBm6k_c5XEkqQ,9599,simonw,2023-03-08T20:22:50Z,2023-03-08T20:23:20Z,OWNER,"Testing this manually:

```
% datasette publish cloudrun content.db --service new-service
Creating temporary tarball archive of 2 file(s) totalling 13.8 MiB before compression.
Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz]
Created [https://cloudbuild.googleapis.com/v1/projects/datasette-222320/locations/global/builds/290f41a4-e29a-443c-a1e5-c54513c6143d].
Logs are available at [ https://console.cloud.google.com/cloud-build/builds/290f41a4-e29a-443c-a1e5-c54513c6143d?project=99025868001 ].
---- REMOTE BUILD OUTPUT ----
starting build ""290f41a4-e29a-443c-a1e5-c54513c6143d""

FETCHSOURCE
Fetching storage object: gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483
Copying gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz#1678306862810483...
/ [1 files][  3.9 MiB/  3.9 MiB]                                                
Operation completed over 1 objects/3.9 MiB.
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
Sending build context to Docker daemon  14.52MB
Step 1/9 : FROM python:3.11.0-slim-bullseye
...
Installing collected packages: rfc3986, typing-extensions, sniffio, PyYAML, python-multipart, pluggy, pint, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, click, certifi, asgiref, aiofiles, uvicorn, Jinja2, janus, click-default-group-wheel, asgi-csrf, anyio, httpcore, httpx, datasette
Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 PyYAML-6.0 aiofiles-23.1.0 anyio-3.6.2 asgi-csrf-0.9 asgiref-3.6.0 certifi-2022.12.7 click-8.1.3 click-default-group-wheel-1.2.2 datasette-0.64.1 h11-0.14.0 httpcore-0.16.3 httpx-0.23.3 hupper-1.11 idna-3.4 itsdangerous-2.1.2 janus-1.0.0 mergedeep-1.3.4 pint-0.20.1 pluggy-1.0.0 python-multipart-0.0.6 rfc3986-1.5.0 sniffio-1.3.0 typing-extensions-4.5.0 uvicorn-0.20.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3 -> 23.0.1
[notice] To update, run: pip install --upgrade pip
Removing intermediate container 8ccebfebebc9
 ---> b972c85b38bb
...
Successfully built 606b7c286d7f
Successfully tagged gcr.io/datasette-222320/datasette-new-service:latest
PUSH
Pushing gcr.io/datasette-222320/datasette-new-service
The push refers to repository [gcr.io/datasette-222320/datasette-new-service]
667b1dc69e5e: Preparing
...
d8ddfcff216f: Pushed
latest: digest: sha256:452daffb2d3d7a8579c2ab39854be285155252c9428b4c1c50caac6a3a269e3f size: 2004
DONE
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ID                                    CREATE_TIME                DURATION  SOURCE                                                                                          IMAGES                                                   STATUS
290f41a4-e29a-443c-a1e5-c54513c6143d  2023-03-08T20:21:03+00:00  39S       gs://datasette-222320_cloudbuild/source/1678306859.271661-805303f364144b6094cc9c8532ab5133.tgz  gcr.io/datasette-222320/datasette-new-service (+1 more)  SUCCESS
Deploying container to Cloud Run service [new-service] in project [datasette-222320] region [us-central1]
✓ Deploying new service... Done.                                                                                                                                                      
  ✓ Creating Revision...                                                                                                                                                              
  ✓ Routing traffic...                                                                                                                                                                
  ✓ Setting IAM Policy...                                                                                                                                                             
Done.                                                                                                                                                                                 
Service [new-service] revision [new-service-00001-zon] has been deployed and is serving 100 percent of traffic.
Service URL: https://new-service-j7hipcg4aq-uc.a.run.app
```
https://new-service-j7hipcg4aq-uc.a.run.app/ was deployed successfully.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems",
https://github.com/simonw/datasette/issues/2036#issuecomment-1460810523,https://api.github.com/repos/simonw/datasette/issues/2036,1460810523,IC_kwDOBm6k_c5XEjMb,9599,simonw,2023-03-08T20:17:01Z,2023-03-08T20:17:01Z,OWNER,"I'm going to solve this by using the service name in that `image_id` instead:

```python
image_id = f""gcr.io/{project}/{service_name}""
```
This is a nasty bug, so I'm going to backport it to a `0.64.2` release as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems",
https://github.com/simonw/datasette/issues/2036#issuecomment-1460809643,https://api.github.com/repos/simonw/datasette/issues/2036,1460809643,IC_kwDOBm6k_c5XEi-r,9599,simonw,2023-03-08T20:16:10Z,2023-03-08T20:16:10Z,OWNER,"I think the code at fault is here:

https://github.com/simonw/datasette/blob/1ad92a1d87d79084ebe524ed186c900ff042328c/datasette/publish/cloudrun.py#L176-L182

That name ends up defaulting to `datasette` - so multiple different projects may end up deploying to the same `image_id`.

What I think happened in the `datasette.io` bug is that this workflow: https://github.com/simonw/simonwillisonblog-backup/blob/bfb573e96d8622ab52b22fdcd54724fe6e59fd24/.github/workflows/backup.yml and this workflow: https://github.com/simonw/datasette.io/blob/4676db5bf4a3fc9f792ee270ec0c59eb902cd2c3/.github/workflows/deploy.yml both happened to run at the exact same time.

And so the image that was pushed to `gcr.io/datasette-222320/datasette:latest` by the `simonw/simonwillisonblog-backup` action was then deployed by the `simonw/datasette.io/` action, which broke the site.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615862295,"`publish cloudrun` reuses image tags, which can lead to very surprising deploy problems",
https://github.com/simonw/datasette/issues/2035#issuecomment-1460808028,https://api.github.com/repos/simonw/datasette/issues/2035,1460808028,IC_kwDOBm6k_c5XEilc,1176293,ar-jan,2023-03-08T20:14:47Z,2023-03-08T20:14:47Z,NONE,"+1, I have been wishing for this feature (also for use with template-sql). It was requested before here #1304.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/pull/1999#issuecomment-1460760116,https://api.github.com/repos/simonw/datasette/issues/1999,1460760116,IC_kwDOBm6k_c5XEW40,9599,simonw,2023-03-08T19:48:52Z,2023-03-08T19:48:52Z,OWNER,"I'm trying to get `http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v` to return the correct results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/issues/2035#issuecomment-1460682625,https://api.github.com/repos/simonw/datasette/issues/2035,1460682625,IC_kwDOBm6k_c5XED-B,9599,simonw,2023-03-08T18:40:57Z,2023-03-08T18:40:57Z,OWNER,Pushed that prototype to a branch: https://github.com/simonw/datasette/commit/0fe844e9adb006a0138e83102ced1329d9155c59 / https://github.com/simonw/datasette/compare/sql-list-parameters,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460679434,https://api.github.com/repos/simonw/datasette/issues/2035,1460679434,IC_kwDOBm6k_c5XEDMK,9599,simonw,2023-03-08T18:39:35Z,2023-03-08T18:39:35Z,OWNER,"I should consider the existing design of magic parameters here: https://docs.datasette.io/en/stable/sql_queries.html#magic-parameters

- `_actor_*`
- `_header_*`
- `_cookie_`
- `_now_epoch`
- `_now_date_utc`
- `_now_datetime_utc`
- `_random_chars_*`

Should this new `id__list` syntax look more like those magic parameters, or is it OK to use `name__magic` syntax here instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460668431,https://api.github.com/repos/simonw/datasette/issues/2035,1460668431,IC_kwDOBm6k_c5XEAgP,9599,simonw,2023-03-08T18:35:34Z,2023-03-08T18:35:34Z,OWNER,"To implement this properly need to do the following:
- Get the page to display multiple `id: [ text input here ]` fields such that re-submission works
- Figure out how this should work for canned queries and for writable canned queries
- Tests that cover queries, canned queries, writable canned queries

And a bonus feature: what if the Datasette UI layer spotted `:id__list` parameters and used them to add a bit of JavaScript that allowed users to click a `+` button next to an `id` form field to add another one?

Also, when a page is re-displayed for on of these queries it could potentially add an extra form field allowing people to add another value.

Though this has an annoying problem: how to tell the difference between an additional `id` input field that the user chose not to populate, v.s. one that is supposed to represent an empty string?

Maybe only support multiple `id` fields for users with JavaScript in order to avoid this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460664619,https://api.github.com/repos/simonw/datasette/issues/2035,1460664619,IC_kwDOBm6k_c5XD_kr,9599,simonw,2023-03-08T18:32:29Z,2023-03-08T18:32:29Z,OWNER,"Got a prototype working:
```diff
diff --git a/datasette/views/database.py b/datasette/views/database.py
index 8d289105..6f9d8a44 100644
--- a/datasette/views/database.py
+++ b/datasette/views/database.py
@@ -226,6 +226,12 @@ class QueryView(DataView):
     ):
         db = await self.ds.resolve_database(request)
         database = db.name
+        # Disallow x__list query string parameters
+        invalid_params = [k for k in request.args if k.endswith(""__list"")]
+        if invalid_params:
+            raise DatasetteError(
+                ""Invalid query string parameters: {}"".format("", "".join(invalid_params))
+            )
         params = {key: request.args.get(key) for key in request.args}
         if ""sql"" in params:
             params.pop(""sql"")
@@ -258,6 +264,11 @@ class QueryView(DataView):
             for named_parameter in named_parameters
             if not named_parameter.startswith(""_"")
         }
+        # Handle any __list parameters
+        for named_parameter in named_parameters:
+            if named_parameter.endswith(""__list""):
+                list_values = request.args.getlist(named_parameter[:-6])
+                params[named_parameter] = json.dumps(list_values)
 
         # Set to blank string if missing from params
         for named_parameter in named_parameters:
```
This isn't yet doing the right thing on form re-submission: it breaks because it attempts to pass through the `?id__list=` invalid parameter. But I did manage to get it to do this through careful editing of the URL:

<img width=""1226"" alt=""image"" src=""https://user-images.githubusercontent.com/9599/223803758-2a1f90bd-bc20-4a77-bdce-4644afe235ba.png"">

That was this URL: `http://127.0.0.1:8034/content?sql=select+%3Aid__list%2C*+from+releases+where+id+in+(select+value+from+json_each(%3Aid__list))&id=62642726&id=18402901&id=38714866`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460659382,https://api.github.com/repos/simonw/datasette/issues/2035,1460659382,IC_kwDOBm6k_c5XD-S2,9599,simonw,2023-03-08T18:28:00Z,2023-03-08T18:28:00Z,OWNER,"Also: `datasette-explain` may need to be updated to understand how to handle this:

`ERROR: conn=<sqlite3.Connection object at 0x102834940>, sql = 'explain select * from releases where id in (select id from json_each(:id__list))', params = None: You did not supply a value for binding parameter :id__list.`
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460654136,https://api.github.com/repos/simonw/datasette/issues/2035,1460654136,IC_kwDOBm6k_c5XD9A4,9599,simonw,2023-03-08T18:25:46Z,2023-03-08T18:25:46Z,OWNER,"Trickiest part of the implementation here is that it needs to know to output three `id` HTML form fields on the page, such that their values are persisted when the form is submitted a second time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460639749,https://api.github.com/repos/simonw/datasette/issues/2035,1460639749,IC_kwDOBm6k_c5XD5gF,9599,simonw,2023-03-08T18:17:31Z,2023-03-08T18:17:31Z,OWNER,"Since we are pre-1.0 it's still OK to implement a feature that disallows `?id__list=` in the URL, but allows `:id__list` in SQL queries to reference the JSON list of parameters.

So I'm going to prototype this as the `:id__list` feature and see how it feels.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460637906,https://api.github.com/repos/simonw/datasette/issues/2035,1460637906,IC_kwDOBm6k_c5XD5DS,9599,simonw,2023-03-08T18:16:31Z,2023-03-08T18:16:31Z,OWNER,"I'm pretty sold on this as a feature now. The main question I have is which of these options to implement:

1. `?id=1&?id=2` results in `:id` in the query being `[""1"", ""2""]` - no additional syntax required
2. `:id` in the query continues to reference just the first of those parameters - but `:id__list` (or some other custom syntax) instead gets `[""1"", ""2""]` - or, if the URL is `?id=1` - gets `[""1""]`

Actually on writing these out I realize that option 2 is the ONLY valid option. It's no good building a query that works against a JSON list if the user might pass just a single ID, `?id=1`, resulting in their query breaking.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460632758,https://api.github.com/repos/simonw/datasette/issues/2035,1460632758,IC_kwDOBm6k_c5XD3y2,9599,simonw,2023-03-08T18:13:49Z,2023-03-08T18:13:49Z,OWNER,"https://github.com/rclement/datasette-dashboards/issues/54 makes the excellent point that the `<select multiple>` default HTML widget produces this exact format of query string:

```html
<form action=""https://www.example.com/"">
<select multiple name=""id"">
  <option>21</option>
  <option>32</option>
  <option>15</option>
  <option>63</option>
</select>
<input type=""submit"">
</form>
```
Submitting that form with the middle two options selected navigates to: `https://www.example.com/?id=32&id=15`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460628199,https://api.github.com/repos/simonw/datasette/issues/2035,1460628199,IC_kwDOBm6k_c5XD2rn,9599,simonw,2023-03-08T18:11:31Z,2023-03-08T18:11:31Z,OWNER,"One variant on this idea: maybe you have to specify in your query that you want it to be the JSON list version, not the single item (first `?id=` parameter version)? Maybe with syntax like this:

    where id in (select value from json_each(:id__list))

Datasette would automatically pass `{""id"": ""11"", ""id__list"": '[""11"", ""32"", ""62""]'}` as arguments to the `db.execute()` method, if the page was called with `?id=11&id=32&id=62`.

This is more explicit, though the syntax is a bit uglier (maybe there's a nicer design for this?). I also worry about `?id__list=` conflicting with this, but I think that's a risk I can take - tell people not to do that, or even block `?id__list=` style parameters entirely.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460621871,https://api.github.com/repos/simonw/datasette/issues/2035,1460621871,IC_kwDOBm6k_c5XD1Iv,9599,simonw,2023-03-08T18:08:25Z,2023-03-08T18:09:04Z,OWNER,"My current preferred solution is to lean into SQLite's JSON support.

What if the query page spotted `?id=11&id=32&id=62` and turned that into  a JSON string called `:id:` with a value of `[""11"", ""32"", ""62""]`?

Note that this is still a string, not a list. This avoids a nasty problem that occurred in PHP world, where `?id[]=1&id[]=2` would result in an actual PHP array object, which often broke underlying code that had expected `$_GET[""id""]` to be a string, not an array.

So in a query you'd be able to do this:

    where id in (select value from json_each(:id))

And then call it with `?id=11&id=32&id=62`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2035#issuecomment-1460618433,https://api.github.com/repos/simonw/datasette/issues/2035,1460618433,IC_kwDOBm6k_c5XD0TB,9599,simonw,2023-03-08T18:06:34Z,2023-03-08T18:06:34Z,OWNER,"One way to do this would be to dynamically generate the `where id in (?, ?, ?)` with the correct number of question marks, then feed in a list from `request.args.getlist(""id"")` - but that would require rewriting the SQL query text to add those question marks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1615692818,Potential feature: special support for `?a=1&a=2` on the query page,
https://github.com/simonw/datasette/issues/2027#issuecomment-1459455356,https://api.github.com/repos/simonw/datasette/issues/2027,1459455356,IC_kwDOBm6k_c5W_YV8,1350673,dmick,2023-03-08T04:42:22Z,2023-03-08T04:42:22Z,NONE,"I managed to make it work by using nginx's 'exact match' (=) combined with 'prefix match'; that is, match explicitly on `/`, and redirect to `/<db>/<table>`, and then have the normal ProxyPath for the unadorned (prefix-matching) `/`.

```
        location = / {
                return 302 /<db>/<table>;
        }
        location / {
                proxy_pass http://127.0.0.1:8001/;
                proxy_set_header Host $host;
        }
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590183272,"How to redirect from ""/"" to a specific db/table",
https://github.com/simonw/datasette/pull/2031#issuecomment-1457243738,https://api.github.com/repos/simonw/datasette/issues/2031,1457243738,IC_kwDOBm6k_c5W28Za,82332573,tmcl-it,2023-03-07T00:05:25Z,2023-03-07T00:12:09Z,NONE,"I've implemented the test (thanks for pointing me in the right direction!).

At [tmcl-it/datasette:0.64.1+row-view-expand-labels](https://github.com/tmcl-it/datasette/tree/0.64.1%2Brow-view-expand-labels) I also have a variant of this patch that applies to the 0.64.x branch. Please let me know if you'd be interested in merging that as well and I'll open another PR.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,Expand foreign key references in row view as well,
https://github.com/simonw/datasette/issues/2033#issuecomment-1457172180,https://api.github.com/repos/simonw/datasette/issues/2033,1457172180,IC_kwDOBm6k_c5W2q7U,25778,eyeseast,2023-03-06T22:54:52Z,2023-03-06T22:54:52Z,CONTRIBUTOR,This would be a nice feature to have with `datasette publish` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1612296210,`datasette install -r requirements.txt`,
https://github.com/simonw/datasette/issues/2033#issuecomment-1457117383,https://api.github.com/repos/simonw/datasette/issues/2033,1457117383,IC_kwDOBm6k_c5W2djH,9599,simonw,2023-03-06T22:28:55Z,2023-03-06T22:28:55Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugins.html#installing-plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1612296210,`datasette install -r requirements.txt`,
https://github.com/simonw/datasette/pull/2031#issuecomment-1456997425,https://api.github.com/repos/simonw/datasette/issues/2031,1456997425,IC_kwDOBm6k_c5W2AQx,9599,simonw,2023-03-06T21:04:27Z,2023-03-06T21:06:34Z,OWNER,"This is a very neat fix, for something I've been wanting for a while.

Add a unit test for the row HTML page - I suggest against this page: https://latest.datasette.io/fixtures/foreign_key_references/1 - and I'll land this PR.

You can model it on this test here: https://github.com/simonw/datasette/blob/a53b893c46453f35decc8c145c138671cee6140c/tests/test_table_html.py#L609-L632

I think adding it to `test_table_html.py` is OK, even though it's technically for the row page and not the table page.

Thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1605481359,Expand foreign key references in row view as well,
https://github.com/simonw/datasette/pull/2028#issuecomment-1456925875,https://api.github.com/repos/simonw/datasette/issues/2028,1456925875,IC_kwDOBm6k_c5W1uyz,22429695,codecov[bot],2023-03-06T20:26:53Z,2023-03-06T20:26:53Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/2028?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`0b4a286`)](https://codecov.io/gh/simonw/datasette/commit/0b4a28691468b5c758df74fa1d72a823813c96bf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11% compared to head [(`a8dde13`)](https://codecov.io/gh/simonw/datasette/pull/2028?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11%.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #2028   +/-   ##
=======================================
  Coverage   92.11%   92.11%           
=======================================
  Files          38       38           
  Lines        5555     5555           
=======================================
  Hits         5117     5117           
  Misses        438      438           
```



Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/2028?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590839187,add Python 3.11 classifier,
https://github.com/simonw/datasette/pull/2028#issuecomment-1456914694,https://api.github.com/repos/simonw/datasette/issues/2028,1456914694,IC_kwDOBm6k_c5W1sEG,9599,simonw,2023-03-06T20:19:37Z,2023-03-06T20:19:37Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590839187,add Python 3.11 classifier,
https://github.com/simonw/datasette/issues/1619#issuecomment-1455196849,https://api.github.com/repos/simonw/datasette/issues/1619,1455196849,IC_kwDOBm6k_c5WvIqx,969875,BryantD,2023-03-05T20:29:55Z,2023-03-05T20:30:14Z,NONE,"I have this same issue, which is happening with both json links and facets. It is not happening with column sort links in the gear popup menus, but it is happening with the sort arrow that results after you use one of those links. I'm using Apache as a proxy to Datasette; the relevant configs are:

```
  ProxyPass /datasette/ http://127.0.0.1:8000/datasette/ nocanon
  ProxyPreserveHost on
```

```
{
	""base_url"": ""/datasette/""
}
```

If it would be useful to get a look at the running installation via the Web, Simon, let me know.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1121583414,JSON link on row page is 404 if base_url setting is used,
https://github.com/simonw/sqlite-utils/issues/433#issuecomment-1444474487,https://api.github.com/repos/simonw/sqlite-utils/issues/433,1444474487,IC_kwDOCGYnMM5WGO53,167893,mcarpenter,2023-02-24T20:57:43Z,2023-02-24T22:22:18Z,CONTRIBUTOR,"I think I see what is happening here, although I haven't quite work out a fix yet. Usually:

* `click.progressbar.render_progress()` renders the cursor invisible on each invocation (update of the bar)
* When the progress bar goes out of scope, the `__exit()__` method is invoked, which calls `render_finish()` to make the cursor re-appear.

(See terminal escape sequences `BEFORE_BAR` and `AFTER_BAR` in click).

However the sqlite-utils `utils.file_progress` context manager wraps `click.progressbar` and yields an instance of a helper class:

``` python
@contextlib.contextmanager     
def file_progress(file, silent=False, **kwargs):
   ...
        with click.progressbar(length=file_length, **kwargs) as bar:
            yield UpdateWrapper(file, bar.update) 
```

The yielded `UpdateWrapper` goes out of scope quickly and `click.progressbar.__exit__()` is called. The cursor is made un-invisible. Hoewever `bar` is still live and so when the caller iterates on the yielded wrapper this invokes the bar's update method, calling `render_progress()`, each time printing the ""make cursor invisible"" escape code. The `progressbar.__exit__` function is not called again, so the cursor doesn't re-appear.

","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1239034903,CLI eats my cursor,
https://github.com/simonw/datasette/issues/2030#issuecomment-1440854834,https://api.github.com/repos/simonw/datasette/issues/2030,1440854834,IC_kwDOBm6k_c5V4bMy,19700859,gk7279,2023-02-22T21:54:39Z,2023-02-22T21:54:39Z,NONE,Thanks @dmick  . I chose to create a firewall rule under my GCP to open the port of interest and datasette works. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1594383280,How to use Datasette with apache webserver on GCP?,
https://github.com/simonw/datasette/issues/2030#issuecomment-1440814680,https://api.github.com/repos/simonw/datasette/issues/2030,1440814680,IC_kwDOBm6k_c5V4RZY,1350673,dmick,2023-02-22T21:22:42Z,2023-02-22T21:22:42Z,NONE,"@gk7279, you had asked in a separate bug about how to redirect web servers in general.  The datasette docs actually have pretty good information on this for both nginx and apache2: https://docs.datasette.io/en/stable/deploying.html#running-datasette-behind-a-proxy
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1594383280,How to use Datasette with apache webserver on GCP?,
https://github.com/simonw/datasette/issues/2027#issuecomment-1440811364,https://api.github.com/repos/simonw/datasette/issues/2027,1440811364,IC_kwDOBm6k_c5V4Qlk,19700859,gk7279,2023-02-22T21:19:47Z,2023-02-22T21:19:47Z,NONE,"yes @dmick . How did you make your public IP redirect to your uvicorn server?

Instead of nginx, I have apache2 on my GCP VM. Any pointers here are helpful too.

Thanks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590183272,"How to redirect from ""/"" to a specific db/table",
https://github.com/simonw/datasette/issues/2027#issuecomment-1440762383,https://api.github.com/repos/simonw/datasette/issues/2027,1440762383,IC_kwDOBm6k_c5V4EoP,1350673,dmick,2023-02-22T20:35:16Z,2023-02-22T20:35:16Z,NONE,"Was that query to me, @gk7279?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590183272,"How to redirect from ""/"" to a specific db/table",
https://github.com/simonw/datasette/issues/2027#issuecomment-1440355080,https://api.github.com/repos/simonw/datasette/issues/2027,1440355080,IC_kwDOBm6k_c5V2hMI,19700859,gk7279,2023-02-22T16:26:41Z,2023-02-22T16:26:41Z,NONE,Can you please help or share your expertise with #2030 ?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1590183272,"How to redirect from ""/"" to a specific db/table",
https://github.com/simonw/datasette/issues/1258#issuecomment-1437671409,https://api.github.com/repos/simonw/datasette/issues/1258,1437671409,IC_kwDOBm6k_c5VsR_x,2670795,brandonrobertz,2023-02-20T23:39:58Z,2023-02-20T23:39:58Z,CONTRIBUTOR,"This is pretty annoying for FTS because sqlite throws an error instead of just doing something like returning all or no results. This makes users who are unfamiliar with SQL and Datasette think the canned query page is broken and is a frequent source of confusion.

To anyone dealing with this: My solution is to modify the canned query so that it returns no results which cues people to fill in the blank parameters.

So instead of `emails_fts match escape_fts(:search))`

My canned queries now look like this:

`emails_fts match escape_fts(iif(:search=="""", ""*"", :search))`

There are no asterisks in my data so the result is always blank.

Ultimately it would be nice to be able to handle this in the metadata. Either making some named parameters required or setting some default values.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",828858421,Allow canned query params to specify default values,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1435318713,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1435318713,IC_kwDOCGYnMM5VjTm5,167893,mcarpenter,2023-02-17T21:55:01Z,2023-02-17T21:55:01Z,CONTRIBUTOR,"Meanwhile, a cheap workaround is to invalidate the registered function cache:
``` python
table.convert(...)
db._registered_functions = set()
table.convert(...)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail,
https://github.com/simonw/datasette/issues/1775#issuecomment-1426158181,https://api.github.com/repos/simonw/datasette/issues/1775,1426158181,IC_kwDOBm6k_c5VAXJl,805751,metamoof,2023-02-10T18:04:40Z,2023-02-10T18:04:40Z,NONE,"Is this where we talk about i18n of results? Or is that a separate thread.

e.g. Having `country_long` show `España` in the Spanish version of the [global power plants demo site](https://global-power-plants.datasettes.com/) instead of `Spain`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1323346408,i18n support,
https://github.com/simonw/datasette/issues/2024#issuecomment-1426031395,https://api.github.com/repos/simonw/datasette/issues/2024,1426031395,IC_kwDOBm6k_c5U_4Mj,9599,simonw,2023-02-10T16:11:53Z,2023-02-10T16:11:53Z,OWNER,Relevant: https://til.simonwillison.net/sqlite/enabling-wal-mode,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1579973223,Mention WAL mode in documentation,
https://github.com/simonw/datasette/issues/2023#issuecomment-1425988018,https://api.github.com/repos/simonw/datasette/issues/2023,1425988018,IC_kwDOBm6k_c5U_tmy,80409402,mlaparie,2023-02-10T15:39:59Z,2023-02-10T15:39:59Z,NONE,"Thanks for confirming my doubts! I removed it after opening this issue, yup, then had another issue with `default_cache_ttl_hashed` which I assume was removed at the same time. Sorry for not trying that before opening the issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1579695809,Error: Invalid setting 'hash_urls' in settings.json in 0.64.1,
https://github.com/simonw/datasette/issues/2023#issuecomment-1425974877,https://api.github.com/repos/simonw/datasette/issues/2023,1425974877,IC_kwDOBm6k_c5U_qZd,193185,cldellow,2023-02-10T15:32:41Z,2023-02-10T15:32:41Z,CONTRIBUTOR,"I think this feature was removed in Datasette 0.61 and moved to a plugin. People who want hashed URLs can use the [datasette-hashed-urls](https://docs.datasette.io/en/stable/performance.html#performance-hashed-urls) plugin to achieve the same affect.

It looks like you're trying to disable hashed urls, so I think you can just remove that config setting and things will work.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1579695809,Error: Invalid setting 'hash_urls' in settings.json in 0.64.1,
https://github.com/simonw/datasette/issues/2022#issuecomment-1424848569,https://api.github.com/repos/simonw/datasette/issues/2022,1424848569,IC_kwDOBm6k_c5U7Xa5,1667631,DavidPratten,2023-02-09T21:13:50Z,2023-02-09T21:13:50Z,NONE,"Nulls in primary keys, does it every time.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1578609658,Error 500 - not clear the cause,
https://github.com/simonw/sqlite-utils/issues/525#issuecomment-1423387341,https://api.github.com/repos/simonw/sqlite-utils/issues/525,1423387341,IC_kwDOCGYnMM5U1yrN,167893,mcarpenter,2023-02-08T23:48:52Z,2023-02-09T00:17:30Z,CONTRIBUTOR,PR below,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1575131737,Repeated calls to `Table.convert()` fail,
https://github.com/simonw/datasette/issues/262#issuecomment-1423067724,https://api.github.com/repos/simonw/datasette/issues/262,1423067724,IC_kwDOBm6k_c5U0kpM,9599,simonw,2023-02-08T18:33:32Z,2023-02-08T18:36:48Z,OWNER,"Just realized that it's useful to be able to tell what parameters were used to generate a page... but reflecting things like `_next` back in the JSON is confusing in the presence of `next`.

So I'm going to add an extra for that information too.

Not sure what to call it though:

- `params` - confusing because in the code that's usually used for params passed to SQL queries
- `query_string` - wouldn't that be a string, not params as a dictionary?

I'm going to experiment with a `request` extra that returns some bits of information about the request.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1422681850,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1422681850,IC_kwDOCGYnMM5UzGb6,21095447,4l1fe,2023-02-08T14:25:50Z,2023-02-08T14:29:09Z,NONE,"I live the patch here for others:

_original code_
```shell
$ which sqlite-utils | xargs cat
```
```python
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from sqlite_utils.cli import cli


if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())
```

_patched/sqlite-utils.py_
```python
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
from sqlite_utils.cli import cli

# New imports
from unittest.mock import patch
from sqlite_utils.cli import VALID_COLUMN_TYPES


if __name__ == '__main__':
    # Choices of the option `--type`
    cli.commands['transform'].params[2].type.types[1].choices.append('DATETIME')

    # The dicts has to be extended with a new type
    with patch.dict('sqlite_utils.db.COLUMN_TYPE_MAPPING', {'DATETIME': 'DATETIME'}),\
         patch('sqlite_utils.cli.VALID_COLUMN_TYPES', VALID_COLUMN_TYPES + (""DATETIME"", )):

        # Command is unchanged
        sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
        sys.exit(cli())
```

And now it's working
```bash
$ sqlite-utils schema events.sqlite cards.chunk.get
CREATE TABLE ""cards.chunk.get"" (
   [id] INTEGER PRIMARY KEY NOT NULL,
   [timestamp] TEXT,
)

$ python patched/sqlite-utils.py transform events.sqlite cards.chunk.get --type timestamp DATETIME

$ sqlite-utils schema events.sqlite cards.chunk.get
CREATE TABLE ""cards.chunk.get"" (
   [id] INTEGER PRIMARY KEY NOT NULL,
   [timestamp] DATETIME,
)
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/datasette/pull/1999#issuecomment-1421988953,https://api.github.com/repos/simonw/datasette/issues/1999,1421988953,IC_kwDOBm6k_c5UwdRZ,9599,simonw,2023-02-08T04:35:44Z,2023-02-08T05:27:48Z,OWNER,"Next step: get `?_next=...` working (it is ignored at the moment, even though the returned JSON includes the `""next""` key).

Then... figure out how to render HTML and other requested formats.

Then get the tests to pass!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft),
https://github.com/simonw/datasette/issues/2019#issuecomment-1421784930,https://api.github.com/repos/simonw/datasette/issues/2019,1421784930,IC_kwDOBm6k_c5Uvrdi,9599,simonw,2023-02-08T01:28:25Z,2023-02-08T01:40:46Z,OWNER,"Rather than duplicate this rather awful hack:

https://github.com/simonw/datasette/blob/0b4a28691468b5c758df74fa1d72a823813c96bf/datasette/views/table.py#L694-L714

I'm tempted to say that the code that calls the new pagination helper needs to ensure that the `sort` or `sort_desc` columns are selected. If it wants to ditch them later (e.g. because they were not included in `?_col=`) it can do that later once the results have come back.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code,
https://github.com/simonw/datasette/issues/2019#issuecomment-1421600789,https://api.github.com/repos/simonw/datasette/issues/2019,1421600789,IC_kwDOBm6k_c5Uu-gV,9599,simonw,2023-02-07T23:12:40Z,2023-02-07T23:16:20Z,OWNER,"Most complicated example of a paginated query: https://latest.datasette.io/fixtures?sql=select%0D%0A++pk1%2C%0D%0A++pk2%2C%0D%0A++content%2C%0D%0A++sortable%2C%0D%0A++sortable_with_nulls%2C%0D%0A++sortable_with_nulls_2%2C%0D%0A++text%0D%0Afrom%0D%0A++sortable%0D%0Awhere%0D%0A++(%0D%0A++++sortable_with_nulls+is+null%0D%0A++++and+(%0D%0A++++++(pk1+%3E+%3Ap0)%0D%0A++++++or+(%0D%0A++++++++pk1+%3D+%3Ap0%0D%0A++++++++and+pk2+%3E+%3Ap1%0D%0A++++++)%0D%0A++++)%0D%0A++)%0D%0Aorder+by%0D%0A++sortable_with_nulls+desc%2C%0D%0A++pk1%2C%0D%0A++pk2%0D%0Alimit%0D%0A++101&p0=h&p1=r

```sql
select
  pk1,
  pk2,
  content,
  sortable,
  sortable_with_nulls,
  sortable_with_nulls_2,
  text
from
  sortable
where
  (
    sortable_with_nulls is null
    and (
      (pk1 > :p0)
      or (
        pk1 = :p0
        and pk2 > :p1
      )
    )
  )
order by
  sortable_with_nulls desc,
  pk1,
  pk2
limit
  101
```
Generated by this page: https://latest.datasette.io/fixtures/sortable?_next=%24null%2Ch%2Cr&_sort_desc=sortable_with_nulls

The `_next=` parameter there decodes as `$null,h,r` - and those components are tilde-encoded, so this can be distinguished from an actual `$null` value which would be represented as `~24null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code,
https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1421571810,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1421571810,IC_kwDOCGYnMM5Uu3bi,167893,mcarpenter,2023-02-07T22:43:09Z,2023-02-07T22:43:09Z,CONTRIBUTOR,"Hey, isn't this essentially the same issue as #448 ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode,
https://github.com/simonw/datasette/issues/2019#issuecomment-1421274434,https://api.github.com/repos/simonw/datasette/issues/2019,1421274434,IC_kwDOBm6k_c5Utu1C,9599,simonw,2023-02-07T18:42:42Z,2023-02-07T18:42:42Z,OWNER,I'm going to build completely separate tests for this in `test_pagination.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1573424830,Refactor out the keyset pagination code,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421177666,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421177666,IC_kwDOCGYnMM5UtXNC,21095447,4l1fe,2023-02-07T17:39:00Z,2023-02-07T17:39:00Z,NONE,">  lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed

I've got your concern. Let's see if we will be replied on it and i'll close the issue some later.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421081939,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421081939,IC_kwDOCGYnMM5Us_1T,193185,cldellow,2023-02-07T16:42:25Z,2023-02-07T16:43:42Z,NONE,"Ha, yes, I might end up making something very niche. That's OK.

I'm building a UI for [Datasette](https://datasette.io/) that lets users make schema changes, so it's important to me that the tool work in a non-surprising way -- if you ask for a column of type X, you should get type X. If the column or table previously had CHECK constraints, they shouldn't be silently removed. And so on. I had hoped that I could just lean on sqlite-utils, but I think it's a little too surprising.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421055590,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421055590,IC_kwDOCGYnMM5Us5Zm,21095447,4l1fe,2023-02-07T16:25:31Z,2023-02-07T16:25:31Z,NONE,"> Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178
> 
> I suspect you could overwrite the datetime entry to achieve what you want

And thank you for pointing me to it. At least, i can make a monkey patch for my need...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421052195,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421052195,IC_kwDOCGYnMM5Us4kj,21095447,4l1fe,2023-02-07T16:23:17Z,2023-02-07T16:23:57Z,NONE,"Isn't your suggestion too fundamental for the utility?

The bigger flexibility, the bigger complexity. Your idea make sense defenitely, but how often do you make schema changes? And how many people could benefit from it, what do you think?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421033725,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421033725,IC_kwDOCGYnMM5Us0D9,193185,cldellow,2023-02-07T16:12:13Z,2023-02-07T16:12:13Z,NONE,"I think the bigger issue is that `sqlite-utils` mixes mechanism (it implements the [12-step way to alter SQLite tables](https://www.sqlite.org/lang_altertable.html#otheralter)) and policy (it has an opinionated stance on what column types should be used).

That might be a design choice to make it accessible to users by providing a reasonable set of defaults, but it doesn't quite fit my use case.

It might make sense to extract a separate library that provides just the mechanisms, and then `sqlite-utils` would sit on top of that library with its opinionated set of policies.

That would be a very big change, though.

I might take a stab at extracting the library, but just for the table schema migration piece, not all the other features that `sqlite-utils` supports. I wouldn't expect `sqlite-utils` to depend on it.

Part of my motivation is that I want to provide some other abilities, too, like support for CHECK constraints. I see that the issue in this repo (https://github.com/simonw/sqlite-utils/issues/358) proposes a bunch of short-hand constraints, which I wouldn't want to accidentally expose to people -- I want a layer that is a 1:1 mapping to SQLite.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1421022917,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1421022917,IC_kwDOCGYnMM5UsxbF,21095447,4l1fe,2023-02-07T16:06:03Z,2023-02-07T16:08:58Z,NONE,"> Do you see a way to enable it without affecting existing users or bumping the major version number?

I don't see a clean solution, only extending code with a side variable that tells us we want to apply advanced types instead of basic.

it could be a similiar command like `tranform-v2 --type column DATETIME` or a cli option `transform --adv-type column DATETIME` along with a dict that contains the advanced types. Then with knowledge that we run an advanced command we take that dictionary somehow, we can wrap the current and new dictionaries by a superdict and work with it everywhere according to the knowledge. This way shouldn't affect users who are using the previous lib versions and it have to be merged in the next major one.

But this way looks a bad design, too messy.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420992261,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420992261,IC_kwDOCGYnMM5Usp8F,193185,cldellow,2023-02-07T15:45:58Z,2023-02-07T15:45:58Z,NONE,"I'd support that, but I'm not the author of this library.

One challenge is that would be a breaking change. Do you see a way to enable it without affecting existing users or bumping the major version number?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420966995,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420966995,IC_kwDOCGYnMM5UsjxT,21095447,4l1fe,2023-02-07T15:29:28Z,2023-02-07T15:29:28Z,NONE,"I could, of course.

Doest it worth bringing such the improvement to the library?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/datasette/pull/564#issuecomment-1420941334,https://api.github.com/repos/simonw/datasette/issues/564,1420941334,IC_kwDOBm6k_c5UsdgW,82988,psychemedia,2023-02-07T15:14:10Z,2023-02-07T15:14:10Z,CONTRIBUTOR,"Is this feature covered by any more recent updates to `datasette`, or via any plugins that you're aware of?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473288428,First proof-of-concept of Datasette Library,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420809773,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420809773,IC_kwDOCGYnMM5Ur9Yt,193185,cldellow,2023-02-07T13:53:01Z,2023-02-07T13:53:01Z,NONE,"Ah, it looks like that is controlled by this dict: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L178

I suspect you could overwrite the datetime entry to achieve what you want","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,
https://github.com/simonw/sqlite-utils/issues/524#issuecomment-1420496447,https://api.github.com/repos/simonw/sqlite-utils/issues/524,1420496447,IC_kwDOCGYnMM5Uqw4_,21095447,4l1fe,2023-02-07T09:57:38Z,2023-02-07T09:57:38Z,NONE,"> That said, it looks like the check is only enforced at the CLI level. If you use the API directly, I think it'll work.

It works, but a column becomes `TEXT`

```python
In [1]: import sqlite_utils
In [2]: db = sqlite_utils.Database('events.sqlite')
In [3]: table = db['cards.chunk.get']
In [4]: table.columns_dict
Out[4]:
{'id': int,
 'timestamp': float,
 'data_chunk_number': int,
 'user_id': str,
 'meta_duplication_source_id': int,
 'context_sort_attribute': str,
 'context_sort_order': str}

In [5]: from datetime import datetime
In [7]: table.transform(types={'timestamp': datetime})
In [8]: table.columns_dict
Out[8]:
{'id': int,
 'timestamp': str,
 'data_chunk_number': int,
 'user_id': str,
 'meta_duplication_source_id': int,
 'context_sort_attribute': str,
 'context_sort_order': str}
```

```bash
❯ sqlite-utils schema events.sqlite cards.chunk.get
CREATE TABLE ""cards.chunk.get"" (
   [id] INTEGER PRIMARY KEY NOT NULL,
   [timestamp] TEXT,
   ...
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1572766460,Transformation type `--type DATETIME`,