html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/issues/2214#issuecomment-1844819002,https://api.github.com/repos/simonw/datasette/issues/2214,1844819002,IC_kwDOBm6k_c5t9bQ6,2874,precipice,2023-12-07T07:36:33Z,2023-12-07T07:36:33Z,NONE,"If I uncheck `expand labels` in the Advanced CSV export dialog, the error does not occur. Re-checking that box and re-running the export does cause the error to occur. ![CleanShot 2023-12-06 at 23 34 58@2x](https://github.com/simonw/datasette/assets/2874/12c6c241-35ce-4ded-8dc7-fc250d809ed9) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029908157,CSV export fails for some `text` foreign key references, https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843579184,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843579184,IC_kwDOCGYnMM5t4skw,9599,simonw,2023-12-06T19:43:55Z,2023-12-06T19:43:55Z,OWNER,"Updated documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-add-column - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#add-column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,str and int as aliases for text and integer, https://github.com/simonw/sqlite-utils/issues/606#issuecomment-1843465748,https://api.github.com/repos/simonw/sqlite-utils/issues/606,1843465748,IC_kwDOCGYnMM5t4Q4U,9599,simonw,2023-12-06T18:36:51Z,2023-12-06T18:36:51Z,OWNER,I'll add `bytes` too - `float` already works. This makes sense because when you are working with the Python API you use `str` and `float` and `bytes` and `int` to specify column types.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2029161033,str and int as aliases for text and integer, https://github.com/simonw/datasette/issues/2213#issuecomment-1843072926,https://api.github.com/repos/simonw/datasette/issues/2213,1843072926,IC_kwDOBm6k_c5t2w-e,536941,fgregg,2023-12-06T15:05:44Z,2023-12-06T15:05:44Z,CONTRIBUTOR,"it probably does not make sense to gzip large sqlite database files on the fly. it can take many seconds to gzip a large file and you either have to have this big thing in memory, or write it to disk, which some deployment environments will not like. i wonder if it would make sense to gzip the databases as part of the datasette publish process. it would be very cool to statically serve those as if they dynamically zipped (i.e. serve the filename example.db, not example.db.zip, and rely on the browser to expand).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2028698018,feature request: gzip compression of database downloads, https://github.com/simonw/sqlite-utils/issues/605#issuecomment-1846554637,https://api.github.com/repos/simonw/sqlite-utils/issues/605,1846554637,IC_kwDOCGYnMM5uEDAN,9599,simonw,2023-12-08T05:07:54Z,2023-12-08T05:07:54Z,OWNER,"Thanks for opening an issue - this should help future Google searchers figure out what's going on here. Another approach here could be to store large integers as `TEXT` in SQLite (or even as `BLOB`). Both storing as `REAL` and storing as `TEXT/BLOB` feel nasty to me, but it looks like SQLite has a hard upper limit of 9223372036854775807 for integers.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2007893839,Insert fails with `Error: Python int too large to convert to SQLite INTEGER`; can we use `NUMERIC` here?, https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1846560096,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1846560096,IC_kwDOCGYnMM5uEEVg,9599,simonw,2023-12-08T05:16:44Z,2023-12-08T05:17:20Z,OWNER,"Also tested this manually like so: ```bash sqlite-utils create-table strict.db strictint id integer size integer --strict sqlite-utils create-table strict.db notstrictint id integer size integer sqlite-utils install sqlite-utils-shell sqlite-utils shell strict.db ``` ``` Attached to strict.db Type 'exit' to exit. sqlite-utils> insert into strictint (size) values (4); 1 row affected sqlite-utils> insert into strictint (size) values ('four'); An error occurred: cannot store TEXT value in INTEGER column strictint.size sqlite-utils> insert into notstrictint (size) values ('four'); 1 row affected sqlite-utils> commit; Done ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support, https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843975536,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843975536,IC_kwDOCGYnMM5t6NVw,16437338,tkhattra,2023-12-07T01:17:05Z,2023-12-07T01:17:05Z,CONTRIBUTOR,Apologies - I pushed a fix that addresses the mypy failures.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support, https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843586503,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843586503,IC_kwDOCGYnMM5t4uXH,9599,simonw,2023-12-06T19:49:10Z,2023-12-06T19:49:29Z,OWNER,"This looks really great on first glance - design is good, implementation is solid, tests and documentation look great. Looks like a couple of `mypy` failures in the tests at the moment: ``` mypy sqlite_utils tests sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type ""type[Table]"", variable has type ""type[View]"") [assignment] tests/test_lookup.py:156: error: Name ""test_lookup_new_table"" already defined on line 5 [no-redef] Found 2 errors in 2 files (checked 54 source files) Error: Process completed with exit code 1. ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support, https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843585454,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843585454,IC_kwDOCGYnMM5t4uGu,22429695,codecov[bot],2023-12-06T19:48:26Z,2023-12-08T05:05:03Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`9286c1b`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/9286c1ba432e890b1bb4b2a1f847b15364c1fa18?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`1698a9d`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.72%. > Report is 1 commits behind head on main. > :exclamation: Current head 1698a9d differs from pull request most recent head 61c6e26. Consider uploading reports for the commit 61c6e26 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #604 +/- ## ========================================== - Coverage 95.77% 95.72% -0.06% ========================================== Files 8 8 Lines 2842 2852 +10 ========================================== + Hits 2722 2730 +8 - Misses 120 122 +2 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/604?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157,Add more STRICT table support,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}" https://github.com/simonw/datasette/pull/2209#issuecomment-1812753347,https://api.github.com/repos/simonw/datasette/issues/2209,1812753347,IC_kwDOBm6k_c5sDGvD,22429695,codecov[bot],2023-11-15T15:31:12Z,2023-11-15T15:31:12Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`c88414b`)](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2209 +/- ## ======================================= Coverage 92.69% 92.69% ======================================= Files 40 40 Lines 6047 6047 ======================================= Hits 5605 5605 Misses 442 442 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2209?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}" https://github.com/simonw/datasette/pull/2209#issuecomment-1812750369,https://api.github.com/repos/simonw/datasette/issues/2209,1812750369,IC_kwDOBm6k_c5sDGAh,198537,rgieseke,2023-11-15T15:29:37Z,2023-11-15T15:29:37Z,CONTRIBUTOR,"Looks like tests are passing now but there is an issue with yaml loading and/or cog. https://github.com/simonw/datasette/actions/runs/6879299298/job/18710911166?pr=2209","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value, https://github.com/simonw/datasette/pull/2209#issuecomment-1812623778,https://api.github.com/repos/simonw/datasette/issues/2209,1812623778,IC_kwDOBm6k_c5sCnGi,198537,rgieseke,2023-11-15T14:22:42Z,2023-11-15T15:24:09Z,CONTRIBUTOR,"Whoops, looks like I forgot to check for other places where the 'facetable' table is used in the tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994861266,Fix query for suggested facets with column named value, https://github.com/simonw/datasette/issues/2208#issuecomment-1812617851,https://api.github.com/repos/simonw/datasette/issues/2208,1812617851,IC_kwDOBm6k_c5sClp7,198537,rgieseke,2023-11-15T14:18:58Z,2023-11-15T14:18:58Z,CONTRIBUTOR,"Without aliases: ![image](https://github.com/simonw/datasette/assets/198537/d9703d3b-9733-4e87-9954-4fc60a07784a) The proposed fix in #2209 also works when the 'value' column is actually facetable (just added another value in the 'value' column). ![image](https://github.com/simonw/datasette/assets/198537/a37a0a1a-c36a-4c78-bdce-01b582637cc6) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1994857251,No suggested facets when a column named 'value' is included, https://github.com/simonw/sqlite-utils/issues/603#issuecomment-1846555822,https://api.github.com/repos/simonw/sqlite-utils/issues/603,1846555822,IC_kwDOCGYnMM5uEDSu,9599,simonw,2023-12-08T05:09:55Z,2023-12-08T05:10:31Z,OWNER,"I'm unable to replicate this issue. This is with a fresh install of `sqlite-utils==3.35.2`: ``` (base) ~ python3.12 Python 3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db[""foo""].insert({""bar"": 1}) >>> import sys >>> sys.version '3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1988525411,Pyhton 3.12 Bug report, https://github.com/simonw/datasette/pull/2206#issuecomment-1801888957,https://api.github.com/repos/simonw/datasette/issues/2206,1801888957,IC_kwDOBm6k_c5rZqS9,22429695,codecov[bot],2023-11-08T13:26:13Z,2023-11-08T13:26:13Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`eec10df`)](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2206 +/- ## ======================================= Coverage 92.69% 92.69% ======================================= Files 40 40 Lines 6047 6047 ======================================= Hits 5605 5605 Misses 442 442 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2206?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1983600865,Bump the python-packages group with 1 update,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}" https://github.com/simonw/datasette/issues/2205#issuecomment-1794054390,https://api.github.com/repos/simonw/datasette/issues/2205,1794054390,IC_kwDOBm6k_c5q7xj2,9599,simonw,2023-11-06T04:09:43Z,2023-11-06T04:10:34Z,OWNER,"That `keep_blank_values=True` is from https://github.com/simonw/datasette/commit/0934844c0b6d124163d0185fb6a41ba5a71433da Commit message: > request.post_vars() no longer discards empty values Relevant test: https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/tests/test_internals_request.py#L19-L27","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1978023780,request.post_vars() method obliterates form keys with multiple values, https://github.com/simonw/datasette/issues/2205#issuecomment-1794052079,https://api.github.com/repos/simonw/datasette/issues/2205,1794052079,IC_kwDOBm6k_c5q7w_v,9599,simonw,2023-11-06T04:06:05Z,2023-11-06T04:08:50Z,OWNER,"It should return a `MultiParams`: https://github.com/simonw/datasette/blob/452a587e236ef642cbc6ae345b58767ea8420cb5/datasette/utils/__init__.py#L900-L917 Change needs to be made before 1.0. ```python return MultiParams(urllib.parse.parse_qs(body.decode(""utf-8""))) ``` Need to remember why I was using `keep_blank_values=True` there and check that using `MultiParams` doesn't conflict with that reason.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1978023780,request.post_vars() method obliterates form keys with multiple values, https://github.com/simonw/datasette/issues/2205#issuecomment-1793880738,https://api.github.com/repos/simonw/datasette/issues/2205,1793880738,IC_kwDOBm6k_c5q7HKi,9599,simonw,2023-11-05T23:26:14Z,2023-11-05T23:26:14Z,OWNER,"I found this problem while trying to use WTForms with this pattern: ```python choices = [(col, col) for col in await db.table_columns(table)] class ConfigForm(Form): template = TextAreaField(""Template"") api_token = PasswordField(""OpenAI API token"") columns = MultiCheckboxField('Columns', choices=choices) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1978023780,request.post_vars() method obliterates form keys with multiple values, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793269219,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793269219,IC_kwDOCGYnMM5q4x3j,9599,simonw,2023-11-04T00:34:33Z,2023-11-04T00:34:33Z,OWNER,"The GIS tests now pass in that container too: ```bash pytest tests/test_gis.py ``` ``` ======================== test session starts ========================= platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0 rootdir: /tmp/sqlite-utils plugins: hypothesis-6.88.1 collected 12 items tests/test_gis.py ............ [100%] ========================= 12 passed in 0.48s ========================= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793268126,IC_kwDOCGYnMM5q4xme,9599,simonw,2023-11-04T00:31:34Z,2023-11-04T00:31:34Z,OWNER,"Testing this manually on macOS using Docker Desk top like this: ```bash docker run -it --rm arm64v8/ubuntu /bin/bash ``` Then inside the container: ```bash uname -m ``` Outputs: `aarch64` Then: ```bash apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y cd /tmp git clone https://github.com/simonw/sqlite-utils cd sqlite-utils python3 -m venv venv source venv/bin/activate pip install -e '.[test]' sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Which output: ``` Traceback (most recent call last): File ""/tmp/sqlite-utils/venv/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1157, in __call__ return self.main(*args, **kwargs) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1078, in main rv = self.invoke(ctx) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 783, in invoke return __callback(*args, **kwargs) File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 1959, in memory _load_extensions(db, load_extension) File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 3232, in _load_extensions if "":"" in ext: TypeError: argument of type 'NoneType' is not iterable ``` Then I ran this: ```bash git checkout -b MikeCoats-spatialite-paths-linux-arm main git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm ``` And now: ```bash sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Outputs: ```json [{""spatialite_version()"": ""5.0.1""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793265952,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793265952,IC_kwDOCGYnMM5q4xEg,9599,simonw,2023-11-04T00:25:34Z,2023-11-04T00:25:34Z,OWNER,The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793264654,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793264654,IC_kwDOCGYnMM5q4wwO,22429695,codecov[bot],2023-11-04T00:22:07Z,2023-11-04T00:27:29Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`622c3a5`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/622c3a5a7dd53a09c029e2af40c2643fe7579340?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b1a6076`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #600 +/- ## ======================================= Coverage 95.77% 95.77% ======================================= Files 8 8 Lines 2840 2840 ======================================= Hits 2720 2720 Misses 120 120 ``` | [Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <ø> (ø)` | | | [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <ø> (ø)` | |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/600?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path,"{""id"": 254, ""slug"": ""codecov"", ""node_id"": ""MDM6QXBwMjU0"", ""owner"": {""login"": ""codecov"", ""id"": 8226205, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjgyMjYyMDU="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/8226205?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/codecov"", ""html_url"": ""https://github.com/codecov"", ""followers_url"": ""https://api.github.com/users/codecov/followers"", ""following_url"": ""https://api.github.com/users/codecov/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/codecov/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/codecov/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/codecov/subscriptions"", ""organizations_url"": ""https://api.github.com/users/codecov/orgs"", ""repos_url"": ""https://api.github.com/users/codecov/repos"", ""events_url"": ""https://api.github.com/users/codecov/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/codecov/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Codecov"", ""description"": ""Codecov provides highly integrated tools to group, merge, archive and compare coverage reports. Whether your team is comparing changes in a pull request or reviewing a single commit, Codecov will improve the code review workflow and quality.\r\n\r\n## Code coverage done right.®\r\n\r\n1. Upload coverage reports from your CI builds.\r\n2. Codecov merges all builds and languages into one beautiful coherent report.\r\n3. Get commit statuses, pull request comments and coverage overlay via our browser extension.\r\n\r\nWhen Codecov merges your uploads it keeps track of the CI provider (inc. build details) and user specified context, e.g. `#unittest` ~ `#smoketest` or `#oldcode` ~ `#newcode`. You can track the `#unittest` coverage independently of other groups. [Learn more here](\r\nhttp://docs.codecov.io/docs/flags)\r\n\r\nThrough **Codecov's Browser Extension** reports overlay directly in GitHub UI to assist in code review in [Chrome](https://chrome.google.com/webstore/detail/codecov/gedikamndpbemklijjkncpnolildpbgo) or Firefox (https://addons.mozilla.org/en-US/firefox/addon/codecov/)\r\n\r\n*Highly detailed* **pull request comments** and *customizable* **commit statuses** will improve your team's workflow and code coverage incrementally.\r\n\r\n**File backed configuration** all through the `codecov.yml`. \r\n\r\n## FAQ\r\n- Do you **merge multiple uploads** to the same commit? **Yes**\r\n- Do you **support multiple languages** in the same project? **Yes**\r\n- Can you **group coverage reports** by project and/or test type? **Yes**\r\n- How does **pricing** work? Only paid users can view reports and post statuses/comments. "", ""external_url"": ""https://codecov.io"", ""html_url"": ""https://github.com/apps/codecov"", ""created_at"": ""2016-09-25T14:18:27Z"", ""updated_at"": ""2023-09-08T15:29:16Z"", ""permissions"": {""administration"": ""read"", ""checks"": ""write"", ""contents"": ""read"", ""emails"": ""read"", ""issues"": ""read"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""write""}, ""events"": [""check_run"", ""check_suite"", ""create"", ""delete"", ""fork"", ""member"", ""membership"", ""organization"", ""public"", ""pull_request"", ""push"", ""release"", ""repository"", ""status"", ""team_add""]}" https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793263638,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793263638,IC_kwDOCGYnMM5q4wgW,9599,simonw,2023-11-04T00:19:58Z,2023-11-04T00:19:58Z,OWNER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379,Add spatialite arm64 linux path, https://github.com/simonw/sqlite-utils/issues/599#issuecomment-1793268750,https://api.github.com/repos/simonw/sqlite-utils/issues/599,1793268750,IC_kwDOCGYnMM5q4xwO,9599,simonw,2023-11-04T00:33:25Z,2023-11-04T00:33:25Z,OWNER,"See details of how I tested this here: - https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126 Short version: having applied this fix, the following command (on simulated `aarch64`): ```bash sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Outputs: ```json [{""spatialite_version()"": ""5.0.1""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1976986318,Cannot find spatialite on arm64 linux, https://github.com/simonw/datasette/pull/2202#issuecomment-1801876943,https://api.github.com/repos/simonw/datasette/issues/2202,1801876943,IC_kwDOBm6k_c5rZnXP,49699333,dependabot[bot],2023-11-08T13:19:00Z,2023-11-08T13:19:00Z,CONTRIBUTOR,Superseded by #2206.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1959278971,Bump the python-packages group with 1 update,"{""id"": 29110, ""slug"": ""dependabot"", ""node_id"": ""MDM6QXBwMjkxMTA="", ""owner"": {""login"": ""github"", ""id"": 9919, ""node_id"": ""MDEyOk9yZ2FuaXphdGlvbjk5MTk="", ""avatar_url"": ""https://avatars.githubusercontent.com/u/9919?v=4"", ""gravatar_id"": """", ""url"": ""https://api.github.com/users/github"", ""html_url"": ""https://github.com/github"", ""followers_url"": ""https://api.github.com/users/github/followers"", ""following_url"": ""https://api.github.com/users/github/following{/other_user}"", ""gists_url"": ""https://api.github.com/users/github/gists{/gist_id}"", ""starred_url"": ""https://api.github.com/users/github/starred{/owner}{/repo}"", ""subscriptions_url"": ""https://api.github.com/users/github/subscriptions"", ""organizations_url"": ""https://api.github.com/users/github/orgs"", ""repos_url"": ""https://api.github.com/users/github/repos"", ""events_url"": ""https://api.github.com/users/github/events{/privacy}"", ""received_events_url"": ""https://api.github.com/users/github/received_events"", ""type"": ""Organization"", ""site_admin"": false}, ""name"": ""Dependabot"", ""description"": """", ""external_url"": ""https://dependabot-api.githubapp.com"", ""html_url"": ""https://github.com/apps/dependabot"", ""created_at"": ""2019-04-16T22:34:25Z"", ""updated_at"": ""2023-10-12T13:35:09Z"", ""permissions"": {""checks"": ""write"", ""contents"": ""write"", ""issues"": ""write"", ""members"": ""read"", ""metadata"": ""read"", ""pull_requests"": ""write"", ""statuses"": ""read"", ""vulnerability_alerts"": ""read"", ""workflows"": ""write""}, ""events"": [""check_suite"", ""issues"", ""issue_comment"", ""label"", ""pull_request"", ""pull_request_review"", ""pull_request_review_comment"", ""repository""]}" https://github.com/simonw/datasette/pull/2202#issuecomment-1777247375,https://api.github.com/repos/simonw/datasette/issues/2202,1777247375,IC_kwDOBm6k_c5p7qSP,22429695,codecov[bot],2023-10-24T13:49:27Z,2023-10-24T13:49:27Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`452a587`)](https://app.codecov.io/gh/simonw/datasette/commit/452a587e236ef642cbc6ae345b58767ea8420cb5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`be4d0f0`)](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2202 +/- ## ======================================= Coverage 92.69% 92.69% ======================================= Files 40 40 Lines 6047 6047 ======================================= Hits 5605 5605 Misses 442 442 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2202?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1959278971,Bump the python-packages group with 1 update, https://github.com/simonw/datasette/pull/2200#issuecomment-1777228352,https://api.github.com/repos/simonw/datasette/issues/2200,1777228352,IC_kwDOBm6k_c5p7lpA,49699333,dependabot[bot],2023-10-24T13:40:25Z,2023-10-24T13:40:25Z,CONTRIBUTOR,Superseded by #2202.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1949756141,Bump the python-packages group with 1 update, https://github.com/simonw/datasette/issues/2199#issuecomment-1760441535,https://api.github.com/repos/simonw/datasette/issues/2199,1760441535,IC_kwDOBm6k_c5o7jS_,9599,simonw,2023-10-12T22:08:42Z,2023-10-12T22:08:42Z,OWNER,Pushed that incomplete code here: https://github.com/datasette/datasette-upgrade,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml, https://github.com/simonw/datasette/issues/2199#issuecomment-1760413191,https://api.github.com/repos/simonw/datasette/issues/2199,1760413191,IC_kwDOBm6k_c5o7cYH,9599,simonw,2023-10-12T21:52:25Z,2023-10-12T21:52:25Z,OWNER,"Demo of that logic: ``` $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-1.yaml New datasette.yaml file will be written to datasette.yaml $ touch metadata-new-1.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette.yaml $ touch datasette.yaml $ datasette upgrade metadata-to-config ../datasette/metadata.json Upgrading ../datasette/metadata.json to new metadata.yaml format New metadata.yaml file will be written to metadata-new-2.yaml New datasette.yaml file will be written to datasette-new.yaml ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml, https://github.com/simonw/datasette/issues/2199#issuecomment-1760412424,https://api.github.com/repos/simonw/datasette/issues/2199,1760412424,IC_kwDOBm6k_c5o7cMI,9599,simonw,2023-10-12T21:51:44Z,2023-10-12T21:51:44Z,OWNER,"Started playing with this plugin idea, now tearing myself away to work on something more important: ```python from datasette import hookimpl import click import pathlib @hookimpl def register_commands(cli): @cli.group() def upgrade(): """""" Apply configuration upgrades to an existing Datasette instance """""" pass @upgrade.command() @click.argument( ""metadata"", type=click.Path(exists=True) ) @click.option( ""new_metadata"", ""-m"", ""--new-metadata"", help=""Path to new metadata.yaml file"", type=click.Path(exists=False) ) @click.option( ""new_datasette"", ""-c"", ""--new-datasette"", help=""Path to new datasette.yaml file"", type=click.Path(exists=False) ) @click.option( ""output_dir"", ""-e"", ""--output-dir"", help=""Directory to write new files to"", type=click.Path(), default=""."" ) def metadata_to_config(metadata, new_metadata, new_datasette, output_dir): """""" Upgrade an existing metadata.json/yaml file to the new metadata.yaml and datasette.yaml split introduced prior to Datasette 1.0. """""" print(""Upgrading {} to new metadata.yaml format"".format(metadata)) output_dir = pathlib.Path(output_dir) if not new_metadata: # Pick a filename for the new metadata.yaml file that does not yet exist new_metadata = pick_filename(""metadata"", output_dir) if not new_datasette: new_datasette = pick_filename(""datasette"", output_dir) print(""New metadata.yaml file will be written to {}"".format(new_metadata)) print(""New datasette.yaml file will be written to {}"".format(new_datasette)) def pick_filename(base, output_dir): options = [""{}.yaml"".format(base), ""{}-new.yaml"".format(base)] i = 0 while True: option = options.pop(0) option_path = output_dir / option if not option_path.exists(): return option_path # If we ran out if not options: i += 1 options = [""{}-new-{}.yaml"".format(base, i)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml, https://github.com/simonw/datasette/issues/2199#issuecomment-1760411937,https://api.github.com/repos/simonw/datasette/issues/2199,1760411937,IC_kwDOBm6k_c5o7cEh,9599,simonw,2023-10-12T21:51:16Z,2023-10-12T21:51:16Z,OWNER,"I think I'm OK with not preserving comments, just because it adds a level of complexity to the tool which I don't think is worth the value it provides. If people want to keep their comments I'm happy to leave them to copy those over by hand.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml, https://github.com/simonw/datasette/issues/2199#issuecomment-1760401731,https://api.github.com/repos/simonw/datasette/issues/2199,1760401731,IC_kwDOBm6k_c5o7ZlD,15178711,asg017,2023-10-12T21:41:42Z,2023-10-12T21:41:42Z,CONTRIBUTOR,"I dig it - I was thinking an Observable notebook where you paste your `metadata.json`/`metadata.yaml` and it would generate the new metadata + datasette.yaml files, but an extensible `datasette upgrade` plugin would be nice for future plugins. One thing to think about: If someone has comments in their original `metadata.yaml`, could we preserve them in the new files? tbh maybe not too important bc if people cared that much they could just copy + paste, and it might be too distracting ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml, https://github.com/simonw/datasette/issues/2199#issuecomment-1760396195,https://api.github.com/repos/simonw/datasette/issues/2199,1760396195,IC_kwDOBm6k_c5o7YOj,9599,simonw,2023-10-12T21:36:25Z,2023-10-12T21:36:25Z,OWNER,"Related idea: how about a `datasette-upgrade` plugin which adds a `datasette upgrade` command that can be used to automate this process? Maybe something like this: ```bash datasette install datasette-upgrade datasette upgrade metadata-to-config metadata.json ``` This would output two new files: `metadata.yaml` and `datasette.yaml`. If files with those names existed already in the current directory they would be called `metadata-new.yaml` and `datasette-new.yaml`. The command would tell you what it did: ``` Your metadata.json file has been rewritten as two files: metadata-new.yaml datasette.yaml Start Datasette like this to try them out: datasette -m metadata-new.yaml -c datasette.yaml ``` The command is `datasette upgrade metadata-to-config` because `metadata-to-config` is the name of the upgrade recipe. The first version of the plugin would only have that single recipe, but we could add more recipes in the future for other upgrades.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml, https://github.com/simonw/datasette/issues/2199#issuecomment-1759952247,https://api.github.com/repos/simonw/datasette/issues/2199,1759952247,IC_kwDOBm6k_c5o5r13,9599,simonw,2023-10-12T16:23:10Z,2023-10-12T16:23:10Z,OWNER,"Some options for where this could go: - Directly in the release notes? I'm not sure about that, those are getting pretty long already. I think the release notes should link to relevant upgrade guides. - On a new page? We could have a ""upgrade instructions"" page in the documentation. - At the bottom of the new https://docs.datasette.io/en/latest/configuration.html page I'm leaning towards the third option at the moment. But... we may also need to provide upgrade instructions for plugin authors. Those could live in a separate area of the documentation though, since issues affecting end-users who configure Datasette and issues affecting plugin authors are unlikely to overlap much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1940346034,Detailed upgrade instructions for metadata.yaml -> datasette.yaml, https://github.com/simonw/datasette/issues/2197#issuecomment-1752096661,https://api.github.com/repos/simonw/datasette/issues/2197,1752096661,IC_kwDOBm6k_c5obt-V,9599,simonw,2023-10-08T16:17:04Z,2023-10-08T16:17:29Z,OWNER,"https://lite.datasette.io/?install=datasette-packages#/-/packages confirms that Datasette Lite still works (`click-default-group-wheel` was originally built to allow it to run) and that it's now using these packages: ``` ""click-default-group"": ""1.2.4"", ""datasette"": ""0.64.5"", ``` Full list: ```json { ""aiofiles"": ""23.2.1"", ""anyio"": ""3.7.1"", ""asgi-csrf"": ""0.9"", ""asgiref"": ""3.7.2"", ""certifi"": ""2022.12.7"", ""click"": ""8.1.3"", ""click-default-group"": ""1.2.4"", ""datasette"": ""0.64.5"", ""datasette-packages"": ""0.2"", ""h11"": ""0.12.0"", ""httpcore"": ""0.15.0"", ""httpx"": ""0.23.0"", ""hupper"": ""1.12"", ""idna"": ""3.4"", ""itsdangerous"": ""2.1.2"", ""janus"": ""1.0.0"", ""Jinja2"": ""3.1.2"", ""MarkupSafe"": ""2.1.2"", ""mergedeep"": ""1.3.4"", ""micropip"": ""0.3.0"", ""packaging"": ""23.0"", ""Pint"": ""0.22"", ""pip"": ""23.2.1"", ""pluggy"": ""1.0.0"", ""pyparsing"": ""3.0.9"", ""python-multipart"": ""0.0.6"", ""PyYAML"": ""6.0"", ""rfc3986"": ""1.5.0"", ""setuptools"": ""67.6.1"", ""sniffio"": ""1.3.0"", ""typing-extensions"": ""4.5.0"", ""uvicorn"": ""0.23.2"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1930008379,click-default-group-wheel dependency conflict, https://github.com/simonw/datasette/issues/2197#issuecomment-1752095961,https://api.github.com/repos/simonw/datasette/issues/2197,1752095961,IC_kwDOBm6k_c5obtzZ,9599,simonw,2023-10-08T16:13:42Z,2023-10-08T16:14:39Z,OWNER,"Confirmed - I ran this in a fresh virtual environment: ```bash pip install --no-cache datasette ``` And now: ```bash pip freeze | grep click ``` ``` click==8.1.7 click-default-group==1.2.4 ``` ```bash datasette --version ``` ``` datasette, version 0.64.5 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1930008379,click-default-group-wheel dependency conflict, https://github.com/simonw/datasette/issues/2197#issuecomment-1752093039,https://api.github.com/repos/simonw/datasette/issues/2197,1752093039,IC_kwDOBm6k_c5obtFv,9599,simonw,2023-10-08T15:59:53Z,2023-10-08T15:59:53Z,OWNER,"Replicated this myself: ```bash cd /tmp mkdir dddd cd dddd pipenv shell pip install datasette pip freeze | grep click ``` ``` click==8.1.7 click-default-group==1.2.4 click-default-group-wheel==1.2.3 ``` Yeah this is bad, I'll ship a `0.64.5` release.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1930008379,click-default-group-wheel dependency conflict, https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793274485,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793274485,IC_kwDOCGYnMM5q4zJ1,9599,simonw,2023-11-04T00:46:55Z,2023-11-04T00:46:55Z,OWNER,"Manually tested. Before: ![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/3bdd30ea-1a54-4fec-b42d-793130a17bc1) After: ![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/015d4e4e-b40c-4a93-81f5-1a4adef69b11) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,Fixed issue #433 - CLI eats cursor, https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793272429,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793272429,IC_kwDOCGYnMM5q4ypt,9599,simonw,2023-11-04T00:40:34Z,2023-11-04T00:40:34Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132,Fixed issue #433 - CLI eats cursor, https://github.com/simonw/sqlite-utils/pull/596#issuecomment-1793274869,https://api.github.com/repos/simonw/sqlite-utils/issues/596,1793274869,IC_kwDOCGYnMM5q4zP1,9599,simonw,2023-11-04T00:47:55Z,2023-11-04T00:47:55Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1919296686,"Fixes mapping for time fields related to mysql, closes #522", https://github.com/simonw/datasette/issues/2196#issuecomment-1760560526,https://api.github.com/repos/simonw/datasette/issues/2196,1760560526,IC_kwDOBm6k_c5o8AWO,1892194,Olshansk,2023-10-13T00:07:07Z,2023-10-13T00:07:07Z,NONE,That worked!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1910269679,Discord invite link returns 401, https://github.com/simonw/datasette/issues/2196#issuecomment-1760417555,https://api.github.com/repos/simonw/datasette/issues/2196,1760417555,IC_kwDOBm6k_c5o7dcT,9599,simonw,2023-10-12T21:54:52Z,2023-10-12T21:54:52Z,OWNER,"I can't replicate this bug. Closing, but please re-open if it's still happening. As an aside, the link I promote is https://datasette.io/discord which redirects: ``` curl -i 'https://datasette.io/discord' HTTP/2 301 location: https://discord.gg/ktd74dm5mw content-type: text/plain x-cloud-trace-context: 8dcfd08d3d1fa44f7ee78568e0f5305e;o=1 date: Thu, 12 Oct 2023 21:54:17 GMT server: Google Frontend content-length: 0 ``` ``` curl -i 'https://discord.gg/ktd74dm5mw' HTTP/2 301 date: Thu, 12 Oct 2023 21:54:28 GMT content-type: text/plain;charset=UTF-8 content-length: 0 location: https://discord.com/invite/ktd74dm5mw strict-transport-security: max-age=31536000; includeSubDomains; preload permissions-policy: interest-cohort=() x-content-type-options: nosniff x-frame-options: DENY x-robots-tag: noindex, nofollow, noarchive, nocache, noimageindex, noodp x-xss-protection: 1; mode=block report-to: {""endpoints"":[{""url"":""https:\/\/a.nel.cloudflare.com\/report\/v3?s=Dzzrf%2FgGkfFxtzSAQ46slMVDLcFjsH9fsvVkzHtgUUiZ891rXAa6LvTRpHK%2BdSMSQ54F57hS9z1mZXXklIbONZW1bfBuFjSK9J4XmjjLjsFUulMXvpjfCLkB6PI%3D""}],""group"":""cf-nel"",""max_age"":604800} nel: {""success_fraction"":0,""report_to"":""cf-nel"",""max_age"":604800} server: cloudflare cf-ray: 815294ddff282511-SJC ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1910269679,Discord invite link returns 401, https://github.com/simonw/datasette/issues/2195#issuecomment-1730458954,https://api.github.com/repos/simonw/datasette/issues/2195,1730458954,IC_kwDOBm6k_c5nJLVK,9599,simonw,2023-09-21T22:57:39Z,2023-09-21T22:57:48Z,OWNER,"Worth noting that it already sets `--cors` automatically without you needing to specify it: https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L374-L374 I wonder if that's actually surprising behaviour that we should change before 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730457374,https://api.github.com/repos/simonw/datasette/issues/2195,1730457374,IC_kwDOBm6k_c5nJK8e,9599,simonw,2023-09-21T22:56:18Z,2023-09-21T22:56:18Z,OWNER,Maybe I should add `--cors` and `--crossdb` to `datasette publish cloudrun` as well?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730446937,https://api.github.com/repos/simonw/datasette/issues/2195,1730446937,IC_kwDOBm6k_c5nJIZZ,9599,simonw,2023-09-21T22:46:42Z,2023-09-21T22:46:52Z,OWNER,"Found more when I [searched for YAML](https://github.com/search?q=datasette+publish+extra-options++language%3AYAML&type=code). Here's the most interesting: https://github.com/labordata/warehouse/blob/0029a72fc1ceae9091932da6566f891167179012/.github/workflows/build.yml#L59 `--extra-options=""--crossdb --setting sql_time_limit_ms 100000 --cors --setting facet_time_limit_ms 500 --setting allow_facet off --setting trace_debug 1""` Uses both `--cors` and `--crossdb`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730441613,https://api.github.com/repos/simonw/datasette/issues/2195,1730441613,IC_kwDOBm6k_c5nJHGN,9599,simonw,2023-09-21T22:42:12Z,2023-09-21T22:42:12Z,OWNER,"https://github.com/search?q=datasette+publish+extra-options+language%3AShell&type=code&l=Shell shows 17 matches, I'll copy in illustrative examples here: ``` --extra-options=""--setting sql_time_limit_ms 5000"" --extra-options=""--config default_cache_ttl:3600 --config hash_urls:1"" --extra-options ""--setting sql_time_limit_ms 3500 --setting default_page_size 20 --setting trace_debug 1"" --extra-options=""--config default_page_size:50 --config sql_time_limit_ms:30000 --config facet_time_limit_ms:10000"" --extra-options=""--setting sql_time_limit_ms 5000"" --extra-options ""--setting suggest_facets off --setting allow_download on --setting truncate_cells_html 0 --setting max_csv_mb 0 --setting sql_time_limit_ms 2000"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730438503,https://api.github.com/repos/simonw/datasette/issues/2195,1730438503,IC_kwDOBm6k_c5nJGVn,9599,simonw,2023-09-21T22:38:10Z,2023-09-21T22:38:10Z,OWNER,"I'd really like to remove `--extra-options`. I think the new design makes that completely obsolete? Maybe it doesn't. You still need `--extra-options` for the `--crossdb` option for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730437934,https://api.github.com/repos/simonw/datasette/issues/2195,1730437934,IC_kwDOBm6k_c5nJGMu,9599,simonw,2023-09-21T22:37:22Z,2023-09-21T22:37:22Z,OWNER,"Here's the full help for Cloud Run at the moment: ```bash datasette publish cloudrun --help ``` ``` Usage: datasette publish cloudrun [OPTIONS] [FILES]... Publish databases to Datasette running on Cloud Run Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve --branch TEXT Install datasette from a GitHub branch e.g. main --template-dir DIRECTORY Path to directory containing custom templates --plugins-dir DIRECTORY Path to directory containing custom plugins --static MOUNT:DIRECTORY Serve static files from this directory at /MOUNT/... --install TEXT Additional packages (e.g. plugins) to install --plugin-secret ... Secrets to pass to plugins, e.g. --plugin- secret datasette-auth-github client_id xxx --version-note TEXT Additional note to show on /-/versions --secret TEXT Secret used for signing secure values, such as signed cookies --title TEXT Title for metadata --license TEXT License label for metadata --license_url TEXT License URL for metadata --source TEXT Source label for metadata --source_url TEXT Source URL for metadata --about TEXT About label for metadata --about_url TEXT About URL for metadata -n, --name TEXT Application name to use when building --service TEXT Cloud Run service to deploy (or over-write) --spatialite Enable SpatialLite extension --show-files Output the generated Dockerfile and metadata.json --memory TEXT Memory to allocate in Cloud Run, e.g. 1Gi --cpu [1|2|4] Number of vCPUs to allocate in Cloud Run --timeout INTEGER Build timeout in seconds --apt-get-install TEXT Additional packages to apt-get install --max-instances INTEGER Maximum Cloud Run instances --min-instances INTEGER Minimum Cloud Run instances --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730437237,https://api.github.com/repos/simonw/datasette/issues/2195,1730437237,IC_kwDOBm6k_c5nJGB1,9599,simonw,2023-09-21T22:36:22Z,2023-09-21T22:36:22Z,OWNER,"I think the actual design of this is pretty simple. Current help starts like this: ``` Usage: datasette publish cloudrun [OPTIONS] [FILES]... Publish databases to Datasette running on Cloud Run Options: -m, --metadata FILENAME Path to JSON/YAML file containing metadata to publish --extra-options TEXT Extra options to pass to datasette serve ``` The `-s` and `-c` short options are not being used. So I think `-c/--config` can point to a JSON or YAML `datasette.yaml` file, and `-s/--setting key value` can mirror the new `-s/--setting` option in `datasette serve` itself (a shortcut for populating the config file directly from the CLI). Here's the relevant help section from `datasette serve`: ``` -m, --metadata FILENAME Path to JSON/YAML file containing license/source metadata -c, --config FILENAME Path to JSON/YAML Datasette configuration file -s, --setting SETTING... nested.key, value setting to use in Datasette configuration ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730313565,https://api.github.com/repos/simonw/datasette/issues/2195,1730313565,IC_kwDOBm6k_c5nIn1d,9599,simonw,2023-09-21T21:16:31Z,2023-09-21T21:16:31Z,OWNER,"The `@add_common_publish_arguments_and_options` decorator described here is bad. If I update it to support a new `config` option all plugins that use it will break. https://github.com/simonw/datasette/blob/f130c7c0a88e50cea4121ea18d1f6db2431b6fab/docs/plugin_hooks.rst#L347-L355 I want to deprecate it and switch to a different, better design to address the same problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2195#issuecomment-1730312128,https://api.github.com/repos/simonw/datasette/issues/2195,1730312128,IC_kwDOBm6k_c5nInfA,9599,simonw,2023-09-21T21:15:11Z,2023-09-21T21:15:11Z,OWNER,"As soon as `datasette publish cloudrun` has this I can re-enable this bit of the demo deploy: https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97 Which should fix this broken demo from https://simonwillison.net/2022/Dec/2/datasette-write-api/ https://todomvc.datasette.io/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907765514,`datasette publish` needs support for the new config/metadata split, https://github.com/simonw/datasette/issues/2194#issuecomment-1730362441,https://api.github.com/repos/simonw/datasette/issues/2194,1730362441,IC_kwDOBm6k_c5nIzxJ,9599,simonw,2023-09-21T22:08:19Z,2023-09-21T22:08:19Z,OWNER,"That worked https://github.com/simonw/datasette/commit/e4f868801a6633400045f59584cfe650961c3fa6 is the latest commit right now and https://latest.datasette.io/-/versions shows that as the deployed version.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2194#issuecomment-1730305920,https://api.github.com/repos/simonw/datasette/issues/2194,1730305920,IC_kwDOBm6k_c5nIl-A,9599,simonw,2023-09-21T21:09:21Z,2023-09-21T21:09:21Z,OWNER,"I'm going to disable this bit of the deploy for the moment, which will break the demo linked to from https://simonwillison.net/2022/Dec/2/datasette-write-api/ https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L91-L97 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2194#issuecomment-1730259871,https://api.github.com/repos/simonw/datasette/issues/2194,1730259871,IC_kwDOBm6k_c5nIauf,9599,simonw,2023-09-21T20:34:09Z,2023-09-21T20:34:09Z,OWNER,"... which raises the challenge that `datasette publish` doesn't yet know what to do with a config file! https://github.com/simonw/datasette/blob/2da1a6acec915b81a16127008fd739c7d6075681/.github/workflows/deploy-latest.yml#L114-L122","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2194#issuecomment-1730258302,https://api.github.com/repos/simonw/datasette/issues/2194,1730258302,IC_kwDOBm6k_c5nIaV-,9599,simonw,2023-09-21T20:32:53Z,2023-09-21T20:33:02Z,OWNER,"Correct usage is now: ```bash python tests/fixtures.py fixtures.db fixtures-config.json fixtures-metadata.json \ plugins --extra-db-filename extra_database.db ``` ``` Test tables written to fixtures.db - metadata written to fixtures-metadata.json - config written to fixtures-config.json Wrote plugin: plugins/register_output_renderer.py Wrote plugin: plugins/view_name.py Wrote plugin: plugins/my_plugin.py Wrote plugin: plugins/messages_output_renderer.py Wrote plugin: plugins/sleep_sql_function.py Wrote plugin: plugins/my_plugin_2.py Test tables written to extra_database.db ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2194#issuecomment-1730256435,https://api.github.com/repos/simonw/datasette/issues/2194,1730256435,IC_kwDOBm6k_c5nIZ4z,9599,simonw,2023-09-21T20:31:22Z,2023-09-21T20:31:31Z,OWNER,"New error: ""Error: Metadata should end with .json"" https://github.com/simonw/datasette/actions/runs/6266720924/job/17018265851 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2194#issuecomment-1730245204,https://api.github.com/repos/simonw/datasette/issues/2194,1730245204,IC_kwDOBm6k_c5nIXJU,9599,simonw,2023-09-21T20:21:42Z,2023-09-21T20:21:42Z,OWNER,"I think I see the problem - it's from here: https://github.com/simonw/datasette/commit/b2ec8717c3619260a1b535eea20e618bf95aa30b#diff-5dbc88d6e5c3615caf10e32a9d6fc6ff683f5b5814948928cb84c3ab91c038b6L770 The `config` and `metadata` Click options are the wrong way round: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L785-L786 https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/tests/fixtures.py#L801","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2194#issuecomment-1730242734,https://api.github.com/repos/simonw/datasette/issues/2194,1730242734,IC_kwDOBm6k_c5nIWiu,9599,simonw,2023-09-21T20:19:29Z,2023-09-21T20:19:29Z,OWNER,"Maybe `plugins/` does not exist? It should have been created by this line: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L41-L42","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2194#issuecomment-1730241813,https://api.github.com/repos/simonw/datasette/issues/2194,1730241813,IC_kwDOBm6k_c5nIWUV,9599,simonw,2023-09-21T20:18:40Z,2023-09-21T20:18:40Z,OWNER,"This looks to be the step that is failing: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/.github/workflows/deploy-latest.yml#L50-L60","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907695234,"Deploy failing with ""plugins/alternative_route.py: Not a directory""", https://github.com/simonw/datasette/issues/2193#issuecomment-1730353006,https://api.github.com/repos/simonw/datasette/issues/2193,1730353006,IC_kwDOBm6k_c5nIxdu,9599,simonw,2023-09-21T21:56:43Z,2023-09-21T21:56:43Z,OWNER,"The test fails as expected now. Closing this issue, will solve the remaining problems in: - #2057","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run", https://github.com/simonw/datasette/issues/2193#issuecomment-1730352111,https://api.github.com/repos/simonw/datasette/issues/2193,1730352111,IC_kwDOBm6k_c5nIxPv,9599,simonw,2023-09-21T21:55:41Z,2023-09-21T21:55:41Z,OWNER,"https://github.com/simonw/datasette/actions/runs/6267146158/job/17019594849 failed on 3.9 this time. ``` plugin_info[""name""] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run", https://github.com/simonw/datasette/issues/2193#issuecomment-1730208566,https://api.github.com/repos/simonw/datasette/issues/2193,1730208566,IC_kwDOBm6k_c5nIOM2,9599,simonw,2023-09-21T19:55:19Z,2023-09-21T19:55:19Z,OWNER,"Yes, the new script seems to work. On Python 3.11: ``` tests/test-datasette-load-plugins.sh echo $? 0 ``` On Python 3.8: ``` tests/test-datasette-load-plugins.sh Test failed: datasette-json-html not found echo $? 1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run", https://github.com/simonw/datasette/issues/2193#issuecomment-1730206629,https://api.github.com/repos/simonw/datasette/issues/2193,1730206629,IC_kwDOBm6k_c5nINul,9599,simonw,2023-09-21T19:53:39Z,2023-09-21T19:53:39Z,OWNER,"[GPT-4 says](https://chat.openai.com/share/65a95561-8f3c-4f9b-a4ab-c2508e87d2a9): > In the script, you're using a subshell `( ... )` to group commands. If you `exit 1` within the subshell, it will only exit the subshell and not the main script. This is why GitHub Actions does not see it as a failure. It suggested doing this instead: ```bash #!/bin/bash PLUGINS=$(datasette plugins) if ! echo ""$PLUGINS"" | jq 'any(.[]; .name == ""datasette-json-html"")' | grep -q true; then echo ""Test failed: datasette-json-html not found"" exit 1 fi PLUGINS2=$(DATASETTE_LOAD_PLUGINS=datasette-init datasette plugins) if ! echo ""$PLUGINS2"" | jq 'any(.[]; .name == ""datasette-json-html"")' | grep -q false; then echo ""Test failed: datasette-json-html should not have been loaded"" exit 1 fi if ! echo ""$PLUGINS2"" | jq 'any(.[]; .name == ""datasette-init"")' | grep -q true; then echo ""Test failed: datasette-init should have been loaded"" exit 1 fi PLUGINS3=$(DATASETTE_LOAD_PLUGINS='' datasette plugins) if ! echo ""$PLUGINS3"" | grep -q '\[\]'; then echo ""Test failed: datasette plugins should have returned []"" exit 1 fi ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run", https://github.com/simonw/datasette/issues/2193#issuecomment-1730203356,https://api.github.com/repos/simonw/datasette/issues/2193,1730203356,IC_kwDOBm6k_c5nIM7c,9599,simonw,2023-09-21T19:51:04Z,2023-09-21T19:51:04Z,OWNER,"The script: https://github.com/simonw/datasette/blob/b0d0a0e5de8bb5b9b6c253e8af451a532266bcf1/tests/test-datasette-load-plugins.sh#L1-L29 I'm not sure why those `exit 1` lines did not cause a CI failure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run", https://github.com/simonw/datasette/issues/2193#issuecomment-1730202533,https://api.github.com/repos/simonw/datasette/issues/2193,1730202533,IC_kwDOBm6k_c5nIMul,9599,simonw,2023-09-21T19:50:22Z,2023-09-21T19:50:22Z,OWNER,"Here's the failure in CI, which did not cause the workflow to fail even though it should have: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907655261,"""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run", https://github.com/simonw/sqlite-utils/issues/595#issuecomment-1733312349,https://api.github.com/repos/simonw/sqlite-utils/issues/595,1733312349,IC_kwDOCGYnMM5nUD9d,123451970,cycle-data,2023-09-25T09:38:13Z,2023-09-25T09:38:57Z,NONE,"Never mind When I created the connection using `sqlite_utils.Database(path)` I just needed to add the following statement right after and it did the trick `self.db.conn.execute(""PRAGMA foreign_keys = ON"")` Hope this helps people in the future 👍 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1907281675,Cascading DELETE not working with Table.delete(pk), https://github.com/simonw/datasette/pull/2192#issuecomment-1726754119,https://api.github.com/repos/simonw/datasette/issues/2192,1726754119,IC_kwDOBm6k_c5m7C1H,22429695,codecov[bot],2023-09-20T01:35:45Z,2023-09-20T01:35:45Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00%`** and project coverage change: **`+0.02%`** :tada: > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`4e6a341`)](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.72%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2192 +/- ## ========================================== + Coverage 92.69% 92.72% +0.02% ========================================== Files 40 40 Lines 6039 6036 -3 ========================================== - Hits 5598 5597 -1 + Misses 441 439 -2 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/table.py](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.31% <100.00%> (+0.51%)` | :arrow_up: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2192/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2192?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1903932086,Stop using parallel SQL queries for tables, https://github.com/simonw/datasette/pull/2191#issuecomment-1728504633,https://api.github.com/repos/simonw/datasette/issues/2191,1728504633,IC_kwDOBm6k_c5nBuM5,9599,simonw,2023-09-20T22:24:51Z,2023-09-20T22:25:16Z,OWNER,"The `{""units"": {""distance"": ""m"", ""frequency"": ""Hz""}}` bit is for the [units feature](https://docs.datasette.io/en/1.0a6/metadata.html#specifying-units-for-a-column) which I've half-disabled already and would like to remove before 1.0, though ideally turning that functionality into a plugin instead (if I can figure out how to do that).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`, `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`", https://github.com/simonw/datasette/pull/2191#issuecomment-1728503623,https://api.github.com/repos/simonw/datasette/issues/2191,1728503623,IC_kwDOBm6k_c5nBt9H,9599,simonw,2023-09-20T22:23:33Z,2023-09-20T22:24:10Z,OWNER,"This is one of the most interesting illustrative examples in the new code: https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L301-L349 Interesting to note that it now has canned queries in it, which include this bit: https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L341-L342 It looks like metadata, but in this case it's configuration. That blur between metadata and configuration at the canned query level still feels a little bit odd to me, but I still think we're going in the right direction with it. Also interesting, from that same file: https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L351-L399 There are a few things in that metadata block that are arguably configuration, not metadata - for example: https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L360 I think `extra_css_urls` is definitely configuration, not metadata. https://github.com/simonw/datasette/blob/f7bdedff779606466b580d8528e5a44509291002/tests/fixtures.py#L369-L395 Most of that stuff is arguably configuration too, with the exception of the `roadside_attractions.columns` bit which is metadata about those columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`, `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`", https://github.com/simonw/datasette/pull/2191#issuecomment-1728498221,https://api.github.com/repos/simonw/datasette/issues/2191,1728498221,IC_kwDOBm6k_c5nBsot,9599,simonw,2023-09-20T22:17:26Z,2023-09-20T22:17:26Z,OWNER,"I tested this locally for permissions like this. `datasette.yml`: ```yaml databases: content: allow: id: root ``` Started Datasette like this: ```bash datasette --root content.db pottery2.db -c datasette.yml ``` As root I could see this (note the padlock): http://127.0.0.1:8001/-/metadata returned `{}` showing that the permissions must have come from the config file instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`, `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`", https://github.com/simonw/datasette/pull/2191#issuecomment-1724480716,https://api.github.com/repos/simonw/datasette/issues/2191,1724480716,IC_kwDOBm6k_c5myXzM,22429695,codecov[bot],2023-09-18T21:28:36Z,2023-10-12T16:15:40Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified lines are covered by tests :white_check_mark: > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`0135e7c`)](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%. > Report is 14 commits behind head on main. > :exclamation: Current head 0135e7c differs from pull request most recent head 18b48f8. Consider uploading reports for the commit 18b48f8 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2191 +/- ## ========================================== - Coverage 92.69% 92.68% -0.02% ========================================== Files 40 40 Lines 6039 6042 +3 ========================================== + Hits 5598 5600 +2 - Misses 441 442 +1 ``` | [Files](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.09% <100.00%> (-0.11%)` | :arrow_down: | | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.36% <100.00%> (+0.01%)` | :arrow_up: | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2191/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2191?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901768721,"Move `permissions`, `allow` blocks, canned queries and more out of `metadata.yaml` and into `datasette.yaml`", https://github.com/simonw/datasette/pull/2190#issuecomment-1759948683,https://api.github.com/repos/simonw/datasette/issues/2190,1759948683,IC_kwDOBm6k_c5o5q-L,9599,simonw,2023-10-12T16:20:41Z,2023-10-12T16:20:41Z,OWNER,I'm going to land this and open a new issue for the upgrade instructions.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json", https://github.com/simonw/datasette/pull/2190#issuecomment-1759947534,https://api.github.com/repos/simonw/datasette/issues/2190,1759947534,IC_kwDOBm6k_c5o5qsO,9599,simonw,2023-10-12T16:19:59Z,2023-10-12T16:19:59Z,OWNER,It would be nice if we could catch that and turn that into a less intimidating Click exception too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json", https://github.com/simonw/datasette/pull/2190#issuecomment-1759947021,https://api.github.com/repos/simonw/datasette/issues/2190,1759947021,IC_kwDOBm6k_c5o5qkN,9599,simonw,2023-10-12T16:19:38Z,2023-10-12T16:19:38Z,OWNER," This looks good and works well. The error from this currently looks like: ``` datasette -m metadata.json -p 8844 Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 33, in sys.exit(load_entry_point('datasette', 'console_scripts', 'datasette')()) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1130, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1055, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages/click/core.py"", line 760, in invoke return __callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 98, in wrapped return fn(*args, **kwargs) File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 546, in serve metadata_data = fail_if_plugins_in_metadata(parse_metadata(metadata.read())) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 1282, in fail_if_plugins_in_metadata raise Exception( Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details. ``` With wrapping: `Exception: Datasette no longer accepts plugin configuration in --metadata. Move your ""plugins"" configuration blocks to a separate file - we suggest calling that datasette..json - and start Datasette with datasette -c datasette..json. See https://docs.datasette.io/en/latest/configuration.html for more details.` I think we should link directly to documentation that tells people how to perform this upgrade.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json", https://github.com/simonw/datasette/pull/2190#issuecomment-1729961503,https://api.github.com/repos/simonw/datasette/issues/2190,1729961503,IC_kwDOBm6k_c5nHR4f,15178711,asg017,2023-09-21T16:56:57Z,2023-09-21T16:56:57Z,CONTRIBUTOR,TODO: add similar checks for permissions/allow/canned queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json", https://github.com/simonw/datasette/pull/2190#issuecomment-1724169693,https://api.github.com/repos/simonw/datasette/issues/2190,1724169693,IC_kwDOBm6k_c5mxL3d,22429695,codecov[bot],2023-09-18T18:39:19Z,2023-09-18T18:39:19Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning: > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fc7dbe0`)](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.67%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2190 +/- ## ========================================== - Coverage 92.69% 92.67% -0.03% ========================================== Files 40 40 Lines 6039 6044 +5 ========================================== + Hits 5598 5601 +3 - Misses 441 443 +2 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <100.00%> (ø)` | | | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (ø)` | | | [datasette/utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.83% <100.00%> (+0.03%)` | :arrow_up: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2190/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2190?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901483874,"Raise an exception if a ""plugins"" block exists in metadata.json", https://github.com/simonw/datasette/issues/2189#issuecomment-1730388418,https://api.github.com/repos/simonw/datasette/issues/2189,1730388418,IC_kwDOBm6k_c5nI6HC,9599,simonw,2023-09-21T22:26:19Z,2023-09-21T22:26:19Z,OWNER,1.0a7 is out with this fix as well now: https://docs.datasette.io/en/1.0a7/changelog.html#a7-2023-09-21,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1730232308,https://api.github.com/repos/simonw/datasette/issues/2189,1730232308,IC_kwDOBm6k_c5nIT_0,9599,simonw,2023-09-21T20:11:16Z,2023-09-21T20:11:16Z,OWNER,"We're planning a breaking change in `1.0a7`: - #2191 Since that's a breaking change I'm going to ship 1.0a7 right now with this fix, then ship that breaking change as `1.0a8` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1730231404,https://api.github.com/repos/simonw/datasette/issues/2189,1730231404,IC_kwDOBm6k_c5nITxs,9599,simonw,2023-09-21T20:10:28Z,2023-09-21T20:10:28Z,OWNER,Release 0.64.4: https://docs.datasette.io/en/stable/changelog.html#v0-64-4,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1730162283,https://api.github.com/repos/simonw/datasette/issues/2189,1730162283,IC_kwDOBm6k_c5nIC5r,9599,simonw,2023-09-21T19:19:47Z,2023-09-21T19:19:47Z,OWNER,"I'm going to release this in `1.0a7`, and I'll backport it to a `0.64.4` release too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1728192688,https://api.github.com/repos/simonw/datasette/issues/2189,1728192688,IC_kwDOBm6k_c5nAiCw,173848,yozlet,2023-09-20T17:53:31Z,2023-09-20T17:53:31Z,NONE,"`/me munches popcorn at a furious rate, utterly entralled`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1726749355,https://api.github.com/repos/simonw/datasette/issues/2189,1726749355,IC_kwDOBm6k_c5m7Bqr,9599,simonw,2023-09-20T01:28:16Z,2023-09-20T01:28:16Z,OWNER,Added a note to that example in the documentation: https://github.com/simonw/datasette/blob/4e6a34179eaedec44c1263275d7592fd83d7e2ac/docs/internals.rst?plain=1#L1320,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724325068,https://api.github.com/repos/simonw/datasette/issues/2189,1724325068,IC_kwDOBm6k_c5mxxzM,9599,simonw,2023-09-18T20:29:41Z,2023-09-18T20:29:41Z,OWNER,"The one other thing affected by this change is this documentation, which suggests a not-actually-safe pattern: https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/docs/internals.rst#L1292-L1321","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724317367,https://api.github.com/repos/simonw/datasette/issues/2189,1724317367,IC_kwDOBm6k_c5mxv63,9599,simonw,2023-09-18T20:25:44Z,2023-09-18T20:25:44Z,OWNER,"My current hunch is that SQLite gets unhappy if multiple threads access the same underlying C object - which sometimes happens with in-memory connections and Datasette presumably because they are faster than file-backed databases. I'm going to remove the `asyncio.gather()` code from the table view. I'll ship a 0.x release with that fix too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724315591,https://api.github.com/repos/simonw/datasette/issues/2189,1724315591,IC_kwDOBm6k_c5mxvfH,9599,simonw,2023-09-18T20:24:30Z,2023-09-18T20:24:30Z,OWNER,"[Using SQLite In Multi-Threaded Applications](https://www.sqlite.org/threadsafe.html) That indicates that there's a SQLite option for ""Serialized"" mode where it's safe to access anything SQLite provides from multiple threads, but as far as I can tell Python doesn't give you an option to turn that mode on or off for a connection - you can read `sqlite3.threadsafet`y to see if that mode was compiled in or not, but not actually change it. On my Mac `sqlite3.threadsafety` returns 1 which means https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety ""Multi-thread: In this mode, SQLite can be safely used by multiple threads provided that no single database connection is used simultaneously in two or more threads."" - it would need to return 3 for that serialized mode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724305169,https://api.github.com/repos/simonw/datasette/issues/2189,1724305169,IC_kwDOBm6k_c5mxs8R,9599,simonw,2023-09-18T20:16:22Z,2023-09-18T20:16:36Z,OWNER,"Looking again at this code: https://github.com/simonw/datasette/blob/6ed7908580fa2ba9297c3225d85c56f8b08b9937/datasette/database.py#L87-L117 `check_same_thread=False` really stands out here. Python docs at https://docs.python.org/3/library/sqlite3.html > **check_same_thread** ([*bool*](https://docs.python.org/3/library/functions.html#bool ""bool"")) -- If `True` (default), [`ProgrammingError`](https://docs.python.org/3/library/sqlite3.html#sqlite3.ProgrammingError ""sqlite3.ProgrammingError"") will be raised if the database connection is used by a thread other than the one that created it. If `False`, the connection may be accessed in multiple threads; write operations may need to be serialized by the user to avoid data corruption. See [`threadsafety`](https://docs.python.org/3/library/sqlite3.html#sqlite3.threadsafety ""sqlite3.threadsafety"") for more information. I think I'm playing with fire by allowing multiple threads to access the same connection without doing my own serialization of those requests. I _do_ do that using the write connection - and in this particular case the bug isn't coming from write queries, it's coming from read queries - but perhaps SQLite has issues with threading for reads, too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724298817,https://api.github.com/repos/simonw/datasette/issues/2189,1724298817,IC_kwDOBm6k_c5mxrZB,9599,simonw,2023-09-18T20:11:26Z,2023-09-18T20:11:26Z,OWNER,"Now that I've confirmed that parallel query execution of the kind introduced in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 can cause hangs (presumably some kind of locking issue) against in-memory databases, some options: 1. Disable parallel execution entirely and rip out related code. 2. Disable parallel execution entirely by leaving that code but having it always behave as if `_noparallel=1` 3. Continue digging and try and find some way to avoid this problem The parallel execution work is something I was playing with last year in the hope of speeding up Datasette pages like the table page which need to execute a bunch of queries - one for each facet, plus one for each column to see if it should be suggested as a facet. I wrote about this at the time here: https://simonwillison.net/2022/May/6/weeknotes/ My hope was that despite Python's GIL this optimization would still help, because the SQLite C module releases the GIL once it gets to SQLite. But... that didn't hold up. It looked like enough work was happening in Python land with the GIL that the optimization didn't improve things. Running the `nogil` fork of Python DID improve things though! I left the code in partly on the hope that the `nogil` fork would be accepted into Python core. ... which it now has! But it will still be a year or two before it fully lands: https://discuss.python.org/t/a-steering-council-notice-about-pep-703-making-the-global-interpreter-lock-optional-in-cpython/30474 So I'm not particularly concerned about dropping the parallel execution. If I do drop it though do I leave the potentially complex code in that relates to it?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724281824,https://api.github.com/repos/simonw/datasette/issues/2189,1724281824,IC_kwDOBm6k_c5mxnPg,9599,simonw,2023-09-18T19:58:06Z,2023-09-18T19:58:06Z,OWNER,I also confirmed that `http://127.0.0.1:8064/airtable_refs/airtable_refs?_noparallel=1` does not trigger the bug but `http://127.0.0.1:8064/airtable_refs/airtable_refs` does.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724278386,https://api.github.com/repos/simonw/datasette/issues/2189,1724278386,IC_kwDOBm6k_c5mxmZy,9599,simonw,2023-09-18T19:55:32Z,2023-09-18T19:55:32Z,OWNER,"OK it looks like it found it! ``` 942411ef946e9a34a2094944d3423cddad27efd3 is the first bad commit commit Author: Simon Willison Date: Tue Apr 26 15:48:56 2022 -0700 Execute some TableView queries in parallel Use ?_noparallel=1 to opt out (undocumented, useful for benchmark comparisons) Refs #1723, #1715 datasette/views/table.py | 93 ++++++++++++++++++++++++++++++++++-------------- 1 file changed, 67 insertions(+), 26 deletions(-) bisect found first bad commit ``` https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 does look like the cause of this problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724276917,https://api.github.com/repos/simonw/datasette/issues/2189,1724276917,IC_kwDOBm6k_c5mxmC1,9599,simonw,2023-09-18T19:54:23Z,2023-09-18T19:54:23Z,OWNER,"Turned out I wasn't running the `datasette` from the current directory, so it was not testing what I intended. FIxed that with `pip install -e .` in the `datasette/` directory. Now I'm seeing some passes, which look like this: ``` running '../airtable-export/testit.sh' INFO: Started server process [77810] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls INFO: 127.0.0.1:59439 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK INFO: 127.0.0.1:59440 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK INFO: 127.0.0.1:59441 - ""GET /airtable_refs/airtable_refs HTTP/1.1"" 200 OK All curl succeeded Killing datasette server with PID 77810 ../airtable-export/testit.sh: line 54: 77810 Killed: 9 datasette pottery2.db -p $port All three curls succeeded. Bisecting: 4 revisions left to test after this (roughly 2 steps) [7463b051cf8d7f856df5eba9f7aa944183ebabe5] Cosmetic tweaks after blacken-docs, refs #1718 running '../airtable-export/testit.sh' INFO: Started server process [77826] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724263390,https://api.github.com/repos/simonw/datasette/issues/2189,1724263390,IC_kwDOBm6k_c5mxive,9599,simonw,2023-09-18T19:44:03Z,2023-09-18T19:44:03Z,OWNER,I knocked it down to 1 retry just to see what happened.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724259229,https://api.github.com/repos/simonw/datasette/issues/2189,1724259229,IC_kwDOBm6k_c5mxhud,9599,simonw,2023-09-18T19:40:56Z,2023-09-18T19:40:56Z,OWNER,I tried it with a path of `/` and everything passed - so it's definitely the path of `/airtable_refs/airtable_refs` (an in-memory database created by an experimental branch of https://github.com/simonw/airtable-export) that triggers the problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724258279,https://api.github.com/repos/simonw/datasette/issues/2189,1724258279,IC_kwDOBm6k_c5mxhfn,9599,simonw,2023-09-18T19:40:13Z,2023-09-18T19:40:13Z,OWNER,"Output while it is running looks like this: ``` running '../airtable-export/testit.sh' INFO: Started server process [75649] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls Running curls Running curls Killing datasette server with PID 75649 ../airtable-export/testit.sh: line 54: 75649 Killed: 9 datasette pottery2.db -p $port Error: Not all curls succeeded after 3 attempts. Bisecting: 155 revisions left to test after this (roughly 7 steps) [247e460e08bf823142f7b84058fe44e43626787f] Update beautifulsoup4 requirement (#1703) running '../airtable-export/testit.sh' INFO: Started server process [75722] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls Running curls Running curls Killing datasette server with PID 75722 ../airtable-export/testit.sh: line 54: 75722 Killed: 9 datasette pottery2.db -p $port Error: Not all curls succeeded after 3 attempts. Bisecting: 77 revisions left to test after this (roughly 6 steps) [3ef47a0896c7e63404a34e465b7160c80eaa571d] Link rel=alternate header for tables and rows running '../airtable-export/testit.sh' INFO: Started server process [75818] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8064 (Press CTRL+C to quit) Running curls ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724257290,https://api.github.com/repos/simonw/datasette/issues/2189,1724257290,IC_kwDOBm6k_c5mxhQK,9599,simonw,2023-09-18T19:39:27Z,2023-09-18T19:44:26Z,OWNER,"I'm now trying this test script: ```bash #!/bin/bash port=8064 # Start datasette server in the background and get its PID datasette pottery2.db -p $port & server_pid=$! # Wait for a moment to ensure the server has time to start up sleep 2 # Initialize counters and parameters retry_count=0 max_retries=3 success_count=0 path=""/airtable_refs/airtable_refs"" # Function to run curl with a timeout function test_curl { # Run the curl command with a timeout of 3 seconds timeout 3s curl -s ""http://localhost:${port}${path}"" > /dev/null if [ $? -eq 0 ]; then # Curl was successful ((success_count++)) fi } # Try three parallel curl requests while [[ $retry_count -lt $max_retries ]]; do # Reset the success counter success_count=0 # Run the curls in parallel echo "" Running curls"" test_curl test_curl test_curl # & test_curl & test_curl & # Wait for all curls to finish #wait # Check the success count if [[ $success_count -eq 3 ]]; then # All curls succeeded, break out of the loop echo "" All curl succeeded"" break fi ((retry_count++)) done # Kill the datasette server echo ""Killing datasette server with PID $server_pid"" kill -9 $server_pid sleep 2 # Print result if [[ $success_count -eq 3 ]]; then echo ""All three curls succeeded."" exit 0 else echo ""Error: Not all curls succeeded after $retry_count attempts."" exit 1 fi ``` I run it like this: ```bash git bisect reset git bisect start git bisect good 0.59.4 git bisect bad 1.0a6 git bisect run ../airtable-export/testit.sh ``` But... it's not having the desired result, I think because the bug is intermittent so each time I run it the bisect spits out a different commit as the one that is to blame.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724159882,https://api.github.com/repos/simonw/datasette/issues/2189,1724159882,IC_kwDOBm6k_c5mxJeK,9599,simonw,2023-09-18T18:32:29Z,2023-09-18T18:32:29Z,OWNER,"This worked, including on macOS even though GPT-4 thought `timeout` would not work there: https://chat.openai.com/share/cc4628e9-5240-4f35-b640-16a9c178b315 ```bash #!/bin/bash # Run the command with a timeout of 5 seconds timeout 5s datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs # Check the exit code from timeout if [ $? -eq 124 ]; then echo ""Error: Command timed out after 5 seconds."" exit 1 fi ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724157182,https://api.github.com/repos/simonw/datasette/issues/2189,1724157182,IC_kwDOBm6k_c5mxIz-,9599,simonw,2023-09-18T18:30:30Z,2023-09-18T18:30:30Z,OWNER,"OK, I can trigger the bug like this: ```bash datasette pottery2.db -p 8045 --get /airtable_refs/airtable_refs ``` Can I write a bash script that fails (and terminates the process) if it takes longer than X seconds?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724089666,https://api.github.com/repos/simonw/datasette/issues/2189,1724089666,IC_kwDOBm6k_c5mw4VC,9599,simonw,2023-09-18T17:49:24Z,2023-09-18T17:49:24Z,OWNER,I switched that particular implementation to using an on-disk database instead of an in-memory database and could no longer recreate the bug.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724084199,https://api.github.com/repos/simonw/datasette/issues/2189,1724084199,IC_kwDOBm6k_c5mw2_n,9599,simonw,2023-09-18T17:47:01Z,2023-09-18T17:47:01Z,OWNER,I managed to trigger it by loading `http://127.0.0.1:8045/airtable_refs/airtable_refs` - which worked - and then hitting refresh on that page a bunch of times until it hung.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724083324,https://api.github.com/repos/simonw/datasette/issues/2189,1724083324,IC_kwDOBm6k_c5mw2x8,9599,simonw,2023-09-18T17:46:21Z,2023-09-18T17:46:21Z,OWNER,"Sometimes it takes a few clicks for the bug to occur, but it does seem to always be within the in-memory database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724081909,https://api.github.com/repos/simonw/datasette/issues/2189,1724081909,IC_kwDOBm6k_c5mw2b1,9599,simonw,2023-09-18T17:45:27Z,2023-09-18T17:45:27Z,OWNER,Maybe it's not related to faceting - I just got it on a hit to `http://127.0.0.1:8045/airtable_refs/airtable_refs` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724072390,https://api.github.com/repos/simonw/datasette/issues/2189,1724072390,IC_kwDOBm6k_c5mw0HG,9599,simonw,2023-09-18T17:39:06Z,2023-09-18T17:39:06Z,OWNER,Landing a version of that test anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724064440,https://api.github.com/repos/simonw/datasette/issues/2189,1724064440,IC_kwDOBm6k_c5mwyK4,9599,simonw,2023-09-18T17:36:00Z,2023-09-18T17:36:00Z,OWNER,"I wrote this test, but it passes: ```python @pytest.mark.asyncio async def test_facet_against_in_memory_database(): ds = Datasette() db = ds.add_memory_database(""mem"") await db.execute_write(""create table t (id integer primary key, name text)"") await db.execute_write_many( ""insert into t (name) values (?)"", [[""one""], [""one""], [""two""]] ) response1 = await ds.client.get(""/mem/t.json"") assert response1.status_code == 200 response2 = await ds.client.get(""/mem/t.json?_facet=name"") assert response2.status_code == 200 assert response2.json() == { ""ok"": True, ""next"": None, ""facet_results"": { ""results"": { ""name"": { ""name"": ""name"", ""type"": ""column"", ""hideable"": True, ""toggle_url"": ""/mem/t.json"", ""results"": [ { ""value"": ""one"", ""label"": ""one"", ""count"": 2, ""toggle_url"": ""http://localhost/mem/t.json?_facet=name&name=one"", ""selected"": False, }, { ""value"": ""two"", ""label"": ""two"", ""count"": 1, ""toggle_url"": ""http://localhost/mem/t.json?_facet=name&name=two"", ""selected"": False, }, ], ""truncated"": False, } }, ""timed_out"": [], }, ""rows"": [ {""id"": 1, ""name"": ""one""}, {""id"": 2, ""name"": ""one""}, {""id"": 3, ""name"": ""two""}, ], ""truncated"": False, } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724055823,https://api.github.com/repos/simonw/datasette/issues/2189,1724055823,IC_kwDOBm6k_c5mwwEP,9599,simonw,2023-09-18T17:31:10Z,2023-09-18T17:31:10Z,OWNER,That line was added in https://github.com/simonw/datasette/commit/942411ef946e9a34a2094944d3423cddad27efd3 which first shipped in 0.62a0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724051886,https://api.github.com/repos/simonw/datasette/issues/2189,1724051886,IC_kwDOBm6k_c5mwvGu,9599,simonw,2023-09-18T17:28:20Z,2023-09-18T17:30:30Z,OWNER,"The bug exhibits when I try to add a facet. I think it's caused by the parallel query execution I added to facets at some point. http://127.0.0.1:8045/airtable_refs/airtable_refs - no error http://127.0.0.1:8045/airtable_refs/airtable_refs?_facet=table_name#facet-table_name - hangs the server Crucial line in the traceback: ``` await gather(execute_facets(), execute_suggested_facets()) ``` From here: https://github.com/simonw/datasette/blob/917272c864ad7b8a00c48c77f5c2944093babb4e/datasette/views/table.py#L568","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724049538,https://api.github.com/repos/simonw/datasette/issues/2189,1724049538,IC_kwDOBm6k_c5mwuiC,9599,simonw,2023-09-18T17:26:44Z,2023-09-18T17:26:44Z,OWNER,"Just managed to get this exception trace: ``` return await self.route_path(scope, receive, send, path) File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/app.py"", line 1354, in route_path response = await view(request, send) File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 134, in view return await self.dispatch_request(request) File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 91, in dispatch_request return await handler(request) File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/base.py"", line 361, in get response_or_template_contexts = await self.data(request, **data_kwargs) File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 158, in data return await self._data_traced(request, default_labels, _next, _size) File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 568, in _data_traced await gather(execute_facets(), execute_suggested_facets()) File ""/Users/simon/.local/share/virtualenvs/airtable-export-Ca4U-3qk/lib/python3.8/site-packages/datasette/views/table.py"", line 177, in _gather_parallel return await asyncio.gather(*args) asyncio.exceptions.CancelledError INFO: 127.0.0.1:64109 - ""GET /airtable_refs/airtable_refs?_facet=table_name&table_name=Sessions HTTP/1.1"" 500 Internal Server Error ^CError in atexit._run_exitfuncs: Traceback (most recent call last): File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/concurrent/futures/thread.py"", line 40, in _python_exit t.join() File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py"", line 1011, in join self._wait_for_tstate_lock() File ""/Users/simon/.pyenv/versions/3.8.17/lib/python3.8/threading.py"", line 1027, in _wait_for_tstate_lock elif lock.acquire(block, timeout): KeyboardInterrupt ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724048314,https://api.github.com/repos/simonw/datasette/issues/2189,1724048314,IC_kwDOBm6k_c5mwuO6,9599,simonw,2023-09-18T17:25:55Z,2023-09-18T17:25:55Z,OWNER,"The good news is that this bug is currently unlikely to affect most users since named in-memory databases (created using `datasette.add_memory_database(""airtable_refs"")` ([docs](https://docs.datasette.io/en/stable/internals.html#add-memory-database-name)) are a pretty obscure feature, only available to plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2189#issuecomment-1724045748,https://api.github.com/repos/simonw/datasette/issues/2189,1724045748,IC_kwDOBm6k_c5mwtm0,9599,simonw,2023-09-18T17:24:07Z,2023-09-18T17:24:07Z,OWNER,"I need reliable steps to reproduce, then I can bisect and figure out which exact version of Datasette introduced the problem. I have a hunch that it relates to changes made to the `datasette/database.py` module, maybe one of these changes here: https://github.com/simonw/datasette/compare/0.61...0.63.1#diff-4e20309c969326a0008dc9237f6807f48d55783315fbfc1e7dfa480b550e16f9","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1901416155,Server hang on parallel execution of queries to named in-memory databases, https://github.com/simonw/datasette/issues/2188#issuecomment-1722848454,https://api.github.com/repos/simonw/datasette/issues/2188,1722848454,IC_kwDOBm6k_c5msJTG,15178711,asg017,2023-09-18T06:58:53Z,2023-09-18T06:58:53Z,CONTRIBUTOR,"Thinking about this more, here a list of things I imagine a ""compile-to-sql"" plugin would want to do: 1. Attach itself to the SQL code editor (switch from SQL -> PRQL/Logica, additional syntax highlighting) 2. Add ""Query using PRQL"" buttons in various parts of Datasette's UI, like `/dbname` page 3. Use `$LANGUAGE=` instead of `sql=` in the JSON API and the SQL results pages 4. Have their own dedicated code editor page 1) and 2) would be difficult to do with current plugin hooks, unless we add the concept of ""slots"" and get the JS plugin support in. 3) could maybe be done with the [`asgi_wrapper(datasette)`](https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette) hook? And 4) ca n be done easily with the `register_routes()` hooks. So it really only sounds like extending the SQL editor will be the hard part. In #2094 I want to add JavaScript plugin hooks for extending the SQL editor, which may work here. If I get the time/motivation, I might try out a `datasette-prql` extension, just because I like playing with it. It'd be really cool if I can get the `asgi_wrapper()` hook to work right there...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1900026059,"Plugin Hooks for ""compile to SQL"" languages", https://github.com/simonw/datasette/issues/2188#issuecomment-1722662413,https://api.github.com/repos/simonw/datasette/issues/2188,1722662413,IC_kwDOBm6k_c5mrb4N,9599,simonw,2023-09-18T02:01:34Z,2023-09-18T02:01:34Z,OWNER,"I'm not interested in these in Datasette core itself, but I think they have a ton of potential for plugins. I wonder what the best way to handle that would be? Right now it's possible to write a plugin that [adds extra routes](https://docs.datasette.io/en/stable/plugin_hooks.html#register-routes-datasette), so someone could build a `/dbname/-/prql?query=xxx` endpoint. If this could return JSON, they could add JavaScript to the `/dbname` page that provided a UI for kicking off one of those queries. Something that could make that more ergonomic might be the plugin hook that allows plugins to add extra HTML to different core database pages - e.g. adding a ""Query this database using PRQL"" button or link or even a full form at the top of that database page. That's this issue here: - #1191","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1900026059,"Plugin Hooks for ""compile to SQL"" languages", https://github.com/simonw/datasette/issues/2186#issuecomment-1721742055,https://api.github.com/repos/simonw/datasette/issues/2186,1721742055,IC_kwDOBm6k_c5mn7Ln,9599,simonw,2023-09-15T19:27:59Z,2023-09-15T19:27:59Z,OWNER,This feels like it might be quite a nice pattern generally - providing optional arguments to plugins and views that can be `await get_x()` called to run an extra calculation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1898927976,Mechanism for register_output_renderer hooks to access full count, https://github.com/simonw/datasette/issues/2186#issuecomment-1721740872,https://api.github.com/repos/simonw/datasette/issues/2186,1721740872,IC_kwDOBm6k_c5mn65I,9599,simonw,2023-09-15T19:26:51Z,2023-09-15T19:27:19Z,OWNER,"Here's where it's called at the moment: https://github.com/simonw/datasette/blob/16f0b6d8222d06682a31b904d0a402c391ae1c1c/datasette/views/base.py#L297-L313 And the docs: https://github.com/simonw/datasette/blob/1.0a6/docs/plugin_hooks.rst#register-output-renderer-datasette I'm tempted to add a `get_count` argument which, when awaited, returns the full count. Then plugins could do this: ```python async def render_notebook(datasette, request, get_count, rows): count = await get_count() # ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1898927976,Mechanism for register_output_renderer hooks to access full count, https://github.com/simonw/datasette/pull/2185#issuecomment-1719468727,https://api.github.com/repos/simonw/datasette/issues/2185,1719468727,IC_kwDOBm6k_c5mfQK3,22429695,codecov[bot],2023-09-14T13:36:07Z,2023-09-19T13:42:35Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage has no change and project coverage change: **`-0.04%`** :warning: > Comparison is base [(`6ed7908`)](https://app.codecov.io/gh/simonw/datasette/commit/6ed7908580fa2ba9297c3225d85c56f8b08b9937?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`fe5f881`)](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2185 +/- ## ========================================== - Coverage 92.69% 92.66% -0.04% ========================================== Files 40 40 Lines 6039 6039 ========================================== - Hits 5598 5596 -2 - Misses 441 443 +2 ``` [see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2185/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2185?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1896578249,Bump the python-packages group with 3 updates, https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714920708,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714920708,IC_kwDOCGYnMM5mN50E,9599,simonw,2023-09-12T03:51:13Z,2023-09-12T03:51:13Z,OWNER,"Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150 I could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works: ```python for table, column, other_table, other_column in table.foreign_keys: # ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,Represent compound foreign keys in table.foreign_keys output, https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714919806,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714919806,IC_kwDOCGYnMM5mN5l-,9599,simonw,2023-09-12T03:49:41Z,2023-09-12T03:49:41Z,OWNER,"Digging in a bit more: ```pycon >>> pprint(list(db.query('PRAGMA foreign_key_list(courses)'))) [{'from': 'campus_name', 'id': 0, 'match': 'NONE', 'on_delete': 'NO ACTION', 'on_update': 'NO ACTION', 'seq': 0, 'table': 'departments', 'to': 'campus_name'}, {'from': 'dept_code', 'id': 0, 'match': 'NONE', 'on_delete': 'NO ACTION', 'on_update': 'NO ACTION', 'seq': 1, 'table': 'departments', 'to': 'dept_code'}] ``` I think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`. Right now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971,Represent compound foreign keys in table.foreign_keys output, https://github.com/simonw/datasette/pull/2183#issuecomment-1718316733,https://api.github.com/repos/simonw/datasette/issues/2183,1718316733,IC_kwDOBm6k_c5ma269,9599,simonw,2023-09-13T21:05:36Z,2023-09-13T21:05:36Z,OWNER,I'm going to land this and make any further documentation tweaks on `main`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support, https://github.com/simonw/datasette/pull/2183#issuecomment-1716801971,https://api.github.com/repos/simonw/datasette/issues/2183,1716801971,IC_kwDOBm6k_c5mVFGz,15178711,asg017,2023-09-13T01:34:01Z,2023-09-13T01:34:01Z,CONTRIBUTOR,"@simonw docs are finished, this is ready for review! One thing: I added ""Configuration"" as a top-level item in the documentation site, at the very bottom. Not sure if this is the best, maybe it can be named ""datasette.yaml Configuration"" or something similar? Mostly because ""Configuration"" by itself can mean many things, but adding ""datasette.yaml"" would make it pretty clear it's about that specific file, and is easier to scan. I'd also be fine with using ""datasette.yaml"" instead of ""datasette.json"", since writing in YAML is much more forgiving (and advanced users will know JSON is also supported) Also, maybe this is a chance to consolidate the docs a bit? I think ""Settings"", ""Configuration"", ""Metadata"", and ""Authentication and permissions"" should possibly be under the same section. Maybe even consolidate the different Plugin pages that exist? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support, https://github.com/simonw/datasette/pull/2183#issuecomment-1714699724,https://api.github.com/repos/simonw/datasette/issues/2183,1714699724,IC_kwDOBm6k_c5mND3M,9599,simonw,2023-09-11T23:01:36Z,2023-09-11T23:02:30Z,OWNER,"On thinking about this further, I'm fine releasing it as another alpha provided it causes Datasette to error loudly with an explanatory message if you attempt to load `-m metadata.json` at a metadata file that includes `""plugins""` configuration. Something like this: ```bash datasette -m metadata.json ``` Outputs: > Datasette no longer accepts plugin configuration in --metadata. Move your `""plugins""` configuration blocks to a separate file - we suggest calling that `datasette.yaml` - and start Datasette with `datasette -c datasette.yaml`. For added usability points, let's have it suggest `datasette.json` if they used `metadata.json` and `datasette.yaml` if they tried to use `metadata.yaml`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support, https://github.com/simonw/datasette/pull/2183#issuecomment-1714544153,https://api.github.com/repos/simonw/datasette/issues/2183,1714544153,IC_kwDOBm6k_c5mMd4Z,22429695,codecov[bot],2023-09-11T20:37:52Z,2023-09-13T20:58:51Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`95.00%`** and project coverage change: **`-0.04%`** :warning: > Comparison is base [(`a4c96d0`)](https://app.codecov.io/gh/simonw/datasette/commit/a4c96d01b27ce7cd06662a024da3547132a7c412?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69% compared to head [(`659dcbd`)](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.66%. > :exclamation: Current head 659dcbd differs from pull request most recent head acca338. Consider uploading reports for the commit acca338 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2183 +/- ## ========================================== - Coverage 92.69% 92.66% -0.04% ========================================== Files 40 40 Lines 6025 6039 +14 ========================================== + Hits 5585 5596 +11 - Misses 440 443 +3 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.19% <95.00%> (-0.24%)` | :arrow_down: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2183?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891212159,`datasette.yaml` plugin support, https://github.com/simonw/datasette/pull/2182#issuecomment-1719451803,https://api.github.com/repos/simonw/datasette/issues/2182,1719451803,IC_kwDOBm6k_c5mfMCb,49699333,dependabot[bot],2023-09-14T13:27:26Z,2023-09-14T13:27:26Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1890593563,Bump the python-packages group with 2 updates, https://github.com/simonw/datasette/pull/2181#issuecomment-1711054840,https://api.github.com/repos/simonw/datasette/issues/2181,1711054840,IC_kwDOBm6k_c5l_J_4,9599,simonw,2023-09-08T04:23:47Z,2023-09-08T04:23:47Z,OWNER,"I've implemented this hook once now in: - https://github.com/datasette/datasette-remote-actors And built and tested a debug tool for it in: - https://github.com/datasette/datasette-debug-actors-from-ids I'm now confident in the design of this plugin hook, I'm going to land it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886812002,actors_from_ids plugin hook and datasette.actors_from_ids() method, https://github.com/simonw/datasette/pull/2181#issuecomment-1710972324,https://api.github.com/repos/simonw/datasette/issues/2181,1710972324,IC_kwDOBm6k_c5l-12k,22429695,codecov[bot],2023-09-08T01:58:44Z,2023-09-08T03:43:43Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00%`** and no project coverage change. > Comparison is base [(`c263704`)](https://app.codecov.io/gh/simonw/datasette/commit/c26370485a4fd4bf130da051be9163d92c57f24f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`e1c5a9d`)](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2181 +/- ## ======================================= Coverage 92.68% 92.69% ======================================= Files 40 40 Lines 6017 6025 +8 ======================================= + Hits 5577 5585 +8 Misses 440 440 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.43% <100.00%> (+0.03%)` | :arrow_up: | | [datasette/hookspecs.py](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2181?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886812002,actors_from_ids plugin hook and datasette.actors_from_ids() method, https://github.com/simonw/datasette/pull/2181#issuecomment-1710969448,https://api.github.com/repos/simonw/datasette/issues/2181,1710969448,IC_kwDOBm6k_c5l-1Jo,9599,simonw,2023-09-08T01:53:27Z,2023-09-08T01:53:27Z,OWNER,"Documentation preview: - https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids - https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886812002,actors_from_ids plugin hook and datasette.actors_from_ids() method, https://github.com/simonw/datasette/issues/2180#issuecomment-1712895084,https://api.github.com/repos/simonw/datasette/issues/2180,1712895084,IC_kwDOBm6k_c5mGLRs,9599,simonw,2023-09-10T17:44:14Z,2023-09-10T17:44:14Z,OWNER,Used by `datasette-short-links` here: https://github.com/datasette/datasette-short-links/blob/468c3e25dbe06a8dcba8edda59bc16a18e126a51/datasette_short_links/__init__.py#L108-L115,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`, https://github.com/simonw/datasette/issues/2180#issuecomment-1711057080,https://api.github.com/repos/simonw/datasette/issues/2180,1711057080,IC_kwDOBm6k_c5l_Ki4,9599,simonw,2023-09-08T04:28:03Z,2023-09-08T04:28:03Z,OWNER,"Landed: - https://docs.datasette.io/en/latest/plugin_hooks.html#actors-from-ids-datasette-actor-ids - https://docs.datasette.io/en/latest/internals.html#await-actors-from-ids-actor-ids","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`, https://github.com/simonw/datasette/issues/2180#issuecomment-1711054624,https://api.github.com/repos/simonw/datasette/issues/2180,1711054624,IC_kwDOBm6k_c5l_J8g,9599,simonw,2023-09-08T04:23:23Z,2023-09-08T04:23:23Z,OWNER,"I've implemented this hook once now in: - https://github.com/datasette/datasette-remote-actors And built and tested a debug tool for it in: - https://github.com/datasette/datasette-debug-actors-from-ids I'm now confident in the design of this plugin hook, I'm going to land it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`, https://github.com/simonw/datasette/issues/2180#issuecomment-1711028355,https://api.github.com/repos/simonw/datasette/issues/2180,1711028355,IC_kwDOBm6k_c5l_DiD,9599,simonw,2023-09-08T03:36:02Z,2023-09-08T03:36:02Z,OWNER,"I shipped the first version of - https://github.com/datasette/datasette-remote-actors/issues/1 When I land this plugin in a Datasette release I need to update that repo to depend on the new alpha.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`, https://github.com/simonw/datasette/issues/2180#issuecomment-1710969339,https://api.github.com/repos/simonw/datasette/issues/2180,1710969339,IC_kwDOBm6k_c5l-1H7,9599,simonw,2023-09-08T01:53:16Z,2023-09-08T01:53:16Z,OWNER,"Documentation preview: - https://datasette--2181.org.readthedocs.build/en/2181/internals.html#await-actors-from-ids-actor-ids - https://datasette--2181.org.readthedocs.build/en/2181/plugin_hooks.html#plugin-hook-actors-from-ids","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`, https://github.com/simonw/datasette/issues/2180#issuecomment-1710947637,https://api.github.com/repos/simonw/datasette/issues/2180,1710947637,IC_kwDOBm6k_c5l-v01,9599,simonw,2023-09-08T01:17:28Z,2023-09-08T01:17:28Z,OWNER,I think this is both a plugin hook and a `await datasette.actors_from_ids(actor_ids)` internal API function that calls it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886791100,Plugin hook: `actors_from_ids()`, https://github.com/simonw/sqlite-utils/pull/593#issuecomment-1710939868,https://api.github.com/repos/simonw/sqlite-utils/issues/593,1710939868,IC_kwDOCGYnMM5l-t7c,22429695,codecov[bot],2023-09-08T01:03:40Z,2023-09-09T00:44:52Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00%`** and no project coverage change. > Comparison is base [(`5d123f0`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/5d123f031fc4fadc98f508e0ef6b7b6671e86155?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77% compared to head [(`b86374f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.77%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #593 +/- ## ======================================= Coverage 95.77% 95.77% ======================================= Files 8 8 Lines 2837 2840 +3 ======================================= + Hits 2717 2720 +3 Misses 120 120 ``` | [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <100.00%> (+<0.01%)` | :arrow_up: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/593?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886783150,".transform() now preserves rowid values, refs #592", https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1712895580,IC_kwDOCGYnMM5mGLZc,9599,simonw,2023-09-10T17:46:41Z,2023-09-10T17:46:41Z,OWNER,"In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`: https://www.sqlite.org/lang_vacuum.html#how_vacuum_works > The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid). So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710935270,IC_kwDOCGYnMM5l-szm,9599,simonw,2023-09-08T00:55:30Z,2023-09-08T00:55:30Z,OWNER,"Yes! That recreated the bug: ``` > assert previous_rows == next_rows E AssertionError: assert equals failed E [ [ E (1, '1', 'Paris'), (1, '1', 'Paris'), E (3, '3', 'New York'), (2, '3', 'New York'), E (4, '4', 'London'), (3, '4', 'London'), E ] ... E ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710934448,IC_kwDOCGYnMM5l-smw,9599,simonw,2023-09-08T00:54:03Z,2023-09-08T00:54:03Z,OWNER,"Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3. If I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710933716,IC_kwDOCGYnMM5l-sbU,9599,simonw,2023-09-08T00:52:42Z,2023-09-08T00:52:42Z,OWNER,"I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key. The reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows. But I tried this test and it passed, too: ```python def test_transform_preserves_rowids(fresh_db): fresh_db[""places""].insert_all( [ {""id"": ""1"", ""name"": ""Paris"", ""country"": ""France""}, {""id"": ""2"", ""name"": ""London"", ""country"": ""UK""}, {""id"": ""3"", ""name"": ""New York"", ""country"": ""USA""}, ], pk=""id"", ) previous_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"") ) # Transform it fresh_db[""places""].transform(column_order=(""country"", ""name"")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"") ) assert previous_rows == next_rows ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710931605,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710931605,IC_kwDOCGYnMM5l-r6V,9599,simonw,2023-09-08T00:49:02Z,2023-09-08T00:49:02Z,OWNER,"I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710930934,IC_kwDOCGYnMM5l-rv2,9599,simonw,2023-09-08T00:47:57Z,2023-09-08T00:47:57Z,OWNER,"That's odd, I wrote a test for this just now and it passes already: ```python def test_transform_preserves_rowids(fresh_db): # Create a rowid table fresh_db[""places""].insert_all( ( {""name"": ""Paris"", ""country"": ""France""}, {""name"": ""London"", ""country"": ""UK""}, {""name"": ""New York"", ""country"": ""USA""}, ), ) assert fresh_db[""places""].use_rowid previous_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, name from places"") ) # Transform it fresh_db[""places""].transform(column_order=(""country"", ""name"")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, name from places"") ) assert previous_rows == next_rows ``` So maybe I'm wrong about the cause of that bug?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493,`table.transform()` should preserve `rowid` values, https://github.com/simonw/datasette/issues/2178#issuecomment-1710879239,https://api.github.com/repos/simonw/datasette/issues/2178,1710879239,IC_kwDOBm6k_c5l-fIH,9599,simonw,2023-09-07T23:20:32Z,2023-09-07T23:20:32Z,OWNER,"To test that locally, use this YAML instead: ```yaml databases: content: allow: id: root tables: releases: allow: true ``` And: ```yaml allow: id: root databases: content: tables: releases: allow: true","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access, https://github.com/simonw/datasette/issues/2178#issuecomment-1710878391,https://api.github.com/repos/simonw/datasette/issues/2178,1710878391,IC_kwDOBm6k_c5l-e63,9599,simonw,2023-09-07T23:19:05Z,2023-09-07T23:19:05Z,OWNER,"This fix didn't work on Datasette Cloud. I used `/-/permissions` to debug it and saw this: ![image](https://github.com/simonw/datasette/assets/9599/61d2bc5f-1f96-41ea-8658-91dfbcb6610c) Only checking `view-table` is not enough: for my instance on Datasette Cloud the view permission check that should have failed was for the database or instance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access, https://github.com/simonw/datasette/issues/2178#issuecomment-1710871095,https://api.github.com/repos/simonw/datasette/issues/2178,1710871095,IC_kwDOBm6k_c5l-dI3,9599,simonw,2023-09-07T23:07:16Z,2023-09-07T23:07:16Z,OWNER,"I ran this: datasette content.db -p 8043 -m fk-auth.yml --root Against this YAML: ```yaml databases: content: tables: users: allow: id: root ``` And it worked as it should - here's a screenshot of an anonymous user and a root user viewing the same page: ![CleanShot 2023-09-07 at 16 05 34@2x](https://github.com/simonw/datasette/assets/9599/3e91da08-107c-421c-8a00-aa650b960c58) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access, https://github.com/simonw/datasette/issues/2178#issuecomment-1710567329,https://api.github.com/repos/simonw/datasette/issues/2178,1710567329,IC_kwDOBm6k_c5l9S-h,9599,simonw,2023-09-07T17:59:59Z,2023-09-07T17:59:59Z,OWNER,Should I put the permission check in that undocumented `datasette.expand_foreign_keys()` method? I think so - it should accept `request.actor` as one of its arguments.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access, https://github.com/simonw/datasette/issues/2178#issuecomment-1710565268,https://api.github.com/repos/simonw/datasette/issues/2178,1710565268,IC_kwDOBm6k_c5l9SeU,9599,simonw,2023-09-07T17:58:04Z,2023-09-07T17:59:06Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/views/table.py#L1132-L1149 Which calls this undocumented method: https://github.com/simonw/datasette/blob/fbcb103c0cb6668018ace539a01a6a1f156e8d6a/datasette/app.py#L938-L973","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886350562,Don't show foreign key links to tables the user cannot access, https://github.com/simonw/datasette/issues/2177#issuecomment-1708777964,https://api.github.com/repos/simonw/datasette/issues/2177,1708777964,IC_kwDOBm6k_c5l2eHs,9599,simonw,2023-09-06T17:04:30Z,2023-09-06T17:04:30Z,OWNER,"Here's the thinking: setting the`DATASETTE_INTERNAL` environment variable for your entire machine will cause any time you run `datasette x.db` to result in a process that shares the same `internal.db` database as other instances. You might run more than one instance at once (I often have 4 or 5 going). This would currently break, because they would over-write each other's catalog tables: https://github.com/simonw/datasette/blob/e4abae3fd7a828625d00c35c316852ffbaa5ef2f/datasette/utils/internal_db.py#L5-L62 The breaking wouldn't be obvious because the catalog tables aren't used by any features yet, but it's still bad. This convinced us that actually we should move those `catalog_` tables OUT of `internal.db`. The `_internal` database will be reserved for plugins that want to use it for caching, storing progress, etc. I think we move them to an in-memory `_catalog` database which is excluded from `ds.databases` (like `_internal` is ) but can be accessed using `datasette.get_catalog_database()` - similar to `datasette.get_internal_database()`. So each instance of Datasette gets its own truly private `_catalog`, which is in-memory and so gets cleared at the end of each process. An interesting thing that came up about a shared `_internal` database is that it provides opportunities for things like a notes plugin which allows you to attach notes to any row in any table in a database... where those notes become available to multiple Datasette instances that you might launch on the same laptop.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884408624,Move schema tables from _internal to _catalog, https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1793278279,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1793278279,IC_kwDOCGYnMM5q40FH,9599,simonw,2023-11-04T00:58:03Z,2023-11-04T00:58:03Z,OWNER,I'm going to abandon this PR and ship the 3.12 testing change directly to `main`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview, https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708695907,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708695907,IC_kwDOCGYnMM5l2KFj,9599,simonw,2023-09-06T16:15:59Z,2023-09-06T16:19:14Z,OWNER,"The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant: - https://github.com/pypa/setuptools/issues/3661 ``` 25h Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' ERROR: Exception: Traceback (most recent call last): ... File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py"", line 697, in get_requires_for_build_wheel return super().get_requires_for_build_wheel(config_settings=cs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 166, in get_requires_for_build_wheel return self._call_hook('get_requires_for_build_wheel', { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 321, in _call_hook raise BackendUnavailable(data.get('traceback', '')) pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py"", line 77, in _build_backend obj = import_module(mod_path) ^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py"", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File """", line 1381, in _gcd_import File """", line 1354, in _find_and_load File """", line 1304, in _find_and_load_unlocked File """", line 488, in _call_with_frames_removed File """", line 1381, in _gcd_import File """", line 1354, in _find_and_load File """", line 1325, in _find_and_load_unlocked File """", line 929, in _load_unlocked File """", line 994, in exec_module File """", line 488, in _call_with_frames_removed File ""/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py"", line 10, in import distutils.core ModuleNotFoundError: No module named 'distutils' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview, https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708693020,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708693020,IC_kwDOCGYnMM5l2JYc,22429695,codecov[bot],2023-09-06T16:14:03Z,2023-11-04T00:54:25Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified and coverable lines are covered by tests :white_check_mark: > Comparison is base [(`347fdc8`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/347fdc865e91b8d3410f49a5c9d5b499fbb594c1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74% compared to head [(`1f14df1`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.74%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #591 +/- ## ======================================= Coverage 95.74% 95.74% ======================================= Files 8 8 Lines 2842 2842 ======================================= Hits 2721 2721 Misses 121 121 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/591?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789,Test against Python 3.12 preview, https://github.com/simonw/datasette/pull/2174#issuecomment-1708728926,https://api.github.com/repos/simonw/datasette/issues/2174,1708728926,IC_kwDOBm6k_c5l2SJe,9599,simonw,2023-09-06T16:36:56Z,2023-09-06T16:36:56Z,OWNER,`DATASETTE_INTERNAL` would be more consistent with `DATASETTE_SECRET`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884330740,Use $DATASETTE_INTERNAL in absence of --internal, https://github.com/simonw/datasette/pull/2174#issuecomment-1708727204,https://api.github.com/repos/simonw/datasette/issues/2174,1708727204,IC_kwDOBm6k_c5l2Ruk,9599,simonw,2023-09-06T16:35:39Z,2023-09-07T23:45:42Z,OWNER,"We can use this here: https://click.palletsprojects.com/en/8.1.x/options/#values-from-environment-variables ```python @click.option(..., envvar=""DATASETTE_INTERNAL"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884330740,Use $DATASETTE_INTERNAL in absence of --internal, https://github.com/simonw/datasette/pull/2174#issuecomment-1708699926,https://api.github.com/repos/simonw/datasette/issues/2174,1708699926,IC_kwDOBm6k_c5l2LEW,22429695,codecov[bot],2023-09-06T16:18:13Z,2023-09-08T00:46:13Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`d75b519`)](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%. > Report is 7 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2174 +/- ## ======================================= Coverage 92.68% 92.68% ======================================= Files 40 40 Lines 6012 6017 +5 ======================================= + Hits 5572 5577 +5 Misses 440 440 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <ø> (ø)` | | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2174/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2174?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884330740,Use $DATASETTE_INTERNAL in absence of --internal, https://github.com/simonw/datasette/pull/2173#issuecomment-1707570378,https://api.github.com/repos/simonw/datasette/issues/2173,1707570378,IC_kwDOBm6k_c5lx3TK,9599,simonw,2023-09-06T02:40:31Z,2023-09-06T02:40:31Z,OWNER,"Surprising error: ``` blacken-docs -l 60 docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.11.4/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.11.4/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.11.4/x64/lib docs/writing_plugins.rst:365: code block parse error Cannot parse: 8:0: ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1883055640,click-default-group>=1.2.3, https://github.com/simonw/datasette/pull/2173#issuecomment-1707570044,https://api.github.com/repos/simonw/datasette/issues/2173,1707570044,IC_kwDOBm6k_c5lx3N8,22429695,codecov[bot],2023-09-06T02:39:59Z,2023-09-06T02:49:01Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`fd083e3`)](https://app.codecov.io/gh/simonw/datasette/commit/fd083e37ec53e7e625111168d324a572344a3b19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`a481ebd`)](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%. > Report is 1 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2173 +/- ## ======================================= Coverage 92.68% 92.68% ======================================= Files 40 40 Lines 6012 6012 ======================================= Hits 5572 5572 Misses 440 440 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2173?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1883055640,click-default-group>=1.2.3, https://github.com/simonw/datasette/pull/2173#issuecomment-1707565495,https://api.github.com/repos/simonw/datasette/issues/2173,1707565495,IC_kwDOBm6k_c5lx2G3,9599,simonw,2023-09-06T02:33:56Z,2023-09-06T02:33:56Z,OWNER,Running tests to see if Pyodide works - that was the reason I switched to my `click-default-group-wheel` package originally.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1883055640,click-default-group>=1.2.3, https://github.com/simonw/sqlite-utils/issues/590#issuecomment-1704387161,https://api.github.com/repos/simonw/sqlite-utils/issues/590,1704387161,IC_kwDOCGYnMM5lluJZ,9599,simonw,2023-09-03T19:50:36Z,2023-09-03T19:50:36Z,OWNER,"Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879214365,Ability to tell if a Database is an in-memory one, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384393,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384393,IC_kwDOCGYnMM5llteJ,9599,simonw,2023-09-03T19:36:34Z,2023-09-03T19:36:34Z,OWNER,"Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b Still needs tests and documentation (and some more thought to make sure it's doing the right thing).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384111,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384111,IC_kwDOCGYnMM5lltZv,9599,simonw,2023-09-03T19:35:03Z,2023-09-03T19:35:03Z,OWNER,"Normally in Python/`sqlite3` you de-register a function by passing `None` to it. You can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else: https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704383901,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704383901,IC_kwDOCGYnMM5lltWd,9599,simonw,2023-09-03T19:34:05Z,2023-09-03T19:34:05Z,OWNER,"For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager. I played with this idea a bit: ```python with db.register_functions(md5, md5_random): db.query(...) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560,Mechanism for de-registering registered SQL functions, https://github.com/simonw/datasette/issues/2168#issuecomment-1712897194,https://api.github.com/repos/simonw/datasette/issues/2168,1712897194,IC_kwDOBm6k_c5mGLyq,9599,simonw,2023-09-10T17:54:07Z,2023-09-10T17:54:07Z,OWNER,"This looks relevant: https://pluggy.readthedocs.io/en/stable/#wrappers > A *hookimpl* can be marked with the `""wrapper""` option, which indicates that the function will be called to *wrap* (or surround) all other normal *hookimpl* calls. A *hook wrapper* can thus execute some code ahead and after the execution of all corresponding non-wrappper *hookimpls*. This could be the perfect mechanism for implementing this hook, although I still need to figure out how it interacts with streaming.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(), https://github.com/simonw/datasette/issues/2168#issuecomment-1701831013,https://api.github.com/repos/simonw/datasette/issues/2168,1701831013,IC_kwDOBm6k_c5lb-Fl,9599,simonw,2023-08-31T21:51:12Z,2023-08-31T21:52:15Z,OWNER,"Need to make sure the design of this takes streaming responses into account. Those could be pretty tricky here. I nice thing about `asgi_wrapper()` is that it handles those already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(), https://github.com/simonw/datasette/issues/2168#issuecomment-1701830241,https://api.github.com/repos/simonw/datasette/issues/2168,1701830241,IC_kwDOBm6k_c5lb95h,9599,simonw,2023-08-31T21:50:18Z,2023-08-31T21:50:18Z,OWNER,"The hook could be called `register_middleware()` and could work like `register_routes()` and `register_commands()`: ```python @hookspec def register_middleware(datasette): """"""Register middleware: returns a list of async def middleware functions"""""" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(), https://github.com/simonw/datasette/issues/2168#issuecomment-1701828197,https://api.github.com/repos/simonw/datasette/issues/2168,1701828197,IC_kwDOBm6k_c5lb9Zl,9599,simonw,2023-08-31T21:48:00Z,2023-08-31T21:48:57Z,OWNER,"A pattern like this could be interesting: ```python async def my_middleware(datasette, request, get_response): # Mess with request here if neccessary response = await get_response(request) # mess with response return response ``` The Django pattern is more complicated but does have that mechanism for running one-time configuration prior to defining the `middleware()` function, which is neat.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(), https://github.com/simonw/datasette/issues/2168#issuecomment-1701826521,https://api.github.com/repos/simonw/datasette/issues/2168,1701826521,IC_kwDOBm6k_c5lb8_Z,9599,simonw,2023-08-31T21:46:13Z,2023-08-31T21:46:13Z,OWNER,"This could even be a pair of hooks - `process_request()` and `process_response()`. Or could take a leaf from Django, which redesigned middleware to use this pattern instead: ```python def simple_middleware(get_response): # One-time configuration and initialization. def middleware(request): # Code to be executed for each request before # the view (and later middleware) are called. response = get_response(request) # Code to be executed for each request/response after # the view is called. return response return middleware ``` Or even borrow an idea from `pytest` where fixtures can `yield` in the middle, like this: ```python @pytest.fixture def sending_user(mail_admin): user = mail_admin.create_user() yield user mail_admin.delete_user(user) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(), https://github.com/simonw/datasette/issues/2168#issuecomment-1701823609,https://api.github.com/repos/simonw/datasette/issues/2168,1701823609,IC_kwDOBm6k_c5lb8R5,9599,simonw,2023-08-31T21:43:06Z,2023-08-31T21:44:13Z,OWNER,"Not sure what to call this. Maybe `app_wrapper()`? Or perhaps it's simpler than that, something like this: ```python @hookspec def process_response(datasette, request, response): """"""Last chance to modify the response before it is returned to the client"""""" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1876353656,Consider a request/response wrapping hook slightly higher level than asgi_wrapper(), https://github.com/simonw/datasette/pull/2166#issuecomment-1701045404,https://api.github.com/repos/simonw/datasette/issues/2166,1701045404,IC_kwDOBm6k_c5lY-Sc,22429695,codecov[bot],2023-08-31T13:31:15Z,2023-09-06T13:25:17Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`05707aa`)](https://app.codecov.io/gh/simonw/datasette/commit/05707aa16b5c6c39fbe48b3176b85a8ffe493938?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68% compared to head [(`7c9df6e`)](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2166 +/- ## ======================================= Coverage 92.68% 92.68% ======================================= Files 40 40 Lines 6012 6012 ======================================= Hits 5572 5572 Misses 440 440 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2166?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1875519316,Bump the python-packages group with 1 update, https://github.com/simonw/datasette/pull/2165#issuecomment-1699910555,https://api.github.com/repos/simonw/datasette/issues/2165,1699910555,IC_kwDOBm6k_c5lUpOb,9599,simonw,2023-08-30T22:05:14Z,2023-08-30T22:05:14Z,OWNER,Documentation preview: https://github.com/simonw/datasette/blob/6321c9c055a640ed6ea98e231dc5813dcde1f773/docs/plugins.rst#controlling-which-plugins-are-loaded,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins, https://github.com/simonw/datasette/pull/2165#issuecomment-1699884314,https://api.github.com/repos/simonw/datasette/issues/2165,1699884314,IC_kwDOBm6k_c5lUi0a,9599,simonw,2023-08-30T21:38:29Z,2023-08-30T21:39:51Z,OWNER,"Here's the reason for that name disparity: https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L54-L65 Note how the `distinfo.project_name` name is used when available. That seems to work for regularly installed plugins but not for plugins loaded via `DATASETTE_LOAD_PLUGINS`. And that's looking things up in `plugin_to_distinfo` which is populated here: https://github.com/simonw/datasette/blob/30b28c8367a9c6870386ea10a202705b40862457/datasette/plugins.py#L37","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins, https://github.com/simonw/datasette/pull/2165#issuecomment-1699812599,https://api.github.com/repos/simonw/datasette/issues/2165,1699812599,IC_kwDOBm6k_c5lURT3,22429695,codecov[bot],2023-08-30T20:43:19Z,2023-08-30T22:05:05Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`26.66%`** and project coverage change: **`-0.17%`** :warning: > Comparison is base [(`30b28c8`)](https://app.codecov.io/gh/simonw/datasette/commit/30b28c8367a9c6870386ea10a202705b40862457?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.84% compared to head [(`5eddf34`)](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.68%. > :exclamation: Current head 5eddf34 differs from pull request most recent head 6321c9c. Consider uploading reports for the commit 6321c9c to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2165 +/- ## ========================================== - Coverage 92.84% 92.68% -0.17% ========================================== Files 40 40 Lines 5998 6012 +14 ========================================== + Hits 5569 5572 +3 - Misses 429 440 +11 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/plugins.py](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `64.58% <26.66%> (-17.77%)` | :arrow_down: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2165?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins, https://github.com/simonw/datasette/pull/2165#issuecomment-1699811810,https://api.github.com/repos/simonw/datasette/issues/2165,1699811810,IC_kwDOBm6k_c5lURHi,9599,simonw,2023-08-30T20:42:41Z,2023-08-30T20:42:41Z,OWNER,"The `load_setuptools_entrypoints()` function in Pluggy [does this](https://github.com/pytest-dev/pluggy/blob/0b41c9766508a46ae666cf281684df3164b3e2a9/src/pluggy/_manager.py#L376): ```python for ep in dist.entry_points: if ( ep.group != group or (name is not None and ep.name != name) # already registered or self.get_plugin(ep.name) or self.is_blocked(ep.name) ): continue plugin = ep.load() self.register(plugin, name=ep.name) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins, https://github.com/simonw/datasette/pull/2165#issuecomment-1699809688,https://api.github.com/repos/simonw/datasette/issues/2165,1699809688,IC_kwDOBm6k_c5lUQmY,9599,simonw,2023-08-30T20:41:06Z,2023-08-30T20:41:06Z,OWNER,"Slight weirdness: I noticed that the output from the `datasette plugins` command looks like this for plugins loaded with the new environment variable: ```json { ""name"": ""datasette_pretty_json"", ""static"": false, ""templates"": false, ""version"": null, ""hooks"": [ ""render_cell"" ] }, ``` That should ideally be `datasette-pretty-json`, not `datasette_pretty_json`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins, https://github.com/simonw/datasette/pull/2165#issuecomment-1699802028,https://api.github.com/repos/simonw/datasette/issues/2165,1699802028,IC_kwDOBm6k_c5lUOus,9599,simonw,2023-08-30T20:35:02Z,2023-08-30T20:35:02Z,OWNER,"Testing this is going to be a bit of a pain. I think I'll add a whole separate test block to CI which installs a couple of plugins and then exercises this feature using `datasette plugins`. I'll use `datasette-init` and `datasette-json-html` just because they are small and simple.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874327336,DATASETTE_LOAD_PLUGINS environment variable for loading specific plugins, https://github.com/simonw/datasette/issues/2164#issuecomment-1699728102,https://api.github.com/repos/simonw/datasette/issues/2164,1699728102,IC_kwDOBm6k_c5lT8rm,9599,simonw,2023-08-30T19:35:32Z,2023-08-30T19:35:47Z,OWNER,"Figured this out so far with the help of GPT-4: https://chat.openai.com/share/0e785865-621b-4fb3-ba05-7449e57c8496 Now this works: ```bash DATASETTE_LOAD_PLUGINS=datasette-write-ui datasette plugins ``` ```json [ { ""name"": ""datasette_write_ui"", ""static"": true, ""templates"": true, ""version"": null, ""hooks"": [ ""extra_template_vars"", ""register_routes"" ] } ] ``` Or multiple plugins: ```bash DATASETTE_LOAD_PLUGINS=datasette-write-ui,datasette-pretty-json datasette plugins ``` Outputs: ```json [ { ""name"": ""datasette_pretty_json"", ""static"": false, ""templates"": false, ""version"": null, ""hooks"": [ ""render_cell"" ] }, { ""name"": ""datasette_write_ui"", ""static"": true, ""templates"": true, ""version"": null, ""hooks"": [ ""extra_template_vars"", ""register_routes"" ] } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1874255116,Ability to only load a specific list of plugins, https://github.com/simonw/datasette/issues/2163#issuecomment-1697818917,https://api.github.com/repos/simonw/datasette/issues/2163,1697818917,IC_kwDOBm6k_c5lMqkl,9599,simonw,2023-08-29T16:56:20Z,2023-08-29T16:56:20Z,OWNER,https://github.com/simonw/datasette/blob/50da908213a0fc405ecd7a40090dfea7a2e7395c/datasette/utils/internal_db.py#L8-L62,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1872043170,Rename core_X to catalog_X in the internals, https://github.com/simonw/datasette/pull/2162#issuecomment-1696710911,https://api.github.com/repos/simonw/datasette/issues/2162,1696710911,IC_kwDOBm6k_c5lIcD_,9599,simonw,2023-08-29T03:23:44Z,2023-08-29T03:23:44Z,OWNER,I'm going to merge this so we can see how it feels.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database", https://github.com/simonw/datasette/pull/2162#issuecomment-1696709110,https://api.github.com/repos/simonw/datasette/issues/2162,1696709110,IC_kwDOBm6k_c5lIbn2,9599,simonw,2023-08-29T03:20:40Z,2023-08-29T03:22:47Z,OWNER,"> However, one important notes about those new `core_` tables: If a `--internal` DB is passed in, that means those `core_` tables will persist across multiple Datasette instances. This wasn't the case before, since `_internal` was always an in-memory database created from scratch. I'm completely happy for the `core_*` tables (or `datasette_*` or some other name) to live in the persisted-to-disk `internal.db` database, even though they're effectively meant to be an in-memory cache. I don't think it causes any harm, and it could even be quite useful to have them visible on disk - other applications could read the `internal.db` database while Datasette itself is running, should they have some weird reason to want to do that! Having those tables stick around in `internal.db` after Datasette shuts down could be useful for other debugging activities as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database", https://github.com/simonw/datasette/pull/2162#issuecomment-1696707458,https://api.github.com/repos/simonw/datasette/issues/2162,1696707458,IC_kwDOBm6k_c5lIbOC,9599,simonw,2023-08-29T03:17:54Z,2023-08-29T03:17:54Z,OWNER,Documentation preview: https://datasette--2162.org.readthedocs.build/en/2162/internals.html#datasette-s-internal-database,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database", https://github.com/simonw/datasette/pull/2162#issuecomment-1696594855,https://api.github.com/repos/simonw/datasette/issues/2162,1696594855,IC_kwDOBm6k_c5lH_un,22429695,codecov[bot],2023-08-29T00:19:29Z,2023-08-29T03:22:21Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00%`** and project coverage change: **`-0.03%`** :warning: > Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`73489ca`)](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.79%. > Report is 1 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2162 +/- ## ========================================== - Coverage 92.82% 92.79% -0.03% ========================================== Files 40 40 Lines 5948 5953 +5 ========================================== + Hits 5521 5524 +3 - Misses 427 429 +2 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `96.87% <ø> (-0.04%)` | :arrow_down: | | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.32% <100.00%> (+0.03%)` | :arrow_up: | | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.53% <100.00%> (-0.06%)` | :arrow_down: | | [datasette/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RhdGFiYXNlLnB5) | `94.72% <100.00%> (+0.04%)` | :arrow_up: | | [datasette/utils/internal\_db.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL2ludGVybmFsX2RiLnB5) | `100.00% <100.00%> (ø)` | | | [datasette/views/database.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.45% <100.00%> (ø)` | | | [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <100.00%> (-0.85%)` | :arrow_down: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2162?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870672704,"Add new `--internal internal.db` option, deprecate legacy `_internal` database", https://github.com/simonw/datasette/pull/2161#issuecomment-1696267473,https://api.github.com/repos/simonw/datasette/issues/2161,1696267473,IC_kwDOBm6k_c5lGvzR,22429695,codecov[bot],2023-08-28T19:38:28Z,2023-08-28T19:39:33Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`75.00%`** and project coverage change: **`+0.05%`** :tada: > Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`a5cbf80`)](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.81%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2161 +/- ## ========================================== + Coverage 92.76% 92.81% +0.05% ========================================== Files 40 40 Lines 5943 5944 +1 ========================================== + Hits 5513 5517 +4 + Misses 430 427 -3 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `82.37% <75.00%> (+1.02%)` | :arrow_up: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2161?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1870345352,"-s/--setting x y gets merged into datasette.yml, refs #2143, #2156", https://github.com/simonw/datasette/pull/2160#issuecomment-1696595326,https://api.github.com/repos/simonw/datasette/issues/2160,1696595326,IC_kwDOBm6k_c5lH_1-,9599,simonw,2023-08-29T00:20:08Z,2023-08-29T00:20:08Z,OWNER,Cog failed!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2160#issuecomment-1696592763,https://api.github.com/repos/simonw/datasette/issues/2160,1696592763,IC_kwDOBm6k_c5lH_N7,9599,simonw,2023-08-29T00:16:38Z,2023-08-29T00:16:38Z,OWNER,"Since this bumps Sphinx I'm manually reviewing the ReadTheDocs preview a bit, looks good to me: https://datasette--2160.org.readthedocs.build/en/2160/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2160#issuecomment-1696586767,https://api.github.com/repos/simonw/datasette/issues/2160,1696586767,IC_kwDOBm6k_c5lH9wP,9599,simonw,2023-08-29T00:08:26Z,2023-08-29T00:08:35Z,OWNER,"https://github.com/simonw/datasette/issues/2148#issuecomment-1689177556 > Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2160#issuecomment-1696586213,https://api.github.com/repos/simonw/datasette/issues/2160,1696586213,IC_kwDOBm6k_c5lH9nl,9599,simonw,2023-08-29T00:07:46Z,2023-08-29T00:07:46Z,OWNER,"I figured out why this was failing in: - https://github.com/simonw/datasette/pull/2148 It's because Sphinx dropped support for Python 3.8. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2160#issuecomment-1695754277,https://api.github.com/repos/simonw/datasette/issues/2160,1695754277,IC_kwDOBm6k_c5lEygl,22429695,codecov[bot],2023-08-28T14:00:17Z,2023-08-29T00:32:52Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`c81b024`)](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82%. > :exclamation: Current head c81b024 differs from pull request most recent head 3529c9c. Consider uploading reports for the commit 3529c9c to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2160 +/- ## ======================================= Coverage 92.82% 92.82% ======================================= Files 40 40 Lines 5948 5948 ======================================= Hits 5521 5521 Misses 427 427 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2160?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1869807874,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/sqlite-utils/issues/588#issuecomment-1694823972,https://api.github.com/repos/simonw/sqlite-utils/issues/588,1694823972,IC_kwDOCGYnMM5lBPYk,9599,simonw,2023-08-28T00:41:54Z,2023-08-28T00:41:54Z,OWNER,Tips on typing `**kwargs`: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1868713944,`table.get(column=value)` option for retrieving things not by their primary key, https://github.com/simonw/datasette/issues/2157#issuecomment-1700291967,https://api.github.com/repos/simonw/datasette/issues/2157,1700291967,IC_kwDOBm6k_c5lWGV_,15178711,asg017,2023-08-31T02:45:56Z,2023-08-31T02:45:56Z,CONTRIBUTOR,"@simonw what do you think about adding a `DATASETTE_INTERNAL_DB_PATH` env variable, where when defined, is the default location of the internal DB? This means when the `--internal` flag is NOT provided, Datasette would check to see if `DATASETTE_INTERNAL_DB_PATH` exists, and if so, uses that as the internal database (and would fallback to an ephemeral memory database) My rationale: some plugins may require, or strongly encourage, a persistent internal database (`datasette-comments`, `datasette-bookmarks`, `datasette-link-shortener`, etc.). However, for users that have a global installation of Datasette (say from `brew install` or a global `pip install`), it would be annoying having to specify `--internal` every time. So instead, they can just add `export DATASETTE_INTERNAL_DB_PATH=""/path/to/internal.db""` to their bashrc/zshrc/whereever to not have to worry about `--internal`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,"Proposal: Make the `_internal` database persistent, customizable, and hidden", https://github.com/simonw/datasette/issues/2157#issuecomment-1692465763,https://api.github.com/repos/simonw/datasette/issues/2157,1692465763,IC_kwDOBm6k_c5k4Ppj,9599,simonw,2023-08-24T21:54:29Z,2023-08-24T21:54:29Z,OWNER,"But yes, I'm a big +1 on this whole plan.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,"Proposal: Make the `_internal` database persistent, customizable, and hidden", https://github.com/simonw/datasette/issues/2157#issuecomment-1692465334,https://api.github.com/repos/simonw/datasette/issues/2157,1692465334,IC_kwDOBm6k_c5k4Pi2,9599,simonw,2023-08-24T21:54:09Z,2023-08-24T21:54:09Z,OWNER,"We discussed this in-person this morning and these notes reflect what we talked about perfectly. I've had so many bugs with plugins that I've written myself that have forgotten to special-case the `_internal` database when looping through `datasette.databases.keys()` - removing it from there entirely would help a lot. Just one tiny disagreement: for `datasette-comments` I think having it store things in `_internal` could be an option, but in most cases I expect users to chose NOT to do that - because being able to join against those tables for more advanced queries is going to be super useful. Show me all rows in `foia_requests` with at least one associated comment in `datasette_comments.comments` kind of tihng.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865869205,"Proposal: Make the `_internal` database persistent, customizable, and hidden", https://github.com/simonw/datasette/issues/2156#issuecomment-1696260446,https://api.github.com/repos/simonw/datasette/issues/2156,1696260446,IC_kwDOBm6k_c5lGuFe,9599,simonw,2023-08-28T19:33:05Z,2023-08-28T19:33:05Z,OWNER,"Now a PR: - #2161 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options, https://github.com/simonw/datasette/issues/2156#issuecomment-1692206200,https://api.github.com/repos/simonw/datasette/issues/2156,1692206200,IC_kwDOBm6k_c5k3QR4,9599,simonw,2023-08-24T18:25:23Z,2023-08-24T18:25:23Z,OWNER,"Ran out of time for this, I'll look at the next step next week.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options, https://github.com/simonw/datasette/issues/2156#issuecomment-1692201647,https://api.github.com/repos/simonw/datasette/issues/2156,1692201647,IC_kwDOBm6k_c5k3PKv,9599,simonw,2023-08-24T18:21:53Z,2023-08-24T18:21:53Z,OWNER,"Oops, that was meant to be a PR. It's just a utility function though so it's safe to land already. I'll do a PR for the actual integration of it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options, https://github.com/simonw/datasette/issues/2156#issuecomment-1692186522,https://api.github.com/repos/simonw/datasette/issues/2156,1692186522,IC_kwDOBm6k_c5k3Lea,9599,simonw,2023-08-24T18:10:04Z,2023-08-24T18:10:04Z,OWNER,I have an implementation in https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514 too - I'm going to land that as a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865649347,datasette -s/--setting option for setting nested configuration options, https://github.com/simonw/datasette/pull/2155#issuecomment-1737906995,https://api.github.com/repos/simonw/datasette/issues/2155,1737906995,IC_kwDOBm6k_c5nllsz,79087,cadeef,2023-09-27T18:44:02Z,2023-09-27T18:44:02Z,NONE,@simonw Any chance we can get this tiny patch merged for an upcoming release?,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865572575,Fix hupper.start_reloader entry point, https://github.com/simonw/datasette/pull/2155#issuecomment-1737363182,https://api.github.com/repos/simonw/datasette/issues/2155,1737363182,IC_kwDOBm6k_c5njg7u,418191,jaywgraves,2023-09-27T13:05:41Z,2023-09-27T13:05:41Z,CONTRIBUTOR,I'm hitting the #2123 issue and I just patched my local version with this and it seems to work fine.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865572575,Fix hupper.start_reloader entry point, https://github.com/simonw/datasette/pull/2154#issuecomment-1697725150,https://api.github.com/repos/simonw/datasette/issues/2154,1697725150,IC_kwDOBm6k_c5lMTre,9599,simonw,2023-08-29T15:54:12Z,2023-08-29T15:54:12Z,OWNER,"In that last commit I also upgraded `Permission` from a named tuple to a dataclass, and added a `implies_can_view=True` private (not documented) option to it. Last step is to refactor the code to use that new property.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/pull/2154#issuecomment-1697612168,https://api.github.com/repos/simonw/datasette/issues/2154,1697612168,IC_kwDOBm6k_c5lL4GI,9599,simonw,2023-08-29T14:57:54Z,2023-08-29T14:57:54Z,OWNER,"The code to refactor is this: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L181-L280 I'm going to turn that into a more general `restrictions_allow_action` function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/pull/2154#issuecomment-1696644066,https://api.github.com/repos/simonw/datasette/issues/2154,1696644066,IC_kwDOBm6k_c5lILvi,9599,simonw,2023-08-29T01:37:47Z,2023-08-29T03:00:25Z,OWNER,"Code for this might be cleaner with a `Restrictions()` class that takes a `""_r""` dictionary to the constructor and can then answer questions like `.any_resource_has_permission(""view-table"")` - where it can resolve aliases etc as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/pull/2154#issuecomment-1696642671,https://api.github.com/repos/simonw/datasette/issues/2154,1696642671,IC_kwDOBm6k_c5lILZv,9599,simonw,2023-08-29T01:35:37Z,2023-08-29T01:35:37Z,OWNER,"Reminder that I also need to confirm that `insert-row` works if you have it at the instance level, the database level or the resource level in `_r`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/pull/2154#issuecomment-1696618784,https://api.github.com/repos/simonw/datasette/issues/2154,1696618784,IC_kwDOBm6k_c5lIFkg,9599,simonw,2023-08-29T00:58:34Z,2023-08-29T00:58:34Z,OWNER,Should this have `implies_can_view=True` too? Probably: https://github.com/simonw/datasette/blob/d64a9896f743f87b673d58859a0ec16685594e79/datasette/default_permissions.py#L20-L22,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/pull/2154#issuecomment-1691845306,https://api.github.com/repos/simonw/datasette/issues/2154,1691845306,IC_kwDOBm6k_c5k14K6,9599,simonw,2023-08-24T14:57:39Z,2023-08-24T14:57:39Z,OWNER,"Notes on manual testing so far - it looks like this might be working! - https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/pull/2154#issuecomment-1691798722,https://api.github.com/repos/simonw/datasette/issues/2154,1691798722,IC_kwDOBm6k_c5k1szC,22429695,codecov[bot],2023-08-24T14:31:42Z,2023-08-29T16:15:12Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00%`** and project coverage change: **`+0.03%`** :tada: > Comparison is base [(`2e28258`)](https://app.codecov.io/gh/simonw/datasette/commit/2e2825869fc2655b5fcadc743f6f9dec7a49bc65?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.82% compared to head [(`3e49fd3`)](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.85%. > Report is 3 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2154 +/- ## ========================================== + Coverage 92.82% 92.85% +0.03% ========================================== Files 40 40 Lines 5948 6008 +60 ========================================== + Hits 5521 5579 +58 - Misses 427 429 +2 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/special.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `94.06% <ø> (-0.85%)` | :arrow_down: | | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.37% <100.00%> (+0.07%)` | :arrow_up: | | [datasette/default\_permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2RlZmF1bHRfcGVybWlzc2lvbnMucHk=) | `97.48% <100.00%> (+0.57%)` | :arrow_up: | | [datasette/permissions.py](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3Blcm1pc3Npb25zLnB5) | `100.00% <100.00%> (ø)` | | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2154/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2154?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/pull/2154#issuecomment-1691788400,https://api.github.com/repos/simonw/datasette/issues/2154,1691788400,IC_kwDOBm6k_c5k1qRw,9599,simonw,2023-08-24T14:25:56Z,2023-08-24T14:25:56Z,OWNER,"Can be tested with: ```bash pip install https://github.com/simonw/datasette/archive/6d57a8c23043e99b27f7a2afbe58f4d58815fd51.zip ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865281760,Cascade for restricted token view-table/view-database/view-instance operations, https://github.com/simonw/datasette/issues/2153#issuecomment-1696355634,https://api.github.com/repos/simonw/datasette/issues/2153,1696355634,IC_kwDOBm6k_c5lHFUy,9599,simonw,2023-08-28T20:19:15Z,2023-08-28T20:19:15Z,OWNER,"Documentation: - https://docs.datasette.io/en/latest/cli-reference.html#datasette-get ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option, https://github.com/simonw/datasette/issues/2153#issuecomment-1691779180,https://api.github.com/repos/simonw/datasette/issues/2153,1691779180,IC_kwDOBm6k_c5k1oBs,9599,simonw,2023-08-24T14:21:03Z,2023-08-24T14:21:03Z,OWNER,"`datasette serve` currently only has a `--get` - for this to be really useful it needs to grow `--post` and maybe other verbs too. Which is a good argument for moving this functionality to `datasette client get ...` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option, https://github.com/simonw/datasette/issues/2153#issuecomment-1691763427,https://api.github.com/repos/simonw/datasette/issues/2153,1691763427,IC_kwDOBm6k_c5k1kLj,9599,simonw,2023-08-24T14:12:43Z,2023-08-24T14:12:43Z,OWNER,Annoying that `datasette client ...` makes a great name both for a plugin that executes simulated queries against a local database (thanks to its similarity to the existing `datasette.client` Python API) but is also the ideal name for a command for running commands as a client of an external Datasette instance!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option, https://github.com/simonw/datasette/issues/2153#issuecomment-1691761685,https://api.github.com/repos/simonw/datasette/issues/2153,1691761685,IC_kwDOBm6k_c5k1jwV,9599,simonw,2023-08-24T14:11:41Z,2023-08-24T14:11:41Z,OWNER,"Another option: implement this as a plugin, providing a new command like `datasette get ...` Or implement `datasette client get ...` as core commands or a plugin - except that clashes with the `datasette client` command that https://github.com/simonw/dclient adds (which is a tool for hitting remote Datasette instances, not running simulated queries through a local one).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option, https://github.com/simonw/datasette/issues/2153#issuecomment-1691753489,https://api.github.com/repos/simonw/datasette/issues/2153,1691753489,IC_kwDOBm6k_c5k1hwR,9599,simonw,2023-08-24T14:07:25Z,2023-08-24T14:09:16Z,OWNER,"Building that `""_r""` array is the main reason this would be useful, but it's also fiddly to get right. `datasette create-token` has a design for that already: https://docs.datasette.io/en/1.0a4/authentication.html#datasette-create-token ``` datasette create-token root \ --secret mysecret \ --all view-instance \ --all view-table \ --database docs view-query \ --resource docs documents insert-row \ --resource docs documents update-row ``` Adding imitations of those options (excluding `--secret`, not needed here) to `datasette serve` would add a LOT of extra options, but it would also make it really convenient to attempt a request with a specific set of restrictions. Not sure if that would be worth the extra `--help` output or not. I feel like the names would have to have a common prefix though. Maybe something like this: ```bash datasette serve data.db --get `/data/mytable.json' \ --actor-id root \ --r-all view-instance \ --r-database data view-query \ --r-resource data documents update-row ``` Other options could be the longer `--restrict-all/--restrict-database/--restrict-resource`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865232341,Datasette --get --actor option, https://github.com/simonw/datasette/pull/2152#issuecomment-1695736691,https://api.github.com/repos/simonw/datasette/issues/2152,1695736691,IC_kwDOBm6k_c5lEuNz,49699333,dependabot[bot],2023-08-28T13:49:35Z,2023-08-28T13:49:35Z,CONTRIBUTOR,Superseded by #2160.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,Bump the python-packages group with 3 updates, https://github.com/simonw/datasette/pull/2152#issuecomment-1691767797,https://api.github.com/repos/simonw/datasette/issues/2152,1691767797,IC_kwDOBm6k_c5k1lP1,9599,simonw,2023-08-24T14:15:10Z,2023-08-24T14:15:10Z,OWNER,This is broken because Sphinx no longer supports Python 3.8.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,Bump the python-packages group with 3 updates, https://github.com/simonw/datasette/pull/2152#issuecomment-1691710474,https://api.github.com/repos/simonw/datasette/issues/2152,1691710474,IC_kwDOBm6k_c5k1XQK,22429695,codecov[bot],2023-08-24T13:45:21Z,2023-08-25T13:18:34Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`527cec6`)](https://app.codecov.io/gh/simonw/datasette/commit/527cec66b0403e689c8fb71fc8b381a1d7a46516?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76% compared to head [(`2e45686`)](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.76%. > :exclamation: Current head 2e45686 differs from pull request most recent head 5dfa305. Consider uploading reports for the commit 5dfa305 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2152 +/- ## ======================================= Coverage 92.76% 92.76% ======================================= Files 40 40 Lines 5943 5919 -24 ======================================= - Hits 5513 5491 -22 + Misses 430 428 -2 ``` [see 3 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2152/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2152?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1865174661,Bump the python-packages group with 3 updates, https://github.com/simonw/datasette/pull/2151#issuecomment-1690746371,https://api.github.com/repos/simonw/datasette/issues/2151,1690746371,IC_kwDOBm6k_c5kxr4D,22429695,codecov[bot],2023-08-23T22:51:57Z,2023-08-23T22:58:13Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`bdf59eb`)](https://app.codecov.io/gh/simonw/datasette/commit/bdf59eb7db42559e538a637bacfe86d39e5d17ca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73% compared to head [(`b895cd2`)](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2151 +/- ## ======================================= Coverage 92.73% 92.73% ======================================= Files 40 40 Lines 5919 5919 ======================================= Hits 5489 5489 Misses 430 430 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2151?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1864112887,Test Datasette on multiple SQLite versions, https://github.com/simonw/datasette/issues/2150#issuecomment-1690438270,https://api.github.com/repos/simonw/datasette/issues/2150,1690438270,IC_kwDOBm6k_c5kwgp-,9599,simonw,2023-08-23T18:27:47Z,2023-08-23T18:27:47Z,OWNER,"I added `outline: 3px dotted pink` to that `form label` block to help spot where else it's being used. Oh interesting, looks like it's over-ridden here too: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default, https://github.com/simonw/datasette/issues/2150#issuecomment-1690435866,https://api.github.com/repos/simonw/datasette/issues/2150,1690435866,IC_kwDOBm6k_c5kwgEa,9599,simonw,2023-08-23T18:25:51Z,2023-08-23T18:25:51Z,OWNER,"Looks like that affects a few forms: The search form on the table page over-rides it already: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default, https://github.com/simonw/datasette/issues/2150#issuecomment-1690432928,https://api.github.com/repos/simonw/datasette/issues/2150,1690432928,IC_kwDOBm6k_c5kwfWg,9599,simonw,2023-08-23T18:23:26Z,2023-08-23T18:23:26Z,OWNER,"That should be scoped just to the query filters form on the table page. I'll fix it in `main` but I'm still going to ship a fix for those plugins separately so they work well before people upgrade to the next Datasette release with this change in it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default, https://github.com/simonw/datasette/issues/2150#issuecomment-1690431509,https://api.github.com/repos/simonw/datasette/issues/2150,1690431509,IC_kwDOBm6k_c5kwfAV,9599,simonw,2023-08-23T18:22:47Z,2023-08-23T18:22:47Z,OWNER,https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/static/app.css#L485-L489,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1863810783,form label { width: 15% } is a bad default, https://github.com/simonw/datasette/pull/2149#issuecomment-1689125244,https://api.github.com/repos/simonw/datasette/issues/2149,1689125244,IC_kwDOBm6k_c5krgF8,9599,simonw,2023-08-23T01:23:27Z,2023-08-23T01:23:27Z,OWNER,"This is a really great start - tests pass, implementation looks clean, the new stubbed documentation page makes sense. Let's land it in `main` and iterate on it further in future PRs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1861812208,"Start a new `datasette.yaml` configuration file, with settings support", https://github.com/simonw/datasette/pull/2149#issuecomment-1688547401,https://api.github.com/repos/simonw/datasette/issues/2149,1688547401,IC_kwDOBm6k_c5kpTBJ,22429695,codecov[bot],2023-08-22T16:30:41Z,2023-08-22T16:30:41Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`76.47%`** and project coverage change: **`+0.10%`** :tada: > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`db720cd`)](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.17%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2149 +/- ## ========================================== + Coverage 92.06% 92.17% +0.10% ========================================== Files 40 40 Lines 5937 5916 -21 ========================================== - Hits 5466 5453 -13 + Misses 471 463 -8 ``` | [Files Changed](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/cli.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `81.02% <25.00%> (+1.20%)` | :arrow_up: | | [datasette/app.py](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.29% <92.30%> (-0.09%)` | :arrow_down: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2149?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1861812208,"Start a new `datasette.yaml` configuration file, with settings support", https://github.com/simonw/datasette/pull/2148#issuecomment-1696591957,https://api.github.com/repos/simonw/datasette/issues/2148,1696591957,IC_kwDOBm6k_c5lH_BV,49699333,dependabot[bot],2023-08-29T00:15:29Z,2023-08-29T00:15:29Z,CONTRIBUTOR,This pull request was built based on a group rule. Closing it will not ignore any of these versions in future pull requests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1696591943,https://api.github.com/repos/simonw/datasette/issues/2148,1696591943,IC_kwDOBm6k_c5lH_BH,9599,simonw,2023-08-29T00:15:27Z,2023-08-29T00:15:27Z,OWNER,"Now solving this here: - #2160 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1689198413,https://api.github.com/repos/simonw/datasette/issues/2148,1689198413,IC_kwDOBm6k_c5krx9N,49699333,dependabot[bot],2023-08-23T02:57:55Z,2023-08-23T02:57:55Z,CONTRIBUTOR,"Looks like this PR has been edited by someone other than Dependabot. That means Dependabot can't rebase it - sorry! If you're happy for Dependabot to recreate it from scratch, overwriting any edits, you can request `@dependabot recreate`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1689198368,https://api.github.com/repos/simonw/datasette/issues/2148,1689198368,IC_kwDOBm6k_c5krx8g,9599,simonw,2023-08-23T02:57:53Z,2023-08-23T02:57:53Z,OWNER,@dependabot rebase,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1689177556,https://api.github.com/repos/simonw/datasette/issues/2148,1689177556,IC_kwDOBm6k_c5krs3U,9599,simonw,2023-08-23T02:44:29Z,2023-08-23T02:44:29Z,OWNER,"Simplest possible solution is to only run the `pip install .[docs]` bit under Python 3.9+, ditto for the docs tests. I think I'll try that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1689175062,https://api.github.com/repos/simonw/datasette/issues/2148,1689175062,IC_kwDOBm6k_c5krsQW,9599,simonw,2023-08-23T02:40:46Z,2023-08-23T02:40:46Z,OWNER,"Here's why the tests are failing: https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/.github/workflows/test.yml#L30-L46 It looks like those tests don't actually need Sphinx installed - they install `pip install -e '.[test,docs]'` to get the other docs dependencies: https://github.com/simonw/datasette/blob/2ce7872e3ba8d07248c194ef554bbdc1df510f32/setup.py#L70-L80","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1689173748,https://api.github.com/repos/simonw/datasette/issues/2148,1689173748,IC_kwDOBm6k_c5krr70,9599,simonw,2023-08-23T02:38:31Z,2023-08-23T02:38:31Z,OWNER,"Sphinx dropped support for Python 3.8 in their version 7.2.0: - https://www.sphinx-doc.org/en/master/changes.html#release-7-2-0-released-aug-17-2023 - https://www.sphinx-doc.org/en/master/internals/release-process.html#python-version-support-policy - https://github.com/sphinx-doc/sphinx/pull/11511 - ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1689133247,https://api.github.com/repos/simonw/datasette/issues/2148,1689133247,IC_kwDOBm6k_c5kriC_,22429695,codecov[bot],2023-08-23T01:36:42Z,2023-08-23T03:09:43Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage has no change and project coverage change: **`+0.01%`** :tada: > Comparison is base [(`17ec309`)](https://app.codecov.io/gh/simonw/datasette/commit/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.71% compared to head [(`a651d3c`)](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.73%. > Report is 2 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2148 +/- ## ========================================== + Coverage 92.71% 92.73% +0.01% ========================================== Files 40 40 Lines 5919 5919 ========================================== + Hits 5488 5489 +1 + Misses 431 430 -1 ``` [see 1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2148/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2148?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/pull/2148#issuecomment-1689127479,https://api.github.com/repos/simonw/datasette/issues/2148,1689127479,IC_kwDOBm6k_c5krgo3,9599,simonw,2023-08-23T01:26:53Z,2023-08-23T01:26:53Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1859415334,"Bump sphinx, furo, blacken-docs dependencies", https://github.com/simonw/datasette/issues/2147#issuecomment-1690955706,https://api.github.com/repos/simonw/datasette/issues/2147,1690955706,IC_kwDOBm6k_c5kye-6,18899,jackowayed,2023-08-24T03:54:35Z,2023-08-24T03:54:35Z,NONE,"That's fair. The best idea I can think of is that if a plugin wanted to limit intensive queries, it could add LIMITs or something. A hook that gives you visibility of queries and maybe the option to reject felt a little more limited than the existing plugin hooks, so I was trying to think of what else one might want to do while looking at to-be-run queries. But without a real motivating example, I see why you don't want to add that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run, https://github.com/simonw/datasette/issues/2147#issuecomment-1689206768,https://api.github.com/repos/simonw/datasette/issues/2147,1689206768,IC_kwDOBm6k_c5krz_w,9599,simonw,2023-08-23T03:06:32Z,2023-08-23T03:06:32Z,OWNER,"I'm less convinced by the ""rewrite the query in some way"" optional idea. What kind of use-cases can you imagine for that? My hunch is that it's much more likely to cause weird breakages than it is to allow for useful plugin extensions, but I'm willing to be convinced otherwise.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run, https://github.com/simonw/datasette/issues/2147#issuecomment-1689206170,https://api.github.com/repos/simonw/datasette/issues/2147,1689206170,IC_kwDOBm6k_c5krz2a,9599,simonw,2023-08-23T03:05:32Z,2023-08-23T03:05:32Z,OWNER,"Interestingly enough there's actually a mechanism that looks like that a bit already: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/views/database.py#L496-L508 That `validate_sql_select()` function is defined here: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L256-L265 Against these constants: https://github.com/simonw/datasette/blob/64fd1d788eeed2624f107ac699f2370590ae1aa3/datasette/utils/__init__.py#L223-L253 Which isn't a million miles away from your suggestion to have a hook that can say if the query should be executed or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run, https://github.com/simonw/datasette/issues/2147#issuecomment-1687433388,https://api.github.com/repos/simonw/datasette/issues/2147,1687433388,IC_kwDOBm6k_c5klDCs,18899,jackowayed,2023-08-22T05:05:33Z,2023-08-22T05:05:33Z,NONE,"Thanks for all this! You're totally right that the ASGI option is doable, if a bit low level and coupled to the current URI design. I'm totally fine with that being the final answer. process_view is interesting and in the general direction of what I had in mind. A somewhat less powerful idea: Is there value in giving a hook for just the query that's about to be run? Maybe I'm thinking a little narrowly about this problem I decided I wanted to solve, but I could see other uses for a hook of the sketch below: ``` def prepare_query(database, table, query): """"""Modify query that is about to be run in some way. Return the (possibly rewritten) query to run, or None to disallow running the query"""""" ``` (Maybe you actually want to return a tuple so there can be an error message when you disallow, or something.) Maybe it's too narrowly useful and some of the other pieces of datasette obviate some of these ideas, but off the cuff I could imagine using it to: * Require a LIMIT. Either fail the query or add the limit if it's not there. * Do logging, like my usecase. * Do other analysis on whether you want to allow the query to run; a linter? query complexity? Definitely feel free to say no, or not now. This is all me just playing around with what datasette and its plugin architecture can do with toy ideas, so don't let me push you to commit to a hook you don't feel confident fits well in the design.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run, https://github.com/simonw/datasette/issues/2147#issuecomment-1686749342,https://api.github.com/repos/simonw/datasette/issues/2147,1686749342,IC_kwDOBm6k_c5kicCe,9599,simonw,2023-08-21T17:33:11Z,2023-08-21T17:33:11Z,OWNER,"I'm definitely open to suggestions for plugin hooks that might make this kind of thing easier. One idea I've been mulling is whether there should be a plugin hook that files on arbitrary views - similar to Django's `process_view` mechanism: https://docs.djangoproject.com/en/4.2/topics/http/middleware/#process-view That would allow people to setup code that runs before or after any of the default views in Datasette. I'm not yet 100% sold on the idea, because I worry about implementing it in a way that guarantees plugins won't break on future releases. But I'm open to considering it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run, https://github.com/simonw/datasette/issues/2147#issuecomment-1686747420,https://api.github.com/repos/simonw/datasette/issues/2147,1686747420,IC_kwDOBm6k_c5kibkc,9599,simonw,2023-08-21T17:31:42Z,2023-08-21T17:34:19Z,OWNER,"Are you talking just about queries submitted to `/database?sql=` using the interface on https://latest.datasette.io/fixtures?sql=select+*+from+facetable or are you interested in queries that are run to power other pages like https://latest.datasette.io/fixtures/facetable as well? I'll assume the former. There are a few ways you could solve this at the moment. The easiest would be with a piece of ASGI middleware that looks for URLs matching `/dbname?sql=...` and logs those. I played with a version of that a few years ago: https://simonwillison.net/2019/Dec/16/logging-sqlite-asgi-middleware/ - see also https://github.com/simonw/asgi-log-to-sqlite Then you can load that middleware from a plugin using https://docs.datasette.io/en/stable/plugin_hooks.html#asgi-wrapper-datasette That feels a bit delicate because it's relying on the URL design not changing, but I'm happy to confirm that URL is going to stay the same for Datasette 1.0 and I have no plans to change it ever. There's also a tracing mechanism built into Datasette itself that you could hook into. The internals of that are documented here: https://docs.datasette.io/en/stable/internals.html#datasette-tracer - but I don't yet consider it a 100% stable API. I don't plan to change it but I won't promise not to either. I used that mechanism in this plugin: https://datasette.io/plugins/datasette-pretty-traces - demonstrated here: https://latest-with-plugins.datasette.io/github/commits?_trace=1 The hackiest way to do this would be to patch Datasette itself and try to replace the `query_view`. This definitely isn't a documented, stable API though and would be very likely to break at arbitrary points in the future. So my recommendation for the moment is the ASGI middleware option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1858228057,Plugin hook for database queries that are run, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096381,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096381,IC_kwDOCGYnMM5kcIe9,9599,simonw,2023-08-19T20:04:32Z,2023-08-19T20:04:32Z,OWNER,I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096284,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096284,IC_kwDOCGYnMM5kcIdc,9599,simonw,2023-08-19T20:03:59Z,2023-08-19T20:03:59Z,OWNER,"Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in: - #577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096129,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096129,IC_kwDOCGYnMM5kcIbB,9599,simonw,2023-08-19T20:03:00Z,2023-08-19T20:03:00Z,OWNER,"Simplest possible recreation of the bug: ```bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute("""""" CREATE TABLE ""logs"" ( [id] INTEGER PRIMARY KEY, [chat_id] INTEGER REFERENCES [log]([id]), [reply_to_id] INTEGER ); """""") db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"") ' ``` That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384,New .add_foreign_key() can break if PRAGMA legacy_alter_table=ON and there's an invalid foreign key reference, https://github.com/simonw/datasette/issues/2145#issuecomment-1686745094,https://api.github.com/repos/simonw/datasette/issues/2145,1686745094,IC_kwDOBm6k_c5kibAG,15178711,asg017,2023-08-21T17:30:01Z,2023-08-21T17:30:01Z,CONTRIBUTOR,"Another point: The new Datasette write API should refuse to insert a row with a NULL primary key. That will likely decrease the likelihood someone find themselves with NULLs in their primary keys, at least with Datasette users. Especially buggy code that uses the write API, like our `datasette-write-ui` bug that led to this issue. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1686683596,https://api.github.com/repos/simonw/datasette/issues/2145,1686683596,IC_kwDOBm6k_c5kiL_M,9599,simonw,2023-08-21T16:49:12Z,2023-08-21T16:49:12Z,OWNER,"Suggestion from @asg017 is that we say that if your row has a null primary key you don't get a link to a row page for that row. Which has some precedent, because our SQL view display doesn't link to row pages at all (since they don't make sense for views): https://latest.datasette.io/fixtures/simple_view","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1685471752,https://api.github.com/repos/simonw/datasette/issues/2145,1685471752,IC_kwDOBm6k_c5kdkII,77071,pkulchenko,2023-08-21T01:07:23Z,2023-08-21T01:07:23Z,NONE,"@simonw, since you're referencing ""rowid"" column by name, I just want to note that there may be an existing rowid column with completely different semantics (https://www.sqlite.org/lang_createtable.html#rowid), which is likely to break this logic. I don't see a good way to detect a proper ""rowid"" name short of checking if there is a field with that name and using the alternative (`_rowid_` or `oid`), which is not ideal, but may work. In terms of the original issue, maybe a way to deal with it is to use rowid by default and then use primary key for WITHOUT ROWID tables (as they are guaranteed to be not null), but I suspect it may require significant changes to the API (and doesn't fully address the issue of what value to pass to indicate NULL when editing records). Would it make sense to generate a random string to indicate NULL values when editing?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684530060,https://api.github.com/repos/simonw/datasette/issues/2145,1684530060,IC_kwDOBm6k_c5kZ-OM,9599,simonw,2023-08-18T23:09:03Z,2023-08-18T23:09:14Z,OWNER,"Ran a quick benchmark on ChatGPT Code Interpreter: https://chat.openai.com/share/8357dc01-a97e-48ae-b35a-f06249935124 Conclusion from there is that this query returns fast no matter how much the table grows: ```sql SELECT EXISTS(SELECT 1 FROM ""nasty"" WHERE ""id"" IS NULL) ``` So detecting if a table contains any null primary keys is definitely feasible without a performance hit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684526447,https://api.github.com/repos/simonw/datasette/issues/2145,1684526447,IC_kwDOBm6k_c5kZ9Vv,9599,simonw,2023-08-18T23:05:02Z,2023-08-18T23:05:02Z,OWNER,How expensive is it to detect if a SQLite table contains at least one `null` primary key?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684525943,https://api.github.com/repos/simonw/datasette/issues/2145,1684525943,IC_kwDOBm6k_c5kZ9N3,9599,simonw,2023-08-18T23:04:14Z,2023-08-18T23:04:14Z,OWNER,"This is hard. I tried this: ```python def path_from_row_pks(row, pks, use_rowid, quote=True): """"""Generate an optionally tilde-encoded unique identifier for a row from its primary keys."""""" if use_rowid or any(row[pk] is None for pk in pks): bits = [row[""rowid""]] else: bits = [ row[pk][""value""] if isinstance(row[pk], dict) else row[pk] for pk in pks ] if quote: bits = [tilde_encode(str(bit)) for bit in bits] else: bits = [str(bit) for bit in bits] return "","".join(bits) ``` The ` if use_rowid or any(row[pk] is None for pk in pks)` bit is new. But I got this error on http://127.0.0.1:8003/nulls/nasty : ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 1364, in run_display_columns_and_rows display_columns, display_rows = await display_columns_and_rows( File ""/Users/simon/Dropbox/Development/datasette/datasette/views/table.py"", line 186, in display_columns_and_rows pk_path = path_from_row_pks(row, pks, not pks, False) File ""/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py"", line 124, in path_from_row_pks bits = [row[""rowid""]] IndexError: No item with that key ``` Because the SQL query I ran to populate the page didn't know that it would need to select `rowid` as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684525054,https://api.github.com/repos/simonw/datasette/issues/2145,1684525054,IC_kwDOBm6k_c5kZ8_-,9599,simonw,2023-08-18T23:02:26Z,2023-08-18T23:02:26Z,OWNER,"Creating a quick test database: ```bash sqlite-utils create-table nulls.db nasty id text --pk id sqlite-utils nulls.db 'insert into nasty (id) values (null)' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684523322,https://api.github.com/repos/simonw/datasette/issues/2145,1684523322,IC_kwDOBm6k_c5kZ8k6,9599,simonw,2023-08-18T22:59:14Z,2023-08-18T22:59:14Z,OWNER,"Except it looks like the Links from other tables section is broken: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684522567,https://api.github.com/repos/simonw/datasette/issues/2145,1684522567,IC_kwDOBm6k_c5kZ8ZH,9599,simonw,2023-08-18T22:58:07Z,2023-08-18T22:58:07Z,OWNER,"Here's a prototype of that: ```diff diff --git a/datasette/app.py b/datasette/app.py index b2644ace..acc55249 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1386,7 +1386,7 @@ class Datasette: ) add_route( RowView.as_view(self), - r""/(?P[^\/\.]+)/(?P
[^/]+?)/(?P[^/]+?)(\.(?P\w+))?$"", + r""/(?P[^\/\.]+)/(?P
[^/]+?)/(?P[A-Za-z0-9\_\-\~]+|\.\d+)(\.(?P\w+))?$"", ) add_route( TableInsertView.as_view(self), @@ -1440,7 +1440,15 @@ class Datasette: async def resolve_row(self, request): db, table_name, _ = await self.resolve_table(request) pk_values = urlsafe_components(request.url_vars[""pks""]) - sql, params, pks = await row_sql_params_pks(db, table_name, pk_values) + + if len(pk_values) == 1 and pk_values[0].startswith("".""): + # It's a special .rowid value + pk_values = (pk_values[0][1:],) + sql, params, pks = await row_sql_params_pks( + db, table_name, pk_values, rowid=True + ) + else: + sql, params, pks = await row_sql_params_pks(db, table_name, pk_values) results = await db.execute(sql, params, truncate=True) row = results.first() if row is None: diff --git a/datasette/utils/__init__.py b/datasette/utils/__init__.py index c388673d..96669281 100644 --- a/datasette/utils/__init__.py +++ b/datasette/utils/__init__.py @@ -1206,9 +1206,12 @@ def truncate_url(url, length): return url[: length - 1] + ""…"" -async def row_sql_params_pks(db, table, pk_values): +async def row_sql_params_pks(db, table, pk_values, rowid=False): pks = await db.primary_keys(table) - use_rowid = not pks + if rowid: + use_rowid = True + else: + use_rowid = not pks select = ""*"" if use_rowid: select = ""rowid, *"" ``` It works: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684505071,https://api.github.com/repos/simonw/datasette/issues/2145,1684505071,IC_kwDOBm6k_c5kZ4Hv,9599,simonw,2023-08-18T22:44:35Z,2023-08-18T22:44:35Z,OWNER,"Also relevant: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/utils/__init__.py#L1147-L1153","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684504398,https://api.github.com/repos/simonw/datasette/issues/2145,1684504398,IC_kwDOBm6k_c5kZ39O,9599,simonw,2023-08-18T22:43:31Z,2023-08-18T22:43:46Z,OWNER,"`(?P[^/]+?)` could instead be a regex that is restricted to the tilde-encoded set of characters, or `\.\d+`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684504051,https://api.github.com/repos/simonw/datasette/issues/2145,1684504051,IC_kwDOBm6k_c5kZ33z,9599,simonw,2023-08-18T22:43:06Z,2023-08-18T22:43:06Z,OWNER,Here's the regex in question at the moment: https://github.com/simonw/datasette/blob/943df09dcca93c3b9861b8c96277a01320db8662/datasette/app.py#L1387-L1390,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684503587,https://api.github.com/repos/simonw/datasette/issues/2145,1684503587,IC_kwDOBm6k_c5kZ3wj,9599,simonw,2023-08-18T22:42:28Z,2023-08-18T22:42:39Z,OWNER,"I could set a rule that extensions (including custom render extensions set by plugins) must not be valid integers, and teach Datasette that `/\.\d+` is the indication of a `rowid`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684503189,https://api.github.com/repos/simonw/datasette/issues/2145,1684503189,IC_kwDOBm6k_c5kZ3qV,9599,simonw,2023-08-18T22:41:51Z,2023-08-18T22:41:51Z,OWNER,"```pycon >>> tilde_encode(""~"") '~7E' >>> tilde_encode(""."") '~2E' >>> tilde_encode(""-"") '-' ``` I think `.` might be the way to do this: /database/table/.4 But... I worry about that colliding with my URL routing code that spots the difference between these: /database/table/.4 /database/table/.4.json /database/table/.4.csv etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684502278,https://api.github.com/repos/simonw/datasette/issues/2145,1684502278,IC_kwDOBm6k_c5kZ3cG,9599,simonw,2023-08-18T22:40:20Z,2023-08-18T22:40:20Z,OWNER,"From reviewing https://simonwillison.net/2022/Mar/19/weeknotes/ unreserved = ALPHA / DIGIT / ""-"" / ""."" / ""_"" / ""~"" That's how I chose the tilde character - but it also suggests that I could use `-` or `.` or `_` for my new `rowid` encoding. So maybe `/database/table/_4` could indicate ""the row with `rowid` of 4"". No, that doesn't work: ```pycon >>> from datasette.utils import tilde_encode >>> tilde_encode(""_"") '_' ``` I need a character which tilde-encoding does indeed encode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684500540,https://api.github.com/repos/simonw/datasette/issues/2145,1684500540,IC_kwDOBm6k_c5kZ3A8,9599,simonw,2023-08-18T22:37:37Z,2023-08-18T22:37:37Z,OWNER,"I just found this and panicked, thinking maybe tilde encoding is a bad idea after all! https://jkorpela.fi/tilde.html But... ""Date of last update: 1999-08-27"" - I think I'm OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684500172,https://api.github.com/repos/simonw/datasette/issues/2145,1684500172,IC_kwDOBm6k_c5kZ27M,9599,simonw,2023-08-18T22:37:04Z,2023-08-18T22:37:04Z,OWNER,"Looking at the way these URLs work: because the components themselves in `a~2Fb,~2Ec-d` are tilde-encoded, any character that's ""safe"" in tilde-encoding could be used to indicate ""this is actually a rowid"". ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684498947,https://api.github.com/repos/simonw/datasette/issues/2145,1684498947,IC_kwDOBm6k_c5kZ2oD,9599,simonw,2023-08-18T22:35:04Z,2023-08-18T22:35:04Z,OWNER,"The most interesting row URL in the fixtures database right now is this one: https://latest.datasette.io/fixtures/compound_primary_key/a~2Fb,~2Ec-d ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684497642,https://api.github.com/repos/simonw/datasette/issues/2145,1684497642,IC_kwDOBm6k_c5kZ2Tq,9599,simonw,2023-08-18T22:32:53Z,2023-08-18T22:32:53Z,OWNER,"Here's a potential solution: make it so ALL `rowid` tables in SQLite can be optionally addressed by their `rowid` instead of by their primary key. Then teach the code that outputs the URL to a row page to spot if there are `null` primary keys and switch to that alternative addressing mechanism instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684497000,https://api.github.com/repos/simonw/datasette/issues/2145,1684497000,IC_kwDOBm6k_c5kZ2Jo,9599,simonw,2023-08-18T22:31:53Z,2023-08-18T22:31:53Z,OWNER,"So it sounds like SQLite does ensure that a `rowid` before it allows a primary key to be null. So one solution here would be to detect a null primary key and switch that table over to using `rowid` URLs instead. The key problem we're trying to solve here after all is how to link to a row: https://latest.datasette.io/fixtures/infinity/1 But when would we run that check? And does every row in the table get a new `/rowid/` URL just because someone messed up and inserted a `null` by mistake?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684495674,https://api.github.com/repos/simonw/datasette/issues/2145,1684495674,IC_kwDOBm6k_c5kZ106,9599,simonw,2023-08-18T22:29:47Z,2023-08-18T22:29:47Z,OWNER,"https://www.sqlite.org/lang_createtable.html#the_primary_key says: >According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid) or the table is a [WITHOUT ROWID](https://www.sqlite.org/withoutrowid.html) table or a [STRICT](https://www.sqlite.org/stricttables.html) table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684494464,https://api.github.com/repos/simonw/datasette/issues/2145,1684494464,IC_kwDOBm6k_c5kZ1iA,9599,simonw,2023-08-18T22:27:51Z,2023-08-18T22:28:40Z,OWNER,"Oh wow, null primary keys are bad news... SQLite lets you insert multiple rows with the same `null` value! ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
>>> db.schema 'CREATE TABLE [foo] (\n [id] TEXT PRIMARY KEY,\n [name] TEXT\n);' >>> db[""foo""].insert({""id"": None, ""name"": ""No ID""}, pk=""id"")
>>> db.schema 'CREATE TABLE [foo] (\n [id] TEXT PRIMARY KEY,\n [name] TEXT\n);' >>> list(db[""foo""].rows) [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}] >>> list(db.query('select * from foo where id = null')) [] >>> list(db.query('select * from foo where id is null')) [{'id': None, 'name': 'No ID'}, {'id': None, 'name': 'No ID'}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/issues/2145#issuecomment-1684384750,https://api.github.com/repos/simonw/datasette/issues/2145,1684384750,IC_kwDOBm6k_c5kZavu,9599,simonw,2023-08-18T20:07:18Z,2023-08-18T20:07:18Z,OWNER,The big challenge here is what the URL to that row page should look like. How can I encode a `None` in a form that can be encoded and decoded without clashing with primary keys that are the string `None` or `null`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857234285,If a row has a primary key of `null` various things break, https://github.com/simonw/datasette/pull/2144#issuecomment-1686366557,https://api.github.com/repos/simonw/datasette/issues/2144,1686366557,IC_kwDOBm6k_c5kg-ld,49699333,dependabot[bot],2023-08-21T13:48:15Z,2023-08-21T13:48:15Z,CONTRIBUTOR,Superseded by #2148.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856760386,Bump the python-packages group with 3 updates, https://github.com/simonw/datasette/pull/2144#issuecomment-1683963463,https://api.github.com/repos/simonw/datasette/issues/2144,1683963463,IC_kwDOBm6k_c5kXz5H,22429695,codecov[bot],2023-08-18T13:58:39Z,2023-08-18T13:58:39Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`3a97755`)](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2144 +/- ## ======================================= Coverage 92.06% 92.06% ======================================= Files 40 40 Lines 5937 5937 ======================================= Hits 5466 5466 Misses 471 471 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2144?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856760386,Bump the python-packages group with 3 updates, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683404978,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683404978,IC_kwDOCGYnMM5kVriy,9599,simonw,2023-08-18T06:13:46Z,2023-08-18T06:13:46Z,OWNER,"I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683398866,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683398866,IC_kwDOCGYnMM5kVqDS,9599,simonw,2023-08-18T06:05:50Z,2023-08-18T06:06:42Z,OWNER,"Options: - Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default? - Read that `PRAGMA` and automatically do that view workaround if it's turned on - Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection? - Try the `transform()` in a transaction, detect the `""error in view""`, `""no such table""`error, if spotted then do the VIEW workaround and try again I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683396150,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683396150,IC_kwDOCGYnMM5kVpY2,9599,simonw,2023-08-18T06:02:18Z,2023-08-18T06:06:31Z,OWNER,"More notes in here: - https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873 Not all Python/SQLite installations exhibit this problem by default! It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view. Here's a one-liner to test if it is on or not: ```bash python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""PRAGMA legacy_alter_table"").fetchall())' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668,.transform() fails to drop column if table is part of a view, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683217284,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683217284,IC_kwDOCGYnMM5kU9uE,9599,simonw,2023-08-18T01:50:21Z,2023-08-18T01:50:21Z,OWNER,"And a test of the `--sql` option: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql ``` Outputs: ```sql CREATE TABLE [places_new_6a705d2f5a13] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent]) SELECT [id], [name], [country], [city], [continent] FROM [places]; DROP TABLE [places]; ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places]; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683212074,IC_kwDOCGYnMM5kU8cq,9599,simonw,2023-08-18T01:43:54Z,2023-08-18T01:43:54Z,OWNER,"Some manual testing: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); ``` ```bash sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); ``` ```bash sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); CREATE TABLE ""places"" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); ``` ```bash sqlite-utils transform /tmp/t.db places --drop-foreign-key country sqlite-utils schema /tmp/t.db places ``` ```sql CREATE TABLE ""places"" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683201239,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683201239,IC_kwDOCGYnMM5kU5zX,9599,simonw,2023-08-18T01:30:46Z,2023-08-18T01:30:46Z,OWNER,"Help can now look like this: ``` --drop-foreign-key TEXT Drop foreign key constraint for this column --add-foreign-key ... Add a foreign key constraint from a column to another table with another column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683200128,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683200128,IC_kwDOCGYnMM5kU5iA,9599,simonw,2023-08-18T01:29:00Z,2023-08-18T01:29:00Z,OWNER,I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683198740,IC_kwDOCGYnMM5kU5MU,9599,simonw,2023-08-18T01:26:47Z,2023-08-18T01:26:47Z,OWNER,"The only CLI feature that supports providing just the column name appears to be this: ```bash sqlite-utils add-foreign-key --help ``` ``` Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE] [OTHER_COLUMN] Add a new foreign key constraint to an existing table Example: sqlite-utils add-foreign-key my.db books author_id authors id WARNING: Could corrupt your database! Back up your database file first. ``` I can drop that WARNING now since I'm not writing to `sqlite_master` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683197882,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683197882,IC_kwDOCGYnMM5kU4-6,9599,simonw,2023-08-18T01:25:53Z,2023-08-18T01:25:53Z,OWNER,"Probably most relevant here is this snippet from: ```bash sqlite-utils create-table --help ``` ``` --default ... Default value that should be set for a column --fk ... Column, other table, other column to set as a foreign key ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683195669,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683195669,IC_kwDOCGYnMM5kU4cV,9599,simonw,2023-08-18T01:24:57Z,2023-08-18T01:24:57Z,OWNER,"Currently: ```bash sqlite-utils transform --help ``` ``` Usage: sqlite-utils transform [OPTIONS] PATH TABLE Transform a table beyond the capabilities of ALTER TABLE Example: sqlite-utils transform mydb.db mytable \ --drop column1 \ --rename column2 column_renamed Options: --type ... Change column type to INTEGER, TEXT, FLOAT or BLOB --drop TEXT Drop this column --rename ... Rename this column to X -o, --column-order TEXT Reorder columns --not-null TEXT Set this column to NOT NULL --not-null-false TEXT Remove NOT NULL from this column --pk TEXT Make this column the primary key --pk-none Remove primary key (convert to rowid table) --default ... Set default value for this column --default-none TEXT Remove default from this column --drop-foreign-key TEXT Drop foreign key constraint for this column --sql Output SQL without executing it --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222,CLI equivalents to `transform(add_foreign_keys=)`, https://github.com/simonw/datasette/issues/2143#issuecomment-1692210044,https://api.github.com/repos/simonw/datasette/issues/2143,1692210044,IC_kwDOBm6k_c5k3RN8,9599,simonw,2023-08-24T18:28:27Z,2023-08-24T18:28:27Z,OWNER,"Just spotted this: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L328-L332 https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L359-L360 Looks to me like that second bit of code doesn't yet handle `datasette.yml` This code does though: https://github.com/simonw/datasette/blob/17ec309e14f9c2e90035ba33f2f38ecc5afba2fa/datasette/app.py#L333-L335 `parse_metadata()` is clearly a bad name for this function: https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/utils/__init__.py#L980-L990 That ` @documented` decorator indicates that it's part of the documented API used by plugin authors: https://docs.datasette.io/en/1.0a4/internals.html#parse-metadata-content So we should rename it to something better like `parse_json_or_yaml()` but keep `parse_metadata` as an undocumented alias for that to avoid any unnecessary plugin breaks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1692182910,https://api.github.com/repos/simonw/datasette/issues/2143,1692182910,IC_kwDOBm6k_c5k3Kl-,9599,simonw,2023-08-24T18:06:57Z,2023-08-24T18:08:17Z,OWNER,"The other thing that could work is something like this: ```bash export AUTH_TOKENS_DB=""tokens"" datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -e plugins.datasette-auth-tokens.manage_tokens_database AUTH_TOKENS_DB ``` So `-e` is an alternative version of `-s` which reads from the named environment variable instead of having the value provided directly as the second value in the pair. I quite like this, because it could replace the really ugly `$ENV` pattern we have in plugin configuration at the moment: https://docs.datasette.io/en/1.0a4/plugins.html#secret-configuration-values ```yaml plugins: datasette-auth-github: client_secret: $env: GITHUB_CLIENT_SECRET ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1692180683,https://api.github.com/repos/simonw/datasette/issues/2143,1692180683,IC_kwDOBm6k_c5k3KDL,9599,simonw,2023-08-24T18:05:17Z,2023-08-24T18:05:17Z,OWNER,"That's a really good call, thanks @rclement - environment variable configuration totally makes sense here. Need to figure out the right syntax for that. Something like this perhaps: ```bash DATASETTE_CONFIG_PLUGINS='{""datasette-ripgrep"": ...}' ``` Hard to know how to make this nestable though. I considered this: ```bash DATASETTE_CONFIG_PLUGINS_DATASETTE_RIPGREP_PATH='/path/to/code/' ``` But that doesn't work, because how does the processing code know that it should split on `_` for most of the tokens but NOT split `DATASETTE_RIPGREP`, instead treating that as `datasette-ripgrep`? I checked and `-` is not a valid character in an environment variable, at least in zsh on macOS: ``` % export FOO_BAR-BAZ=1 export: not valid in this context: FOO_BAR-BAZ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1691094870,https://api.github.com/repos/simonw/datasette/issues/2143,1691094870,IC_kwDOBm6k_c5kzA9W,1238873,rclement,2023-08-24T06:43:40Z,2023-08-24T06:43:40Z,NONE,"If I may, the ""path-like"" configuration is great but one thing that would be even greater: allowing the same configuration to be provided using environment variables. For instance: ``` datasette -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}' ``` could also be provided using: ``` export DS_PLUGINS_DATASETTE-COMPLEX-PLUGIN_CONFIGS='{""foo"": [1,2,3], ""bar"": ""baz""}' datasette ``` (I do not like mixing `-` and `_` in env vars but I do not have a best easily reversible example at the moment) FYI, you could take some inspiration from another great open source data project, Metabase: https://www.metabase.com/docs/latest/configuring-metabase/config-file https://www.metabase.com/docs/latest/configuring-metabase/environment-variables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690800641,https://api.github.com/repos/simonw/datasette/issues/2143,1690800641,IC_kwDOBm6k_c5kx5IB,9599,simonw,2023-08-24T00:11:16Z,2023-08-24T00:11:16Z,OWNER,"> @simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case. That's a neat example thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690800119,https://api.github.com/repos/simonw/datasette/issues/2143,1690800119,IC_kwDOBm6k_c5kx4_3,9599,simonw,2023-08-24T00:10:32Z,2023-08-24T00:39:00Z,OWNER,"Something notable about this design is that, because the values in the key-value pairs are treated as JSON first and then strings only if they don't parse cleanly as JSON, it's possible to represent any structure (including nesting structures) using this syntax. You can do things like this if you need to (settings for an imaginary plugin): ```bash datasette data.db \ -s plugins.datasette-complex-plugin.configs '{""foo"": [1,2,3], ""bar"": ""baz""}' ``` Which would be equivalent to: ```yaml plugins: datasette-complex-plugin: configs: foo: - 1 - 2 - 3 bar: baz ``` This is a bit different from a previous attempt I made at the same problem: https://github.com/simonw/json-flatten - that used syntax like `foo.bar.[0]$int = 1` to specify an integer as the first item of an array, which is much more complex. That previous design was meant to support round-trips, so you could take any nested JSON object and turn it into an HTMl form or query string where every value can have its own form field, then turn the result back again. For the `datasette -s key value` feature we don't need round-tripping with individual values each editable on their own, so we can go with something much simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690799608,https://api.github.com/repos/simonw/datasette/issues/2143,1690799608,IC_kwDOBm6k_c5kx434,77071,pkulchenko,2023-08-24T00:09:47Z,2023-08-24T00:10:41Z,NONE,"@simonw, FWIW, I do exactly the same thing for one of my projects (both to allow multiple configuration files to be passed on the command line and setting individual values) and it works quite well for me and my users. I even use the same parameter name for both (https://studio.zerobrane.com/doc-configuration#configuration-via-command-line), but I understand why you may want to use different ones for files and individual values. There is one small difference that I accept code snippets, but I don't think it matters much in this case.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690792514,https://api.github.com/repos/simonw/datasette/issues/2143,1690792514,IC_kwDOBm6k_c5kx3JC,9599,simonw,2023-08-24T00:00:16Z,2023-08-24T00:02:55Z,OWNER,"I've been thinking about what it might look like to allow command-line arguments to be used to define _any_ of the configuration options in `datasette.yml`, as alternative and more convenient syntax. Here's what I've come up with: ``` datasette \ -s settings.sql_time_limit_ms 1000 \ -s plugins.datasette-auth-tokens.manage_tokens true \ -s plugins.datasette-auth-tokens.manage_tokens_database tokens \ -s plugins.datasette-ripgrep.path ""/home/simon/code-to-search"" \ -s databases.mydatabase.tables.example_table.sort created \ mydatabase.db tokens.db ``` Which would be equivalent to `datasette.yml` containing this: ```yaml plugins: datasette-auth-tokens: manage_tokens: true manage_tokens_database: tokens datasette-ripgrep: path: /home/simon/code-to-search databases: mydatabase: tables: example_table: sort: created settings: sql_time_limit_ms: 1000 ``` Here's a prototype implementation of this: ```python import json from typing import Any, List, Tuple def _handle_pair(key: str, value: str) -> dict: """""" Turn a key-value pair into a nested dictionary. foo, bar => {'foo': 'bar'} foo.bar, baz => {'foo': {'bar': 'baz'}} foo.bar, [1, 2, 3] => {'foo': {'bar': [1, 2, 3]}} foo.bar, ""baz"" => {'foo': {'bar': 'baz'}} foo.bar, '{""baz"": ""qux""}' => {'foo': {'bar': ""{'baz': 'qux'}""}} """""" try: value = json.loads(value) except json.JSONDecodeError: # If it doesn't parse as JSON, treat it as a string pass keys = key.split('.') result = current_dict = {} for k in keys[:-1]: current_dict[k] = {} current_dict = current_dict[k] current_dict[keys[-1]] = value return result def _combine(base: dict, update: dict) -> dict: """""" Recursively merge two dictionaries. """""" for key, value in update.items(): if isinstance(value, dict) and key in base and isinstance(base[key], dict): base[key] = _combine(base[key], value) else: base[key] = value return base def handle_pairs(pairs: List[Tuple[str, Any]]) -> dict: """""" Parse a list of key-value pairs into a nested dictionary. """""" result = {} for key, value in pairs: parsed_pair = _handle_pair(key, value) result = _combine(result, parsed_pair) return result ``` Exercised like this: ```python print(json.dumps(handle_pairs([ (""settings.sql_time_limit_ms"", ""1000""), (""plugins.datasette-auth-tokens.manage_tokens"", ""true""), (""plugins.datasette-auth-tokens.manage_tokens_database"", ""tokens""), (""plugins.datasette-ripgrep.path"", ""/home/simon/code-to-search""), (""databases.mydatabase.tables.example_table.sort"", ""created""), ]), indent=4)) ``` Output: ```json { ""settings"": { ""sql_time_limit_ms"": 1000 }, ""plugins"": { ""datasette-auth-tokens"": { ""manage_tokens"": true, ""manage_tokens_database"": ""tokens"" }, ""datasette-ripgrep"": { ""path"": ""/home/simon/code-to-search"" } }, ""databases"": { ""mydatabase"": { ""tables"": { ""example_table"": { ""sort"": ""created"" } } } } } ``` Note that `-s` isn't currently an option for `datasette serve`. `--setting key value` IS an existing option, but it isn't completely compatible with this because it maps directly just to settings. Although... we could keep compatibility by saying that if you call `--setting known_setting value` and that `known_setting` is in this list then we treat it as if you said `-s settings.known_setting value` instead: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/app.py#L114-L204","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1690787394,https://api.github.com/repos/simonw/datasette/issues/2143,1690787394,IC_kwDOBm6k_c5kx15C,9599,simonw,2023-08-23T23:52:02Z,2023-08-23T23:52:02Z,OWNER,"> This also makes it simple to separate out secrets. > > `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml` Having multiple configs that combine in that way is a really interesting direction. > To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight. I'm very keen on separating out the ""metadata"" - where metadata is the slimmest possible set of things, effectively the data license and the source and the column and table descriptions - from everything else, mainly because I want metadata to be able to travel with the data. One idea that's been discussed before is having an optional mechanism for storing metadata in the SQLite database file itself - potentially in a `_datasette_metadata` table. That way you could distribute a DB file and anyone who opened it in Datasette would also see the correct metadata about it. That's why I'm so keen on splitting out metadata from all of the other stuff - settings and plugin configuration and authentication rules. So really it becomes ""true metadata"" v.s. ""all of the other junk that's accumulated in metadata and `settings.json`"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685263948,https://api.github.com/repos/simonw/datasette/issues/2143,1685263948,IC_kwDOBm6k_c5kcxZM,11784304,dvizard,2023-08-20T11:50:10Z,2023-08-20T11:50:10Z,NONE,"This also makes it simple to separate out secrets. `datasette --config settings.yaml --config secrets.yaml --config db-docs.yaml --config db-fixtures.yaml` settings.yaml ``` settings: default_page_size: 10 max_returned_rows: 3000 sql_time_limit_ms"": 8000 plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages ``` secrets.yaml ``` plugins: datasette-auth-github: client_secret: SUCH_SECRET ``` db-docs.yaml ``` databases: docs: permissions: create-table: id: editor ``` db-fixtures.yaml ``` databases: fixtures: tables: no_primary_key: hidden: true queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; title: Search neighborhoods description_html: |-

This demonstrates basic LIKE search ```","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685260624,https://api.github.com/repos/simonw/datasette/issues/2143,1685260624,IC_kwDOBm6k_c5kcwlQ,11784304,dvizard,2023-08-20T11:31:16Z,2023-08-20T11:31:16Z,NONE,https://pypi.org/project/deep-chainmap/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685260244,https://api.github.com/repos/simonw/datasette/issues/2143,1685260244,IC_kwDOBm6k_c5kcwfU,11784304,dvizard,2023-08-20T11:29:00Z,2023-08-20T11:29:00Z,NONE,https://docs.python.org/3/library/collections.html#collections.ChainMap,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1685259985,https://api.github.com/repos/simonw/datasette/issues/2143,1685259985,IC_kwDOBm6k_c5kcwbR,11784304,dvizard,2023-08-20T11:27:21Z,2023-08-20T11:27:21Z,NONE,"To chime in from a poweruser perspective: I'm worried that this is an overengineering trap. Yes, the current solution is somewhat messy. But there are datasette-wide settings, there are database-scope settings, there are table-scope settings etc, but then there are database-scope metadata and table-scope metadata. Trying to cleanly separate ""settings"" from ""configuration"" is, I believe, an uphill fight. Even separating db/table-scope settings from pure descriptive metadata is not always easy. Like, do canned queries belong to database metadata or to settings? Do I need two separate files for this? One pragmatic solution I used in a project is stacking yaml configuration files. Basically, have an arbitrary number of yaml or json settings files that you load in a specified order. Every file adds to the corresponding settings in the earlier-loaded file (if it already existed). I implemented this myself but found later that there is an existing Python ""cascading dict"" type of thing, I forget what it's called. There is a bit of a challenge deciding whether there is ""replacement"" or ""addition"" (I think I pragmatically ran `update` on the second level of the dict but better solutions are certainly possible). This way, one allows separation of settings into different blocks, while not imposing a specific idea of what belongs where that might not apply equally to all cases. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684496274,https://api.github.com/repos/simonw/datasette/issues/2143,1684496274,IC_kwDOBm6k_c5kZ1-S,15178711,asg017,2023-08-18T22:30:45Z,2023-08-18T22:30:45Z,CONTRIBUTOR,"> That said, I do really like a bias towards settings that can be changed at runtime Does this include things like `--settings` values or plugin config? I can totally see being able to update metadata without restarting, but not sure if that would work well with `--setting`, plugin config, or auth/permissions stuff. Well it could work with `--setting` and auth/permissions, with a lot of core changes. But changing plugin config on the fly could be challenging, for plugin authors. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684488526,https://api.github.com/repos/simonw/datasette/issues/2143,1684488526,IC_kwDOBm6k_c5kZ0FO,9599,simonw,2023-08-18T22:18:39Z,2023-08-18T22:18:39Z,OWNER,"> Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings. > Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings. I'm not a fan of that. I feel like software history is full of examples of projects that implemented configuration-as-code and then later regretted it - the most recent example is `setup.py` in Python turning into `pyproject.yaml`, but I feel like I've seen that pattern play out elsewhere too. I don't think having people dynamically generate JSON/YAML for their configuration is a big burden. I'd have to see some very compelling use-cases to convince me otherwise. That said, I do really like a bias towards settings that can be changed at runtime. Datasette has suffered a bit from some settings that can't be easily changed at runtime already - hence my gnarly https://github.com/simonw/datasette-remote-metadata plugin. For things like Datasette Cloud for example the more people can configure without rebooting their container the better! I don't think live reconfiguration at runtime is incompatible with JSON/YAML configuration though. Caddy is one of my favourite examples of software that can be entirely re-configured at runtime by POSTING a big blob of JSON to it: https://caddyserver.com/docs/quick-starts/api ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684485591,https://api.github.com/repos/simonw/datasette/issues/2143,1684485591,IC_kwDOBm6k_c5kZzXX,9599,simonw,2023-08-18T22:14:35Z,2023-08-18T22:14:35Z,OWNER,"Actually there is one thing that I'm not comfortable about with respect to the existing design: the way the database / tables stuff is nested. They assume that the user will attach the database to Datasette using a fixed name - `docs.db` or whatever. But what if we want to support users downloading databases from each other and attaching them to Datasette where those DBs might carry some of their own configuration? Moving metadata into the databases makes sense there, but what about database-specific settings like the default sort order for a table, or configured canned queries? Having those tied to the filename of the database itself feels unpleasant to me. But how else could we handle this?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426,https://api.github.com/repos/simonw/datasette/issues/2143,1684484426,IC_kwDOBm6k_c5kZzFK,9599,simonw,2023-08-18T22:12:52Z,2023-08-18T22:12:52Z,OWNER,"Yeah, I'm convinced by that. There's not point in having both `settings.json` and `datasette.json`. I like `datasette.json` ( / `datasette.yml`) as a name. That can be the file that lives in your config directory too, so if you run `datasette .` in a folder containing `datasette.yml` all of those settings get picked up. Here's a thought for how it could look - I'll go with the YAML format because I expect that to be the default most people use, just because it supports multi-line strings better. I based this on the big example at https://docs.datasette.io/en/1.0a3/metadata.html#using-yaml-for-metadata - and combined some bits from https://docs.datasette.io/en/1.0a3/authentication.html as well. ```yaml title: Demonstrating Metadata from YAML description_html: |-

This description includes a long HTML string

  • YAML is better for embedding HTML strings than JSON!
settings: default_page_size: 10 max_returned_rows: 3000 sql_time_limit_ms"": 8000 databases: docs: permissions: create-table: id: editor fixtures: tables: no_primary_key: hidden: true queries: neighborhood_search: sql: |- select neighborhood, facet_cities.name, state from facetable join facet_cities on facetable.city_id = facet_cities.id where neighborhood like '%' || :text || '%' order by neighborhood; title: Search neighborhoods description_html: |-

This demonstrates basic LIKE search permissions: debug-menu: id: '*' plugins: datasette-ripgrep: path: /usr/local/lib/python3.11/site-packages ``` I'm inclined to say we try to be a super-set of the existing `metadata.yml` format, at least where it makes sense to do so. That way the upgrade path is smooth for people. Also, I don't think the format itself is terrible - it's the name that's the big problem. In this example I've mixed in one extra concept: that `settings:` block with a bunch of settings in it. There are some things in there that look a little bit like metadata - the `title` and `description_html` fields. But _are they_ metadata? The title and description of the overall instance feels like it could be described as general configuration. The stuff for the `query` should live where the query itself is defined. Note that queries can be defined by a plugin hook too: https://docs.datasette.io/en/1.0a3/plugin_hooks.html#canned-queries-datasette-database-actor What do you think? Is this the right direction, or are you thinking there's a more radical redesign that would make sense here?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684205563,https://api.github.com/repos/simonw/datasette/issues/2143,1684205563,IC_kwDOBm6k_c5kYu_7,15178711,asg017,2023-08-18T17:12:54Z,2023-08-18T17:12:54Z,CONTRIBUTOR,"Another option would be, instead of flat `datasette.json`/`datasette.yaml` files, we could instead use a Python file, like `datasette_config.py`. That way one could dynamically generate config (ex dev vs prod, auto-discover credentials, etc.). Kinda like Django settings. Though I imagine Python imports might make this complex to do, and json/yaml is already supported and pretty easy to write ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1684202932,https://api.github.com/repos/simonw/datasette/issues/2143,1684202932,IC_kwDOBm6k_c5kYuW0,15178711,asg017,2023-08-18T17:10:21Z,2023-08-18T17:10:21Z,CONTRIBUTOR,"I agree with all your points! I think the best solution would be having a `datasette.json` config file, where you ""configure"" your datasette instances, with settings, permissions/auth, plugin configuration, and table settings (sortable column, label columns, etc.). Which #2093 would do. Then optionally, you have a `metadata.json`, or use `datasette_metadata`, or some other plugin to define metadata (ex the future [sqlite-docs](https://github.com/asg017/sqlite-docs) plugin). Everything in `datasette.json` could also be overwritten by CLI flags, like `--setting key value`, `--plugin xxxx key value`. We could even completely remove `settings.json` in favor or just `datasette.json`. Mostly because I think the less files the better, especially if they have generic names like `settings.json` or `config.json`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1683443891,https://api.github.com/repos/simonw/datasette/issues/2143,1683443891,IC_kwDOBm6k_c5kV1Cz,9599,simonw,2023-08-18T06:58:15Z,2023-08-18T06:58:15Z,OWNER,"Hah, that `--plugin-secret` thing was a messy solution I came up with to the problem that all metadata is visible at `/-/metadata` - so if you need to stash a secret you need a way to keep it not-visible in there! Hence the whole `$env` mess: https://docs.datasette.io/en/stable/plugins.html#secret-configuration-values ```json { ""plugins"": { ""datasette-auth-github"": { ""client_secret"": { ""$env"": ""GITHUB_CLIENT_SECRET"" } } } } ``` If configuration and metadata were separate we could ditch that whole messy situation - configuration can stay hidden, metadata can stay public. Though I have been thinking that Datasette might benefit from a ""secrets"" mechanism that's separate from configuration and metadata... kind of like what LLM has: https://llm.datasette.io/en/stable/help.html#llm-keys-help","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1683440597,https://api.github.com/repos/simonw/datasette/issues/2143,1683440597,IC_kwDOBm6k_c5kV0PV,9599,simonw,2023-08-18T06:54:49Z,2023-08-18T06:54:49Z,OWNER,"A related point that I've been considering a lot recently: it turns out that sometimes I really want to define settings on the CLI instead of in a file, purely for convenience. It's pretty annoying when I want to try out a new plugin but I have to create a dedicated `metadata.yml` file for it just to setup a single option - I'd love to have the option to be able to run this instead: ```bash datasette data.db --plugin-setting datasette-upload-csvs default-database data ``` So maybe there's a world in which all of the settings can be applied in a `datasette.yml` file OR with command-line options. That gets trickier when you need to pass a nested structure or similar, but we could always support those as JSON: ```bash datasette data.db --plugin-setting datasette-emoji-reactions emoji '[""😼"", ""🐺""]' ``` Note that we kind of have precedent for this in `datasette publish`: https://docs.datasette.io/en/stable/publish.html#custom-metadata-and-plugins ```bash datasette publish heroku my_database.db \ --name my-heroku-app-demo \ --install=datasette-auth-github \ --plugin-secret datasette-auth-github client_id your_client_id \ --plugin-secret datasette-auth-github client_secret your_client_secret ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1683435579,https://api.github.com/repos/simonw/datasette/issues/2143,1683435579,IC_kwDOBm6k_c5kVzA7,9599,simonw,2023-08-18T06:49:39Z,2023-08-18T06:49:39Z,OWNER,"My ideal situation then would be something like this: - Metadata itself is VERY clearly described, including sensible rules for metadata inheritance where it makes sense. There is a `datasette.X` method for accessing it which is much more intuitive than `datasette.metadata()`. - It's possible that method should be an `async` method, because that would support things like plugins that lookup metadata in database tables better. - All templates etc switch to the new, clean, intuitive metadata mechanism before 1.0. - I'm interested in the option of metadata being able to live in a `_datasette_metadata` table in the databases themselves - either as a plugin or as a core feature. I think it makes a lot of sense for metadata to optionally live with the data that it describes. - Configuration gets split from metadata. The stuff that configures Datasette no longer lives in the `metadata.yml` file - it lives in `config.yml` (or even `datasette.yml`). Currently we have three types of things: - Metadata - information about the data - Configuration - stuff like ""these columns should be sortable"" and ""this is configured as `fts_table`"" and suchlike - Settings - the stuff that you pass to `datasette --setting x y` on server start. Should settings and configuration be separate? I'm not 100% sure that they should - maybe those two concepts should be combined somehow. Configuration directory mode needs to be considered too: https://docs.datasette.io/en/stable/settings.html#configuration-directory-mode - interestingly it already has a thing where it can pick up settings from a `settings.json` file - where settings are things like `datasette --setting sql_time_limit_ms 4000`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1683429959,https://api.github.com/repos/simonw/datasette/issues/2143,1683429959,IC_kwDOBm6k_c5kVxpH,9599,simonw,2023-08-18T06:43:33Z,2023-08-18T15:19:07Z,OWNER,"The single biggest design challenge I've had with metadata relates to how it should or should not be inherited. If you apply a license to a Datasette instance, it feels like that should flow down to cover all of the databases and all of the tables within those databases. If the license is at the database level, it should cover all tables. But... should source do the same thing? I made it behave the same way as license, but it's presumably common for one database to have a single license but multiple different sources of data. Then there's title - should that inherit? It feels like title should apply to only one level - you may want a title that applies to the instance, then a different custom title for databases and tables. Here's the current state of play for metadata: https://docs.datasette.io/en/1.0a3/metadata.html So there's `title` and `description` - and I'll be honest, I'm not 100% sure even I understand how those should be inherited down by tables/etc. There's `description_html` which over-rides the `description` if it is set. It's a useful customization hack, but a bit surprising. Then there are these six: - `license` - `license_url` - `source` - `source_url` - `about` - `about_url` I added `about` later than the others, because I realized that plenty of my own projects needed a link to an article explaining them somewhere - e.g. https://scotrail.datasette.io/ Tables can also have column descriptions - just a string for each column. There's a demo of those here: https://latest.datasette.io/fixtures/roadside_attractions And then there's all of the other stuff, most of which feels much more like ""settings"" than ""metadata"": - `sort: created` - the custom sort order - `size: 10` for a custom page size for a specific table - `sortable_columns` to set which columns can be used to sort - `hidden: true` to hide a table - `label_column: title` is an interesting one - it lets you hint to Datasette which column should be displayed when there is a foreign key relationship. It's sort-of-metadata and sort-of-a-setting. - `facets` sets default facets, see https://docs.datasette.io/en/1.0a3/facets.html#facets-in-metadata - `facet_size` sets the number of facets to display - `fts_table` and `fts_pk` can be used to configure FTS, especially for views: https://docs.datasette.io/en/1.0a3/full_text_search.html And the authentication stuff! `allow` and `allow_sql` blocks: https://docs.datasette.io/en/1.0a3/authentication.html#defining-permissions-with-allow-blocks And the new `permissions` key in the 1.0 alphas: https://docs.datasette.io/en/1.0a3/authentication.html#other-permissions-in-metadata I think that might be everything (excluding the `plugins` settings stuff, which is also a bad fit for metadata.) And to make things even more confusing... I believe you can add arbitrary key/value pairs to your metadata and then use them in your templates! I think I've heard from at least one person who uses that ability.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/datasette/issues/2143#issuecomment-1683420879,https://api.github.com/repos/simonw/datasette/issues/2143,1683420879,IC_kwDOBm6k_c5kVvbP,9599,simonw,2023-08-18T06:33:24Z,2023-08-18T15:15:34Z,OWNER,"I completely agree: metadata is a mess, and it deserves our attention. > 1. Metadata cannot be updated without re-starting the entire Datasette instance. That's not completely true - there are hacks around that. I have a plugin that applies one set of gnarly hacks for that here: https://github.com/simonw/datasette-remote-metadata - it's pretty grim though! > 2. The `metadata.json`/`metadata.yaml` has become a kitchen sink of unrelated (imo) features like plugin config, authentication config, canned queries 100% this: it's a complete mess. Datasette used to have a `datasette --config foo:bar` mechanism, which I deprecated in favour of `datasette --setting foo bar` partly because I wanted to free up `--config` for pointing at a real config file, so we could stop dropping everything in `--metadata metadata.yml`. > 3. The Python APIs for defining extra metadata are a bit awkward (the `datasette.metadata()` class, `get_metadata()` hook, etc.) Yes, they're not pretty at all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855885427,De-tangling Metadata before Datasette 1.0, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683164661,IC_kwDOCGYnMM5kUw31,9599,simonw,2023-08-18T00:45:53Z,2023-08-18T00:45:53Z,OWNER,"More updated documentation: - https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform - https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145819,IC_kwDOCGYnMM5kUsRb,9599,simonw,2023-08-18T00:17:26Z,2023-08-18T00:17:26Z,OWNER,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145110,IC_kwDOCGYnMM5kUsGW,9599,simonw,2023-08-18T00:16:28Z,2023-08-18T00:16:48Z,OWNER,"One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`: https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708 I should write tests for them too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683143723,IC_kwDOCGYnMM5kUrwr,9599,simonw,2023-08-18T00:14:52Z,2023-08-18T00:14:52Z,OWNER,"Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times: > The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries. > > `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up. > > Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`. That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683139304,IC_kwDOCGYnMM5kUqro,9599,simonw,2023-08-18T00:09:56Z,2023-08-18T00:09:56Z,OWNER,"Upgrading `flake8` locally replicated the error: ``` pip install -U flake8 flake8 ``` ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683138953,IC_kwDOCGYnMM5kUqmJ,9599,simonw,2023-08-18T00:09:20Z,2023-08-18T00:09:20Z,OWNER,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop: ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683137259,IC_kwDOCGYnMM5kUqLr,9599,simonw,2023-08-18T00:06:59Z,2023-08-18T00:06:59Z,OWNER,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,simonw,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures: ``` sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[, ] = ..."") [var-annotated] sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform"" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] Found 5 errors in 1 file (checked 56 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122490,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122490,IC_kwDOCGYnMM5kUmk6,22429695,codecov[bot],2023-08-17T23:45:44Z,2023-08-18T00:46:07Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`92.85%`** and project coverage change: **`-0.07%`** :warning: > Comparison is base [(`1dc6b5a`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/1dc6b5aa644a92d3654f7068110ed7930989ce71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.82% compared to head [(`2915050`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.76%. > Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #584 +/- ## ========================================== - Coverage 95.82% 95.76% -0.07% ========================================== Files 8 8 Lines 2829 2834 +5 ========================================== + Hits 2711 2714 +3 - Misses 118 120 +2 ``` | [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <92.85%> (-0.15%)` | :arrow_down: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,simonw,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line: ```python fresh_db.create_table(""breeds"", {""name"": str}) ``` Has this schema: ```sql CREATE TABLE [breeds] ( [name] TEXT ); ``` SQLite creates an invisible `rowid` column for it automatically. On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added: ```sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); ``` But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884 Here's what the debugger reveals about this code: ```python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) ``` ``` (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,simonw,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem: ```python from sqlite_utils import Database fresh_db = Database(memory=True) fresh_db.create_table(""dogs"", {""name"": str}) fresh_db.create_table(""breeds"", {""name"": str}) fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"") ``` Traceback: ``` Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql self.db.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,simonw,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,simonw,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left: ``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): > raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) E sqlite_utils.db.AlterError: No such column: breeds.rowid sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys, https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,simonw,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on: - #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914,Get rid of test.utils.collapse_whitespace, https://github.com/simonw/datasette/pull/2142#issuecomment-1683950031,https://api.github.com/repos/simonw/datasette/issues/2142,1683950031,IC_kwDOBm6k_c5kXwnP,49699333,dependabot[bot],2023-08-18T13:49:24Z,2023-08-18T13:49:24Z,CONTRIBUTOR,"Looks like these dependencies are updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1854970601,Bump the python-packages group with 2 updates, https://github.com/simonw/datasette/pull/2142#issuecomment-1682272492,https://api.github.com/repos/simonw/datasette/issues/2142,1682272492,IC_kwDOBm6k_c5kRXDs,22429695,codecov[bot],2023-08-17T13:18:08Z,2023-08-17T13:18:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`6b617f7`)](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2142 +/- ## ======================================= Coverage 92.06% 92.06% ======================================= Files 40 40 Lines 5937 5937 ======================================= Hits 5466 5466 Misses 471 471 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1854970601,Bump the python-packages group with 2 updates, https://github.com/simonw/datasette/pull/2141#issuecomment-1682256251,https://api.github.com/repos/simonw/datasette/issues/2141,1682256251,IC_kwDOBm6k_c5kRTF7,49699333,dependabot[bot],2023-08-17T13:07:43Z,2023-08-17T13:07:43Z,CONTRIBUTOR,"Looks like blacken-docs is updatable in another way, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1853289039,Bump the python-packages group with 1 update, https://github.com/simonw/datasette/pull/2141#issuecomment-1680660343,https://api.github.com/repos/simonw/datasette/issues/2141,1680660343,IC_kwDOBm6k_c5kLNd3,22429695,codecov[bot],2023-08-16T13:55:22Z,2023-08-16T13:55:22Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`e2be94a`)](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2141 +/- ## ======================================= Coverage 92.06% 92.06% ======================================= Files 40 40 Lines 5937 5937 ======================================= Hits 5466 5466 Misses 471 471 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2141?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1853289039,Bump the python-packages group with 1 update, https://github.com/simonw/datasette/issues/2140#issuecomment-1675157445,https://api.github.com/repos/simonw/datasette/issues/2140,1675157445,IC_kwDOBm6k_c5j2N_F,9599,simonw,2023-08-11T17:45:54Z,2023-08-11T17:51:59Z,OWNER,"Affected pages: - https://docs.datasette.io/en/latest/authentication.html - https://docs.datasette.io/en/latest/changelog.html - https://docs.datasette.io/en/latest/cli-reference.html - https://docs.datasette.io/en/latest/contributing.html - https://docs.datasette.io/en/latest/custom_templates.html - https://docs.datasette.io/en/latest/deploying.html - https://docs.datasette.io/en/latest/facets.html - https://docs.datasette.io/en/latest/full_text_search.html - https://docs.datasette.io/en/latest/installation.html - https://docs.datasette.io/en/latest/plugin_hooks.html - https://docs.datasette.io/en/latest/publish.html - https://docs.datasette.io/en/latest/settings.html - https://docs.datasette.io/en/latest/spatialite.html - https://docs.datasette.io/en/latest/sql_queries.html e.g. on https://docs.datasette.io/en/latest/authentication.html#using-the-root-actor ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1847201263,Remove all remaining documentation instances of '$ ', https://github.com/simonw/datasette/issues/2139#issuecomment-1674210009,https://api.github.com/repos/simonw/datasette/issues/2139,1674210009,IC_kwDOBm6k_c5jymrZ,9599,simonw,2023-08-11T04:52:44Z,2023-08-11T04:52:44Z,OWNER,"I'll fix these too: - https://github.com/simonw/datasette/issues/2119 - https://github.com/simonw/datasette/issues/568","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1846076261,border-color: ##ff0000 bug - two hashes, https://github.com/simonw/datasette/issues/2139#issuecomment-1674115279,https://api.github.com/repos/simonw/datasette/issues/2139,1674115279,IC_kwDOBm6k_c5jyPjP,9599,simonw,2023-08-11T01:27:30Z,2023-08-11T01:27:30Z,OWNER,"https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/database.html#L13 Looks like I should be doing this instead: https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1846076261,border-color: ##ff0000 bug - two hashes, https://github.com/simonw/datasette/issues/2138#issuecomment-1672366410,https://api.github.com/repos/simonw/datasette/issues/2138,1672366410,IC_kwDOBm6k_c5jrklK,9599,simonw,2023-08-10T00:39:16Z,2023-08-10T00:39:16Z,OWNER,"Here's what it looks like when it works: And here's what you get if there's a SQL error in the `on_success_message_sql` query itself: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1844213115,on_success_message_sql option for writable canned queries, https://github.com/simonw/datasette/issues/2138#issuecomment-1672353428,https://api.github.com/repos/simonw/datasette/issues/2138,1672353428,IC_kwDOBm6k_c5jrhaU,9599,simonw,2023-08-10T00:20:56Z,2023-08-10T00:20:56Z,OWNER,"Goal here is to customize the message displayed after a query has succeeded. e.g. ```yaml databases: mydatabase: queries: add_name: sql: INSERT INTO names (name) VALUES (:name) write: true on_success_message_sql: 'select ''Name inserted: '' || :name' on_success_redirect: /mydatabase/names ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1844213115,on_success_message_sql option for writable canned queries, https://github.com/simonw/datasette/issues/2137#issuecomment-1671983228,https://api.github.com/repos/simonw/datasette/issues/2137,1671983228,IC_kwDOBm6k_c5jqHB8,9599,simonw,2023-08-09T19:02:47Z,2023-08-09T19:02:47Z,OWNER,"As part of this I'd like to refactor the JSON renderer code - I started attempting that here: https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2 In particular I want to get rid of that `data` argument.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843821954,Redesign row default JSON, https://github.com/simonw/datasette/issues/2136#issuecomment-1671982215,https://api.github.com/repos/simonw/datasette/issues/2136,1671982215,IC_kwDOBm6k_c5jqGyH,9599,simonw,2023-08-09T19:02:19Z,2023-08-09T19:02:19Z,OWNER,"I'm going with a bit of a hacky workaround for the moment, until I get a chance to refactor this: - #2137 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`, https://github.com/simonw/datasette/issues/2136#issuecomment-1671964003,https://api.github.com/repos/simonw/datasette/issues/2136,1671964003,IC_kwDOBm6k_c5jqCVj,9599,simonw,2023-08-09T18:53:10Z,2023-08-09T18:53:18Z,OWNER,"I made an initial attempt at this in a branch, but it broke too much other stuff. https://github.com/simonw/datasette/commit/f3944608cc000a2542b4fe9a7a89c866391924f2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`, https://github.com/simonw/datasette/issues/2136#issuecomment-1671861623,https://api.github.com/repos/simonw/datasette/issues/2136,1671861623,IC_kwDOBm6k_c5jppV3,9599,simonw,2023-08-09T17:34:35Z,2023-08-09T17:34:35Z,OWNER,"I'll make `?_extra=columns` work in: - #2113","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`, https://github.com/simonw/datasette/issues/2136#issuecomment-1671848664,https://api.github.com/repos/simonw/datasette/issues/2136,1671848664,IC_kwDOBm6k_c5jpmLY,9599,simonw,2023-08-09T17:24:37Z,2023-08-09T17:24:37Z,OWNER,"It also bothers me that `ok` comes first in one but not first in the other - I'd like the order of keys in the JSON to be consistent, even though order isn't meant to mean anything.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843710170,Query view shouldn't return `columns`, https://github.com/simonw/datasette/issues/2135#issuecomment-1671844664,https://api.github.com/repos/simonw/datasette/issues/2135,1671844664,IC_kwDOBm6k_c5jplM4,9599,simonw,2023-08-09T17:21:22Z,2023-08-09T17:21:22Z,OWNER,"OK, I've done the small changes - still need to document the big JSON changes though. Those aren't covered by the regular documentation either so I need to update that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843600087,Release notes for 1.0a3, https://github.com/simonw/datasette/issues/2135#issuecomment-1671753753,https://api.github.com/repos/simonw/datasette/issues/2135,1671753753,IC_kwDOBm6k_c5jpPAZ,9599,simonw,2023-08-09T16:26:17Z,2023-08-09T16:28:19Z,OWNER,"- Dropped support for Python 3.7 - New `Justfile` - #1970 - `datasette.render_template()` now accepts a `Context` subclass as an alternative to a dictionary - #2106 - #2007 - https://github.com/simonw/datasette/issues/2130 - https://github.com/simonw/datasette/issues/2079 - https://github.com/simonw/datasette/issues/1153","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843600087,Release notes for 1.0a3, https://github.com/simonw/datasette/issues/2135#issuecomment-1671750913,https://api.github.com/repos/simonw/datasette/issues/2135,1671750913,IC_kwDOBm6k_c5jpOUB,9599,simonw,2023-08-09T16:24:26Z,2023-08-09T16:24:26Z,OWNER,"Remember not to include changes from this: https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/docs/changelog.rst?plain=1#L9-L31 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843600087,Release notes for 1.0a3, https://github.com/simonw/datasette/issues/2134#issuecomment-1672384439,https://api.github.com/repos/simonw/datasette/issues/2134,1672384439,IC_kwDOBm6k_c5jro-3,9599,simonw,2023-08-10T01:05:56Z,2023-08-10T01:22:46Z,OWNER,"Now live at https://latest.datasette.io/counters ![increment](https://github.com/simonw/datasette/assets/9599/ffb3625b-b61b-4189-80e3-7b4e8b411f14) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io, https://github.com/simonw/datasette/issues/2134#issuecomment-1671581610,https://api.github.com/repos/simonw/datasette/issues/2134,1671581610,IC_kwDOBm6k_c5jok-q,9599,simonw,2023-08-09T15:01:14Z,2023-08-10T00:19:56Z,OWNER,Or... how about if the `on_success_message` option could define a SQL query to be executed to generate that message? Maybe `on_success_message_sql`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io, https://github.com/simonw/datasette/issues/2134#issuecomment-1671526372,https://api.github.com/repos/simonw/datasette/issues/2134,1671526372,IC_kwDOBm6k_c5joXfk,9599,simonw,2023-08-09T14:39:54Z,2023-08-09T14:39:54Z,OWNER,"Alternatively, what about if there was a custom SQL function available during canned write queries for setting the output message? Then the fact that canned queries don't return a table view wouldn't be a problem. Bit weird though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io, https://github.com/simonw/datasette/issues/2134#issuecomment-1671503163,https://api.github.com/repos/simonw/datasette/issues/2134,1671503163,IC_kwDOBm6k_c5joR07,9599,simonw,2023-08-09T14:32:30Z,2023-08-09T14:32:30Z,OWNER,"Annoying thing about this plugin is that you don't see the new counter value when you submit the increment or decrement query. Maybe canned queries should support SQL multiple statements? Could return the result of the last one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io, https://github.com/simonw/datasette/issues/2134#issuecomment-1671500965,https://api.github.com/repos/simonw/datasette/issues/2134,1671500965,IC_kwDOBm6k_c5joRSl,9599,simonw,2023-08-09T14:31:43Z,2023-08-09T14:31:43Z,OWNER,"Here's a simple plugin that can do this: ```python from datasette import hookimpl @hookimpl def startup(datasette): db = datasette.add_memory_database(""counters"") async def inner(): await db.execute_write(""create table if not exists counters (name text primary key, value integer)"") await db.execute_write(""insert or ignore into counters (name, value) values ('counter', 0)"") return inner @hookimpl def canned_queries(database): if database == ""counters"": return { ""increment"": { ""sql"": ""update counters set value = value + 1 where name = 'counter'"", ""write"": True, }, ""decrement"": { ""sql"": ""update counters set value = value - 1 where name = 'counter'"", ""write"": True, }, } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1843391585,Add writable canned query demo to latest.datasette.io, https://github.com/simonw/datasette/issues/2133#issuecomment-1672360472,https://api.github.com/repos/simonw/datasette/issues/2133,1672360472,IC_kwDOBm6k_c5jrjIY,54462,HaveF,2023-08-10T00:31:24Z,2023-08-10T00:31:24Z,NONE,"It looks very nice now. Finally, no more manual installation of plugins one by one. Thank you, Simon! ❤️ ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1672224611,https://api.github.com/repos/simonw/datasette/issues/2133,1672224611,IC_kwDOBm6k_c5jrB9j,9599,simonw,2023-08-09T22:07:43Z,2023-08-09T22:07:43Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugins.html#seeing-what-plugins-are-installed,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1672076671,https://api.github.com/repos/simonw/datasette/issues/2133,1672076671,IC_kwDOBm6k_c5jqd1_,9599,simonw,2023-08-09T20:09:31Z,2023-08-09T20:09:31Z,OWNER,I'm going to have it only return plugins with a version other than `None`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1672076284,https://api.github.com/repos/simonw/datasette/issues/2133,1672076284,IC_kwDOBm6k_c5jqdv8,9599,simonw,2023-08-09T20:09:12Z,2023-08-09T20:09:12Z,OWNER,"Problem - if there are plugins in the plugins directory it was returning this: ``` messages_output_renderer.py==None my_plugin.py==None my_plugin_2.py==None register_output_renderer.py==None sleep_sql_function.py==None view_name.py==None ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1671722482,https://api.github.com/repos/simonw/datasette/issues/2133,1671722482,IC_kwDOBm6k_c5jpHXy,9599,simonw,2023-08-09T16:06:28Z,2023-08-09T16:06:28Z,OWNER,"Plugin configuration itself currently lives in `metadata.json` which I don't like - that file was supposed to be for metadata about your databases, it's a bit of an accident that it grew configuration options as well. I'm hoping to fix that for Datasette 1.0, but I'm still figuring out how to do that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1671720761,https://api.github.com/repos/simonw/datasette/issues/2133,1671720761,IC_kwDOBm6k_c5jpG85,9599,simonw,2023-08-09T16:05:33Z,2023-08-09T16:05:33Z,OWNER,"Honestly the `--all` option isn't particularly useful - it's really more of a debugging tool for myself than something that I expect anyone else to use. The purpose of `datasette plugins` generally is to help people confirm that plugins installed correctly. It's effectively the CLI alternative to hitting https://datasette.io/-/plugins","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1671649530,https://api.github.com/repos/simonw/datasette/issues/2133,1671649530,IC_kwDOBm6k_c5jo1j6,54462,HaveF,2023-08-09T15:41:14Z,2023-08-09T15:41:14Z,NONE,"Yes, using this approach(`datasette install -r requirements.txt`) will result in more consistency. I'm curious about the results of the `datasette plugins --all` command. Where will we use the output of this command? Will it include configuration information for these plugins in the future? If so, will we need to consider the configuration of these plugins in addition to installing them on different computers?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1671634654,https://api.github.com/repos/simonw/datasette/issues/2133,1671634654,IC_kwDOBm6k_c5jox7e,9599,simonw,2023-08-09T15:32:14Z,2023-08-09T15:32:14Z,OWNER,"```diff diff --git a/datasette/cli.py b/datasette/cli.py index 32266888..370b722c 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -223,15 +223,22 @@ pm.hook.publish_subcommand(publish=publish) @cli.command() @click.option(""--all"", help=""Include built-in default plugins"", is_flag=True) +@click.option( + ""--requirements"", help=""Output requirements.txt of installed plugins"", is_flag=True +) @click.option( ""--plugins-dir"", type=click.Path(exists=True, file_okay=False, dir_okay=True), help=""Path to directory containing custom plugins"", ) -def plugins(all, plugins_dir): +def plugins(all, requirements, plugins_dir): """"""List currently installed plugins"""""" app = Datasette([], plugins_dir=plugins_dir) - click.echo(json.dumps(app._plugins(all=all), indent=4)) + if requirements: + for plugin in app._plugins(): + click.echo(""{}=={}"".format(plugin[""name""], plugin[""version""])) + else: + click.echo(json.dumps(app._plugins(all=all), indent=4)) @cli.command() ``` Output: ``` % datasette plugins --requirements datasette-codespaces==0.1.1 datasette-graphql==2.2 datasette-json-html==1.0.1 datasette-pretty-json==0.2.2 datasette-x-forwarded-host==0.1 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2133#issuecomment-1671628602,https://api.github.com/repos/simonw/datasette/issues/2133,1671628602,IC_kwDOBm6k_c5jowc6,9599,simonw,2023-08-09T15:28:42Z,2023-08-09T15:28:42Z,OWNER,"Neat idea! We already have this command: datasette install -r requirements.txt So maybe `datasetet plugins --requirements` could generate a `requirements.txt` file which could be used there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841501975,[feature request]`datasette install plugins.json` options, https://github.com/simonw/datasette/issues/2132#issuecomment-1669640626,https://api.github.com/repos/simonw/datasette/issues/2132,1669640626,IC_kwDOBm6k_c5jhLGy,9599,simonw,2023-08-08T13:39:54Z,2023-08-08T13:39:54Z,OWNER,https://github.com/simonw/datasette/blob/1377a290cd85ba8d3338b1da47c4665ed4c6c625/datasette/views/database.py#L509-L517,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1841343173,Get form fields on query page working again , https://github.com/simonw/datasette/issues/2131#issuecomment-1671626876,https://api.github.com/repos/simonw/datasette/issues/2131,1671626876,IC_kwDOBm6k_c5jowB8,9599,simonw,2023-08-09T15:27:40Z,2023-08-09T15:27:40Z,OWNER,Another place that needs fixing: https://github.com/simonw/datasette/blob/26be9f0445b753fb84c802c356b0791a72269f25/datasette/views/database.py#L244-L249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840417903,Refactor code that supports templates_considered comment, https://github.com/simonw/datasette/issues/2130#issuecomment-1668793100,https://api.github.com/repos/simonw/datasette/issues/2130,1668793100,IC_kwDOBm6k_c5jd8MM,9599,simonw,2023-08-08T01:51:54Z,2023-08-08T01:51:54Z,OWNER,Updated documentation is here: https://docs.datasette.io/en/latest/plugin_hooks.html#register-output-renderer-datasette,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840329615,Render plugin mechanism needs `error` and `truncated` fields, https://github.com/simonw/datasette/issues/2130#issuecomment-1668695958,https://api.github.com/repos/simonw/datasette/issues/2130,1668695958,IC_kwDOBm6k_c5jdkeW,9599,simonw,2023-08-07T23:21:07Z,2023-08-07T23:21:16Z,OWNER,"Main place the code needs updating is here: https://github.com/simonw/datasette/blob/5139c0886a7f6bb94d317ba0665aa6e728716028/datasette/views/base.py#L301-L310 Plus the documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840329615,Render plugin mechanism needs `error` and `truncated` fields, https://github.com/simonw/datasette/issues/2129#issuecomment-1668690232,https://api.github.com/repos/simonw/datasette/issues/2129,1668690232,IC_kwDOBm6k_c5jdjE4,9599,simonw,2023-08-07T23:13:24Z,2023-08-07T23:13:24Z,OWNER,This may end up being a renderer plugin hook design question too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1840324765,CSV ?sql= should indicate errors, https://github.com/simonw/datasette/pull/2128#issuecomment-1668143992,https://api.github.com/repos/simonw/datasette/issues/2128,1668143992,IC_kwDOBm6k_c5jbdt4,22429695,codecov[bot],2023-08-07T15:58:08Z,2023-08-07T15:58:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`adf54f5`)](https://app.codecov.io/gh/simonw/datasette/commit/adf54f5c808c16e9ad812c3a4376f35cf829a93c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`935d336`)](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2128 +/- ## ======================================= Coverage 92.38% 92.38% ======================================= Files 39 39 Lines 5803 5803 ======================================= Hits 5361 5361 Misses 442 442 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2128?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1839766197,"Bump blacken-docs, furo, blacken-docs", https://github.com/simonw/datasette/issues/2127#issuecomment-1672404276,https://api.github.com/repos/simonw/datasette/issues/2127,1672404276,IC_kwDOBm6k_c5jrt00,9599,simonw,2023-08-10T01:30:25Z,2023-08-10T01:30:25Z,OWNER,"Here's the prototype of adding documentation for these using Cog: https://github.com/simonw/datasette/issues/1510#issuecomment-1610512875 ![](https://user-images.githubusercontent.com/9599/249324964-79210a43-716c-4470-a62d-01aa871973ec.png)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838469176,Context base class to support documenting the context, https://github.com/simonw/datasette/issues/2127#issuecomment-1668792576,https://api.github.com/repos/simonw/datasette/issues/2127,1668792576,IC_kwDOBm6k_c5jd8EA,9599,simonw,2023-08-08T01:51:14Z,2023-08-08T01:51:14Z,OWNER,"OK, I've landed the first example of this now. I'm cautiously optimistic about it, but I think I need to use it in a few more places AND add the actual code that uses it for documentation before I'm completely happy and close this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838469176,Context base class to support documenting the context, https://github.com/simonw/datasette/issues/2127#issuecomment-1667011607,https://api.github.com/repos/simonw/datasette/issues/2127,1667011607,IC_kwDOBm6k_c5jXJQX,9599,simonw,2023-08-07T00:02:49Z,2023-08-07T00:02:49Z,OWNER,"This is pretty neat: https://github.com/simonw/datasette/blob/2f9038a831a3510d4c9ab39a12d96259b3a55bc7/datasette/views/database.py#L157-L206","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838469176,Context base class to support documenting the context, https://github.com/simonw/datasette/issues/2126#issuecomment-1674242356,https://api.github.com/repos/simonw/datasette/issues/2126,1674242356,IC_kwDOBm6k_c5jyuk0,36199671,ctsrc,2023-08-11T05:52:29Z,2023-08-11T05:52:29Z,NONE,"I see :) yeah, I’m on the stable version installed from homebrew on macOS","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838266862,Permissions in metadata.yml / metadata.json, https://github.com/simonw/datasette/issues/2126#issuecomment-1672385674,https://api.github.com/repos/simonw/datasette/issues/2126,1672385674,IC_kwDOBm6k_c5jrpSK,9599,simonw,2023-08-10T01:07:43Z,2023-08-10T01:07:43Z,OWNER,"What version of Datasette are you running? That feature was added in Datasette 1.0a2, so if you're on the current stable release you won't have it yet.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838266862,Permissions in metadata.yml / metadata.json, https://github.com/simonw/datasette/issues/2126#issuecomment-1666912107,https://api.github.com/repos/simonw/datasette/issues/2126,1666912107,IC_kwDOBm6k_c5jWw9r,36199671,ctsrc,2023-08-06T16:27:34Z,2023-08-06T16:27:34Z,NONE,"And in similar fashion, how can I assign the `edit-tiddlywiki` permission to my user `myuser` in `metadata.yml` / `metadata.json`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1838266862,Permissions in metadata.yml / metadata.json, https://github.com/simonw/datasette/pull/2125#issuecomment-1668187546,https://api.github.com/repos/simonw/datasette/issues/2125,1668187546,IC_kwDOBm6k_c5jboWa,49699333,dependabot[bot],2023-08-07T16:20:26Z,2023-08-07T16:20:26Z,CONTRIBUTOR,"Looks like sphinx is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1833193570,Bump sphinx from 6.1.3 to 7.1.2, https://github.com/simonw/datasette/pull/2125#issuecomment-1662228666,https://api.github.com/repos/simonw/datasette/issues/2125,1662228666,IC_kwDOBm6k_c5jE5i6,22429695,codecov[bot],2023-08-02T13:36:22Z,2023-08-02T13:36:22Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`730f554`)](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2125 +/- ## ======================================= Coverage 92.38% 92.38% ======================================= Files 39 39 Lines 5803 5803 ======================================= Hits 5361 5361 Misses 442 442 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2125?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1833193570,Bump sphinx from 6.1.3 to 7.1.2, https://github.com/dogsheep/dogsheep-photos/pull/38#issuecomment-1656694944,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/38,1656694944,IC_kwDOD079W85ivyig,319473,coldclimate,2023-07-29T10:01:19Z,2023-07-29T10:01:19Z,NONE,Duplicate of https://github.com/dogsheep/dogsheep-photos/pull/36 - closing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1827427757,photos-to-sql not found?, https://github.com/dogsheep/dogsheep-photos/pull/38#issuecomment-1656694854,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/38,1656694854,IC_kwDOD079W85ivyhG,319473,coldclimate,2023-07-29T10:00:45Z,2023-07-29T10:00:45Z,NONE,Ran across https://github.com/dogsheep/dogsheep-photos/issues/33 which is the same subject. My PR just fixes docs,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1827427757,photos-to-sql not found?, https://github.com/simonw/datasette/pull/2124#issuecomment-1662215579,https://api.github.com/repos/simonw/datasette/issues/2124,1662215579,IC_kwDOBm6k_c5jE2Wb,49699333,dependabot[bot],2023-08-02T13:28:43Z,2023-08-02T13:28:43Z,CONTRIBUTOR,Superseded by #2125.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1826424151,Bump sphinx from 6.1.3 to 7.1.1, https://github.com/simonw/datasette/pull/2124#issuecomment-1655691977,https://api.github.com/repos/simonw/datasette/issues/2124,1655691977,IC_kwDOBm6k_c5ir9rJ,22429695,codecov[bot],2023-07-28T13:33:54Z,2023-07-28T13:33:54Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`9388102`)](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2124 +/- ## ======================================= Coverage 92.38% 92.38% ======================================= Files 39 39 Lines 5803 5803 ======================================= Hits 5361 5361 Misses 442 442 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2124?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1826424151,Bump sphinx from 6.1.3 to 7.1.1, https://github.com/simonw/datasette/issues/2123#issuecomment-1723362847,https://api.github.com/repos/simonw/datasette/issues/2123,1723362847,IC_kwDOBm6k_c5muG4f,6523121,hueyy,2023-09-18T13:02:46Z,2023-09-18T13:02:46Z,NONE,"Can confirm that this bug can be reproduced as follows: ``` docker run datasetteproject/datasette datasette serve --reload ``` which produces the following output: > Starting monitor for PID 10. > Error: Invalid value for '[FILES]...': Path 'serve' does not exist.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1825007061,datasette serve when invoked with --reload interprets the serve command as a file, https://github.com/simonw/datasette/issues/2123#issuecomment-1689207309,https://api.github.com/repos/simonw/datasette/issues/2123,1689207309,IC_kwDOBm6k_c5kr0IN,9599,simonw,2023-08-23T03:07:27Z,2023-08-23T03:07:27Z,OWNER,"> I'm happy to debug and land a patch if it's welcome. Yes please! What an odd bug.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1825007061,datasette serve when invoked with --reload interprets the serve command as a file, https://github.com/simonw/datasette/pull/2121#issuecomment-1668186872,https://api.github.com/repos/simonw/datasette/issues/2121,1668186872,IC_kwDOBm6k_c5jboL4,49699333,dependabot[bot],2023-08-07T16:20:19Z,2023-08-07T16:20:19Z,CONTRIBUTOR,"Looks like furo is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1824399610,Bump furo from 2023.3.27 to 2023.7.26, https://github.com/simonw/datasette/pull/2121#issuecomment-1653666188,https://api.github.com/repos/simonw/datasette/issues/2121,1653666188,IC_kwDOBm6k_c5ikPGM,22429695,codecov[bot],2023-07-27T13:48:43Z,2023-07-27T13:48:43Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`0818182`)](https://app.codecov.io/gh/simonw/datasette/commit/08181823990a71ffa5a1b57b37259198eaa43e06?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`3aca01f`)](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2121 +/- ## ======================================= Coverage 92.38% 92.38% ======================================= Files 39 39 Lines 5803 5803 ======================================= Hits 5361 5361 Misses 442 442 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2121?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1824399610,Bump furo from 2023.3.27 to 2023.7.26, https://github.com/simonw/datasette/issues/2119#issuecomment-1674227353,https://api.github.com/repos/simonw/datasette/issues/2119,1674227353,IC_kwDOBm6k_c5jyq6Z,9599,simonw,2023-08-11T05:25:45Z,2023-08-11T05:25:45Z,OWNER,"Demo: - https://latest.datasette.io/ - https://latest.datasette.io/fixtures - https://latest.datasette.io/fixtures/facetable - https://latest.datasette.io/fixtures/pragma_cache_size - https://latest.datasette.io/fixtures/attraction_characteristic/2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823393475,"database color shows only on index page, not other pages", https://github.com/simonw/datasette/issues/2119#issuecomment-1674211804,https://api.github.com/repos/simonw/datasette/issues/2119,1674211804,IC_kwDOBm6k_c5jynHc,9599,simonw,2023-08-11T04:56:28Z,2023-08-11T04:56:28Z,OWNER,"That `database.color` is only available in `index.html` at the moment thanks to this: https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/views/index.py#L108-L110","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823393475,"database color shows only on index page, not other pages", https://github.com/simonw/datasette/issues/2119#issuecomment-1674210215,https://api.github.com/repos/simonw/datasette/issues/2119,1674210215,IC_kwDOBm6k_c5jymun,9599,simonw,2023-08-11T04:53:11Z,2023-08-11T04:53:11Z,OWNER,"Relevant template snippet: https://github.com/simonw/datasette/blob/33251d04e78d575cca62bb59069bb43a7d924746/datasette/templates/index.html#L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823393475,"database color shows only on index page, not other pages", https://github.com/simonw/datasette/pull/2118#issuecomment-1668789134,https://api.github.com/repos/simonw/datasette/issues/2118,1668789134,IC_kwDOBm6k_c5jd7OO,9599,simonw,2023-08-08T01:46:12Z,2023-08-08T01:46:12Z,OWNER,"I'm going to squash merge this even though it's a bit big, because I want a clean passing commit in the `main` branch.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views, https://github.com/simonw/datasette/pull/2118#issuecomment-1668786770,https://api.github.com/repos/simonw/datasette/issues/2118,1668786770,IC_kwDOBm6k_c5jd6pS,9599,simonw,2023-08-08T01:42:55Z,2023-08-08T01:42:55Z,OWNER,"> UPDATE: Fixed the `_size=max` bit, but I've not replicated the behaviour where it adds `?_labels=on` if there are expandable columns yet. It looks like that behaviour is only relevant to table views, and it's already implemented - https://latest.datasette.io/fixtures/roadside_attraction_characteristics links to https://latest.datasette.io/fixtures/roadside_attraction_characteristics.csv?_labels=on&_size=max","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views, https://github.com/simonw/datasette/pull/2118#issuecomment-1668781558,https://api.github.com/repos/simonw/datasette/issues/2118,1668781558,IC_kwDOBm6k_c5jd5X2,9599,simonw,2023-08-08T01:34:42Z,2023-08-08T01:34:42Z,OWNER,"3 failures left: ``` =================================================================== short test summary info ==================================================================== FAILED tests/test_html.py::test_query_json_csv_export_links - assert 'CSV' in '\n\n\n fixtures: select 1\n ' in '\n\n\n fixtures: select * from... FAILED tests/test_html.py::test_query_page_truncates - assert equals failed FAILED tests/test_html.py::test_templates_considered[/fixtures-database-fixtures.html, *database.html] - assert '<!-- Templates considered: database-fixtures.html, *database.html -->' in '<!DOCTYPE html>\n<html>\n<head>\n <title>fixtures\n CSV' in '\n\n\n fixtures: select 1\n ' in '\n\n\n fixtures\n CSV' in '\n\n\n fixtures: select 1\n ' in '\n\n\n fixtures: select * from... FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures-expected_decoded_object1] - AssertionError: assert equals failed FAILED tests/test_plugins.py::test_view_names[/fixtures-database] - AssertionError: assert equals failed FAILED tests/test_plugins.py::test_view_names[/fixtures?sql=select+1-database] - AssertionError: assert equals failed FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures-expected_extra_body_script1] - AssertionError: assert equals failed FAILED tests/test_html.py::test_base_url_config[False-/fixtures?sql=select+1] - AssertionError: { FAILED tests/test_table_api.py::test_max_returned_rows - KeyError: 'query' FAILED tests/test_html.py::test_alternate_url_json[/fixtures-http://localhost/fixtures.json] - KeyError: 'link' ============================================================================ 15 failed, 1297 passed, 2 skipped, 1 xfailed in 58.15s ============================================================================ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views, https://github.com/simonw/datasette/pull/2118#issuecomment-1668568445,https://api.github.com/repos/simonw/datasette/issues/2118,1668568445,IC_kwDOBm6k_c5jdFV9,9599,simonw,2023-08-07T20:57:46Z,2023-08-07T20:57:46Z,OWNER,17 failing tests now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views, https://github.com/simonw/datasette/pull/2118#issuecomment-1652681136,https://api.github.com/repos/simonw/datasette/issues/2118,1652681136,IC_kwDOBm6k_c5igemw,9599,simonw,2023-07-26T23:30:44Z,2023-07-26T23:30:44Z,OWNER,"The `_shape=` stuff should use `json_renderer` instead - that's how the table view did it: https://github.com/simonw/datasette/commit/d97e82df3c8a3f2e97038d7080167be9bb74a68d#diff-5c9ef29c33ed0fde413565b23fa258d60fc3a2bb205b016db9e915c9bd5ecfb3 https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/table.py#L822-L850 Instead of: https://github.com/simonw/datasette/blob/c3e3ecf590ca5fa61b00aba4c78599e33d370b60/datasette/views/database.py#L239-L288","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823352380,New JSON design for query views, https://github.com/simonw/sqlite-utils/issues/581#issuecomment-1652496702,https://api.github.com/repos/simonw/sqlite-utils/issues/581,1652496702,IC_kwDOCGYnMM5ifxk-,9599,simonw,2023-07-26T21:07:45Z,2023-07-26T21:07:45Z,OWNER,Docs: https://sqlite-utils.datasette.io/en/latest/cli.html#using-the-debugger,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823160748,`sqlite-utils convert --pdb` option, https://github.com/simonw/datasette/issues/2117#issuecomment-1668799233,https://api.github.com/repos/simonw/datasette/issues/2117,1668799233,IC_kwDOBm6k_c5jd9sB,9599,simonw,2023-08-08T02:02:07Z,2023-08-08T02:02:07Z,OWNER,I just need to pass `view_name` to `datasette.render_template()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822982933,Figure out what to do about `DatabaseView.name`, https://github.com/simonw/datasette/issues/2116#issuecomment-1652315926,https://api.github.com/repos/simonw/datasette/issues/2116,1652315926,IC_kwDOBm6k_c5ifFcW,9599,simonw,2023-07-26T18:43:17Z,2023-07-26T18:43:17Z,OWNER,"Tests pass, and manually tested like this too: ```bash datasette -i pelicans.db ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function, https://github.com/simonw/datasette/issues/2116#issuecomment-1652304289,https://api.github.com/repos/simonw/datasette/issues/2116,1652304289,IC_kwDOBm6k_c5ifCmh,9599,simonw,2023-07-26T18:33:07Z,2023-07-26T18:33:07Z,OWNER,"This runs three tests: ```bash pytest -k test_database_download ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function, https://github.com/simonw/datasette/issues/2116#issuecomment-1652303471,https://api.github.com/repos/simonw/datasette/issues/2116,1652303471,IC_kwDOBm6k_c5ifCZv,9599,simonw,2023-07-26T18:32:24Z,2023-07-26T18:32:24Z,OWNER,https://github.com/simonw/datasette/blob/dc5171eb1b1d9f1d55e367f8a4d93edb55a43351/datasette/views/database.py#L172-L212,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822949756,Turn DatabaseDownload into an async view function, https://github.com/simonw/datasette/issues/2114#issuecomment-1670470516,https://api.github.com/repos/simonw/datasette/issues/2114,1670470516,IC_kwDOBm6k_c5jkVt0,9599,simonw,2023-08-09T00:09:55Z,2023-08-09T00:10:32Z,OWNER,"I decided to figure out where the `_size=` parameter here comes from: https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/database.py#L576-L588 It took some spelunking, but it looks like I added that five years ago as part of CSV streaming in https://github.com/simonw/datasette/commit/fc3660cfad7668dbce6ead12766e048fc1f78b11 It's there purely to allow CSV streaming to set `_size=""max""`. And since CSV streaming is still just a table thing, not a query thing, I can drop it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822940263,Implement canned queries against new query JSON work, https://github.com/simonw/datasette/issues/2114#issuecomment-1670454222,https://api.github.com/repos/simonw/datasette/issues/2114,1670454222,IC_kwDOBm6k_c5jkRvO,9599,simonw,2023-08-08T23:44:42Z,2023-08-08T23:44:42Z,OWNER,"The tests for this are still passing: ```bash pytest tests/test_canned_queries.py ``` Because I left in the old code: https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L752-L767 https://github.com/simonw/datasette/blob/cd57b0f71234273156cb1eba3f9153b9e27ac14d/datasette/views/table.py#L701-L727","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822940263,Implement canned queries against new query JSON work, https://github.com/simonw/datasette/issues/2114#issuecomment-1652321419,https://api.github.com/repos/simonw/datasette/issues/2114,1652321419,IC_kwDOBm6k_c5ifGyL,9599,simonw,2023-07-26T18:48:03Z,2023-07-26T18:48:03Z,OWNER,"This is also where I'll bring back writable canned queries: https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L281-L334","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822940263,Implement canned queries against new query JSON work, https://github.com/simonw/datasette/issues/2113#issuecomment-1671862711,https://api.github.com/repos/simonw/datasette/issues/2113,1671862711,IC_kwDOBm6k_c5jppm3,9599,simonw,2023-08-09T17:35:22Z,2023-08-09T17:35:22Z,OWNER,"Need `?_extra=columns` too since I'm removing `""columns""` from the default query JSON in https://github.com/simonw/datasette/issues/2136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822939274,Implement and document extras for the new query view page, https://github.com/simonw/datasette/issues/2113#issuecomment-1671725144,https://api.github.com/repos/simonw/datasette/issues/2113,1671725144,IC_kwDOBm6k_c5jpIBY,9599,simonw,2023-08-09T16:07:58Z,2023-08-09T16:07:58Z,OWNER,Punting this out of the 1.0a3 alpha for the moment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822939274,Implement and document extras for the new query view page, https://github.com/simonw/datasette/issues/2113#issuecomment-1668983779,https://api.github.com/repos/simonw/datasette/issues/2113,1668983779,IC_kwDOBm6k_c5jeqvj,9599,simonw,2023-08-08T06:27:42Z,2023-08-08T06:27:42Z,OWNER,"I've now shipped the new query JSON without any extras: https://latest.datasette.io/fixtures.json?sql=select+*+from+facetable There aren't actually that many extras that make sense for the query page. These are the most obvious: - `query` - adding a `{""sql"": ""..."", ""params"": ""..""}` block, as seen on https://latest.datasette.io/fixtures/facetable.json?_extra=query&_size=0 - `duration_ms` to show the duration - which isn't an extra on table right now but should be. - `metadata` - https://latest.datasette.io/fixtures/facetable.json?_extra=metadata&_size=0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822939274,Implement and document extras for the new query view page, https://github.com/simonw/datasette/issues/2112#issuecomment-1668798642,https://api.github.com/repos/simonw/datasette/issues/2112,1668798642,IC_kwDOBm6k_c5jd9iy,9599,simonw,2023-08-08T02:01:01Z,2023-08-08T02:01:01Z,OWNER,Shipped: https://latest.datasette.io/fixtures?sql=select+*+from+facetable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=..., https://github.com/simonw/datasette/issues/2112#issuecomment-1652751140,https://api.github.com/repos/simonw/datasette/issues/2112,1652751140,IC_kwDOBm6k_c5igvsk,9599,simonw,2023-07-27T00:52:53Z,2023-07-27T00:52:53Z,OWNER,"Maybe I teach `datasette.render_template()` to take a dataclass instance as an optional alternative to a dictionary, and if it gets one it turns it into something that works well in the template context.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=..., https://github.com/simonw/datasette/issues/2112#issuecomment-1652746185,https://api.github.com/repos/simonw/datasette/issues/2112,1652746185,IC_kwDOBm6k_c5igufJ,9599,simonw,2023-07-27T00:46:37Z,2023-07-27T00:46:37Z,OWNER,"Got Claude to generate this code: ```python from dataclasses import dataclass, field @dataclass class QueryContext: database: str = field(metadata={""help"": ""The name of the database being queried""}) query: dict = field(metadata={""help"": ""The SQL query object containing the `sql` string""}) canned_query: str = field(metadata={""help"": ""The name of the canned query if this is a canned query""}) private: bool = field(metadata={""help"": ""Boolean indicating if this is a private database""}) urls: dict = field(metadata={""help"": ""Object containing URL helpers like `database()`""}) canned_write: bool = field(metadata={""help"": ""Boolean indicating if this canned query allows writes""}) db_is_immutable: bool = field(metadata={""help"": ""Boolean indicating if this database is immutable""}) error: str = field(metadata={""help"": ""Any query error message""}) hide_sql: bool = field(metadata={""help"": ""Boolean indicating if the SQL should be hidden""}) show_hide_link: str = field(metadata={""help"": ""The URL to toggle showing/hiding the SQL""}) show_hide_text: str = field(metadata={""help"": ""The text for the show/hide SQL link""}) editable: bool = field(metadata={""help"": ""Boolean indicating if the SQL can be edited""}) allow_execute_sql: bool = field(metadata={""help"": ""Boolean indicating if custom SQL can be executed""}) tables: list = field(metadata={""help"": ""List of table objects in the database""}) named_parameter_values: dict = field(metadata={""help"": ""Dictionary of parameter names/values""}) csrftoken: callable = field(metadata={""help"": ""Function to generate a CSRF token""}) edit_sql_url: str = field(metadata={""help"": ""URL to edit the SQL for a canned query""}) display_rows: list = field(metadata={""help"": ""List of result rows to display""}) columns: list = field(metadata={""help"": ""List of column names""}) renderers: dict = field(metadata={""help"": ""Dictionary of renderer name to URL""}) url_csv: str = field(metadata={""help"": ""URL for CSV export""}) metadata: dict = field(metadata={""help"": ""Metadata about the query/database""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=..., https://github.com/simonw/datasette/issues/2112#issuecomment-1652723029,https://api.github.com/repos/simonw/datasette/issues/2112,1652723029,IC_kwDOBm6k_c5igo1V,9599,simonw,2023-07-27T00:25:49Z,2023-07-27T00:25:49Z,OWNER,"Getting the query page working with all of the necessary HTML context variables is a slog. I uploaded `query.html` to Claude and asked it ""List of every template context variable with a detailed description of each one"": <img width=""518"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/9c7f3292-d3f3-49bc-9263-dc39bed5aa58""> > Based on the template code provided, here is a list of the template context variables and a description of each one: > > - `database` - The name of the database being queried. > - `query` - The SQL query object containing the `sql` string. > - `canned_query` - The name of the canned query if this is a canned query. > - `private` - Boolean indicating if this is a private database. > - `urls` - Object containing URL helpers like `database()`. > - `canned_write` - Boolean indicating if this canned query allows writes. > - `db_is_immutable` - Boolean indicating if this database is immutable. > - `error` - Any query error message. > - `hide_sql` - Boolean indicating if the SQL should be hidden. > - `show_hide_link` - The URL to toggle showing/hiding the SQL. > - `show_hide_text` - The text for the show/hide SQL link. > - `editable` - Boolean indicating if the SQL can be edited. > - `allow_execute_sql` - Boolean indicating if custom SQL can be executed. > - `tables` - List of table objects in the database. > - `named_parameter_values` - Dictionary of parameter names/values. > - `csrftoken()` - Function to generate a CSRF token. > - `edit_sql_url` - URL to edit the SQL for a canned query. > - `display_rows` - List of result rows to display. > - `columns` - List of column names. > - `renderers` - Dictionary of renderer name to URL. > - `url_csv` - URL for CSV export. > - `metadata` - Metadata about the query/database. > - `crumbs` - Breadcrumb navigation object. > > So in summary, it includes variables for the database, query, results, URLs, permissions, and metadata to power the template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=..., https://github.com/simonw/datasette/issues/2112#issuecomment-1652722208,https://api.github.com/repos/simonw/datasette/issues/2112,1652722208,IC_kwDOBm6k_c5igoog,9599,simonw,2023-07-27T00:24:50Z,2023-07-27T00:25:16Z,OWNER,"I changed my mind, I'm not going to do this with a format hook. Also: https://github.com/simonw/datasette/issues/2109#issuecomment-1652715823 I've decided to let HTML have its own special context. I'm going to try formalizing that as a `@dataclass` with fields with `metadata={""help"": ...}` to see what that looks like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822938661,Build HTML version of /content?sql=..., https://github.com/simonw/datasette/issues/2111#issuecomment-1668798261,https://api.github.com/repos/simonw/datasette/issues/2111,1668798261,IC_kwDOBm6k_c5jd9c1,9599,simonw,2023-08-08T02:00:22Z,2023-08-08T02:00:37Z,OWNER,"Shipped: https://latest.datasette.io/_memory.json?sql=select+sqlite_version() ```json { ""rows"": [ { ""sqlite_version()"": ""3.34.1"" } ], ""columns"": [ ""sqlite_version()"" ], ""ok"": true, ""truncated"": false } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=..., https://github.com/simonw/datasette/issues/2111#issuecomment-1652407208,https://api.github.com/repos/simonw/datasette/issues/2111,1652407208,IC_kwDOBm6k_c5ifbuo,9599,simonw,2023-07-26T19:54:20Z,2023-07-26T19:54:20Z,OWNER,"I implemented `/content` and `/content.json` but I left `/content?sql=...` as this: https://github.com/simonw/datasette/blob/2e40a506ad45b44fd7642474f630a31ef18b5911/datasette/views/database.py#L220-L221","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=..., https://github.com/simonw/datasette/issues/2111#issuecomment-1652354606,https://api.github.com/repos/simonw/datasette/issues/2111,1652354606,IC_kwDOBm6k_c5ifO4u,9599,simonw,2023-07-26T19:16:35Z,2023-07-26T19:17:45Z,OWNER,"I just noticed that this URL: https://latest.datasette.io/fixtures.csv Returns a 500 error right now! <img width=""275"" alt=""image"" src=""https://github.com/simonw/datasette/assets/9599/bd560821-07ba-4e8f-8472-9e677294687d""> It's fine with a `?sql=` query: https://latest.datasette.io/fixtures.csv?sql=select+*+from+facetable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=..., https://github.com/simonw/datasette/issues/2111#issuecomment-1652318269,https://api.github.com/repos/simonw/datasette/issues/2111,1652318269,IC_kwDOBm6k_c5ifGA9,9599,simonw,2023-07-26T18:45:23Z,2023-07-26T18:45:23Z,OWNER,"To avoid confusion I'll start by having `/content` return a HTML ""TODO: implement this"" message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822937426,Implement new /content.json?sql=..., https://github.com/simonw/datasette/issues/2110#issuecomment-1652330111,https://api.github.com/repos/simonw/datasette/issues/2110,1652330111,IC_kwDOBm6k_c5ifI5_,9599,simonw,2023-07-26T18:55:31Z,2023-07-26T18:55:31Z,OWNER,Changed my mind on this - I'm going to have the `query_view` mapped to `/db` but have the first code on there notice if `?sql=` is missing and return a `database_view()` function instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822936521,Merge database index page and query view, https://github.com/simonw/datasette/issues/2109#issuecomment-1652721111,https://api.github.com/repos/simonw/datasette/issues/2109,1652721111,IC_kwDOBm6k_c5igoXX,9599,simonw,2023-07-27T00:23:28Z,2023-07-27T00:23:37Z,OWNER,"Once again I'm tempted to formalize the extra HTML context as a `dataclass` so I can ensure it is documented correctly. Here's an example of Hugging Face doing that: https://github.com/huggingface/transformers/blob/1689aea73346816b936b84932e12b774974e61a6/src/transformers/training_args.py#L622C1-L624 ```python from dataclasses import dataclass, field @dataclass class TrainingArguments: output_dir: str = field( metadata={""help"": ""The output directory where the model predictions and checkpoints will be written.""}, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working, https://github.com/simonw/datasette/issues/2109#issuecomment-1652715823,https://api.github.com/repos/simonw/datasette/issues/2109,1652715823,IC_kwDOBm6k_c5ignEv,9599,simonw,2023-07-27T00:17:04Z,2023-07-27T00:18:07Z,OWNER,"New decision: I had originally decided that the HTML view would just use data that could otherwise be extracted from the JSON view if you fed in enough extras. I've changed my mind. I'm OK with the HTML view getting a few bonus things available in its context, provided those are clearly documented for template authors. This should help avoid me having to make many changes to the templates themselves. HTML-specific stuff will be things like `csrftoken()` and `edit_sql_url` and `show_hide_link` and `database_color` and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working, https://github.com/simonw/datasette/issues/2109#issuecomment-1652325193,https://api.github.com/repos/simonw/datasette/issues/2109,1652325193,IC_kwDOBm6k_c5ifHtJ,9599,simonw,2023-07-26T18:51:15Z,2023-07-26T18:51:15Z,OWNER,"Here's the code I'm going to be entirely replacing: https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/database.py#L213-L530 Plus this weird class in `views/table.py`: https://github.com/simonw/datasette/blob/18dd88ee4d78fe9d760e9da96028ae06d938a85c/datasette/views/table.py#L701-L749","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working, https://github.com/simonw/datasette/issues/2109#issuecomment-1652295866,https://api.github.com/repos/simonw/datasette/issues/2109,1652295866,IC_kwDOBm6k_c5ifAi6,9599,simonw,2023-07-26T18:26:18Z,2023-07-26T18:26:18Z,OWNER,"I'm going to do this work in a fresh branch, copying bits over from here as needed: - https://github.com/simonw/datasette/pull/2053","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working, https://github.com/simonw/datasette/issues/2109#issuecomment-1652294920,https://api.github.com/repos/simonw/datasette/issues/2109,1652294920,IC_kwDOBm6k_c5ifAUI,9599,simonw,2023-07-26T18:25:34Z,2023-07-26T18:25:34Z,OWNER,"OK, these issues will do for the plan.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1822934563,Plan for getting the new JSON format query views working, https://github.com/simonw/datasette/pull/2107#issuecomment-1655678215,https://api.github.com/repos/simonw/datasette/issues/2107,1655678215,IC_kwDOBm6k_c5ir6UH,49699333,dependabot[bot],2023-07-28T13:23:16Z,2023-07-28T13:23:16Z,CONTRIBUTOR,Superseded by #2124.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1820346348,Bump sphinx from 6.1.3 to 7.1.0, https://github.com/simonw/datasette/pull/2107#issuecomment-1649863533,https://api.github.com/repos/simonw/datasette/issues/2107,1649863533,IC_kwDOBm6k_c5iVutt,22429695,codecov[bot],2023-07-25T13:37:07Z,2023-07-25T13:37:07Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`278ac91`)](https://app.codecov.io/gh/simonw/datasette/commit/278ac91a4d68da333a6ed74a2c75b3ec9db8bd19?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38% compared to head [(`7fc79ba`)](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.38%. <details><summary>Additional details and impacted files</summary> ```diff @@ Coverage Diff @@ ## main #2107 +/- ## ======================================= Coverage 92.38% 92.38% ======================================= Files 39 39 Lines 5805 5805 ======================================= Hits 5363 5363 Misses 442 442 ``` </details> [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2107?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1820346348,Bump sphinx from 6.1.3 to 7.1.0, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1668113177,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1668113177,IC_kwDOCGYnMM5jbWMZ,25778,eyeseast,2023-08-07T15:41:49Z,2023-08-07T15:41:49Z,CONTRIBUTOR,I wonder if this should be two hooks: input and output. The current `--csv` (and `--tsv`) options apply to both. Haven't looked at how it's implemented. Or maybe it's one hook that returns a format for reading and for writing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648339661,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648339661,IC_kwDOCGYnMM5iP6rN,25778,eyeseast,2023-07-24T17:44:30Z,2023-07-24T17:44:30Z,CONTRIBUTOR,"> A related feature would be support for plugins to add new ways of ingesting data - currently sqlite-utils insert works against JSON, newline-JSON, CSV and TSV. This is my goal, to have one plugin that handles input and output symmetrically. I'd like to be able to do something like this: ```sh sqlite-utils insert data.db table file.geojson --format geojson # ... explore and manipulate in Datasette sqlite-utils query data.db ... --format geojson > output.geojson ``` This would work especially well with [datasette-query-files](https://github.com/eyeseast/datasette-query-files), since I already have the queries I need saved in standalone SQL files. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648325682,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648325682,IC_kwDOCGYnMM5iP3Qy,9599,simonw,2023-07-24T17:33:10Z,2023-07-24T17:33:10Z,OWNER,"A related feature would be support for plugins to add new ways of _ingesting_ data - currently `sqlite-utils insert` works against JSON, newline-JSON, CSV and TSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648324312,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648324312,IC_kwDOCGYnMM5iP27Y,9599,simonw,2023-07-24T17:32:21Z,2023-07-24T17:32:21Z,OWNER,I'm currently leaning very slightly towards `--format geojson` - and explaining in the docs that `--format` is for formats added by plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648323482,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648323482,IC_kwDOCGYnMM5iP2ua,9599,simonw,2023-07-24T17:31:56Z,2023-07-24T17:31:56Z,OWNER,"The main blocker here is coming up with a design. The challenge is cleanly integrating it with the existing format options: https://github.com/simonw/sqlite-utils/blob/8bee14588687b66c54c7a3dfae5de2f9cc7cac3f/docs/cli-reference.rst#L114-L132 I'd like to avoid breaking backwards compatibility. The most obvious solution is to add a `--format x` option. This is slightly confusing as `--fmt` already exists. Or... `--fmt` could be enhanced to handle plugins too - and the existing set of formats (from the `tabulate` integration) could be refactored to use the new hook. If I ever do release `sqlite-utils 4` I'm tempted to move that tabulate stuff into a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294,Plugin hook for adding new output formats, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1684235760,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1684235760,IC_kwDOCGYnMM5kY2Xw,9599,simonw,2023-08-18T17:43:11Z,2023-08-18T17:43:11Z,OWNER,"Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it: https://github.com/simonw/sqlite-utils-fast-fks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,simonw,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133 Needs updating like so: ```diff diff --git a/tests/test_extract.py b/tests/test_extract.py index 70ad0cf..fd52534 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db): assert fresh_db[""tree""].schema == ( 'CREATE TABLE ""tree"" (\n' "" [name] TEXT,\n"" - "" [common_name_latin_name_id] INTEGER,\n"" - "" FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n"" + "" [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n"" "")"" ) assert ( ``` Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in. I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683076325,IC_kwDOCGYnMM5kUbTl,9599,simonw,2023-08-17T22:48:36Z,2023-08-17T22:48:36Z,OWNER,"I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism. It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,simonw,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,simonw,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this: ```python drop_foreign_keys: Optional[Iterable] = None, ``` Should be this: ```python drop_foreign_keys: Optional[Iterable[str]] = None, ``` Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,simonw,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments: ```python def transform( self, *, # ... # This one exists already: drop_foreign_keys: Optional[Iterable] = None, # These two are new. This one specifies keys to add: add_foreign_keys: Optional[ForeignKeysType] = None, # Or this one causes them all to be replaced with the new definitions: foreign_keys: Optional[ForeignKeysType] = None, ``` There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,simonw,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149 At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`. Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,simonw,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others: - https://github.com/simonw/llm/issues/60 - https://github.com/simonw/llm/issues/116 - https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,simonw,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work. It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872 Improving this code to support adding foreign keys as well would be pretty simple. And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`, https://github.com/dogsheep/dogsheep-beta/issues/37#issuecomment-1646950438,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/37,1646950438,IC_kwDOC8SPRc5iKngm,10352819,rprimet,2023-07-23T20:18:26Z,2023-07-23T20:18:26Z,NONE,"My bad, although I could not find how to use `url`s, I was using a function call syntax for the filters instead of jinja's pipe syntax; closing this issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817281557,cannot use jinja filters in display?, https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646753678,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646753678,IC_kwDOCGYnMM5iJ3eO,9599,simonw,2023-07-23T05:49:51Z,2023-07-23T05:49:51Z,OWNER,Done - bottom of https://sqlite-utils.datasette.io/en/latest/changelog.html#id116,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,Backfill the release notes prior to 0.4, https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646752814,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646752814,IC_kwDOCGYnMM5iJ3Qu,9599,simonw,2023-07-23T05:43:13Z,2023-07-23T05:43:13Z,OWNER,"Using `llm` in my `sqlite-utils` local checkout: ```bash git diff 0.2 0.3 | llm -s 'release notes as markdown bullet points' ``` > - Added a new `Table` class in the `db` module which represents a table in a SQLite database > - The `Table` class has properties that provide information about the table, such as column names, foreign keys, and schema > - The `Table` class also has a `count` property that returns the number of rows in the table > - The `Table` class has a `create` method that can be used to create a new table with specified columns, primary key, and foreign keys > - Updated the `Database` class in the `__init__` module to include the `Table` class and expose it via the `__all__` variable > - Added new tests for creating tables and introspecting tables ```bash git diff 0.3 0.3.1 | llm -s 'release notes as markdown bullet points' ``` > - Updated PyPI version badge to display the latest version > - Updated Travis CI badge to link to the master branch > - Added a badge for documentation status > - Added a license badge > - Added a description of the package > - Added installation instructions > - Added a link to the documentation > - Added a ""Related projects"" section and linked to Datasette and csvs-to-sqlite > - Updated the version in setup.py to 0.3.1 ```bash git diff 0.3.1 0.4 | llm -s 'release notes as markdown bullet points' ``` > - Added an `enable_fts` method to the `Table` class that enables full-text search on specified columns > - Added a `populate_fts` method to the `Table` class that refreshes the search index for full-text search > - Added a `search` method to the `Table` class that allows searching the table using full-text search > - Updated the `setup.py` script to version 0.4 > - Updated the `db.py` file to include the `enable_fts`, `populate_fts`, and `search` methods > - Added test cases for the `enable_fts`, `populate_fts`, and `search` methods in the `test_enable_fts.py` file > - Added test cases for the `count`, `columns`, and `schema` properties in the `test_introspect.py` file > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390,Backfill the release notes prior to 0.4, https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646688288,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646688288,IC_kwDOCGYnMM5iJngg,9599,simonw,2023-07-22T23:08:22Z,2023-07-22T23:08:22Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,Python API ability to opt-out of connection plugins, https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646687461,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646687461,IC_kwDOCGYnMM5iJnTl,9599,simonw,2023-07-22T23:01:44Z,2023-07-22T23:01:44Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568,Python API ability to opt-out of connection plugins, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688748,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688748,IC_kwDOCGYnMM5iJnns,9599,simonw,2023-07-22T23:12:03Z,2023-07-22T23:13:14Z,OWNER,"Just tried this out by adding the example from the docs to my `sqlite-utils-hello-world` plugin and running: ```bash sqlite-utils install -e ../sqlite-utils-hello-world ``` ``` Obtaining file:///Users/simon/Dropbox/Development/sqlite-utils-hello-world ... ``` Then: ```bash sqlite-utils memory ""select hello('simon')"" ``` ```json [{""hello('simon')"": ""Hello, simon!""}] ``` Also: ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query(""select hello('simon')"")) [{""hello('simon')"": 'Hello, simon!'}] >>> db2 = sqlite_utils.Database(memory=True, execute_plugins=False) >>> list(db2.query(""select hello('simon')"")) Traceback (most recent call last): File ""<stdin>"", line 1, in <module> File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 494, in query cursor = self.execute(sql, params or tuple()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 512, in execute return self.conn.execute(sql, parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sqlean.dbapi2.OperationalError: no such function: hello ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688339,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688339,IC_kwDOCGYnMM5iJnhT,9599,simonw,2023-07-22T23:08:49Z,2023-07-22T23:08:49Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#prepare-connection-conn,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646686477,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646686477,IC_kwDOCGYnMM5iJnEN,9599,simonw,2023-07-22T22:52:56Z,2023-07-22T22:52:56Z,OWNER,"Alex built this in: - #573","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185,`prepare_connection()` plugin hook, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646687103,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646687103,IC_kwDOCGYnMM5iJnN_,9599,simonw,2023-07-22T22:58:35Z,2023-07-22T22:58:35Z,OWNER,"https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn <img width=""773"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/3211a37a-c30d-47f9-bc10-c2c3240b6f6e""> ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686675,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686675,IC_kwDOCGYnMM5iJnHT,9599,simonw,2023-07-22T22:54:38Z,2023-07-22T22:54:38Z,OWNER,"<img width=""779"" alt=""image"" src=""https://github.com/simonw/sqlite-utils/assets/9599/c4e5769d-f1a3-427b-b260-882fce5c7205""> Glitch in the rendered documentation from https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686382,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686382,IC_kwDOCGYnMM5iJnCu,22429695,codecov[bot],2023-07-22T22:52:22Z,2023-07-22T22:56:49Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00`**% and project coverage change: **`+0.03`** :tada: > Comparison is base [(`86a352f`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/86a352f8b713ca30a65a2048170bd510d529d8c4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.09% compared to head [(`faf398f`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.12%. <details><summary>Additional details and impacted files</summary> ```diff @@ Coverage Diff @@ ## main #573 +/- ## ========================================== + Coverage 96.09% 96.12% +0.03% ========================================== Files 8 8 Lines 2794 2816 +22 ========================================== + Hits 2685 2707 +22 Misses 109 109 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/\_\_init\_\_.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL19faW5pdF9fLnB5) | `100.00% <100.00%> (ø)` | | | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.36% <100.00%> (+0.01%)` | :arrow_up: | | [sqlite\_utils/hookspecs.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) </details> [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/573?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686332,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686332,IC_kwDOCGYnMM5iJnB8,9599,simonw,2023-07-22T22:52:01Z,2023-07-22T22:52:01Z,OWNER,I was literally seconds away from shipping version 3.34 but I this looks good so I'm going to try and get it in there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522,feat: Implement a prepare_connection plugin hook, https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646681192,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646681192,IC_kwDOCGYnMM5iJlxo,9599,simonw,2023-07-22T22:12:08Z,2023-07-22T22:12:08Z,OWNER,Found it: https://github.com/simonw/sqlite-utils/blob/18f190e28334d821be78a1dbbf31d7610fc1f9c1/.github/workflows/test.yml#L29-L31,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910,Don't test Python 3.7 against textual, https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646660777,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646660777,IC_kwDOCGYnMM5iJgyp,9599,simonw,2023-07-22T19:59:13Z,2023-07-22T19:59:13Z,OWNER,I don't know where that is coming from! I can't see `textual` as a dependency we are pulling in anywhere.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910,Don't test Python 3.7 against textual, https://github.com/simonw/sqlite-utils/issues/571#issuecomment-1646682686,https://api.github.com/repos/simonw/sqlite-utils/issues/571,1646682686,IC_kwDOCGYnMM5iJmI-,9599,simonw,2023-07-22T22:23:47Z,2023-07-22T22:23:47Z,OWNER,"Demo: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].insert({""name"": ""Cleo""}) <Table foo (name)> >>> print(db[""foo""].schema) CREATE TABLE [foo] ( [name] TEXT ) >>> db[""foo""].transform(not_null={""name""}) <Table foo (name)> >>> print(db[""foo""].schema) CREATE TABLE ""foo"" ( [name] TEXT NOT NULL ) >>> db[""foo""].transform(types={""name"": int}, keep_table=""kept"") <Table foo (name)> >>> print(db.schema) CREATE TABLE ""kept"" ( [name] TEXT NOT NULL ); CREATE TABLE ""foo"" ( [name] INTEGER NOT NULL ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816876211,`.transform(keep_table=...)` option, https://github.com/simonw/datasette/issues/2106#issuecomment-1652298879,https://api.github.com/repos/simonw/datasette/issues/2106,1652298879,IC_kwDOBm6k_c5ifBR_,9599,simonw,2023-07-26T18:28:33Z,2023-07-26T18:28:33Z,OWNER,"Applied the same fix as here: - https://github.com/simonw/llm/issues/136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816857442,`datasette install -e` option, https://github.com/simonw/datasette/issues/2106#issuecomment-1646648262,https://api.github.com/repos/simonw/datasette/issues/2106,1646648262,IC_kwDOBm6k_c5iJdvG,9599,simonw,2023-07-22T18:42:20Z,2023-07-22T18:42:20Z,OWNER,"Tested this locally with: ```bash datasette install -e ../datasette-graphql ``` Running `datasette plugins` confirmed that the plugin had been installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816857442,`datasette install -e` option, https://github.com/simonw/datasette/issues/2106#issuecomment-1646646931,https://api.github.com/repos/simonw/datasette/issues/2106,1646646931,IC_kwDOBm6k_c5iJdaT,9599,simonw,2023-07-22T18:34:04Z,2023-07-22T18:34:04Z,OWNER,Here's the diff for adding it to `sqlite-utils`: https://github.com/simonw/sqlite-utils/commit/ef31210bf06f920e0890e171c3198f0b0dc8d72d,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816857442,`datasette install -e` option, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646654275,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646654275,IC_kwDOCGYnMM5iJfND,9599,simonw,2023-07-22T19:19:35Z,2023-07-22T19:19:35Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646645990,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646645990,IC_kwDOCGYnMM5iJdLm,9599,simonw,2023-07-22T18:29:40Z,2023-07-22T18:30:16Z,OWNER,"Example plugin: https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf ```bash sqlite-utils install https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf/archive/cd7960e476c441a3c5f619e2a44a641a39b91467.zip ``` `pyproject.toml`: ```toml [project] name = ""sqlite-utils-hello-world"" version = ""0.1"" [project.entry-points.sqlite_utils] hello_world = ""sqlite_utils_hello_world"" ``` `sqlite_utils_hello_world.py`: ```python import click import sqlite_utils @sqlite_utils.hookimpl def register_commands(cli): @cli.command() def hello_world(): ""Say hello world"" click.echo(""Hello world!"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646643676,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646643676,IC_kwDOCGYnMM5iJcnc,9599,simonw,2023-07-22T18:18:24Z,2023-07-22T18:18:24Z,OWNER,"Here's where I added that to LLM: https://github.com/simonw/llm/commit/a396950f7934e82a9968703bb3ce9ab7ab62f7f8 - https://github.com/simonw/llm/issues/49","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402,register_command plugin hook, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646655272,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646655272,IC_kwDOCGYnMM5iJfco,9599,simonw,2023-07-22T19:25:35Z,2023-07-22T19:25:35Z,OWNER,"Here's why that test broke: https://github.com/simonw/sqlite-utils/blob/58b577279fcd5ef6ce88f88b28668dffebfe7f44/sqlite_utils/db.py#L960-L964 I added an extra `if self[name].exists()` check to the `db.create_table()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646654818,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646654818,IC_kwDOCGYnMM5iJfVi,9599,simonw,2023-07-22T19:22:40Z,2023-07-22T19:22:40Z,OWNER,"I think this broke a test: ``` $ pytest tests/test_tracer.py =============================================== test session starts ================================================ platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0 rootdir: /Users/simon/Dropbox/Development/sqlite-utils plugins: icdiff-0.6, hypothesis-6.68.2 collected 2 items tests/test_tracer.py F. [100%] ===================================================== FAILURES ===================================================== ___________________________________________________ test_tracer ____________________________________________________ def test_tracer(): collected = [] db = Database( memory=True, tracer=lambda sql, params: collected.append((sql, params)) ) db[""dogs""].insert({""name"": ""Cleopaws""}) db[""dogs""].enable_fts([""name""]) db[""dogs""].search(""Cleopaws"") > assert collected == [ (""PRAGMA recursive_triggers=on;"", None), (""select name from sqlite_master where type = 'view'"", None), (""select name from sqlite_master where type = 'table'"", None), (""select name from sqlite_master where type = 'view'"", None), (""CREATE TABLE [dogs] (\n [name] TEXT\n);\n "", None), (""select name from sqlite_master where type = 'view'"", None), (""INSERT INTO [dogs] ([name]) VALUES (?);"", [""Cleopaws""]), (""select name from sqlite_master where type = 'view'"", None), ( ""CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n [name],\n content=[dogs]\n)"", None, ), ( ""INSERT INTO [dogs_fts] (rowid, [name])\n SELECT rowid, [name] FROM [dogs];"", None, ), (""select name from sqlite_master where type = 'view'"", None), ] E assert equals failed E [ [ E ('PRAGMA recursive_triggers=on;', None), ('PRAGMA recursive_triggers=on;', None), E ( E ""select name from sqlite_master where type = E 'view'"", E None, ... E E ...Full output truncated (13 lines hidden), use '-vv' to show tests/test_tracer.py:12: AssertionError ============================================= short test summary info ============================================== FAILED tests/test_tracer.py::test_tracer - assert equals failed =========================================== 1 failed, 1 passed in 0.05s ============================================ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653610,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653610,IC_kwDOCGYnMM5iJfCq,9599,simonw,2023-07-22T19:14:56Z,2023-07-22T19:14:56Z,OWNER,"Manual testing of CLI command as well: ``` $ sqlite-utils create-table /tmp/f.db foo id integer $ sqlite-utils create-table /tmp/f.db foo id integer Error: Table ""foo"" already exists. Use --replace to delete and replace it. $ sqlite-utils create-table /tmp/f.db foo id integer --replace $ sqlite-utils create-table /tmp/f.db foo id $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER ); $ sqlite-utils create-table /tmp/f.db foo id integer name str --transform Error: column types must be one of ('INTEGER', 'TEXT', 'FLOAT', 'BLOB') $ sqlite-utils create-table /tmp/f.db foo id integer name text --transform $ sqlite-utils schema /tmp/f.db CREATE TABLE ""foo"" ( [id] INTEGER, [name] TEXT ); $ sqlite-utils create-table /tmp/f.db foo id integer name text --ignore $ sqlite-utils create-table /tmp/f.db foo id integer name text --replace $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653382,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653382,IC_kwDOCGYnMM5iJe_G,9599,simonw,2023-07-22T19:13:20Z,2023-07-22T19:13:20Z,OWNER,"Demo: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].create({""id"": int}) <Table foo (id)> >>> db[""foo""].create({""id"": int}) Traceback (most recent call last): File ""<stdin>"", line 1, in <module> File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1647, in create self.db.create_table( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1030, in create_table self.execute(sql) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 510, in execute return self.conn.execute(sql) ^^^^^^^^^^^^^^^^^^^^^^ sqlean.dbapi2.OperationalError: table [foo] already exists >>> db[""foo""].create({""id"": int}, ignore=True) <Table foo (id)> >>> db[""foo""].create({""id"": int, ""name"": str}, replace=True) <Table foo (id, name)> >>> db[""foo""].create({""id"": int, ""name"": str, ""age"": int}, transform=True) <Table foo (id, name, age)> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646652105,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646652105,IC_kwDOCGYnMM5iJerJ,9599,simonw,2023-07-22T19:05:13Z,2023-07-22T19:05:13Z,OWNER,I think this is `replace=True` and `ignore=True` to match the CLI. And refactoring the CLI to use them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642959,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642959,IC_kwDOCGYnMM5iJccP,9599,simonw,2023-07-22T18:14:49Z,2023-07-22T18:14:49Z,OWNER,Here's where those are implemented for the `create-table` CLI command: https://github.com/simonw/sqlite-utils/blob/f7af23837deab5c98dae9441d1f68318065d7d8c/sqlite_utils/cli.py#L1543-L1564,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642666,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642666,IC_kwDOCGYnMM5iJcXq,9599,simonw,2023-07-22T18:13:19Z,2023-07-22T18:13:19Z,OWNER,"https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table ```bash sqlite-utils create-table ... --replace ``` That also has `--ignore`: ``` --ignore If table already exists, do nothing --replace If table already exists, replace it --transform If table already exists, try to transform the schema ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056,"table.create(..., replace=True)", https://github.com/simonw/datasette/issues/2105#issuecomment-1646247246,https://api.github.com/repos/simonw/datasette/issues/2105,1646247246,IC_kwDOBm6k_c5iH71O,2235371,aki-k,2023-07-21T21:16:37Z,2023-07-21T21:17:09Z,NONE,"I must be doing something wrong. On page https://192.168.1.3:5432/datasette-llm/logs/_llm_migrations there's the option to filter the results, there's an Apply button. It also tries to load an URL with an extra URL element in it: https://192.168.1.3:5432/datasette-llm/datasette-llm/logs/_llm_migrations?_sort=name&name__contains=initial","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1811824307,When reverse proxying datasette with nginx an URL element gets erronously added, https://github.com/simonw/datasette/issues/2105#issuecomment-1643873232,https://api.github.com/repos/simonw/datasette/issues/2105,1643873232,IC_kwDOBm6k_c5h-4PQ,2235371,aki-k,2023-07-20T12:53:00Z,2023-07-20T12:53:34Z,NONE,"I forgot to add that I followed these instructions to set up the python llm: https://simonwillison.net/2023/Jul/18/accessing-llama-2/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1811824307,When reverse proxying datasette with nginx an URL element gets erronously added, https://github.com/simonw/datasette/issues/2105#issuecomment-1642013043,https://api.github.com/repos/simonw/datasette/issues/2105,1642013043,IC_kwDOBm6k_c5h3yFz,2235371,aki-k,2023-07-19T12:41:36Z,2023-07-19T12:41:36Z,NONE,"The same problem can be seen in the links: Advanced export JSON shape: default, array, newline-delimited, object","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1811824307,When reverse proxying datasette with nginx an URL element gets erronously added, https://github.com/simonw/datasette/issues/2104#issuecomment-1641082395,https://api.github.com/repos/simonw/datasette/issues/2104,1641082395,IC_kwDOBm6k_c5h0O4b,15178711,asg017,2023-07-18T22:41:37Z,2023-07-18T22:41:37Z,CONTRIBUTOR,"For filtering virtual table's ""shadow tables"" (ex the FTS5 _content and most the spatialite tables), you can use `pragma_table_list` (first appeared in SQLite 3.37 (2021-11-27), which has a `type` column that calls out `type=""shadow""` tables https://www.sqlite.org/pragma.html#pragma_table_list","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1808215339,Tables starting with an underscore should be treated as hidden, https://github.com/simonw/datasette/issues/2104#issuecomment-1638552567,https://api.github.com/repos/simonw/datasette/issues/2104,1638552567,IC_kwDOBm6k_c5hqlP3,9599,simonw,2023-07-17T17:14:20Z,2023-07-17T17:14:20Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/database.py#L391-L451,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1808215339,Tables starting with an underscore should be treated as hidden, https://github.com/simonw/datasette/issues/2102#issuecomment-1696378239,https://api.github.com/repos/simonw/datasette/issues/2102,1696378239,IC_kwDOBm6k_c5lHK1_,9599,simonw,2023-08-28T20:38:01Z,2023-08-28T20:38:01Z,OWNER,"I want to test ""for this set of restrictions, does a GET/POST to this path return 200 or 403""?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1696361304,https://api.github.com/repos/simonw/datasette/issues/2102,1696361304,IC_kwDOBm6k_c5lHGtY,9599,simonw,2023-08-28T20:23:47Z,2023-08-28T20:24:35Z,OWNER,"Here's an existing relevant test: https://github.com/simonw/datasette/blob/2e2825869fc2655b5fcadc743f6f9dec7a49bc65/tests/test_permissions.py#L616-L666 It's not quite right for this new set of tests though, since they need to be exercising actual endpoints (`/.json` etc) in order to check that this works correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691842259,https://api.github.com/repos/simonw/datasette/issues/2102,1691842259,IC_kwDOBm6k_c5k13bT,9599,simonw,2023-08-24T14:55:54Z,2023-08-24T14:55:54Z,OWNER,"So what's needed to finish this is: - Tests that demonstrate that nothing is revealed that shouldn't be by tokens restricted in this way - Similar tests for other permissions like `create-table` that check that they work (and don't also need `view-instance` etc). - Documentation","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691824713,https://api.github.com/repos/simonw/datasette/issues/2102,1691824713,IC_kwDOBm6k_c5k1zJJ,9599,simonw,2023-08-24T14:45:49Z,2023-08-24T14:45:49Z,OWNER,"I tested this out against a Datasette Cloud instance. I created a restricted token and tested it like this: ```bash curl -H ""Authorization: Bearer $TOKEN"" \ 'https://$INSTANCE/-/actor.json' | jq ``` ```json { ""actor"": { ""id"": ""245"", ""token"": ""dsatok"", ""token_id"": 2, ""_r"": { ""r"": { ""data"": { ""all_stocks"": [ ""vt"" ] } } } } } ``` It can access the `all_stocks` demo table: ```bash curl -H ""Authorization: Bearer $TOKEN"" \ 'https://$INSTANCE/data/all_stocks.json?_size=1' | jq ``` ```json { ""ok"": true, ""next"": ""1"", ""rows"": [ { ""rowid"": 1, ""Date"": ""2013-01-02"", ""Open"": 79.12, ""High"": 79.29, ""Low"": 77.38, ""Close"": 78.43, ""Volume"": 140124866, ""Name"": ""AAPL"" } ], ""truncated"": false } ``` Accessing the database returns just information about that table, even though other tables exist: ```bash curl -H ""Authorization: Bearer $TOKEN"" \ 'https://$INSTANCE/data.json?_size=1' ``` ```json { ""database"": ""data"", ""private"": true, ""path"": ""/data"", ""size"": 3796992, ""tables"": [ { ""name"": ""all_stocks"", ""columns"": [ ""Date"", ""Open"", ""High"", ""Low"", ""Close"", ""Volume"", ""Name"" ], ""primary_keys"": [], ""count"": 8813, ""hidden"": false, ""fts_table"": null, ""foreign_keys"": { ""incoming"": [], ""outgoing"": [] }, ""private"": true } ], ""hidden_count"": 0, ""views"": [], ""queries"": [], ""allow_execute_sql"": false, ""table_columns"": {} } ``` And hitting the top-level `/.json` thing does the same - it reveals that table but not any of the other tables or databases: ```bash curl -H ""Authorization: Bearer $TOKEN"" \ 'https://$INSTANCE/.json?_size=1' ``` ```json { ""data"": { ""name"": ""data"", ""hash"": null, ""color"": ""8d777f"", ""path"": ""/data"", ""tables_and_views_truncated"": [ { ""name"": ""all_stocks"", ""columns"": [ ""Date"", ""Open"", ""High"", ""Low"", ""Close"", ""Volume"", ""Name"" ], ""primary_keys"": [], ""count"": null, ""hidden"": false, ""fts_table"": null, ""num_relationships_for_sorting"": 0, ""private"": false } ], ""tables_and_views_more"": false, ""tables_count"": 1, ""table_rows_sum"": 0, ""show_table_row_counts"": false, ""hidden_table_rows_sum"": 0, ""hidden_tables_count"": 0, ""views_count"": 0, ""private"": false } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691758168,https://api.github.com/repos/simonw/datasette/issues/2102,1691758168,IC_kwDOBm6k_c5k1i5Y,9599,simonw,2023-08-24T14:09:45Z,2023-08-24T14:09:45Z,OWNER,I'm going to implement this in a branch to make it easier to test out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691045051,https://api.github.com/repos/simonw/datasette/issues/2102,1691045051,IC_kwDOBm6k_c5ky0y7,9599,simonw,2023-08-24T05:51:59Z,2023-08-24T05:51:59Z,OWNER,"With that fix in place, this works: ```bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ ""_r"": { ""r"": { ""fixtures"": { ""facetable"": [ ""vt"" ] } } }, ""a"": ""user"" }' ``` But this fails, because it's for a table not explicitly listed: ```bash datasette fixtures.db --get '/fixtures/searchable.json' --actor '{ ""_r"": { ""r"": { ""fixtures"": { ""facetable"": [ ""vt"" ] } } }, ""a"": ""user"" }' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691044283,https://api.github.com/repos/simonw/datasette/issues/2102,1691044283,IC_kwDOBm6k_c5ky0m7,9599,simonw,2023-08-24T05:51:02Z,2023-08-24T05:51:02Z,OWNER,"Also need to confirm that permissions like `insert-row`, `delete-row`, `create-table` etc don't also need special cases to ensure they get through the `view-instance` etc checks, if those exist for those actions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691043475,https://api.github.com/repos/simonw/datasette/issues/2102,1691043475,IC_kwDOBm6k_c5ky0aT,9599,simonw,2023-08-24T05:50:04Z,2023-08-24T05:50:04Z,OWNER,"On first test this seems to work! ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 63a66c3c..9303dac8 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -187,6 +187,30 @@ def permission_allowed_actor_restrictions(datasette, actor, action, resource): return None _r = actor.get(""_r"") + # Special case for view-instance: it's allowed if there are any view-database + # or view-table permissions defined + if action == ""view-instance"": + database_rules = _r.get(""d"") or {} + for rules in database_rules.values(): + if ""vd"" in rules or ""view-database"" in rules: + return None + # Now check resources + resource_rules = _r.get(""r"") or {} + for _database, resources in resource_rules.items(): + for rules in resources.values(): + if ""vt"" in rules or ""view-table"" in rules: + return None + + # Special case for view-database: it's allowed if there are any view-table permissions + # defined within that database + if action == ""view-database"": + database_name = resource + resource_rules = _r.get(""r"") or {} + resources_in_database = resource_rules.get(database_name) or {} + for rules in resources_in_database.values(): + if ""vt"" in rules or ""view-table"" in rules: + return None + # Does this action have an abbreviation? to_check = {action} permission = datasette.permissions.get(action) ``` Needs a LOT of testing to make sure what it's doing is sensible though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691037971,https://api.github.com/repos/simonw/datasette/issues/2102,1691037971,IC_kwDOBm6k_c5kyzET,9599,simonw,2023-08-24T05:42:47Z,2023-08-24T05:42:47Z,OWNER,"I applied a fun trick to help test this out: ```diff diff --git a/datasette/cli.py b/datasette/cli.py index 58f89c1c..830f47ef 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -445,6 +445,10 @@ def uninstall(packages, yes): ""--token"", help=""API token to send with --get requests"", ) +@click.option( + ""--actor"", + help=""Actor to use for --get requests"", +) @click.option(""--version-note"", help=""Additional note to show on /-/versions"") @click.option(""--help-settings"", is_flag=True, help=""Show available settings"") @click.option(""--pdb"", is_flag=True, help=""Launch debugger on any errors"") @@ -499,6 +503,7 @@ def serve( root, get, token, + actor, version_note, help_settings, pdb, @@ -611,7 +616,10 @@ def serve( headers = {} if token: headers[""Authorization""] = ""Bearer {}"".format(token) - response = client.get(get, headers=headers) + cookies = {} + if actor: + cookies[""ds_actor""] = client.actor_cookie(json.loads(actor)) + response = client.get(get, headers=headers, cookies=cookies) click.echo(response.text) exit_code = 0 if response.status == 200 else 1 sys.exit(exit_code) ``` This adds a `--actor` option to `datasette ... --get /path` which makes it easy to test an API endpoint using a fake actor with a set of `_r` restrictions. With that in place I can try this, with a token that has view-instance and view-database and view-table: ```bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ ""_r"": { ""a"": [ ""vi"" ], ""d"": { ""fixtures"": [ ""vd"" ] }, ""r"": { ""fixtures"": { ""facetable"": [ ""vt"" ] } } }, ""a"": ""user"" }' ``` Or this, with a token that just has view-table but is missing the view-database and view-instance: ```bash datasette fixtures.db --get '/fixtures/facetable.json' --actor '{ ""_r"": { ""r"": { ""fixtures"": { ""facetable"": [ ""vt"" ] } } }, ""a"": ""user"" }' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1691036559,https://api.github.com/repos/simonw/datasette/issues/2102,1691036559,IC_kwDOBm6k_c5kyyuP,9599,simonw,2023-08-24T05:40:53Z,2023-08-24T05:40:53Z,OWNER,"There might be an easier way to solve this. Here's some permission checks that run when hitting `/fixtures/facetable.json`: ``` permission_allowed: action=view-table, resource=('fixtures', 'facetable'), actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'} File ""/datasette/views/table.py"", line 727, in table_view_traced view_data = await table_view_data( File ""/datasette/views/table.py"", line 875, in table_view_data visible, private = await datasette.check_visibility( File ""/datasette/app.py"", line 890, in check_visibility await self.ensure_permissions(actor, permissions) permission_allowed: action=view-database, resource=fixtures, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'} File ""/datasette/views/table.py"", line 727, in table_view_traced view_data = await table_view_data( File ""/datasette/views/table.py"", line 875, in table_view_data visible, private = await datasette.check_visibility( File ""/datasette/app.py"", line 890, in check_visibility await self.ensure_permissions(actor, permissions) permission_allowed: action=view-instance, resource=<None>, actor={'_r': {'a': ['vi'], 'd': {'fixtures': ['vd']}, 'r': {'fixtures': {'facetable': ['vt']}}}, 'a': 'user'} File ""/datasette/views/table.py"", line 727, in table_view_traced view_data = await table_view_data( File ""/datasette/views/table.py"", line 875, in table_view_data visible, private = await datasette.check_visibility( File ""/datasette/app.py"", line 890, in check_visibility await self.ensure_permissions(actor, permissions) ``` That's with a token that has the view instance, view database and view table permissions required. But... what if the restrictions logic said that if you have view-table you automatically also get view-database and view-instance? Would that actually let people do anything they shouldn't be able to do? I don't think it would even let them see a list of tables that they weren't allowed to visit, so it might be OK. I'll try that and see how it works.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1690705243,https://api.github.com/repos/simonw/datasette/issues/2102,1690705243,IC_kwDOBm6k_c5kxh1b,9599,simonw,2023-08-23T22:03:54Z,2023-08-23T22:03:54Z,OWNER,Idea: `datasette-permissions-debug` plugin which simply prints out a stacktrace for every permission check so you can see where in the code they are.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1690703764,https://api.github.com/repos/simonw/datasette/issues/2102,1690703764,IC_kwDOBm6k_c5kxheU,9599,simonw,2023-08-23T22:02:14Z,2023-08-23T22:02:14Z,OWNER,"Built this new test: ```python @pytest.mark.asyncio async def test_view_table_token_can_access_table(perms_ds): actor = { ""id"": ""restricted-token"", ""token"": ""dstok"", # Restricted to just view-table on perms_ds_two/t1 ""_r"": {""r"": {""perms_ds_two"": {""t1"": [""vt""]}}}, } cookies = {""ds_actor"": perms_ds.client.actor_cookie(actor)} response = await perms_ds.client.get(""/perms_ds_two/t1.json"", cookies=cookies) assert response.status_code == 200 ``` The test fails. Running it with `pytest --pdb` let me do this: ``` (Pdb) from pprint import pprint (Pdb) pprint(perms_ds._permission_checks) deque([{'action': 'view-table', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': ('perms_ds_two', 't1'), 'result': None, 'used_default': True, 'when': '2023-08-23T21:59:45.117155'}, {'action': 'view-database', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': 'perms_ds_two', 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.117189'}, {'action': 'view-instance', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': None, 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.126751'}, {'action': 'debug-menu', 'actor': {'_r': {'r': {'perms_ds_two': {'t1': ['vt']}}}, 'id': 'restricted-token', 'token': 'dstok'}, 'resource': None, 'result': False, 'used_default': False, 'when': '2023-08-23T21:59:45.126777'}], maxlen=200) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1690693830,https://api.github.com/repos/simonw/datasette/issues/2102,1690693830,IC_kwDOBm6k_c5kxfDG,9599,simonw,2023-08-23T21:51:52Z,2023-08-23T21:52:58Z,OWNER,"This is the hook in question: https://github.com/simonw/datasette/blob/bdf59eb7db42559e538a637bacfe86d39e5d17ca/datasette/hookspecs.py#L108-L110 - `True` means they are allowed to access it. You only need a single`True` from a plugin to allow it. - `False` means they are not, and just one `False` from a plugin will deny it (even if another one returned `True` I think) - `None` means that the plugin has no opinion on this question.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1640064620,https://api.github.com/repos/simonw/datasette/issues/2102,1640064620,IC_kwDOBm6k_c5hwWZs,9599,simonw,2023-07-18T11:47:21Z,2023-07-18T11:47:21Z,OWNER,"I think I've figured out the problem here. The question being asked is ""can this actor access this resource, which is within this database within this instance"". The answer to this question needs to consider the full set of questions at once - yes they can access within this instance IF they have access to the specified table and that's the table being asked about. But the questions are currently being asked independently, which means the plugin hook acting on `view-instance` can't see that the answer here should be yes because it's actually about a table that the actor has explicit permission to view. So I think I may need to redesign the plugin hook to always see the full hierarchy of checks, not just a single check at a time. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1638567228,https://api.github.com/repos/simonw/datasette/issues/2102,1638567228,IC_kwDOBm6k_c5hqo08,9599,simonw,2023-07-17T17:24:19Z,2023-07-17T17:25:12Z,OWNER,"Confirmed that this is an issue with regular Datasette signed tokens as well. I created one on https://latest.datasette.io/-/create-token with these details: ```json { ""_r"": { ""r"": { ""fixtures"": { ""sortable"": [ ""vt"" ] } } }, ""a"": ""root"", ""d"": 3600, ""t"": 1689614483 } ``` Run like this: ``` curl -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsInQiOjE2ODk2MTQ0ODMsImQiOjM2MDAsIl9yIjp7InIiOnsiZml4dHVyZXMiOnsic29ydGFibGUiOlsidnQiXX19fX0.n-VGxxawz1Q0WK7sqLfhXUgcvY0' \ https://latest.datasette.io/fixtures/sortable.json ``` Returned an HTML Forbidden page: ```html <!DOCTYPE html> <html> <head> <title>Forbidden ... ``` Same token againts `/-/actor.json` returns: ```json { ""actor"": { ""id"": ""root"", ""token"": ""dstok"", ""_r"": { ""r"": { ""fixtures"": { ""sortable"": [ ""vt"" ] } } }, ""token_expires"": 1689618083 } } ``` Reminder - `""_r""` means restrict, `""r""` means resource.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636093730,https://api.github.com/repos/simonw/datasette/issues/2102,1636093730,IC_kwDOBm6k_c5hhM8i,9599,simonw,2023-07-14T16:26:27Z,2023-07-14T16:32:49Z,OWNER,"Here's that crucial comment: > If _r is defined then we use those to further restrict the actor. > >Crucially, we only use this to say NO (return False) - we never use it to return YES (True) because that might over-ride other restrictions placed on this actor So that's why I implemented it like this. The goal here is to be able to issue a token which can't do anything _more_ than the actor it is associated with, but CAN be configured to do less. So I think the solution here is for the `_r` checking code to perhaps implement its own view cascade logic - it notices if you have `view-table` and consequently fails to block `view-table` and `view-instance`. I'm not sure that's going to work though - would that mean that granting `view-table` grants `view-database` in a surprising and harmful way? Maybe that's OK: if you have `view-database` but permission checks fail for individual tables and queries you shouldn't be able to see a thing that you shouldn't. Need to verify that though. Also, do `Permission` instances have enough information to implement this kind of cascade without hard-coding anything? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636053060,https://api.github.com/repos/simonw/datasette/issues/2102,1636053060,IC_kwDOBm6k_c5hhDBE,9599,simonw,2023-07-14T15:51:36Z,2023-07-14T16:14:05Z,OWNER,"This might only be an issue with the code that checks `_r` on actors. https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/default_permissions.py#L185-L222 Added in https://github.com/simonw/datasette/commit/bcc781f4c50a8870e3389c4e60acb625c34b0317 - refs: - #1855 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636042066,https://api.github.com/repos/simonw/datasette/issues/2102,1636042066,IC_kwDOBm6k_c5hhAVS,9599,simonw,2023-07-14T15:41:54Z,2023-07-14T15:42:32Z,OWNER,"I tried some code spelunking and came across https://github.com/simonw/datasette/commit/d6e03b04302a0852e7133dc030eab50177c37be7 which says: > - If you have table permission but not database permission you can now view the table page Refs: - #832 Which suggests that my initial design decision wasn't what appears to be implemented today. Needs more investigation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636040164,https://api.github.com/repos/simonw/datasette/issues/2102,1636040164,IC_kwDOBm6k_c5hg_3k,9599,simonw,2023-07-14T15:40:21Z,2023-07-14T15:40:21Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/app.py#L822-L855","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2102#issuecomment-1636036312,https://api.github.com/repos/simonw/datasette/issues/2102,1636036312,IC_kwDOBm6k_c5hg-7Y,9599,simonw,2023-07-14T15:37:14Z,2023-07-14T15:37:14Z,OWNER,"I think I made this decision because I was thinking about default deny: obviously if a user has been denied access to a database. It doesn't make sense that they could access tables within it. But now that I am spending more time with authentication tokens, which default to denying everything, except for the things that you have explicitly listed, this policy, no longer makes as much sense. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1805076818,API tokens with view-table but not view-database/view-instance cannot access the table, https://github.com/simonw/datasette/issues/2101#issuecomment-1634443907,https://api.github.com/repos/simonw/datasette/issues/2101,1634443907,IC_kwDOBm6k_c5ha6KD,9599,simonw,2023-07-13T15:24:17Z,2023-07-13T15:24:17Z,OWNER,https://github.com/simonw/datasette/blob/0f7192b6154edb576c41b55bd3f2a3f53e5f436a/datasette/views/table.py#L486-L506,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1803264272,alter: true support for JSON write API, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646687219,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646687219,IC_kwDOCGYnMM5iJnPz,9599,simonw,2023-07-22T22:59:36Z,2023-07-22T22:59:36Z,OWNER,Now that we have two plugin hooks I'm closing this issue (we can open other issues for further hooks).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646686424,IC_kwDOCGYnMM5iJnDY,9599,simonw,2023-07-22T22:52:34Z,2023-07-22T22:52:34Z,OWNER,Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646656283,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646656283,IC_kwDOCGYnMM5iJfsb,25778,eyeseast,2023-07-22T19:32:24Z,2023-07-22T19:32:24Z,CONTRIBUTOR,Cool. I might try to add a geojson plugin that handles both input and output. That would help me out a lot. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646654383,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646654383,IC_kwDOCGYnMM5iJfOv,9599,simonw,2023-07-22T19:20:16Z,2023-07-22T19:20:16Z,OWNER,"Here's documentation for the new plugins mechanism, including a very short tutorial on writing a new plugin (inspired by https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html): https://sqlite-utils.datasette.io/en/latest/plugins.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643450,IC_kwDOCGYnMM5iJcj6,9599,simonw,2023-07-22T18:17:18Z,2023-07-22T18:17:18Z,OWNER,I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643379,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643379,IC_kwDOCGYnMM5iJciz,9599,simonw,2023-07-22T18:16:54Z,2023-07-22T18:16:54Z,OWNER,"> Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins? Yeah, it could do! That's not a terrible idea to be honest, those things have really been proliferating. Alternatively, they could each register themselves as plugins in addition - so if you install e.g. `pocket-to-sqlite` you could then optionally also run it as `sqlite-utils pocket-to-sqlite ...` The benefit there is for people who install `sqlite-utils` from Homebrew, where it gets its own virtual environment. They could run: ```bash brew install sqlite-utils sqlite-utils install pocket-to-sqlite sqlite-utils pocket-to-sqlite ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1642808866,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1642808866,IC_kwDOCGYnMM5h60Yi,25778,eyeseast,2023-07-19T21:54:27Z,2023-07-19T21:54:27Z,CONTRIBUTOR,Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638926655,IC_kwDOCGYnMM5hsAk_,9599,simonw,2023-07-17T21:42:37Z,2023-07-17T21:42:37Z,OWNER,"I really like this. I'm also interested in: - Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette - Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638910473,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638910473,IC_kwDOCGYnMM5hr8oJ,15178711,asg017,2023-07-17T21:27:41Z,2023-07-17T21:27:41Z,CONTRIBUTOR,"Another use-case: I want to make a `sqlite-utils` plugin that'll help me insert data into Datasette. ```bash sqlite-utils insert-datasette \ --token $DATASETTE_API_KEY \ https://latest.datasette.io/fixtures/my-table \ 'select ...' ``` This could also be a datasette plugin (ex `datasette upload-data ...`, but you can also think of `sqlite-utils` plugins that upload to S3, a postgres table, other DBMS's, etc.)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744,Plugin system, https://github.com/simonw/datasette/pull/2098#issuecomment-1668186815,https://api.github.com/repos/simonw/datasette/issues/2098,1668186815,IC_kwDOBm6k_c5jboK_,49699333,dependabot[bot],2023-08-07T16:20:18Z,2023-08-07T16:20:18Z,CONTRIBUTOR,"Looks like blacken-docs is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1796830110,Bump blacken-docs from 1.14.0 to 1.15.0, https://github.com/simonw/datasette/pull/2098#issuecomment-1646250291,https://api.github.com/repos/simonw/datasette/issues/2098,1646250291,IC_kwDOBm6k_c5iH8kz,22429695,codecov[bot],2023-07-21T21:21:00Z,2023-07-21T21:27:25Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`3a51ca9`)](https://app.codecov.io/gh/simonw/datasette/commit/3a51ca901427ac93b0ed0638fd1f82fb647a299a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39% compared to head [(`3d45262`)](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.39%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2098 +/- ## ======================================= Coverage 92.39% 92.39% ======================================= Files 39 39 Lines 5802 5802 ======================================= Hits 5361 5361 Misses 441 441 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2098?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1796830110,Bump blacken-docs from 1.14.0 to 1.15.0, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627598570,IC_kwDOCGYnMM5hAy7q,9599,simonw,2023-07-09T04:13:34Z,2023-07-09T04:13:34Z,OWNER,On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627597872,IC_kwDOCGYnMM5hAyww,9599,simonw,2023-07-09T04:09:56Z,2023-07-09T04:09:56Z,OWNER,"Thanks, looks like a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,`--no-headers` doesn't work on most formats, https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627564127,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627564127,IC_kwDODLZ_YM5hAqhf,9599,simonw,2023-07-09T01:19:42Z,2023-07-09T01:19:42Z,MEMBER,https://github.com/dogsheep/pocket-to-sqlite/tree/0.2.3 and https://pypi.org/project/pocket-to-sqlite/0.2.3/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml, https://github.com/dogsheep/pocket-to-sqlite/issues/12#issuecomment-1627563202,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/12,1627563202,IC_kwDODLZ_YM5hAqTC,9599,simonw,2023-07-09T01:14:27Z,2023-07-09T01:14:27Z,MEMBER,I tested this locally with `python -m build` and then `pip install ...whl` in a fresh virtual environment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795187493,Switch to pyproject.toml, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646659809,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646659809,IC_kwDOCGYnMM5iJgjh,9599,simonw,2023-07-22T19:53:56Z,2023-07-22T19:53:56Z,OWNER,"CLI documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#renaming-a-table - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#rename-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646658978,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646658978,IC_kwDOCGYnMM5iJgWi,9599,simonw,2023-07-22T19:48:54Z,2023-07-22T19:48:54Z,OWNER,"Python method documentation: - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.rename_table - https://sqlite-utils.datasette.io/en/latest/python-api.html#renaming-a-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657849,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646657849,IC_kwDOCGYnMM5iJgE5,9599,simonw,2023-07-22T19:42:02Z,2023-07-22T19:42:02Z,OWNER,"Manually testing new `rename-table` command: ``` $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT ); $ sqlite-utils rename-table /tmp/f.db bad-table hi Error: Table ""bad-table"" could not be renamed. no such table: bad-table $ sqlite-utils rename-table /tmp/f.db foo foo Error: Table ""foo"" could not be renamed. there is already another table or index with this name: foo $ sqlite-utils rename-table /tmp/f.db foo bar $ sqlite-utils schema /tmp/f.db CREATE TABLE ""bar"" ( [id] INTEGER, [name] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646657324,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646657324,IC_kwDOCGYnMM5iJf8s,9599,simonw,2023-07-22T19:39:06Z,2023-07-22T19:39:06Z,OWNER,"> Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one. I think `keep_table=""name_of_table""` is good for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1646656246,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1646656246,IC_kwDOCGYnMM5iJfr2,9599,simonw,2023-07-22T19:32:07Z,2023-07-22T19:32:07Z,OWNER,"Trying out a simple first implementation: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db[""foo""].insert({""id"": 1}) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [foo] ( [id] INTEGER ); None select name from sqlite_master where type = 'view' None INSERT INTO [foo] ([id]) VALUES (?); [1] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([foo]) None
>>> db.rename_table(""foo"", ""baz"") ALTER TABLE [foo] RENAME TO [baz] None >>> print(db.schema) select sql from sqlite_master where sql is not null None CREATE TABLE ""baz"" ( [id] INTEGER ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/565#issuecomment-1618380888,https://api.github.com/repos/simonw/sqlite-utils/issues/565,1618380888,IC_kwDOCGYnMM5gdohY,9599,simonw,2023-07-03T14:09:11Z,2023-07-03T14:09:31Z,OWNER,"For the CLI: ```bash sqlite-utils rename-table data.db old_table_name new_table_name ``` For the Python code, should it go on Table or on Database? ```python db[""foo""].rename_table(""bar"") db.rename_table(""foo"", ""bar"") ``` I think I like the second better, it's slightly more clear. Also need a design for an option for the `.transform()` method to indicate that the new table should be created with a new name without dropping the old one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1786258502,Table renaming: db.rename_table() and sqlite-utils rename-table, https://github.com/simonw/sqlite-utils/issues/563#issuecomment-1617395444,https://api.github.com/repos/simonw/sqlite-utils/issues/563,1617395444,IC_kwDOCGYnMM5gZ370,9599,simonw,2023-07-03T05:44:43Z,2023-07-03T05:44:43Z,OWNER,Documentation at the bottom of this section: https://sqlite-utils.datasette.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1785360409,`--empty-null` option when importing CSV, https://github.com/simonw/sqlite-utils/issues/562#issuecomment-1616782404,https://api.github.com/repos/simonw/sqlite-utils/issues/562,1616782404,IC_kwDOCGYnMM5gXiRE,9599,simonw,2023-07-02T19:24:14Z,2023-07-02T19:26:39Z,OWNER,"[Dataclasses](https://docs.python.org/3/library/dataclasses.html) were added in Python 3.7 and `sqlite-utils` was originally written for Python 3.6 - but both 3.6 and 3.7 are EOL now. The thing that makes Dataclasses particularly interesting is the potential to use type annotations with them to help specify the types of the related SQLite columns. Example for https://datasette.io/content/users ```sql CREATE TABLE [users] ( [login] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [avatar_url] TEXT, [gravatar_id] TEXT, [html_url] TEXT, [type] TEXT, [site_admin] INTEGER, [name] TEXT ); ``` And the dataclass: ```python from dataclasses import dataclass @dataclass class User: id: int login: str node_id: str avatar_url: str gravatar_id: str html_url: str type: str site_admin: int name: str ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1784794489,Explore the intersection between sqlite-utils and dataclasses, https://github.com/simonw/datasette/issues/2093#issuecomment-1688532012,https://api.github.com/repos/simonw/datasette/issues/2093,1688532012,IC_kwDOBm6k_c5kpPQs,15178711,asg017,2023-08-22T16:21:40Z,2023-08-22T16:21:40Z,CONTRIBUTOR,"OK Here's the gameplan for this, which is closely tied to #2143 : - We will add a new `datasette.json`/`datasette.yaml` configuration file to datasette, which combines settings/plugin config/permissions/canned queries into a new file format - Metadata will NOT be a part of this file - TOML support is not planned, but maybe we can create a separate issue for support TOML with JSON/YAML - The `settings.json` file will be deprecated, and the `--config` arg will be brought back. - Command line arguments can still be used to overwrite values (ex `--setting` will overwrite settings in `datasette.yaml` The format of `datasette.json` will follow what Simon listed here: https://github.com/simonw/datasette/issues/2143#issuecomment-1684484426 Here's the current implementation plan: 1. Add a new `--config` flag and port over `""settings""` into a new datasette.json config file, remove settings.json 2. Add top-level plugin config support to `datasette.json` 3. Figure out database/table structure of config `datasette.json` 4. Port over database/table level plugin config support `datasette.json` 5. Port over permissions/auth settings to `datasette.json` 6. Deprecate non-metadata values in `metadata.json`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1616286848,https://api.github.com/repos/simonw/datasette/issues/2093,1616286848,IC_kwDOBm6k_c5gVpSA,15178711,asg017,2023-07-02T02:17:46Z,2023-07-02T02:17:46Z,CONTRIBUTOR,"Storing metadata in the database won't be required. I imagine there'll be many different ways to store metadata, including any possible `datasette_metadata` or sqlite-docs, or the older metadata.json way. The next question will be how precedence should work - i'd imagine metadata.json > plugins > datasette_metadata > sqlite-docs","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1616195496,https://api.github.com/repos/simonw/datasette/issues/2093,1616195496,IC_kwDOBm6k_c5gVS-o,273509,terinjokes,2023-07-02T00:06:54Z,2023-07-02T00:07:17Z,NONE,"I'm not keen on requiring metadata to be within the database. I commonly have multiple DBs, from various sources, and having one config file to provide the metadata works out very well. I use Datasette with databases where I'm not the original source, needing to mutate them to add a metadata table or sqlite-docs makes me uncomfortable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1614652001,https://api.github.com/repos/simonw/datasette/issues/2093,1614652001,IC_kwDOBm6k_c5gPaJh,9599,simonw,2023-06-30T13:27:13Z,2023-06-30T13:27:13Z,OWNER,"I agree, settings in the DB doesn't make sense but metadata does. On the JSON v YAML v TOML issue I just spotted Caddy has a concept of config adapters which they use to resolve exactly that problem: https://caddyserver.com/docs/config-adapters","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1613896210,https://api.github.com/repos/simonw/datasette/issues/2093,1613896210,IC_kwDOBm6k_c5gMhoS,15178711,asg017,2023-06-29T22:53:33Z,2023-06-29T22:53:33Z,CONTRIBUTOR,"Maybe we can have a separate issue for revamping `metadata.json`? A `datasette_metadata` table or the `sqlite-docs` extension seem like two reasonable additions that we can work through. Storing metadata inside a SQLite database makes sense, but I don't think storing `datasette.*` style config (ex ports, settings, etc.) inside a SQLite DB makes sense, since it's very environment-dependent","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1613895188,https://api.github.com/repos/simonw/datasette/issues/2093,1613895188,IC_kwDOBm6k_c5gMhYU,15178711,asg017,2023-06-29T22:51:53Z,2023-06-29T22:51:53Z,CONTRIBUTOR,"I agree with not liking `metadata.json` stuff in a `datasette.*` config file. Editing description of a table/column in a file like `datasette.*` seems odd to me. Though since plugin configuration currently lives in `metadata.json`, I think it should be removed from there and placed in `datasette.*`, at least for top-level config like `datasette-auth-github`'s config. Keeping `metadata.json` strictly for documentation/licensing/column units makes sense to me, but anything plugin related should be in some config file, like `datasette.*`. And ya, supporting both `datasette.*` and CLI flags makes a lot of sense to me. Any `--setting` flag should override anything in `datasette.*` for easier debugging, with possibly a warning message so people don't get confused. Same with `--port` and a port defined in `datasette.*`","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1613889979,https://api.github.com/repos/simonw/datasette/issues/2093,1613889979,IC_kwDOBm6k_c5gMgG7,9599,simonw,2023-06-29T22:44:08Z,2023-06-30T13:25:39Z,OWNER,"I do like also being able to set options using command line options though - for things like SQL time limits I'd much rather be able to throw on `--setting sql_time_limit_ms 10000` than have to save a config file to disk. So I'd want to support both. Which maybe means also having a way to set plugin options with CLI options. `datasette publish` kind of has that ability already: ``` datasette publish heroku my_database.db \ --name my-heroku-app-demo \ --install=datasette-auth-github \ --plugin-secret datasette-auth-github client_id your_client_id \ --plugin-secret datasette-auth-github client_secret your_client_secret ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2093#issuecomment-1613887492,https://api.github.com/repos/simonw/datasette/issues/2093,1613887492,IC_kwDOBm6k_c5gMfgE,9599,simonw,2023-06-29T22:40:25Z,2023-06-29T22:40:25Z,OWNER,"I'm strongly in favour of combining settings, configuration and plugin configuration. I'm not keen on mixing in metadata as well - that feels like a different concept to me, and I'm unhappy with how that's already had things like plugin settings leak into it. I'm not yet sold on TOML - I actually find it less intuitive than YAML, surprisingly. They all have their warts I guess. Datasette already has the ability to consume JSON or YAML for metadata - maybe it could grow TOML support too? That way users could have a `datasette.json` or `datasette.yaml` or `datasette.toml` file depending on their preference. In terms of metadata: since that's means to be driven by a plugin hook anyway, maybe one of the potential sources of metadata is a `metadata` nested object in that `datasette.*` configuration file. Or you can have it in a separate `metadata.json` or bundled into the SQLite database or some other plugin-driven mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781530343,"Proposal: Combine settings, metadata, static, etc. into a single `datasette.yaml` File", https://github.com/simonw/datasette/issues/2092#issuecomment-1613381990,https://api.github.com/repos/simonw/datasette/issues/2092,1613381990,IC_kwDOBm6k_c5gKkFm,9599,simonw,2023-06-29T15:23:04Z,2023-06-29T15:26:28Z,OWNER,"Felt lazy: ```bash symbex test_homepage -f tests/test_api.py | \ llm -m 4 --system 'Change all of the == some integer tests in this code to isinstance(thing, int) instead' ``` Output: ```python # File: tests/test_api.py Line: 26 @pytest.mark.asyncio async def test_homepage(ds_client): response = await ds_client.get(""/.json"") assert response.status_code == 200 assert ""application/json; charset=utf-8"" == response.headers[""content-type""] data = response.json() assert isinstance(data.keys(), int) d = data[""fixtures""] assert d[""name""] == ""fixtures"" assert isinstance(d[""tables_count""], int) assert isinstance(len(d[""tables_and_views_truncated""]), int) assert d[""tables_and_views_more""] is True # 4 hidden FTS tables + no_primary_key (hidden in metadata) assert isinstance(d[""hidden_tables_count""], int) # 201 in no_primary_key, plus 6 in other hidden tables: assert isinstance(d[""hidden_table_rows_sum""], int), data assert isinstance(d[""views_count""], int) ``` I'll use most of that and delete the obsoleted comments.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781047747,test_homepage intermittent failure, https://github.com/simonw/datasette/issues/2092#issuecomment-1613375407,https://api.github.com/repos/simonw/datasette/issues/2092,1613375407,IC_kwDOBm6k_c5gKiev,9599,simonw,2023-06-29T15:20:52Z,2023-06-29T15:21:05Z,OWNER,I'm going to remove this assertion entirely. The homepage JSON needs a refactor anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781047747,test_homepage intermittent failure, https://github.com/simonw/datasette/issues/2091#issuecomment-1690423878,https://api.github.com/repos/simonw/datasette/issues/2091,1690423878,IC_kwDOBm6k_c5kwdJG,9599,simonw,2023-08-23T18:18:18Z,2023-08-23T18:18:18Z,OWNER,"Dupe of: - #2097 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7, https://github.com/simonw/datasette/issues/2091#issuecomment-1613369355,https://api.github.com/repos/simonw/datasette/issues/2091,1613369355,IC_kwDOBm6k_c5gKhAL,9599,simonw,2023-06-29T15:18:34Z,2023-06-29T15:18:34Z,OWNER,Posted on the Glitch feedback forum about this here: https://support.glitch.com/t/upgrade-python-version-from-3-7-which-is-now-eol-to-something-more-recent/63011,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7, https://github.com/simonw/datasette/issues/2091#issuecomment-1613360413,https://api.github.com/repos/simonw/datasette/issues/2091,1613360413,IC_kwDOBm6k_c5gKe0d,9599,simonw,2023-06-29T15:13:04Z,2023-06-29T15:13:04Z,OWNER,"One problem: https://glitch.com/ still provides 3.7: ``` $ python3 --version Python 3.7.10 ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781022369,Drop support for Python 3.7, https://github.com/simonw/datasette/issues/2090#issuecomment-1613346412,https://api.github.com/repos/simonw/datasette/issues/2090,1613346412,IC_kwDOBm6k_c5gKbZs,9599,simonw,2023-06-29T15:05:04Z,2023-06-29T15:05:04Z,OWNER,"Decided to fix just those ""Ambiguous variable name"" ones: ```bash ruff check . | grep E741 ``` Then iterated through and fixed them all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781005740,Adopt ruff for linting, https://github.com/simonw/datasette/issues/2090#issuecomment-1613339404,https://api.github.com/repos/simonw/datasette/issues/2090,1613339404,IC_kwDOBm6k_c5gKZsM,9599,simonw,2023-06-29T15:01:01Z,2023-06-29T15:01:20Z,OWNER,"I tried it just now and got some interesting results. I dropped in a `ruff.toml` file: ```toml line-length = 160 ``` Because the default line length limit of 88 was causing a lot of noisy errors. Then run: ```bash pip install ruff ruff check . ``` Plenty of warnings about unused imports - running `ruff check . --fix` fixed those automatically, but I think I still need to manually review them as some might be imports which are deliberate and should be in `__all__` to ensure they are visible from that module as well. Some lines in tests are longer than even 160 chars, e.g.: https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L673-L681 These can have ` # noqa: E501` added to the end of those lines to skip the check for them. That got it down to: ``` % ruff check . datasette/views/table.py:23:5: F811 Redefinition of unused `format_bytes` from line 19 run_tests.py:2:5: E401 Multiple imports on one line tests/test_api.py:591:40: F811 Redefinition of unused `app_client_no_files` from line 7 tests/test_api.py:629:35: F811 Redefinition of unused `app_client_no_files` from line 7 tests/test_api.py:635:54: F811 Redefinition of unused `app_client_with_dot` from line 8 tests/test_api.py:661:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 9 tests/test_api.py:759:25: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10 tests/test_api.py:892:28: F811 Redefinition of unused `app_client_larger_cache_size` from line 11 tests/test_api.py:928:5: F811 Redefinition of unused `app_client_with_cors` from line 12 tests/test_api.py:929:5: F811 Redefinition of unused `app_client_two_attached_databases_one_immutable` from line 10 tests/test_api.py:969:38: F811 Redefinition of unused `app_client_two_attached_databases` from line 13 tests/test_api.py:976:39: F811 Redefinition of unused `app_client_conflicting_database_names` from line 14 tests/test_api.py:987:38: F811 Redefinition of unused `app_client_immutable_and_inspect_file` from line 15 tests/test_api.py:1002:24: F811 Redefinition of unused `app_client` from line 6 tests/test_csv.py:67:33: F811 Redefinition of unused `app_client_with_cors` from line 6 tests/test_csv.py:157:21: F811 Redefinition of unused `app_client_csv_max_mb_one` from line 5 tests/test_csv.py:198:20: F811 Redefinition of unused `app_client_with_trace` from line 7 tests/test_csv.py:209:53: F811 Redefinition of unused `app_client_with_trace` from line 7 tests/test_csv.py:215:53: F811 Redefinition of unused `app_client_with_trace` from line 7 tests/test_filters.py:102:11: F811 Redefinition of unused `test_through_filters_from_request` from line 81 tests/test_html.py:19:19: F811 Redefinition of unused `app_client_two_attached_databases` from line 7 tests/test_html.py:175:25: F811 Redefinition of unused `app_client_shorter_time_limit` from line 6 tests/test_html.py:469:51: F811 Redefinition of unused `app_client` from line 4 tests/test_html.py:797:26: F811 Redefinition of unused `app_client_base_url_prefix` from line 5 tests/test_html.py:840:44: F811 Redefinition of unused `app_client_base_url_prefix` from line 5 tests/test_html.py:850:51: F811 Redefinition of unused `app_client_base_url_prefix` from line 5 tests/test_pagination.py:50:43: F821 Undefined name `parse_next` tests/test_pagination.py:82:7: F811 Redefinition of unused `KeysetPaginator` from line 36 tests/test_plugins.py:115:15: E741 Ambiguous variable name: `l` tests/test_plugins.py:482:161: E501 Line too long (170 > 160 characters) tests/test_plugins.py:543:29: E741 Ambiguous variable name: `l` tests/test_plugins.py:563:161: E501 Line too long (170 > 160 characters) tests/test_plugins.py:940:62: E741 Ambiguous variable name: `l` tests/test_table_api.py:739:5: F811 Redefinition of unused `app_client_returned_rows_matches_page_size` from line 6 tests/test_table_api.py:1066:45: F811 Redefinition of unused `app_client_with_trace` from line 5 tests/test_table_html.py:484:29: E741 Ambiguous variable name: `l` tests/test_table_html.py:524:29: E741 Ambiguous variable name: `l` tests/test_table_html.py:675:161: E501 Line too long (165 > 160 characters) tests/test_table_html.py:897:161: E501 Line too long (164 > 160 characters) tests/test_table_html.py:902:161: E501 Line too long (164 > 160 characters) tests/test_utils.py:141:161: E501 Line too long (176 > 160 characters) Found 41 errors. ``` Those ""Redefinition of unused `app_client_two_attached_databases`"" lines are caused because of the fixtures pattern I'm using here: https://github.com/simonw/datasette/blob/99ba05118891db9dc30f1dca22ad6709775560de/tests/test_html.py#L3-L20 I could fix that by getting rid of `fixtures.py` and moving those into `conftest.py`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1781005740,Adopt ruff for linting, https://github.com/simonw/datasette/issues/2089#issuecomment-1613316722,https://api.github.com/repos/simonw/datasette/issues/2089,1613316722,IC_kwDOBm6k_c5gKUJy,9599,simonw,2023-06-29T14:48:10Z,2023-06-29T14:48:10Z,OWNER,Spell check is passing now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613315851,https://api.github.com/repos/simonw/datasette/issues/2089,1613315851,IC_kwDOBm6k_c5gKT8L,9599,simonw,2023-06-29T14:47:38Z,2023-06-29T14:47:38Z,OWNER,"Confirmed, this was a 2.2.5 change: https://github.com/codespell-project/codespell/releases/tag/v2.2.5 > - Add displaing->displaying by [@peternewman](https://github.com/peternewman) in [#2808](https://github.com/codespell-project/codespell/pull/2808)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613307716,https://api.github.com/repos/simonw/datasette/issues/2089,1613307716,IC_kwDOBm6k_c5gKR9E,9599,simonw,2023-06-29T14:42:23Z,2023-06-29T14:42:23Z,OWNER,"Yes, upgrading locally got me the correct version and the test failure: ``` % pip install -U codespell Requirement already satisfied: codespell in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.10/site-packages (2.2.2) Collecting codespell Downloading codespell-2.2.5-py3-none-any.whl (242 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 4.9 MB/s eta 0:00:00 Installing collected packages: codespell Attempting uninstall: codespell Found existing installation: codespell 2.2.2 Uninstalling codespell-2.2.2: Successfully uninstalled codespell-2.2.2 Successfully installed codespell-2.2.5 % codespell docs/metadata.rst docs/metadata.rst:192: displaing ==> displaying ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613306787,https://api.github.com/repos/simonw/datasette/issues/2089,1613306787,IC_kwDOBm6k_c5gKRuj,9599,simonw,2023-06-29T14:41:47Z,2023-06-29T14:41:47Z,OWNER,"Looks like in CI it's running 2.2.5: ``` Collecting codespell (from datasette==1.0a2) Downloading codespell-2.2.5-py3-none-any.whl (242 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.7/242.7 kB 31.1 MB/s eta 0:00:00 ``` But on my laptop it's 2.2.2: ``` % codespell --version 2.2.2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/datasette/issues/2089#issuecomment-1613305070,https://api.github.com/repos/simonw/datasette/issues/2089,1613305070,IC_kwDOBm6k_c5gKRTu,9599,simonw,2023-06-29T14:40:44Z,2023-06-29T14:40:44Z,OWNER,"I'm not sure why I can't duplicate this failure in my local development environment: ``` % codespell docs/metadata.rst ``` It finds no errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1780973290,codespell test failure, https://github.com/simonw/sqlite-utils/issues/561#issuecomment-1610040517,https://api.github.com/repos/simonw/sqlite-utils/issues/561,1610040517,IC_kwDOCGYnMM5f90TF,9599,simonw,2023-06-27T18:44:31Z,2023-06-27T18:44:38Z,OWNER,"Got this working: ```bash sqlite-utils insert /tmp/playground.db Playground_Submission_Data \ ~/Downloads/Playground_Submission_Data.csv --csv --stop-after 2000 ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1777548699,`--stop-after` option for `insert` and `upsert` commands, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606315321,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606315321,IC_kwDOCGYnMM5fvm05,9599,simonw,2023-06-25T23:18:33Z,2023-06-25T23:18:33Z,OWNER,Documentation preview: https://sqlite-utils--560.org.readthedocs.build/en/560/installation.html#alternatives-to-sqlite3,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606310630,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606310630,IC_kwDOCGYnMM5fvlrm,9599,simonw,2023-06-25T23:06:07Z,2023-06-25T23:06:07Z,OWNER,"Filed an issue about the above with `pysqlite3` (which `sqlean.py` is based on) here: - https://github.com/coleifer/pysqlite3/issues/58","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606297356,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606297356,IC_kwDOCGYnMM5fvicM,9599,simonw,2023-06-25T22:42:41Z,2023-06-25T22:42:41Z,OWNER,"Yes that does seem to do the trick: ```pycon >>> import sqlean >>> db = sqlean.connect(""/tmp/4.db"") >>> db.execute('PRAGMA journal_mode;').fetchall() [('delete',)] >>> db.isolation_level '' >>> db.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction >>> db.isolation_level = None >>> db.isolation_level >>> db.execute('PRAGMA journal_mode=wal;') ``` Weird how `isolation_level` of empty string causes the error, but setting that to `None` fixes the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606294627,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606294627,IC_kwDOCGYnMM5fvhxj,9599,simonw,2023-06-25T22:40:10Z,2023-06-25T22:40:10Z,OWNER,I suspect this has something to do with `autocommit` mode in `sqlite3` - which I may be able to turn off by setting `con.isolation_level = None`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606293382,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606293382,IC_kwDOCGYnMM5fvheG,9599,simonw,2023-06-25T22:34:47Z,2023-06-25T22:34:47Z,OWNER,"```pycon >>> import sqlite3 >>> db = sqlite3.connect(""/tmp/1.db"") >>> db.execute('PRAGMA journal_mode=wal;') >>> import sqlean >>> db2 = sqlean.connect(""/tmp/2.db"") >>> db2.execute('PRAGMA journal_mode=wal;') Traceback (most recent call last): File """", line 1, in sqlean.dbapi2.OperationalError: cannot change into wal mode from within a transaction ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606290917,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606290917,IC_kwDOCGYnMM5fvg3l,9599,simonw,2023-06-25T22:32:28Z,2023-06-25T22:32:28Z,OWNER,"I've fixed most of the test failures, but I still need to fix this one: > cannot change into wal mode from within a transaction","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606273005,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606273005,IC_kwDOCGYnMM5fvcft,9599,simonw,2023-06-25T21:47:47Z,2023-06-25T21:47:47Z,OWNER,I can use https://github.com/simonw/sqlite-dump as an optional dependency to handle the missing `.iterdump()` method.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270887,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270887,IC_kwDOCGYnMM5fvb-n,9599,simonw,2023-06-25T21:37:12Z,2023-06-26T08:21:00Z,OWNER,"On my own laptop I got a crash running the tests - details here: - https://github.com/nalgeon/sqlean.py/issues/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606270055,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606270055,IC_kwDOCGYnMM5fvbxn,9599,simonw,2023-06-25T21:31:56Z,2023-06-25T21:31:56Z,OWNER,"Lots of failures now that I'm trying to run the tests against `sqlean.py` on macOS and Python 3.10: https://github.com/simonw/sqlite-utils/actions/runs/5371800108/jobs/9744802953 A bunch of these, because `pysqlite3` chooses not to implement `.iterdump()`: ``` @pytest.fixture def db_to_analyze_path(db_to_analyze, tmpdir): path = str(tmpdir / ""test.db"") db = sqlite3.connect(path) > db.executescript(""\n"".join(db_to_analyze.conn.iterdump())) E AttributeError: 'sqlean.dbapi2.Connection' object has no attribute 'iterdump' ``` Also some of these: ``` def test_analyze_whole_database(db): assert set(db.table_names()) == {""one_index"", ""two_indexes""} db.analyze() > assert set(db.table_names()) == {""one_index"", ""two_indexes"", ""sqlite_stat1""} E AssertionError: assert {'one_index',...'two_indexes'} == {'one_index',...'two_indexes'} E Extra items in the left set: E 'sqlite_stat4' E Full diff: E - {'two_indexes', 'sqlite_stat1', 'one_index'} E + {'two_indexes', 'sqlite_stat1', 'sqlite_stat4', 'one_index'} E ? ++++++++++++++++ ``` Apparently `sqlean.py` adds a `sqlite_stat4` table that the tests are not expecting. Plus some errors that look like this: ``` def test_enable_wal(): runner = CliRunner() dbs = [""test.db"", ""test2.db""] with runner.isolated_filesystem(): for dbname in dbs: db = Database(dbname) db[""t""].create({""pk"": int}, pk=""pk"") assert db.journal_mode == ""delete"" result = runner.invoke(cli.cli, [""enable-wal""] + dbs) > assert 0 == result.exit_code E AssertionError: assert 0 == 1 E + where 1 = .exit_code ``` Test summary: ``` ============ 13 failed, 909 passed, 16 skipped, 2 errors in 19.29s ============= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/sqlite-utils/pull/560#issuecomment-1606237836,https://api.github.com/repos/simonw/sqlite-utils/issues/560,1606237836,IC_kwDOCGYnMM5fvT6M,22429695,codecov[bot],2023-06-25T19:49:45Z,2023-06-26T08:20:59Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`80.55`**% and project coverage change: **`-0.15`** :warning: > Comparison is base [(`2747257`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/2747257a3334d55e890b40ec58fada57ae8cfbfd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`5e7d27e`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.22%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #560 +/- ## ========================================== - Coverage 96.36% 96.22% -0.15% ========================================== Files 6 6 Lines 2726 2752 +26 ========================================== + Hits 2627 2648 +21 - Misses 99 104 +5 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/utils.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `94.56% <63.63%> (-0.62%)` | :arrow_down: | | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <86.36%> (-0.20%)` | :arrow_down: | | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <100.00%> (ø)` | |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/560?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1773458985,Use sqlean if available in environment, https://github.com/simonw/datasette/issues/2087#issuecomment-1636134091,https://api.github.com/repos/simonw/datasette/issues/2087,1636134091,IC_kwDOBm6k_c5hhWzL,653549,adarshp,2023-07-14T17:02:03Z,2023-07-14T17:02:03Z,NONE,"@asg017 - the docs say that the autodetection only occurs in configuration directory mode. I for one would also be interested in the `--settings settings.json` feature. For context, I am developing a large database for use with Datasette, but the database lives in a different network volume than my source code, since the volume in which my source code lives is aggressively backed up, while the location where the database lives is meant for temporary files and is not as aggressively backed up (since the backups would get unreasonably large).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1765870617,`--settings settings.json` option, https://github.com/simonw/datasette/issues/2087#issuecomment-1616853644,https://api.github.com/repos/simonw/datasette/issues/2087,1616853644,IC_kwDOBm6k_c5gXzqM,15178711,asg017,2023-07-02T22:00:48Z,2023-07-02T22:00:48Z,CONTRIBUTOR,"I just saw in the docs that Dasette auto-detects `settings.json`: > settings.json - settings that would normally be passed using --setting - here they should be stored as a JSON object of key/value pairs > [*Source*](https://docs.datasette.io/en/stable/settings.html#:~:text=settings.json%20%2D%20settings%20that%20would%20normally%20be%20passed%20using%20%2D%2Dsetting%20%2D%20here%20they%20should%20be%20stored%20as%20a%20JSON%20object%20of%20key/value%20pairs)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1765870617,`--settings settings.json` option, https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1590531892,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1590531892,IC_kwDOCGYnMM5ezZc0,7908073,chapmanjacobd,2023-06-14T06:09:21Z,2023-06-14T06:09:21Z,CONTRIBUTOR,"I put together a [simple script](https://github.com/chapmanjacobd/library/blob/42129c5ebe15f9d74653c0f5ca4ed0c991d383e0/xklb/scripts/dedupe_db.py) to upsert and remove duplicate rows based on business keys. If anyone has similar problems with above this might help ``` CREATE TABLE my_table ( id INTEGER PRIMARY KEY, column1 TEXT, column2 TEXT, column3 TEXT ); INSERT INTO my_table (column1, column2, column3) VALUES ('Value 1', 'Duplicate 1', 'Duplicate A'), ('Value 2', 'Duplicate 2', 'Duplicate B'), ('Value 3', 'Duplicate 2', 'Duplicate C'), ('Value 4', 'Duplicate 3', 'Duplicate D'), ('Value 5', 'Duplicate 3', 'Duplicate E'), ('Value 6', 'Duplicate 3', 'Duplicate F'); ``` ``` library dedupe-db test.db my_table --bk column2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands, https://github.com/simonw/sqlite-utils/issues/557#issuecomment-1577355134,https://api.github.com/repos/simonw/sqlite-utils/issues/557,1577355134,IC_kwDOCGYnMM5eBId-,7908073,chapmanjacobd,2023-06-05T19:26:26Z,2023-06-05T19:26:26Z,CONTRIBUTOR,"this isn't really actionable... I'm just being a whiny baby. I have tasted the milk of being able to use `upsert_all`, `insert_all`, etc without having to write DDL to create tables. The meat of the issue is that SQLITE doesn't make rowid stable between vacuums so it is not possible to take shortcuts","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740150327,Aliased ROWID option for tables created from alter=True commands, https://github.com/simonw/sqlite-utils/issues/556#issuecomment-1575310378,https://api.github.com/repos/simonw/sqlite-utils/issues/556,1575310378,IC_kwDOCGYnMM5d5VQq,601708,mcint,2023-06-04T01:21:15Z,2023-06-04T01:21:15Z,CONTRIBUTOR,"I've resolved my use, with the line-buffered output and while read loop for line buffered input, but I leave this here so the incremental saving or line-buffered use-case can be explicitly handled or rejected (or deferred).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1740026046,Support storing incrementally piped values, https://github.com/simonw/sqlite-utils/issues/555#issuecomment-1592047502,https://api.github.com/repos/simonw/sqlite-utils/issues/555,1592047502,IC_kwDOCGYnMM5e5LeO,7908073,chapmanjacobd,2023-06-14T22:00:10Z,2023-06-14T22:01:57Z,CONTRIBUTOR,"You may want to try doing a performance comparison between this and just selecting all the ids with few constraints and then doing the filtering within python. That might seem like a lazy-programmer, inefficient way but queries with large resultsets are a different profile than what databases like SQLITE are designed for. That is not to say that SQLITE is slow or that python is always faster but when you start reading >20% of an index there is an equilibrium that is reached. Especially when adding in writing extra temp tables and stuff to memory/disk. And especially given the `NOT IN` style of query... You may also try chunking like this: ```py def chunks(lst, n) -> Generator: for i in range(0, len(lst), n): yield lst[i : i + n] SQLITE_PARAM_LIMIT = 32765 data = [] chunked = chunks(video_ids, consts.SQLITE_PARAM_LIMIT) for ids in chunked: data.expand( list( db.query( f""""""SELECT * from videos WHERE id in ("""""" + "","".join([""?""] * len(ids)) + "")"", (*ids,), ) ) ) ``` but that actually won't work with your `NOT IN` requirements. You need to query the full resultset to check any row. Since you are doing stuff with files/videos in SQLITE you might be interested in my side project: https://github.com/chapmanjacobd/library","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1733198948,Filter table by a large bunch of ids, https://github.com/simonw/datasette/pull/2080#issuecomment-1563650990,https://api.github.com/repos/simonw/datasette/issues/2080,1563650990,IC_kwDOBm6k_c5dM2uu,9599,simonw,2023-05-26T00:08:59Z,2023-05-26T00:08:59Z,OWNER,"I'm not going to document this yet, I want to let it bake for a bit longer first.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class, https://github.com/simonw/datasette/pull/2080#issuecomment-1563629348,https://api.github.com/repos/simonw/datasette/issues/2080,1563629348,IC_kwDOBm6k_c5dMxck,22429695,codecov[bot],2023-05-25T23:31:10Z,2023-05-26T00:07:34Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`95.45`**% and project coverage change: **`+0.01`** :tada: > Comparison is base [(`b49fa44`)](https://app.codecov.io/gh/simonw/datasette/commit/b49fa446d683ddcaf6faf2944dacc0d866bf2d70?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`df5fd73`)](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41%. > :exclamation: Current head df5fd73 differs from pull request most recent head e990fbc. Consider uploading reports for the commit e990fbc to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2080 +/- ## ========================================== + Coverage 92.40% 92.41% +0.01% ========================================== Files 39 39 Lines 5768 5790 +22 ========================================== + Hits 5330 5351 +21 - Misses 438 439 +1 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/views/base.py](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `92.97% <95.45%> (+0.18%)` | :arrow_up: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2080?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class, https://github.com/simonw/datasette/pull/2080#issuecomment-1563626231,https://api.github.com/repos/simonw/datasette/issues/2080,1563626231,IC_kwDOBm6k_c5dMwr3,9599,simonw,2023-05-25T23:25:17Z,2023-05-25T23:25:17Z,OWNER,I'm going to try using this for the `/-/patterns` page.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726603778,New View base class, https://github.com/simonw/datasette/issues/2079#issuecomment-1563607291,https://api.github.com/repos/simonw/datasette/issues/2079,1563607291,IC_kwDOBm6k_c5dMsD7,9599,simonw,2023-05-25T22:56:28Z,2023-05-25T22:56:28Z,OWNER,Wrote this up as a TIL: https://til.simonwillison.net/http/testing-cors-max-age,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563597589,https://api.github.com/repos/simonw/datasette/issues/2079,1563597589,IC_kwDOBm6k_c5dMpsV,9599,simonw,2023-05-25T22:42:07Z,2023-05-25T22:42:07Z,OWNER,"Mystery solved as to why I wasn't seeing this work: I had ""Disable Cache"" checked! I ran this experiment after un-checking that box: ```javascript fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // And run it again fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // Now try a thing that doesn't serve that max-age header yet: fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // And a second time but within 5s fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // Third time after waiting longer than 5s fetch('https://latest-with-plugins.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); // Try that original one again - still within the 1hr cache time fetch('https://latest.datasette.io/ephemeral/foo/1/-/update', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ test: 'test' }) }); ``` The results show that the cache of 1hr was being obeyed for `latest.datasette.io` while the `latest-with-plugins.datasette.io` default cache of 5s was being obeyed too. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563588199,https://api.github.com/repos/simonw/datasette/issues/2079,1563588199,IC_kwDOBm6k_c5dMnZn,9599,simonw,2023-05-25T22:29:47Z,2023-05-25T22:30:12Z,OWNER,"https://fetch.spec.whatwg.org/#http-access-control-max-age says: > Indicates the number of seconds (5 by default) the information provided by the [Access-Control-Allow-Methods](https://fetch.spec.whatwg.org/#http-access-control-allow-methods) and [Access-Control-Allow-Headers](https://fetch.spec.whatwg.org/#http-access-control-allow-headers) [headers](https://fetch.spec.whatwg.org/#concept-header) can be cached. So there was already a 5s cache anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563587230,https://api.github.com/repos/simonw/datasette/issues/2079,1563587230,IC_kwDOBm6k_c5dMnKe,9599,simonw,2023-05-25T22:28:20Z,2023-05-25T22:28:20Z,OWNER,"Weird... after the deploy went out: But the request did indeed get the new header: So I'm not sure why it's making multiple `POST` requests like that. Maybe it's because the attempted `POST` failed with a 404?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563565407,https://api.github.com/repos/simonw/datasette/issues/2079,1563565407,IC_kwDOBm6k_c5dMh1f,9599,simonw,2023-05-25T22:09:53Z,2023-05-25T22:09:53Z,OWNER,Updated docs: https://docs.datasette.io/en/latest/json_api.html#enabling-cors,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563563438,https://api.github.com/repos/simonw/datasette/issues/2079,1563563438,IC_kwDOBm6k_c5dMhWu,9599,simonw,2023-05-25T22:08:28Z,2023-05-25T22:08:28Z,OWNER,"I ran this on https://www.example.com/ twice using the console: ```javascript fetch( `https://latest.datasette.io/ephemeral/foo/1/-/update`, { method: ""POST"", mode: ""cors"", headers: { Authorization: `Bearer tok`, ""Content-Type"": ""application/json"", }, body: JSON.stringify({update: {blah: 1}}), } ) .then((r) => r.json()) .then((data) => { console.log(data); }); ``` And got this in the network pane: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563558915,https://api.github.com/repos/simonw/datasette/issues/2079,1563558915,IC_kwDOBm6k_c5dMgQD,9599,simonw,2023-05-25T22:04:41Z,2023-05-25T22:04:41Z,OWNER,I'm going with 3600 for 1 hour instead of 600 for 10 minutes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2079#issuecomment-1563547097,https://api.github.com/repos/simonw/datasette/issues/2079,1563547097,IC_kwDOBm6k_c5dMdXZ,9599,simonw,2023-05-25T21:51:38Z,2023-05-25T21:51:38Z,OWNER,"Also need to update this documentation: https://github.com/simonw/datasette/blob/9584879534ff0556e04e4c420262972884cac87b/docs/json_api.rst?plain=1#L453-L465 Or maybe make that automated via `cog`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726531350,Datasette should serve Access-Control-Max-Age, https://github.com/simonw/datasette/issues/2078#issuecomment-1563625093,https://api.github.com/repos/simonw/datasette/issues/2078,1563625093,IC_kwDOBm6k_c5dMwaF,9599,simonw,2023-05-25T23:23:15Z,2023-05-25T23:23:15Z,OWNER,"Rest of the work on this will happen in the PR: - #2080","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563522011,https://api.github.com/repos/simonw/datasette/issues/2078,1563522011,IC_kwDOBm6k_c5dMXPb,9599,simonw,2023-05-25T21:22:30Z,2023-05-25T21:22:30Z,OWNER,"This is bad: ```python async def __call__(self, request, datasette): try: handler = getattr(self, request.method.lower()) return await handler(request, datasette) except AttributeError: return await self.method_not_allowed(request) ``` Because it hides any `AttributeError` exceptions that might occur in the view code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563511171,https://api.github.com/repos/simonw/datasette/issues/2078,1563511171,IC_kwDOBm6k_c5dMUmD,9599,simonw,2023-05-25T21:11:20Z,2023-05-25T21:13:05Z,OWNER,I'm going to call this `VerbView` for the moment. Might even rename it to `View` later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563498048,https://api.github.com/repos/simonw/datasette/issues/2078,1563498048,IC_kwDOBm6k_c5dMRZA,9599,simonw,2023-05-25T20:57:52Z,2023-05-25T20:58:13Z,OWNER,"Here's a new `BaseView` class that automatically populates `OPTIONS` based on available methods: ```python class BaseView: async def head(self, *args, **kwargs): try: response = await self.get(*args, **kwargs) response.body = b"""" return response except AttributeError: raise async def method_not_allowed(self, request): if ( request.path.endswith("".json"") or request.headers.get(""content-type"") == ""application/json"" ): response = Response.json( {""ok"": False, ""error"": ""Method not allowed""}, status=405 ) else: response = Response.text(""Method not allowed"", status=405) return response async def options(self, request, *args, **kwargs): response = Response.text(""ok"") response.headers[""allow""] = "", "".join( method.upper() for method in (""head"", ""get"", ""post"", ""put"", ""patch"", ""delete"") if hasattr(self, method) ) return response async def __call__(self, request, datasette): try: handler = getattr(self, request.method.lower()) return await handler(request, datasette) except AttributeError: return await self.method_not_allowed(request) class DemoView(BaseView): async def get(self, datasette, request): return Response.text(""Hello there! {} - {}"".format(datasette, request)) post = get ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563488929,https://api.github.com/repos/simonw/datasette/issues/2078,1563488929,IC_kwDOBm6k_c5dMPKh,9599,simonw,2023-05-25T20:48:12Z,2023-05-25T20:48:39Z,OWNER,"Actually no need for that extra level of parameter detection: `BaseView.__call__` should _always_ take `datasette, request` - `scope` and `receive` are both available on `request`, and `send` is only needed if you're not planning on returning a `Response` object. So the `get` and `post` and suchlike methods should take `datasette` and `request` too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563444296,https://api.github.com/repos/simonw/datasette/issues/2078,1563444296,IC_kwDOBm6k_c5dMERI,9599,simonw,2023-05-25T20:06:08Z,2023-05-25T20:06:08Z,OWNER,"This prototype seems to work well: ```diff diff --git a/datasette/app.py b/datasette/app.py index d7dace67..ed0edf28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -17,6 +17,7 @@ import secrets import sys import threading import time +import types import urllib.parse from concurrent import futures from pathlib import Path @@ -1266,6 +1267,8 @@ class Datasette: # TODO: /favicon.ico and /-/static/ deserve far-future cache expires add_route(favicon, ""/favicon.ico"") + add_route(wrap_view(DemoView, self), '/demo') + add_route( asgi_static(app_root / ""datasette"" / ""static""), r""/-/static/(?P.*)$"" ) @@ -1673,8 +1676,46 @@ def _cleaner_task_str(task): return _cleaner_task_str_re.sub("""", s) -def wrap_view(view_fn, datasette): - @functools.wraps(view_fn) +class DemoView: + async def __call__(self, datasette, request): + return Response.text(""Hello there! {} - {}"".format(datasette, request)) + +def wrap_view(view_fn_or_class, datasette): + is_function = isinstance(view_fn_or_class, types.FunctionType) + if is_function: + return wrap_view_function(view_fn_or_class, datasette) + else: + if not isinstance(view_fn_or_class, type): + raise ValueError(""view_fn_or_class must be a function or a class"") + return wrap_view_class(view_fn_or_class, datasette) + + +def wrap_view_class(view_class, datasette): + async def async_view_for_class(request, send): + instance = view_class() + if inspect.iscoroutinefunction(instance.__call__): + return await async_call_with_supported_arguments( + instance.__call__, + scope=request.scope, + receive=request.receive, + send=send, + request=request, + datasette=datasette, + ) + else: + return call_with_supported_arguments( + instance.__call__, + scope=request.scope, + receive=request.receive, + send=send, + request=request, + datasette=datasette, + ) + + return async_view_for_class + + +def wrap_view_function(view_fn, datasette): async def async_view_fn(request, send): if inspect.iscoroutinefunction(view_fn): response = await async_call_with_supported_arguments( ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563419066,https://api.github.com/repos/simonw/datasette/issues/2078,1563419066,IC_kwDOBm6k_c5dL-G6,9599,simonw,2023-05-25T19:42:16Z,2023-05-25T19:43:08Z,OWNER,"Maybe what I want here is the ability to register classes with the router - and have the router know that if it's a class it should instantiate it via its constructor and then await `__call__` it. The neat thing about it is that it can reduce the risk of having a class instance that accidentally shares state between requests. It also encourages that each class only responds based on the `datasette, request, ...` objects that are passed to its methods.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563359114,https://api.github.com/repos/simonw/datasette/issues/2078,1563359114,IC_kwDOBm6k_c5dLveK,9599,simonw,2023-05-25T18:47:57Z,2023-05-25T18:47:57Z,OWNER,"Oops, that broke everything: ``` @documented async def await_me_maybe(value: typing.Any) -> typing.Any: ""If value is callable, call it. If awaitable, await it. Otherwise return it."" > if callable(value): E TypeError: 'module' object is not callable ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563329245,https://api.github.com/repos/simonw/datasette/issues/2078,1563329245,IC_kwDOBm6k_c5dLoLd,9599,simonw,2023-05-25T18:26:47Z,2023-05-25T18:28:08Z,OWNER,"With type hints and a namedtuple: ```python import asyncio import types from typing import NamedTuple, Any class CallableStatus(NamedTuple): is_callable: bool is_async_callable: bool def check_callable(obj: Any) -> CallableStatus: if not callable(obj): return CallableStatus(False, False) if isinstance(obj, type): # It's a class return CallableStatus(True, False) if isinstance(obj, types.FunctionType): return CallableStatus(True, asyncio.iscoroutinefunction(obj)) if hasattr(obj, ""__call__""): return CallableStatus(True, asyncio.iscoroutinefunction(obj.__call__)) assert False, ""obj {} is somehow callable with no __call__ method"".format(repr(obj)) ``` ```python for thing in ( async_func, non_async_func, AsyncClass(), NotAsyncClass(), ClassNoCall(), AsyncClass, NotAsyncClass, ClassNoCall, ): print(thing, check_callable(thing)) ``` ``` CallableStatus(is_callable=True, is_async_callable=True) CallableStatus(is_callable=True, is_async_callable=False) <__main__.AsyncClass object at 0x106ba7490> CallableStatus(is_callable=True, is_async_callable=True) <__main__.NotAsyncClass object at 0x106740150> CallableStatus(is_callable=True, is_async_callable=False) <__main__.ClassNoCall object at 0x10676d910> CallableStatus(is_callable=False, is_async_callable=False) CallableStatus(is_callable=True, is_async_callable=False) CallableStatus(is_callable=True, is_async_callable=False) CallableStatus(is_callable=True, is_async_callable=False) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563326000,https://api.github.com/repos/simonw/datasette/issues/2078,1563326000,IC_kwDOBm6k_c5dLnYw,9599,simonw,2023-05-25T18:23:38Z,2023-05-25T18:23:38Z,OWNER,I don't like that `is_callable()` implies a single boolean result but actually returns a pair. I'll call it `check_callable(obj)` instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563318598,https://api.github.com/repos/simonw/datasette/issues/2078,1563318598,IC_kwDOBm6k_c5dLllG,9599,simonw,2023-05-25T18:17:03Z,2023-05-25T18:21:25Z,OWNER,"I think I want that to return `(is_callable, is_async)` - so I can both test if the thing can be called AND if it should be awaited in the same operation (without any exceptions). I tried this: ```python def is_callable(obj): ""Returns (is_callable, is_async_callable)"" if not callable(obj): return False, False if isinstance(obj, types.FunctionType): return True, asyncio.iscoroutinefunction(obj) if hasattr(obj, '__call__'): return True, asyncio.iscoroutinefunction(obj.__call__) return False, False ``` ```python for thing in ( async_func, non_async_func, AsyncClass(), NotAsyncClass(), ClassNoCall(), AsyncClass, NotAsyncClass, ClassNoCall ): print(thing, is_callable(thing)) ``` And got: ``` (True, True) (True, False) <__main__.AsyncClass object at 0x106cce490> (True, True) <__main__.NotAsyncClass object at 0x106ccf710> (True, False) <__main__.ClassNoCall object at 0x106ccc810> (False, False) (True, True) (True, False) (True, False) ``` Which is almost right, but I don't like that `AsyncClass` is shown as callable (which it is, since it's a class) and awaitable (which it is not - the `__call__` method may be async but calling the class constructor is not). So I'm going to detect classes using `isinstance(obj, type)`. ```python def is_callable(obj): ""Returns (is_callable, is_async_callable)"" if not callable(obj): return False, False if isinstance(obj, type): # It's a class return True, False if isinstance(obj, types.FunctionType): return True, asyncio.iscoroutinefunction(obj) if hasattr(obj, '__call__'): return True, asyncio.iscoroutinefunction(obj.__call__) assert False, ""obj {} somehow is callable with no __call__ method"".format(obj) ``` I am reasonably confident the `AssertionError` can never be raised. And now: ``` (True, True) (True, False) <__main__.AsyncClass object at 0x106ccfa50> (True, True) <__main__.NotAsyncClass object at 0x106ccc8d0> (True, False) <__main__.ClassNoCall object at 0x106cd7690> (False, False) (True, False) (True, False) (True, False) ``` Which is what I wanted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563308919,https://api.github.com/repos/simonw/datasette/issues/2078,1563308919,IC_kwDOBm6k_c5dLjN3,9599,simonw,2023-05-25T18:08:34Z,2023-05-25T18:08:34Z,OWNER,"After much fiddling this seems to work: ```python import asyncio, types def is_async_callable(obj): if not callable(obj): raise ValueError(""Object is not callable"") if isinstance(obj, types.FunctionType): return asyncio.iscoroutinefunction(obj) if hasattr(obj, '__call__'): return asyncio.iscoroutinefunction(obj.__call__) raise ValueError(""Not a function and has no __call__ attribute"") ``` Tested like so: ```python class AsyncClass: async def __call__(self): pass class NotAsyncClass: def __call__(self): pass class ClassNoCall: pass async def async_func(): pass def non_async_func(): pass for thing in (AsyncClass(), NotAsyncClass(), ClassNoCall(), async_func, non_async_func): try: print(thing, is_async_callable(thing)) except Exception as ex: print(thing, ex) ``` ``` <__main__.AsyncClass object at 0x106c32150> True <__main__.NotAsyncClass object at 0x106c32390> False <__main__.ClassNoCall object at 0x106c32750> Object is not callable True False ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563294669,https://api.github.com/repos/simonw/datasette/issues/2078,1563294669,IC_kwDOBm6k_c5dLfvN,9599,simonw,2023-05-25T17:57:06Z,2023-05-25T17:57:06Z,OWNER,"I may need to be able to detect if a class instance has an `async def __call__` method - I think I can do that like so: ```python def iscoroutinefunction(obj): if inspect.iscoroutinefunction(obj): return True if hasattr(obj, '__call__') and inspect.iscoroutinefunction(obj.__call__): return True return False ``` From https://github.com/encode/starlette/issues/886#issuecomment-606585152","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563292373,https://api.github.com/repos/simonw/datasette/issues/2078,1563292373,IC_kwDOBm6k_c5dLfLV,9599,simonw,2023-05-25T17:55:12Z,2023-05-25T17:55:30Z,OWNER,"So I think subclasses of `BaseView` need to offer a callable which accepts all five of the DI arguments - `datasette`, `request`, `scope`, `send`, `receive` - and then makes a decision based on the HTTP verb as to which method of the class to call. Those methods themselves can accept a subset of those parameters and will only be sent on to them. Having two layers of parameter detection feels a little bit untidy, but I think it will work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563283939,https://api.github.com/repos/simonw/datasette/issues/2078,1563283939,IC_kwDOBm6k_c5dLdHj,9599,simonw,2023-05-25T17:47:38Z,2023-05-25T17:47:38Z,OWNER,"The idea behind `wrap_view()` is dependency injection - it's mainly used by plugins: https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-routes-datasette But I like the pattern so I started using it for some of Datasette's own features. I should use it for _all_ of Datasette's own features. But I still like the way `BaseView` helps with running different code for GET/POST/etc verbs.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/datasette/issues/2078#issuecomment-1563282327,https://api.github.com/repos/simonw/datasette/issues/2078,1563282327,IC_kwDOBm6k_c5dLcuX,9599,simonw,2023-05-25T17:46:05Z,2023-05-25T17:46:05Z,OWNER,"Here's what `wrap_view()` does: https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1676-L1700 It's used e.g. here: https://github.com/simonw/datasette/blob/49184c569cd70efbda4f3f062afef3a34401d8d5/datasette/app.py#L1371-L1375 The `BaseView` thing meanwhile works like this: https://github.com/simonw/datasette/blob/d97e82df3c8a3f2e97038d7080167be9bb74a68d/datasette/views/base.py#L56-L157","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1726236847,Resolve the difference between `wrap_view()` and `BaseView`, https://github.com/simonw/sqlite-utils/issues/554#issuecomment-1557607516,https://api.github.com/repos/simonw/sqlite-utils/issues/554,1557607516,IC_kwDOCGYnMM5c1zRc,1231935,xavdid,2023-05-22T17:18:33Z,2023-05-22T17:18:33Z,NONE,"Oh and for context - this goes away if I use `.upsert` instead of `insert(..., ignore=True)`, but I don't want to update the value if it's written, just do an insert if it's new. The code is basically: ```py def save_items(table, items): db[""users""].insert(build_user(items[0]), pk=""id"",ignore=True) db[table].insert_all(items) if comments := fetch_comments(): save_items('comments', comments) if posts := fetch_posts(): save_items('posts', posts) ``` So either `comments` or `post` could create the relevant user if those items exist. In cases where they _both_ exist, I get this error. I need the `pk` because either call could create the table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1720096994,"`IndexError` when doing `.insert(..., pk='id')` after `insert_all`", https://github.com/simonw/datasette/pull/2077#issuecomment-1653652665,https://api.github.com/repos/simonw/datasette/issues/2077,1653652665,IC_kwDOBm6k_c5ikLy5,49699333,dependabot[bot],2023-07-27T13:40:52Z,2023-07-27T13:40:52Z,CONTRIBUTOR,Superseded by #2121.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20, https://github.com/simonw/datasette/pull/2077#issuecomment-1613290899,https://api.github.com/repos/simonw/datasette/issues/2077,1613290899,IC_kwDOBm6k_c5gKN2T,9599,simonw,2023-06-29T14:32:16Z,2023-06-29T14:32:16Z,OWNER,@dependabot recreate,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20, https://github.com/simonw/datasette/pull/2077#issuecomment-1557289070,https://api.github.com/repos/simonw/datasette/issues/2077,1557289070,IC_kwDOBm6k_c5c0lhu,22429695,codecov[bot],2023-05-22T14:08:33Z,2023-06-29T14:40:35Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`ede6203`)](https://app.codecov.io/gh/simonw/datasette/commit/ede62036180993dbd9d4e5d280fc21c183cda1c3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`9785c4f`)](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2077 +/- ## ======================================= Coverage 92.40% 92.40% ======================================= Files 39 39 Lines 5803 5803 ======================================= Hits 5362 5362 Misses 441 441 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2077?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1719759468,Bump furo from 2023.3.27 to 2023.5.20, https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556288300,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556288300,IC_kwDOCGYnMM5cwxMs,9599,simonw,2023-05-21T20:48:01Z,2023-05-21T20:48:01Z,OWNER,If https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries looks good I can merge this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs, https://github.com/simonw/sqlite-utils/pull/553#issuecomment-1556287870,https://api.github.com/repos/simonw/sqlite-utils/issues/553,1556287870,IC_kwDOCGYnMM5cwxF-,22429695,codecov[bot],2023-05-21T20:45:58Z,2023-05-21T20:57:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`e240133`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/e240133b11588d31dc22c632f7a7ca636c72978d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36% compared to head [(`0b81794`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%. > :exclamation: Current head 0b81794 differs from pull request most recent head 21036a5. Consider uploading reports for the commit 21036a5 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #553 +/- ## ======================================= Coverage 96.36% 96.36% ======================================= Files 6 6 Lines 2726 2726 ======================================= Hits 2627 2627 Misses 99 99 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/553?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718635018,Reformatted CLI examples in docs, https://github.com/simonw/sqlite-utils/issues/552#issuecomment-1556292204,https://api.github.com/repos/simonw/sqlite-utils/issues/552,1556292204,IC_kwDOCGYnMM5cwyJs,9599,simonw,2023-05-21T21:05:15Z,2023-05-21T21:05:15Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/installation.html#setting-up-shell-completion,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718612569,Document how to setup shell auto-completion, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556291915,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556291915,IC_kwDOCGYnMM5cwyFL,9599,simonw,2023-05-21T21:04:03Z,2023-05-21T21:04:03Z,OWNER,Now live at https://sqlite-utils.datasette.io/en/latest/cli.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556288270,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556288270,IC_kwDOCGYnMM5cwxMO,9599,simonw,2023-05-21T20:47:51Z,2023-05-21T20:47:51Z,OWNER,This page has all of the changes: https://sqlite-utils--553.org.readthedocs.build/en/553/cli.html#running-sql-queries,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556287599,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556287599,IC_kwDOCGYnMM5cwxBv,9599,simonw,2023-05-21T20:44:55Z,2023-05-21T20:44:55Z,OWNER,"Put this in a PR so I can preview it: - #553 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556265772,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556265772,IC_kwDOCGYnMM5cwrss,9599,simonw,2023-05-21T19:16:15Z,2023-05-21T19:16:15Z,OWNER,"Another option: That's using this markup: ``` Newline-delimited JSON ~~~~~~~~~~~~~~~~~~~~~~ Use ``--nl`` to get back newline-delimited JSON objects: .. code-block:: bash sqlite-utils dogs.db ""select * from dogs"" --nl .. code-block:: output {""id"": 1, ""age"": 4, ""name"": ""Cleo""} {""id"": 2, ""age"": 2, ""name"": ""Pancakes""} ``` And this extra CSS: ```css .highlight-output .highlight { border-left: 9px solid #30c94f; } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556263182,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556263182,IC_kwDOCGYnMM5cwrEO,9599,simonw,2023-05-21T19:06:48Z,2023-05-21T19:06:48Z,OWNER,"I could split them up into two blocks like this: I do miss the visual indication that one of these is the command and one is the output though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/551#issuecomment-1556262574,https://api.github.com/repos/simonw/sqlite-utils/issues/551,1556262574,IC_kwDOCGYnMM5cwq6u,9599,simonw,2023-05-21T19:04:59Z,2023-05-21T19:04:59Z,OWNER,"I wrote the docs like this because early examples include both the command and its output: https://sqlite-utils.datasette.io/en/stable/cli.html#returning-json ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718607907,Make as many examples in the CLI docs as possible copy-and-pastable, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556255309,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556255309,IC_kwDOCGYnMM5cwpJN,9599,simonw,2023-05-21T18:42:25Z,2023-05-21T18:42:25Z,OWNER,Tests passed here: https://github.com/simonw/sqlite-utils/actions/runs/5039119716,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556250236,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556250236,IC_kwDOCGYnMM5cwn58,9599,simonw,2023-05-21T18:25:26Z,2023-05-21T18:25:26Z,OWNER,"Relevant issues: - https://github.com/python/importlib_metadata/issues/406 - https://github.com/PyCQA/flake8/issues/1701 It looks to me like this is only a problem for `flake8` on Python 3.7 - 3.8 and higher work OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/issues/550#issuecomment-1556249984,https://api.github.com/repos/simonw/sqlite-utils/issues/550,1556249984,IC_kwDOCGYnMM5cwn2A,9599,simonw,2023-05-21T18:24:48Z,2023-05-21T18:24:48Z,OWNER,"This is blocking: - #549","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718595700,AttributeError: 'EntryPoints' object has no attribute 'get' for flake8 on Python 3.7, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556242262,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556242262,IC_kwDOCGYnMM5cwl9W,9599,simonw,2023-05-21T18:00:05Z,2023-05-21T18:00:05Z,OWNER,"Failing `mypy` test: https://github.com/simonw/sqlite-utils/actions/runs/5038983349/jobs/9036828465 ``` sqlite_utils/cli.py:37: error: Skipping analyzing ""trogon"": module is installed, but missing library stubs or py.typed marker [import] sqlite_utils/cli.py:37: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 52 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241812,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241812,IC_kwDOCGYnMM5cwl2U,9599,simonw,2023-05-21T17:58:25Z,2023-05-21T17:58:25Z,OWNER,Documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/pull/549#issuecomment-1556241555,https://api.github.com/repos/simonw/sqlite-utils/issues/549,1556241555,IC_kwDOCGYnMM5cwlyT,22429695,codecov[bot],2023-05-21T17:57:24Z,2023-05-21T18:28:44Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`83.33`**% and project coverage change: **`+0.06`** :tada: > Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`948692a`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.36%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #549 +/- ## ========================================== + Coverage 96.30% 96.36% +0.06% ========================================== Files 6 6 Lines 2707 2726 +19 ========================================== + Hits 2607 2627 +20 + Misses 100 99 -1 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.22% <83.33%> (-0.03%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/549?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718586377,TUI powered by Trogon, https://github.com/simonw/sqlite-utils/issues/548#issuecomment-1556231832,https://api.github.com/repos/simonw/sqlite-utils/issues/548,1556231832,IC_kwDOCGYnMM5cwjaY,9599,simonw,2023-05-21T17:24:13Z,2023-05-21T17:24:13Z,OWNER,"Oh, I see why that is now: https://github.com/simonw/sqlite-utils/blob/6027f3ea6939a399aeef2578fca17efec0e539df/sqlite_utils/cli.py#L2670-L2679 This is because of the following command: sqlite-utils analyze-tables table1 table2 --column x Since you can pass multiple tables AND multiple columns, the tool currently assumes that the column(s) you specify may be available on a subset of the provided tables. I'm going to change this so if the column is not on ANY of those tables you get an error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718576761,analyze-tables should validate provide --column names, https://github.com/simonw/sqlite-utils/issues/547#issuecomment-1556228395,https://api.github.com/repos/simonw/sqlite-utils/issues/547,1556228395,IC_kwDOCGYnMM5cwikr,9599,simonw,2023-05-21T17:11:15Z,2023-05-21T17:11:15Z,OWNER,"This will be a cosmetic change to the CLI output only - the options to save data to the database and the Python API function will continue to return `[(None, 158)]`. I can add an optimization though to avoid running the SQL count query if we know that it's all `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718572201,No need to show common values if everything is null, https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213396,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213396,IC_kwDOCGYnMM5cwe6U,9599,simonw,2023-05-21T15:58:12Z,2023-05-21T16:18:46Z,OWNER,"Documentation preview: - https://sqlite-utils--546.org.readthedocs.build/en/546/cli.html#cli-analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/cli-reference.html#analyze-tables - https://sqlite-utils--546.org.readthedocs.build/en/546/python-api.html#analyzing-a-column - https://sqlite-utils--546.org.readthedocs.build/en/546/reference.html#sqlite_utils.db.Table.analyze_column","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least", https://github.com/simonw/sqlite-utils/pull/546#issuecomment-1556213031,https://api.github.com/repos/simonw/sqlite-utils/issues/546,1556213031,IC_kwDOCGYnMM5cwe0n,22429695,codecov[bot],2023-05-21T15:56:05Z,2023-05-21T16:18:03Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`93.75`**% and no project coverage change. > Comparison is base [(`b3b100d`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/b3b100d7f5b2a76ccd4bfe8b0301a29e321d0375?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.30% compared to head [(`9f23e68`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.31%. > :exclamation: Current head 9f23e68 differs from pull request most recent head 2eca17d. Consider uploading reports for the commit 2eca17d to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #546 +/- ## ======================================= Coverage 96.30% 96.31% ======================================= Files 6 6 Lines 2707 2712 +5 ======================================= + Hits 2607 2612 +5 Misses 100 100 ``` | [Impacted Files](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.37% <90.90%> (+<0.01%)` | :arrow_up: | | [sqlite\_utils/cli.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2NsaS5weQ==) | `95.26% <100.00%> (+0.01%)` | :arrow_up: |
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/546?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718550688,"Analyze tables options: --common-limit, --no-most, --no-least", https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556269616,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556269616,IC_kwDOCGYnMM5cwsow,9599,simonw,2023-05-21T19:33:13Z,2023-05-21T19:33:13Z,OWNER,Now released: https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556247818,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556247818,IC_kwDOCGYnMM5cwnUK,9599,simonw,2023-05-21T18:17:46Z,2023-05-21T18:17:46Z,OWNER,Draft documentation: https://sqlite-utils--549.org.readthedocs.build/en/549/cli.html#cli-tui,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556210844,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556210844,IC_kwDOCGYnMM5cweSc,9599,simonw,2023-05-21T15:44:10Z,2023-05-21T15:44:10Z,OWNER,"It looks like `nargs=-1` on a positional argument isn't yet supported - opened an issue here: - https://github.com/Textualize/trogon/issues/4","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556191894,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556191894,IC_kwDOCGYnMM5cwZqW,9599,simonw,2023-05-21T14:20:14Z,2023-05-21T14:20:14Z,OWNER,"Opened a feature request for customizing the help and command name: - https://github.com/Textualize/trogon/issues/2","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556190531,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556190531,IC_kwDOCGYnMM5cwZVD,9599,simonw,2023-05-21T14:13:43Z,2023-05-21T14:13:43Z,OWNER,"OK, this works! ![trogon](https://github.com/simonw/sqlite-utils/assets/9599/2ae194c5-ec82-471a-9d1b-b01b3f2632f3) To try it out, install that branch from GitHub: pip install https://github.com/simonw/sqlite-utils/archive/refs/heads/trogon.zip Then run this: sqlite-utils install trogon And this: sqlite-utils tui ","{""total_count"": 5, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 3, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/545#issuecomment-1556189823,https://api.github.com/repos/simonw/sqlite-utils/issues/545,1556189823,IC_kwDOCGYnMM5cwZJ_,9599,simonw,2023-05-21T14:09:59Z,2023-05-21T14:09:59Z,OWNER,"I don't want to add `trogon` as a default dependency because it's a little heavy - it pulls in all of Rich and Textual as well. People who use `sqlite-utils` just for its Python API won't benefit from this - it's a CLI feature only. But I have a `sqlite-utils install ...` command for helping people to install packages into the same virtual environment as `sqlite-utils` no matter how they installed that tool: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install So I can treat Trogon as an optional dependency and add the `sqlite-utils tui` command only if that package is also installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718517882,Try out Trogon for a tui interface, https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556225788,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556225788,IC_kwDOCGYnMM5cwh78,9599,simonw,2023-05-21T17:02:05Z,2023-05-21T17:02:05Z,OWNER,"New docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#cli-analyze-tables - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#analyze-tables - https://sqlite-utils.datasette.io/en/latest/python-api.html#analyzing-a-column - https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Table.analyze_column New help output: ``` % sqlite-utils analyze-tables --help Usage: sqlite-utils analyze-tables [OPTIONS] PATH [TABLES]... Analyze the columns in one or more tables Example: sqlite-utils analyze-tables data.db trees Options: -c, --column TEXT Specific columns to analyze --save Save results to _analyze_tables table --common-limit INTEGER How many common values --no-most Skip most common values --no-least Skip least common values --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/sqlite-utils/issues/544#issuecomment-1556211643,https://api.github.com/repos/simonw/sqlite-utils/issues/544,1556211643,IC_kwDOCGYnMM5cwee7,9599,simonw,2023-05-21T15:48:17Z,2023-05-21T15:48:17Z,OWNER,I generated the commit message in https://github.com/simonw/sqlite-utils/commit/1c1991b447a1ddd3d61d9d4a8a1d6a9da47ced20 using `git diff | llm --system 'describe this change'`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1718515590,New options for analyze-tables --common-limit --no-most and --no-least, https://github.com/simonw/datasette/pull/2075#issuecomment-1649849249,https://api.github.com/repos/simonw/datasette/issues/2075,1649849249,IC_kwDOBm6k_c5iVrOh,49699333,dependabot[bot],2023-07-25T13:28:35Z,2023-07-25T13:28:35Z,CONTRIBUTOR,Superseded by #2107.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1710164693,Bump sphinx from 6.1.3 to 7.0.1, https://github.com/simonw/datasette/pull/2075#issuecomment-1547944971,https://api.github.com/repos/simonw/datasette/issues/2075,1547944971,IC_kwDOBm6k_c5cQ8QL,22429695,codecov[bot],2023-05-15T14:12:20Z,2023-05-15T14:12:20Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`49184c5`)](https://app.codecov.io/gh/simonw/datasette/commit/49184c569cd70efbda4f3f062afef3a34401d8d5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40% compared to head [(`b99e1d3`)](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.40%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2075 +/- ## ======================================= Coverage 92.40% 92.40% ======================================= Files 38 38 Lines 5751 5751 ======================================= Hits 5314 5314 Misses 437 437 ```
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2075?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1710164693,Bump sphinx from 6.1.3 to 7.0.1, https://github.com/simonw/datasette/issues/2073#issuecomment-1546119773,https://api.github.com/repos/simonw/datasette/issues/2073,1546119773,IC_kwDOBm6k_c5cJ-pd,9599,simonw,2023-05-12T18:24:07Z,2023-05-12T18:24:07Z,OWNER,"Here's a demo of this breaking in Datasette Lite: https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data/baseline?_filter_column_1=is_baseline&_filter_op_1=exact&_filter_value_1=1&_filter_column_2=&_filter_op_2=notnull__1&_filter_value_2=1&_filter_column=&_filter_op=exact&_filter_value=&_sort=&_facet=is_baseline Here's a SQL query that demonstrates the underlying issue: ```sql select 'working', count(*) from baseline where is_baseline = 1 union all select 'broken', count(*) from baseline where is_baseline = '1' ``` https://lite.datasette.io/?sql=https://gist.github.com/simonw/261564c0ca01567df6eeb9b222b8be84&json=https%3A%2F%2Fcdn.jsdelivr.net%2Fnpm%2Fweb-features%2Findex.json#/data?sql=select+%27working%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+1%0Aunion+all%0Aselect+%27broken%27%2C+count%28*%29+from+baseline+where+is_baseline+%3D+%271%27 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1708030220,Faceting doesn't work against integer columns in views, https://github.com/simonw/datasette/issues/2073#issuecomment-1546117538,https://api.github.com/repos/simonw/datasette/issues/2073,1546117538,IC_kwDOBm6k_c5cJ-Gi,9599,simonw,2023-05-12T18:21:38Z,2023-05-12T18:21:38Z,OWNER,"https://latest.datasette.io/fixtures doesn't currently have a view with any integer columns in it, making this bug harder to demonstrate there. I can't replicate the bug using https://datasette.io/content/plugins?_facet=stargazers_count&stargazers_count=3 - I would expect that not to work correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1708030220,Faceting doesn't work against integer columns in views, https://github.com/simonw/datasette/issues/2070#issuecomment-1540494121,https://api.github.com/repos/simonw/datasette/issues/2070,1540494121,IC_kwDOBm6k_c5b0hMp,9599,simonw,2023-05-09T16:25:00Z,2023-05-09T16:25:00Z,OWNER,Can now be used here: https://github.com/simonw/datasette/actions/workflows/deploy-branch-preview.yml,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1702354223,Mechanism for deploying a preview of a branch using Vercel, https://github.com/simonw/datasette/issues/2070#issuecomment-1540491751,https://api.github.com/repos/simonw/datasette/issues/2070,1540491751,IC_kwDOBm6k_c5b0gnn,9599,simonw,2023-05-09T16:23:12Z,2023-05-09T16:23:12Z,OWNER,Added a actions `BRANCH_PREVIEW_VERCEL_TOKEN` secret to this repository.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1702354223,Mechanism for deploying a preview of a branch using Vercel, https://github.com/simonw/sqlite-utils/issues/542#issuecomment-1539052467,https://api.github.com/repos/simonw/sqlite-utils/issues/542,1539052467,IC_kwDOCGYnMM5bvBOz,9599,simonw,2023-05-08T21:07:41Z,2023-05-08T21:07:41Z,OWNER,"Relevant commits (will mostly revert these): - https://github.com/simonw/sqlite-utils/commit/455c35b512895c19bf922c2b804d750d27cb8dbd - https://github.com/simonw/sqlite-utils/commit/e0ec4c345129996011951e400388fd74865f65a2","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700936245,Remove `skip_false=True` and `--no-skip-false` in `sqlite-utils` 4.0, https://github.com/simonw/sqlite-utils/issues/541#issuecomment-1538963959,https://api.github.com/repos/simonw/sqlite-utils/issues/541,1538963959,IC_kwDOCGYnMM5burn3,9599,simonw,2023-05-08T19:59:34Z,2023-05-08T19:59:34Z,OWNER,"There are 8 failing tests left: ``` ==== short test summary info ==== FAILED tests/test_cli_memory.py::test_memory_csv[False-test] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_csv[False-t1] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_tsv[False] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_dump[extra_args0] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_cli_memory.py::test_memory_two_files_with_same_stem - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedate] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bcaca0> FAILED tests/test_recipes.py::test_dateparse_errors[None-parsedatetime] - pytest.PytestUnraisableExceptionWarning: Exception ignored in: .convert_value at 0x106bc9620> ERROR tests/test_cli.py::test_upsert_analyze - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]> ==== 8 failed, 894 passed, 4 skipped, 1 error in 6.27s ==== ``` Full traceback here: https://gist.github.com/simonw/b40b3e814729d6c02a0302a84ce54d9e","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1700840265,Get tests to pass with `pytest -Werror`, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537744000,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537744000,IC_kwDOCGYnMM5bqByA,42327,pquentin,2023-05-08T04:56:12Z,2023-05-08T04:56:12Z,NONE,"Hey @simonw, urllib3 maintainer here :wave: Sorry for breaking your CI. I understand you may prefer to pin the Python version, but note that specifying just `python: ""3""` will get you the latest. We use that in urllib3: https://github.com/urllib3/urllib3/blob/main/.readthedocs.yml I can open PRs to sqlite-utils / datasette if you're interested","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537514069,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537514069,IC_kwDOCGYnMM5bpJpV,9599,simonw,2023-05-07T18:40:18Z,2023-05-07T18:40:18Z,OWNER,"https://docs.readthedocs.io/en/stable/config-file/v2.html suggests: ```yaml build: os: ubuntu-22.04 tools: python: ""3.11"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513912,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513912,IC_kwDOCGYnMM5bpJm4,9599,simonw,2023-05-07T18:39:29Z,2023-05-07T18:39:29Z,OWNER,"https://readthedocs.org/projects/sqlite-utils/builds/20513034/ said: > Problem in your project's configuration. Invalid ""python.version"": expected one of (2, 2.7, 3, 3.5, 3.6, 3.7, 3.8, pypy3.5), got 3.11","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/540#issuecomment-1537513653,https://api.github.com/repos/simonw/sqlite-utils/issues/540,1537513653,IC_kwDOCGYnMM5bpJi1,9599,simonw,2023-05-07T18:37:59Z,2023-05-07T18:38:19Z,OWNER,"Useful comment here: - https://github.com/urllib3/urllib3/issues/2168#issuecomment-1537360928 > I faced the same issue. I switched to a different Python kernel (3.11.2) and it worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699184583,sphinx.builders.linkcheck build error, https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537514610,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537514610,IC_kwDOCGYnMM5bpJxy,9599,simonw,2023-05-07T18:43:24Z,2023-05-07T18:43:24Z,OWNER,"Documentation: - https://sqlite-utils.datasette.io/en/latest/cli.html#returning-raw-data-such-as-binary-content - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#memory","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507676,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507676,IC_kwDOCGYnMM5bpIFc,9599,simonw,2023-05-07T18:09:43Z,2023-05-07T18:09:54Z,OWNER,"This worked: ```bash sqlite-utils memory /tmp/books3.json:nl \ 'select name from books3' --raw-lines > titles.txt ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507525,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507525,IC_kwDOCGYnMM5bpIDF,9599,simonw,2023-05-07T18:09:09Z,2023-05-07T18:09:09Z,OWNER,"I'm tempted to upgrade `--raw` to do this instead, but that would be a breaking change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/sqlite-utils/issues/539#issuecomment-1537507394,https://api.github.com/repos/simonw/sqlite-utils/issues/539,1537507394,IC_kwDOCGYnMM5bpIBC,9599,simonw,2023-05-07T18:08:44Z,2023-05-07T18:08:44Z,OWNER,"Prototype: ```diff diff --git a/docs/cli-reference.rst b/docs/cli-reference.rst index 153e5f9..c830518 100644 --- a/docs/cli-reference.rst +++ b/docs/cli-reference.rst @@ -124,6 +124,7 @@ See :ref:`cli_query`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --functions TEXT Python code defining one or more custom SQL functions @@ -192,6 +193,7 @@ See :ref:`cli_memory`. --json-cols Detect JSON cols and output them as JSON, not escaped strings -r, --raw Raw output, first column of first row + --raw-lines Raw output, first column of each row -p, --param ... Named :parameters for SQL query --encoding TEXT Character encoding for CSV input, defaults to utf-8 diff --git a/sqlite_utils/cli.py b/sqlite_utils/cli.py index d25b1df..da0e4b6 100644 --- a/sqlite_utils/cli.py +++ b/sqlite_utils/cli.py @@ -1653,6 +1653,7 @@ def drop_view(path, view, ignore, load_extension): ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1677,6 +1678,7 @@ def query( fmt, json_cols, raw, + raw_lines, param, load_extension, functions, @@ -1700,7 +1702,19 @@ def query( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) @@ -1728,6 +1742,7 @@ def query( ) @output_options @click.option(""-r"", ""--raw"", is_flag=True, help=""Raw output, first column of first row"") +@click.option(""--raw-lines"", is_flag=True, help=""Raw output, first column of each row"") @click.option( ""-p"", ""--param"", @@ -1773,6 +1788,7 @@ def memory( fmt, json_cols, raw, + raw_lines, param, encoding, no_detect_types, @@ -1879,12 +1895,36 @@ def memory( _register_functions(db, functions) _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ) def _execute_query( - db, sql, param, raw, table, csv, tsv, no_headers, fmt, nl, arrays, json_cols + db, + sql, + param, + raw, + raw_lines, + table, + csv, + tsv, + no_headers, + fmt, + nl, + arrays, + json_cols, ): with db.conn: try: @@ -1903,6 +1943,13 @@ def _execute_query( sys.stdout.buffer.write(data) else: sys.stdout.write(str(data)) + elif raw_lines: + for row in cursor: + data = row[0] + if isinstance(data, bytes): + sys.stdout.buffer.write(data + b""\n"") + else: + sys.stdout.write(str(data) + ""\n"") elif fmt or table: print( tabulate.tabulate( ``` Needs tests and more documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1699174055,"`--raw-lines` option, like `--raw` for multiple lines", https://github.com/simonw/datasette/issues/2069#issuecomment-1537277919,https://api.github.com/repos/simonw/datasette/issues/2069,1537277919,IC_kwDOBm6k_c5boP_f,31861128,yqlbu,2023-05-07T03:17:35Z,2023-05-07T03:17:35Z,NONE,"Some updates: I notice that there is an option in the CLI where we can explicitly set `immutable` mode when spinning up the server ```console Options: -i, --immutable PATH Database files to open in immutable mode ``` Then, the question is - how can I disable immutable mode in the deployed instance on Vercel?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1698865182,[BUG] Cannot insert new data to deployed instance, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538975545,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538975545,IC_kwDOCGYnMM5buuc5,1231935,xavdid,2023-05-08T20:06:35Z,2023-05-08T20:06:35Z,NONE,"perfect, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538921774,IC_kwDOCGYnMM5buhUu,9599,simonw,2023-05-08T19:24:41Z,2023-05-08T19:24:41Z,OWNER,That fix seems to work!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538910894,IC_kwDOCGYnMM5buequ,9599,simonw,2023-05-08T19:16:52Z,2023-05-08T19:17:00Z,OWNER,"How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`? Something like this: ```python [ ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538903556,IC_kwDOCGYnMM5buc4E,9599,simonw,2023-05-08T19:11:24Z,2023-05-08T19:13:23Z,OWNER,"I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though. This would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538893329,IC_kwDOCGYnMM5buaYR,9599,simonw,2023-05-08T19:04:47Z,2023-05-08T19:04:47Z,OWNER,"This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`. One fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html But... > UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04). I still want to support SQLite versions earlier than that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538889482,IC_kwDOCGYnMM5buZcK,9599,simonw,2023-05-08T19:02:38Z,2023-05-08T19:02:38Z,OWNER,"Here's the code at fault: https://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538887361,IC_kwDOCGYnMM5buY7B,9599,simonw,2023-05-08T19:01:20Z,2023-05-08T19:01:20Z,OWNER,"Here's the problem: ```python import sqlite3 db = sqlite3.connect("":memory:"") db.execute('create table foo (id integer primary key, name not null)') db.execute('insert into foo (id) values (1)') ``` Produces: ``` IntegrityError: NOT NULL constraint failed: foo.name ``` But this: ```python db.execute('insert or ignore into foo (id) values (1)') ``` Completes without an exception.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538801855,IC_kwDOCGYnMM5buEC_,9599,simonw,2023-05-08T18:00:17Z,2023-05-08T18:00:17Z,OWNER,"From time in the debugger, after creating the table it ends up doing this: ``` (Pdb) queries_and_params [ ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]), ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1]) ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817,https://api.github.com/repos/simonw/sqlite-utils/issues/538,1538793817,IC_kwDOCGYnMM5buCFZ,9599,simonw,2023-05-08T17:55:10Z,2023-05-08T17:55:10Z,OWNER,"Confirmed - I added this test and it fails: ```python def test_upsert_all_not_null(fresh_db): # https://github.com/simonw/sqlite-utils/issues/538 fresh_db[""comments""].upsert_all( [{""id"": 1, ""name"": ""Cleo""}], pk=""id"", not_null=[""name""], ) assert list(fresh_db[""comments""].rows) == [{""id"": 1, ""name"": ""Cleo""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1695428235,`table.upsert_all` fails to write rows when `not_null` is present, https://github.com/simonw/datasette/pull/2068#issuecomment-1547911570,https://api.github.com/repos/simonw/datasette/issues/2068,1547911570,IC_kwDOBm6k_c5cQ0GS,49699333,dependabot[bot],2023-05-15T13:59:35Z,2023-05-15T13:59:35Z,CONTRIBUTOR,Superseded by #2075.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1690842199,Bump sphinx from 6.1.3 to 7.0.0, https://github.com/simonw/datasette/issues/2067#issuecomment-1532304714,https://api.github.com/repos/simonw/datasette/issues/2067,1532304714,IC_kwDOBm6k_c5bVR1K,39538958,justmars,2023-05-03T00:16:03Z,2023-05-03T00:16:03Z,NONE,"Curiously, after running commands on the database that was litestream-restored, datasette starts to work again, e.g. ```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db datasette data/db.sqlite # fails (OperationalError described above) ``` ```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db sqlite-utils enable-wal data/db.sqlite datasette data/db.sqlite # works ``` ```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db sqlite-utils optimize data/db.sqlite datasette data/db.sqlite # works ``` ```sh litestream restore -o data/db.sqlite s3://mytestbucketxx/db sqlite3 data/db.sqlite "".clone test.db"" datasette test.db # works ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1690765434,Litestream-restored db: errors on 3.11 and 3.10.8; but works on py3.10.7 and 3.10.6, https://github.com/simonw/datasette/issues/2066#issuecomment-1524680160,https://api.github.com/repos/simonw/datasette/issues/2066,1524680160,IC_kwDOBm6k_c5a4MXg,9599,simonw,2023-04-27T04:27:50Z,2023-04-27T04:27:50Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524675817,https://api.github.com/repos/simonw/datasette/issues/2066,1524675817,IC_kwDOBm6k_c5a4LTp,9599,simonw,2023-04-27T04:21:03Z,2023-04-27T04:21:03Z,OWNER,I went with this to generate the long string: https://github.com/simonw/datasette/blob/0b0c5cd7a94fe3f151a3e10261b5c84ee64f2f18/tests/test_csv.py#L157-L176,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524669124,https://api.github.com/repos/simonw/datasette/issues/2066,1524669124,IC_kwDOBm6k_c5a4JrE,9599,simonw,2023-04-27T04:10:44Z,2023-04-27T04:10:52Z,OWNER,"I need an alternative way of generating a long string with a shorter URL. ```sql select group_concat('abcabcabc', '') from json_each(json_array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)) ``` https://latest.datasette.io/_memory?sql=select+group_concat%28%27abcabcabc%27%2C+%27%27%29+from+json_each%28json_array%281%2C+2%2C+3%2C+4%2C+5%2C+6%2C+7%2C+8%2C+9%2C+10%29%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524666049,https://api.github.com/repos/simonw/datasette/issues/2066,1524666049,IC_kwDOBm6k_c5a4I7B,9599,simonw,2023-04-27T04:06:18Z,2023-04-27T04:06:18Z,OWNER,"Most recent `httpx` release is 0.24 a couple of weeks ago. Here's what changed in that version: https://github.com/encode/httpx/compare/0.23.3...0.24.0 It looks like that URL limit is new: https://github.com/encode/httpx/commit/57daabf673705954afa94686c0002801c93d31f3#diff-78d8d93b5dd4c77d99c3e2b46b7286ba71a8fd60e92d8bd4eee5fb200b4f87bfR149-R155 ```python def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult: # Initial basic checks on allowable URLs. # --------------------------------------- # Hard limit the maximum allowable URL length. if len(url) > MAX_URL_LENGTH: raise InvalidURL(""URL too long"") ``` https://github.com/encode/httpx/blob/32e25497a36e6222cc64a758c98275b450dac28d/httpx/_urlparse.py#L153-L155","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524660603,https://api.github.com/repos/simonw/datasette/issues/2066,1524660603,IC_kwDOBm6k_c5a4Hl7,9599,simonw,2023-04-27T04:02:55Z,2023-04-27T04:02:55Z,OWNER,"In the debugger: ``` (Pdb) MAX_URL_LENGTH 65536 ``` Weird this only seems to be a problem with `httpx` on Python 3.7 though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524659084,https://api.github.com/repos/simonw/datasette/issues/2066,1524659084,IC_kwDOBm6k_c5a4HOM,9599,simonw,2023-04-27T04:02:07Z,2023-04-27T04:02:07Z,OWNER,"This is the failing test: https://github.com/simonw/datasette/blob/249fcf8e3e2a90e763f41b080c1b9ec8017f5005/tests/test_csv.py#L156-L167","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524655203,https://api.github.com/repos/simonw/datasette/issues/2066,1524655203,IC_kwDOBm6k_c5a4GRj,9599,simonw,2023-04-27T03:59:56Z,2023-04-27T03:59:56Z,OWNER,"In a fresh Datasette checkout I ran: pipenv shell --python /Users/simon/.pyenv/versions/3.7.16/bin/python That gave me a virtual environment with 3.7.16 Python. Then I ran: pip install -e '.[test]' Weirdly that gave me a `which pytest` of `/opt/homebrew/bin/pytest` which ran the tests on 3.11. I figured out the location of the virtual environment with `which python` and then ran this: ``` % /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/bin/pytest tests/test_csv.py ============================================================================================== test session starts =============================================================================================== platform darwin -- Python 3.7.16, pytest-7.3.1, pluggy-1.0.0 SQLite: 3.39.5 rootdir: /private/tmp/datasette configfile: pytest.ini plugins: asyncio-0.21.0, timeout-2.1.0, xdist-3.2.1, anyio-3.6.2 asyncio: mode=strict collected 15 items tests/test_csv.py ..........F.... [100%] ==================================================================================================== FAILURES ==================================================================================================== ________________________________________________________________________________________________ test_max_csv_mb _________________________________________________________________________________________________ app_client_csv_max_mb_one = def test_max_csv_mb(app_client_csv_max_mb_one): response = app_client_csv_max_mb_one.get( ( ""/fixtures.csv?sql=select+'{}'+"" ""from+compound_three_primary_keys&_stream=1&_size=max"" > ).format(""abcdefg"" * 10000) ) /private/tmp/datasette/tests/test_csv.py:161: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:240: in __call__ return call_result.result() /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:428: in result return self.__get_result() /Users/simon/.pyenv/versions/3.7.16/lib/python3.7/concurrent/futures/_base.py:384: in __get_result raise self._exception /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/asgiref/sync.py:306: in main_wrap result = await self.awaitable(*args, **kwargs) /private/tmp/datasette/datasette/utils/testing.py:76: in get headers=headers, /private/tmp/datasette/datasette/utils/testing.py:167: in _request content=post_body, /private/tmp/datasette/datasette/app.py:1787: in request method, self._fix(path, avoid_path_rewrites), **kwargs /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:1528: in request extensions=extensions, /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:346: in build_request url = self._merge_url(url) /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_client.py:376: in _merge_url merge_url = URL(url) /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urls.py:113: in __init__ self._uri_reference = urlparse(url, **kwargs) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ url = ""http://localhost/fixtures.csv?sql=select+'abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcde...gabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg'+from+compound_three_primary_keys&_stream=1&_size=max"" kwargs = {} def urlparse(url: str = """", **kwargs: typing.Optional[str]) -> ParseResult: # Initial basic checks on allowable URLs. # --------------------------------------- # Hard limit the maximum allowable URL length. if len(url) > MAX_URL_LENGTH: > raise InvalidURL(""URL too long"") E httpx.InvalidURL: URL too long /Users/simon/.local/share/virtualenvs/datasette-cZYvnUqY/lib/python3.7/site-packages/httpx/_urlparse.py:155: InvalidURL ============================================================================================ short test summary info ============================================================================================= FAILED tests/test_csv.py::test_max_csv_mb - httpx.InvalidURL: URL too long ========================================================================================== 1 failed, 14 passed in 1.83s ========================================================================================== (datasette) simon@Simons-MacBook-Pro datasette % ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524648995,https://api.github.com/repos/simonw/datasette/issues/2066,1524648995,IC_kwDOBm6k_c5a4Ewj,9599,simonw,2023-04-27T03:56:42Z,2023-04-27T03:57:11Z,OWNER,"I don't have 3.7 locally. Trying to install it with `pyenv`. brew install pyenv Then: pyenv install --list | grep 3.7 Installing: pyenv install 3.7.16 Output: Installed Python-3.7.16 to /Users/simon/.pyenv/versions/3.7.16 So the executable is `/Users/simon/.pyenv/versions/3.7.16/bin/python`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524638233,https://api.github.com/repos/simonw/datasette/issues/2066,1524638233,IC_kwDOBm6k_c5a4CIZ,9599,simonw,2023-04-27T03:50:51Z,2023-04-27T03:50:51Z,OWNER,Failure was on 3.7. I'll try that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2066#issuecomment-1524637376,https://api.github.com/repos/simonw/datasette/issues/2066,1524637376,IC_kwDOBm6k_c5a4B7A,9599,simonw,2023-04-27T03:50:19Z,2023-04-27T03:50:19Z,OWNER,"Having trouble replicating this on my laptop. I tried a fresh virtual environment with fresh packages (in case this is a `httpx` change) but this passed: pytest tests/test_csv.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686042269,Failing test: httpx.InvalidURL: URL too long, https://github.com/simonw/datasette/issues/2065#issuecomment-1524709988,https://api.github.com/repos/simonw/datasette/issues/2065,1524709988,IC_kwDOBm6k_c5a4Tpk,9599,simonw,2023-04-27T05:09:36Z,2023-04-27T05:09:36Z,OWNER,"That fixed it - after installing `main.zip` again I ran this and it worked: ~/.rye/tools/main-zip/bin/datasette install datasette-graphql ``` % ~/.rye/tools/main-zip/bin/datasette plugins [ { ""name"": ""datasette-graphql"", ""static"": true, ""templates"": true, ""version"": ""2.2"", ""hooks"": [ ""database_actions"", ""extra_template_vars"", ""menu_links"", ""register_routes"", ""startup"", ""table_actions"" ] } ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/datasette/issues/2065#issuecomment-1524707628,https://api.github.com/repos/simonw/datasette/issues/2065,1524707628,IC_kwDOBm6k_c5a4TEs,9599,simonw,2023-04-27T05:06:44Z,2023-04-27T05:06:44Z,OWNER,"I need `pip` as a dependency too: ``` % ~/.rye/tools/main-zip/bin/datasette install datasette-graphql Traceback (most recent call last): File ""/Users/simon/.rye/tools/main-zip/bin/datasette"", line 8, in sys.exit(cli()) ^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/click/core.py"", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/.rye/tools/main-zip/lib/python3.11/site-packages/datasette/cli.py"", line 365, in install run_module(""pip"", run_name=""__main__"") File """", line 222, in run_module File """", line 142, in _get_module_details ImportError: No module named pip ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/datasette/issues/2065#issuecomment-1524699863,https://api.github.com/repos/simonw/datasette/issues/2065,1524699863,IC_kwDOBm6k_c5a4RLX,9599,simonw,2023-04-27T04:56:22Z,2023-04-27T04:56:22Z,OWNER,Turned this into a TIL: https://til.simonwillison.net/python/rye,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/datasette/issues/2065#issuecomment-1524616740,https://api.github.com/repos/simonw/datasette/issues/2065,1524616740,IC_kwDOBm6k_c5a384k,9599,simonw,2023-04-27T03:38:21Z,2023-04-27T03:38:21Z,OWNER,"Tried this: rye install https://github.com/simonw/datasette/archive/refs/heads/main.zip But got this error: Error: Expected one of `@`, `(`, `<`, `=`, `>`, `~`, `!`, `;`, found `:` I instead downloaded that file and ran: rye install main.zip This worked! And now: ``` ~/.rye/tools/main-zip/bin/datasette --version ``` ``` datasette, version 1.0a2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1686033652,Datasette cannot be installed with Rye, https://github.com/simonw/datasette/pull/2064#issuecomment-1529737426,https://api.github.com/repos/simonw/datasette/issues/2064,1529737426,IC_kwDOBm6k_c5bLfDS,49699333,dependabot[bot],2023-05-01T13:58:50Z,2023-05-01T13:58:50Z,CONTRIBUTOR,Superseded by #2068.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1683229834,Bump sphinx from 6.1.3 to 6.2.1, https://github.com/simonw/datasette/pull/2063#issuecomment-1521837780,https://api.github.com/repos/simonw/datasette/issues/2063,1521837780,IC_kwDOBm6k_c5atWbU,49699333,dependabot[bot],2023-04-25T13:57:52Z,2023-04-25T13:57:52Z,CONTRIBUTOR,Superseded by #2064.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1681339696,Bump sphinx from 6.1.3 to 6.2.0, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539157643,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539157643,IC_kwDOCGYnMM5bva6L,9599,simonw,2023-05-08T22:45:09Z,2023-05-08T22:45:21Z,OWNER,"Here's an example from the new tests: https://github.com/simonw/sqlite-utils/blob/a75abeb61b91a28650d3b9933e7ec80ad0d92529/tests/test_create.py#L291-L307","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1539055393,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1539055393,IC_kwDOCGYnMM5bvB8h,9599,simonw,2023-05-08T21:10:06Z,2023-05-08T21:10:06Z,OWNER,Thanks!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/sqlite-utils/pull/537#issuecomment-1506200813,https://api.github.com/repos/simonw/sqlite-utils/issues/537,1506200813,IC_kwDOCGYnMM5Zxszt,22429695,codecov[bot],2023-04-13T01:45:22Z,2023-04-13T01:45:22Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage: **`100.00`**% and no project coverage change. > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`a75abeb`)](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #537 +/- ## ======================================= Coverage 96.25% 96.25% ======================================= Files 6 6 Lines 2671 2673 +2 ======================================= + Hits 2571 2573 +2 Misses 100 100 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.33% <100.00%> (+<0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/537?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665200812,Support self-referencing FKs in `Table.create`, https://github.com/simonw/datasette/issues/2059#issuecomment-1506174353,https://api.github.com/repos/simonw/datasette/issues/2059,1506174353,IC_kwDOBm6k_c5ZxmWR,9599,simonw,2023-04-13T01:13:00Z,2023-04-13T01:13:00Z,OWNER,"Can you provide a URL to an example, and/or a screenshot of this? Is it a browser warning or is it a warning from Heroku itself?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1665053646,"""Deceptive site ahead"" alert on Heroku deployment", https://github.com/simonw/datasette/issues/2058#issuecomment-1507264934,https://api.github.com/repos/simonw/datasette/issues/2058,1507264934,IC_kwDOBm6k_c5Z1wmm,1138559,esagara,2023-04-13T16:35:21Z,2023-04-13T16:35:21Z,NONE,"I tried deploying the fix you submitted, but still getting the same errors. I can past the logs here if needed, but I really don't see anything new in them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1506203550,https://api.github.com/repos/simonw/datasette/issues/2058,1506203550,IC_kwDOBm6k_c5Zxtee,547438,cephillips,2023-04-13T01:48:21Z,2023-04-13T01:48:21Z,NONE,Really interesting how you are using ChatGPT in this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504426792,https://api.github.com/repos/simonw/datasette/issues/2058,1504426792,IC_kwDOBm6k_c5Zq7so,9599,simonw,2023-04-12T02:02:42Z,2023-04-12T02:02:42Z,OWNER,"I tightened up the benchmark (it was measuring the time taken to create the tables too) and got this: ![image](https://user-images.githubusercontent.com/9599/231328328-85ca35ac-a11b-46f4-b132-dae367103570.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504328395,https://api.github.com/repos/simonw/datasette/issues/2058,1504328395,IC_kwDOBm6k_c5ZqjrL,9599,simonw,2023-04-12T00:28:38Z,2023-04-12T00:28:38Z,OWNER,"Here's a much better chart, which shows that MD5 performance unsurprisingly gets worse as the number of tables increases while `schema_version` remains constant: ![image](https://user-images.githubusercontent.com/9599/231316778-513bd99f-5ea4-495c-b86d-c572a7106369.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504315697,https://api.github.com/repos/simonw/datasette/issues/2058,1504315697,IC_kwDOBm6k_c5Zqgkx,9599,simonw,2023-04-12T00:16:22Z,2023-04-12T00:27:12Z,OWNER,"I got ChatGPT (code execution alpha) to run a micro-benchmark for me. This was the conclusion: > The benchmark using `PRAGMA schema_version` is approximately 1.36 times faster than the benchmark using `hashlib.md5` for the case with 100 tables. For the case with 200 tables, the benchmark using `PRAGMA schema_version` is approximately 2.33 times faster than the benchmark using `hashlib.md5`. Here's the chart it drew me: ![image](https://user-images.githubusercontent.com/9599/231315366-3a12b6d3-08d7-419d-a1fd-36eb24da0d85.png) (It's a pretty rubbish chart though, it only took measurements at 100 and 200 and drew a line between the two, I should have told it to measure every 10 and plot that) And the full transcript: https://gist.github.com/simonw/2fc46effbfbe49e6de0bcfdc9e31b235 The benchmark looks good enough on first glance that I don't feel the need to be more thorough with it. `PRAGMA schema_version` is faster, but not so fast that I feel like the MD5 hack is worth worrying about too much. I'm tempted to add something to the `/-/versions` page that tries to identify if this is a problem or not though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504298448,https://api.github.com/repos/simonw/datasette/issues/2058,1504298448,IC_kwDOBm6k_c5ZqcXQ,9599,simonw,2023-04-12T00:04:01Z,2023-04-12T00:04:01Z,OWNER,"Here's a potential workaround: when I store the schema versions, I could also score an MD5 hash of the full schema (`select group_concat(sql) from sqlite_master`). When I read the schema version with `PRAGMA schema_version` I could catch that exception and, if I see it, I could calculate that MD5 hash again as a fallback and use that to determine if the schema has changed instead. The performance overhead of this needs investigating - how much more expensive is `md5(... that SQL query result)` compared to just `PRAGMA schema_version`, especially on a database with a lot of tables?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504295345,https://api.github.com/repos/simonw/datasette/issues/2058,1504295345,IC_kwDOBm6k_c5Zqbmx,9599,simonw,2023-04-12T00:01:42Z,2023-04-12T00:02:26Z,OWNER,"Here's the relevant code: https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/app.py#L421-L437 This function is called on almost every request (everything that subclasses `BaseView` at least - need to remember that for the refactor in #2053 etc). https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/datasette/views/base.py#L101-L103 It uses `PRAGMA schema_version` as a cheap way to determine if the schema has changed, in which case it needs to refresh the internal schema tables. This was already the cause of a subtle bug here: - #1231 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504292145,https://api.github.com/repos/simonw/datasette/issues/2058,1504292145,IC_kwDOBm6k_c5Zqa0x,9599,simonw,2023-04-11T23:58:59Z,2023-04-11T23:58:59Z,OWNER,Asked on the SQLite Forum if anyone has seen this before: https://sqlite.org/forum/forumpost/793a2ed75b,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2058#issuecomment-1504291892,https://api.github.com/repos/simonw/datasette/issues/2058,1504291892,IC_kwDOBm6k_c5Zqaw0,9599,simonw,2023-04-11T23:58:45Z,2023-04-11T23:58:45Z,OWNER,"I thought it might relate to the ""defensive mode"" issue described here: - https://github.com/simonw/sqlite-utils/issues/235 But I have since determined that the Datasette official Docker image does NOT run anything in defensive mode, so I don't think it's related to that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1663399821,"500 ""attempt to write a readonly database"" error caused by ""PRAGMA schema_version""", https://github.com/simonw/datasette/issues/2057#issuecomment-1730363182,https://api.github.com/repos/simonw/datasette/issues/2057,1730363182,IC_kwDOBm6k_c5nIz8u,9599,simonw,2023-09-21T22:09:10Z,2023-09-21T22:09:10Z,OWNER,Tests all pass now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730356422,https://api.github.com/repos/simonw/datasette/issues/2057,1730356422,IC_kwDOBm6k_c5nIyTG,9599,simonw,2023-09-21T22:01:00Z,2023-09-21T22:01:00Z,OWNER,Tested that locally with Python 3.9 from `pyenv` and it worked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730353462,https://api.github.com/repos/simonw/datasette/issues/2057,1730353462,IC_kwDOBm6k_c5nIxk2,9599,simonw,2023-09-21T21:57:17Z,2023-09-21T21:57:17Z,OWNER,"Still fails in Python 3.9: https://github.com/simonw/datasette/actions/runs/6266752548/job/17018363302 ``` plugin_info[""name""] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' Test failed: datasette-json-html should not have been loaded ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730250337,https://api.github.com/repos/simonw/datasette/issues/2057,1730250337,IC_kwDOBm6k_c5nIYZh,9599,simonw,2023-09-21T20:26:12Z,2023-09-21T20:26:12Z,OWNER,That does seem to fix the problem! ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730247545,https://api.github.com/repos/simonw/datasette/issues/2057,1730247545,IC_kwDOBm6k_c5nIXt5,9599,simonw,2023-09-21T20:23:47Z,2023-09-21T20:23:47Z,OWNER,Hunch: https://pypi.org/project/importlib-metadata/ may help here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730226107,https://api.github.com/repos/simonw/datasette/issues/2057,1730226107,IC_kwDOBm6k_c5nISe7,9599,simonw,2023-09-21T20:06:19Z,2023-09-21T20:06:19Z,OWNER,"No that's not it actually, it's something else. Got to this point: ```bash DATASETTE_LOAD_PLUGINS=datasette-init python -i $(which datasette) plugins ``` That fails and drops me into a debugger: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 186, in plugins app = Datasette([], plugins_dir=plugins_dir) File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 405, in __init__ for plugin in get_plugins() File ""/Users/simon/Dropbox/Development/datasette/datasette/plugins.py"", line 89, in get_plugins plugin_info[""name""] = distinfo.name or distinfo.project_name AttributeError: 'PathDistribution' object has no attribute 'name' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730219703,https://api.github.com/repos/simonw/datasette/issues/2057,1730219703,IC_kwDOBm6k_c5nIQ63,9599,simonw,2023-09-21T20:01:54Z,2023-09-21T20:01:54Z,OWNER,"The problem is here: ``` 86 distinfo = plugin_to_distinfo.get(plugin) 87 if distinfo is None: 88 breakpoint() 89 -> assert False 90 if distinfo.name is None: 91 breakpoint() 92 assert False 93 if distinfo: 94 plugin_info[""version""] = distinfo.version (Pdb) distinfo (Pdb) plugin ``` That `plugin_to_distinfo` is missing some stuff.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730214654,https://api.github.com/repos/simonw/datasette/issues/2057,1730214654,IC_kwDOBm6k_c5nIPr-,9599,simonw,2023-09-21T19:59:51Z,2023-09-21T19:59:51Z,OWNER,"So the problem is the `get_plugins()` function returning plugins with `None` for their name: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/plugins.py#L61-L91","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730212597,https://api.github.com/repos/simonw/datasette/issues/2057,1730212597,IC_kwDOBm6k_c5nIPL1,9599,simonw,2023-09-21T19:58:38Z,2023-09-21T19:58:38Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/80a9cd9620fddf2695d12d8386a91e7c6b145ef2/datasette/app.py#L1127-L1146,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730211445,https://api.github.com/repos/simonw/datasette/issues/2057,1730211445,IC_kwDOBm6k_c5nIO51,9599,simonw,2023-09-21T19:57:44Z,2023-09-21T19:57:44Z,OWNER,"In the debugger: ``` >>> import pdb >>> pdb.pm() > /Users/simon/Dropbox/Development/datasette/datasette/app.py(1136)_plugins() -> ps.sort(key=lambda p: p[""name""]) (Pdb) ps [{'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['prepare_connection', 'render_cell'], 'version': '1.0.1'}, {'name': None, 'static_path': None, 'templates_path': None, 'hooks': ['startup'], 'version': '0.2'}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730210728,https://api.github.com/repos/simonw/datasette/issues/2057,1730210728,IC_kwDOBm6k_c5nIOuo,9599,simonw,2023-09-21T19:57:08Z,2023-09-21T19:57:08Z,OWNER,"In my Python 3.8 environment I ran: ```bash datasette install datasette-init datasette-json-html ``` And now `datasette plugins` produces this error: ``` File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 192, in plugins click.echo(json.dumps(app._plugins(all=all), indent=4)) File ""/Users/simon/Dropbox/Development/datasette/datasette/app.py"", line 1136, in _plugins ps.sort(key=lambda p: p[""name""]) TypeError: '<' not supported between instances of 'NoneType' and 'NoneType' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730201226,https://api.github.com/repos/simonw/datasette/issues/2057,1730201226,IC_kwDOBm6k_c5nIMaK,9599,simonw,2023-09-21T19:49:20Z,2023-09-21T19:49:20Z,OWNER,"That passed on 3.8 but should have failed: https://github.com/simonw/datasette/actions/runs/6266341481/job/17017099801 - the ""Test DATASETTE_LOAD_PLUGINS"" test shows errors but did not fail the CI run.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730188367,https://api.github.com/repos/simonw/datasette/issues/2057,1730188367,IC_kwDOBm6k_c5nIJRP,9599,simonw,2023-09-21T19:38:28Z,2023-09-21T19:40:38Z,OWNER,"I'll imitate `certbot`: https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot-ci/setup.py#L9 ```python 'importlib_resources>=1.3.1; python_version < ""3.9""', ``` Looks like `1.3` is the minimum version needed for compatibility with the 3.9 standard library, according to https://github.com/python/importlib_resources/blob/main/README.rst#compatibility https://github.com/certbot/certbot/blob/694c758db7fcd8410b5dadcd136c61b3eb028fdc/certbot/certbot/_internal/constants.py#L13C29-L16 ```python if sys.version_info >= (3, 9): # pragma: no cover import importlib.resources as importlib_resources else: # pragma: no cover import importlib_resources ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730185322,https://api.github.com/repos/simonw/datasette/issues/2057,1730185322,IC_kwDOBm6k_c5nIIhq,9599,simonw,2023-09-21T19:35:49Z,2023-09-21T19:35:49Z,OWNER,I think I can fix this using https://importlib-resources.readthedocs.io/en/latest/using.html - maybe as a dependency only installed if the Python version is less than 3.9.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730183405,https://api.github.com/repos/simonw/datasette/issues/2057,1730183405,IC_kwDOBm6k_c5nIIDt,9599,simonw,2023-09-21T19:34:09Z,2023-09-21T19:34:09Z,OWNER,"Confirmed: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files > `importlib.resources.files(package)` > [...] > New in version 3.9.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1730171241,https://api.github.com/repos/simonw/datasette/issues/2057,1730171241,IC_kwDOBm6k_c5nIFFp,9599,simonw,2023-09-21T19:27:25Z,2023-09-21T19:27:25Z,OWNER,"This broke in Python 3.8: ``` if plugin.__name__ not in DEFAULT_PLUGINS: try: if (importlib.resources.files(plugin.__name__) / ""static"").is_dir(): E AttributeError: module 'importlib.resources' has no attribute 'files' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1722323967,https://api.github.com/repos/simonw/datasette/issues/2057,1722323967,IC_kwDOBm6k_c5mqJP_,9599,simonw,2023-09-16T21:54:33Z,2023-09-16T21:54:33Z,OWNER,Just found this migration guide: https://importlib-metadata.readthedocs.io/en/latest/migration.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1722266942,https://api.github.com/repos/simonw/datasette/issues/2057,1722266942,IC_kwDOBm6k_c5mp7U-,9599,simonw,2023-09-16T16:38:27Z,2023-09-16T16:38:27Z,OWNER,"The `importlib.metadata.entry_points()` function is pretty interesting: ```pycon >>> import importlib.metadata >>> from pprint import pprint >>> pprint(importlib.metadata.entry_points()) {'babel.checkers': [EntryPoint(name='num_plurals', value='babel.messages.checkers:num_plurals', group='babel.checkers'), EntryPoint(name='python_format', value='babel.messages.checkers:python_format', group='babel.checkers')], 'babel.extractors': [EntryPoint(name='jinja2', value='jinja2.ext:babel_extract[i18n]', group='babel.extractors'), EntryPoint(name='ignore', value='babel.messages.extract:extract_nothing', group='babel.extractors'), EntryPoint(name='javascript', value='babel.messages.extract:extract_javascript', group='babel.extractors'), EntryPoint(name='python', value='babel.messages.extract:extract_python', group='babel.extractors')], 'console_scripts': [EntryPoint(name='datasette', value='datasette.cli:cli', group='console_scripts'), EntryPoint(name='normalizer', value='charset_normalizer.cli.normalizer:cli_detect', group='console_scripts'), EntryPoint(name='pypprint', value='pprintpp:console', group='console_scripts'), EntryPoint(name='cog', value='cogapp:main', group='console_scripts'), EntryPoint(name='icdiff', value='icdiff:start', group='console_scripts'), EntryPoint(name='pycodestyle', value='pycodestyle:_main', group='console_scripts'), EntryPoint(name='sphinx-autobuild', value='sphinx_autobuild.__main__:main', group='console_scripts'), EntryPoint(name='sphinx-apidoc', value='sphinx.ext.apidoc:main', group='console_scripts'), EntryPoint(name='sphinx-autogen', value='sphinx.ext.autosummary.generate:main', group='console_scripts'), EntryPoint(name='sphinx-build', value='sphinx.cmd.build:main', group='console_scripts'), EntryPoint(name='sphinx-quickstart', value='sphinx.cmd.quickstart:main', group='console_scripts'), EntryPoint(name='sphinx-to-sqlite', value='sphinx_to_sqlite.cli:cli', group='console_scripts'), EntryPoint(name='pybabel', value='babel.messages.frontend:main', group='console_scripts'), EntryPoint(name='docutils', value='docutils.__main__:main', group='console_scripts'), EntryPoint(name='isort', value='isort.main:main', group='console_scripts'), EntryPoint(name='isort-identify-imports', value='isort.main:identify_imports_main', group='console_scripts'), EntryPoint(name='hupper', value='hupper.cli:main', group='console_scripts'), EntryPoint(name='sqlite-utils', value='sqlite_utils.cli:cli', group='console_scripts'), EntryPoint(name='py.test', value='pytest:console_main', group='console_scripts'), EntryPoint(name='pytest', value='pytest:console_main', group='console_scripts'), EntryPoint(name='pyflakes', value='pyflakes.api:main', group='console_scripts'), EntryPoint(name='livereload', value='livereload.cli:main', group='console_scripts'), EntryPoint(name='uvicorn', value='uvicorn.main:main', group='console_scripts'), EntryPoint(name='httpx', value='httpx:main', group='console_scripts'), EntryPoint(name='flake8', value='flake8.main.cli:main', group='console_scripts'), EntryPoint(name='blacken-docs', value='blacken_docs:main', group='console_scripts'), EntryPoint(name='pip', value='pip._internal.cli.main:main', group='console_scripts'), EntryPoint(name='pip3', value='pip._internal.cli.main:main', group='console_scripts'), EntryPoint(name='pip3.10', value='pip._internal.cli.main:main', group='console_scripts'), EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts'), EntryPoint(name='pygmentize', value='pygments.cmdline:main', group='console_scripts'), EntryPoint(name='black', value='black:patched_main', group='console_scripts'), EntryPoint(name='blackd', value='blackd:patched_main [d]', group='console_scripts'), EntryPoint(name='codespell', value='codespell_lib:_script_main', group='console_scripts'), EntryPoint(name='tabulate', value='tabulate:_main', group='console_scripts')], 'datasette': [EntryPoint(name='debug_permissions', value='datasette_debug_permissions', group='datasette'), EntryPoint(name='codespaces', value='datasette_codespaces', group='datasette'), EntryPoint(name='vega', value='datasette_vega', group='datasette'), EntryPoint(name='x_forwarded_host', value='datasette_x_forwarded_host', group='datasette'), EntryPoint(name='json_html', value='datasette_json_html', group='datasette'), EntryPoint(name='datasette_write_ui', value='datasette_write_ui', group='datasette'), EntryPoint(name='pretty_json', value='datasette_pretty_json', group='datasette'), EntryPoint(name='graphql', value='datasette_graphql', group='datasette')], 'distutils.commands': [EntryPoint(name='compile_catalog', value='babel.messages.frontend:compile_catalog', group='distutils.commands'), EntryPoint(name='extract_messages', value='babel.messages.frontend:extract_messages', group='distutils.commands'), EntryPoint(name='init_catalog', value='babel.messages.frontend:init_catalog', group='distutils.commands'), EntryPoint(name='update_catalog', value='babel.messages.frontend:update_catalog', group='distutils.commands'), EntryPoint(name='isort', value='isort.setuptools_commands:ISortCommand', group='distutils.commands'), EntryPoint(name='alias', value='setuptools.command.alias:alias', group='distutils.commands'), EntryPoint(name='bdist_egg', value='setuptools.command.bdist_egg:bdist_egg', group='distutils.commands'), EntryPoint(name='bdist_rpm', value='setuptools.command.bdist_rpm:bdist_rpm', group='distutils.commands'), EntryPoint(name='build', value='setuptools.command.build:build', group='distutils.commands'), EntryPoint(name='build_clib', value='setuptools.command.build_clib:build_clib', group='distutils.commands'), EntryPoint(name='build_ext', value='setuptools.command.build_ext:build_ext', group='distutils.commands'), EntryPoint(name='build_py', value='setuptools.command.build_py:build_py', group='distutils.commands'), EntryPoint(name='develop', value='setuptools.command.develop:develop', group='distutils.commands'), EntryPoint(name='dist_info', value='setuptools.command.dist_info:dist_info', group='distutils.commands'), EntryPoint(name='easy_install', value='setuptools.command.easy_install:easy_install', group='distutils.commands'), EntryPoint(name='editable_wheel', value='setuptools.command.editable_wheel:editable_wheel', group='distutils.commands'), EntryPoint(name='egg_info', value='setuptools.command.egg_info:egg_info', group='distutils.commands'), EntryPoint(name='install', value='setuptools.command.install:install', group='distutils.commands'), EntryPoint(name='install_egg_info', value='setuptools.command.install_egg_info:install_egg_info', group='distutils.commands'), EntryPoint(name='install_lib', value='setuptools.command.install_lib:install_lib', group='distutils.commands'), EntryPoint(name='install_scripts', value='setuptools.command.install_scripts:install_scripts', group='distutils.commands'), EntryPoint(name='rotate', value='setuptools.command.rotate:rotate', group='distutils.commands'), EntryPoint(name='saveopts', value='setuptools.command.saveopts:saveopts', group='distutils.commands'), EntryPoint(name='sdist', value='setuptools.command.sdist:sdist', group='distutils.commands'), EntryPoint(name='setopt', value='setuptools.command.setopt:setopt', group='distutils.commands'), EntryPoint(name='test', value='setuptools.command.test:test', group='distutils.commands'), EntryPoint(name='upload_docs', value='setuptools.command.upload_docs:upload_docs', group='distutils.commands'), EntryPoint(name='bdist_wheel', value='wheel.bdist_wheel:bdist_wheel', group='distutils.commands')], 'distutils.setup_keywords': [EntryPoint(name='message_extractors', value='babel.messages.frontend:check_message_extractors', group='distutils.setup_keywords'), EntryPoint(name='cffi_modules', value='cffi.setuptools_ext:cffi_modules', group='distutils.setup_keywords'), EntryPoint(name='dependency_links', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'), EntryPoint(name='eager_resources', value='setuptools.dist:assert_string_list', group='distutils.setup_keywords'), EntryPoint(name='entry_points', value='setuptools.dist:check_entry_points', group='distutils.setup_keywords'), EntryPoint(name='exclude_package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'), EntryPoint(name='extras_require', value='setuptools.dist:check_extras', group='distutils.setup_keywords'), EntryPoint(name='include_package_data', value='setuptools.dist:assert_bool', group='distutils.setup_keywords'), EntryPoint(name='install_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'), EntryPoint(name='namespace_packages', value='setuptools.dist:check_nsp', group='distutils.setup_keywords'), EntryPoint(name='package_data', value='setuptools.dist:check_package_data', group='distutils.setup_keywords'), EntryPoint(name='packages', value='setuptools.dist:check_packages', group='distutils.setup_keywords'), EntryPoint(name='python_requires', value='setuptools.dist:check_specifier', group='distutils.setup_keywords'), EntryPoint(name='setup_requires', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'), EntryPoint(name='test_loader', value='setuptools.dist:check_importable', group='distutils.setup_keywords'), EntryPoint(name='test_runner', value='setuptools.dist:check_importable', group='distutils.setup_keywords'), EntryPoint(name='test_suite', value='setuptools.dist:check_test_suite', group='distutils.setup_keywords'), EntryPoint(name='tests_require', value='setuptools.dist:check_requirements', group='distutils.setup_keywords'), EntryPoint(name='use_2to3', value='setuptools.dist:invalid_unless_false', group='distutils.setup_keywords'), EntryPoint(name='zip_safe', value='setuptools.dist:assert_bool', group='distutils.setup_keywords')], 'egg_info.writers': [EntryPoint(name='PKG-INFO', value='setuptools.command.egg_info:write_pkg_info', group='egg_info.writers'), EntryPoint(name='dependency_links.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'), EntryPoint(name='depends.txt', value='setuptools.command.egg_info:warn_depends_obsolete', group='egg_info.writers'), EntryPoint(name='eager_resources.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'), EntryPoint(name='entry_points.txt', value='setuptools.command.egg_info:write_entries', group='egg_info.writers'), EntryPoint(name='namespace_packages.txt', value='setuptools.command.egg_info:overwrite_arg', group='egg_info.writers'), EntryPoint(name='requires.txt', value='setuptools.command.egg_info:write_requirements', group='egg_info.writers'), EntryPoint(name='top_level.txt', value='setuptools.command.egg_info:write_toplevel_names', group='egg_info.writers')], 'flake8.extension': [EntryPoint(name='C90', value='mccabe:McCabeChecker', group='flake8.extension'), EntryPoint(name='E', value='flake8.plugins.pycodestyle:pycodestyle_logical', group='flake8.extension'), EntryPoint(name='F', value='flake8.plugins.pyflakes:FlakesChecker', group='flake8.extension'), EntryPoint(name='W', value='flake8.plugins.pycodestyle:pycodestyle_physical', group='flake8.extension')], 'flake8.report': [EntryPoint(name='default', value='flake8.formatting.default:Default', group='flake8.report'), EntryPoint(name='pylint', value='flake8.formatting.default:Pylint', group='flake8.report'), EntryPoint(name='quiet-filename', value='flake8.formatting.default:FilenameOnly', group='flake8.report'), EntryPoint(name='quiet-nothing', value='flake8.formatting.default:Nothing', group='flake8.report')], 'pylama.linter': [EntryPoint(name='isort', value='isort.pylama_isort:Linter', group='pylama.linter')], 'pytest11': [EntryPoint(name='icdiff', value='pytest_icdiff', group='pytest11'), EntryPoint(name='asyncio', value='pytest_asyncio.plugin', group='pytest11'), EntryPoint(name='xdist', value='xdist.plugin', group='pytest11'), EntryPoint(name='xdist.looponfail', value='xdist.looponfail', group='pytest11'), EntryPoint(name='timeout', value='pytest_timeout', group='pytest11'), EntryPoint(name='anyio', value='anyio.pytest_plugin', group='pytest11')], 'setuptools.finalize_distribution_options': [EntryPoint(name='keywords', value='setuptools.dist:Distribution._finalize_setup_keywords', group='setuptools.finalize_distribution_options'), EntryPoint(name='parent_finalize', value='setuptools.dist:_Distribution.finalize_options', group='setuptools.finalize_distribution_options')], 'sphinx.html_themes': [EntryPoint(name='alabaster', value='alabaster', group='sphinx.html_themes'), EntryPoint(name='basic-ng', value='sphinx_basic_ng', group='sphinx.html_themes'), EntryPoint(name='furo', value='furo', group='sphinx.html_themes')], 'sqlite_utils': [EntryPoint(name='hello_world', value='sqlite_utils_hello_world', group='sqlite_utils')]} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1722266513,https://api.github.com/repos/simonw/datasette/issues/2057,1722266513,IC_kwDOBm6k_c5mp7OR,9599,simonw,2023-09-16T16:36:09Z,2023-09-16T16:36:09Z,OWNER,"Now I need to switch out `pkg_resources` in `plugins.py`: https://github.com/simonw/datasette/blob/852f5014853943fa27f43ddaa2d442545b3259fb/datasette/plugins.py#L33-L74","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1722265848,https://api.github.com/repos/simonw/datasette/issues/2057,1722265848,IC_kwDOBm6k_c5mp7D4,9599,simonw,2023-09-16T16:32:42Z,2023-09-16T16:32:42Z,OWNER,"Here's the exception it uses: ```pycon >>> importlib.metadata.version(""datasette"") '1.0a6' >>> importlib.metadata.version(""datasette2"") Traceback (most recent call last): File """", line 1, in File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 996, in version return distribution(distribution_name).version File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 969, in distribution return Distribution.from_name(distribution_name) File ""/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/importlib/metadata/__init__.py"", line 548, in from_name raise PackageNotFoundError(name) importlib.metadata.PackageNotFoundError: No package metadata was found for datasette2 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1722258980,https://api.github.com/repos/simonw/datasette/issues/2057,1722258980,IC_kwDOBm6k_c5mp5Yk,9599,simonw,2023-09-16T15:56:45Z,2023-09-16T15:56:45Z,OWNER,"Weird, I still can't get the warning to show even with this: ```python @pytest.mark.asyncio async def test_plugin_is_installed(): datasette = Datasette(memory=True) class DummyPlugin: __name__ = ""DummyPlugin"" @hookimpl def actors_from_ids(self, datasette, actor_ids): return {} try: pm.register(DummyPlugin(), name=""DummyPlugin"") response = await datasette.client.get(""/-/plugins.json"") assert response.status_code == 200 installed_plugins = {p[""name""] for p in response.json()} assert ""DummyPlugin"" in installed_plugins finally: pm.unregister(name=""ReturnNothingPlugin"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1722257328,https://api.github.com/repos/simonw/datasette/issues/2057,1722257328,IC_kwDOBm6k_c5mp4-w,9599,simonw,2023-09-16T15:47:32Z,2023-09-16T15:47:32Z,OWNER,"Frustrating that this warning doesn't show up in the Datasette test suite itself. It shows up in plugin test suites that run this test: ```python @pytest.mark.asyncio async def test_plugin_is_installed(): datasette = Datasette(memory=True) response = await datasette.client.get(""/-/plugins.json"") assert response.status_code == 200 installed_plugins = {p[""name""] for p in response.json()} assert ""datasette-chronicle"" in installed_plugins ``` If you run that test inside Datasette core `installed_plugins` is an empty set, which presumably is why the warning doesn't get triggered there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1503838640,https://api.github.com/repos/simonw/datasette/issues/2057,1503838640,IC_kwDOBm6k_c5ZosGw,9599,simonw,2023-04-11T17:48:23Z,2023-04-11T17:48:23Z,OWNER,"> This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory. I was right about that: ```pycon >>> importlib.resources.files('datasette_graphql') PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql') >>> importlib.resources.files('datasette_graphql').joinpath(""static"") PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static') >>> p = importlib.resources.files('datasette_graphql').joinpath(""static"") >>> p PosixPath('/Users/simon/.local/share/virtualenvs/datasette-big-local-6Yn-280V/lib/python3.11/site-packages/datasette_graphql/static') >>> p.is_ p.is_absolute() p.is_char_device() p.is_fifo() p.is_mount() p.is_reserved() p.is_symlink() p.is_block_device() p.is_dir() p.is_file() p.is_relative_to( p.is_socket() >>> p.is_dir() True >>> p.is_file() False ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1503833906,https://api.github.com/repos/simonw/datasette/issues/2057,1503833906,IC_kwDOBm6k_c5Zoq8y,9599,simonw,2023-04-11T17:44:16Z,2023-04-11T17:45:45Z,OWNER,"Another prompt: > How to fix this: > > `pkg_resources.get_distribution(package).version` Response: ```python import importlib.metadata # Get the version number of the specified package package_version = importlib.metadata.version(package) ``` That seems to work: ```pycon >>> import importlib.metadata >>> importlib.metadata.version(""datasette"") '0.64.2' >>> importlib.metadata.version(""pluggy"") '1.0.0' >>> importlib.metadata.version(""not-a-package"") ... importlib.metadata.PackageNotFoundError: No package metadata was found for not-a-package ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/issues/2057#issuecomment-1503832422,https://api.github.com/repos/simonw/datasette/issues/2057,1503832422,IC_kwDOBm6k_c5Zoqlm,9599,simonw,2023-04-11T17:42:57Z,2023-04-11T17:46:42Z,OWNER,"I ran this prompt against ChatGPT with the Browsing alpha: > ```python > if pkg_resources.resource_isdir(plugin.__name__, ""static""): > static_path = pkg_resources.resource_filename( > plugin.__name__, ""static"" > ) > if pkg_resources.resource_isdir(plugin.__name__, ""templates""): > templates_path = pkg_resources.resource_filename( > plugin.__name__, ""templates"" > ) > ``` > This code gives a deprecation warning in Python 3.11 - fix it It looked up the fix for me: And suggested: ```python import importlib.resources # Replace pkg_resources.resource_isdir with importlib.resources.files().is_file() if importlib.resources.files(plugin.__name__).joinpath(""static"").is_file(): static_path = importlib.resources.as_file( importlib.resources.files(plugin.__name__).joinpath(""static"") ) if importlib.resources.files(plugin.__name__).joinpath(""templates"").is_file(): templates_path = importlib.resources.as_file( importlib.resources.files(plugin.__name__).joinpath(""templates"") ) ``` This looks wrong to me - I would expect something like `is_directory()` not `is_file()` for telling if `static/` is a directory.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1662951875,DeprecationWarning: pkg_resources is deprecated as an API, https://github.com/simonw/datasette/pull/2056#issuecomment-1509852821,https://api.github.com/repos/simonw/datasette/issues/2056,1509852821,IC_kwDOBm6k_c5Z_oaV,3709715,cclauss,2023-04-15T14:24:45Z,2023-04-15T14:24:45Z,NONE,Status?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506485287,https://api.github.com/repos/simonw/datasette/issues/2056,1506485287,IC_kwDOBm6k_c5ZyyQn,3709715,cclauss,2023-04-13T07:29:38Z,2023-04-13T07:41:55Z,NONE,"Ruff (written in Rust, not Python) is a 23MB executable so the time to download and pip install it dwarfs its runtime. Let's run ruff with and without GitHub Actions pip cache side-by-side to see the relative performance. Once you approve the workflows below, `ruff_with_cache` should echo `cache-hit = false` but if you rerun that job hopefully it should echo `cache-hit = true`. That will be the execution time that we are interested to compare. There are two great problems in computer science: ;-) 1. Naming things 2. Cache invalidation 3. Off-by-one errors For 2., https://github.com/actions/setup-python#caching-packages-dependencies is vital reading. Only _exactly pinned requirements_ can be cached. Currently in `setup.py` the only pinned dependencies are: 1. Sphinx==6.1.3 2. furo==2023.3.27 3. black==23.3.0 4. blacken-docs==1.13.0 # but unpinned elsewhere in `setup.py` This means that there will be very few cache hits in the current actions. See the link below to print out cache hits: https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#cache-hit > PyPI uses Fastly's CDN to quickly serve content to end-users, allowing us to minimize our hosting infrastructure and obscure possible downtime. -- https://pypi.org/sponsors I would be shocked if Fastly does not have beefy CDN nodes in the same datacenters where GitHub Actions run so GHA requests to download `ruff` probably never hit a PyPI server. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506179555,https://api.github.com/repos/simonw/datasette/issues/2056,1506179555,IC_kwDOBm6k_c5Zxnnj,9599,simonw,2023-04-13T01:21:05Z,2023-04-13T01:22:08Z,OWNER,"OK, I'm sold - this is a really neat improvement. One thing to change in the PR: right now it runs `pip install --user ruff` on every commit, which hits PyPI to install the package. I prefer to avoid hitting PyPI every time, so I like to use the GitHub Actions cache. My usual pattern for that looks like this: https://github.com/simonw/datasette/blob/5890a20c374fb0812d88c9b0ef26a838bfa06c76/.github/workflows/test-pyodide.yml#L16-L20 Then a separate command that runs `pip install ...` will benefit from that cache. Are you OK to make that change?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506177857,https://api.github.com/repos/simonw/datasette/issues/2056,1506177857,IC_kwDOBm6k_c5ZxnNB,9599,simonw,2023-04-13T01:18:18Z,2023-04-13T01:18:18Z,OWNER,"Cool - and now https://github.com/simonw/datasette/pull/2056/files is showing me those inline annotations: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506177115,https://api.github.com/repos/simonw/datasette/issues/2056,1506177115,IC_kwDOBm6k_c5ZxnBb,9599,simonw,2023-04-13T01:17:16Z,2023-04-13T01:17:16Z,OWNER,"Here are the failures: https://github.com/simonw/datasette/actions/runs/4684460653/jobs/8300630794?pr=2056 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/datasette/pull/2056#issuecomment-1506175208,https://api.github.com/repos/simonw/datasette/issues/2056,1506175208,IC_kwDOBm6k_c5Zxmjo,9599,simonw,2023-04-13T01:14:13Z,2023-04-13T01:14:13Z,OWNER,https://github.com/simonw/datasette/actions/runs/4664796647/jobs/8300596121?pr=2056 it's pretty fast - that finished in 9s.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1661860507,GitHub Action to lint Python code with ruff, https://github.com/simonw/sqlite-utils/pull/536#issuecomment-1500893216,https://api.github.com/repos/simonw/sqlite-utils/issues/536,1500893216,IC_kwDOCGYnMM5ZddAg,22429695,codecov[bot],2023-04-08T13:35:42Z,2023-04-08T13:35:42Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch and project coverage have no change. > Comparison is base [(`c0251cc`)](https://codecov.io/gh/simonw/sqlite-utils/commit/c0251cc9271260de73b4227859a51fab9b4cb745?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25% compared to head [(`cea05dc`)](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 96.25%.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #536 +/- ## ======================================= Coverage 96.25% 96.25% ======================================= Files 6 6 Lines 2671 2671 ======================================= Hits 2571 2571 Misses 100 100 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/utils.py](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL3V0aWxzLnB5) | `95.13% <ø> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/sqlite-utils/pull/536?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1659525418,Add paths for homebrew on Apple silicon, https://github.com/simonw/datasette/issues/2054#issuecomment-1500608101,https://api.github.com/repos/simonw/datasette/issues/2054,1500608101,IC_kwDOBm6k_c5ZcXZl,9599,simonw,2023-04-07T20:14:38Z,2023-04-07T20:14:38Z,OWNER,"Ooh that one's really interesting - very different from the others: ```ruby # app.rb require ""roda"" class App < Roda route do |r| r.root do ""Home page"" end r.on ""pages"" do r.get "":slug"" do |slug| ""Page: #{slug}"" end end r.on ""news"" do r.get "":yyyy/:mm/:dd"" do |yyyy, mm, dd| ""News for #{yyyy}/#{mm}/#{dd}"" end end end end # config.ru require_relative ""app"" run App.freeze.app ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499797384,https://api.github.com/repos/simonw/datasette/issues/2054,1499797384,IC_kwDOBm6k_c5ZZReI,6213,dsisnero,2023-04-07T00:46:50Z,2023-04-07T00:46:50Z,NONE,you should have a look at Roda written in ruby . ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499604822,https://api.github.com/repos/simonw/datasette/issues/2054,1499604822,IC_kwDOBm6k_c5ZYidW,9599,simonw,2023-04-06T20:48:19Z,2023-04-06T20:48:55Z,OWNER,"I actually quite like that. I could even use `@classmethod` and have utility methods defined on that class that both `get()` and `post()` could call. The crucial rule here is NO INSTANCE STATE - that's what makes routing to classes particularly damaging, and encourages code that's hard to maintain.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499604066,https://api.github.com/repos/simonw/datasette/issues/2054,1499604066,IC_kwDOBm6k_c5ZYiRi,9599,simonw,2023-04-06T20:47:30Z,2023-04-06T20:47:30Z,OWNER,"I'm contemplating a new approach: using a class with static methods. Something like this: ```python class TableView(MethodRouter): @staticmethod async def get(request): return Response.text(""GET"") @staticmethod async def post(request): return Response.text(""POST"") ``` So effectively the class is just there to bundle together verb implementations, and to provide a `route(request)` method which knows how to dispatch them to the right place. It can offer default HEAD and OPTIONS methods too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499596941,https://api.github.com/repos/simonw/datasette/issues/2054,1499596941,IC_kwDOBm6k_c5ZYgiN,9599,simonw,2023-04-06T20:39:35Z,2023-04-06T20:39:35Z,OWNER,"I think there are two main approaches: - Map path to a function, that function then behaves differently on GET/POST/etc - Map method AND path to a function - the `app.get('/example')` v.s. `app.post('/example')` ones The decorator approach is calling out to me a bit now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499530921,https://api.github.com/repos/simonw/datasette/issues/2054,1499530921,IC_kwDOBm6k_c5ZYQap,9599,simonw,2023-04-06T19:41:52Z,2023-04-06T19:42:09Z,OWNER,Turned this into a TIL: https://til.simonwillison.net/gpt3/gpt4-api-design,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499491258,https://api.github.com/repos/simonw/datasette/issues/2054,1499491258,IC_kwDOBm6k_c5ZYGu6,9599,simonw,2023-04-06T19:00:16Z,2023-04-06T20:40:16Z,OWNER,"I asked GPT-4 to compare URL routing based on path and verb approaches in different languages and frameworks and spit out some examples. Full transcript here: https://gist.github.com/simonw/fa2379b97420404a81b0fcdb4db79657 Initial prompt: > Show me code examples of different web frameworks in Python and JavaScript and Go illustrating how HTTP routing works - in particular the problem of mapping an incoming HTTP request to some code based on both the URL path and the HTTP verb > > Don't show installation instructions, just get straight to the code snippets Flask: ```python from flask import Flask app = Flask(__name__) @app.route('/example', methods=['GET']) def get_example(): return ""This is a GET request."" @app.route('/example', methods=['POST']) def post_example(): return ""This is a POST request."" if __name__ == '__main__': app.run() ``` Express: ```javascript const express = require('express'); const app = express(); app.get('/example', (req, res) => { res.send('This is a GET request.'); }); app.post('/example', (req, res) => { res.send('This is a POST request.'); }); app.listen(3000, () => { console.log('Server is running on port 3000'); }); ``` Gin: ```go package main import ( ""github.com/gin-gonic/gin"" ) func main() { r := gin.Default() r.GET(""/example"", func(c *gin.Context) { c.String(200, ""This is a GET request."") }) r.POST(""/example"", func(c *gin.Context) { c.String(200, ""This is a POST request."") }) r.Run() } ``` Actix-web: ```rust use actix_web::{web, App, HttpResponse, HttpServer, Responder}; async fn get_example() -> impl Responder { HttpResponse::Ok().body(""This is a GET request."") } async fn post_example() -> impl Responder { HttpResponse::Ok().body(""This is a POST request."") } #[actix_web::main] async fn main() -> std::io::Result<()> { HttpServer::new(|| { App::new() .route(""/example"", web::get().to(get_example)) .route(""/example"", web::post().to(post_example)) }) .bind(""127.0.0.1:8080"")? .run() .await } ``` Koa: ```javascript const Koa = require('koa'); const Router = require('koa-router'); const app = new Koa(); const router = new Router(); router.get('/example', (ctx) => { ctx.body = 'This is a GET request.'; }); router.post('/example', (ctx) => { ctx.body = 'This is a POST request.'; }); app.use(router.routes()); app.use(router.allowedMethods()); app.listen(3000, () => { console.log('Server is running on port 3000'); }); ``` Django: ```python from django.http import HttpResponse from django.views import View class ExampleView(View): def get(self, request): return HttpResponse(""This is a GET request."") def post(self, request): return HttpResponse(""This is a POST request."") ``` ```python from django.urls import path from .views import ExampleView urlpatterns = [ path('example/', ExampleView.as_view(), name='example'), ] ``` Rocket: ```rust #![feature(proc_macro_hygiene, decl_macro)] #[macro_use] extern crate rocket; use rocket::response::content; #[get(""/example"")] fn get_example() -> content::Plain<&'static str> { content::Plain(""This is a GET request."") } #[post(""/example"")] fn post_example() -> content::Plain<&'static str> { content::Plain(""This is a POST request."") } #[launch] fn rocket() -> _ { rocket::build().mount(""/"", routes![get_example, post_example]) } ``` Elixir/Phoenix: ```elixir defmodule ExampleAppWeb.Router do use ExampleAppWeb, :router pipeline :api do plug :accepts, [""json""] end scope ""/api"", ExampleAppWeb do pipe_through :api get ""/example"", ExampleController, :get_example post ""/example"", ExampleController, :post_example end end ``` Bottle: ```python from bottle import Bottle, run app = Bottle() @app.route('/example', method='GET') def get_example(): return ""This is a GET request."" @app.route('/example', method='POST') def post_example(): return ""This is a POST request."" if __name__ == '__main__': run(app, host='localhost', port=8080) ``` FastAPI: ```python from fastapi import FastAPI app = FastAPI() @app.get(""/example"") async def get_example(): return {""message"": ""This is a GET request.""} @app.post(""/example"") async def post_example(): return {""message"": ""This is a POST request.""} if __name__ == ""__main__"": import uvicorn uvicorn.run(app, host=""127.0.0.1"", port=8000) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499467703,https://api.github.com/repos/simonw/datasette/issues/2054,1499467703,IC_kwDOBm6k_c5ZYA-3,9599,simonw,2023-04-06T18:37:22Z,2023-04-06T18:57:22Z,OWNER,"I've been hoping to move entirely away from class-based views, but now I'm wondering if that's the right decision given the need to support HTTP verbs. It is cleaner to have a class that has methods for each verb, or to have a single function that can behave differently depending on the verb? Or should I have a mechanism for dispatching to separate functions based on the verbs that isn't implemented in a base class? The trouble with using base classes is that I've already shown that they tempt me to put logic in weird places, which makes code harder to modify later on. The thing here is that the ONLY thing I want to use the base class for is as an HTTP verb switcher - which makes me doubt that I should risk taking on the other temptations of having a base class.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499465648,https://api.github.com/repos/simonw/datasette/issues/2054,1499465648,IC_kwDOBm6k_c5ZYAew,9599,simonw,2023-04-06T18:35:03Z,2023-04-06T18:35:03Z,OWNER,"There are actually five classes that subclass `DataView`: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/row.py#L16 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L34 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L172 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/database.py#L215 https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/table.py#L72 I don't think `DatabaseView` and `DatabaseDownload` should have subclassed that at all, since they don't return a table of data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499462324,https://api.github.com/repos/simonw/datasette/issues/2054,1499462324,IC_kwDOBm6k_c5ZX_q0,9599,simonw,2023-04-06T18:31:56Z,2023-04-06T18:31:56Z,OWNER,"The `DataView` class does a LOT of work - mostly involving CSV responses. https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L160-L544 It has a `redirect()` method with some complex logic and CORS handling: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L163-L172 It uses this method a lot, which has to be over-ridden in the `TableView` and `RowView` and `QueryView` subclasses: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L174-L175 This method: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L180 Is the bulk of the complexity, because it knows how to both turn a list of SQLite rows into a CSV file but also knows how to call `.data()` repeatedly with different pagination arguments in order to stream CSV back for a large table. The `async def get()` method for GET requests is also very complicated. It mainly handles format stuff - knowing how to render HTML v.s. JSON v.s. CSV v.s. other formats specified using this plugin hook: https://docs.datasette.io/en/1.0a2/plugin_hooks.html#register-output-renderer-datasette Plus it catches interrupted queries and returns a special error page for those (and other error messages too): https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L381-L408 It adds the time taken to execute the queries: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L410-L411","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499457291,https://api.github.com/repos/simonw/datasette/issues/2054,1499457291,IC_kwDOBm6k_c5ZX-cL,9599,simonw,2023-04-06T18:26:45Z,2023-04-06T18:26:45Z,OWNER," Here's `BaseView`: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L56-L145 It has methods for the `options`, `get`, `post`, `delete`, `put`, `patch` and `head` HTTP verbs, most defaulting to returinng a 405 Method not allowed message in plain text or JSON, depending on this check: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L71-L81 Also adds CORS headers to anything if CORS mode is on: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L106-L107 And adds the `database_color` (weirdly) and the `select_templates` variables to the template context: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L112-L122 And has special code for setting the `Link: ...; rel=""alternate""` HTTP header: https://github.com/simonw/datasette/blob/8b9d7fdbd8de7e74414cc29e3005382669a812dc/datasette/views/base.py#L124-L136","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499457201,https://api.github.com/repos/simonw/datasette/issues/2054,1499457201,IC_kwDOBm6k_c5ZX-ax,9599,simonw,2023-04-06T18:26:39Z,2023-04-06T18:26:39Z,OWNER,These classes - `TableView` and `RowView` and `QueryView` - all subclass `DataView` which subclasses` BaseView`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/issues/2054#issuecomment-1499452122,https://api.github.com/repos/simonw/datasette/issues/2054,1499452122,IC_kwDOBm6k_c5ZX9La,9599,simonw,2023-04-06T18:21:51Z,2023-04-06T18:21:51Z,OWNER,"I'm going to make notes against the code in the most recent alpha release, ignoring the recent work I did to refactor `TableView`. https://github.com/simonw/datasette/tree/1.0a2/datasette/views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1657861026,"Make detailed notes on how table, query and row views work right now", https://github.com/simonw/datasette/pull/2053#issuecomment-1652296467,https://api.github.com/repos/simonw/datasette/issues/2053,1652296467,IC_kwDOBm6k_c5ifAsT,9599,simonw,2023-07-26T18:26:44Z,2023-07-26T18:26:44Z,OWNER,"I'm abandoning this branch in favour of a fresh attempt, described here: - https://github.com/simonw/datasette/issues/2109 I'll copy bits and pieces of this branch across as-needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1651904060,https://api.github.com/repos/simonw/datasette/issues/2053,1651904060,IC_kwDOBm6k_c5idg48,9599,simonw,2023-07-26T14:19:00Z,2023-07-26T15:25:15Z,OWNER,"Worth noting that the `register_output_renderer()` is actually pretty easy to extend, because it returns a dictionary which could have more keys (like the required set of extras) added to it: ```python @hookimpl def register_output_renderer(datasette): return { ""extension"": ""test"", ""render"": render_demo, ""can_render"": can_render_demo, # Optional } ``` https://docs.datasette.io/en/0.64.3/plugin_hooks.html#register-output-renderer-datasette One slight hiccup with that plugin hook is this: > rows - list of `sqlite3.Row` objects I could turn that into a Datasette defined object that behaves like a [sqlite3.Row](https://docs.python.org/3/library/sqlite3.html#sqlite3.Row) though, which would give me extra flexibility in the future. A bit tricky though since it's implemented in C for performance: https://github.com/python/cpython/blob/b0202a4e5d6b629ba5acbc703e950f08ebaf07df/Modules/_sqlite/row.c Pasted that into Claude for the following explanation: > - pysqlite_Row is the structure defining the Row object. It contains the tuple of data (self->data) and description of columns (self->description). > - pysqlite_row_new() is the constructor which creates a new Row object given a cursor and tuple of data. > - pysqlite_row_dealloc() frees the memory when Row object is deleted. > - pysqlite_row_keys() returns the column names of the row. > - pysqlite_row_length() and pysqlite_row_subscript() implement sequence like behavior to access row elements by index. > - pysqlite_row_subscript() also allows accessing by column name by doing a lookup in description. > - pysqlite_row_hash() and pysqlite_row_richcompare() implement equality checks and hash function. I could use protocols in Python to make my own `DatasetteRow` which can be used interchangeably with `sqlite3.Row` - https://docs.python.org/3/library/typing.html#typing.Protocol Turned this into a TIL: https://til.simonwillison.net/python/protocols","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1651894668,https://api.github.com/repos/simonw/datasette/issues/2053,1651894668,IC_kwDOBm6k_c5idemM,9599,simonw,2023-07-26T14:14:34Z,2023-07-26T14:14:34Z,OWNER,"Another point of confusion is how `/content` sometimes serves the database index page (with a list of tables) and sometimes solves the results of a query. I could resolve this by turning the information on the index page into extras, which can optionally be requested any time a query is run but default to being shown if there is no query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1651883505,https://api.github.com/repos/simonw/datasette/issues/2053,1651883505,IC_kwDOBm6k_c5idb3x,9599,simonw,2023-07-26T14:08:20Z,2023-07-26T14:08:20Z,OWNER,"I think the hardest part of getting this working is dealing with the different formats. Idea: refactor `.html` as a format (since it's by far the most complex) and tweak the plugin hook a bit as part of that, then use what I learn from that to get the other formats working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1651874649,https://api.github.com/repos/simonw/datasette/issues/2053,1651874649,IC_kwDOBm6k_c5idZtZ,9599,simonw,2023-07-26T14:03:37Z,2023-07-26T14:03:37Z,OWNER,"Big chunk of commented-out code I just removed: ```python import pdb pdb.set_trace() if isinstance(output, dict) and output.get(""ok"") is False: # TODO: Other error codes? response.status_code = 400 if datasette.cors: add_cors_headers(response.headers) return response # registry = Registry( # extra_count, # extra_facet_results, # extra_facets_timed_out, # extra_suggested_facets, # facet_instances, # extra_human_description_en, # extra_next_url, # extra_columns, # extra_primary_keys, # run_display_columns_and_rows, # extra_display_columns, # extra_display_rows, # extra_debug, # extra_request, # extra_query, # extra_metadata, # extra_extras, # extra_database, # extra_table, # extra_database_color, # extra_table_actions, # extra_filters, # extra_renderers, # extra_custom_table_templates, # extra_sorted_facet_results, # extra_table_definition, # extra_view_definition, # extra_is_view, # extra_private, # extra_expandable_columns, # extra_form_hidden_args, # ) results = await registry.resolve_multi( [""extra_{}"".format(extra) for extra in extras] ) data = { ""ok"": True, ""next"": next_value and str(next_value) or None, } data.update( { key.replace(""extra_"", """"): value for key, value in results.items() if key.startswith(""extra_"") and key.replace(""extra_"", """") in extras } ) raw_sqlite_rows = rows[:page_size] data[""rows""] = [dict(r) for r in raw_sqlite_rows] private = False if canned_query: # Respect canned query permissions visible, private = await datasette.check_visibility( request.actor, permissions=[ (""view-query"", (database, canned_query)), (""view-database"", database), ""view-instance"", ], ) if not visible: raise Forbidden(""You do not have permission to view this query"") else: await datasette.ensure_permissions(request.actor, [(""execute-sql"", database)]) # If there's no sql, show the database index page if not sql: return await database_index_view(request, datasette, db) validate_sql_select(sql) # Extract any :named parameters named_parameters = named_parameters or await derive_named_parameters(db, sql) named_parameter_values = { named_parameter: params.get(named_parameter) or """" for named_parameter in named_parameters if not named_parameter.startswith(""_"") } # Set to blank string if missing from params for named_parameter in named_parameters: if named_parameter not in params and not named_parameter.startswith(""_""): params[named_parameter] = """" extra_args = {} if params.get(""_timelimit""): extra_args[""custom_time_limit""] = int(params[""_timelimit""]) if _size: extra_args[""page_size""] = _size templates = [f""query-{to_css_class(database)}.html"", ""query.html""] if canned_query: templates.insert( 0, f""query-{to_css_class(database)}-{to_css_class(canned_query)}.html"", ) query_error = None # Execute query - as write or as read if write: raise NotImplementedError(""Write queries not yet implemented"") # if request.method == ""POST"": # # If database is immutable, return an error # if not db.is_mutable: # raise Forbidden(""Database is immutable"") # body = await request.post_body() # body = body.decode(""utf-8"").strip() # if body.startswith(""{"") and body.endswith(""}""): # params = json.loads(body) # # But we want key=value strings # for key, value in params.items(): # params[key] = str(value) # else: # params = dict(parse_qsl(body, keep_blank_values=True)) # # Should we return JSON? # should_return_json = ( # request.headers.get(""accept"") == ""application/json"" # or request.args.get(""_json"") # or params.get(""_json"") # ) # if canned_query: # params_for_query = MagicParameters(params, request, self.ds) # else: # params_for_query = params # ok = None # try: # cursor = await self.ds.databases[database].execute_write( # sql, params_for_query # ) # message = metadata.get( # ""on_success_message"" # ) or ""Query executed, {} row{} affected"".format( # cursor.rowcount, """" if cursor.rowcount == 1 else ""s"" # ) # message_type = self.ds.INFO # redirect_url = metadata.get(""on_success_redirect"") # ok = True # except Exception as e: # message = metadata.get(""on_error_message"") or str(e) # message_type = self.ds.ERROR # redirect_url = metadata.get(""on_error_redirect"") # ok = False # if should_return_json: # return Response.json( # { # ""ok"": ok, # ""message"": message, # ""redirect"": redirect_url, # } # ) # else: # self.ds.add_message(request, message, message_type) # return self.redirect(request, redirect_url or request.path) # else: # async def extra_template(): # return { # ""request"": request, # ""db_is_immutable"": not db.is_mutable, # ""path_with_added_args"": path_with_added_args, # ""path_with_removed_args"": path_with_removed_args, # ""named_parameter_values"": named_parameter_values, # ""canned_query"": canned_query, # ""success_message"": request.args.get(""_success"") or """", # ""canned_write"": True, # } # return ( # { # ""database"": database, # ""rows"": [], # ""truncated"": False, # ""columns"": [], # ""query"": {""sql"": sql, ""params"": params}, # ""private"": private, # }, # extra_template, # templates, # ) # Not a write rows = [] if canned_query: params_for_query = MagicParameters(params, request, datasette) else: params_for_query = params try: results = await datasette.execute( database, sql, params_for_query, truncate=True, **extra_args ) columns = [r[0] for r in results.description] rows = list(results.rows) except sqlite3.DatabaseError as e: query_error = e results = None columns = [] allow_execute_sql = await datasette.permission_allowed( request.actor, ""execute-sql"", database ) format_ = request.url_vars.get(""format"") or ""html"" if format_ == ""csv"": raise NotImplementedError(""CSV format not yet implemented"") elif format_ in datasette.renderers.keys(): # Dispatch request to the correct output format renderer # (CSV is not handled here due to streaming) result = call_with_supported_arguments( datasette.renderers[format_][0], datasette=datasette, columns=columns, rows=rows, sql=sql, query_name=None, database=db.name, table=None, request=request, view_name=""table"", # TODO: should this be ""query""? # These will be deprecated in Datasette 1.0: args=request.args, data={ ""rows"": rows, }, # TODO what should this be? ) result = await await_me_maybe(result) if result is None: raise NotFound(""No data"") if isinstance(result, dict): r = Response( body=result.get(""body""), status=result.get(""status_code"") or 200, content_type=result.get(""content_type"", ""text/plain""), headers=result.get(""headers""), ) elif isinstance(result, Response): r = result # if status_code is not None: # # Over-ride the status code # r.status = status_code else: assert False, f""{result} should be dict or Response"" elif format_ == ""html"": headers = {} templates = [f""query-{to_css_class(database)}.html"", ""query.html""] template = datasette.jinja_env.select_template(templates) alternate_url_json = datasette.absolute_url( request, datasette.urls.path(path_with_format(request=request, format=""json"")), ) headers.update( { ""Link"": '{}; rel=""alternate""; type=""application/json+datasette""'.format( alternate_url_json ) } ) r = Response.html( await datasette.render_template( template, dict( data, append_querystring=append_querystring, path_with_replaced_args=path_with_replaced_args, fix_path=datasette.urls.path, settings=datasette.settings_dict(), # TODO: review up all of these hacks: alternate_url_json=alternate_url_json, datasette_allow_facet=( ""true"" if datasette.setting(""allow_facet"") else ""false"" ), is_sortable=any(c[""sortable""] for c in data[""display_columns""]), allow_execute_sql=await datasette.permission_allowed( request.actor, ""execute-sql"", resolved.db.name ), query_ms=1.2, select_templates=[ f""{'*' if template_name == template.name else ''}{template_name}"" for template_name in templates ], ), request=request, view_name=""table"", ), headers=headers, ) else: assert False, ""Invalid format: {}"".format(format_) # if next_url: # r.headers[""link""] = f'<{next_url}>; rel=""next""' return r ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1565058994,https://api.github.com/repos/simonw/datasette/issues/2053,1565058994,IC_kwDOBm6k_c5dSOey,9599,simonw,2023-05-26T23:13:02Z,2023-05-26T23:13:02Z,OWNER,"I should have an extra called `extra_html_context` which bundles together all of the weird extra stuff needed by the HTML template, and is then passed as the root context when the template is rendered (with the other stuff from extras patched into it).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563793781,https://api.github.com/repos/simonw/datasette/issues/2053,1563793781,IC_kwDOBm6k_c5dNZl1,9599,simonw,2023-05-26T04:27:55Z,2023-05-26T04:27:55Z,OWNER,"I should split out a `canned_query.html` template too, as something that extends the `query.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563667574,https://api.github.com/repos/simonw/datasette/issues/2053,1563667574,IC_kwDOBm6k_c5dM6x2,9599,simonw,2023-05-26T00:40:22Z,2023-05-26T00:40:22Z,OWNER,"Or maybe... - `BaseQueryView(View)` - knows how to render the results of a SQL query - `QueryView(BaseQueryView)` - renders from `?sql=` - `CannedQueryView(BaseQueryView)` - renders for a named canned query And then later perhaps: - `RowQueryView(BaseQueryView)` - renders the `select * from t where pk = ?` - `TableQueryView(BaseQueryView)` - replaces the super complex existing `TableView`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563663925,https://api.github.com/repos/simonw/datasette/issues/2053,1563663925,IC_kwDOBm6k_c5dM541,9599,simonw,2023-05-26T00:32:47Z,2023-05-26T00:35:47Z,OWNER,"I'm going to entirely split canned queries off from `?sql=` queries - they share a bunch of code right now which is just making everything much harder to follow. I'll refactor their shared bits into functions that they both call. Or _maybe_ I'll try having `CannedQueryView` as a subclass of `QueryView`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563663616,https://api.github.com/repos/simonw/datasette/issues/2053,1563663616,IC_kwDOBm6k_c5dM50A,9599,simonw,2023-05-26T00:32:08Z,2023-05-26T00:32:08Z,OWNER,"Now that I have the new `View` subclass from #2078 I want to use it to simplify this code. Challenge: there are several things to consider here: - The `/db` page without `?sql=` displays a list of tables in that database - With `?sql=` it shows the query results for that query (or an error) - If it's a `/db/name-of-canned-query` it works a bit like the query page, but executes a canned query instead of the `?sql=` query - POST `/db/name-of-canned-query` is support for writable canned queries","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1563285150,https://api.github.com/repos/simonw/datasette/issues/2053,1563285150,IC_kwDOBm6k_c5dLdae,9599,simonw,2023-05-25T17:48:50Z,2023-05-25T17:49:52Z,OWNER,"Uncommitted experimental code: ```diff diff --git a/datasette/views/database.py b/datasette/views/database.py index 455ebd1f..85775433 100644 --- a/datasette/views/database.py +++ b/datasette/views/database.py @@ -909,12 +909,13 @@ async def query_view( elif format_ in datasette.renderers.keys(): # Dispatch request to the correct output format renderer # (CSV is not handled here due to streaming) + print(data) result = call_with_supported_arguments( datasette.renderers[format_][0], datasette=datasette, - columns=columns, - rows=rows, - sql=sql, + columns=data[""rows""][0].keys(), + rows=data[""rows""], + sql='', query_name=None, database=db.name, table=None, @@ -923,7 +924,7 @@ async def query_view( # These will be deprecated in Datasette 1.0: args=request.args, data={ - ""rows"": rows, + ""rows"": data[""rows""], }, # TODO what should this be? ) result = await await_me_maybe(result) diff --git a/docs/index.rst b/docs/index.rst index 5a9cc7ed..254ed3da 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -57,6 +57,7 @@ Contents settings introspection custom_templates + template_context plugins writing_plugins plugin_hooks ``` Where `docs/template_context.rst` looked like this: ```rst .. _template_context: Template context ================ .. currentmodule:: datasette.context This page describes the variables made available to templates used by Datasette to render different pages of the application. .. autoclass:: QueryContext :members: ``` And `datasette/context.py` had this: ```python from dataclasses import dataclass @dataclass class QueryContext: """""" Used by the ``/database`` page when showing the results of a SQL query """""" id: int ""Id is a thing"" rows: list[dict] ""Name is another thing"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/datasette/pull/2053#issuecomment-1498279469,https://api.github.com/repos/simonw/datasette/issues/2053,1498279469,IC_kwDOBm6k_c5ZTe4t,9599,simonw,2023-04-05T23:28:53Z,2023-04-05T23:28:53Z,OWNER,"Table errors page currently does this: ```json { ""ok"": false, ""error"": ""no such column: blah"", ""status"": 400, ""title"": ""Invalid SQL"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1656432059,WIP new JSON for queries, https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592617271,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592617271,IC_kwDOCGYnMM5e7Wk3,13780613,erlend-aasland,2023-06-15T08:39:49Z,2023-06-15T08:39:49Z,NONE,"> piping to `jq` is good enough usually ... or `python -m json.tool`[^1], if you don't have `jq` installed. [^1]: no fancy colouring, like `jq`; only pretty-printing","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,rows: --transpose or psql extended view-like functionality, https://github.com/simonw/sqlite-utils/issues/535#issuecomment-1592052320,https://api.github.com/repos/simonw/sqlite-utils/issues/535,1592052320,IC_kwDOCGYnMM5e5Mpg,7908073,chapmanjacobd,2023-06-14T22:05:28Z,2023-06-14T22:05:28Z,CONTRIBUTOR,piping to `jq` is good enough usually,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1655860104,rows: --transpose or psql extended view-like functionality, https://github.com/simonw/datasette/pull/2052#issuecomment-1760552652,https://api.github.com/repos/simonw/datasette/issues/2052,1760552652,IC_kwDOBm6k_c5o7-bM,9599,simonw,2023-10-12T23:59:21Z,2023-10-12T23:59:21Z,OWNER,I'm landing this despite the cog failures. I'll fix them on main if I have to.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1760545012,https://api.github.com/repos/simonw/datasette/issues/2052,1760545012,IC_kwDOBm6k_c5o78j0,9599,simonw,2023-10-12T23:48:16Z,2023-10-12T23:48:16Z,OWNER,Oh! I think I broke Cog on `main` and these tests are running against this branch rebased against main.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1760542865,https://api.github.com/repos/simonw/datasette/issues/2052,1760542865,IC_kwDOBm6k_c5o78CR,9599,simonw,2023-10-12T23:44:53Z,2023-10-12T23:45:15Z,OWNER,"Weird, the `cog` check is failing in CI. ``` Run cog --check docs/*.rst cog --check docs/*.rst shell: /usr/bin/bash -e {0} env: pythonLocation: /opt/hostedtoolcache/Python/3.9.18/x64 PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib/pkgconfig Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.18/x64 LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.18/x64/lib Check failed Checking docs/authentication.rst Checking docs/binary_data.rst Checking docs/changelog.rst Checking docs/cli-reference.rst Checking docs/configuration.rst (changed) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1722943484,https://api.github.com/repos/simonw/datasette/issues/2052,1722943484,IC_kwDOBm6k_c5msgf8,30934,20after4,2023-09-18T08:14:47Z,2023-09-18T08:14:47Z,NONE,This is such a well thought out contribution. I don't think I've seen such a thoroughly considered PR on any project in recent memory.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1632867333,https://api.github.com/repos/simonw/datasette/issues/2052,1632867333,IC_kwDOBm6k_c5hU5QF,22429695,codecov[bot],2023-07-12T16:38:27Z,2023-10-12T23:52:24Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report All modified lines are covered by tests :white_check_mark: > Comparison is base [(`3feed1f`)](https://app.codecov.io/gh/simonw/datasette/commit/3feed1f66e2b746f349ee56970a62246a18bb164?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.46% compared to head [(`8ae479c`)](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.69%. > Report is 112 commits behind head on main.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2052 +/- ## ========================================== + Coverage 92.46% 92.69% +0.22% ========================================== Files 38 40 +2 Lines 5750 6047 +297 ========================================== + Hits 5317 5605 +288 - Misses 433 442 +9 ``` [see 19 files with indirect coverage changes](https://app.codecov.io/gh/simonw/datasette/pull/2052/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/datasette/pull/2052?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1630776144,https://api.github.com/repos/simonw/datasette/issues/2052,1630776144,IC_kwDOBm6k_c5hM6tQ,9020979,hydrosquall,2023-07-11T12:54:03Z,2023-07-11T12:54:03Z,CONTRIBUTOR,"Thanks for the review and the code pointers @simonw - I've made the suggested edits, fixed the renamed variable, and confirmed that the panels still render on the `table` and `database` views. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1629337927,https://api.github.com/repos/simonw/datasette/issues/2052,1629337927,IC_kwDOBm6k_c5hHblH,9599,simonw,2023-07-10T16:43:38Z,2023-07-10T16:44:23Z,OWNER,"I tried running this locally just now. I made one edit: ```diff diff --git a/demos/plugins/example_js_manager_plugins.py b/demos/plugins/example_js_manager_plugins.py index 7bdb9f3f..f9dfa8e6 100644 --- a/demos/plugins/example_js_manager_plugins.py +++ b/demos/plugins/example_js_manager_plugins.py @@ -15,6 +15,6 @@ def extra_js_urls(view_name): if view_name in PERMITTED_VIEWS: return [ { - ""url"": f""/-/demos/plugins/static/table-example-plugins.js"", + ""url"": f""/static/table-example-plugins.js"", } ] ``` And then started it running like this: ```bash wget https://datasette.io/content.db ``` ```bash datasette content.db \ --plugins-dir demos/plugins \ --static static:datasette/demos/plugins/static ``` It didn't quite work for me - I got this error on a table page: And this error on a query page: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1616095810,https://api.github.com/repos/simonw/datasette/issues/2052,1616095810,IC_kwDOBm6k_c5gU6pC,15178711,asg017,2023-07-01T20:31:31Z,2023-07-01T20:31:31Z,CONTRIBUTOR,"> Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory? I just did a github search for `user:simonw ""def extra_js_urls(""` ! Though I'm sure other plugins made by people other than Simon also exist out there https://github.com/search?q=user%3Asimonw+%22def+extra_js_urls%28%22&type=code","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1615997736,https://api.github.com/repos/simonw/datasette/issues/2052,1615997736,IC_kwDOBm6k_c5gUiso,9020979,hydrosquall,2023-07-01T16:55:24Z,2023-07-01T16:55:24Z,CONTRIBUTOR,"> Ok @hydrosquall a couple things before this PR should be good to go: Thank you @asg017 ! I've pushed both suggested changes onto this branch. > Not sure how difficult it'll be to inject it server-side If we are OK with having a build system, it would free me up to do do many things! We could make datasette-manager.js a server-side rendered file as a ""template"" instead of having it as a static JS file, but I'm not sure it's worth the extra jump in complexity / loss of syntax highlighting in the JS file. In the short-term, I could see an intermediary solution where a unit test in the preferred language was able to read both `version.py` and `datasette-manager.js`, and make sure that the strings versions are in sync. (This assumes that we want the manager and datasette's versions to be synced, and not decoupled). Since the version is not changing very often, a ""manual sync"" might be good enough. > In terms of how to integrate this into Datasette, a few options I can see working: This sounds good to me. I'm not sure how to add a settings flag, but will be interested to see the PR that adds support for it. > I'm also curious to see how ""plugins for a plugin' would work I'm comfortable to wait until we have a realistic usecase for this. In the short term, I think we could give plugins a way to grant access to a ""public API of other plugins"", and also ask to be notified when plugins with other names have loaded, but don't picture the datasette manager getting more involved than that. > here's a list of Simon's Datasette plugins that use ""extra_js_urls()"" Neat, thanks for compiling this list! Just curious, is there a query that can be used to compile this programmatically, or did you identify these through memory? > I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.) I look forward to trying this out 👍 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1613778296,https://api.github.com/repos/simonw/datasette/issues/2052,1613778296,IC_kwDOBm6k_c5gME14,15178711,asg017,2023-06-29T20:36:09Z,2023-06-29T20:36:09Z,CONTRIBUTOR,"Ok @hydrosquall a couple things before this PR should be good to go: - Can we move `datasette/static/table-example-plugins.js` into `demos/plugins/static`? - For `datasetteManager.VERSION`, can we fill that in or just comment it out for now? Not sure how difficult it'll be to inject it server-side. I imagine we could also have a small build process with esbuild/rollup that just injects a version string into `manager.js` directly, so we don't have to worry about server-rendering (but that can be a future PR) In terms of how to integrate this into Datasette, a few options I can see working: - Push this as-is and figure it out before the next release - Hide this feature behind a settings flag (`--setting unstable-js-plugins on`) and use that setting to hide/show `` in `base.html` I'll let @simonw decide which one to work with. I kindof like the idea of having an ""unstable"" opt-in process to enable JS plugins, to give us time to try it out with a wide variety of plugins until we feel its ready. I'm also curious to see how ""plugins for a plugin' would work, like #1542. For example, if the leaflet plugin showed default markers, but also included its own hook for other plugins to add more markers/styling. I'm imagine that the individual plugin would re-create their own plugin system compared to this, since handling ""plugins of plugins"" at the top with Datasette seems really convoluted. Also for posterity, here's a list of Simon's Datasette plugins that use ""extra_js_urls()"", which probably means they can be ported/re-written to use this new plugin system: - [`datasette-vega`](https://github.com/simonw/datasette-vega/blob/00de059ab1ef77394ba9f9547abfacf966c479c4/datasette_vega/__init__.py#L25) - [`datasette-cluster-map`](https://github.com/simonw/datasette-cluster-map/blob/795d25ad9ff6cba0307191f44fecc8f8070bef5c/datasette_cluster_map/__init__.py#L14) - [`datasette-leaflet-geojson`](https://github.com/simonw/datasette-leaflet-geojson/blob/64713aa497750400b9ac2c12e8bb6ffab8eb77f3/datasette_leaflet_geojson/__init__.py#L47) - [`datasette-pretty-traces`](https://github.com/simonw/datasette-pretty-traces/blob/5219d65eca3d7d7a73bb9d3120df42fe046a1315/datasette_pretty_traces/__init__.py#L5) - [`datasette-youtube-embed`](https://github.com/simonw/datasette-youtube-embed/blob/4b4a0d7e58ebe15f47e9baf68beb9908c1d899da/datasette_youtube_embed/__init__.py#L55) - [`datasette-leaflet-freedraw`](https://github.com/simonw/datasette-leaflet-freedraw/blob/8f28c2c2080ec9d29f18386cc6a2573a1c8fbde7/datasette_leaflet_freedraw/__init__.py#L66) - [`datasette-hovercards`](https://github.com/simonw/datasette-hovercards/blob/9439ba46b7140fb03223faff0d21aeba5615a287/datasette_hovercards/__init__.py#L5) - [`datasette-mp3-audio`](https://github.com/simonw/datasette-mp3-audio/blob/4402168792f452a46ab7b488e40ec49cd4b12185/datasette_mp3_audio/__init__.py#L6) - [`datasette-geojson-map`](https://github.com/simonw/datasette-geojson-map/blob/32af5f1fd1a07278bbf8071fbb20a61e0f613246/datasette_geojson_map/__init__.py#L30)","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1606352600,https://api.github.com/repos/simonw/datasette/issues/2052,1606352600,IC_kwDOBm6k_c5fvv7Y,15178711,asg017,2023-06-26T00:17:04Z,2023-06-26T00:17:04Z,CONTRIBUTOR,":wave: would love to see this get merged soon! I want to make a javascript plugin on top of the code-mirror editor to make a few things nicer (function auto-complete, table/column descriptions, etc.), and this would help out a bunch","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1585149909,https://api.github.com/repos/simonw/datasette/issues/2052,1585149909,IC_kwDOBm6k_c5ee3fV,9020979,hydrosquall,2023-06-09T21:35:00Z,2023-06-09T21:35:00Z,CONTRIBUTOR,"Thanks @cldellow for the thoughtful comments! These are all things that I'll keep in mind as we figure out how/if this API is actually used by plugin authors once it's actually out in the world. > Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation. Ah, I understand now! Thanks for explaining. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1548617257,https://api.github.com/repos/simonw/datasette/issues/2052,1548617257,IC_kwDOBm6k_c5cTgYp,193185,cldellow,2023-05-15T21:32:20Z,2023-05-15T21:32:20Z,CONTRIBUTOR,"> Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like makeColumnActions or makeAboveTablePanelConfigs supported returning a promise of arrays instead only returning plain arrays? The latter - that you could return a promise of arrays, so it parallels the [""await me maybe"" pattern in Datasette](https://simonwillison.net/2020/Sep/2/await-me-maybe/), where you can return either a value, a callable or an awaitable. > I have a hunch that what you're describing might be achievable without adding Promises to the API with something Oops, I did a poor job explaining. Yes, this would work - but it requires me to continue to communicate the column names out of band (in order to fetch the facet data per-column before registering my plugin), vs being able to re-use them from the plugin implementation. This isn't that big of a deal - it'd be a nice ergonomic improvement, but nowhere near as a big of an improvement as having an officially sanctioned way to add stuff to the column menus in the first place. This could also be layered on in a future commit without breaking v1 users, too, so it's not at all urgent. > especially if those lines are encapsulated by a function we provide (maybe something that's available on the window provided by Datasette as an inline script tag Ah, this is maybe the the key point. Since it's all hosted inside Datasette, Datasette can provide some arbitrary sugar to make it easier to work with. My experience with async scripts in JS is that people sometimes don't understand the race conditions inherent to them. If they copy/paste from a tutorial, it does just work. But then they'll delete half the code, and by chance it still works on their machine/Datasette templates, and now someone's headed for an annoying debugging session -- maybe them, maybe someone else who tries to re-use their plugin. Again, a fairly minor thing, though.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1546362374,https://api.github.com/repos/simonw/datasette/issues/2052,1546362374,IC_kwDOBm6k_c5cK54G,9020979,hydrosquall,2023-05-12T22:09:03Z,2023-05-12T22:09:03Z,CONTRIBUTOR,"Hey @cldellow , thanks for the thoughtful feedback and describing the ""lazy facets"" feature! It sounds like the [postTask](https://developer.mozilla.org/en-US/docs/Web/API/Scheduler/postTask) API might be relevant for the types of network request scheduling you have in mind. Addressing your points inline below: > It might also be nice if the plugins could return Promises. Were you picturing that the whole plugin config object could be returned as a promise, or that the individual hooks (like `makeColumnActions` or `makeAboveTablePanelConfigs` supported returning a promise of arrays instead only returning plain arrays? I think what you're describing can be achievable, but I want to make sure I do so in a way that addresses your need / keeps the complexity of the plugin core system at a level this is approachable . I have a hunch that what you're describing might be achievable without adding Promises to the API with something like ``` fetch('/api/with-custom-facets').then(myFacets => { // reusing the go() idiom go(manager, myFacets); }) ``` but I'd like to confirm if that's the case before investigating adding support. > bulletproof plugin registration code that is robust against the order in which the script tags load Yes, I think what you wrote looks right to me! While it looks a little bit verbose compared to the second example, I'm hoping we can mitigate the cost of that during this API incubation phase by making it an easy-to-copy paste code snippet. I haven't heard of the GA queing pattern before, thanks for the example. I won't have time to implement of proof of concept in the next few weeks, but I took some time to think through the pros/cons to decide whether we may want to add this in a future release: I can see that this approach brings advantages - Plugin developers don't need to know the name of the datasette initialization event to start their plugin - Pushing a function to an array probably is easier (definitely more concise) than adding a document event listener - One less event listener sitting in memory It also has some minor costs - A malicious plugin could choose to (or accidentally) mess with the order of the queue if multiple scripts are lined up - Some risk in encouraging people to mutate global state - (not a cost, more a moot point): changing this API may not make a meaningful difference if we're discussing whether people enter 2 vs 5 lines of code, especially if those lines are encapsulated by a function we provide (maybe something that's available on the `window` provided by Datasette as an inline script tag). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1651082214,"feat: Javascript Plugin API (Custom panels, column menu items with JS actions)", https://github.com/simonw/datasette/pull/2052#issuecomment-1530822437,https://api.github.com/repos/simonw/datasette/issues/2052,1530822437,IC_kwDOBm6k_c5bPn8l,193185,cldellow,2023-05-02T03:35:30Z,2023-05-02T16:02:38Z,CONTRIBUTOR,"Also, just checking - is this how I'd write bulletproof plugin registration code that is robust against the order in which the script tags load (eg if both my code and the Datasette code are loaded via a `\n\n\n\n\n', ...} == {'1+1': 2, 'c... 0xXXX>', ...} FAILED tests/test_permissions.py::test_permissions_checked[/fixtures/simple_primary_key-permissions3] - assert 500 in (200, 403) FAILED tests/test_table_html.py::test_table_csv_json_export_interface - assert 500 == 200 FAILED tests/test_table_html.py::test_table_metadata - assert 500 == 200 FAILED tests/test_html.py::test_css_classes_on_body[/fixtures/simple_primary_key-expected_classes3] - assert 500 == 200 FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert 500 == 200 FAILED tests/test_plugins.py::test_hook_table_actions[simple_view] - AssertionError: assert [] == [{'href': '/'...simple_view'}] ERROR tests/test_internals_database.py::test_execute_write_fn_connection_exception - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_mtime_ns_is_none_for_memory - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_is_mutable - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_database_memory_name - Failed: Timeout >1.0s ERROR tests/test_internals_database.py::test_in_memory_databases_forbid_writes - Failed: Timeout >1.0s =============== 21 failed, 1275 passed, 2 skipped, 1 xfailed, 6 errors in 59.18s ================ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1480163485,https://api.github.com/repos/simonw/datasette/issues/1999,1480163485,IC_kwDOBm6k_c5YOYCd,9599,simonw,2023-03-22T19:48:00Z,2023-03-22T19:48:00Z,OWNER,"Getting close now! Only 13 failures left, mostly relating to CSV. ``` FAILED tests/test_csv.py::test_table_csv - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_cors_headers - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_no_header - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_blob_columns - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_download - assert 500 == 200 FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_plugins.py::test_hook_extra_css_urls[/fixtures/sortable-expected_decoded_object2] - AssertionError: assert {'added': 15,...ortable', ...} == {'added': 15,...ortable', ...} FAILED tests/test_plugins.py::test_hook_register_facet_classes - KeyError: 'suggested_facets' FAILED tests/test_csv.py::test_csv_trace - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_plugins.py::test_hook_extra_body_script[/fixtures/sortable-expected_extra_body_script2] - AssertionError: assert {'added': 15,...ixtures', ...} == {'added': 15,...ixtures', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_all_parameters - assert {'1+1': 2, 'c... 0xXXX>', ...} == {'1+1': 2, 'c... 0xXXX>', ...} =============== 13 failed, 1287 passed, 2 skipped, 1 xfailed in 61.57s (0:01:01) ================ ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1480154453,https://api.github.com/repos/simonw/datasette/issues/1999,1480154453,IC_kwDOBm6k_c5YOV1V,9599,simonw,2023-03-22T19:41:39Z,2023-03-22T19:43:04Z,OWNER,"To replace this code: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L122 Maybe `datasette.render_template()` should optionally accept a list of templates. https://docs.datasette.io/en/stable/internals.html#await-render-template-template-context-none-request-none - turns out it does already: > If this is a list of template file names then the first one that exists will be loaded and rendered. It doesn't have an easy way to populate that `select_templates` debug template variable though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1477082852,https://api.github.com/repos/simonw/datasette/issues/1999,1477082852,IC_kwDOBm6k_c5YCn7k,9599,simonw,2023-03-20T23:27:25Z,2023-03-20T23:27:25Z,OWNER,"Urgh getting CSV to work is going to be _so hard_, because the logic for that currently lives in a huge chunk of code in `BaseView` which depends on the old design of the `data()` method: https://github.com/simonw/datasette/blob/4bb49848697e40b8b9a1557be42b8e59eac965b3/datasette/views/base.py#L177-L343","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476898261,https://api.github.com/repos/simonw/datasette/issues/1999,1476898261,IC_kwDOBm6k_c5YB63V,9599,simonw,2023-03-20T20:37:52Z,2023-03-20T20:37:52Z,OWNER,"Manual testing spotted a bug. `/content/repos.json?owner=9599&_facet_array=topics` - does not return a `facet_results` key. `/content/repos.json?owner=9599&_facet_array=topics&_facet=owner` does.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476860334,https://api.github.com/repos/simonw/datasette/issues/1999,1476860334,IC_kwDOBm6k_c5YBxmu,9599,simonw,2023-03-20T20:06:56Z,2023-03-20T22:09:03Z,OWNER,"`pytest -n 8` locally says: ``` FAILED tests/test_canned_queries.py::test_canned_query_form_csrf_hidden_field[add_name_specify_id-True] - assert '\n\n\n Error 500\n details.open = false);\n});\n\n\n\n Error 500\n details.open = false);\n});\n' in '\n\n\n Error 500\n details.open = false);\... FAILED tests/test_canned_queries.py::test_insert - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_insert_error - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[False-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_magic_parameters_csrf_json[True-True] - KeyError: 'ds_csrftoken' FAILED tests/test_canned_queries.py::test_vary_header - KeyError: 'vary' FAILED tests/test_csv.py::test_csv_trace - AssertionError: assert 'application/...charset=utf-8' == 'text/html; charset=utf-8' FAILED tests/test_csv.py::test_table_csv - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_blob_columns - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_download - AssertionError: assert 'application/...charset=utf-8' == 'text/csv; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_no_header - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_stream - assert 1 == 101 FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_counts - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_stream_does_not_calculate_facets - AttributeError: 'NoneType' object has no attribute 'text' FAILED tests/test_csv.py::test_table_csv_with_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_csv.py::test_table_csv_with_nullable_labels - AssertionError: assert 'application/...charset=utf-8' == 'text/plain; charset=utf-8' FAILED tests/test_facets.py::test_array_facet_handle_duplicate_tags - KeyError: 'facet_results' FAILED tests/test_facets.py::test_conflicting_facet_names_json - AssertionError: assert {'results', 'timed_out'} == {'created', '...gs', 'tags_2'} FAILED tests/test_facets.py::test_facet_size - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_json_array_with_blanks_and_nulls - KeyError: 'suggested_facets' FAILED tests/test_facets.py::test_other_types_of_facet_in_metadata - assert 'created (date)\n' in '\n\n\n fixtures: facetable: 15 rows\n \n\n\n\n' in '\n\n\n fixtures: simple_primary_key: 5 rows\n ' in '\n\n\n fixtures: table/with/slashes.csv: 1 row\n ', ...} FAILED tests/test_plugins.py::test_hook_register_output_renderer_no_parameters - assert b'Hello' == b'{""ok"": true... ""n"": null}]}' FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_broken_value - assert 200 == 500 FAILED tests/test_plugins.py::test_hook_register_output_renderer_returning_response - assert {'next': None...', ...}, ...]} == {'this_is': 'json'} === 39 failed, 1259 passed, 2 skipped, 1 xfailed in 58.07s === ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476854645,https://api.github.com/repos/simonw/datasette/issues/1999,1476854645,IC_kwDOBm6k_c5YBwN1,9599,simonw,2023-03-20T20:02:24Z,2023-03-20T20:02:24Z,OWNER,"``` def test_routes(routes, path, expected_class, expected_matches): match, view = resolve_routes(routes, path) if expected_class is None: assert match is None else: > assert view.view_class.__name__ == expected_class E AttributeError: 'function' object has no attribute 'view_class' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1476851525,https://api.github.com/repos/simonw/datasette/issues/1999,1476851525,IC_kwDOBm6k_c5YBvdF,9599,simonw,2023-03-20T19:59:51Z,2023-03-20T20:01:40Z,OWNER,"Three failures in `test_html.py`: ``` FAILED tests/test_html.py::test_templates_considered[/fixtures/simple_primary_key-table-fixtures-simple_primary_key.html, *table.html] - assert '' in '\n\n\n fixtures: simple_primary_key: 5 rows\n ' in '\n\n\n fixtures: table/with/slashes.csv: 1 row\n \xa0']] == [['']] FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels - assert [['']] == [['']] FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['']] == [['']] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/infinity-expected_column_options0] - AssertionError: assert ['- column -'...wid', 'value'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/primary_key_multiple_columns-expected_column_options1] - AssertionError: assert ['- column -'...', 'content2'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_column_options[/fixtures/compound_primary_key-expected_column_options2] - AssertionError: assert ['- column -'...2', 'content'] == ['- column -'] FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns - AssertionError: assert ['- column -'] == ['- column -'...nulls_2', ...] FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references - assert [['']] == [['']] FAILED tests/test_table_html.py::test_view_html - assert None is not None FAILED tests/test_table_html.py::test_extra_where_clauses - assert [('_where', ""...'_city_id=1')] == [] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10-expected_hidden0] - AssertionError: assert [] == [('_size', '10')] FAILED tests/test_table_html.py::test_other_hidden_form_fields[/fixtures/facetable?_size=10&_ignore=1&_ignore=2-expected_hidden1] - AssertionError: assert [] == [('_size', '1...ignore', '2')] FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated[/fixtures/searchable?_sort=text2&_where=1-expected_hidden3] - AssertionError: assert [] == [('_where', '1')] FAILED tests/test_table_html.py::test_binary_data_display_in_table - assert [['']] == [['']] FAILED tests/test_table_html.py::test_metadata_sort - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_metadata_sort_desc - AttributeError: 'NoneType' object has no attribute 'string' FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood-2-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 2 FAILED tests/test_table_html.py::test_facet_more_links[5-/fixtures/facetable?_facet=_neighborhood&_facet_size=50-5-True-/fixtures/facetable?_facet=_neighborhood&_facet_size=max] - assert 0 == 5 FAILED tests/test_table_html.py::test_facet_total - assert 500 == 200 ``` Deduped that's 30 tests: ``` FAILED tests/test_table_html.py::test_add_filter_redirects FAILED tests/test_table_html.py::test_binary_data_display_in_table FAILED tests/test_table_html.py::test_compound_primary_key_with_foreign_key_references FAILED tests/test_table_html.py::test_csv_json_export_links_include_labels_if_foreign_keys FAILED tests/test_table_html.py::test_empty_search_parameter_gets_removed FAILED tests/test_table_html.py::test_existing_filter_redirects FAILED tests/test_table_html.py::test_extra_where_clauses FAILED tests/test_table_html.py::test_facet_more_links FAILED tests/test_table_html.py::test_facet_total FAILED tests/test_table_html.py::test_facets_persist_through_filter_form FAILED tests/test_table_html.py::test_metadata_sort FAILED tests/test_table_html.py::test_metadata_sort_desc FAILED tests/test_table_html.py::test_next_does_not_persist_in_hidden_field FAILED tests/test_table_html.py::test_other_hidden_form_fields FAILED tests/test_table_html.py::test_reflected_hidden_form_fields FAILED tests/test_table_html.py::test_rowid_sortable_no_primary_key FAILED tests/test_table_html.py::test_search_and_sort_fields_not_duplicated FAILED tests/test_table_html.py::test_searchable_view_persists_fts_table FAILED tests/test_table_html.py::test_sort_by_desc_redirects FAILED tests/test_table_html.py::test_sort_links FAILED tests/test_table_html.py::test_table_csv_json_export_interface FAILED tests/test_table_html.py::test_table_html_compound_primary_key FAILED tests/test_table_html.py::test_table_html_disable_foreign_key_links_with_labels FAILED tests/test_table_html.py::test_table_html_filter_form_column_options FAILED tests/test_table_html.py::test_table_html_filter_form_still_shows_nocol_columns FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column FAILED tests/test_table_html.py::test_table_html_foreign_key_links FAILED tests/test_table_html.py::test_table_html_no_primary_key FAILED tests/test_table_html.py::test_table_html_simple_primary_key FAILED tests/test_table_html.py::test_view_html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1475074025,https://api.github.com/repos/simonw/datasette/issues/1999,1475074025,IC_kwDOBm6k_c5X69fp,9599,simonw,2023-03-19T02:14:28Z,2023-03-19T02:14:51Z,OWNER,"I had to replicate quite a bit of this logic from `base.py`: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L526-L544 I should refactor this when I move the canned / arbitrary query views away from that base class too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1475016834,https://api.github.com/repos/simonw/datasette/issues/1999,1475016834,IC_kwDOBm6k_c5X6viC,9599,simonw,2023-03-18T22:30:31Z,2023-03-18T22:30:31Z,OWNER,"`test_paginate_using_link_header` will be tricky. The reason the tests are failing is that `json_renderer()` attempts to populate the `link` header using `data[""next_url""]` - but that's not present unless `?_extra=next_url` has been passed: https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/renderer.py#L101-L102 But I can only rely on `next` being present, not `next_url`. I thought I could maybe assemble the `link` header using `next`, by turning that into a `next_url` link - but there's some custom logic which kicks in for pagination against SQL views (offset/limit pagination) to calculate the `next_url` which isn't easily replicable at the `json_renderer()` layer, in the `_next_value_and_url()` utility function: https://github.com/simonw/datasette/blob/2f38479dcc81f11a4362f4e28511fa88afc34e61/datasette/views/table.py#L2275-L2282 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1475003292,https://api.github.com/repos/simonw/datasette/issues/1999,1475003292,IC_kwDOBm6k_c5X6sOc,9599,simonw,2023-03-18T21:46:20Z,2023-03-18T21:46:20Z,OWNER,"Now 25 failures in `test_table_api.py`: ``` FAILED tests/test_table_api.py::test_expand_labels - assert {'2': {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', '_city_id': 1, '_neighborhood': 'Dogpatch', 'tags': '[""tag1"", ""tag3""]', 'complex_array'... FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_co... FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1, 'value': inf}, {'rowid': 2, 'value': -inf}, {'rowid': 3, 'value': 1.5}] == [{'rowid': 1, 'value': None}, {'rowid': 2, 'value': None}, {'rowid': 3, 'value'... FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'pk': '1', 'foreign_key_with_label': '1', 'foreign_key_with_blank_label': '3', 'foreign_key_with_no_label': '1', 'foreign_key_compound_pk1': 'a', 'foreign_key_compoun... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'rowid': 1, 'data': ""b'\\x15\\x1c\\x02\\xc7\\xad\\x05\\xfe'""}, {'rowid': 2, 'data': ""b'\\x15\\x1c\\x03\\xc7\\xad\\x05\\xfe'""}, {'rowid': 3, 'data': None}] == [{'rowid': 1, 'data': {'... FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}\n{""rowid"": 3, ""data"": null}] - assert '{""ok"": false, ""error"": ""Object of type bytes is not JSON serializable"", ""status"": 500, ""title"": null}' == '{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid""... FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 25 failed, 86 passed, 1 xfailed in 7.18s ============================================================================= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1474704790,https://api.github.com/repos/simonw/datasette/issues/1999,1474704790,IC_kwDOBm6k_c5X5jWW,9599,simonw,2023-03-18T04:52:59Z,2023-03-18T04:52:59Z,OWNER,"Here are the next set of tests to get passing: ``` % pytest tests/test_table_api.py ``` ``` FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id-expected_facet_results0] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=state&_facet=_city_id&state=MI-expected_facet_results1] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int-expected_facet_results2] - KeyError: 'name' FAILED tests/test_table_api.py::test_facets[/fixtures/facetable.json?_facet=planet_int&planet_int=1-expected_facet_results3] - KeyError: 'name' FAILED tests/test_table_api.py::test_suggested_facets - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_allow_facet_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_suggest_facets_off - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nofacet[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[True] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nosuggest[False] - KeyError: 'suggested_facets' FAILED tests/test_table_api.py::test_nocount[True-None] - KeyError: 'count' FAILED tests/test_table_api.py::test_nocount[False-15] - KeyError: 'count' FAILED tests/test_table_api.py::test_expand_labels - AssertionError: assert {'13': {'_cit...:00:00', ...}} == {'13': {'_cit...:00:00', ...}} FAILED tests/test_table_api.py::test_expand_label - AssertionError: assert {'1': {'forei...l': '1', ...}} == {'1': {'forei...': '1'}, ...}} FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=invalid-max-age=5] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=10-max-age=10] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_ttl_parameter[/fixtures/facetable.json?_ttl=0-no-cache] - KeyError: 'Cache-Control' FAILED tests/test_table_api.py::test_infinity_returned_as_null - AssertionError: assert [{'rowid': 1,...'value': 1.5}] == [{'rowid': 1,...'value': 1.5}] FAILED tests/test_table_api.py::test_null_and_compound_foreign_keys_are_not_expanded - AssertionError: assert [{'foreign_ke...': None, ...}] == [{'foreign_ke...': None, ...}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array-expected_json0-None] - assert [{'data': ""b'..., 'rowid': 3}] == [{'data': {'$..., 'rowid': 3}] FAILED tests/test_table_api.py::test_binary_data_in_json[/fixtures/binary_data.json?_shape=array&_nl=on-None-{""rowid"": 1, ""data"": {""$base64"": true, ""encoded"": ""FRwCx60F/g==""}}\n{""rowid"": 2, ""data"": {""$base64"": true, ""encoded"": ""FRwDx60F/g==""}}\n{""rowid"": 3, ""data"": null}] - assert '{""ok"": false...title"": null}' == '{""rowid"": 1,...""data"": null}' FAILED tests/test_table_api.py::test_paginate_using_link_header[] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrays] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=arrayfirst] - assert 400 == 200 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=object] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=objects] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array] - assert 1 == 21 FAILED tests/test_table_api.py::test_paginate_using_link_header[?_shape=array&_nl=on] - assert 1 == 21 FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=created-expected_columns0] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=created-expected_columns1] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created-expected_columns2] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=state-expected_columns3] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_col=state&_col=created&_nocol=created-expected_columns4] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/facetable.json?_nocol=state&_facet=state-expected_columns5] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_nocol=content-expected_columns6] - KeyError: 'columns' FAILED tests/test_table_api.py::test_col_nocol[/fixtures/simple_view.json?_col=content-expected_columns7] - KeyError: 'columns' ============================================================================= 38 failed, 73 passed, 1 xfailed in 7.25s ============================================================================= ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463113856,https://api.github.com/repos/simonw/datasette/issues/1999,1463113856,IC_kwDOBm6k_c5XNViA,9599,simonw,2023-03-10T02:13:15Z,2023-03-10T02:13:15Z,OWNER,Idea for if this change ends up making a bunch of breaking changes to the templates (which I think it should) - I can generate a GitHub diff link between the old and new templates and include that link in the 1.0 upgrade documentation to help people who wrote custom templates see what they might need to change - with minimal effort from myself to document those changes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463024951,https://api.github.com/repos/simonw/datasette/issues/1999,1463024951,IC_kwDOBm6k_c5XM_03,9599,simonw,2023-03-10T00:17:58Z,2023-03-10T00:17:58Z,OWNER,"Renderers have an impact on three different pages: query results, table page and row page. The row page feature is incomplete though: https://congress-legislators.datasettes.com/legislators/social_media/A000055 Why is there no `.csv` link there?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463023674,https://api.github.com/repos/simonw/datasette/issues/1999,1463023674,IC_kwDOBm6k_c5XM_g6,9599,simonw,2023-03-10T00:16:03Z,2023-03-10T00:16:03Z,OWNER,"I also need to figure out the `renderers` stuff, so I can link to the right URLs for CSV and JSON and other formats: https://github.com/simonw/datasette/blob/6d07a7da1531cd749844fc6827d9a1e57009b2ea/datasette/views/base.py#L474-L518","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463022397,https://api.github.com/repos/simonw/datasette/issues/1999,1463022397,IC_kwDOBm6k_c5XM_M9,9599,simonw,2023-03-10T00:14:21Z,2023-03-10T00:14:21Z,OWNER,"Ironically the thing I most need right now is comprehensive documentation of what variables are being passed to the templates! One big challenge is that I need to untangle the template context that happens in `BaseView` - I'm hacking that together at the moment, but I need a real answer for how that should work in a world in which view functions aren't using that base class. https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/views/base.py#L110-L145","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463021383,https://api.github.com/repos/simonw/datasette/issues/1999,1463021383,IC_kwDOBm6k_c5XM-9H,9599,simonw,2023-03-10T00:12:50Z,2023-03-10T00:12:50Z,OWNER,"Now at 34 failed, 34 passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1463005744,https://api.github.com/repos/simonw/datasette/issues/1999,1463005744,IC_kwDOBm6k_c5XM7Iw,9599,simonw,2023-03-09T23:52:15Z,2023-03-09T23:52:23Z,OWNER,"I need to figure out what to do about `extra_context_from_filters` - which was previously passed straight to the HTML context. https://github.com/simonw/datasette/blob/11f7feb7a3f7166c71389786880863d60ed3d165/datasette/views/table.py#L406-L422","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1462997800,https://api.github.com/repos/simonw/datasette/issues/1999,1462997800,IC_kwDOBm6k_c5XM5Mo,9599,simonw,2023-03-09T23:39:47Z,2023-03-09T23:39:47Z,OWNER,"Found a neat trick: ```diff diff --git a/datasette/app.py b/datasette/app.py index 186f192d..40416713 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -23,7 +23,13 @@ from pathlib import Path from markupsafe import Markup, escape from itsdangerous import URLSafeSerializer -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader +from jinja2 import ( + ChoiceLoader, + Environment, + FileSystemLoader, + PrefixLoader, + StrictUndefined, +) from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound @@ -394,7 +400,10 @@ class Datasette: ] ) self.jinja_env = Environment( - loader=template_loader, autoescape=True, enable_async=True + loader=template_loader, + autoescape=True, + enable_async=True, + undefined=StrictUndefined, ) self.jinja_env.filters[""escape_css_string""] = escape_css_string self.jinja_env.filters[""quote_plus""] = urllib.parse.quote_plus ``` This causes Jinja to raise a hard error if there are any variables referenced in the template that are not available in the context. It's helping me spot things that are still missing, rather than just relying on failed unit tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461161256,https://api.github.com/repos/simonw/datasette/issues/1999,1461161256,IC_kwDOBm6k_c5XF40o,9599,simonw,2023-03-09T02:10:07Z,2023-03-09T02:10:07Z,OWNER,"Just ran into a `no such table: columns` error - which I think is because my new view code sometimes bypasses calling this method (currently done in `BaseView.dispatch_request()`): https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/base.py#L101-L103","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461148579,https://api.github.com/repos/simonw/datasette/issues/1999,1461148579,IC_kwDOBm6k_c5XF1uj,9599,simonw,2023-03-09T01:54:10Z,2023-03-09T01:55:33Z,OWNER,Or... I could temporarily build a quick additional `CannedQueryView` subclass that just does the necessary bits to get the existing code to work. I'm going to try that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461148254,https://api.github.com/repos/simonw/datasette/issues/1999,1461148254,IC_kwDOBm6k_c5XF1pe,9599,simonw,2023-03-09T01:53:41Z,2023-03-09T01:53:41Z,OWNER,"Solving this is proving difficult: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L1500-L1503 The problem is that calling `.data()` on `QueryView` only works here because we expect to ourselves be inside a `.data()` method, with all of the existing magic that knows how to render things that are returned by that. So I may need to substantially re-engineer how `QueryView` works in order to get this to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461075648,https://api.github.com/repos/simonw/datasette/issues/1999,1461075648,IC_kwDOBm6k_c5XFj7A,9599,simonw,2023-03-09T00:24:22Z,2023-03-09T00:24:22Z,OWNER,`127.0.0.1:8001/fixtures/neighborhood_search` fails because the forwarding to a canned query does not yet work.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461074526,https://api.github.com/repos/simonw/datasette/issues/1999,1461074526,IC_kwDOBm6k_c5XFjpe,9599,simonw,2023-03-09T00:23:06Z,2023-03-09T00:23:06Z,OWNER," pytest tests/test_table_html.py Currently 44 failed, 24 passed in 7.53s Failures here: https://gist.github.com/simonw/df0a52cd7d820b776dc3dfc50e7cb778","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461070937,https://api.github.com/repos/simonw/datasette/issues/1999,1461070937,IC_kwDOBm6k_c5XFixZ,9599,simonw,2023-03-09T00:18:52Z,2023-03-09T00:19:36Z,OWNER,"I managed to get HTML view working! I did it by continuing to add more things to the extras and the `_html` bundle until the page loaded for me: ```diff async def extra_extras(): ""Available ?_extra= blocks"" return { @@ -1981,6 +2053,14 @@ async def extra_extras(): ""query"", ""display_columns"", ""display_rows"", + ""database"", + ""table"", + ""database_color"", + ""table_actions"", + ""filters"", + ""renderers"", + ""custom_table_templates"", + ""sorted_facet_results"", ] } @@ -2006,6 +2086,14 @@ async def extra_extras(): extra_query, extra_metadata, extra_extras, + extra_database, + extra_table, + extra_database_color, + extra_table_actions, + extra_filters, + extra_renderers, + extra_custom_table_templates, + extra_sorted_facet_results, ) ``` I'll probably refactor this into something cleaner, and maybe but a bunch of them in a `""html""` dictionary and update the templates to use `{{ html.filters }}` or similar. Will look at that once the tests are passing. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461047607,https://api.github.com/repos/simonw/datasette/issues/1999,1461047607,IC_kwDOBm6k_c5XFdE3,9599,simonw,2023-03-08T23:51:46Z,2023-03-08T23:51:46Z,OWNER,"This feels quite nice: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461044477,https://api.github.com/repos/simonw/datasette/issues/1999,1461044477,IC_kwDOBm6k_c5XFcT9,9599,simonw,2023-03-08T23:47:26Z,2023-03-08T23:47:26Z,OWNER,"I want to package together all of the extras that are needed for the HTML format. A few options for doing that: - Introduce `?_extra=_html` where the leading underscore indicates that this is a ""bundle"" of extras, then define a bundle that's everything needed for the HTML renderer - Have some other mechanism whereby different renderers can request a bundle of extras. I'm leaning towards the first option. I'll try that and see what it looks like.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461023559,https://api.github.com/repos/simonw/datasette/issues/1999,1461023559,IC_kwDOBm6k_c5XFXNH,9599,simonw,2023-03-08T23:23:02Z,2023-03-08T23:23:02Z,OWNER,"To get this unblocked, I'm going to allow myself to pass non-JSON-serializable objects to the HTML template version of things. If I can get that working (and get the existing tests to pass) I can consider a later change that makes those JSON serializable - or admit that it's OK for the templates to have non-JSON data passed to them and figure out how best to document those variables independently from the JSON documentation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1461002039,https://api.github.com/repos/simonw/datasette/issues/1999,1461002039,IC_kwDOBm6k_c5XFR83,9599,simonw,2023-03-08T22:58:16Z,2023-03-08T23:02:09Z,OWNER,"The reason for that `Row` thing is that it allows custom templates that do things like this: https://docs.datasette.io/en/stable/changelog.html#easier-custom-templates-for-table-rows ```html+jinja {% for row in display_rows %}

{{ row[""title""] }}

{{ row[""description""] }}

Category: {{ row.display(""category_id"") }}

{% endfor %} ``` Is that a good design? the `.display()` thing feels weird - I wonder if anyone has ever actually used that. It's documented here: https://docs.datasette.io/en/0.64.2/custom_templates.html#custom-templates > If you want to output the rendered HTML version of a column, including any links to foreign keys, you can use `{{ row.display(""column_name"") }}`. I can't see any examples of anyone using it in this code search: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+row.display It is however useful to have some kind of abstraction layer here that insulates the SQLite `Row` object, since having an extra layer will help if Datasette ever grows support for alternative database backends such as DuckDB or PostgreSQL.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460988975,https://api.github.com/repos/simonw/datasette/issues/1999,1460988975,IC_kwDOBm6k_c5XFOwv,9599,simonw,2023-03-08T22:42:57Z,2023-03-08T22:42:57Z,OWNER,"Aside idea: it might be interesting if there were ""lazy"" template variables available in the context: things that are not actually executed unless a template author requests them. Imagine if `metadata` was a lazy template reference, such that custom templates that don't display any metadata don't trigger it to be resolved (which might involve additional database queries some day).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460986533,https://api.github.com/repos/simonw/datasette/issues/1999,1460986533,IC_kwDOBm6k_c5XFOKl,9599,simonw,2023-03-08T22:40:28Z,2023-03-08T22:40:28Z,OWNER,"Figuring out what to do with `display_columns_and_rows()` is hard. That returns rows as this special kind of object, which is designed to be accessed from the HTML templates: https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L45-L71","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460970807,https://api.github.com/repos/simonw/datasette/issues/1999,1460970807,IC_kwDOBm6k_c5XFKU3,9599,simonw,2023-03-08T22:31:49Z,2023-03-08T22:33:03Z,OWNER,"For the HTML version, I need to decide where all of the stuff that happens in `async def extra_template()` is going to live. I think it's another one of those extra functions, triggered for `?_extra=context`. https://github.com/simonw/datasette/blob/96e94f9b7b2db53865e61390bcce6761727f26d8/datasette/views/table.py#L813-L912","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460943097,https://api.github.com/repos/simonw/datasette/issues/1999,1460943097,IC_kwDOBm6k_c5XFDj5,9599,simonw,2023-03-08T22:09:24Z,2023-03-08T22:09:47Z,OWNER,"The ease with which I added that `?_extra=query` feature in https://github.com/simonw/datasette/pull/1999/commits/96e94f9b7b2db53865e61390bcce6761727f26d8 made me feel really confident that this architecture is going in the right direction. ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8d3bb2c930..3e1db9c85f 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1913,6 +1913,13 @@ async def extra_request(): ""args"": request.args._data, } + async def extra_query(): + ""Details of the underlying SQL query"" + return { + ""sql"": sql, + ""params"": params, + } + async def extra_extras(): ""Available ?_extra= blocks"" return { @@ -1938,6 +1945,7 @@ async def extra_extras(): extra_primary_keys, extra_debug, extra_request, + extra_query, extra_extras, ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460916405,https://api.github.com/repos/simonw/datasette/issues/1999,1460916405,IC_kwDOBm6k_c5XE9C1,9599,simonw,2023-03-08T21:43:27Z,2023-03-08T21:43:27Z,OWNER,"Just noticed that `_json=colname` is not working, and that's because it's handled by the renderer here: https://github.com/simonw/datasette/blob/56b0758a5fbf85d01ff80a40c9b028469d7bb65f/datasette/renderer.py#L29-L40 But that's not currently being called by my new code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460907148,https://api.github.com/repos/simonw/datasette/issues/1999,1460907148,IC_kwDOBm6k_c5XE6yM,9599,simonw,2023-03-08T21:34:30Z,2023-03-08T21:34:30Z,OWNER,"I'm going to hold off on that refactor until later, when I have tests to show me if the refactor works or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460906741,https://api.github.com/repos/simonw/datasette/issues/1999,1460906741,IC_kwDOBm6k_c5XE6r1,9599,simonw,2023-03-08T21:34:08Z,2023-03-08T21:34:08Z,OWNER,"So maybe I can refactor it to look a bit more like this: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1602-L1604 One thing that's useful here is that `is_view` is handled early, like this: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L466-L472 So if I omit the `is_view` bit from the extracted function I can simplify more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469,https://api.github.com/repos/simonw/datasette/issues/1999,1460905469,IC_kwDOBm6k_c5XE6X9,9599,simonw,2023-03-08T21:32:56Z,2023-03-08T21:32:56Z,OWNER,"I'd really like to extract this ugly logic out into a helper function: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1617-L1684 I copied it in and asked ChatGPT to ""List all of the variable that are either modified or assigned to by the above code"": > Modified variables: > > offset > where_clauses > params > order_by > > Assigned variables: > > sort_value > next_by_pk_clauses > order_by (when sort or sort_desc is not None) Then I asked which variables were used as inputs, and argued with it a bit about whether it should be counting functions. Eventually got to this: > My apologies for the oversight. Here are the variables needed as input by the above code, excluding any functions: > > _next > is_view > sort > sort_desc > use_rowid > pks > order_by Note that `use_rowid` is actually defined earlier in terms of two of those other variables: https://github.com/simonw/datasette/blob/db1a88f4e17a1f50bdaa681e8beddb2276503e7c/datasette/views/table.py#L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460760116,https://api.github.com/repos/simonw/datasette/issues/1999,1460760116,IC_kwDOBm6k_c5XEW40,9599,simonw,2023-03-08T19:48:52Z,2023-03-08T19:48:52Z,OWNER,"I'm trying to get `http://127.0.0.1:8001/fixtures/compound_three_primary_keys?_next=a,d,v` to return the correct results.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1460759358,https://api.github.com/repos/simonw/datasette/issues/1999,1460759358,IC_kwDOBm6k_c5XEWs-,9599,simonw,2023-03-08T19:48:13Z,2023-03-20T18:47:12Z,OWNER,"Breaking this down into smaller steps: - [x] Get `?_next=` working - [x] Implement extensions - so `.json` is needed again for the JSON version, and anything without an extension is passed through a new code path for HTML - [ ] That HTML view should only access JSON data, which can be seen by using `.context` - this will require a lot of updates to templates (it may be necessary to still provide access to some helper functions though). This will form the basis of the ambition to fully document the template context. - [ ] Get a bunch of the existing table HTML and JSON tests to pass - [ ] Use those tests to refactor the nasty `_next` code, see https://github.com/simonw/datasette/pull/1999#issuecomment-1460905469 - [ ] Figure out how the [register_output_renderer(datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#register-output-renderer-datasette) plugin hook should work","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1421988953,https://api.github.com/repos/simonw/datasette/issues/1999,1421988953,IC_kwDOBm6k_c5UwdRZ,9599,simonw,2023-02-08T04:35:44Z,2023-02-08T05:27:48Z,OWNER,"Next step: get `?_next=...` working (it is ignored at the moment, even though the returned JSON includes the `""next""` key). Then... figure out how to render HTML and other requested formats. Then get the tests to pass!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1399343659,https://api.github.com/repos/simonw/datasette/issues/1999,1399343659,IC_kwDOBm6k_c5TaEor,9599,simonw,2023-01-21T22:19:20Z,2023-02-06T23:02:12Z,OWNER,"HTML mode needs a list of renderers so it can show links to `.geojson` etc - can do that as a hidden extra (maybe called `renderers`), repeating this code: https://github.com/simonw/datasette/blob/e4ebef082de90db4e1b8527abc0d582b7ae0bc9d/datasette/views/base.py#L477-L497","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1399341658,https://api.github.com/repos/simonw/datasette/issues/1999,1399341658,IC_kwDOBm6k_c5TaEJa,9599,simonw,2023-01-21T22:06:29Z,2023-01-21T22:07:30Z,OWNER,"Relevant: - #1101 - #1672 - #1062","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/pull/1999#issuecomment-1399341151,https://api.github.com/repos/simonw/datasette/issues/1999,1399341151,IC_kwDOBm6k_c5TaEBf,9599,simonw,2023-01-21T22:03:20Z,2023-01-21T22:03:20Z,OWNER,"I think I'm going to have to write a new view function from scratch which completely ignores the existing BaseView/DataView/TableView hierarchy. Here's what I get on the incoming request: ``` (Pdb) request.url, request.full_path, request.host, request.url_vars ('http://127.0.0.1:8001/content/repos.json', '/content/repos.json', '127.0.0.1:8001', {'database': 'content', 'table': 'repos', 'format': 'json'}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551694938,?_extra= support (draft), https://github.com/simonw/datasette/issues/1998#issuecomment-1398768399,https://api.github.com/repos/simonw/datasette/issues/1998,1398768399,IC_kwDOBm6k_c5TX4MP,9599,simonw,2023-01-20T18:19:06Z,2023-01-20T18:19:06Z,OWNER,"Simplest solution would be to ditch the `version_option` decorator and roll a custom option based on it instead, imitating what this code does: https://github.com/pallets/click/blob/7586834cab38c5592d9d6de3ee0ebe75d4353bfb/src/click/decorators.py#L413-L524","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551113681,`datasette --version` should also show the SQLite version, https://github.com/simonw/datasette/issues/1998#issuecomment-1398767813,https://api.github.com/repos/simonw/datasette/issues/1998,1398767813,IC_kwDOBm6k_c5TX4DF,9599,simonw,2023-01-20T18:18:27Z,2023-01-20T18:18:27Z,OWNER,"Fell down a bit of a rabbit hole trying to figure out how to get Click's `version_option()` to evaluate a custom message. Got this far: ```python class _VersionMessage(UserString): @property def data(self): return ""%(prog)s, version %(version)s (SQLite {})"".format( sqlite3.connect("":memory:"").execute(""select sqlite_version()"").fetchone()[0] ) @data.setter def data(self, value): pass @click.group(cls=DefaultGroup, default=""serve"", default_if_no_args=True) @click.version_option(version=__version__, message=_VersionMessage("""")) def cli(): """""" Datasette is an open source multi-tool for exploring and publishing data \b About Datasette: https://datasette.io/ Full documentation: https://docs.datasette.io/ """""" ``` But now: ``` % datasette --version %(prog)s, version %(version)s (SQLite 3.40.1) ``` I was trying to avoid running that `select sqlite_version()` thing unless the `--version` option was used.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1551113681,`datasette --version` should also show the SQLite version, https://github.com/simonw/datasette/pull/1996#issuecomment-1396933348,https://api.github.com/repos/simonw/datasette/issues/1996,1396933348,IC_kwDOBm6k_c5TQ4Lk,22429695,codecov[bot],2023-01-19T12:55:57Z,2023-01-19T12:55:57Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.11**% // Head: **92.11**% // Increases project coverage by **`+0.00%`** :tada: > Coverage data is based on head [(`96b8b7f`)](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`6a352e9`)](https://codecov.io/gh/simonw/datasette/commit/6a352e99ab988dbf8fd22a100049caa6ad33f1ec?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1996 +/- ## ======================================= Coverage 92.11% 92.11% ======================================= Files 38 38 Lines 5555 5556 +1 ======================================= + Hits 5117 5118 +1 Misses 438 438 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.87% <100.00%> (+<0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1996?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1538342965,Document custom json encoder, https://github.com/simonw/datasette/issues/1994#issuecomment-1399957897,https://api.github.com/repos/simonw/datasette/issues/1994,1399957897,IC_kwDOBm6k_c5TcamJ,201897,julienma,2023-01-23T08:21:08Z,2023-01-23T08:21:08Z,NONE,"Me too, on a M1. Not sure if it's compatible?","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 1, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1536851861,Stuck on loading screen, https://github.com/simonw/datasette/pull/1992#issuecomment-1384039611,https://api.github.com/repos/simonw/datasette/issues/1992,1384039611,IC_kwDOBm6k_c5SfsS7,22429695,codecov[bot],2023-01-16T13:12:32Z,2023-03-29T06:10:23Z,NONE,"## [Codecov](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Patch coverage has no change and project coverage change: **`-0.31`** :warning: > Comparison is base [(`c025b01`)](https://codecov.io/gh/simonw/datasette/commit/c025b0180fe344e4f28177ecd7cf388d492d3064?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.41% compared to head [(`bfaa59b`)](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.11%. > :exclamation: Current head bfaa59b differs from pull request most recent head 64dd698. Consider uploading reports for the commit 64dd698 to get more accurate results
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1992 +/- ## ========================================== - Coverage 92.41% 92.11% -0.31% ========================================== Files 38 38 Lines 5750 5555 -195 ========================================== - Hits 5314 5117 -197 - Misses 436 438 +2 ``` [see 9 files with indirect coverage changes](https://codecov.io/gh/simonw/datasette/pull/1992/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/simonw/datasette/pull/1992?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1534904478,Bump blacken-docs from 1.12.1 to 1.13.0, https://github.com/simonw/datasette/issues/1989#issuecomment-1491357104,https://api.github.com/repos/simonw/datasette/issues/1989,1491357104,IC_kwDOBm6k_c5Y5E2w,1231935,xavdid,2023-03-31T06:17:23Z,2023-03-31T06:18:05Z,NONE,"I'm running into a similar use case as pax above- I made a `nice` view that just has the data I'm interested in (which doesn't include the `id`, since it's not important in this case). But, by excluding `id` from the view, I can't do fts queries against it because the view has no `id` field to tie to `rowid`: ``` ERROR: conn=, sql = 'select time, text, permalink, num_children from nice where id in (select rowid from items_fts where items_fts match :search) limit 101', params = {'search': 'whatever'}: no such column: id ``` It works fine when I include `id` in my view, but now my `nice` view is cluttered up. Would be great to hide it permanently in the `config.json`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1531991339,Suggestion: Hiding columns, https://github.com/simonw/datasette/issues/1989#issuecomment-1402347667,https://api.github.com/repos/simonw/datasette/issues/1989,1402347667,IC_kwDOBm6k_c5TliCT,116795,pax,2023-01-24T17:48:59Z,2023-01-24T17:48:59Z,NONE,"The problem (in my particular use case) with using a VIEW is that I'd need one of the columns to be searchable – but that ([enable-fts](https://github.com/simonw/datasette-search-all)) doesn't work with views :/ __ side-suggestion: I don't know how feasible this might be, but when one column (or table) would be marked as hidden, could the _Download SQLite DB_ link take that into account? 🧐","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1531991339,Suggestion: Hiding columns, https://github.com/simonw/datasette/issues/1989#issuecomment-1397734424,https://api.github.com/repos/simonw/datasette/issues/1989,1397734424,IC_kwDOBm6k_c5TT7wY,9599,simonw,2023-01-19T23:18:45Z,2023-01-19T23:18:45Z,OWNER,"There's a `?_nocol=x` argument you can use to hide a column when you link to a table, but that won't help you if you need to hide the column for privacy reasons: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments One solution right now is to define a SQL view for the things that you DO want people to be able to see, and then use Datasette's permission system to hide the tables (`'""allow"": false` in metadata for each table) but show the views. If you want to redact specific columns there's a plugin for doing that: https://datasette.io/plugins/datasette-mask-columns This does make sense as more of a core Datasette feature though - tagging it as a feature suggestion.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1531991339,Suggestion: Hiding columns, https://github.com/simonw/datasette/issues/1988#issuecomment-1379502161,https://api.github.com/repos/simonw/datasette/issues/1988,1379502161,IC_kwDOBm6k_c5SOYhR,9599,simonw,2023-01-11T21:24:24Z,2023-01-11T21:25:05Z,OWNER,Renaming `sql()` to `template_sql()` in https://github.com/simonw/datasette-template-sql and shipping a 2.0 release of that plugin (it is currently at 1.0.2) would be a sensible short-term fix too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,Reconsider pattern where plugins could break existing template context, https://github.com/simonw/datasette/issues/1988#issuecomment-1379500131,https://api.github.com/repos/simonw/datasette/issues/1988,1379500131,IC_kwDOBm6k_c5SOYBj,9599,simonw,2023-01-11T21:22:31Z,2023-01-11T21:22:31Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/6a352e99ab988dbf8fd22a100049caa6ad33f1ec/datasette/app.py#L1138-L1149 https://github.com/simonw/datasette/blob/6a352e99ab988dbf8fd22a100049caa6ad33f1ec/datasette/app.py#L1193-L1195","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,Reconsider pattern where plugins could break existing template context, https://github.com/simonw/datasette/issues/1988#issuecomment-1379495504,https://api.github.com/repos/simonw/datasette/issues/1988,1379495504,IC_kwDOBm6k_c5SOW5Q,9599,simonw,2023-01-11T21:18:00Z,2023-01-11T21:18:25Z,OWNER,"A few options: - Ensure that the explicit template context overrides anything that plugins might do - so in this case the `sql(...)` function would not be available on that page. This would break people who have custom templates that use that function though. - Encourage naming conventions where functions and variables from plugins are less likely to interfere with existing functionality. - Completely change how templates work, so you never have a variable called `{{ sql }}` - it is always accessed via some parent object instead, such as `{{ page.sql }}`. That last option actually fits quite well with my efforts to unify template rendering with JSON (and JSON extras) so it might be the best way to address this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,Reconsider pattern where plugins could break existing template context, https://github.com/simonw/datasette/issues/1988#issuecomment-1379492499,https://api.github.com/repos/simonw/datasette/issues/1988,1379492499,IC_kwDOBm6k_c5SOWKT,9599,simonw,2023-01-11T21:15:01Z,2023-01-11T21:20:14Z,OWNER,"This is tricky. `datasette-template-sql` registers a custom `sql()` function, but @cldellow spotted a situation where this would interfere with an existing `{{ sql }}` variable in a template context! https://github.com/simonw/datasette-template-sql/blob/70483fd94bb0349761830e37c9ab928e8966841a/datasette_template_sql/__init__.py ```python @hookimpl def extra_template_vars(datasette): async def execute_sql(sql, args=None, database=None): db = datasette.get_database(database) return (await db.execute(sql, args)).rows return {""sql"": execute_sql} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529707837,Reconsider pattern where plugins could break existing template context, https://github.com/simonw/datasette/issues/1987#issuecomment-1379313564,https://api.github.com/repos/simonw/datasette/issues/1987,1379313564,IC_kwDOBm6k_c5SNqec,9599,simonw,2023-01-11T18:29:25Z,2023-01-11T18:29:25Z,OWNER,"Fixed: - https://docs.datasette.io/en/latest/installation.html#using-pip - https://docs.datasette.io/en/stable/installation.html#using-pip - https://docs.datasette.io/en/0.64.1/installation.html#using-pip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,installpython3.com is now a spam website, https://github.com/simonw/datasette/issues/1987#issuecomment-1379311498,https://api.github.com/repos/simonw/datasette/issues/1987,1379311498,IC_kwDOBm6k_c5SNp-K,9599,simonw,2023-01-11T18:27:30Z,2023-01-11T18:27:46Z,OWNER,"I'm going to leave the bad link in the old versions, because I can't figure out a way to update them using ReadTheDocs - shipping 0.64.1 will at least mean that both `/latest/` and `/stable/` have a good link.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,installpython3.com is now a spam website, https://github.com/simonw/datasette/issues/1987#issuecomment-1379282756,https://api.github.com/repos/simonw/datasette/issues/1987,1379282756,IC_kwDOBm6k_c5SNi9E,9599,simonw,2023-01-11T18:04:22Z,2023-01-11T18:04:22Z,OWNER,"That change is now live on https://docs.datasette.io/en/latest/installation.html#using-pip I may need to ship a new stable dot-release to update the `/en/stable/` page though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,installpython3.com is now a spam website, https://github.com/simonw/datasette/issues/1987#issuecomment-1379276484,https://api.github.com/repos/simonw/datasette/issues/1987,1379276484,IC_kwDOBm6k_c5SNhbE,9599,simonw,2023-01-11T18:00:35Z,2023-01-11T18:00:35Z,OWNER,"Next challenge: how to best deploy this. Ideally I'd like to fix all of the older versions of the docs too - pages like https://docs.datasette.io/en/0.59.4/installation.html#using-pip Not sure if ReadTheDocs has a mechanism for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1529452371,installpython3.com is now a spam website, https://github.com/simonw/datasette/issues/1985#issuecomment-1378253346,https://api.github.com/repos/simonw/datasette/issues/1985,1378253346,IC_kwDOBm6k_c5SJnoi,9599,simonw,2023-01-11T05:18:05Z,2023-01-11T05:18:05Z,OWNER,`Datasette(path)` should either work or it should return an easily debugged error message.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1528448642,Don't let Datasette(path) without a list cause weird errors, https://github.com/simonw/datasette/pull/1984#issuecomment-1376619691,https://api.github.com/repos/simonw/datasette/issues/1984,1376619691,IC_kwDOBm6k_c5SDYyr,9599,simonw,2023-01-10T02:02:16Z,2023-01-10T02:02:16Z,OWNER,Preview at https://datasette--1984.org.readthedocs.build/en/1984/ looks good to me.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1526635374,Upgrade Sphinx, https://github.com/simonw/datasette/issues/1983#issuecomment-1375810027,https://api.github.com/repos/simonw/datasette/issues/1983,1375810027,IC_kwDOBm6k_c5SATHr,25778,eyeseast,2023-01-09T15:35:58Z,2023-01-09T15:35:58Z,CONTRIBUTOR,"Yes please, and thank you. I realized I was maybe getting myself in trouble using that, but I think it's a good way to standardize JSON handling.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,Make CustomJSONEncoder a documented public API, https://github.com/simonw/datasette/issues/1983#issuecomment-1375797856,https://api.github.com/repos/simonw/datasette/issues/1983,1375797856,IC_kwDOBm6k_c5SAQJg,9599,simonw,2023-01-09T15:29:34Z,2023-01-09T15:29:34Z,OWNER,"I had forgotten this had special handling for cursors! The default representation of a row as a tuple rather than a dictionary might not be the right thing to do based on: - #782 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,Make CustomJSONEncoder a documented public API, https://github.com/simonw/datasette/issues/1983#issuecomment-1375795441,https://api.github.com/repos/simonw/datasette/issues/1983,1375795441,IC_kwDOBm6k_c5SAPjx,9599,simonw,2023-01-09T15:27:53Z,2023-01-09T15:27:53Z,OWNER,https://github.com/simonw/datasette/blob/7b48664d751e59222c01c97a2739d53d6363bdda/datasette/utils/__init__.py#L163-L178,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525815985,Make CustomJSONEncoder a documented public API, https://github.com/simonw/datasette/pull/1982#issuecomment-1376620851,https://api.github.com/repos/simonw/datasette/issues/1982,1376620851,IC_kwDOBm6k_c5SDZEz,49699333,dependabot[bot],2023-01-10T02:03:18Z,2023-01-10T02:03:18Z,CONTRIBUTOR,"Looks like sphinx is up-to-date now, so this is no longer needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1525560504,Bump sphinx from 5.3.0 to 6.1.2, https://github.com/simonw/datasette/issues/1981#issuecomment-1375142205,https://api.github.com/repos/simonw/datasette/issues/1981,1375142205,IC_kwDOBm6k_c5R9wE9,9599,simonw,2023-01-09T06:05:44Z,2023-01-09T06:05:44Z,OWNER,"I want the labels to take as much space as they need, but line up with each other. If they are more than half the width of the page then the form fields should break to another line.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524983536,Canned query field labels truncated, https://github.com/simonw/datasette/issues/1980#issuecomment-1375062405,https://api.github.com/repos/simonw/datasette/issues/1980,1375062405,IC_kwDOBm6k_c5R9cmF,9599,simonw,2023-01-09T03:23:53Z,2023-01-09T03:23:53Z,OWNER,"Some potential solutions: - Make it so the primary key is _always_ sortable - not a bad idea, the `sortable_columns` thing was designed to protect against sorting GIANT tables by a non-indexed column, and the primary key always has an index - Make sure that ` ``` But `id` is not a valid sort option any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524867951,"""Cannot sort table by id"" when sortable_columns is used", https://github.com/simonw/datasette/issues/1979#issuecomment-1374659874,https://api.github.com/repos/simonw/datasette/issues/1979,1374659874,IC_kwDOBm6k_c5R76Ui,9599,simonw,2023-01-07T23:59:11Z,2023-01-07T23:59:11Z,OWNER,I back-ported this fix to `0.63.x` as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,More useful error message if enable_load_extension is not available, https://github.com/simonw/datasette/issues/1979#issuecomment-1374659233,https://api.github.com/repos/simonw/datasette/issues/1979,1374659233,IC_kwDOBm6k_c5R76Kh,9599,simonw,2023-01-07T23:56:25Z,2023-01-07T23:56:25Z,OWNER,"``` % datasette --load-extension foo Error: Your Python installation does not have the ability to load SQLite extensions. More information: https://datasette.io/help/extensions ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,More useful error message if enable_load_extension is not available, https://github.com/simonw/datasette/issues/1979#issuecomment-1374658468,https://api.github.com/repos/simonw/datasette/issues/1979,1374658468,IC_kwDOBm6k_c5R75-k,9599,simonw,2023-01-07T23:53:41Z,2023-01-07T23:53:41Z,OWNER,https://datasette.io/help/extensions now redirects to this new section of documentation: https://docs.datasette.io/en/latest/installation.html#a-note-about-extensions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,More useful error message if enable_load_extension is not available, https://github.com/simonw/datasette/issues/1979#issuecomment-1374657204,https://api.github.com/repos/simonw/datasette/issues/1979,1374657204,IC_kwDOBm6k_c5R75q0,9599,simonw,2023-01-07T23:48:42Z,2023-01-07T23:48:42Z,OWNER,Once I ship the next release I should change that `/help/extensions` link to go to `/stable/` and not `/latest/`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,More useful error message if enable_load_extension is not available, https://github.com/simonw/datasette/issues/1979#issuecomment-1374582375,https://api.github.com/repos/simonw/datasette/issues/1979,1374582375,IC_kwDOBm6k_c5R7nZn,9599,simonw,2023-01-07T19:22:39Z,2023-01-07T19:22:39Z,OWNER,"This helps: ```diff diff --git a/datasette/cli.py b/datasette/cli.py index 2b61292b..ea98879c 100644 --- a/datasette/cli.py +++ b/datasette/cli.py @@ -4,13 +4,15 @@ import click from click import formatting from click.types import CompositeParamType from click_default_group import DefaultGroup +import functools import json import os import pathlib import shutil from subprocess import call -import sys from runpy import run_module +import sys +import textwrap import webbrowser from .app import ( OBSOLETE_SETTINGS, @@ -126,7 +128,7 @@ class Setting(CompositeParamType): def sqlite_extensions(fn): - return click.option( + fn = click.option( ""sqlite_extensions"", ""--load-extension"", type=LoadExtension(), @@ -134,6 +136,25 @@ def sqlite_extensions(fn): multiple=True, help=""Path to a SQLite extension to load, and optional entrypoint"", )(fn) + # Wrap it in a custom error handler + @functools.wraps(fn) + def wrapped(*args, **kwargs): + try: + return fn(*args, **kwargs) + except AttributeError as e: + if ""enable_load_extension"" in str(e): + raise click.ClickException( + textwrap.dedent( + """""" + Your Python installation does not have the ability to load SQLite extensions. + + More information: https://docs.datasette.io/en/stable/installation.html#extensions + """""" + ).strip() + ) + raise + + return wrapped @click.group(cls=DefaultGroup, default=""serve"", default_if_no_args=True) ``` Need to write help for that to link to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1524076587,More useful error message if enable_load_extension is not available, https://github.com/simonw/datasette/issues/1978#issuecomment-1375708725,https://api.github.com/repos/simonw/datasette/issues/1978,1375708725,IC_kwDOBm6k_c5R_6Y1,25778,eyeseast,2023-01-09T14:30:00Z,2023-01-09T14:30:00Z,CONTRIBUTOR,Totally missed that issue. I can close this as a duplicate.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522778923,Document datasette.urls.row and row_blob, https://github.com/simonw/datasette/issues/1978#issuecomment-1374916593,https://api.github.com/repos/simonw/datasette/issues/1978,1374916593,IC_kwDOBm6k_c5R84_x,9599,simonw,2023-01-08T20:06:16Z,2023-01-08T20:06:16Z,OWNER,"Related issue - I'm not 100% settled on the design for these yet: - https://github.com/simonw/datasette/issues/1048#issuecomment-1179757912","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522778923,Document datasette.urls.row and row_blob, https://github.com/simonw/datasette/pull/1977#issuecomment-1375596856,https://api.github.com/repos/simonw/datasette/issues/1977,1375596856,IC_kwDOBm6k_c5R_fE4,49699333,dependabot[bot],2023-01-09T13:06:14Z,2023-01-09T13:06:14Z,CONTRIBUTOR,Superseded by #1982.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522552817,Bump sphinx from 5.3.0 to 6.1.1, https://github.com/simonw/datasette/pull/1977#issuecomment-1373600402,https://api.github.com/repos/simonw/datasette/issues/1977,1373600402,IC_kwDOBm6k_c5R33qS,22429695,codecov[bot],2023-01-06T13:09:51Z,2023-01-06T13:09:51Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1977?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.15**% // Head: **92.15**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`a4eebee`)](https://codecov.io/gh/simonw/datasette/pull/1977?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`c41278b`)](https://codecov.io/gh/simonw/datasette/commit/c41278b46f7936b4b1a8a14bf285bed82c81c609?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1977 +/- ## ======================================= Coverage 92.15% 92.15% ======================================= Files 38 38 Lines 5542 5542 ======================================= Hits 5107 5107 Misses 435 435 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1977?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1522552817,Bump sphinx from 5.3.0 to 6.1.1, https://github.com/simonw/datasette/pull/1976#issuecomment-1373592231,https://api.github.com/repos/simonw/datasette/issues/1976,1373592231,IC_kwDOBm6k_c5R31qn,49699333,dependabot[bot],2023-01-06T13:02:15Z,2023-01-06T13:02:15Z,CONTRIBUTOR,Superseded by #1977.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1520712722,Bump sphinx from 5.3.0 to 6.1.0, https://github.com/simonw/datasette/pull/1976#issuecomment-1372203113,https://api.github.com/repos/simonw/datasette/issues/1976,1372203113,IC_kwDOBm6k_c5Ryihp,22429695,codecov[bot],2023-01-05T13:15:04Z,2023-01-05T13:15:04Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1976?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.15**% // Head: **92.15**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`af76abc`)](https://codecov.io/gh/simonw/datasette/pull/1976?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`c41278b`)](https://codecov.io/gh/simonw/datasette/commit/c41278b46f7936b4b1a8a14bf285bed82c81c609?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1976 +/- ## ======================================= Coverage 92.15% 92.15% ======================================= Files 38 38 Lines 5542 5542 ======================================= Hits 5107 5107 Misses 435 435 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1976?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1520712722,Bump sphinx from 5.3.0 to 6.1.0, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1539109587,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1539109587,IC_kwDOCGYnMM5bvPLT,9599,simonw,2023-05-08T22:00:46Z,2023-05-08T22:00:46Z,OWNER,"> Hey, isn't this essentially the same issue as #448 ? Yes it is, good catch!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1421571810,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1421571810,IC_kwDOCGYnMM5Uu3bi,167893,mcarpenter,2023-02-07T22:43:09Z,2023-02-07T22:43:09Z,CONTRIBUTOR,"Hey, isn't this essentially the same issue as #448 ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/sqlite-utils/issues/520#issuecomment-1369333759,https://api.github.com/repos/simonw/sqlite-utils/issues/520,1369333759,IC_kwDOCGYnMM5Rnl__,9599,simonw,2023-01-03T02:23:43Z,2023-01-03T02:23:43Z,OWNER,"The documentation here does at least say the following: https://sqlite-utils.datasette.io/en/3.30/python-api.html#reading-rows-from-a-file > - **fp** (*BinaryIO*) -- a file-like object containing binary data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516644980,rows_from_file() raises confusing error if file-like object is not in binary mode, https://github.com/simonw/datasette/pull/1974#issuecomment-1372188571,https://api.github.com/repos/simonw/datasette/issues/1974,1372188571,IC_kwDOBm6k_c5Rye-b,49699333,dependabot[bot],2023-01-05T13:02:40Z,2023-01-05T13:02:40Z,CONTRIBUTOR,Superseded by #1976.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516376583,Bump sphinx from 5.3.0 to 6.0.0, https://github.com/simonw/datasette/pull/1974#issuecomment-1368935738,https://api.github.com/repos/simonw/datasette/issues/1974,1368935738,IC_kwDOBm6k_c5RmE06,22429695,codecov[bot],2023-01-02T13:12:30Z,2023-01-02T13:12:30Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1974?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.10**% // Head: **92.10**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`1e213a3`)](https://codecov.io/gh/simonw/datasette/pull/1974?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`572bdb5`)](https://codecov.io/gh/simonw/datasette/commit/572bdb5b8095bbdc5ec2aa95400055e49ea028cb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1974 +/- ## ======================================= Coverage 92.10% 92.10% ======================================= Files 38 38 Lines 5538 5538 ======================================= Hits 5101 5101 Misses 437 437 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
[:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1974?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1516376583,Bump sphinx from 5.3.0 to 6.0.0, https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464796494,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24,1464796494,IC_kwDOC8tyDs5XTwVO,956433,Mjboothaus,2023-03-11T02:23:42Z,2023-03-11T02:23:42Z,NONE,@simonw - maybe put in some error handling to trap for poorly formed XML (from Apple engineers) so that it suggests that there are problems with export.zip rather than odd looking Python errors :),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515883470,DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 , https://github.com/dogsheep/healthkit-to-sqlite/issues/24#issuecomment-1464786643,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/24,1464786643,IC_kwDOC8tyDs5XTt7T,956433,Mjboothaus,2023-03-11T02:01:27Z,2023-03-11T02:01:27Z,NONE,Thanks for reporting this and providing a solution -- I was puzzled by this error when I revisited my walking data and experienced this issues. I haven't tried the fix yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515883470,DOC: xml.etree.ElementTree.ParseError due to healthkit version 12 , https://github.com/simonw/datasette/issues/1973#issuecomment-1407523547,https://api.github.com/repos/simonw/datasette/issues/1973,1407523547,IC_kwDOBm6k_c5T5Rrb,193185,cldellow,2023-01-29T00:40:31Z,2023-01-29T00:40:31Z,CONTRIBUTOR,"A +1 for switching to `CustomRow`: I think you currently only get a `CustomRow` if the result set had a column that was an fkey ([this code](https://github.com/simonw/datasette/blob/3c352b7132ef09b829abb69a0da0ad00be5edef9/datasette/views/table.py#L667-L682)) Otherwise you get vanilla `sqlite3.Row`s, which will fail if you try to access `.columns` or lookup the cell by name, which surprised me recently","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,render_cell plugin hook's row object is not a sqlite.Row, https://github.com/simonw/datasette/issues/1973#issuecomment-1369044959,https://api.github.com/repos/simonw/datasette/issues/1973,1369044959,IC_kwDOBm6k_c5Rmfff,193185,cldellow,2023-01-02T15:41:40Z,2023-01-02T15:41:40Z,CONTRIBUTOR,"Thanks for the response! Yes, it does seem like a pretty nice developer experience--both the automagical labelling of fkeys, and the ability to index the row by column name in addition to column index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,render_cell plugin hook's row object is not a sqlite.Row, https://github.com/simonw/datasette/issues/1973#issuecomment-1368936146,https://api.github.com/repos/simonw/datasette/issues/1973,1368936146,IC_kwDOBm6k_c5RmE7S,9599,simonw,2023-01-02T13:13:03Z,2023-01-02T13:13:03Z,OWNER,It's quite handy that the label for foreign keys is made available to the custom render function here - if that was properly documented (and maybe had a slightly more intuitive API) I think it might be a good solution here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,render_cell plugin hook's row object is not a sqlite.Row, https://github.com/simonw/datasette/issues/1973#issuecomment-1368935320,https://api.github.com/repos/simonw/datasette/issues/1973,1368935320,IC_kwDOBm6k_c5RmEuY,9599,simonw,2023-01-02T13:12:00Z,2023-01-02T13:12:00Z,OWNER,"Thanks for catching this! Definitely a bug. This might be a good opportunity to reconsider how his should work though. It's possible that Datasette will eventually support alternative database backends - PostgreSQL, DuckDB and ClickHouse are all on my radar here. If that happens, this plugin hook returning a SQLite Row wouldn't make sense. Some kind of thing abstraction layer - maybe CustomRow, maybe something a little bit better - might be a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515815014,render_cell plugin hook's row object is not a sqlite.Row, https://github.com/simonw/datasette/issues/1971#issuecomment-1368266904,https://api.github.com/repos/simonw/datasette/issues/1971,1368266904,IC_kwDOBm6k_c5RjhiY,9599,simonw,2022-12-31T19:09:48Z,2022-12-31T19:09:48Z,OWNER,"For the moment I'm going to pin to Sphinx 5.3.0 Note that I got this warning there: ``` [sphinx-autobuild] > sphinx-build -b html /Users/simon/Dropbox/Development/datasette/docs /Users/simon/Dropbox/Development/datasette/docs/_build Running Sphinx v5.3.0 loading pickled environment... done WARNING: extlinks: Sphinx-6.0 will require a caption string to contain exactly one '%s' and all other '%' need to be escaped as '%%'. ``` So I should probably fix that in advance of the 6.0 upgrade.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515185383,Upgrade for Sphinx 6.0 (once Furo has support for it), https://github.com/simonw/datasette/issues/1971#issuecomment-1368266658,https://api.github.com/repos/simonw/datasette/issues/1971,1368266658,IC_kwDOBm6k_c5Rjhei,9599,simonw,2022-12-31T19:07:19Z,2022-12-31T19:07:19Z,OWNER,"So it looks like Furo doesn't work with Sphinx 6.0 yet - there' s a fix on `main` but it hasn't been released: - https://github.com/pradyunsg/furo/discussions/569","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515185383,Upgrade for Sphinx 6.0 (once Furo has support for it), https://github.com/simonw/datasette/issues/1971#issuecomment-1368266457,https://api.github.com/repos/simonw/datasette/issues/1971,1368266457,IC_kwDOBm6k_c5RjhbZ,9599,simonw,2022-12-31T19:05:26Z,2022-12-31T19:05:26Z,OWNER,"Ran this locally: pip install -U Sphinx Got these warnings: ``` Installing collected packages: Sphinx Attempting uninstall: Sphinx Found existing installation: Sphinx 5.1.1 Uninstalling Sphinx-5.1.1: Successfully uninstalled Sphinx-5.1.1 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. sphinx-basic-ng 0.0.1a12 requires sphinx<6.0,>=4.0, but you have sphinx 6.0.0 which is incompatible. furo 2022.6.21 requires sphinx<6.0,>=4.0, but you have sphinx 6.0.0 which is incompatible. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515185383,Upgrade for Sphinx 6.0 (once Furo has support for it), https://github.com/simonw/datasette/issues/1970#issuecomment-1368268233,https://api.github.com/repos/simonw/datasette/issues/1970,1368268233,IC_kwDOBm6k_c5Rjh3J,9599,simonw,2022-12-31T19:22:58Z,2022-12-31T19:22:58Z,OWNER,"https://latest.datasette.io/_internal/databases shows the fix now: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515182998,"Path ""None"" in _internal database table", https://github.com/simonw/datasette/issues/1970#issuecomment-1368264882,https://api.github.com/repos/simonw/datasette/issues/1970,1368264882,IC_kwDOBm6k_c5RjhCy,9599,simonw,2022-12-31T18:52:01Z,2022-12-31T18:52:01Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/e03aed00026cc2e59c09ca41f69a247e1a85cc89/datasette/app.py#L418-L424,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1515182998,"Path ""None"" in _internal database table", https://github.com/simonw/sqlite-utils/pull/519#issuecomment-1539058795,https://api.github.com/repos/simonw/sqlite-utils/issues/519,1539058795,IC_kwDOCGYnMM5bvCxr,9599,simonw,2023-05-08T21:12:52Z,2023-05-08T21:12:52Z,OWNER,"This is a really neat fix, thank you.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1505568103,Fixes breaking DEFAULT values, https://github.com/simonw/datasette/pull/1967#issuecomment-1368267484,https://api.github.com/repos/simonw/datasette/issues/1967,1368267484,IC_kwDOBm6k_c5Rjhrc,9599,simonw,2022-12-31T19:15:50Z,2022-12-31T19:15:50Z,OWNER,"My Firefox tab before: And after: ","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1503010009,Add favicon to documentation, https://github.com/simonw/datasette/pull/1967#issuecomment-1368265354,https://api.github.com/repos/simonw/datasette/issues/1967,1368265354,IC_kwDOBm6k_c5RjhKK,9599,simonw,2022-12-31T18:56:35Z,2022-12-31T18:56:35Z,OWNER,"Thanks for this! Datasette has a favicon in this file: https://github.com/simonw/datasette/blob/2b4a8aa0833068da25072b6247bf349949ca00ee/datasette/static/favicon.png - so I'll copy that over and use it here as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1503010009,Add favicon to documentation, https://github.com/simonw/datasette/issues/1966#issuecomment-1368267406,https://api.github.com/repos/simonw/datasette/issues/1966,1368267406,IC_kwDOBm6k_c5RjhqO,9599,simonw,2022-12-31T19:15:10Z,2022-12-31T19:15:10Z,OWNER,"It's working again now - not sure what happened there, I was paying for Heroku for that already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1501900064,Broken link to live demo in Getting started docs, https://github.com/simonw/datasette/pull/1965#issuecomment-1356827218,https://api.github.com/repos/simonw/datasette/issues/1965,1356827218,IC_kwDOBm6k_c5Q34pS,9599,simonw,2022-12-18T16:01:36Z,2022-12-18T16:01:36Z,OWNER,Will link to this from my TIL shortly.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1501843596,Detect server start/stop more reliably., https://github.com/simonw/datasette/pull/1965#issuecomment-1356827167,https://api.github.com/repos/simonw/datasette/issues/1965,1356827167,IC_kwDOBm6k_c5Q34of,9599,simonw,2022-12-18T16:01:22Z,2022-12-18T16:01:22Z,OWNER,"This is great, thank you!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1501843596,Detect server start/stop more reliably., https://github.com/simonw/datasette/issues/1964#issuecomment-1356697705,https://api.github.com/repos/simonw/datasette/issues/1964,1356697705,IC_kwDOBm6k_c5Q3ZBp,9599,simonw,2022-12-18T06:37:23Z,2022-12-18T06:37:23Z,OWNER,"I'm certain the two other cog menus (the app menu on the right of the nav bar and the column action menus) have the same problem. Would be great to figure out the right ARIA attributes for these too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1501778647,Cog menu is not keyboard accessible (also no ARIA), https://github.com/simonw/datasette/issues/1963#issuecomment-1356652057,https://api.github.com/repos/simonw/datasette/issues/1963,1356652057,IC_kwDOBm6k_c5Q3N4Z,9599,simonw,2022-12-18T03:23:22Z,2022-12-18T03:23:22Z,OWNER,https://pypi.org/project/datasette/0.63.3/ is released.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1501713288,0.63.3 bugfix release, https://github.com/simonw/datasette/issues/1963#issuecomment-1356651943,https://api.github.com/repos/simonw/datasette/issues/1963,1356651943,IC_kwDOBm6k_c5Q3N2n,9599,simonw,2022-12-18T03:23:03Z,2022-12-18T03:23:03Z,OWNER,"Oh that's annoying... every step in publish succeeded except the static docs one: https://github.com/simonw/datasette/actions/runs/3723015082/jobs/6314292722 This means the documentation database used to update the search engine on https://datasette.io/ won't reflect the very latest changelog. I'm OK with that - I'll fix this workflow so that next time I publish a release this will work correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1501713288,0.63.3 bugfix release, https://github.com/simonw/datasette/issues/1962#issuecomment-1355691595,https://api.github.com/repos/simonw/datasette/issues/1962,1355691595,IC_kwDOBm6k_c5QzjZL,9599,simonw,2022-12-16T21:53:45Z,2022-12-16T21:55:29Z,OWNER,"Also need an alternative mechanism to this convenience for getting CSRF tokens before a POST: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/datasette/utils/testing.py#L90-L103 One option would be adding that mechanism to `datasette.client.post(...)` - as a `_csrf_from=` parameter (with an underscore because it's mainly intended for use in tests, though perhaps that's a weird convention that I shouldn't introduce).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1500636982,"Alternative, async-friendly pattern for `make_app_client()` and similar - fully retire `TestClient`", https://github.com/simonw/datasette/pull/1960#issuecomment-1356478575,https://api.github.com/repos/simonw/datasette/issues/1960,1356478575,IC_kwDOBm6k_c5Q2jhv,9599,simonw,2022-12-17T21:47:48Z,2022-12-17T21:47:48Z,OWNER,"Stick a twig in it, this will do for the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1356476886,https://api.github.com/repos/simonw/datasette/issues/1960,1356476886,IC_kwDOBm6k_c5Q2jHW,9599,simonw,2022-12-17T21:37:05Z,2022-12-17T21:37:05Z,OWNER,"I think this test may be to blame: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/test_plugins.py#L950-L972 It's over-riding `_metadata_local` and then failing to set it back to original in a `finally:` block at the end.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1356476583,https://api.github.com/repos/simonw/datasette/issues/1960,1356476583,IC_kwDOBm6k_c5Q2jCn,9599,simonw,2022-12-17T21:34:51Z,2022-12-17T21:34:51Z,OWNER,"These are all the places that tests touch `_metadata_local` at the moment: ``` (venv) root@76a81d2417f5:/tmp/datasette/tests# rg _metadata_local test_facets.py 596: ds._metadata_local = { 605: ds._metadata_local[""databases""][""test_facet_size""][""tables""][""neighbourhoods""][ test_permissions.py 62: padlock_client.ds._metadata_local[""allow""] = allow 77: del padlock_client.ds._metadata_local[""allow""] 522: cascade_app_client.ds._metadata_local = updated_metadata 533: cascade_app_client.ds._metadata_local = previous_metadata 549: previous_metadata = cascade_app_client.ds._metadata_local 551: cascade_app_client.ds._metadata_local = metadata 566: cascade_app_client.ds._metadata_local = previous_metadata 842: perms_ds._metadata_local = updated_metadata 849: perms_ds._metadata_local = previous_metadata test_html.py 1114: orig = ds_client.ds._metadata_local 1115: ds_client.ds._metadata_local = metadata 1123: ds_client.ds._metadata_local = orig test_plugins.py 1034: ds_client.ds._metadata_local = { ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1356056018,https://api.github.com/repos/simonw/datasette/issues/1960,1356056018,IC_kwDOBm6k_c5Q08XS,9599,simonw,2022-12-17T05:33:26Z,2022-12-17T05:33:26Z,OWNER,"I think I've found the problem. The failing test is this one: `paginated_view.json-201-9` That's this example: https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L179-L180 Why is it expected to take 9 pages and not 5, when the definition of that view is this: ```sql CREATE VIEW paginated_view AS SELECT content, '- ' || content || ' -' AS content_extra FROM no_primary_key; ``` Because `paginated_view` has extra configuration in `metadata.json`: https://github.com/simonw/datasette/blob/5ee954e34b6eb762ccecbdb2be0791d0166fd19c/tests/fixtures.py#L357 So this test is showing that `metadata` can be used to set an alternative page size for a view. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1356038242,https://api.github.com/repos/simonw/datasette/issues/1960,1356038242,IC_kwDOBm6k_c5Q04Bi,9599,simonw,2022-12-17T04:56:22Z,2022-12-17T04:57:04Z,OWNER,"May have spotted the problem with that `test_paginate_tables_and_views` test: ``` (Pdb) path, expected_rows, expected_pages (None, 201, 9) ``` `path` should not be `None` here: https://github.com/simonw/datasette/blob/cede1efeedbc3d928397d53d5a1611eecc598fde/tests/test_table_api.py#L175-L212 No that's not it either - `path` is reassigned on purpose.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355823260,https://api.github.com/repos/simonw/datasette/issues/1960,1355823260,IC_kwDOBm6k_c5Q0Dic,9599,simonw,2022-12-16T23:36:07Z,2022-12-16T23:36:07Z,OWNER,"I ran `pytest --pdb -x` to drop into the debugger on the first failing test. ``` assert expected_rows == len(fetched) > assert expected_pages == count E assert 9 == 5 ``` That's in `test_paginate_tables_and_views()`. On a hunch, I checked the current settings: ``` (Pdb) ds_client.ds._settings {'default_page_size': 50, 'max_returned_rows': 100, 'max_insert_rows': 100, 'num_sql_threads': 1, 'sql_time_limit_ms': 200, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'allow_facet': True, 'allow_download': True, 'allow_signed_tokens': True, 'max_signed_tokens_ttl': 0, 'suggest_facets': True, 'default_cache_ttl': 5, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'trace_debug': False, 'base_url': '/'} ``` Looks like something changed `default_page_size` to 50 and forgot to change it back!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355685828,https://api.github.com/repos/simonw/datasette/issues/1960,1355685828,IC_kwDOBm6k_c5Qzh_E,9599,simonw,2022-12-16T21:50:01Z,2022-12-16T21:50:01Z,OWNER,"Looks like that `@pytest.mark.ds_client` mark I've been using isn't necessary - I added that so I could easily run `pytest -m ds_client` to execute all tests that I had ported to the new feature, but actually this achieves the same thing: pytest -k ds_client So I'm going to remove the mark.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355478743,https://api.github.com/repos/simonw/datasette/issues/1960,1355478743,IC_kwDOBm6k_c5QyvbX,9599,simonw,2022-12-16T19:27:12Z,2022-12-16T19:27:12Z,OWNER,"Bad news: they're definitely caused by tests that are subtly affected by other tests. This passes without errors: pytest -k test_paginate_tables_and_views","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client, https://github.com/simonw/datasette/pull/1960#issuecomment-1355475671,https://api.github.com/repos/simonw/datasette/issues/1960,1355475671,IC_kwDOBm6k_c5QyurX,9599,simonw,2022-12-16T19:26:08Z,2022-12-16T19:26:08Z,OWNER,"Great news! The test failures I got running on my laptop (with that fresh Ubuntu Docker image) look like they match the failures I saw in CI: ``` ======== short test summary info ======== FAILED tests/test_table_api.py::test_paginate_tables_and_views[/fixtures/paginated_view.json-201-9] - assert 9 == 5 FAILED tests/test_table_api.py::test_sortable_columns_metadata - KeyError: 'error' FAILED tests/test_table_api.py::test_searchable_views[/fixtures/searchable_view_configured_by_metadata.json?_search=weasel-expected_rows0] - AssertionError: assert [[2, 'terry d...sel', 'puma']] == [[1, 'barry c...sel', 'puma']] FAILED tests/test_table_api.py::test_unit_filters - KeyError: 'distance' FAILED tests/test_table_api.py::test_custom_query_with_unicode_characters - AssertionError: assert {'error': 'Ta...'title': None} == [{'id': 1, 'n...n Francisco'}] FAILED tests/test_table_html.py::test_sort_links - AssertionError: assert [{'a_href': N...', ...}}, ...] == [{'a_href': N...', ...}}, ...] FAILED tests/test_table_html.py::test_table_html_no_primary_key - AttributeError: 'NoneType' object has no attribute 'strip' FAILED tests/test_table_html.py::test_table_html_foreign_key_custom_label_column - assert [['']] == [['']] FAILED tests/test_table_html.py::test_view_html - assert upper_content is None FAILED tests/test_table_html.py::test_table_metadata - AssertionError: assert 'This <em&...t; is escaped' == 'simple_primary_key' FAILED tests/test_table_html.py::test_metadata_sort - AssertionError: assert ['id', 'name\xa0▼'] == ['id\xa0▼', 'name'] FAILED tests/test_table_html.py::test_metadata_sort_desc - AssertionError: assert ['pk\xa0▲', 'name'] == ['pk\xa0▼', 'name'] FAILED tests/test_table_html.py::test_column_metadata - AttributeError: 'NoneType' object has no attribute 'findAll' ======== 13 failed, 1279 passed, 3 skipped, 57 warnings in 572.40s (0:09:32) ======== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1499150951,Port as many tests as possible to async def tests against ds_client,
\xa0bb\xa0\xa01