html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/datasette/pull/1893#issuecomment-1316256386,https://api.github.com/repos/simonw/datasette/issues/1893,1316256386,IC_kwDOBm6k_c5OdHqC,95570,bgrins,2022-11-16T03:18:06Z,2022-11-16T03:18:06Z,CONTRIBUTOR,"> If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values. This version ""sort of"" works when on the main database page where the template passes the relevant data https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25 by doing this and passing that into the `schema` object: ``` let TABLES_DATA = []; {% if tables is defined %} TABLES_DATA = {{ tables | tojson(indent=2) }}; {% endif %} // Turn into an object, shaped like https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L27. const TABLES_SCHEMA = Object.fromEntries( new Map( TABLES_DATA.map((table) => { return [table.name, table.columns]; }) ).entries() ); ``` But there are a number of papercuts with it - it's not escaping table names with spaces (likely be fixable from the data being passed into the view) but mainly it doesn't seem to autocomplete columns. I think it might only want to do it when you first type the table name from my read of https://github.com/codemirror/lang-sql/blob/ebf115fffdbe07f91465ccbd82868c587f8182bc/test/test-complete.ts#L37. It's possible I'm just passing something wrong, but it may end up being something that needs feature work upstream. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316253186,https://api.github.com/repos/simonw/datasette/issues/1893,1316253186,IC_kwDOBm6k_c5OdG4C,9599,simonw,2022-11-16T03:16:36Z,2022-11-16T03:16:36Z,OWNER,Yeah I haven't written this down anywhere but Datasette definitely has an undocumented preference for lower-case SQL.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1890#issuecomment-1316242752,https://api.github.com/repos/simonw/datasette/issues/1890,1316242752,IC_kwDOBm6k_c5OdEVA,9599,simonw,2022-11-16T03:10:52Z,2022-11-16T03:12:47Z,OWNER,"https://bugs.webkit.org/show_bug.cgi?id=201768 - "" Datalist option's label not used"" - marked as RESOLVED FIXED on March 31st 2020. The commit: https://trac.webkit.org/changeset/259330/webkit And here's the test mirrored on GitHub: https://cs.github.com/qtwebkit/webkit-mirror/blob/cc3fcd0b4bad1f7cf77c26e34aa01d16618d6d5e/LayoutTests/fast/forms/datalist/datalist-option-labels.html?q=datalist-option-labels.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/pull/1893#issuecomment-1316243602,https://api.github.com/repos/simonw/datasette/issues/1893,1316243602,IC_kwDOBm6k_c5OdEiS,95570,bgrins,2022-11-16T03:11:46Z,2022-11-16T03:11:46Z,CONTRIBUTOR,Was just reviewing the SQL options and there's an [upperCaseKeywords](https://github.com/codemirror/lang-sql#user-content-sqlconfig.uppercasekeywords) if we'd rather have SELECT vs select. Datasette seems to prefer lowercase so probably best to keep it as-is,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1890#issuecomment-1316240839,https://api.github.com/repos/simonw/datasette/issues/1890,1316240839,IC_kwDOBm6k_c5OdD3H,9599,simonw,2022-11-16T03:09:11Z,2022-11-16T03:09:11Z,OWNER,"Here's a polyfill for ``: https://github.com/mfranzke/datalist-polyfill It shouldn't be necessary now that Safari has shipped support (apparently added in https://developer.apple.com/documentation/safari-release-notes/safari-12_1-release-notes#3130314 Safari 12.1 in March 2019). But it does look like Safari doesn't support differing `label` and `value` attributes, though documentation about this is hard to come by.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/pull/1893#issuecomment-1316236448,https://api.github.com/repos/simonw/datasette/issues/1893,1316236448,IC_kwDOBm6k_c5OdCyg,9599,simonw,2022-11-16T03:04:57Z,2022-11-16T03:04:57Z,OWNER,If you rebase from `main` you should get the fix for that test failure.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1890#issuecomment-1316233532,https://api.github.com/repos/simonw/datasette/issues/1890,1316233532,IC_kwDOBm6k_c5OdCE8,9599,simonw,2022-11-16T03:00:58Z,2022-11-16T03:00:58Z,OWNER,"Oops, introduced a test failure: ``` def test_table_html_foreign_key_facets(app_client): response = app_client.get( ""/fixtures/foreign_key_references?_facet=foreign_key_with_blank_label"" ) assert response.status == 200 > assert ( '
  • ' ""- 1
  • "" ) in response.text E assert '
  • - 1
  • ' in '\n\n\n fixtures: foreign_key_references: 2 rows\n \n\n\n\n\n' E + where '\n\n\n fixtures: foreign_key_references: 2 rows\n \n\n\n\n\n' = .text ``` Need to fix this test: https://github.com/simonw/datasette/blob/eac028d3f77aa5473a5fcf59240635a1bca80f7d/tests/test_table_html.py#L616-L624","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/pull/1893#issuecomment-1316232588,https://api.github.com/repos/simonw/datasette/issues/1893,1316232588,IC_kwDOBm6k_c5OdB2M,9599,simonw,2022-11-16T03:00:04Z,2022-11-16T03:00:04Z,OWNER,"Oops, the tests are failing because of a test failure I introduced here: - #1890","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316231560,https://api.github.com/repos/simonw/datasette/issues/1893,1316231560,IC_kwDOBm6k_c5OdBmI,9599,simonw,2022-11-16T02:59:00Z,2022-11-16T02:59:00Z,OWNER,"The resize handle doesn't appear on Mobile Safari on iPhone - I don't think that particularly matters though. The textarea does get a weird border around it when focused on iPhone though. Focused: ![BF34E8FB-E35C-4CAB-9BFB-8EEF7E29B16C_1_201_a](https://user-images.githubusercontent.com/9599/202072748-c85bab94-a039-4ed6-8185-3cac25c78ed3.jpeg) Not focused: ![31A5CF38-D540-4A1A-8A7D-E29453D150F4_1_201_a](https://user-images.githubusercontent.com/9599/202072744-d9f0ea62-13b7-46ff-afe1-6d88d7fb8b53.jpeg) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316227073,https://api.github.com/repos/simonw/datasette/issues/1893,1316227073,IC_kwDOBm6k_c5OdAgB,9599,simonw,2022-11-16T02:54:22Z,2022-11-16T02:54:32Z,OWNER,"If you can get a version of this working with table and column autocompletion just using a static JavaScript object in the source code with the right tables and columns, I'm happy to take on the work of turning that static object into something that Datasette includes in the page itself with all of the correct values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316141764,https://api.github.com/repos/simonw/datasette/issues/1893,1316141764,IC_kwDOBm6k_c5OcrrE,9599,simonw,2022-11-16T01:26:59Z,2022-11-16T01:26:59Z,OWNER,"Resizing works great for me - and the page automatically sizes the editor to fit an existing query, e.g. on https://datasette-pr-1893.vercel.app/fixtures?sql=select+id%2C+content%2C+content2%0D%0A++from+primary_key_multiple_columns_explicit_label%0D%0A++order+by+id%0D%0A++limit+101","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316137982,https://api.github.com/repos/simonw/datasette/issues/1893,1316137982,IC_kwDOBm6k_c5Ocqv-,9599,simonw,2022-11-16T01:23:47Z,2022-11-16T01:23:47Z,OWNER,"Autocomplete here looks promising (I've wanted that to work for years!), but it does currently show a whole bunch of suggestions which aren't part of the SQLite SQL dialect: ![autocomplete](https://user-images.githubusercontent.com/9599/202060211-51ec9f45-bc52-459a-a729-27fc2faadff9.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316135244,https://api.github.com/repos/simonw/datasette/issues/1893,1316135244,IC_kwDOBm6k_c5OcqFM,9599,simonw,2022-11-16T01:21:41Z,2022-11-16T01:21:41Z,OWNER,"I just deployed a demo instance like this (using the commit hash from this PR): ```bash datasette publish vercel fixtures.db \ --branch 544f7025900b78f63c34b9985522271ba5fd9c0f \ --project datasette-pr-1893 \ --scope datasette \ --about 'PR 1893' \ --about_url https://github.com/simonw/datasette/pull/1893 ``` Here's the result: https://datasette-pr-1893.vercel.app/fixtures","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1316041828,https://api.github.com/repos/simonw/datasette/issues/1893,1316041828,IC_kwDOBm6k_c5OcTRk,95570,bgrins,2022-11-15T23:51:35Z,2022-11-15T23:51:35Z,CONTRIBUTOR,"I experimented with autocompleting the actual schema in https://github.com/bgrins/datasette/commit/8431c98850c7a552dbcde2a4dd0c3dc942a97d25, but it would need some work (current problems with it listed in the commit message there)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1315869946,https://api.github.com/repos/simonw/datasette/issues/1893,1315869946,IC_kwDOBm6k_c5ObpT6,95570,bgrins,2022-11-15T21:12:38Z,2022-11-15T21:12:38Z,CONTRIBUTOR,https://github.com/Sphinxxxx/cm-resize isn't compatible with 6. There's a suggestion to try using CSS resize in https://discuss.codemirror.net/t/resizing-codemirror-6/3265/2,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1315869040,https://api.github.com/repos/simonw/datasette/issues/1893,1315869040,IC_kwDOBm6k_c5ObpFw,95570,bgrins,2022-11-15T21:11:42Z,2022-11-15T21:11:42Z,CONTRIBUTOR,"extraKeys is done - Shift+Enter is added in the helper function, and it appears that the Tab behavior now defaults to what the `Tab: false` setting was doing (allowing it to escape to the form)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/pull/1893#issuecomment-1315853097,https://api.github.com/repos/simonw/datasette/issues/1893,1315853097,IC_kwDOBm6k_c5OblMp,95570,bgrins,2022-11-15T20:55:40Z,2022-11-15T20:55:40Z,CONTRIBUTOR,Should also minify the bundled output,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450363982,"Upgrade to CodeMirror 6, add SQL autocomplete", https://github.com/simonw/datasette/issues/1892#issuecomment-1315814786,https://api.github.com/repos/simonw/datasette/issues/1892,1315814786,IC_kwDOBm6k_c5Obb2C,4399499,ocdtrekkie,2022-11-15T20:14:38Z,2022-11-15T20:14:38Z,NONE,"I have no particular point, I just want to say being around for the 1.0 release of Datasette seems historic and legendary to witness.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1863#issuecomment-1315812212,https://api.github.com/repos/simonw/datasette/issues/1863,1315812212,IC_kwDOBm6k_c5ObbN0,9599,simonw,2022-11-15T20:12:02Z,2022-11-15T20:12:02Z,OWNER,"If the update succeeds it will return `{""ok"": true}`. For consistency with `/db/table/-/insert` you can pass `""return"": true` and it will return a `""row""` key with the now-updated full row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1315809867,https://api.github.com/repos/simonw/datasette/issues/1863,1315809867,IC_kwDOBm6k_c5ObapL,9599,simonw,2022-11-15T20:09:44Z,2022-11-15T20:09:44Z,OWNER,"I'm also not going to implement `""alter"": true` yet (which would add any missing columns based on the update) - I'll hold that off for a later feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1315809260,https://api.github.com/repos/simonw/datasette/issues/1863,1315809260,IC_kwDOBm6k_c5Obafs,9599,simonw,2022-11-15T20:09:11Z,2022-11-15T20:09:11Z,OWNER,"I'm going to use the error format I've been experimenting with here: - #1875 ```json { ""type"": ""https://example.net/validation-error"", ""title"": ""Your request is not valid."", ""errors"": [ { ""detail"": ""must be a positive integer"", ""pointer"": ""#/age"" }, { ""detail"": ""must be 'green', 'red' or 'blue'"", ""pointer"": ""#/profile/color"" } ] } ``` I'm not quite ready to commit to a `type` URL though, so I'll leave that to be solved later should I fully embrace that RFC.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1315808062,https://api.github.com/repos/simonw/datasette/issues/1863,1315808062,IC_kwDOBm6k_c5ObaM-,9599,simonw,2022-11-15T20:08:04Z,2022-11-15T20:08:04Z,OWNER,"The initial design I'm going to implement will look like this: ``` POST /db/table/1/-/update Authorization: Bearer xxx Content-Type: application/json ``` ```json { ""update"": { ""name"": ""New name"" } } ``` Any fields that are not yet columns will return an error. Should it enforce types, in as much as an integer column should have a JSON integer passed to it, or should it allow strings containing valid integers? I'm going to allow strings, mainly as a workaround for the fact that JavaScript integers have a maximum size.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1892#issuecomment-1315805498,https://api.github.com/repos/simonw/datasette/issues/1892,1315805498,IC_kwDOBm6k_c5ObZk6,9599,simonw,2022-11-15T20:05:30Z,2022-11-15T20:05:30Z,OWNER,"One slight concern: https://latest.datasette.io/ will increasingly reflect a version that isn't the most recent production release. I might setup https://stable.datasette.io/ as a demo instance of the most recent non-alpha release to compensate for that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1892#issuecomment-1315804535,https://api.github.com/repos/simonw/datasette/issues/1892,1315804535,IC_kwDOBm6k_c5ObZV3,9599,simonw,2022-11-15T20:04:38Z,2022-11-15T20:04:38Z,OWNER,"I'll do this after the 1.0a0 release: - #1708","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450312343,Merge 1.0-dev branch back to main, https://github.com/simonw/datasette/issues/1708#issuecomment-1095675839,https://api.github.com/repos/simonw/datasette/issues/1708,1095675839,IC_kwDOBm6k_c5BTq-_,9599,simonw,2022-04-11T23:06:30Z,2022-11-15T19:57:53Z,OWNER,"# Datasette 1.0 alpha 1 This alpha release is a preview of Datasette 1.0. Datasette 1.0 marks a significant milestone in the project: it is the point from which various aspects of Datasette can be considered ""stable"", in that code developed against them should expect not to be broken by future releases in the 1.x series. This will hold true until the next major version release, Datasette 2.0 - which we hope to hold off releasing for as long as possible. The following Datasette components should be be considered stable after 1.0: - The plugin API. Plugins developed against 1.0 should continue to work unmodified throughout the 1.x series. - The JSON API. Code written that interacts with Datasette's default JSON web API should continue to work. - The template context. If you build custom templates against Datasette your custom pages should continue to work. Note that none of these components will cease to introduce new features. New plugin hooks, new JSON APIs and new template context variables can be introduced without breaking existing code. Since this alpha release previews features that will be frozen for 1.0, please test this thoroughly against your existing Datasette projects. You can install the alpha using: pip install datasette==1.0a0 ## JSON API changes The most significant changes introduced in this new alpha concern Datasette's JSON API. The default JSON returned by the `/database/table.json` endpoint has changed. It now returns an object with two keys: `rows` - which contains a list of objects representing the rows in the table or query, and `more` containing a `boolean` that shows if there are more rows or if this object contains them all. ```json { ""rows"": [{ ""id"": 1, ""name"": ""Name 1"" }, { ""id"": 2, ""name"": ""Name 2"" }], ""more"": false } ``` [ Initially I thought about going with `next_url`, which would be `null` if you have reached the last page of records. Maybe that would be better? But since `next_url` cannot be provided on query pages, should this be part of the default format at all? ] ## Use ?_extra= to retrieve extra fields The default format can be expanded using one or more `?_extra=` parameters. This takes names of extra keys you would like to include. These can be comma-separated or `?_extra=` can be applied multiple times. For example: /database/table.json?_extra=total This adds a `""total"": 124` field to the returned JSON. [ Question: if you do `?_facet=foo` then do you still need to do `?_extra=facets` - I think not? ]","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1200649124,Datasette 1.0 alpha upcoming release notes, https://github.com/simonw/datasette/issues/1890#issuecomment-1314891228,https://api.github.com/repos/simonw/datasette/issues/1890,1314891228,IC_kwDOBm6k_c5OX6Xc,9599,simonw,2022-11-15T07:23:01Z,2022-11-15T07:23:01Z,OWNER,"Annoying: Mobile Safari doesn't seem to support separate labels and values. I should probably disable this feature on that browser, at least for foreign key facets (for the moment).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314856513,https://api.github.com/repos/simonw/datasette/issues/1890,1314856513,IC_kwDOBm6k_c5OXx5B,9599,simonw,2022-11-15T06:56:29Z,2022-11-15T06:56:29Z,OWNER,"Looks like I can fix that like so: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314850524,https://api.github.com/repos/simonw/datasette/issues/1890,1314850524,IC_kwDOBm6k_c5OXwbc,9599,simonw,2022-11-15T06:48:37Z,2022-11-15T06:48:37Z,OWNER,"Spotted a bug with this on https://latest.datasette.io/fixtures/facetable?_facet=_city_id - the `_city_id` column is a foreign key, so you need to type `1` or `2` - but the autocomplete list shows the full text names for the cities.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314849867,https://api.github.com/repos/simonw/datasette/issues/1890,1314849867,IC_kwDOBm6k_c5OXwRL,9599,simonw,2022-11-15T06:47:51Z,2022-11-15T06:47:51Z,OWNER,Demo now live here: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=party - select `party` and start typing.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314848432,https://api.github.com/repos/simonw/datasette/issues/1890,1314848432,IC_kwDOBm6k_c5OXv6w,9599,simonw,2022-11-15T06:46:08Z,2022-11-15T06:46:08Z,OWNER,Wrote a TIL about ``: https://til.simonwillison.net/html/datalist,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1862#issuecomment-1314845667,https://api.github.com/repos/simonw/datasette/issues/1862,1314845667,IC_kwDOBm6k_c5OXvPj,9599,simonw,2022-11-15T06:42:09Z,2022-11-15T06:42:32Z,OWNER,"I implemented this as part of `/db/-/create`. https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1890#issuecomment-1314835740,https://api.github.com/repos/simonw/datasette/issues/1890,1314835740,IC_kwDOBm6k_c5OXs0c,9599,simonw,2022-11-15T06:30:26Z,2022-11-15T06:30:26Z,OWNER,That prototype actually works really well! I'm going to add that to `table.js`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314833881,https://api.github.com/repos/simonw/datasette/issues/1890,1314833881,IC_kwDOBm6k_c5OXsXZ,9599,simonw,2022-11-15T06:27:21Z,2022-11-15T06:27:21Z,OWNER,"Here's a prototype: ```javascript function createDataLists() { var facetResults = document.querySelectorAll("".facet-results [data-column]""); Array.from(facetResults).forEach(function (facetResult) { // Use link text from all links in the facet result var linkTexts = Array.from( facetResult.querySelectorAll(""li:not(.facet-truncated) a"") ).map(function (link) { return link.textContent; }); // Create a datalist element var datalist = document.createElement(""datalist""); datalist.id = ""datalist-"" + facetResult.dataset.column; // Create an option element for each link text linkTexts.forEach(function (linkText) { var option = document.createElement(""option""); option.value = linkText; datalist.appendChild(option); }); // Add the datalist to the facet result facetResult.appendChild(datalist); }); } createDataLists(); // When any select with name=_filter_column changes, update the datalist document.body.addEventListener(""change"", function (event) { if (event.target.name === ""_filter_column"") { event.target .closest("".filter-row"") .querySelector("".filter-value"") .setAttribute(""list"", ""datalist-"" + event.target.value); } }); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314829751,https://api.github.com/repos/simonw/datasette/issues/1890,1314829751,IC_kwDOBm6k_c5OXrW3,9599,simonw,2022-11-15T06:20:50Z,2022-11-15T06:20:50Z,OWNER,"This finds the right links on the page: document.querySelectorAll('.facet-results [data-column] li:not(.facet-truncated) a')","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314825019,https://api.github.com/repos/simonw/datasette/issues/1890,1314825019,IC_kwDOBm6k_c5OXqM7,9599,simonw,2022-11-15T06:13:36Z,2022-11-15T06:13:36Z,OWNER,"This could start out as a purely JavaScript enhancement for pages that already figured out the available values through faceting, like you suggested.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314823752,https://api.github.com/repos/simonw/datasette/issues/1890,1314823752,IC_kwDOBm6k_c5OXp5I,9599,simonw,2022-11-15T06:11:49Z,2022-11-15T06:11:49Z,OWNER,"I tried this out on https://congress-legislators.datasettes.com/legislators/legislator_terms for the `party` column - here's the demo: ![datalist](https://user-images.githubusercontent.com/9599/201839812-db887ce0-c4b9-432c-8620-5ac73f222a63.gif) I made this work by dropping the following HTML into the page in the browser DevTools: ```html ``` And then adding `list=""party""` to the input element in the filter form.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1890#issuecomment-1314821337,https://api.github.com/repos/simonw/datasette/issues/1890,1314821337,IC_kwDOBm6k_c5OXpTZ,9599,simonw,2022-11-15T06:08:19Z,2022-11-15T06:08:19Z,OWNER,"Oh interesting... this doesn't even need to be attached to the visible faceting feature, necessarily: Datasette could try to detect when a column has a limited number of options (which the faceting code handles already) and could turn those into an auto-complete interface. There's actually a native HTML element for this these days: the `` https://developer.mozilla.org/en-US/docs/Web/HTML/Element/datalist","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1448143294,Autocomplete text entry for filter values that correspond to facets, https://github.com/simonw/datasette/issues/1882#issuecomment-1314813205,https://api.github.com/repos/simonw/datasette/issues/1882,1314813205,IC_kwDOBm6k_c5OXnUV,9599,simonw,2022-11-15T06:00:41Z,2022-11-15T06:00:41Z,OWNER,"Documentation: - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table - https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table-from-example-data Wrote a TIL about how I wrote some of those tests with Copilot: https://til.simonwillison.net/gpt3/writing-test-with-copilot","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1886#issuecomment-1314627077,https://api.github.com/repos/simonw/datasette/issues/1886,1314627077,IC_kwDOBm6k_c5OW54F,11788561,jrdmb,2022-11-15T01:19:54Z,2022-11-15T01:19:54Z,NONE,"Datasette usage comments for its 5th anniversary celebration: I use Datasette and related tools for a Cosmology Researcher Talks database app project, which is [described in the github Readme](https://github.com/jrdmb/cosmotalks-datasette#readme) The app hosted on the Google Cloud Run service also uses other Datasette-related tools developed by Simon - datasette-render-markdown, csvs-to-sqlite, datasette-template-sql, and datasette-block-robots. This is one of two apps used for querying the talks database, each has it pros/cons as described in the github Readme. At present, over 170 different sites that host cosmology talks are scraped to collect new talks for import into the sqlite database. The shot-scraper and sqlite-utils tools are a major help for this. I also use the Mastodon API to get my favorites, toots, and boosts into a local database so I can do searches on the data. This was done on Twitter and was then extended to the Mastodon data. Again, sqlite-utils is an important tool for this. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1875#issuecomment-1314620086,https://api.github.com/repos/simonw/datasette/issues/1875,1314620086,IC_kwDOBm6k_c5OW4K2,9599,simonw,2022-11-15T01:09:56Z,2022-11-15T01:09:56Z,OWNER,"Rough initial prototype: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index 8b987221..518ac578 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1103,19 +1103,30 @@ class TableInsertView(BaseView): except json.JSONDecodeError as e: return _errors([""Invalid JSON: {}"".format(e)]) if not isinstance(data, dict): - return _errors([""JSON must be a dictionary""]) + return _errors([{""detail"": ""JSON must be a dictionary"", ""pointer"": ""#/""}]) keys = data.keys() # keys must contain ""row"" or ""rows"" if ""row"" not in keys and ""rows"" not in keys: return _errors(['JSON must have one or other of ""row"" or ""rows""']) rows = [] + was_single_row = False if ""row"" in keys: if ""rows"" in keys: - return _errors(['Cannot use ""row"" and ""rows"" at the same time']) + return _errors( + [ + { + ""detail"": 'Cannot use ""row"" and ""rows"" at the same time', + ""pointer"": ""#/row"", + } + ] + ) + was_single_row = True row = data[""row""] if not isinstance(row, dict): - return _errors(['""row"" must be a dictionary']) + return _errors( + [{""detail"": '""row"" must be a dictionary', ""pointer"": ""#/row""}] + ) rows = [row] data[""return""] = True else: @@ -1152,9 +1163,12 @@ class TableInsertView(BaseView): invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append( - ""Row {} has invalid columns: {}"".format( - i, "", "".join(sorted(invalid_columns)) - ) + { + ""detail"": ""Invalid columns: {}"".format( + "", "".join(sorted(invalid_columns)) + ), + ""pointer"": ""#/blah/"", + } ) if errors: return _errors(errors) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314615592,https://api.github.com/repos/simonw/datasette/issues/1875,1314615592,IC_kwDOBm6k_c5OW3Eo,9599,simonw,2022-11-15T01:04:28Z,2022-11-15T01:04:28Z,OWNER,"Worth noting this bit in RFC 7807: > The fictional problem type here defines the ""errors"" extension, an > array that describes the details of each validation error. Each > member is an object containing ""detail"" to describe the issue, and > ""pointer"" to locate the problem within the request's content using a > JSON Pointer [JSON-POINTER]. So the list of `""errors""` with JSON Pointer isn't technically part of the spec, it's an imaginary extension. It fits what I need to do though, so I'm inclined to stick with it anyway.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314545407,https://api.github.com/repos/simonw/datasette/issues/1875,1314545407,IC_kwDOBm6k_c5OWl7_,9599,simonw,2022-11-14T23:30:34Z,2022-11-14T23:30:34Z,OWNER,TIL: https://til.simonwillison.net/json/json-pointer,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314491884,https://api.github.com/repos/simonw/datasette/issues/1875,1314491884,IC_kwDOBm6k_c5OWY3s,9599,simonw,2022-11-14T22:26:11Z,2022-11-14T22:26:54Z,OWNER,"Spec looks pretty simple: > A JSON Pointer is a Unicode string (see [RFC4627], Section 3) > containing a sequence of zero or more reference tokens, each prefixed > by a `/` (%x2F) character. > > Because the characters `~` (%x7E) and `/` (%x2F) have special > meanings in JSON Pointer, `~` needs to be encoded as `~0` and `/` > needs to be encoded as `~1` when these characters appear in a > reference token.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314491150,https://api.github.com/repos/simonw/datasette/issues/1875,1314491150,IC_kwDOBm6k_c5OWYsO,9599,simonw,2022-11-14T22:25:20Z,2022-11-14T22:25:20Z,OWNER,"That's using JSON Pointer: https://www.rfc-editor.org/rfc/rfc6901 There's a Python library for that here https://github.com/stefankoegl/python-json-pointer/blob/master/jsonpointer.py - which looks simple and clean and well maintained and documented, but it only handles the ""what is at this pointer within this JSON object"" case - I need to generate the correct JSON pointer to explain where my error is. So I think I'll end up hand-rolling this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1875#issuecomment-1314488010,https://api.github.com/repos/simonw/datasette/issues/1875,1314488010,IC_kwDOBm6k_c5OWX7K,9599,simonw,2022-11-14T22:21:43Z,2022-11-14T22:21:43Z,OWNER,"Here's the most relevant example from the RFC spec: ``` POST /details HTTP/1.1 Host: account.example.com Accept: application/json ``` ```json { ""age"": 42.3, ""profile"": { ""color"": ""yellow"" } } ``` ``` HTTP/1.1 400 Bad Request Content-Type: application/problem+json Content-Language: en ``` ```json { ""type"": ""https://example.net/validation-error"", ""title"": ""Your request is not valid."", ""errors"": [ { ""detail"": ""must be a positive integer"", ""pointer"": ""#/age"" }, { ""detail"": ""must be 'green', 'red' or 'blue'"", ""pointer"": ""#/profile/color"" } ] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1886#issuecomment-1314455003,https://api.github.com/repos/simonw/datasette/issues/1886,1314455003,IC_kwDOBm6k_c5OWP3b,17053189,sachaj,2022-11-14T21:51:11Z,2022-11-14T21:51:11Z,NONE,"Happy Birthday Datasette! I am a librarian at the Université du Québec à Montréal (UQAM) and I've been using Datasette to publish excerpts of our library data. There are several use cases I'm working with as a proof of concept : 1. New titles list : based on reports of recent acquisitions by subject, discipline, etc. 2. List of all UQAM theses and dissertations : based on an extract of bibliographic records 3. List of all publications by UQAM Authors : based on an extract of bibliographic records See our prototype under construction here : https://datasette-bib.uqam.ca/ (some bits and pieces have been translated into French) Datasette is amazing, there is so much potential here for libraries. Thanks to Simon and all the contributors for this outstanding effort. Also sqlite-utils deserves special mention as incredibly handy and useful.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1314241058,https://api.github.com/repos/simonw/datasette/issues/1886,1314241058,IC_kwDOBm6k_c5OVboi,25778,eyeseast,2022-11-14T19:06:35Z,2022-11-14T19:06:35Z,CONTRIBUTOR,"This probably counts as a case study: https://github.com/eyeseast/spatial-data-cooking-show. Even has video. Seriously, though, this workflow has become integral to my work with reporters and editors across USA TODAY Network. Very often, I get sent a folder of data in mixed formats, with a vague ask of how we should communicate some part of it to users. Datasette and its constellation of tools makes it easy to get a quick look at that data, run exploratory queries, map it and ask questions to figure out what's important to show. And then I export a version of the data that's exactly what I need for display. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1314223118,https://api.github.com/repos/simonw/datasette/issues/1886,1314223118,IC_kwDOBm6k_c5OVXQO,639730,virtadpt,2022-11-14T18:51:20Z,2022-11-14T18:51:20Z,NONE,I use Datasette to analyze blocklists by using csv-to-sqlite to pull their contents into a database and Datasette to look around through them. I also use its REST API to query said database as part of filtering out garbage from domains found in those blocklists.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1884#issuecomment-1314066229,https://api.github.com/repos/simonw/datasette/issues/1884,1314066229,IC_kwDOBm6k_c5OUw81,25778,eyeseast,2022-11-14T16:48:35Z,2022-11-14T16:48:35Z,CONTRIBUTOR,"I'm realizing I don't know if a virtual table will ever return a count. Maybe it depends on the implementation. For these three, just checking now, it'll always return zero. That said, I'm not sure there's any downside to having them return zero and caching that. (They're hidden, too.) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1314054300,https://api.github.com/repos/simonw/datasette/issues/1884,1314054300,IC_kwDOBm6k_c5OUuCc,9599,simonw,2022-11-14T16:40:06Z,2022-11-14T16:40:06Z,OWNER,"I wonder if there are any reasons that inspect SHOULD try to count virtual tables? Like are there any likely uses for a cirial table where the count is both interesting and likely to be accessed often enough that it's worth caching? I have an issue open to add a setting to disable table counts entirely: - #1818 Maybe that should be expanded to automatically disable row counts for virtual tables entirely? Which would mean no count would be shown for them in the UI. If you desperately wanted a count you would then have to run a count(*) query against them explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1313962183,https://api.github.com/repos/simonw/datasette/issues/1884,1313962183,IC_kwDOBm6k_c5OUXjH,25778,eyeseast,2022-11-14T15:46:32Z,2022-11-14T15:46:32Z,CONTRIBUTOR,"It does work, though I think it's probably still worth excluding virtual tables that will always be zero. Here's the same inspection as before, now with `--load-extension spatialite`: ```json { ""alltheplaces"": { ""hash"": ""0843cfe414439ab903c22d1121b7ddbc643418c35c7f0edbcec82ef1452411df"", ""size"": 963375104, ""file"": ""alltheplaces.db"", ""tables"": { ""spatial_ref_sys"": { ""count"": 6215 }, ""spatialite_history"": { ""count"": 18 }, ""sqlite_sequence"": { ""count"": 2 }, ""geometry_columns"": { ""count"": 3 }, ""spatial_ref_sys_aux"": { ""count"": 6164 }, ""views_geometry_columns"": { ""count"": 0 }, ""virts_geometry_columns"": { ""count"": 0 }, ""geometry_columns_statistics"": { ""count"": 3 }, ""views_geometry_columns_statistics"": { ""count"": 0 }, ""virts_geometry_columns_statistics"": { ""count"": 0 }, ""geometry_columns_field_infos"": { ""count"": 0 }, ""views_geometry_columns_field_infos"": { ""count"": 0 }, ""virts_geometry_columns_field_infos"": { ""count"": 0 }, ""geometry_columns_time"": { ""count"": 3 }, ""geometry_columns_auth"": { ""count"": 3 }, ""views_geometry_columns_auth"": { ""count"": 0 }, ""virts_geometry_columns_auth"": { ""count"": 0 }, ""data_licenses"": { ""count"": 10 }, ""sql_statements_log"": { ""count"": 0 }, ""states"": { ""count"": 56 }, ""counties"": { ""count"": 3234 }, ""idx_states_geometry_rowid"": { ""count"": 56 }, ""idx_states_geometry_node"": { ""count"": 3 }, ""idx_states_geometry_parent"": { ""count"": 2 }, ""idx_counties_geometry_rowid"": { ""count"": 3234 }, ""idx_counties_geometry_node"": { ""count"": 98 }, ""idx_counties_geometry_parent"": { ""count"": 97 }, ""idx_places_geometry_rowid"": { ""count"": 1236796 }, ""idx_places_geometry_node"": { ""count"": 38163 }, ""idx_places_geometry_parent"": { ""count"": 38162 }, ""places"": { ""count"": 1332609 }, ""SpatialIndex"": { ""count"": 0 }, ""ElementaryGeometries"": { ""count"": 0 }, ""KNN"": { ""count"": 0 }, ""idx_states_geometry"": { ""count"": 56 }, ""idx_counties_geometry"": { ""count"": 3234 }, ""idx_places_geometry"": { ""count"": 1236796 } } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1886#issuecomment-1313271719,https://api.github.com/repos/simonw/datasette/issues/1886,1313271719,IC_kwDOBm6k_c5ORu-n,124274,lucapette,2022-11-14T08:25:12Z,2022-11-14T08:25:12Z,NONE,"Nothing spectacular yet but I think this falls under ""cool/cute application of datasette"": [improving fakedata performance for fun](https://lucapette.me/writing/improving-fakedata-performance-for-fun/). tl;dr I used datasette to visualize benchmarking data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1313252879,https://api.github.com/repos/simonw/datasette/issues/1886,1313252879,IC_kwDOBm6k_c5ORqYP,883348,adipasquale,2022-11-14T08:10:23Z,2022-11-14T08:10:23Z,CONTRIBUTOR,"Hi @simonw and thanks for the great tools you're publishing, your dedication is inspiring! I work for the French Ministry of Culture on a surveying tool for objects protected for their historical value. It is part of a program building modern public services called [beta.gouv.fr](https://beta.gouv.fr/). In that context I'm using data published by the Ministry that I have ingested into datasette and published on a free Fly instance : https://collectif-objets-datasette.fly.dev . I have also ingested another data set with infos about french cities on this instance so that I can perform joined queries. The surveying tool synchronizes its data regularly from this datasette instance, and I also use it to perform queries when asked generic questions about the distribution of objects. (The data is not very accessible as it's undocumented and for internal usage mostly)","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1850#issuecomment-1313156167,https://api.github.com/repos/simonw/datasette/issues/1850,1313156167,IC_kwDOBm6k_c5ORSxH,9599,simonw,2022-11-14T06:23:39Z,2022-11-14T06:23:39Z,OWNER,The API explorer is now live here: https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1313155712,https://api.github.com/repos/simonw/datasette/issues/1850,1313155712,IC_kwDOBm6k_c5ORSqA,9599,simonw,2022-11-14T06:22:57Z,2022-11-14T06:22:57Z,OWNER,"I think the ability to create tokens should be protected by a `create-tokens` permission, not just a global setting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1888#issuecomment-1313139657,https://api.github.com/repos/simonw/datasette/issues/1888,1313139657,IC_kwDOBm6k_c5OROvJ,9599,simonw,2022-11-14T06:04:48Z,2022-11-14T06:04:48Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447439985,API explorer should take immutability into account, https://github.com/simonw/datasette/issues/1866#issuecomment-1313128913,https://api.github.com/repos/simonw/datasette/issues/1866,1313128913,IC_kwDOBm6k_c5ORMHR,9599,simonw,2022-11-14T05:48:22Z,2022-11-14T05:48:22Z,OWNER,"I changed my mind about the `""return_rows"": true` option - I'm going to rename it to `""return"": true`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1874#issuecomment-1313127054,https://api.github.com/repos/simonw/datasette/issues/1874,1313127054,IC_kwDOBm6k_c5ORLqO,9599,simonw,2022-11-14T05:45:00Z,2022-11-14T05:45:00Z,OWNER,"Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=&method=POST ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1871#issuecomment-1313125870,https://api.github.com/repos/simonw/datasette/issues/1871,1313125870,IC_kwDOBm6k_c5ORLXu,9599,simonw,2022-11-14T05:42:50Z,2022-11-14T05:42:50Z,OWNER,Demo: https://latest-1-0-dev.datasette.io/-/api#path=%2Ffixtures%2Ffacetable%2F-%2Fdrop&json=%7B%22confirm%22%3A+true%7D&method=POST,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1874#issuecomment-1313125123,https://api.github.com/repos/simonw/datasette/issues/1874,1313125123,IC_kwDOBm6k_c5ORLMD,9599,simonw,2022-11-14T05:41:20Z,2022-11-14T05:42:23Z,OWNER,"I also changed the confirmation JSON returned by this endpoint to add the `database` and `table` like so: ```json { ""ok"": true, ""database"": ""data"", ""table"": ""docs"", ""row_count"": 1, ""message"": ""Pass \""confirm\"": true to confirm"" } ``` Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1874#issuecomment-1313119558,https://api.github.com/repos/simonw/datasette/issues/1874,1313119558,IC_kwDOBm6k_c5ORJ1G,9599,simonw,2022-11-14T05:30:27Z,2022-11-14T05:30:27Z,OWNER,Found a bug: you get a 500 error if you try this against an immutable database.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/datasette/issues/1850#issuecomment-1313115059,https://api.github.com/repos/simonw/datasette/issues/1850,1313115059,IC_kwDOBm6k_c5ORIuz,9599,simonw,2022-11-14T05:21:30Z,2022-11-14T05:21:30Z,OWNER,New documentation for these features currently lives here: https://docs.datasette.io/en/1.0-dev/json_api.html#the-json-write-api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1875#issuecomment-1313114283,https://api.github.com/repos/simonw/datasette/issues/1875,1313114283,IC_kwDOBm6k_c5ORIir,9599,simonw,2022-11-14T05:20:00Z,2022-11-14T05:20:00Z,OWNER,"I started a conversation about JSON error standards on Mastodon here: https://fedi.simonwillison.net/web/@simon/109338725610487457 Quite a few people pointed to this RFC independently.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430797211,Figure out design for JSON errors (consider RFC 7807), https://github.com/simonw/datasette/issues/1887#issuecomment-1313113642,https://api.github.com/repos/simonw/datasette/issues/1887,1313113642,IC_kwDOBm6k_c5ORIYq,9599,simonw,2022-11-14T05:18:51Z,2022-11-14T05:18:51Z,OWNER,Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,Add a confirm step to the drop table API, https://github.com/simonw/datasette/issues/1887#issuecomment-1313097713,https://api.github.com/repos/simonw/datasette/issues/1887,1313097713,IC_kwDOBm6k_c5OREfx,9599,simonw,2022-11-14T05:00:54Z,2022-11-14T05:00:54Z,OWNER,"I'm going to add a `""confirm"": true` option to the API. Without that, it returns a note about how many rows will be deleted.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447388809,Add a confirm step to the drop table API, https://github.com/simonw/datasette/issues/1871#issuecomment-1313097057,https://api.github.com/repos/simonw/datasette/issues/1871,1313097057,IC_kwDOBm6k_c5OREVh,9599,simonw,2022-11-14T04:59:28Z,2022-11-14T04:59:28Z,OWNER,In playing with the API explorer just now I realized it's way too easy to accidentally drop a table using it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1313072900,https://api.github.com/repos/simonw/datasette/issues/1871,1313072900,IC_kwDOBm6k_c5OQ-cE,9599,simonw,2022-11-14T04:15:50Z,2022-11-14T04:15:50Z,OWNER,"For the example links - I'm going to have these at the bottom of the page so you don't have to scroll past them. Ideally these would take the user's permissions into account. This could make the page expensive to load, but I'm going to risk it for the moment. Something like this then: > - data > - /data/-/create - create table > - /data/table1/-/insert - insert into table1 > - /data/table1/-/drop - drop table1 I won't bother with per-row demo links (for update and delete) because there could be thousands of them for each table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1313062699,https://api.github.com/repos/simonw/datasette/issues/1871,1313062699,IC_kwDOBm6k_c5OQ78r,9599,simonw,2022-11-14T04:03:29Z,2022-11-14T04:12:41Z,OWNER,"Two things left before I close this issue: - [x] I want to preserve the state of the forms in the URL - probably after a `#` - [ ] Instead of hard-coding the current examples, I want to provide a list of links which populate the forms","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1886#issuecomment-1313052863,https://api.github.com/repos/simonw/datasette/issues/1886,1313052863,IC_kwDOBm6k_c5OQ5i_,9599,simonw,2022-11-14T03:40:50Z,2022-11-14T03:40:50Z,OWNER,"Tim Sherratt on Twitter: https://twitter.com/wragge/status/1591930345469153282 > Where do I start? The [#GLAMWorkbench](https://twitter.com/hashtag/GLAMWorkbench?src=hashtag_click) now includes a number of examples where GLAM data is harvested, processed, and then made available for exploration via Datasette. > > https://glam-workbench.net/ > > For example the GLAM Name Index Search brings together 10+ million entries from 240 indexes and provides an aggregated search using the Datasette search-all plugin: > > https://glam-workbench.net/name-search/ > > Most recently I converted PDFs of the Tasmanian Postal Directories to a big Datasette instance: https://updates.timsherratt.org/2022/09/15/from-pdfs-to.html the process is documented and reusable.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1886#issuecomment-1312898318,https://api.github.com/repos/simonw/datasette/issues/1886,1312898318,IC_kwDOBm6k_c5OQT0O,19851673,eigenfoo,2022-11-14T00:52:16Z,2022-11-14T00:52:16Z,NONE,"I'm a cryptic crossword enthusiast and have spent a lot of time scraping and parsing cryptic crossword clues from various blogs, forums and publications. The result is over **half a million clues from cryptic crosswords over the past twelve years**, including the clue, answer, puzzle date, puzzle name and a link to the original source. This is all hosted using Datasette, which has been a delight to use: https://cryptics.georgeho.org/ This dataset is a significant work of crossword archivism and scholarship, as acquiring historical crosswords and structuring their contents require focused effort and tedious cleaning that few are willing to do for such trivial data - for example, according to [this 2004 selection guide](https://cryptics.georgeho.org/static/documents/Selection_AppendixE_v2.pdf), the Library of Congress explicitly does not collect crossword puzzles. Anecdotally, I know that many constructors/setters of cryptic crosswords use this dataset as a resource, and some even simply call it ""the database"" - this is probably one of the most impactful data projects I've worked on!","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1871#issuecomment-1312822353,https://api.github.com/repos/simonw/datasette/issues/1871,1312822353,IC_kwDOBm6k_c5OQBRR,9599,simonw,2022-11-13T21:07:40Z,2022-11-13T21:07:40Z,OWNER,I'm going to need extra code to toggle POST closed when GET opens and vice-versa.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312821031,https://api.github.com/repos/simonw/datasette/issues/1871,1312821031,IC_kwDOBm6k_c5OQA8n,9599,simonw,2022-11-13T21:02:06Z,2022-11-13T21:03:11Z,OWNER,"Actually no, I'm going to add a class of `details-menu` to the other details elements that SHOULD be closed. That way custom templates using `
    ` won't close in a surprising way.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312816451,https://api.github.com/repos/simonw/datasette/issues/1871,1312816451,IC_kwDOBm6k_c5OP_1D,9599,simonw,2022-11-13T20:39:26Z,2022-11-13T20:39:34Z,OWNER,I'm going to add a special `no-auto-close` class to these and teach that code not to close them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1312816292,https://api.github.com/repos/simonw/datasette/issues/1871,1312816292,IC_kwDOBm6k_c5OP_yk,9599,simonw,2022-11-13T20:38:42Z,2022-11-13T20:38:42Z,OWNER,"The current API explorer uses details/summary elements for the GET and POST dialogs. I only want one of these to be open at a time, to reflect that you can make either a GET or a POST. I just noticed that clicking anywhere else on the page closes both elements, which isn't what I want to happen. Turns out that's because of this code I added as part of Datasette's menu implementation! https://github.com/simonw/datasette/blob/9f54f00a50a4d950cfd69a0ff3526ae82c858826/datasette/templates/_close_open_menus.html#L2-L15","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1886#issuecomment-1312814245,https://api.github.com/repos/simonw/datasette/issues/1886,1312814245,IC_kwDOBm6k_c5OP_Sl,2090382,noslouch,2022-11-13T20:28:26Z,2022-11-13T20:28:26Z,NONE,"I work at The Wall Street Journal as a computational journalist and serve as our self-appointed Datasette evangelist. They say that to a hammer everything looks like a nail, but the reality is newsrooms find themselves in a sea of nails! I've only got a couple public projects that I can share, but happy to offer you a look at some of the internal projects. More often than not the internal projects stay internal because the reporting doesn't lead anywhere or I can't convince an editor to greenlight it. But imho that's the beauty of datasette: a (relatively) painless mechanism to see if there's any there there. - [WSJ Inflation Tracker](wsj.com/inflationtracker) - I scraped the oscars website and turned it into a datasette instance and ran the numbers on [best actress/best picture overlap ](https://www.wsj.com/livecoverage/oscars-academy-awards-2022/card/the-best-actress-nominees-aren-t-in-any-best-pictures-contenders-when-is-the-last-time-that-happened--mDxvbLug3rq84pxLE8gY) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1447050738,"Call for birthday presents: if you're using Datasette, let us know how you're using it here", https://github.com/simonw/datasette/issues/1882#issuecomment-1312582512,https://api.github.com/repos/simonw/datasette/issues/1882,1312582512,IC_kwDOBm6k_c5OPGtw,9599,simonw,2022-11-12T22:11:18Z,2022-11-12T22:11:18Z,OWNER,"I like this: ```json { ""ok"": true, ""database"": ""data"", ""table"": ""agai2n"", ""table_url"": ""http://127.0.0.1:8001/data/agai2n"", ""schema"": ""CREATE TABLE [agai2n] (\n [hello] INTEGER\n)"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312581121,https://api.github.com/repos/simonw/datasette/issues/1882,1312581121,IC_kwDOBm6k_c5OPGYB,9599,simonw,2022-11-12T22:01:32Z,2022-11-12T22:01:32Z,OWNER,I'm going to change it to `table` in the output AND the input.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312581008,https://api.github.com/repos/simonw/datasette/issues/1882,1312581008,IC_kwDOBm6k_c5OPGWQ,9599,simonw,2022-11-12T22:00:52Z,2022-11-12T22:00:52Z,OWNER,"Tried out my prototype in the API explorer: The `""name""` on the output is bothering me a bit - should it be `table_name` or `table` instead? Problem is I really like `name` for the input, so should it be consistent to have the same name on the output here, or should I aim for consistency with other endpoints that might return `table` rather than the ambiguous `name` elsewhere?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312580348,https://api.github.com/repos/simonw/datasette/issues/1882,1312580348,IC_kwDOBm6k_c5OPGL8,9599,simonw,2022-11-12T21:55:54Z,2022-11-12T21:56:45Z,OWNER,"What should this API return? I think the name of the table (`name`), the URL to that table (`table_url` - for consistency with how faceting API works already) and the schema of the table (`schema`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312575048,https://api.github.com/repos/simonw/datasette/issues/1882,1312575048,IC_kwDOBm6k_c5OPE5I,9599,simonw,2022-11-12T21:22:58Z,2022-11-12T21:22:58Z,OWNER,Need to validate the table name. SQLite supports almost any table name - but they can't contain a newline character and cannot start with `sqlite_` - according to https://stackoverflow.com/questions/3694276/what-are-valid-table-names-in-sqlite/43049720#43049720,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1312556044,https://api.github.com/repos/simonw/datasette/issues/1882,1312556044,IC_kwDOBm6k_c5OPAQM,9599,simonw,2022-11-12T19:29:11Z,2022-11-12T19:29:11Z,OWNER,"Thought of an edge-case: with `sqlite-utils` one feature I really like is that I can pipe data into it without caring if the table already exists or not: cat data.json | sqlite-utils insert my.db mytable - How could this new API support that? I thought about adding a `""create"": true` option to `/db/table/-/insert` which creates the table if it doesn't already exist, but if I do that I'll need to start adding other options to that endpoint - to set the primary key, add foreign keys and suchlike - which would be ignored except for the cases where the table was being created from scratch. This doesn't feel right to me - I want to keep those options here, on `/db/-/create`. One idea I had was to implement it such that you can call `/db/-/create` multiple times for the same table, but only if you are using the `""rows""` option. If so, and if the rows can be safely inserted, it would let you do that. But instead, I'm going to outsource this to the CLI tool I plan to write that feeds data into this API. I'm already planning to use that tool for CSV inserts (so the API doesn't need to accept CSV directly). I think it's a good place for other usability enhancements like ""insert this, creating the table if it does not exist"" as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1878#issuecomment-1312534826,https://api.github.com/repos/simonw/datasette/issues/1878,1312534826,IC_kwDOBm6k_c5OO7Eq,18738650,stevecrawshaw,2022-11-12T17:34:58Z,2022-11-12T17:34:58Z,NONE,"Hi Simon. I have just started experimenting with datasette in earnest, looking at it's suitability for air quality open data. A bulk upsert \ upsert_all would be very useful for me in enabling real time data to be pushed from a sql server database with FME server to a datasette db. An hourly process queries the last 2 hours of data and pushes that to my database, inserting new data and updating existing combinations of pk siteid and date_time. This is already implemented on our current [open data portal](https://opendata.bristol.gov.uk/explore/dataset/air-quality-data-continuous/table/?disjunctive.location&sort=date_time). Excited to see your progress with this! Thank you for this amazing software.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1883#issuecomment-1311437901,https://api.github.com/repos/simonw/datasette/issues/1883,1311437901,IC_kwDOBm6k_c5OKvRN,31312775,mattmalcher,2022-11-11T09:20:21Z,2022-11-11T09:20:21Z,NONE,Amazing - thank you for fixing and releasing that so quickly and for showing your process! <3 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311314981,https://api.github.com/repos/simonw/datasette/issues/1883,1311314981,IC_kwDOBm6k_c5OKRQl,9599,simonw,2022-11-11T07:15:48Z,2022-11-11T07:15:48Z,OWNER,I released that fix in Datasette 0.63.1: https://docs.datasette.io/en/stable/changelog.html#v0-63-1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311299535,https://api.github.com/repos/simonw/datasette/issues/1883,1311299535,IC_kwDOBm6k_c5OKNfP,9599,simonw,2022-11-11T06:54:58Z,2022-11-11T06:54:58Z,OWNER,"This time deployed with: ``` cd demos/apache-proxy fly deploy --build-arg DATASETTE_REF=8d9a957c6329d26cc1e417b5d6911640d74765eb ``` To ensure the exact commit with the fix. And that fixed it! ``` % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:54:45 GMT server: Fly/b1863e2e7 (2022-11-09) location: /prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1 link: ; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJQGBSXBR7E53TY0EKMQ9PA-sjc ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311292463,https://api.github.com/repos/simonw/datasette/issues/1883,1311292463,IC_kwDOBm6k_c5OKLwv,9599,simonw,2022-11-11T06:44:24Z,2022-11-11T06:44:24Z,OWNER,"Modifying that test to the following does indeed cause a failure: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( ""/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"" ) assert response.status == 302 assert ( response.headers[""location""] == ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"" ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311291632,https://api.github.com/repos/simonw/datasette/issues/1883,1311291632,IC_kwDOBm6k_c5OKLjw,9599,simonw,2022-11-11T06:43:00Z,2022-11-11T06:43:00Z,OWNER,"https://datasette-apache-proxy-demo.datasette.io/prefix/-/asgi-scope is useful: It confirms that `/prefix/` is nowhere to be seen in the incoming request data: ``` 'path': '/-/asgi-scope', 'query_string': b'', 'raw_path': b'/-/asgi-scope', ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311290115,https://api.github.com/repos/simonw/datasette/issues/1883,1311290115,IC_kwDOBm6k_c5OKLMD,9599,simonw,2022-11-11T06:40:14Z,2022-11-11T06:41:56Z,OWNER,"I modified that config file to have this line instead: ``` ProxyPass /prefix/ http://127.0.0.1:8001/ nocanon ``` And then deployed it by running: flyctl deploy --build-arg DATASETTE_REF=main This does NOT seem to have fixed the bug: ``` ~ % curl -i 'https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid' HTTP/2 302 date: Fri, 11 Nov 2022 06:40:01 GMT server: Fly/b1863e2e7 (2022-11-09) location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 link: ; rel=preload content-type: text/plain x-proxied-by: Apache2 Debian via: 2 fly.io fly-request-id: 01GHJPNCF51CJ626EWZEHK2CH9-sjc ``` https://datasette-apache-proxy-demo.datasette.io/prefix/-/versions seems to confirm that this is the latest deployed version (0.63), so it looks like the deploy worked.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311286593,https://api.github.com/repos/simonw/datasette/issues/1883,1311286593,IC_kwDOBm6k_c5OKKVB,9599,simonw,2022-11-11T06:34:09Z,2022-11-11T06:34:09Z,OWNER,"https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxypass includes this note: > Normally, mod_proxy will canonicalise ProxyPassed URLs. But this may be incompatible with some backends, particularly those that make use of *PATH_INFO*. The optional *nocanon* keyword suppresses this and passes the URL path ""raw"" to the backend.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311284537,https://api.github.com/repos/simonw/datasette/issues/1883,1311284537,IC_kwDOBm6k_c5OKJ05,9599,simonw,2022-11-11T06:30:38Z,2022-11-11T06:30:38Z,OWNER,"Is there a chance that it's Apache that's messing with that `location:` header here, not Datasette? https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/demos/apache-proxy/000-default.conf#L7-L13","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311283301,https://api.github.com/repos/simonw/datasette/issues/1883,1311283301,IC_kwDOBm6k_c5OKJhl,9599,simonw,2022-11-11T06:28:38Z,2022-11-11T06:29:33Z,OWNER,"`path_with_added_args(request, redirect_params)` should be preserving the current path from the request. https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/utils/__init__.py#L273-L286","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311282970,https://api.github.com/repos/simonw/datasette/issues/1883,1311282970,IC_kwDOBm6k_c5OKJca,9599,simonw,2022-11-11T06:28:05Z,2022-11-11T06:28:05Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/bbaab3b38ec2ce5944239ffbe2dd53328df40fff/datasette/views/table.py#L227-L249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311280709,https://api.github.com/repos/simonw/datasette/issues/1883,1311280709,IC_kwDOBm6k_c5OKI5F,9599,simonw,2022-11-11T06:25:27Z,2022-11-11T06:25:27Z,OWNER,"I tried adding this test but it passed! I expected it to fail: ```python def test_base_url_affects_filter_redirects(app_client_base_url_prefix): response = app_client_base_url_prefix.get( ""/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid"" ) assert response.status == 302 assert ( response.headers[""location""] == ""/prefix/fixtures/binary_data?_sort=rowid&rowid__exact=1"" ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311278678,https://api.github.com/repos/simonw/datasette/issues/1883,1311278678,IC_kwDOBm6k_c5OKIZW,9599,simonw,2022-11-11T06:22:37Z,2022-11-11T06:22:37Z,OWNER,"If you view source on that page the HTML looks correct: ```html
    ``` (I just added a test that confirms this too.) But... it looks like the bug is in the redirection code. https://datasette-apache-proxy-demo.datasette.io/prefix/fixtures/binary_data?_filter_column=rowid&_filter_op=exact&_filter_value=1&_sort=rowid returns the following: location: /fixtures/binary_data?_sort=rowid&rowid__exact=1 Which is incorrect.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1883#issuecomment-1311273461,https://api.github.com/repos/simonw/datasette/issues/1883,1311273461,IC_kwDOBm6k_c5OKHH1,9599,simonw,2022-11-11T06:16:08Z,2022-11-11T06:16:08Z,OWNER,"Great catch, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435917503,Errors when using table filters behind a proxy, https://github.com/simonw/datasette/issues/1880#issuecomment-1311273063,https://api.github.com/repos/simonw/datasette/issues/1880,1311273063,IC_kwDOBm6k_c5OKHBn,9599,simonw,2022-11-11T06:15:28Z,2022-11-11T06:15:28Z,OWNER,"The `_internal` database is intended to help Datasette handle much larger attached databases. Right now Datasette attempts to show every database on the https://latest.datasette.io/ index page and every table on the https://latest.datasette.io/fixtures database index page - but these are not paginated. If you had a database containing 1,000 tables the database index page would get pretty slow. So I want to be able to paginate (and search) those. But to paginate them it's useful to have them in a database table itself, since then I can paginate using SQL. My plan for `_internal` is to use it to implement those advanced browsing features. I've not completed this work yet though. See this issue for more details on that: - #417","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1880#issuecomment-1311271298,https://api.github.com/repos/simonw/datasette/issues/1880,1311271298,IC_kwDOBm6k_c5OKGmC,9599,simonw,2022-11-11T06:12:29Z,2022-11-11T06:12:29Z,OWNER,"I think you may have misunderstood this feature. This is talking about the `_internal` in-memory database, which maintains a set of tables that list the databases and tables that are attached to Datasette. They're not a copy of the data itself - just a list of table names, column names and database names. You can see what that database looks like by signing in as root - running `datasette --root` and clicking the link. Or you can see an example here: - Click the button on https://latest.datasette.io/login-as-root - Now visit https://latest.datasette.io/_internal For the example instance that looks like this: The two most interesting tables in there are these ones: As you can see, it's just the table schema itself and the columns that make up the tables. Even if you have hundreds of databases connected each with hundreds of tables this should still only add up to a few MB of RAM.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1884#issuecomment-1311269045,https://api.github.com/repos/simonw/datasette/issues/1884,1311269045,IC_kwDOBm6k_c5OKGC1,9599,simonw,2022-11-11T06:08:28Z,2022-11-11T06:08:28Z,OWNER,Does that work if you add `--load-extension spatialite`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1884#issuecomment-1309735529,https://api.github.com/repos/simonw/datasette/issues/1884,1309735529,IC_kwDOBm6k_c5OEPpp,25778,eyeseast,2022-11-10T03:57:23Z,2022-11-10T03:57:23Z,CONTRIBUTOR,Here's how to get a list of virtual tables: https://stackoverflow.com/questions/46617118/how-to-fetch-names-of-virtual-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1439009231,Exclude virtual tables from datasette inspect, https://github.com/simonw/datasette/issues/1871#issuecomment-1309650806,https://api.github.com/repos/simonw/datasette/issues/1871,1309650806,IC_kwDOBm6k_c5OD692,3556,davidbgk,2022-11-10T01:38:58Z,2022-11-10T01:38:58Z,CONTRIBUTOR,"> Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth. > > This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly. I agree (that's what I did with the previous insert plugin), maybe a complete example using `fetch()` in the documentation would be valuable as a “Getting started with the API” or similar?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1304539296,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1304539296,IC_kwDOCGYnMM5NwbCg,559711,ryascott,2022-11-05T12:40:12Z,2022-11-05T12:40:12Z,NONE,"I had the problem this morning when running: `Python==3.9.6 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30 ` I upgraded to: `Python ==3.10.8 sqlite3.sqlite_version==3.37.2 sqlite-utils==3.30 ` and the error did not appear anymore. Hope this helps Ryan ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304320521,IC_kwDOCGYnMM5NvloJ,7908073,chapmanjacobd,2022-11-04T22:54:09Z,2022-11-04T22:59:54Z,CONTRIBUTOR,I ran `PRAGMA integrity_check` and it returned `ok`. but then I tried restoring from a backup and I didn't get this `IntegrityError: constraint failed` error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed", https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945,https://api.github.com/repos/simonw/sqlite-utils/issues/511,1304078945,IC_kwDOCGYnMM5Nuqph,7908073,chapmanjacobd,2022-11-04T19:38:36Z,2022-11-04T20:13:17Z,CONTRIBUTOR,"Even more bizarre, the source db only has one record and the target table has no conflicting record: ``` 875 0.3s lb:/ (main|✚2) [0|0]🌺 sqlite-utils tube_71.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq [ { ""size"": null, ""time_created"": null, ""play_count"": 1, ""language"": null, ""view_count"": null, ""width"": null, ""height"": null, ""fps"": null, ""average_rating"": null, ""live_status"": null, ""age_limit"": null, ""uploader"": null, ""time_played"": 0, ""path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"", ""id"": ""088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"", ""ie_key"": ""ArchiveOrg"", ""playlist_path"": ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz"", ""duration"": 1424.05, ""tags"": null, ""title"": ""074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv"" } ] 875 0.3s lb:/ (main|✚2) [0|0]🥧 sqlite-utils video.db 'select * from media where path = ""https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz""' | jq [] ``` I've been able to use this code successfully several times before so not sure what's causing the issue. I guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1436539554,"[insert_all, upsert_all] IntegrityError: constraint failed", https://github.com/simonw/datasette/issues/1882#issuecomment-1302818784,https://api.github.com/repos/simonw/datasette/issues/1882,1302818784,IC_kwDOBm6k_c5Np2_g,9599,simonw,2022-11-04T00:25:18Z,2022-11-04T16:12:39Z,OWNER,"On that basis I think the core API design should change to this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { ""name"": ""my new table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ] ""pk"": ""id"" } ``` This leaves room for a `""rows"": []` key at the root too. Having that as a child of `""table""` felt unintuitive to me, and I didn't like the way this looked either: ```json { ""table"": { ""name"": ""my_new_table"" }, ""rows"": [ {""id"": 1, ""title"": ""Title""} ] } ``` Weird to have the table `name` nested inside `table` when `rows` wasn't.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293,https://api.github.com/repos/simonw/sqlite-utils/issues/50,1303660293,IC_kwDOCGYnMM5NtEcF,7908073,chapmanjacobd,2022-11-04T14:38:36Z,2022-11-04T14:38:36Z,CONTRIBUTOR,where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",473083260,"""Too many SQL variables"" on large inserts", https://github.com/simonw/datasette/issues/1217#issuecomment-1303301786,https://api.github.com/repos/simonw/datasette/issues/1217,1303301786,IC_kwDOBm6k_c5Nrs6a,31312775,mattmalcher,2022-11-04T11:37:52Z,2022-11-04T11:37:52Z,NONE,"All seems to work well, but there are some glitches to do with proxies, see #1883 . Excited to use this :)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",802513359,Possible to deploy as a python app (for Rstudio connect server)?, https://github.com/simonw/datasette/issues/1217#issuecomment-1303299509,https://api.github.com/repos/simonw/datasette/issues/1217,1303299509,IC_kwDOBm6k_c5NrsW1,31312775,mattmalcher,2022-11-04T11:35:13Z,2022-11-04T11:35:13Z,NONE,"The following worked for deployment to RStudio / Posit Connect An app.py along the lines of: ```python from pathlib import Path from datasette.app import Datasette example_db = Path(__file__).parent / ""data"" / ""example.db"" # use connect 'Content URL' setting here to set app to /datasette/ ds = Datasette(files=[example_db], settings={""base_url"": ""/datasette/""}) ds._startup_invoked = True ds_app = ds.app() ``` Then to deploy, from within a virtualenv with `rsconnect-python` ```sh rsconnect write-manifest fastapi -p $VIRTUAL_ENV/bin/python -e app:ds_app -o . rsconnect deploy manifest manifest.json -n -t ""Example Datasette"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",802513359,Possible to deploy as a python app (for Rstudio connect server)?, https://github.com/simonw/datasette/issues/1882#issuecomment-1302818153,https://api.github.com/repos/simonw/datasette/issues/1882,1302818153,IC_kwDOBm6k_c5Np21p,9599,simonw,2022-11-04T00:23:58Z,2022-11-04T00:23:58Z,OWNER,"I made a decision here that this endpoint should also accept an optional `""rows"": [...]` list which is used to automatically create the table using a schema derived from those example rows (which then get inserted): - https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807,https://api.github.com/repos/simonw/datasette/issues/1862,1302817807,IC_kwDOBm6k_c5Np2wP,9599,simonw,2022-11-04T00:23:13Z,2022-11-04T00:23:13Z,OWNER,"I don't like this on `/db/table/-/insert` - I think it makes more sense to optionally pass a `""rows""` key to the `/db/-/create` endpoint instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1862#issuecomment-1302817500,https://api.github.com/repos/simonw/datasette/issues/1862,1302817500,IC_kwDOBm6k_c5Np2rc,9599,simonw,2022-11-04T00:22:31Z,2022-11-04T00:22:31Z,OWNER,"Maybe this is a feature added to the existing `/db/table/-/insert` endpoint? Bit weird that you can call that endpoint for a table that doesn't exist yet, but it fits the `sqlite-utils` way of creating tables which I've found very pleasant over the past few years. So perhaps the API looks like this: ``` POST ///-/insert Content-Type: application/json Authorization: Bearer dstok_ { ""create_table"": true, ""rows"": [ { ""column1"": ""value1"", ""column2"": ""value2"" }, { ""column1"": ""value3"", ""column2"": ""value4"" } ] } ``` The `create_table` option will cause the table to be created if it doesn't already exist. That means I probably also need a `""pk"": ""...""` column for setting a primary key if the table is being created ... and maybe other options that I invent for this other feature too? - #1882","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1871#issuecomment-1302815105,https://api.github.com/repos/simonw/datasette/issues/1871,1302815105,IC_kwDOBm6k_c5Np2GB,9599,simonw,2022-11-04T00:17:23Z,2022-11-04T00:17:23Z,OWNER,"I'll probably enhance it a bit more though, I want to provide a UI that lists all the tables you can explore and lets you click to pre-fill the forms with them. Though at that point what should I do about the other endpoints? Probably list those too. Gets a bit complex, especially with the row-level update and delete endpoints.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1302814693,https://api.github.com/repos/simonw/datasette/issues/1871,1302814693,IC_kwDOBm6k_c5Np1_l,9599,simonw,2022-11-04T00:16:36Z,2022-11-04T00:16:36Z,OWNER,"I can close this issue once I fix it so it no longer hard-codes a potentially invalid example endpoint: https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L24-L26 https://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L34-L35","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1881#issuecomment-1302813449,https://api.github.com/repos/simonw/datasette/issues/1881,1302813449,IC_kwDOBm6k_c5Np1sJ,9599,simonw,2022-11-04T00:14:07Z,2022-11-04T00:14:07Z,OWNER,"Tool is now live here: https://latest-1-0-dev.datasette.io/-/permissions Needs root perms, so access this first: https://latest-1-0-dev.datasette.io/login-as-root","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1302812918,https://api.github.com/repos/simonw/datasette/issues/1881,1302812918,IC_kwDOBm6k_c5Np1j2,9599,simonw,2022-11-04T00:13:05Z,2022-11-04T00:13:05Z,OWNER,Has tests now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1863#issuecomment-1302790013,https://api.github.com/repos/simonw/datasette/issues/1863,1302790013,IC_kwDOBm6k_c5Npv99,9599,simonw,2022-11-03T23:32:30Z,2022-11-03T23:32:30Z,OWNER,"I'm not going to allow updates to primary keys. If you need to do that, you can instead delete the record and then insert a new one with the new primary keys you wanted - or maybe use a custom SQL query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185,https://api.github.com/repos/simonw/datasette/issues/1851,1294224185,IC_kwDOBm6k_c5NJEs5,9599,simonw,2022-10-27T23:18:24Z,2022-11-03T23:26:05Z,OWNER,"So new API design is: ``` POST /db/table/-/insert Authorization: Bearer xxx Content-Type: application/json { ""row"": { ""id"": 1, ""name"": ""New record"" } } ``` Returns: ``` 201 Created { ""row"": [{ ""id"": 1, ""name"": ""New record"" }] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302785086,https://api.github.com/repos/simonw/datasette/issues/1863,1302785086,IC_kwDOBm6k_c5Npuw-,9599,simonw,2022-11-03T23:24:33Z,2022-11-03T23:24:56Z,OWNER,"Thinking more about validation: I'm considering if this should validate that columns which are defined as SQLite foreign keys are being updated to values that exist in those other tables. I like the sound of this. It seems like a sensible default behaviour for Datasette. And it fits with the fact that Datasette treats foreign keys specially elsewhere in the interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302760549,https://api.github.com/repos/simonw/datasette/issues/1863,1302760549,IC_kwDOBm6k_c5Npoxl,9599,simonw,2022-11-03T22:43:04Z,2022-11-03T23:21:31Z,OWNER,"The `id=(int, ...)` thing is weird, but is apparently Pydantic syntax for a required field? https://cs.github.com/starlite-api/starlite/blob/28ddc847c4cb072f0d5d21a9ecd5259711f12ec9/docs/usage/11-data-transfer-objects.md#L161 confirms: > 1. For required fields use a tuple of type + ellipsis, for example `(str, ...)`. > 2. For optional fields use a tuple of type + `None`, for example `(str, None)` > 3. To set a default value use a tuple of type + default value, for example `(str, ""Hello World"")`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302760382,https://api.github.com/repos/simonw/datasette/issues/1863,1302760382,IC_kwDOBm6k_c5Npou-,9599,simonw,2022-11-03T22:42:47Z,2022-11-03T22:42:47Z,OWNER,"```python print(create_model('document', id=(int, ...), title=(str, None)).schema_json(indent=2)) ``` ```json { ""title"": ""document"", ""type"": ""object"", ""properties"": { ""id"": { ""title"": ""Id"", ""type"": ""integer"" }, ""title"": { ""title"": ""Title"", ""type"": ""string"" } }, ""required"": [ ""id"" ] } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1863#issuecomment-1302759174,https://api.github.com/repos/simonw/datasette/issues/1863,1302759174,IC_kwDOBm6k_c5NpocG,9599,simonw,2022-11-03T22:40:47Z,2022-11-03T22:40:47Z,OWNER,"I'm considering Pydantic for this, see: - https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350 In particular the `create_model()` method: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation This would give me good validation. It would also, weirdly, give me the ability to output JSON schema. Maybe I could have this as the JSON schema for a row? `/db/table/-/json-schema`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029242,Update a single record in an existing table, https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350,https://api.github.com/repos/simonw/datasette/issues/1882,1302716350,IC_kwDOBm6k_c5Npd--,9599,simonw,2022-11-03T21:51:14Z,2022-11-03T22:35:54Z,OWNER,"Validating this JSON object is getting a tiny bit complex. I'm tempted to adopt https://pydantic-docs.helpmanual.io/ at this point. The `create_model` example on https://stackoverflow.com/questions/66168517/generate-dynamic-model-using-pydantic/66168682#66168682 is particularly relevant, especially when I work on this issue: - #1863 ```python from pydantic import create_model d = {""strategy"": {""name"": ""test_strat2"", ""periods"": 10}} Strategy = create_model(""Strategy"", **d[""strategy""]) print(Strategy.schema_json(indent=2)) ``` `create_model()`: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302721916,https://api.github.com/repos/simonw/datasette/issues/1882,1302721916,IC_kwDOBm6k_c5NpfV8,9599,simonw,2022-11-03T21:58:50Z,2022-11-03T21:59:17Z,OWNER,"Mocked up a quick HTML+JavaScript form for creating that JSON structure using some iteration against Copilot prompts: ```html
    /* JSON format:
    {
      ""table"": {
          ""name"": ""my new table"",
          ""columns"": [
              {
                  ""name"": ""id"",
                  ""type"": ""integer""
              },
              {
                  ""name"": ""title"",
                  ""type"": ""text""
              }
          ]
         ""pk"": ""id""
      }
    }
    
    HTML form with Javascript for creating this JSON:
    */


    Current columns:

      ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1882#issuecomment-1302715662,https://api.github.com/repos/simonw/datasette/issues/1882,1302715662,IC_kwDOBm6k_c5Npd0O,9599,simonw,2022-11-03T21:50:27Z,2022-11-03T21:50:27Z,OWNER,"API design for this: ``` POST /db/-/create Authorization: Bearer xxx Content-Type: application/json { ""table"": { ""name"": ""my new table"", ""columns"": [ { ""name"": ""id"", ""type"": ""integer"" }, { ""name"": ""title"", ""type"": ""text"" } ] ""pk"": ""id"" } } ``` Supported column types are: - `integer` - `text` - `float` (even though SQLite calls it a ""real"") - `blob` This matches my design for `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1435294468,`/db/-/create` API for creating tables, https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026,https://api.github.com/repos/simonw/datasette/issues/1843,1302679026,IC_kwDOBm6k_c5NpU3y,9599,simonw,2022-11-03T21:22:42Z,2022-11-03T21:22:42Z,OWNER,Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384,https://api.github.com/repos/simonw/datasette/issues/1843,1302678384,IC_kwDOBm6k_c5NpUtw,9599,simonw,2022-11-03T21:21:59Z,2022-11-03T21:21:59Z,OWNER,"I added extra debug info to `/-/threads` to see this for myself: ```diff diff --git a/datasette/app.py b/datasette/app.py index 02bd38f1..16579e28 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -969,6 +969,13 @@ class Datasette: ""threads"": [ {""name"": t.name, ""ident"": t.ident, ""daemon"": t.daemon} for t in threads ], + ""file_connections"": { + db.name: [ + [dict(r) for r in conn.execute(""pragma database_list"").fetchall()] + for conn in db._all_file_connections + ] + for db in self.databases.values() + }, } # Only available in Python 3.7+ if hasattr(asyncio, ""all_tasks""): ``` Output after hitting refresh on a few `/fixtures` tables to ensure more threads started: ``` ""file_connections"": { ""_internal"": [], ""fixtures"": [ [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ], [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ], [ { ""seq"": 0, ""name"": ""main"", ""file"": ""/Users/simon/Dropbox/Development/datasette/fixtures.db"" } ] ] }, ``` I decided not to ship this feature though as it leaks the names of internal database files.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332,https://api.github.com/repos/simonw/datasette/issues/1843,1302634332,IC_kwDOBm6k_c5NpJ9c,9599,simonw,2022-11-03T20:34:56Z,2022-11-03T20:34:56Z,OWNER,"Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem. I'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330,https://api.github.com/repos/simonw/datasette/issues/1843,1302574330,IC_kwDOBm6k_c5No7T6,9599,simonw,2022-11-03T19:30:22Z,2022-11-03T19:30:22Z,OWNER,"This is affecting me a lot at the moment, on my laptop (runs fine in CI). Here's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`: ```diff diff --git a/tests/conftest.py b/tests/conftest.py index f4638a14..21d433c1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,6 +1,7 @@ import httpx import os import pathlib +import psutil import pytest import re import subprocess @@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory): yield ds_proc, uds # Shut it down at the end of the pytest session ds_proc.terminate() + + +def pytest_runtest_teardown(item: pytest.Item) -> None: + open_files = psutil.Process().open_files() + assert len(open_files) < 5 ``` The first error I get from this with `pytest --pdb -x` is here: ``` tests/test_api.py ............E >>>>> traceback >>>>> item = def pytest_runtest_teardown(item: pytest.Item) -> None: open_files = psutil.Process().open_files() > assert len(open_files) < 5 E AssertionError: assert 5 < 5 E + where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)]) /Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError >>>>> entering PDB >>>>> >>>>> PDB post_mortem (IO-capturing turned off) >>>>> > /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown() -> assert len(open_files) < 5 ``` That's this test: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673 Which uses this fixture: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231 Which calls this function: https://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L105-L122 So now I'm suspicious that, even though the fixture is meant to be session scoped, the way I'm using `with tempfile.TemporaryDirectory() as tmpdir:` is causing a whole load of files to be created and held open which are not later closed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1855#issuecomment-1301646670,https://api.github.com/repos/simonw/datasette/issues/1855,1301646670,IC_kwDOBm6k_c5NlY1O,9599,simonw,2022-11-03T05:11:26Z,2022-11-03T05:11:26Z,OWNER,That still needs comprehensive tests before I land it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1301646493,https://api.github.com/repos/simonw/datasette/issues/1855,1301646493,IC_kwDOBm6k_c5NlYyd,9599,simonw,2022-11-03T05:11:06Z,2022-11-03T05:11:06Z,OWNER,"Built a prototype of the above: ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 32b0c758..f68aa38f 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -6,8 +6,8 @@ import json import time -@hookimpl(tryfirst=True) -def permission_allowed(datasette, actor, action, resource): +@hookimpl(tryfirst=True, specname=""permission_allowed"") +def permission_allowed_default(datasette, actor, action, resource): async def inner(): if action in ( ""permissions-debug"", @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource): return inner +@hookimpl(specname=""permission_allowed"") +def permission_allowed_actor_restrictions(actor, action, resource): + if actor is None: + return None + _r = actor.get(""_r"") + if not _r: + # No restrictions, so we have no opinion + return None + action_initials = """".join([word[0] for word in action.split(""-"")]) + # If _r is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = _r.get(""a"") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if action_initials in all_allowed: + return None + # How about for the current database? + if action in (""view-database"", ""view-database-download"", ""execute-sql""): + database_allowed = _r.get(""d"", {}).get(resource) + if database_allowed is not None: + assert isinstance(database_allowed, list) + if action_initials in database_allowed: + return None + # Or the current table? That's any time the resource is (database, table) + if not isinstance(resource, str) and len(resource) == 2: + database, table = resource + table_allowed = _r.get(""t"", {}).get(database, {}).get(table) + # TODO: What should this do for canned queries? + if table_allowed is not None: + assert isinstance(table_allowed, list) + if action_initials in table_allowed: + return None + # This action is not specifically allowed, so reject it + return False + + @hookimpl def actor_from_request(datasette, request): prefix = ""dstok_"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1881#issuecomment-1301639741,https://api.github.com/repos/simonw/datasette/issues/1881,1301639741,IC_kwDOBm6k_c5NlXI9,9599,simonw,2022-11-03T04:58:21Z,2022-11-03T04:58:21Z,OWNER,"The whole `database_name` or `(database_name, table_name)` tuple for resource is a bit of a code smell. Maybe this is a chance to tidy that up too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301639370,https://api.github.com/repos/simonw/datasette/issues/1881,1301639370,IC_kwDOBm6k_c5NlXDK,9599,simonw,2022-11-03T04:57:21Z,2022-11-03T04:57:21Z,OWNER,"The plugin hook would be called `register_permissions()`, for consistency with `register_routes()` and `register_commands()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301638918,https://api.github.com/repos/simonw/datasette/issues/1881,1301638918,IC_kwDOBm6k_c5NlW8G,9599,simonw,2022-11-03T04:56:06Z,2022-11-03T04:56:06Z,OWNER,"I've also introduced a new concept of a permission abbreviation, which like the permission name needs to be globally unique. That's a problem for plugins - they might just be able to guarantee that their permission long-form name is unique among other plugins (through sensible naming conventions) but the thing where they declare a initial-letters-only abbreviation is far more risky. I think abbreviations are optional - they are provided for core permissions but plugins are advised not to use them. Also Datasette could check that the installed plugins do not provide conflicting permissions on startup and refuse to start if they do.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301638156,https://api.github.com/repos/simonw/datasette/issues/1881,1301638156,IC_kwDOBm6k_c5NlWwM,9599,simonw,2022-11-03T04:54:00Z,2022-11-03T04:54:00Z,OWNER,"If I have the permissions defined like this: ```python PERMISSIONS = ( Permission(""view-instance"", ""vi"", False, False, True), Permission(""view-database"", ""vd"", True, False, True), Permission(""view-database-download"", ""vdd"", True, False, True), Permission(""view-table"", ""vt"", True, True, True), Permission(""view-query"", ""vq"", True, True, True), Permission(""insert-row"", ""ir"", True, True, False), Permission(""delete-row"", ""dr"", True, True, False), Permission(""drop-table"", ""dt"", True, True, False), Permission(""execute-sql"", ""es"", True, False, True), Permission(""permissions-debug"", ""pd"", False, False, False), Permission(""debug-menu"", ""dm"", False, False, False), ) ``` Instead of just calling them by their undeclared names in places like this: ```python await self.ds.permission_allowed( request.actor, ""execute-sql"", database, default=True ) ``` On the one hand I can ditch that confusing `default=True` option - whether a permission is on by default becomes a characteristic of that `Permission()` itself, which feels much neater. On the other hand though, plugins that introduce their own permissions - like https://datasette.io/plugins/datasette-edit-schema - will need a way to register those permissions with Datasette core. Probably another plugin hook.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301635906,https://api.github.com/repos/simonw/datasette/issues/1881,1301635906,IC_kwDOBm6k_c5NlWNC,9599,simonw,2022-11-03T04:48:09Z,2022-11-03T04:48:09Z,OWNER,"I built this prototype on the http://127.0.0.1:8001/-/allow-debug page, which is open to anyone to visit. But... I just realized that using this tool can leak information - you can use it to guess the names of invisible databases and tables and run theoretical permission checks against them. Using the tool also pollutes the list of permission checks that show up on the root-anlo `/-/permissions` page. So.... I'm going to restrict the usage of this tool to users with access to `/-/permissions` and put it on that page instead. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1881#issuecomment-1301635340,https://api.github.com/repos/simonw/datasette/issues/1881,1301635340,IC_kwDOBm6k_c5NlWEM,9599,simonw,2022-11-03T04:46:41Z,2022-11-03T04:46:41Z,OWNER,"Built this prototype: ![prototype](https://user-images.githubusercontent.com/9599/199649219-f146e43b-bfb5-45e6-9777-956f21a79887.gif) In building it I realized I needed to know which permissions took a table, a database, both or neither. So I had to bake that into the code. Here's the prototype so far (which includes a prototype of the logic for the `_r` field on actor, see #1855): ```diff diff --git a/datasette/default_permissions.py b/datasette/default_permissions.py index 32b0c758..f68aa38f 100644 --- a/datasette/default_permissions.py +++ b/datasette/default_permissions.py @@ -6,8 +6,8 @@ import json import time -@hookimpl(tryfirst=True) -def permission_allowed(datasette, actor, action, resource): +@hookimpl(tryfirst=True, specname=""permission_allowed"") +def permission_allowed_default(datasette, actor, action, resource): async def inner(): if action in ( ""permissions-debug"", @@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource): return inner +@hookimpl(specname=""permission_allowed"") +def permission_allowed_actor_restrictions(actor, action, resource): + if actor is None: + return None + _r = actor.get(""_r"") + if not _r: + # No restrictions, so we have no opinion + return None + action_initials = """".join([word[0] for word in action.split(""-"")]) + # If _r is defined then we use those to further restrict the actor + # Crucially, we only use this to say NO (return False) - we never + # use it to return YES (True) because that might over-ride other + # restrictions placed on this actor + all_allowed = _r.get(""a"") + if all_allowed is not None: + assert isinstance(all_allowed, list) + if action_initials in all_allowed: + return None + # How about for the current database? + if action in (""view-database"", ""view-database-download"", ""execute-sql""): + database_allowed = _r.get(""d"", {}).get(resource) + if database_allowed is not None: + assert isinstance(database_allowed, list) + if action_initials in database_allowed: + return None + # Or the current table? That's any time the resource is (database, table) + if not isinstance(resource, str) and len(resource) == 2: + database, table = resource + table_allowed = _r.get(""t"", {}).get(database, {}).get(table) + # TODO: What should this do for canned queries? + if table_allowed is not None: + assert isinstance(table_allowed, list) + if action_initials in table_allowed: + return None + # This action is not specifically allowed, so reject it + return False + + @hookimpl def actor_from_request(datasette, request): prefix = ""dstok_"" diff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html index 0f1b30f0..ae43f0f5 100644 --- a/datasette/templates/allow_debug.html +++ b/datasette/templates/allow_debug.html @@ -35,7 +35,7 @@ p.message-warning {

      Use this tool to try out different actor and allow combinations. See Defining permissions with ""allow"" blocks for documentation.

      -
      +

      @@ -55,4 +55,82 @@ p.message-warning { {% if result == ""False"" %}

      Result: deny

      {% endif %} +

      Test permission check

      + +

      This tool lets you simulate an actor and a permission check for that actor.

      + + + +
      +

      + +
      +
      +

      +

      + +

      +

      +
      +
      + +
      + + + + + {% endblock %} diff --git a/datasette/views/special.py b/datasette/views/special.py index 9922a621..d46fc280 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -1,6 +1,8 @@ import json +from datasette.permissions import PERMISSIONS from datasette.utils.asgi import Response, Forbidden from datasette.utils import actor_matches_allow, add_cors_headers +from datasette.permissions import PERMISSIONS from .base import BaseView import secrets import time @@ -138,9 +140,34 @@ class AllowDebugView(BaseView): ""error"": ""\n\n"".join(errors) if errors else """", ""actor_input"": actor_input, ""allow_input"": allow_input, + ""permissions"": PERMISSIONS, }, ) + async def post(self, request): + vars = await request.post_vars() + actor = json.loads(vars[""actor""]) + permission = vars[""permission""] + resource_1 = vars[""resource_1""] + resource_2 = vars[""resource_2""] + resource = [] + if resource_1: + resource.append(resource_1) + if resource_2: + resource.append(resource_2) + resource = tuple(resource) + result = await self.ds.permission_allowed( + actor, permission, resource, default=""USE_DEFAULT"" + ) + return Response.json( + { + ""actor"": actor, + ""permission"": permission, + ""resource"": resource, + ""result"": result, + } + ) + class MessagesDebugView(BaseView): name = ""messages_debug"" ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365,Tool for simulating permission checks against actors, https://github.com/simonw/datasette/issues/1855#issuecomment-1301594495,https://api.github.com/repos/simonw/datasette/issues/1855,1301594495,IC_kwDOBm6k_c5NlMF_,9599,simonw,2022-11-03T03:11:17Z,2022-11-03T03:11:17Z,OWNER,"Maybe the way to do this is through a new standard mechanism on the actor: a set of additional restrictions, e.g.: ``` { ""id"": ""root"", ""_r"": { ""a"": [""ir"", ""ur"", ""dr""], ""d"": { ""fixtures"": [""ir"", ""ur"", ""dr""] }, ""t"": { ""fixtures"": { ""searchable"": [""ir""] } } } ``` `""a""` is ""all permissions"" - these apply to everything. `""d""` permissions only apply to the specified database `""t""` permissions only apply to the specified table The way this works is there's a default [permission_allowed(datasette, actor, action, resource)](https://docs.datasette.io/en/stable/plugin_hooks.html#id25) hook which only consults these, and crucially just says NO if those rules do not match. In this way it would apply as an extra layer of permission rules over the defaults (which for this `root` instance would all return yes).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1880#issuecomment-1301043042,https://api.github.com/repos/simonw/datasette/issues/1880,1301043042,IC_kwDOBm6k_c5NjFdi,525934,amitkoth,2022-11-02T18:20:14Z,2022-11-02T18:20:14Z,NONE,"Follow on question - is all memory use @simonw - for both datasette and SQLlite confined to the ""query time"" itself i.e. the memory use is relevant only to a particular transaction or query - and then subsequently released?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1433576351,Datasette with many and large databases > Memory use, https://github.com/simonw/datasette/issues/1871#issuecomment-1299607082,https://api.github.com/repos/simonw/datasette/issues/1871,1299607082,IC_kwDOBm6k_c5Ndm4q,9599,simonw,2022-11-02T05:45:31Z,2022-11-02T05:45:31Z,OWNER,"I'm going to add a link to the Datasette API docs for the current running version of Datasette, e.g. to https://docs.datasette.io/en/0.63/json_api.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299600257,https://api.github.com/repos/simonw/datasette/issues/1871,1299600257,IC_kwDOBm6k_c5NdlOB,9599,simonw,2022-11-02T05:36:40Z,2022-11-02T05:36:40Z,OWNER,"The API Explorer should definitely link to the `/-/create-token` page for users who have permission though. And it should probably go in the Datasette application menu?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299599461,https://api.github.com/repos/simonw/datasette/issues/1871,1299599461,IC_kwDOBm6k_c5NdlBl,9599,simonw,2022-11-02T05:35:36Z,2022-11-02T05:36:15Z,OWNER,"Here's a slightly wild idea: what if there was a button on `/-/api` that you could click to turn on ""API explorer mode"" for the rest of the Datasette interface - which sets a cookie, and that cookie means you then see ""API explorer"" links in all sorts of other relevant places in the Datasette UI (maybe tucked away in cog menus). Only reason I don't want to show these to everyone is that I don't think this is a very user-friendly feature: if you don't know what an API is I don't want to expose you to it unnecessarily.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299598570,https://api.github.com/repos/simonw/datasette/issues/1871,1299598570,IC_kwDOBm6k_c5Ndkzq,9599,simonw,2022-11-02T05:34:28Z,2022-11-02T05:34:28Z,OWNER,"This is pretty useful now. Two features I still want to add: - The ability to link to the API explorer such that the form is pre-filled with material from the URL. Need to guard against clickjacking first though, so no-one can link to it in an invisible iframe and trick the user into hitting POST. - Some kind of list of endpoints so people can click links to start using the API explorer. A list of every table the user can write to with each of their `/db/table/-/insert` endpoints for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299597066,https://api.github.com/repos/simonw/datasette/issues/1871,1299597066,IC_kwDOBm6k_c5NdkcK,9599,simonw,2022-11-02T05:32:22Z,2022-11-02T05:32:22Z,OWNER,"Demo of the latest API explorer: ![explorer](https://user-images.githubusercontent.com/9599/199406184-1292df42-25ea-4daf-8b54-ca26170ec1ea.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299388341,https://api.github.com/repos/simonw/datasette/issues/1871,1299388341,IC_kwDOBm6k_c5Ncxe1,9599,simonw,2022-11-02T00:24:28Z,2022-11-02T00:25:00Z,OWNER,"I want JSON syntax highlighting. https://github.com/luyilin/json-format-highlight is an MIT licensed tiny highlighter that looks decent for this. https://unpkg.com/json-format-highlight@1.0.1/dist/json-format-highlight.js","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1299349741,https://api.github.com/repos/simonw/datasette/issues/1871,1299349741,IC_kwDOBm6k_c5NcoDt,9599,simonw,2022-11-01T23:22:55Z,2022-11-01T23:22:55Z,OWNER,"It's weird that the API explorer only lets you explore POST APIs. It should probably also let you explore GET APIs, or be renamed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1879#issuecomment-1299098458,https://api.github.com/repos/simonw/datasette/issues/1879,1299098458,IC_kwDOBm6k_c5Nbqta,9599,simonw,2022-11-01T20:27:40Z,2022-11-01T20:33:52Z,OWNER,"https://github.com/simonw/datasette-x-forwarded-host/blob/main/datasette_x_forwarded_host/__init__.py could happen in core controlled by: `--setting trust_forwarded_host 1`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299102108,https://api.github.com/repos/simonw/datasette/issues/1879,1299102108,IC_kwDOBm6k_c5Nbrmc,9599,simonw,2022-11-01T20:30:54Z,2022-11-01T20:33:06Z,OWNER,One idea: add a `/-/debug` page (or `/-/tips` or `/-/checks`) which shows the incoming requests headers and could even detect if there's an `x-forwarded-host` header that isn't being repeated and show a tip on how to fix that.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299102755,https://api.github.com/repos/simonw/datasette/issues/1879,1299102755,IC_kwDOBm6k_c5Nbrwj,9599,simonw,2022-11-01T20:31:37Z,2022-11-01T20:31:37Z,OWNER,And some JavaScript that can spot if Datasette thinks it is being served over HTTP when it's actually being served over HTTPS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299096850,https://api.github.com/repos/simonw/datasette/issues/1879,1299096850,IC_kwDOBm6k_c5NbqUS,9599,simonw,2022-11-01T20:26:12Z,2022-11-01T20:26:12Z,OWNER,"The other relevant plugin here is https://datasette.io/plugins/datasette-x-forwarded-host Maybe that should be rolled into core too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1879#issuecomment-1299090678,https://api.github.com/repos/simonw/datasette/issues/1879,1299090678,IC_kwDOBm6k_c5Nboz2,9599,simonw,2022-11-01T20:20:28Z,2022-11-01T20:20:28Z,OWNER,My first step in debugging these is to install https://datasette.io/plugins/datasette-debug-asgi - but now I'm thinking maybe something like that should be part of core.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432037325,Make it easier to fix URL proxy problems, https://github.com/simonw/datasette/issues/1862#issuecomment-1299073433,https://api.github.com/repos/simonw/datasette/issues/1862,1299073433,IC_kwDOBm6k_c5NbkmZ,9599,simonw,2022-11-01T20:04:31Z,2022-11-01T20:04:31Z,OWNER,"It really feels like this should be accompanied by a `/db/-/create` API for creating tables. I had to add that to `sqlite-utils` eventually (initially it only supported creating by passing in an example document): https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1878#issuecomment-1299071456,https://api.github.com/repos/simonw/datasette/issues/1878,1299071456,IC_kwDOBm6k_c5NbkHg,9599,simonw,2022-11-01T20:02:43Z,2022-11-01T20:02:43Z,OWNER,"Note that ""update"" is partially covered by the `replace` option to `/-/insert`, added here: - https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704,/db/table/-/upsert API, https://github.com/simonw/datasette/issues/1873#issuecomment-1298919552,https://api.github.com/repos/simonw/datasette/issues/1873,1298919552,IC_kwDOBm6k_c5Na_CA,9599,simonw,2022-11-01T18:11:27Z,2022-11-01T18:11:27Z,OWNER,"I forgot to document `ignore` and `replace`. Also I need to add tests that cover: - Forgetting to include a primary key on a non-autoincrement table - Compound primary keys - Rowid only tables with and without rowid specified I think my validation logic here will get caught out by the fact that `rowid` does not show up as a valid column name: https://github.com/simonw/datasette/blob/9bec7c38eb93cde5afb16df9bdd96aea2a5b0459/datasette/views/table.py#L1151-L1160 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1298905135,https://api.github.com/repos/simonw/datasette/issues/1873,1298905135,IC_kwDOBm6k_c5Na7gv,9599,simonw,2022-11-01T17:59:59Z,2022-11-01T17:59:59Z,OWNER,"It's a bit surprising that you can send `""ignore"": true, ""return_rows"": true` and the returned `""inserted""` key will list rows that were NOT inserted (since they were ignored). Three options: 1. Ignore that and document it 2. Fix it so `""inserted""` only returns rows that were actually inserted (bit tricky) 3. Change the name of `""inserted""` to something else I'm picking 3 - I'm going to change it to be called `""rows""` instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1298885451,https://api.github.com/repos/simonw/datasette/issues/1873,1298885451,IC_kwDOBm6k_c5Na2tL,9599,simonw,2022-11-01T17:42:20Z,2022-11-01T17:42:20Z,OWNER,"Design decision: ```json { ""rows"": [{""id"": 1, ""title"": ""The title""}], ""ignore"": true } ``` Or `""replace"": true`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298879701,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298879701,IC_kwDOCGYnMM5Na1TV,9599,simonw,2022-11-01T17:37:13Z,2022-11-01T17:37:13Z,OWNER,"The question I was originally trying to answer here was this: how many rows were actually inserted by that call to `.insert_all()`? I don't know that `.rowcount` would ever be useful here, since the ""correct"" answer depends on other factors - had I determined to ignore or replace records with a primary key that matches an existing record for example? So I think if people need `rowcount` they can get it by using a `cursor` directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1298877872,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1298877872,IC_kwDOCGYnMM5Na02w,9599,simonw,2022-11-01T17:35:30Z,2022-11-01T17:35:30Z,OWNER,"This may not make sense. First, `.last_rowid` is a property on table - but that doesn't make sense for `rowcount` since it should clearly be a property on the database itself (you can run a query directly using `db.execute()` without going through a `Table` object). So I tried this prototype: ```diff diff --git a/docs/python-api.rst b/docs/python-api.rst index 206e5e6..78d3a8d 100644 --- a/docs/python-api.rst +++ b/docs/python-api.rst @@ -186,6 +186,15 @@ The ``db.query(sql)`` function executes a SQL query and returns an iterator over # {'name': 'Cleo'} # {'name': 'Pancakes'} +After executing a query the ``db.rowcount`` property on that database instance will reflect the number of rows affected by any insert, update or delete operations performed by that query: + +.. code-block:: python + + db = Database(memory=True) + db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) + print(db.rowcount) + # Outputs: 2 + .. _python_api_execute: db.execute(sql, params) diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a06f4b7..c19c2dd 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -294,6 +294,8 @@ class Database: _counts_table_name = ""_counts"" use_counts_table = False + # Number of rows inserted, updated or deleted + rowcount: Optional[int] = None def __init__( self, @@ -480,9 +482,11 @@ class Database: if self._tracer: self._tracer(sql, parameters) if parameters is not None: - return self.conn.execute(sql, parameters) + cursor = self.conn.execute(sql, parameters) else: - return self.conn.execute(sql) + cursor = self.conn.execute(sql) + self.rowcount = cursor.rowcount + return cursor def executescript(self, sql: str) -> sqlite3.Cursor: """""" ``` But this happens: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}])
      >>> db.rowcount -1 ``` Turning on query tracing demonstrates why: ```pycon >>> db = Database(memory=True, tracer=print) PRAGMA recursive_triggers=on; None >>> db[""dogs""].insert_all([{""name"": ""Cleo""}, {""name"": ""Pancakes""}]) select name from sqlite_master where type = 'view' None select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'view' None CREATE TABLE [dogs] ( [name] TEXT ); None select name from sqlite_master where type = 'view' None INSERT INTO [dogs] ([name]) VALUES (?), (?); ['Cleo', 'Pancakes'] select name from sqlite_master where type = 'table' None select name from sqlite_master where type = 'table' None PRAGMA table_info([dogs]) None
      >>> ``` The `.insert_all()` function does a bunch of other queries too, so `.rowcount` is quickly over-ridden by the same result from extra queries that it executed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/datasette/issues/1876#issuecomment-1298856054,https://api.github.com/repos/simonw/datasette/issues/1876,1298856054,IC_kwDOBm6k_c5Navh2,9599,simonw,2022-11-01T17:16:01Z,2022-11-01T17:16:01Z,OWNER,`ta.style.height = ta.scrollHeight + 'px'` is an easy way to do that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1431786951,SQL query should wrap on SQL interrupted screen, https://github.com/simonw/datasette/issues/1876#issuecomment-1298854321,https://api.github.com/repos/simonw/datasette/issues/1876,1298854321,IC_kwDOBm6k_c5NavGx,9599,simonw,2022-11-01T17:14:33Z,2022-11-01T17:14:33Z,OWNER,"I could use a `textarea` here (would need to figure out a neat pattern to expand it to fit the query): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1431786951,SQL query should wrap on SQL interrupted screen, https://github.com/simonw/sqlite-utils/issues/507#issuecomment-1297859539,https://api.github.com/repos/simonw/sqlite-utils/issues/507,1297859539,IC_kwDOCGYnMM5NW8PT,7908073,chapmanjacobd,2022-11-01T00:40:16Z,2022-11-01T00:40:16Z,CONTRIBUTOR,"Ideally people could fix their data if they run into this issue. If you are using filenames try [convmv](https://linux.die.net/man/1/convmv) ``` convmv --preserve-mtimes -f utf8 -t utf8 --notest -i -r . ``` maybe this script will also help: ```py import argparse, shutil from pathlib import Path import ftfy from xklb import utils from xklb.utils import log def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser() parser.add_argument(""paths"", nargs='*') parser.add_argument(""--verbose"", ""-v"", action=""count"", default=0) args = parser.parse_args() log.info(utils.dict_filter_bool(args.__dict__)) return args def rename_invalid_paths() -> None: args = parse_args() for path in args.paths: log.info(path) for p in sorted([str(p) for p in Path(path).rglob(""*"")], key=len): fixed = ftfy.fix_text(p, uncurl_quotes=False).replace(""\r\n"", ""\n"").replace(""\r"", ""\n"").replace(""\n"", """") if p != fixed: try: shutil.move(p, fixed) except FileNotFoundError: log.warning(""FileNotFound. %s"", p) else: log.info(fixed) if __name__ == ""__main__"": rename_invalid_paths() ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430325103,conn.execute: UnicodeEncodeError: 'utf-8' codec can't encode character, https://github.com/simonw/sqlite-utils/pull/508#issuecomment-1297754631,https://api.github.com/repos/simonw/sqlite-utils/issues/508,1297754631,IC_kwDOCGYnMM5NWioH,22429695,codecov[bot],2022-10-31T22:14:48Z,2022-10-31T22:53:59Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **96.25**% // Head: **96.09**% // Decreases project coverage by **`-0.15%`** :warning: > Coverage data is based on head [(`2d6a149`)](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`529110e`)](https://codecov.io/gh/simonw/sqlite-utils/commit/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 63.63% of modified lines in pull request are covered. > :exclamation: Current head 2d6a149 differs from pull request most recent head 43a8c4c. Consider uploading reports for the commit 43a8c4c to get more accurate results
      Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #508 +/- ## ========================================== - Coverage 96.25% 96.09% -0.16% ========================================== Files 4 4 Lines 2401 2407 +6 ========================================== + Hits 2311 2313 +2 - Misses 90 94 +4 ``` | [Impacted Files](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [sqlite\_utils/db.py](https://codecov.io/gh/simonw/sqlite-utils/pull/508/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `96.79% <63.63%> (-0.30%)` | :arrow_down: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
      [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/sqlite-utils/pull/508?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1430563092,Allow surrogates in parameters, https://github.com/simonw/sqlite-utils/issues/448#issuecomment-1297703307,https://api.github.com/repos/simonw/sqlite-utils/issues/448,1297703307,IC_kwDOCGYnMM5NWWGL,167893,mcarpenter,2022-10-31T21:23:51Z,2022-10-31T21:27:32Z,CONTRIBUTOR,"The Windows aspect is a red herring: OP's sample above produces the same error on Linux. (Though I don't know what's going on with the CI). The same error can also be obtained by passing an `io` from a file opened in non-binary mode (`'r'` as opposed to `'rb'`) to `rows_from_file()`. This is how I got here. The fix for my case is easy: open the file in mode `'rb'`. The analagous fix for OP's problem also works: use `BytesIO` in place of `StringIO`. Minimal test case (derived from [utils.py](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/utils.py#L304)): ``` python import io from typing import cast #fp = io.StringIO(""id,name\n1,Cleo"") # error fp = io.BytesIO(bytes(""id,name\n1,Cleo"", encoding='utf-8')) # okay reader = io.BufferedReader(cast(io.RawIOBase, fp)) reader.peek(1) # exception thrown here ``` I see the signature of `rows_from_file()` correctly has `fp: BinaryIO` but I guess you'd need either a runtime type check for that (not all `io`s have `mode()`), or to catch the `AttributeError` on `peek()` to produce a better error for users. Neither option is ideal. Some thoughts on testing binary-ness of `io`s in this SO question: https://stackoverflow.com/questions/44584829/how-to-determine-if-file-is-opened-in-binary-or-text-mode","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1279144769,Reading rows from a file => AttributeError: '_io.StringIO' object has no attribute 'readinto', https://github.com/dogsheep/twitter-to-sqlite/issues/61#issuecomment-1297201971,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/61,1297201971,IC_kwDODEm0Qs5NUbsz,3153638,Profpatsch,2022-10-31T14:47:58Z,2022-10-31T14:47:58Z,NONE,There’s also a limit of 3200 tweets. I wonder if that can be circumvented somehow.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1077560091,"Data Pull fails for ""Essential"" level access to the Twitter API (for Documentation)", https://github.com/simonw/datasette/issues/1864#issuecomment-1296403316,https://api.github.com/repos/simonw/datasette/issues/1864,1296403316,IC_kwDOBm6k_c5NRYt0,9599,simonw,2022-10-31T00:39:43Z,2022-10-31T00:39:43Z,OWNER,"It looks like SQLite has features for this already: https://www.sqlite.org/foreignkeys.html#fk_actions > Foreign key ON DELETE and ON UPDATE clauses are used to configure actions that take place when deleting rows from the parent table (ON DELETE), or modifying the parent key values of existing rows (ON UPDATE). A single foreign key constraint may have different actions configured for ON DELETE and ON UPDATE. Foreign key actions are similar to triggers in many ways. On that basis, I'm not going to implement anything additional in the `.../-/delete` endpoint relating to foreign keys. Developers who want special treatment of them can do that with a combination of a plugin (maybe I'll build a `datasette-enable-foreign-keys` plugin) and tables created using those `ON DELETE` clauses.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1864#issuecomment-1296402071,https://api.github.com/repos/simonw/datasette/issues/1864,1296402071,IC_kwDOBm6k_c5NRYaX,9599,simonw,2022-10-31T00:37:09Z,2022-10-31T00:37:09Z,OWNER,"I need to think about what happens if you delete a row that is the target of a foreign key from another row. https://www.sqlite.org/foreignkeys.html#fk_enable shows that SQLite will only actively enforce these relationships (e.g. throw an error if you try to delete a row that is referenced by another row) if you first run `PRAGMA foreign_keys = ON;` against the connection. > Foreign key constraints are disabled by default (for backwards compatibility), so must be enabled separately for each [database connection](https://www.sqlite.org/c3ref/sqlite3.html). (Note, however, that future releases of SQLite might change so that foreign key constraints enabled by default. Careful developers will not make any assumptions about whether or not foreign keys are enabled by default but will instead enable or disable them as necessary.) I don't actually believe that the SQLite maintainers will ever make that the default though. Datasette doesn't turn these on at the moment, but it could be turned on by a `prepare_connection()` plugin. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1864#issuecomment-1296375536,https://api.github.com/repos/simonw/datasette/issues/1864,1296375536,IC_kwDOBm6k_c5NRR7w,9599,simonw,2022-10-30T23:17:11Z,2022-10-30T23:17:11Z,OWNER,I'm a bit nervous about calling `.delete()` with the `pk_values` - can I be sure they are in the correct order? https://github.com/simonw/datasette/blob/00632ded30e7cf9f0cf9478680645d1dabe269ae/datasette/views/row.py#L188-L190,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1864#issuecomment-1296375310,https://api.github.com/repos/simonw/datasette/issues/1864,1296375310,IC_kwDOBm6k_c5NRR4O,9599,simonw,2022-10-30T23:16:19Z,2022-10-30T23:16:19Z,OWNER,Still needs tests that cover compound primary keys and rowid tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425029275,Delete a single record from an existing table, https://github.com/simonw/datasette/issues/1874#issuecomment-1296363981,https://api.github.com/repos/simonw/datasette/issues/1874,1296363981,IC_kwDOBm6k_c5NRPHN,9599,simonw,2022-10-30T22:19:47Z,2022-10-30T22:19:47Z,OWNER,Documentation: https://docs.datasette.io/en/1.0-dev/json_api.html#dropping-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429030341,API to drop a table, https://github.com/simonw/sqlite-utils/issues/506#issuecomment-1296358636,https://api.github.com/repos/simonw/sqlite-utils/issues/506,1296358636,IC_kwDOCGYnMM5NRNzs,9599,simonw,2022-10-30T21:52:11Z,2022-10-30T21:52:11Z,OWNER,This could work in a similar way to `db.insert(...).last_rowid`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1429029604,Make `cursor.rowcount` accessible (wontfix), https://github.com/simonw/datasette/issues/1873#issuecomment-1296343716,https://api.github.com/repos/simonw/datasette/issues/1873,1296343716,IC_kwDOBm6k_c5NRKKk,9599,simonw,2022-10-30T20:24:55Z,2022-10-30T20:24:55Z,OWNER,"I think the key feature I need here is going to be the equivalent of `ignore=True` and `replace=True` for dealing with primary key collisions, see https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite_utils.db.Table.insert","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343317,https://api.github.com/repos/simonw/datasette/issues/1873,1296343317,IC_kwDOBm6k_c5NRKEV,9599,simonw,2022-10-30T20:22:40Z,2022-10-30T20:22:40Z,OWNER,"So maybe they're not actually worth worrying about separately, because they are guaranteed to have a primary key set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343173,https://api.github.com/repos/simonw/datasette/issues/1873,1296343173,IC_kwDOBm6k_c5NRKCF,9599,simonw,2022-10-30T20:21:54Z,2022-10-30T20:22:20Z,OWNER,"One last case to consider: `WITHOUT ROWID` tables. https://www.sqlite.org/withoutrowid.html > By default, every row in SQLite has a special column, usually called the ""[rowid](https://www.sqlite.org/lang_createtable.html#rowid)"", that uniquely identifies that row within the table. However if the phrase ""WITHOUT ROWID"" is added to the end of a [CREATE TABLE](https://www.sqlite.org/lang_createtable.html) statement, then the special ""rowid"" column is omitted. There are sometimes space and performance advantages to omitting the rowid. > > ... > > Every WITHOUT ROWID table must have a [PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#primkeyconst). An error is raised if a CREATE TABLE statement with the WITHOUT ROWID clause lacks a PRIMARY KEY.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296343014,https://api.github.com/repos/simonw/datasette/issues/1873,1296343014,IC_kwDOBm6k_c5NRJ_m,9599,simonw,2022-10-30T20:21:01Z,2022-10-30T20:21:01Z,OWNER,"Actually, for simplicity I'm going to say that you can always set the primary key, even for auto-incrementing primary key columns... but you cannot set it on pure `rowid` columns.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296342814,https://api.github.com/repos/simonw/datasette/issues/1873,1296342814,IC_kwDOBm6k_c5NRJ8e,9599,simonw,2022-10-30T20:20:05Z,2022-10-30T20:20:05Z,OWNER,"Some notes on what Datasette does already https://latest.datasette.io/fixtures/tags.json?_shape=array returns: ```json [ { ""tag"": ""canine"" }, { ""tag"": ""feline"" } ] ``` That table is defined [like this](https://latest.datasette.io/fixtures/tags): ```sql CREATE TABLE tags ( tag TEXT PRIMARY KEY ); ``` Here's a `rowid` table with no explicit primary key: https://latest.datasette.io/fixtures/binary_data https://latest.datasette.io/fixtures/binary_data.json?_shape=array ```json [ { ""rowid"": 1, ""data"": { ""$base64"": true, ""encoded"": ""FRwCx60F/g=="" } }, { ""rowid"": 2, ""data"": { ""$base64"": true, ""encoded"": ""FRwDx60F/g=="" } }, { ""rowid"": 3, ""data"": null } ] ``` ```sql CREATE TABLE binary_data ( data BLOB ); ``` https://latest.datasette.io/fixtures/simple_primary_key has a text primary key: https://latest.datasette.io/fixtures/simple_primary_key.json?_shape=array ```json [ { ""id"": ""1"", ""content"": ""hello"" }, { ""id"": ""2"", ""content"": ""world"" }, { ""id"": ""3"", ""content"": """" }, { ""id"": ""4"", ""content"": ""RENDER_CELL_DEMO"" }, { ""id"": ""5"", ""content"": ""RENDER_CELL_ASYNC"" } ] ``` ```sql CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ); ``` https://latest.datasette.io/fixtures/compound_primary_key is a compound primary key. https://latest.datasette.io/fixtures/compound_primary_key.json?_shape=array ```json [ { ""pk1"": ""a"", ""pk2"": ""b"", ""content"": ""c"" }, { ""pk1"": ""a/b"", ""pk2"": "".c-d"", ""content"": ""c"" } ] ``` ```sql CREATE TABLE compound_primary_key ( pk1 varchar(30), pk2 varchar(30), content text, PRIMARY KEY (pk1, pk2) ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341469,https://api.github.com/repos/simonw/datasette/issues/1873,1296341469,IC_kwDOBm6k_c5NRJnd,9599,simonw,2022-10-30T20:13:50Z,2022-10-30T20:13:50Z,OWNER,"I checked and SQLite itself does allow you to set the `rowid` on that kind of table - it then increments from whatever you inserted: ``` % sqlite3 /tmp/t.db SQLite version 3.39.4 2022-09-07 20:51:41 Enter "".help"" for usage hints. sqlite> create table docs (title text); sqlite> insert into docs (title) values ('one'); sqlite> select rowid, title from docs; 1|one sqlite> insert into docs (rowid, title) values (3, 'three'); sqlite> select rowid, title from docs; 1|one 3|three sqlite> insert into docs (title) values ('another'); sqlite> select rowid, title from docs; 1|one 3|three 4|another ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1873#issuecomment-1296341055,https://api.github.com/repos/simonw/datasette/issues/1873,1296341055,IC_kwDOBm6k_c5NRJg_,9599,simonw,2022-10-30T20:11:47Z,2022-10-30T20:12:30Z,OWNER,"If a table has an auto-incrementing primary key, should you be allowed to insert records with an explicit key into it? I'm torn on this one. It's something you can do with direct database access, but it's something I very rarely want to do. I'm inclined to disallow it and say that if you want that you can get it using a writable canned query instead. Likewise, I'm not going to provide a way to set the `rowid` explicitly on a freshly inserted row.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339386,https://api.github.com/repos/simonw/datasette/issues/1871,1296339386,IC_kwDOBm6k_c5NRJG6,9599,simonw,2022-10-30T20:03:04Z,2022-10-30T20:03:04Z,OWNER,"I do need to skip CSRF for these API calls. I'm going to start out by doing that using the `skip_csrf()` hook to skip CSRF checks on anything with a `content-type: application/json` request header. ```python @hookimpl def skip_csrf(scope): if scope[""type""] == ""http"": headers = scope.get(""headers"") if dict(headers).get(b'content-type') == b'application/json': return True ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296339205,https://api.github.com/repos/simonw/datasette/issues/1871,1296339205,IC_kwDOBm6k_c5NRJEF,9599,simonw,2022-10-30T20:02:05Z,2022-10-30T20:02:05Z,OWNER,"Realized the API explorer doesn't need the API key piece at all - it can work with standard cookie-based auth. This also reflects how most plugins are likely to use this API, where they'll be adding JavaScript that uses `fetch()` to call the write API directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296131872,https://api.github.com/repos/simonw/datasette/issues/1871,1296131872,IC_kwDOBm6k_c5NQWcg,9599,simonw,2022-10-30T06:27:56Z,2022-10-30T06:27:56Z,OWNER,Initial prototype API explorer is now live at https://latest-1-0-dev.datasette.io/-/api,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1873#issuecomment-1296131681,https://api.github.com/repos/simonw/datasette/issues/1873,1296131681,IC_kwDOBm6k_c5NQWZh,9599,simonw,2022-10-30T06:27:12Z,2022-10-30T06:27:12Z,OWNER,Relevant TODO: https://github.com/simonw/datasette/blob/c35859ae3df163406f1a1895ccf9803e933b2d8e/datasette/views/table.py#L1131-L1135,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428630253,Ensure insert API has good tests for rowid and compound primark key tables, https://github.com/simonw/datasette/issues/1872#issuecomment-1296131343,https://api.github.com/repos/simonw/datasette/issues/1872,1296131343,IC_kwDOBm6k_c5NQWUP,9599,simonw,2022-10-30T06:26:01Z,2022-10-30T06:26:01Z,OWNER,"Good spot fixing that! Sorry about this - it was a change in Datasette 0.63 which should have been better called out. My goal for Datasette 1.0 (which I aim to have out by the end of the year) is to introduce a formal process for avoiding problems like this, with very clear documentation when something like this might happen.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/issues/1871#issuecomment-1296130073,https://api.github.com/repos/simonw/datasette/issues/1871,1296130073,IC_kwDOBm6k_c5NQWAZ,9599,simonw,2022-10-30T06:20:56Z,2022-10-30T06:20:56Z,OWNER,"That initial prototype looks like this: It currently shows the returned JSON from the API in an `alert()`. Next I should make that part of the page instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296126389,https://api.github.com/repos/simonw/datasette/issues/1871,1296126389,IC_kwDOBm6k_c5NQVG1,9599,simonw,2022-10-30T06:04:48Z,2022-10-30T06:04:48Z,OWNER,"This is even more important now I have pushed: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1871#issuecomment-1296114136,https://api.github.com/repos/simonw/datasette/issues/1871,1296114136,IC_kwDOBm6k_c5NQSHY,9599,simonw,2022-10-30T05:15:40Z,2022-10-30T05:15:40Z,OWNER,"Host it at `/-/api` It's an input box with a path in and a textarea you can put JSON in, plus a submit button to post the request. It lists the API endpoints you can use - click on a link to populate the form field plus a example. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1427293909,API explorer tool, https://github.com/simonw/datasette/issues/1872#issuecomment-1296080804,https://api.github.com/repos/simonw/datasette/issues/1872,1296080804,IC_kwDOBm6k_c5NQJ-k,192568,mroswell,2022-10-30T03:06:32Z,2022-10-30T03:06:32Z,CONTRIBUTOR,"I updated datasette-publish-vercel to 0.14.2 in requirements.txt And the site is back up! Is there a way that we can get some sort of notice when something like this will have critical impact on website function?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/issues/1872#issuecomment-1296076803,https://api.github.com/repos/simonw/datasette/issues/1872,1296076803,IC_kwDOBm6k_c5NQJAD,192568,mroswell,2022-10-30T02:50:34Z,2022-10-30T02:50:34Z,CONTRIBUTOR,"should this issue be under https://github.com/simonw/datasette-publish-vercel/issues ? Perhaps I just need to update: datasette-publish-vercel==0.11 in requirements.txt? I'll try that and see what happens... ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1428560020,"SITE-BUSTING ERROR: ""render_template() called before await ds.invoke_startup()""", https://github.com/simonw/datasette/pull/1870#issuecomment-1295667649,https://api.github.com/repos/simonw/datasette/issues/1870,1295667649,IC_kwDOBm6k_c5NOlHB,536941,fgregg,2022-10-29T00:52:43Z,2022-10-29T00:53:43Z,CONTRIBUTOR,"> Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command? Somehow, `datasette serve -i data.db` will lead to the `data.db` being modified, which will trigger a [copy-on-write](https://docs.docker.com/storage/storagedriver/#the-copy-on-write-cow-strategy) of `data.db` into the read-write layer of the container. I don't understand **how** that happens. it kind of feels like a bug in sqlite, but i can't quite follow the sqlite code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1295660092,https://api.github.com/repos/simonw/datasette/issues/1870,1295660092,IC_kwDOBm6k_c5NOjQ8,9599,simonw,2022-10-29T00:25:26Z,2022-10-29T00:25:26Z,OWNER,"Saw your comment here too: https://github.com/simonw/datasette/issues/1480#issuecomment-1271101072 > switching from `immutable=1` to `mode=ro` completely addressed this. see https://github.com/simonw/datasette/issues/1836#issuecomment-1271100651 for details. So maybe we need a special case for containers that are intended to be run using Docker - the ones produced by `datasette package` and `datasette publish cloudrun`? Those are cases where the `-i` option should actually be opened in read-only mode, not immutable mode. Maybe a `datasette serve --irw data.db` option for opening a file in immutable-but-actually-read-only mode? Bit ugly though. I should run some benchmarks to figure out if `immutable` really does offer significant performance benefits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1295657771,https://api.github.com/repos/simonw/datasette/issues/1870,1295657771,IC_kwDOBm6k_c5NOisr,9599,simonw,2022-10-29T00:19:03Z,2022-10-29T00:19:03Z,OWNER,"Just saw your comment here: https://github.com/simonw/datasette/issues/1836#issuecomment-1272357976 > when you are running from docker, you **always** will want to run as `mode=ro` because the same thing that is causing duplication in the inspect layer will cause duplication in the final container read/write layer when `datasette serve` runs. I don't understand this. My mental model of how Docker works is that the image itself is created using `docker build`... but then when the image runs later on (`docker run`) the image itself isn't touched at all. Are you saying that I can build a container, but then when I run it and it does `datasette serve -i data.db ...` it will somehow modify the image, or create a new modified filesystem layer in the runtime environment, as a result of running that `serve` command?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/issues/1866#issuecomment-1295200988,https://api.github.com/repos/simonw/datasette/issues/1866,1295200988,IC_kwDOBm6k_c5NMzLc,9599,simonw,2022-10-28T16:29:55Z,2022-10-28T16:29:55Z,OWNER,"I wonder if there's something clever I could do here within a transaction? Start a transaction. Write out a temporary in-memory table with all of the existing primary keys in the table. Run the bulk insert. Then run `select pk from table where pk not in (select pk from old_pks)` to see what has changed. I don't think that's going to work well for large tables. I'm going to go with not returning inserted rows by default, unless you pass a special option requesting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1294408928,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1294408928,IC_kwDOCGYnMM5NJxzg,39538958,justmars,2022-10-28T03:36:56Z,2022-10-28T03:37:50Z,NONE,"With respect to the typing of Table class itself, my interim solution: ```python from sqlite_utils.db import Table def tbl(self, table_name: str) -> Table: tbl = self.db[table_name] if isinstance(tbl, Table): return tbl raise Exception(f""Missing {table_name=}"") ``` With respect to @chapmanjacobd concern on the `DEFAULT` being an empty class, have also been using `# type: ignore`, e.g. ```python @classmethod def insert_list(cls, areas: list[str]): return meta.tbl(meta.Areas).insert_all( ({""area"": a} for a in areas), ignore=True # type: ignore ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/datasette/issues/1866#issuecomment-1294316640,https://api.github.com/repos/simonw/datasette/issues/1866,1294316640,IC_kwDOBm6k_c5NJbRg,9599,simonw,2022-10-28T01:51:40Z,2022-10-28T01:51:40Z,OWNER,"This needs to support the following: - Rows do not include a primary key - one is assigned by the database - Rows provide their own primary key, any clashes are errors - Rows provide their own primary key, clashes are silently ignored - Rows provide their own primary key, replacing any existing records","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294306071,https://api.github.com/repos/simonw/datasette/issues/1866,1294306071,IC_kwDOBm6k_c5NJYsX,9599,simonw,2022-10-28T01:37:14Z,2022-10-28T01:37:59Z,OWNER,"Quick crude benchmark: ```python import sqlite3 db = sqlite3.connect("":memory:"") def create_table(db, name): db.execute(f""create table {name} (id integer primary key, title text)"") create_table(db, ""single"") create_table(db, ""multi"") create_table(db, ""bulk"") def insert_singles(titles): inserted = [] for title in titles: cursor = db.execute(f""insert into single (title) values (?)"", [title]) inserted.append((cursor.lastrowid, title)) return inserted def insert_many(titles): db.executemany(f""insert into multi (title) values (?)"", ((t,) for t in titles)) def insert_bulk(titles): db.execute(""insert into bulk (title) values {}"".format( "", "".join(""(?)"" for _ in titles) ), titles) titles = [""title {}"".format(i) for i in range(1, 10001)] ``` Then in iPython I ran these: ``` In [14]: %timeit insert_singles(titles) 23.8 ms ± 535 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [13]: %timeit insert_many(titles) 12 ms ± 520 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [12]: %timeit insert_bulk(titles) 2.59 ms ± 25 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) ``` So the bulk insert really is a lot faster - 3ms compared to 24ms for single inserts, so ~8x faster.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1294296767,https://api.github.com/repos/simonw/datasette/issues/1866,1294296767,IC_kwDOBm6k_c5NJWa_,9599,simonw,2022-10-28T01:22:25Z,2022-10-28T01:23:09Z,OWNER,"Nasty catch on this one: I wanted to return the IDs of the freshly inserted rows. But... the `insert_all()` method I was planning to use from `sqlite-utils` doesn't appear to have a way of doing that: https://github.com/simonw/sqlite-utils/blob/529110e7d8c4a6b1bbf5fb61f2e29d72aa95a611/sqlite_utils/db.py#L2813-L2835 SQLite itself added a `RETURNING` statement which might help, but that is only available from version 3.35 released in March 2021: https://www.sqlite.org/lang_returning.html - which isn't commonly available yet. https://latest.datasette.io/-/versions right now shows 3.34, and https://lite.datasette.io/#/-/versions shows 3.27.2 (from Feb 2019). Two options then: 1. Even for bulk inserts do one insert at a time so I can use `cursor.lastrowid` to get the ID of the inserted record. This isn't terrible since SQLite is very fast, but it may still be a big performance hit for large inserts. 2. Don't return the list of inserted rows for bulk inserts 3. Default to not returning the list of inserted rows for bulk inserts, but allow the user to request that - in which case we use the slower path That third option might be the way to go here. I should benchmark first to figure out how much of a difference this actually makes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/pull/1870#issuecomment-1294285471,https://api.github.com/repos/simonw/datasette/issues/1870,1294285471,IC_kwDOBm6k_c5NJTqf,536941,fgregg,2022-10-28T01:06:03Z,2022-10-28T01:06:03Z,CONTRIBUTOR,"as far as i can tell, [this is where the ""immutable"" argument is used](https://github.com/sqlite/sqlite/blob/c97bb14fab566f6fa8d967c8fd1e90f3702d5b73/src/pager.c#L4926-L4931) in sqlite: ```c pPager->noLock = sqlite3_uri_boolean(pPager->zFilename, ""nolock"", 0); if( (iDc & SQLITE_IOCAP_IMMUTABLE)!=0 || sqlite3_uri_boolean(pPager->zFilename, ""immutable"", 0) ){ vfsFlags |= SQLITE_OPEN_READONLY; goto act_like_temp_file; } ``` so it does set the read only flag, but then has a goto.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/issues/1866#issuecomment-1294282263,https://api.github.com/repos/simonw/datasette/issues/1866,1294282263,IC_kwDOBm6k_c5NJS4X,9599,simonw,2022-10-28T01:00:42Z,2022-10-28T01:00:42Z,OWNER,"I'm going to set the limit at 1,000 rows inserted at a time. I'll make this configurable using a new `max_insert_rows` setting (for consistency with `max_returned_rows`).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294281451,https://api.github.com/repos/simonw/datasette/issues/1851,1294281451,IC_kwDOBm6k_c5NJSrr,9599,simonw,2022-10-28T00:59:25Z,2022-10-28T00:59:25Z,OWNER,"I'm going to use this endpoint for bulk inserts too, so I'm closing this issue and continuing the work here: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/pull/1870#issuecomment-1294238862,https://api.github.com/repos/simonw/datasette/issues/1870,1294238862,IC_kwDOBm6k_c5NJISO,22429695,codecov[bot],2022-10-27T23:44:25Z,2022-10-27T23:44:25Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.55**% // Head: **92.55**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`4faa4fd`)](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`bf00b0b`)](https://codecov.io/gh/simonw/datasette/commit/bf00b0b59b6692bdec597ac9db4e0b497c5a47b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.
      Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1870 +/- ## ======================================= Coverage 92.55% 92.55% ======================================= Files 35 35 Lines 4432 4432 ======================================= Hits 4102 4102 Misses 330 330 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1870/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.30% <ø> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
      [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1870?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/pull/1870#issuecomment-1294237783,https://api.github.com/repos/simonw/datasette/issues/1870,1294237783,IC_kwDOBm6k_c5NJIBX,536941,fgregg,2022-10-27T23:42:18Z,2022-10-27T23:42:18Z,CONTRIBUTOR,Relevant sqlite forum thread: https://www.sqlite.org/forum/forumpost/02f7bda329f41e30451472421cf9ce7f715b768ce3db02797db1768e47950d48,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426379903,"don't use immutable=1, only mode=ro", https://github.com/simonw/datasette/issues/1851#issuecomment-1289712350,https://api.github.com/repos/simonw/datasette/issues/1851,1289712350,IC_kwDOBm6k_c5M33Le,9599,simonw,2022-10-24T22:28:39Z,2022-10-27T23:18:48Z,OWNER,"API design: (**UPDATE: this was [later changed to POST /db/table/-/insert](https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185)) ``` POST /db/table Authorization: Bearer xxx Content-Type: application/json { ""row"": { ""id"": 1, ""name"": ""New record"" } } ``` Returns: ``` 201 Created { ""row"": { ""id"": 1, ""name"": ""New record"" } } ``` You can omit optional fields in the input, including the ID field. The returned object will always include all fields - and will even include `rowid` if your object doesn't have a primary key of its own. I decided to use `""row""` as the key in both request and response, to preserve space for other future keys - one that tells you that the table has been created, for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1869#issuecomment-1294181485,https://api.github.com/repos/simonw/datasette/issues/1869,1294181485,IC_kwDOBm6k_c5NI6Rt,9599,simonw,2022-10-27T22:24:37Z,2022-10-27T22:24:37Z,OWNER,"https://docs.datasette.io/en/stable/changelog.html#v0-63 Annotated release notes: https://simonwillison.net/2022/Oct/27/datasette-0-63/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/issues/1786#issuecomment-1294116493,https://api.github.com/repos/simonw/datasette/issues/1786,1294116493,IC_kwDOBm6k_c5NIqaN,9599,simonw,2022-10-27T21:50:12Z,2022-10-27T21:50:12Z,OWNER,Demo in Datasette Lite: https://lite.datasette.io/#/fixtures?sql=select%0A++pk1%2C%0A++pk2%2C%0A++content%2C%0A++sortable%2C%0A++sortable_with_nulls%2C%0A++sortable_with_nulls_2%2C%0A++text%0Afrom%0A++sortable%0Aorder+by%0A++pk1%2C%0A++pk2%0Alimit%0A++101,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1342430983,Adjust height of textarea for no JS case, https://github.com/simonw/datasette/issues/1869#issuecomment-1294105558,https://api.github.com/repos/simonw/datasette/issues/1869,1294105558,IC_kwDOBm6k_c5NInvW,9599,simonw,2022-10-27T21:44:13Z,2022-10-27T21:44:13Z,OWNER,I'm going to do annotated release notes for this one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/issues/1869#issuecomment-1294056552,https://api.github.com/repos/simonw/datasette/issues/1869,1294056552,IC_kwDOBm6k_c5NIbxo,9599,simonw,2022-10-27T21:00:02Z,2022-10-27T21:02:25Z,OWNER,"Those release notes as markdown: ### Features - Now tested against Python 3.11. Docker containers used by `datasette publish` and `datasette package` both now use that version of Python. ([#1853](https://github.com/simonw/datasette/issues/1853)) - `--load-extension` option now supports entrypoints. Thanks, Alex Garcia. ([#1789](https://github.com/simonw/datasette/pull/1789)) - Facet size can now be set per-table with the new `facet_size` table metadata option. ([#1804](https://github.com/simonw/datasette/issues/1804)) - The [truncate_cells_html](https://docs.datasette.io/en/stable/settings.html#setting-truncate-cells-html) setting now also affects long URLs in columns. ([#1805](https://github.com/simonw/datasette/issues/1805)) - The non-JavaScript SQL editor textarea now increases height to fit the SQL query. ([#1786](https://github.com/simonw/datasette/issues/1786)) - Facets are now displayed with better line-breaks in long values. Thanks, Daniel Rech. ([#1794](https://github.com/simonw/datasette/pull/1794)) - The `settings.json` file used in [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) is now validated on startup. ([#1816](https://github.com/simonw/datasette/issues/1816)) - SQL queries can now include leading SQL comments, using `/* ... */` or `-- ...` syntax. Thanks, Charles Nepote. ([#1860](https://github.com/simonw/datasette/issues/1860)) - SQL query is now re-displayed when terminated with a time limit error. ([#1819](https://github.com/simonw/datasette/issues/1819)) - The [inspect data](https://docs.datasette.io/en/stable/performance.html#performance-inspect) mechanism is now used to speed up server startup - thanks, Forest Gregg. ([#1834](https://github.com/simonw/datasette/issues/1834)) - In [Configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir) databases with filenames ending in `.sqlite` or `.sqlite3` are now automatically added to the Datasette instance. ([#1646](https://github.com/simonw/datasette/issues/1646)) - Breadcrumb navigation display now respects the current user's permissions. ([#1831](https://github.com/simonw/datasette/issues/1831)) ### Plugin hooks and internals - The [prepare_jinja2_environment(env, datasette)](https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hook-prepare-jinja2-environment) plugin hook now accepts an optional `datasette` argument. Hook implementations can also now return an `async` function which will be awaited automatically. ([#1809](https://github.com/simonw/datasette/issues/1809)) - `Database(is_mutable=)` now defaults to `True`. ([#1808](https://github.com/simonw/datasette/issues/1808)) - The [datasette.check_visibility()](https://docs.datasette.io/en/stable/internals.html#datasette-check-visibility) method now accepts an optional `permissions=` list, allowing it to take multiple permissions into account at once when deciding if something should be shown as public or private. This has been used to correctly display padlock icons in more places in the Datasette interface. ([#1829](https://github.com/simonw/datasette/issues/1829)) - Datasette no longer enforces upper bounds on its dependencies. ([#1800](https://github.com/simonw/datasette/issues/1800)) ### Documentation - New tutorial: [Cleaning data with sqlite-utils and Datasette](https://datasette.io/tutorials/clean-data). - Screenshots in the documentation are now maintained using [shot-scraper](https://shot-scraper.datasette.io/), as described in [Automating screenshots for the Datasette documentation using shot-scraper](https://simonwillison.net/2022/Oct/14/automating-screenshots/). ([#1844](https://github.com/simonw/datasette/issues/1844)) - More detailed command descriptions on the [CLI reference](https://docs.datasette.io/en/stable/cli-reference.html#cli-reference) page. ([#1787](https://github.com/simonw/datasette/issues/1787)) - New documentation on [Running Datasette using OpenRC](https://docs.datasette.io/en/stable/deploying.html#deploying-openrc) - thanks, Adam Simpson. ([#1825](https://github.com/simonw/datasette/pull/1825))","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426253476,Release 0.63, https://github.com/simonw/datasette/pull/1835#issuecomment-1294049178,https://api.github.com/repos/simonw/datasette/issues/1835,1294049178,IC_kwDOBm6k_c5NIZ-a,9599,simonw,2022-10-27T20:51:30Z,2022-10-27T20:51:30Z,OWNER,"See also: - https://github.com/simonw/datasette/pull/1837","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400121355,use inspect data for hash and file size, https://github.com/simonw/datasette/pull/1837#issuecomment-1294048849,https://api.github.com/repos/simonw/datasette/issues/1837,1294048849,IC_kwDOBm6k_c5NIZ5R,9599,simonw,2022-10-27T20:51:08Z,2022-10-27T20:51:08Z,OWNER,"Yeah this is better, thanks!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1400431789,Make hash and size a lazy property, https://github.com/simonw/datasette/pull/1839#issuecomment-1294034011,https://api.github.com/repos/simonw/datasette/issues/1839,1294034011,IC_kwDOBm6k_c5NIWRb,9599,simonw,2022-10-27T20:34:37Z,2022-10-27T20:34:37Z,OWNER,@dependabot rebase,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1401155623,Bump black from 22.8.0 to 22.10.0, https://github.com/simonw/datasette/issues/1851#issuecomment-1294012583,https://api.github.com/repos/simonw/datasette/issues/1851,1294012583,IC_kwDOBm6k_c5NIRCn,9599,simonw,2022-10-27T20:11:22Z,2022-10-27T20:11:22Z,OWNER,"And the response to `""inserted"": [{...}]` - it will be the same for bulk inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294012084,https://api.github.com/repos/simonw/datasette/issues/1851,1294012084,IC_kwDOBm6k_c5NIQ60,9599,simonw,2022-10-27T20:10:47Z,2022-10-27T20:10:47Z,OWNER,"I'm going to change the incoming JSON back to `{""row"": {...}}` - no need to POST `{""insert"": ...}` to something with `/-/insert` in the URL already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1294009354,https://api.github.com/repos/simonw/datasette/issues/1851,1294009354,IC_kwDOBm6k_c5NIQQK,9599,simonw,2022-10-27T20:07:42Z,2022-10-27T20:07:42Z,OWNER,"Need to implement the new URL design from: - #1868 This is now going to be `/db/table/-/insert` - and it will eventually handle bulk inserts as well as single inserts.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1868#issuecomment-1294008733,https://api.github.com/repos/simonw/datasette/issues/1868,1294008733,IC_kwDOBm6k_c5NIQGd,9599,simonw,2022-10-27T20:07:01Z,2022-10-27T20:07:01Z,OWNER,I'm happy with this `/db/table/-/action` design for the moment. Will review it once I've built it to see if I still like it!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294008282,https://api.github.com/repos/simonw/datasette/issues/1868,1294008282,IC_kwDOBm6k_c5NIP_a,9599,simonw,2022-10-27T20:06:34Z,2022-10-27T20:06:34Z,OWNER,"I'm going to stick with one `/-/insert` endpoint which handles both single row inserts AND multiple row inserts I think - partly because I don't want to build both `/-/upsert` and `/-/upsert-many`, I'd rather just have `/-/upsert`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294007024,https://api.github.com/repos/simonw/datasette/issues/1868,1294007024,IC_kwDOBm6k_c5NIPrw,9599,simonw,2022-10-27T20:05:44Z,2022-10-27T20:05:52Z,OWNER,"So given this scheme, the URL design would look like this: - `POST /db/table/-/insert` - insert a single row - `POST /db/table/-/insert-many` - insert multiple rows (might just keep that on `/-/insert` with a JSON array rather than object though) - `POST /db/table/-/drop` - drop a table - `POST /db/table/-/alter` - alter a table - `POST /db/table/-/upsert` - upsert, https://sqlite-utils.datasette.io/en/stable/python-api.html#upserting-data - `POST /db/table/-/create` - could be an endpoint for explicitly creating a table, or should that live at `/db/-/create` instead? And for rows (`pks` here since compound primary keys are supported): - `POST /db/table/pks/-/update` - update row - `POST /db/table/pks/-/delete` - delete row","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294004308,https://api.github.com/repos/simonw/datasette/issues/1868,1294004308,IC_kwDOBm6k_c5NIPBU,9599,simonw,2022-10-27T20:03:08Z,2022-10-27T20:03:08Z,OWNER,The other option here would be to lean into custom HTTP verbs like `DELETE` and `PATCH`. I'm not sold on those: they've never given me any convincing wins over just using `POST` for the many times I've encountered them in my career to date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1868#issuecomment-1294003701,https://api.github.com/repos/simonw/datasette/issues/1868,1294003701,IC_kwDOBm6k_c5NIO31,9599,simonw,2022-10-27T20:02:26Z,2022-10-27T20:02:26Z,OWNER,"The problem with the above design is that I want to support a bunch of different actions that can be taken against a table: - insert a single row - insert multiple rows - bulk update rows - rename table - alter table - drop table I could have ALL of those be a `POST /db/table` with different JSON root keys (`{""drop"": true}` for example, but this raises two problems: 1. Server logs that only show `POST /db/table` will be less useful, they won't reveal what action was performed 2. What happens if you send `{""insert"": {""title"": ""New record""}, ""drop"": true}`? Does that return an error, or does it perform both of those actions? This is already slightly confusing in that `POST /db/name-of-query` is the existing API for executing a writable canned query: https://docs.datasette.io/en/stable/sql_queries.html#json-api-for-writable-canned-queries So I'm ready to consider other design options. Initial thoughts on possible designs (for the single row insert case, but could be expanded to cover other verbs): - `POST /db/table?action=insert` - `POST /db/table?nsert` - `POST /db/table/-/insert` I quite like that third one: it feels consistent with the existing `/-/actor` etc pages that Datasette serves already. There's one slight confusion here in that it overlaps with the URL for a row with a primary key of `""-""` - which is currently at `/db/table/-` - but that might be OK. Especially if I say that child pages of rows must theselves use the `/-/` pattern. So to update or delet a row you would use: - `POST /db/table/row/-/update` - `POST /db/table/row/-/delete` So a row with primary key `-` would end up as `/db/table/row/-/-/update` - which I think is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426195437,Design URLs for the write API, https://github.com/simonw/datasette/issues/1851#issuecomment-1293996735,https://api.github.com/repos/simonw/datasette/issues/1851,1293996735,IC_kwDOBm6k_c5NINK_,9599,simonw,2022-10-27T19:54:53Z,2022-10-27T19:54:53Z,OWNER,"Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292997608,https://api.github.com/repos/simonw/datasette/issues/1851,1292997608,IC_kwDOBm6k_c5NEZPo,9599,simonw,2022-10-27T04:54:53Z,2022-10-27T19:05:50Z,OWNER,"I'm going to change the design of this to: ``` { ""insert"": { ""title"" :""..."" } } ``` Renaming `""row""` to `""insert""`. This will be consistent with adding `""drop"": true` for dropping a table, and maybe other verbs like for modifying the schema. The API response will look like this: ```json { ""inserted_row"": { ""id"": 1, ""title"": ""..."" } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1860#issuecomment-1293939737,https://api.github.com/repos/simonw/datasette/issues/1860,1293939737,IC_kwDOBm6k_c5NH_QZ,9599,simonw,2022-10-27T18:57:37Z,2022-10-27T18:57:37Z,OWNER,The new code is now live at https://latest.datasette.io/fixtures,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293928738,https://api.github.com/repos/simonw/datasette/issues/1860,1293928738,IC_kwDOBm6k_c5NH8ki,9599,simonw,2022-10-27T18:46:31Z,2022-10-27T18:46:31Z,OWNER,I think mine has a better pattern for handling `/* ... anything in here that isn't */ ... */`,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293928230,https://api.github.com/repos/simonw/datasette/issues/1860,1293928230,IC_kwDOBm6k_c5NH8cm,9599,simonw,2022-10-27T18:46:03Z,2022-10-27T18:46:03Z,OWNER,"Here's yours on Debuggex: https://www.debuggex.com/r/HjdJryTy9ezGsuWK ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293926417,https://api.github.com/repos/simonw/datasette/issues/1860,1293926417,IC_kwDOBm6k_c5NH8AR,9599,simonw,2022-10-27T18:44:20Z,2022-10-27T18:45:21Z,OWNER,"Hah, I just came up with this one - we were clearly working on this at the same time! `^\s*((?:\-\-.*?\n\s*)|(?:\/\*((?!\*\/)[\s\S])*\*\/)\s*)*\s*select\b` https://www.debuggex.com/r/Rbw-UWD9PdOU2GyO ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1293912781,https://api.github.com/repos/simonw/datasette/issues/1860,1293912781,IC_kwDOBm6k_c5NH4rN,562352,CharlesNepote,2022-10-27T18:31:15Z,2022-10-27T18:31:15Z,NONE,"Here is my suggestion: `^\s*((?:\-\-.*?\n\s*)|(?:/\*.*?(?=\*/)\*/\s*))*select\b` See the following test: https://regex101.com/r/Doeqqa/1 And here I played all your tests: https://regexr.com/713ir ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1866#issuecomment-1293893789,https://api.github.com/repos/simonw/datasette/issues/1866,1293893789,IC_kwDOBm6k_c5NH0Cd,9599,simonw,2022-10-27T18:13:00Z,2022-10-27T18:13:00Z,OWNER,If people care about that kind of thing they could always push all of their inserts to a table called `_tablename` and then atomically rename that once they've uploaded all of the data (assuming I provide an atomic-rename-this-table mechanism).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293892818,https://api.github.com/repos/simonw/datasette/issues/1866,1293892818,IC_kwDOBm6k_c5NHzzS,9599,simonw,2022-10-27T18:12:02Z,2022-10-27T18:12:02Z,OWNER,"There's one catch with batched inserts: if your CLI tool fails half way through you could end up with a partially populated table - since a bunch of batches will have succeeded first. I think that's OK. In the future I may want to come up with a way to run multiple batches of inserts inside a single transaction, but I can ignore that for the first release of this feature.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293891876,https://api.github.com/repos/simonw/datasette/issues/1866,1293891876,IC_kwDOBm6k_c5NHzkk,9599,simonw,2022-10-27T18:11:05Z,2022-10-27T18:11:05Z,OWNER,Likewise for newline-delimited JSON. While it's tempting to want to accept that as an ingest format (because it's nice to generate and stream) I think it's better to have a client application that can turn a stream of newline-delimited JSON into batched JSON inserts.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293891191,https://api.github.com/repos/simonw/datasette/issues/1866,1293891191,IC_kwDOBm6k_c5NHzZ3,9599,simonw,2022-10-27T18:10:22Z,2022-10-27T18:10:22Z,OWNER,"So for the moment I'm just going to concentrate on the JSON API. I can consider CSV variants later on, or as plugins, or both.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293890684,https://api.github.com/repos/simonw/datasette/issues/1866,1293890684,IC_kwDOBm6k_c5NHzR8,9599,simonw,2022-10-27T18:09:52Z,2022-10-27T18:09:52Z,OWNER,"Should this API accept CSV/TSV etc in addition to JSON? I'm torn on this one. My initial instinct is that it should not - and there should instead be a Datasette client library / CLI tool you can use that knows how to turn CSV into batches of JSON calls for when you want to upload a CSV file. I don't think the usability of `curl https://datasette/db/table -F 'data=@path/to/file.csv' -H 'Authentication: Bearer xxx'` is particularly great compared to something like`datasette client insert https://datasette/ db table file.csv --csv` (where the command version could store API tokens for you too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1866#issuecomment-1293887808,https://api.github.com/repos/simonw/datasette/issues/1866,1293887808,IC_kwDOBm6k_c5NHylA,9599,simonw,2022-10-27T18:07:02Z,2022-10-27T18:07:02Z,OWNER,"Error handling is really important here. What should happen if you submit 100 records and one of them has some kind of validation error? How should that error be reported back to you? I'm inclined to say that it defaults to all-or-nothing in a transaction - but there should be a `""continue_on_error"": true` option (or similar) which causes it to insert the ones that are valid while reporting back the ones that are invalid.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426001541,API for bulk inserting records into a table, https://github.com/simonw/datasette/issues/1860#issuecomment-1293863145,https://api.github.com/repos/simonw/datasette/issues/1860,1293863145,IC_kwDOBm6k_c5NHsjp,562352,CharlesNepote,2022-10-27T17:43:37Z,2022-10-27T17:43:37Z,NONE,"Sorry I forgot the `-- comments like that`. I'm afraid there is an issue in your regexp, see: https://regex101.com/r/pyubJf/1 I guess I can fix it. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1862#issuecomment-1293857306,https://api.github.com/repos/simonw/datasette/issues/1862,1293857306,IC_kwDOBm6k_c5NHrIa,9599,simonw,2022-10-27T17:38:17Z,2022-10-27T17:38:17Z,OWNER,"Strongly related to: - #1866","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425011030,"Create a new table from one or more records, `sqlite-utils` style", https://github.com/simonw/datasette/issues/1865#issuecomment-1293568194,https://api.github.com/repos/simonw/datasette/issues/1865,1293568194,IC_kwDOBm6k_c5NGkjC,9599,simonw,2022-10-27T13:58:26Z,2022-10-27T13:58:26Z,OWNER,"Here's the issue where I started doing this: - #849","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1425682079,Stop syncing main to master, https://github.com/simonw/datasette/issues/849#issuecomment-649908756,https://api.github.com/repos/simonw/datasette/issues/849,649908756,MDEyOklzc3VlQ29tbWVudDY0OTkwODc1Ng==,9599,simonw,2020-06-26T02:09:09Z,2022-10-27T13:57:08Z,OWNER,"I mentioned this issue here: https://simonwillison.net/2020/Jun/26/weeknotes-plugins-sqlite-generate/ Repositories created by following the README in https://github.com/simonw/datasette-template and https://github.com/simonw/click-app have a `main` branch instead of `master` so I have a few examples live now. https://github.com/simonw/datasette-saved-queries is one example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",639072811,Rename master branch to main, https://github.com/simonw/datasette/issues/1851#issuecomment-1292999579,https://api.github.com/repos/simonw/datasette/issues/1851,1292999579,IC_kwDOBm6k_c5NEZub,9599,simonw,2022-10-27T04:59:06Z,2022-10-27T04:59:12Z,OWNER,"I should probably refactor this to use `sqlite-utils`, since I'm going to want to use that later for the feature that automatically creates tables. Might make it easier to solve the rowid issues too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292996181,https://api.github.com/repos/simonw/datasette/issues/1851,1292996181,IC_kwDOBm6k_c5NEY5V,9599,simonw,2022-10-27T04:51:47Z,2022-10-27T04:51:47Z,OWNER,Also need a test for invalid JSON (currently triggers a 500 HTML error).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1855#issuecomment-1292962813,https://api.github.com/repos/simonw/datasette/issues/1855,1292962813,IC_kwDOBm6k_c5NEQv9,9599,simonw,2022-10-27T04:31:40Z,2022-10-27T04:31:40Z,OWNER,"My hunch on this is that anyone with that level of complex permissions requirements needs to be using a custom authentication plugin which includes much more concrete token rules, rather than the default signed stateless token implementation that ships with Datasette core.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1855#issuecomment-1292959886,https://api.github.com/repos/simonw/datasette/issues/1855,1292959886,IC_kwDOBm6k_c5NEQCO,9599,simonw,2022-10-27T04:30:07Z,2022-10-27T04:30:07Z,OWNER,"Here's an interesting edge-case to consider: what if a user creates themselves a token for a specific table, then deletes that table, and waits for another user to create a table of the same name... and then uses their previously created token to write to the table that someone else created? Not sure if this is a threat I need to actively consider, but it's worth thinking a little bit about the implications of such a thing - since there will be APIs that allow users to create tables, and there may be cases where people want to have a concept of users ""owning"" specific tables. This is probably something that could be left for plugins to solve, but it still needs to be understood and potentially documented. There may even be a world in which tracking the timestamp at which a table was created becomes useful - because that could then be baked into API tokens, such that a token created BEFORE the table was created does not grant access to that table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1851#issuecomment-1292952121,https://api.github.com/repos/simonw/datasette/issues/1851,1292952121,IC_kwDOBm6k_c5NEOI5,9599,simonw,2022-10-27T04:24:09Z,2022-10-27T04:24:20Z,OWNER,"And come up with a whole bunch of tests for weird table shapes, surprising column names, different types etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292951833,https://api.github.com/repos/simonw/datasette/issues/1851,1292951833,IC_kwDOBm6k_c5NEOEZ,9599,simonw,2022-10-27T04:23:40Z,2022-10-27T04:23:40Z,OWNER,Also need to think about transactions - it should use them!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292939146,https://api.github.com/repos/simonw/datasette/issues/1851,1292939146,IC_kwDOBm6k_c5NEK-K,9599,simonw,2022-10-27T04:00:17Z,2022-10-27T04:23:15Z,OWNER,"Documentation for this first draft of the API: https://docs.datasette.io/en/1.0-dev/json_api.html#inserting-a-single-row It currently returns errors as HTML - it needs to return errors as JSON. Also the errors need comprehensive test coverage. I'm also worried about what happens if you use it on a table that doesn't use an integer primary key - need to check that. I think this code may break: https://github.com/simonw/datasette/blob/51c436fed29205721dcf17fa31d7e7090d34ebb8/datasette/views/table.py#L155-L171 Plus will `rowid` tables without an explicit primary key return the `rowid` column? They should.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1850#issuecomment-1292940011,https://api.github.com/repos/simonw/datasette/issues/1850,1292940011,IC_kwDOBm6k_c5NELLr,9599,simonw,2022-10-27T04:01:59Z,2022-10-27T04:01:59Z,OWNER,"Working on that first ""insert row"" implementation: - https://github.com/simonw/datasette/issues/1851 Has made it very clear to me that I should go the whole hog and build the basic form-based interface for this as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1858#issuecomment-1292709818,https://api.github.com/repos/simonw/datasette/issues/1858,1292709818,IC_kwDOBm6k_c5NDS-6,9599,simonw,2022-10-26T22:07:04Z,2022-10-26T22:07:04Z,OWNER,"New token design: ```json { ""a"": ""actor-id"", ""t"": ""creation timestamp as integer"", ""d"": ""intended duration in seconds, or blank if no duration set"" } ``` This is in place of the `""e"": ""expiry timestamp""` design I've built so far.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1858#issuecomment-1292708227,https://api.github.com/repos/simonw/datasette/issues/1858,1292708227,IC_kwDOBm6k_c5NDSmD,9599,simonw,2022-10-26T22:05:34Z,2022-10-26T22:05:34Z,OWNER,"I just realized this can't easily affect the `datasette create-token` command because it doesn't currently accept the `--setting` option, so it wouldn't know what `max_signed_tokens_ttl` was. More to the point: even if it did, someone could abuse their knowledge of the secret to create a signed non-expiring token even on servers that didn't want to support those. So I actually need to redesign the token format: it needs to store the timestamp when the token was created and the intended duration, NOT the timestamp that the token expires at. Otherwise it's not possible for servers to enforce `max_signed_tokens_ttl` - someone could always create a token with a custom `expires_at` timestamp on it outside of the configured limit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1858#issuecomment-1292687774,https://api.github.com/repos/simonw/datasette/issues/1858,1292687774,IC_kwDOBm6k_c5NDNme,9599,simonw,2022-10-26T21:44:57Z,2022-10-26T21:44:57Z,OWNER,"I'm going for consistency with `max_csv_mb` and `max_returned_rows` and `allow_signed_tokens` and `default_cache_ttl`. So `max_signed_tokens_ttl`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1860#issuecomment-1292685478,https://api.github.com/repos/simonw/datasette/issues/1860,1292685478,IC_kwDOBm6k_c5NDNCm,9599,simonw,2022-10-26T21:42:35Z,2022-10-26T21:42:35Z,OWNER,"That's deployed to https://latest.datasette.io/ now - some examples: - https://latest.datasette.io/fixtures?sql=--+one+kind+of+comment%0D%0Aselect+*+from+searchable - https://latest.datasette.io/fixtures?sql=%2F*+Multi%0D%0A++line+comment+*%2F%0D%0Aselect+*+from+searchable - https://latest.datasette.io/fixtures?sql=%2F*+Both+kinds+*%2F%0D%0A--+of+comment%0D%0A%2F*+and+more+*%2F%0D%0A--+and+more+and+more%0D%0Aselect+*+from+searchable","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292679567,https://api.github.com/repos/simonw/datasette/issues/1860,1292679567,IC_kwDOBm6k_c5NDLmP,9599,simonw,2022-10-26T21:36:25Z,2022-10-26T21:36:25Z,OWNER,I'm never 100% sure how to tell if a regular expression includes a nasty denial of service attack - are there any inputs that could cause this new regex to execute in quadratic time or similar?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292678657,https://api.github.com/repos/simonw/datasette/issues/1860,1292678657,IC_kwDOBm6k_c5NDLYB,9599,simonw,2022-10-26T21:35:23Z,2022-10-26T21:35:37Z,OWNER,Here are the new tests - each of these should now work: https://github.com/simonw/datasette/blob/55a709c480a1e7401b4ff6208f37a2cf7c682183/tests/test_utils.py#L170-L175,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292674919,https://api.github.com/repos/simonw/datasette/issues/1860,1292674919,IC_kwDOBm6k_c5NDKdn,9599,simonw,2022-10-26T21:31:22Z,2022-10-26T21:31:22Z,OWNER,"I'm experimenting with this: ```python # Allow SQL to start with a /* */ or -- comment comment_re = ( # Start of string, then any amount of whitespace r'^(\s*' + # Comment that starts with -- and ends at a newline r'(?:\-\-.*?\n\s*)' + # Comment that starts with /* and ends with */ r'|(?:/\*[\s\S]*?\*/)' + # Whitespace r')*\s*' ) allowed_sql_res = [ re.compile(comment_re + r""select\b""), re.compile(comment_re + r""explain\s+select\b""), re.compile(comment_re + r""explain\s+query\s+plan\s+select\b""), re.compile(comment_re + r""with\b""), re.compile(comment_re + r""explain\s+with\b""), re.compile(comment_re + r""explain\s+query\s+plan\s+with\b""), ] ``` This should allow any number of comments of either type as a suffix to the allowed SQL patterns. Needs extensive unit tests! I'm not massively worried if it has a flaw in it though, since this is part of Datasette's defense in depth: if a non-SELECT query sneaks through it still shouldn't be able to cause any damage as the database connection is read-only or immutable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1860#issuecomment-1292659986,https://api.github.com/repos/simonw/datasette/issues/1860,1292659986,IC_kwDOBm6k_c5NDG0S,9599,simonw,2022-10-26T21:14:26Z,2022-10-26T21:15:22Z,OWNER,"Yeah we should fix this. https://www.sqlite.org/lang_comment.html - SQLite also supports `-- style` comments. I like how explicit the documentation is here: > SQL comments begin with two consecutive ""-"" characters (ASCII 0x2d) and extend up to and including the next newline character (ASCII 0x0a) or until the end of input, whichever comes first. > > C-style comments begin with ""/*"" and extend up to and including the next ""*/"" character pair or until the end of input, whichever comes first. C-style comments can span multiple lines. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1849#issuecomment-1292654852,https://api.github.com/repos/simonw/datasette/issues/1849,1292654852,IC_kwDOBm6k_c5NDFkE,9599,simonw,2022-10-26T21:08:44Z,2022-10-26T21:08:44Z,OWNER,"Generally though we should expect that people might try to use `render_template(...)` without passing a `request`, so Datasette core should be able to handle this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1292654522,https://api.github.com/repos/simonw/datasette/issues/1849,1292654522,IC_kwDOBm6k_c5NDFe6,9599,simonw,2022-10-26T21:08:20Z,2022-10-26T21:08:20Z,OWNER,"From the stack trace in Sentry: So this happened because a custom plugin tried to render `forbidden.html` without passing in the `request`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1292653219,https://api.github.com/repos/simonw/datasette/issues/1849,1292653219,IC_kwDOBm6k_c5NDFKj,9599,simonw,2022-10-26T21:06:56Z,2022-10-26T21:06:56Z,OWNER,"This was a hit to an authenticated page where the incoming user WAS logged in but did not have permission to view that specific page. Code in question: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/app.py#L634-L640","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1851#issuecomment-1292592210,https://api.github.com/repos/simonw/datasette/issues/1851,1292592210,IC_kwDOBm6k_c5NC2RS,25778,eyeseast,2022-10-26T20:03:46Z,2022-10-26T20:03:46Z,CONTRIBUTOR,"Yeah, every time I see something cool done with triggers, I remember that I need to start using triggers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292544296,https://api.github.com/repos/simonw/datasette/issues/1851,1292544296,IC_kwDOBm6k_c5NCqko,9599,simonw,2022-10-26T19:33:34Z,2022-10-26T19:33:34Z,OWNER,That trigger solution is pretty neat!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1292519956,https://api.github.com/repos/simonw/datasette/issues/1851,1292519956,IC_kwDOBm6k_c5NCkoU,15178711,asg017,2022-10-26T19:20:33Z,2022-10-26T19:20:33Z,CONTRIBUTOR,"> This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can. @eyeseast Maybe you could do this with triggers? Like you can insert JSON-friendly data into a ""raw"" table, and create a trigger that transforms that inserted data into the proper table Here's an example: ```sql -- meant to be updated from a Datasette insert create table points_raw(longitude int, latitude int); -- the target table with proper spatliate geometries create table points(point geometry); CREATE TRIGGER insert_points_raw INSERT ON points_raw BEGIN insert into points(point) values (makepoint(new.longitude, new.latitude)) END; ``` You could then POST a new row to `points_raw` like this: ``` POST /db/points_raw Authorization: Bearer xxx Content-Type: application/json { ""row"": { ""longitude"": 27.64356, ""latitude"": -47.29384 } } ``` Then SQLite with run the trigger and insert a new row in `points` with the correct geometry point. Downside is you'd have duplicated data with `points_raw`, but maybe it could be a `TEMP` table (or have a cron that deletes all rows from that table every so often?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/sqlite-utils/pull/499#issuecomment-1292401308,https://api.github.com/repos/simonw/sqlite-utils/issues/499,1292401308,IC_kwDOCGYnMM5NCHqc,7908073,chapmanjacobd,2022-10-26T17:54:26Z,2022-10-26T17:54:51Z,CONTRIBUTOR,"The problem with how it is currently is that the transformed fts table _will_ return incorrect results (unless the table was only 1 row or something), even if create_triggers was enabled previously. Maybe the simplest solution is to disable fts on a transformed table rather than try to recreate it? Thoughts?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1405196044,feat: recreate fts triggers after table transform, https://github.com/simonw/datasette/issues/1860#issuecomment-1292390996,https://api.github.com/repos/simonw/datasette/issues/1860,1292390996,IC_kwDOBm6k_c5NCFJU,562352,CharlesNepote,2022-10-26T17:43:41Z,2022-10-26T17:43:41Z,NONE,"I guess the issue is here: https://github.com/simonw/datasette/blob/9676b2deb07cff20247ba91dad3e84a4ab0b00d1/datasette/utils/__init__.py#L209 Here is a working regexp allowing it: ```diff - re.compile(r""^select\b""), + re.compile(r""^\s*(/\*.+?(?=\*/)\*/\s*)*select""), ``` `^\s*`: beginning by 0 or an infinite number of \s (spaces, tabs, newlines...) `(/\*.+?(?=\*/)\*/\s*)*`: 0 or an infinite number of chars beginning by `/*` and ending to the next occurrence of `*/` followed by 0 or an infinite number of \s You can play with the regexp here: https://regex101.com/r/aESXDL/3 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1424378012,SQL query field can't begin by a comment, https://github.com/simonw/datasette/issues/1855#issuecomment-1291485444,https://api.github.com/repos/simonw/datasette/issues/1855,1291485444,IC_kwDOBm6k_c5M-oEE,9599,simonw,2022-10-26T04:30:34Z,2022-10-26T04:30:34Z,OWNER,"I'm going to delay working on this until after I have some of the write APIs built to try it against: - #1851","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089,`datasette create-token` ability to create tokens with a reduced set of permissions, https://github.com/simonw/datasette/issues/1859#issuecomment-1291484749,https://api.github.com/repos/simonw/datasette/issues/1859,1291484749,IC_kwDOBm6k_c5M-n5N,9599,simonw,2022-10-26T04:29:43Z,2022-10-26T04:29:43Z,OWNER,"Documentation: - https://docs.datasette.io/en/1.0-dev/authentication.html#datasette-create-token - https://docs.datasette.io/en/1.0-dev/cli-reference.html#datasette-create-token","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1843#issuecomment-1291467084,https://api.github.com/repos/simonw/datasette/issues/1843,1291467084,IC_kwDOBm6k_c5M-jlM,9599,simonw,2022-10-26T04:03:49Z,2022-10-26T04:03:49Z,OWNER,"This time I'm suspicious that there are open SQLite files tucked away in thread locals hidden inside my thread pool executor: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L24 https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/database.py#L204-L214","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1291466613,https://api.github.com/repos/simonw/datasette/issues/1843,1291466613,IC_kwDOBm6k_c5M-jd1,9599,simonw,2022-10-26T04:02:56Z,2022-10-26T04:02:56Z,OWNER,Just saw this error again!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1859#issuecomment-1291439998,https://api.github.com/repos/simonw/datasette/issues/1859,1291439998,IC_kwDOBm6k_c5M-c9-,9599,simonw,2022-10-26T03:15:13Z,2022-10-26T03:15:13Z,OWNER,Reads from `DATASETTE_SECRET` or accepts `--secret` for the signing secret.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1859#issuecomment-1291439875,https://api.github.com/repos/simonw/datasette/issues/1859,1291439875,IC_kwDOBm6k_c5M-c8D,9599,simonw,2022-10-26T03:14:58Z,2022-10-26T03:14:58Z,OWNER,"Initial design: datasette create-token Or: datasette create-token --expire-after 10m","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423369494,datasette create-token CLI command, https://github.com/simonw/datasette/issues/1858#issuecomment-1291435464,https://api.github.com/repos/simonw/datasette/issues/1858,1291435464,IC_kwDOBm6k_c5M-b3I,9599,simonw,2022-10-26T03:07:16Z,2022-10-26T03:07:16Z,OWNER,"This setting will disable the ""Token never expires"" option: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423364990,`max_signed_tokens_ttl` setting for a maximum duration on API tokens, https://github.com/simonw/datasette/issues/1852#issuecomment-1291406219,https://api.github.com/repos/simonw/datasette/issues/1852,1291406219,IC_kwDOBm6k_c5M-UuL,9599,simonw,2022-10-26T02:19:54Z,2022-10-26T02:59:52Z,OWNER,"I'm going to split the remaining work into separate issues: - [x] #1856 - [ ] #1855 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291431132,https://api.github.com/repos/simonw/datasette/issues/1852,1291431132,IC_kwDOBm6k_c5M-azc,9599,simonw,2022-10-26T02:59:50Z,2022-10-26T02:59:50Z,OWNER,Documentation: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1850#issuecomment-1291430992,https://api.github.com/repos/simonw/datasette/issues/1850,1291430992,IC_kwDOBm6k_c5M-axQ,9599,simonw,2022-10-26T02:59:33Z,2022-10-26T02:59:33Z,OWNER,I started the documentation for the API tokens mechanism here: https://docs.datasette.io/en/1.0-dev/authentication.html#api-tokens,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1857#issuecomment-1291418546,https://api.github.com/repos/simonw/datasette/issues/1857,1291418546,IC_kwDOBm6k_c5M-Xuy,9599,simonw,2022-10-26T02:38:35Z,2022-10-26T02:38:35Z,OWNER,"I'm going to set a convention that an actor signed in via a token should set `""token"": ""something""` as a key. Then the `/-/create-token` view can reject those actors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423347412,Prevent API tokens from using /-/create-token to create more tokens, https://github.com/simonw/datasette/issues/1850#issuecomment-1291417755,https://api.github.com/repos/simonw/datasette/issues/1850,1291417755,IC_kwDOBm6k_c5M-Xib,9599,simonw,2022-10-26T02:36:52Z,2022-10-26T02:36:58Z,OWNER,"I'm going to set a convention that `""token"": ""something""` in an actor means that they were authenticated by a token. `""token"": ""dstok""` for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1291417100,https://api.github.com/repos/simonw/datasette/issues/1850,1291417100,IC_kwDOBm6k_c5M-XYM,9599,simonw,2022-10-26T02:35:32Z,2022-10-26T02:35:32Z,OWNER,"It strikes me that users should NOT be able to use a token to create additional tokens. The current design actually does allow that, since the `dstok_` Bearer token can be used to authenticate calls to the `/-/create-token` page. So I think I need a mechanism whereby that page can only allow access to users authenticated by cookie. Not obvious how to do that though, since Datasette's authentication actor system is designed to abstract that detail away!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1856#issuecomment-1291410747,https://api.github.com/repos/simonw/datasette/issues/1856,1291410747,IC_kwDOBm6k_c5M-V07,9599,simonw,2022-10-26T02:27:05Z,2022-10-26T02:27:05Z,OWNER,"Because of that I think this is a better name: --setting allow_signed_tokens off","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1856#issuecomment-1291410331,https://api.github.com/repos/simonw/datasette/issues/1856,1291410331,IC_kwDOBm6k_c5M-Vub,9599,simonw,2022-10-26T02:26:19Z,2022-10-26T02:26:19Z,OWNER,"It's a bit confusing that a setting called `allow_create_tokens` also causes incoming `dstok_` tokens to be ignored. Is it confusing enough that I should pick a different name for the setting though?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1856#issuecomment-1291409312,https://api.github.com/repos/simonw/datasette/issues/1856,1291409312,IC_kwDOBm6k_c5M-Veg,9599,simonw,2022-10-26T02:24:49Z,2022-10-26T02:24:49Z,OWNER,"The effect of this setting will be: - `/-/create-tokens` interface is no longer available - Incoming `dstok_` tokens are no longer respected by the following code: https://github.com/simonw/datasette/blob/b29e487bc3fde6418bf45bda7cfed2e081ff03fb/datasette/default_permissions.py#L52-L72","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336122,allow_signed_tokens setting for disabling API signed token mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291397623,https://api.github.com/repos/simonw/datasette/issues/1852,1291397623,IC_kwDOBm6k_c5M-Sn3,9599,simonw,2022-10-26T02:11:40Z,2022-10-26T02:11:40Z,OWNER,"Built a prototype of the `actor_from_request()` hook for this and now: ``` % curl http://127.0.0.1:8001/-/actor.json -H 'Authorization: Bearer dstok_eyJhIjoicm9vdCIsImUiOm51bGx9.6O1OxgNTFkAU6uw7xNcmXYX949A' {""actor"": {""id"": ""root"", ""dstok"": true}} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291392887,https://api.github.com/repos/simonw/datasette/issues/1852,1291392887,IC_kwDOBm6k_c5M-Rd3,9599,simonw,2022-10-26T02:04:48Z,2022-10-26T02:04:48Z,OWNER,"Implemented that `dstok_` prefix and the thing where only the `actor[""id""]` is copied to the `""a""` field.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291290451,https://api.github.com/repos/simonw/datasette/issues/1852,1291290451,IC_kwDOBm6k_c5M94dT,9599,simonw,2022-10-26T00:49:56Z,2022-10-26T00:49:56Z,OWNER,Prefix: `dstok_` - for Datasette signed token.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291289369,https://api.github.com/repos/simonw/datasette/issues/1852,1291289369,IC_kwDOBm6k_c5M94MZ,9599,simonw,2022-10-26T00:47:46Z,2022-10-26T00:47:46Z,OWNER,"The tokens also need something that can be used to differentiate them from alternative token mechanisms that other plugins might provide. Maybe a prefix before the signed value. Prefixes are also useful for scanning to check they were not accidentally committed to source control.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272280,https://api.github.com/repos/simonw/datasette/issues/1852,1291272280,IC_kwDOBm6k_c5M90BY,9599,simonw,2022-10-26T00:16:09Z,2022-10-26T00:46:21Z,OWNER,"Other options: - `--setting default_api_tokens off` - `--setting signed_api_tokens off` - `--setting allow_create_token off` These feel inconsistent because they don't use the `allow_` prefix - but they're also a bit less ugly to look at. I like that last one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291281243,https://api.github.com/repos/simonw/datasette/issues/1852,1291281243,IC_kwDOBm6k_c5M92Nb,9599,simonw,2022-10-26T00:32:21Z,2022-10-26T00:32:21Z,OWNER,"Rather than duplicating the entire actor into the ""a"" field, maybe just copy the actor ID? Would need to restrict token creation to just actors with an ID set. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291277913,https://api.github.com/repos/simonw/datasette/issues/1852,1291277913,IC_kwDOBm6k_c5M91ZZ,4399499,ocdtrekkie,2022-10-26T00:26:11Z,2022-10-26T00:26:11Z,NONE,"> On that basis, I think the model described above where tokens mainly work to provide an ""act on behalf of this actor"" - but with optional additional constraints - is a good one. This is what we do for Sandstorm essentially and I fully agree it's the right way to do API tokens in multiuser systems. Constraints will definitely be important though. I know I want a token to submit error reports programmatically, but I wouldn't want that token to convey my right to delete tables and records, Little Bobby Tables is out there somewhere, and he's all grown up now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291274835,https://api.github.com/repos/simonw/datasette/issues/1852,1291274835,IC_kwDOBm6k_c5M90pT,9599,simonw,2022-10-26T00:20:48Z,2022-10-26T00:22:26Z,OWNER,"Tests failed because I added a view without also adding documentation! I forgot that the deploy still goes out for branches other than `main` even if the tests aren't passing: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/.github/workflows/deploy-latest.yml#L34-L38","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291273609,https://api.github.com/repos/simonw/datasette/issues/1852,1291273609,IC_kwDOBm6k_c5M90WJ,9599,simonw,2022-10-26T00:18:40Z,2022-10-26T00:18:40Z,OWNER,"Another thought about tokens that can act on behalf of the user. Imagine a user has permission to access a table. They create a token that can create that table... but then their permission is revoked. It would be bad if they could still use that token they created earlier to access that table! On that basis, I think the model described above where tokens mainly work to provide an ""act on behalf of this actor"" - but with optional additional constraints - is a good one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272612,https://api.github.com/repos/simonw/datasette/issues/1852,1291272612,IC_kwDOBm6k_c5M90Gk,9599,simonw,2022-10-26T00:16:53Z,2022-10-26T00:16:53Z,OWNER,Next step: make these tokens actually do something.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291272414,https://api.github.com/repos/simonw/datasette/issues/1852,1291272414,IC_kwDOBm6k_c5M90De,9599,simonw,2022-10-26T00:16:28Z,2022-10-26T00:16:28Z,OWNER,If I'm going to change the naming conventions for settings I should do it before Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291271580,https://api.github.com/repos/simonw/datasette/issues/1852,1291271580,IC_kwDOBm6k_c5M9z2c,9599,simonw,2022-10-26T00:14:49Z,2022-10-26T00:15:06Z,OWNER,"If I'm going to have a setting to disable this feature I need to decide what it will be called. Closest existing setting is this one, since it's for a feature that is turned on by default: datasette mydatabase.db --setting allow_download off So maybe this? datasette mydatabase.db --setting allow_signed_api_tokens off I like `allow_signed_api_tokens` more than `allow_api_tokens` because if you install a plugin such as https://datasette.io/plugins/datasette-auth-tokens then API tokens will work even though you disabled this default signed token feature. `allow_signed_api_tokens` does feel a bit clumsy/verbose though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291270227,https://api.github.com/repos/simonw/datasette/issues/1852,1291270227,IC_kwDOBm6k_c5M9zhT,9599,simonw,2022-10-26T00:12:18Z,2022-10-26T00:12:18Z,OWNER,Demo is now live at https://latest-1-0-dev.datasette.io/-/create-token - visit https://latest-1-0-dev.datasette.io/login-as-root first to sign in.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291269607,https://api.github.com/repos/simonw/datasette/issues/1852,1291269607,IC_kwDOBm6k_c5M9zXn,9599,simonw,2022-10-26T00:11:15Z,2022-10-26T00:11:15Z,OWNER,"If you click ""Create token"" for ""Token never expires"" multiple times you currently get exactly the same token each time, since it's just a signed token containing a copy of your actor dictionary. I'm not sure if I like that. I could give each token a random ID (maybe using `secrets.token_hex()`) such that different tokens have different identities, which would be useful for logging and auditing and maybe even revocation at some point in the future.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291268380,https://api.github.com/repos/simonw/datasette/issues/1852,1291268380,IC_kwDOBm6k_c5M9zEc,9599,simonw,2022-10-26T00:09:06Z,2022-10-26T00:09:06Z,OWNER,"Demo: ![token-demo](https://user-images.githubusercontent.com/9599/197904595-e5651d6c-bafc-4124-b762-71ad94c06ced.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291243333,https://api.github.com/repos/simonw/datasette/issues/1852,1291243333,IC_kwDOBm6k_c5M9s9F,9599,simonw,2022-10-25T23:25:13Z,2022-10-25T23:25:13Z,OWNER,"A `/-/debug-token` page that can take a token and decode it to show you how long until it expires, what actor it represents and the permissions it has will be useful as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291234262,https://api.github.com/repos/simonw/datasette/issues/1852,1291234262,IC_kwDOBm6k_c5M9qvW,9599,simonw,2022-10-25T23:11:23Z,2022-10-25T23:11:23Z,OWNER,I'm going to build an initial `/-/create-token` interface which just bakes a token with the current actor in it and an optional expiry timestamp. I'll try the limited permissions thing later.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291233652,https://api.github.com/repos/simonw/datasette/issues/1852,1291233652,IC_kwDOBm6k_c5M9ql0,9599,simonw,2022-10-25T23:10:20Z,2022-10-25T23:10:44Z,OWNER,"In which case the token would need to duplicate the current `actor` and then add extra constraints. So maybe the token design looks like this: ```json { ""a"": { ""copy_of"": ""actor_creating_token""}, ""p"": { ""t"": ""... the thing designed earlier, with those permissions in it"" }, ""e"": ""integer timestamp when token expires"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291232589,https://api.github.com/repos/simonw/datasette/issues/1852,1291232589,IC_kwDOBm6k_c5M9qVN,9599,simonw,2022-10-25T23:08:37Z,2022-10-25T23:08:37Z,OWNER,"... so maybe there's a way to create a token that inherits the exact permissions of the actor that created the token? That could even be a default mode for tokens, with an option to then further restrict permissions if desired.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1291231651,https://api.github.com/repos/simonw/datasette/issues/1852,1291231651,IC_kwDOBm6k_c5M9qGj,9599,simonw,2022-10-25T23:07:17Z,2022-10-25T23:07:17Z,OWNER,"Interesting challenge: what permissions should users be allowed to grant to tokens? Clearly a user should not be able to create a token with a permission that the user themselves does not have. And should there be a permission that allows people to create tokens? I think so.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1851#issuecomment-1291228502,https://api.github.com/repos/simonw/datasette/issues/1851,1291228502,IC_kwDOBm6k_c5M9pVW,25778,eyeseast,2022-10-25T23:02:10Z,2022-10-25T23:02:10Z,CONTRIBUTOR,That's reasonable. Canned queries and custom endpoints are certainly going to give more room for specific needs. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1852#issuecomment-1291227942,https://api.github.com/repos/simonw/datasette/issues/1852,1291227942,IC_kwDOBm6k_c5M9pMm,9599,simonw,2022-10-25T23:01:18Z,2022-10-25T23:01:18Z,OWNER,"Datasette currently defaults to having everything public-readable by default, unless a permission plugin changes that default. In thinking more about this API mechanism, I realized that it might be good to have a mode where Datasette _doesn't_ default to public everything. Maybe `datasette --private` to start it like that? Might even be an opportunity to get rid of the current slightly confusing mechanism where permission checks can announce that they should default to true: https://github.com/simonw/datasette/blob/c7dd76c26257ded5bcdfd0570e12412531b8b88f/datasette/views/database.py#L152-L154","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1851#issuecomment-1291226367,https://api.github.com/repos/simonw/datasette/issues/1851,1291226367,IC_kwDOBm6k_c5M9oz_,9599,simonw,2022-10-25T22:58:30Z,2022-10-25T22:58:30Z,OWNER,"The `datasette insert` concept included plugin support, with the idea of being able to support things like SpatiaLite files: - #1160 I think this API mechanism is going to be a bit less exciting than that - it will be low-level for inserting rows, and if you want to do something fancier you can use a canned query that feeds incoming GeoJSON to a SpatiaLite function instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291216193,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291216193,IC_kwDOCGYnMM5M9mVB,9599,simonw,2022-10-25T22:41:16Z,2022-10-25T22:41:16Z,OWNER,Tweeted about it here: https://twitter.com/simonw/status/1585038766678609921,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/505#issuecomment-1291203911,https://api.github.com/repos/simonw/sqlite-utils/issues/505,1291203911,IC_kwDOCGYnMM5M9jVH,9599,simonw,2022-10-25T22:21:02Z,2022-10-25T22:21:02Z,OWNER,"- Now tested against Python 3.11. ([#502](https://github.com/simonw/sqlite-utils/issues/502)) - New `table.search_sql(include_rank=True)` option, which adds a `rank` column to the generated SQL. Thanks, Jacob Chapman. ([#480](https://github.com/simonw/sqlite-utils/pull/480)) - Progress bars now display for newline-delimited JSON files using the `--nl` option. Thanks, Mischa Untaga. ([#485](https://github.com/simonw/sqlite-utils/issues/485)) - New `db.close()` method. ([#504](https://github.com/simonw/sqlite-utils/issues/504)) - Conversion functions passed to [table.convert(...)](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-convert) can now return lists or dictionaries, which will be inserted into the database as JSON strings. ([#495](https://github.com/simonw/sqlite-utils/issues/495)) - `sqlite-utils install` and `sqlite-utils uninstall` commands for installing packages into the same virtual environment as `sqlite-utils`, [described here](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-install). ([#483](https://github.com/simonw/sqlite-utils/issues/483)) - New [sqlite_utils.utils.flatten()](https://sqlite-utils.datasette.io/en/stable/reference.html#reference-utils-flatten) utility function. ([#500](https://github.com/simonw/sqlite-utils/issues/500)) - Documentation on [using Just](https://sqlite-utils.datasette.io/en/stable/contributing.html#contributing-just) to run tests, linters and build documentation. - Documentation now covers the [Release process](https://sqlite-utils.datasette.io/en/stable/contributing.html#release-process) for this package.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423182778,Release sqlite-utils 3.30, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291170072,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291170072,IC_kwDOCGYnMM5M9bEY,9599,simonw,2022-10-25T21:36:12Z,2022-10-25T21:36:12Z,OWNER,"I was going to suggest using `db.table(name)` instead of `db[name]` - but it looks like that method will have the same problem: https://github.com/simonw/sqlite-utils/blob/defa2974c6d3abc19be28d6b319649b8028dc966/sqlite_utils/db.py#L497-L506 I could change `sqlite-utils` so `db.table(name)` always returns a table and you need to call `db.view(name)` if you want to access a view - that would require bumping to 4.0 though. I'm not convinced that's the best approach here either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/496#issuecomment-1291167887,https://api.github.com/repos/simonw/sqlite-utils/issues/496,1291167887,IC_kwDOCGYnMM5M9aiP,9599,simonw,2022-10-25T21:33:25Z,2022-10-25T21:33:25Z,OWNER,"I do care about this, but I'm not hugely experienced with types yet so I'm open to suggestions about how to do it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393202060,devrel/python api: Pylance type hinting, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1291166273,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1291166273,IC_kwDOCGYnMM5M9aJB,9599,simonw,2022-10-25T21:31:15Z,2022-10-25T21:31:15Z,OWNER,"Based on the docs here I tried the following too: https://docutils.sourceforge.io/docs/user/smartquotes.html#description - `\--` - `\\--` - `\\-\\-` - `\-\-` But none of them had the desired effect in this particular piece of markup: the :ref:`insert \--convert ` I think because this is text inside a `:ref:` block, not regular text. Consider the following: The \--convert and the :ref:`insert \--convert ` and It's rendered like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291159549,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291159549,IC_kwDOCGYnMM5M9Yf9,9599,simonw,2022-10-25T21:23:01Z,2022-10-25T21:23:01Z,OWNER,"I've decided not to explicitly document this, since it's consistent with how other parts of the library work already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291152433,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291152433,IC_kwDOCGYnMM5M9Wwx,9599,simonw,2022-10-25T21:14:54Z,2022-10-25T21:14:54Z,OWNER,"There is a case where the function can return a dictionary at the moment: `multi=True` ```python table.convert( ""title"", lambda v: {""upper"": v.upper(), ""lower"": v.lower()}, multi=True ) ``` But I think this change is still compatible with that. if you don't use `multi=True` then the return value will be stringified. If you DO use `multi=True` then something like this could work: ```python table.convert( ""title"", lambda v: {""upper"": {""str"": v.upper()}, ""lower"": {""str"": v.lower()}}, multi=True ) ``` This would result in a `upper` and `lower` column, each containing the JSON string `{""str"": ""UPPERCASE""}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/495#issuecomment-1291149509,https://api.github.com/repos/simonw/sqlite-utils/issues/495,1291149509,IC_kwDOCGYnMM5M9WDF,9599,simonw,2022-10-25T21:12:11Z,2022-10-25T21:12:11Z,OWNER,"This makes sense to me. There are other places in the codebase where JSON is automatically stringified: https://github.com/simonw/sqlite-utils/blob/c7e4308e6f49d929704163531632e558f9646e4a/sqlite_utils/db.py#L2759-L2766 I don't see why the return value from a convert function shouldn't do the same thing. Since this will result in previous errors working, I don't think it warrants a major version bump either.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392690202,Support JSON values returned from .convert() functions, https://github.com/simonw/sqlite-utils/issues/497#issuecomment-1291146850,https://api.github.com/repos/simonw/sqlite-utils/issues/497,1291146850,IC_kwDOCGYnMM5M9VZi,9599,simonw,2022-10-25T21:09:28Z,2022-10-25T21:09:28Z,OWNER,"Yeah, `table.columns` and `table.columns_dict` are meant to handle this: https://sqlite-utils.datasette.io/en/stable/python-api.html#columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1393212964,column_names, https://github.com/simonw/sqlite-utils/issues/504#issuecomment-1291136971,https://api.github.com/repos/simonw/sqlite-utils/issues/504,1291136971,IC_kwDOCGYnMM5M9S_L,9599,simonw,2022-10-25T21:00:29Z,2022-10-25T21:00:29Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite_utils.db.Database.close,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423069384,"db.close() method, calling db.conn.close()", https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291124413,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291124413,IC_kwDOCGYnMM5M9P69,9599,simonw,2022-10-25T20:47:34Z,2022-10-25T20:47:34Z,OWNER,TIL about this: https://til.simonwillison.net/python/os-remove-windows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291122389,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291122389,IC_kwDOCGYnMM5M9PbV,9599,simonw,2022-10-25T20:45:43Z,2022-10-25T20:45:43Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291115986,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291115986,IC_kwDOCGYnMM5M9N3S,9599,simonw,2022-10-25T20:39:24Z,2022-10-25T20:39:24Z,OWNER,"Used `psutil` to confirm that closing a SQLite connection closes the underlying file: https://til.simonwillison.net/python/too-many-open-files-psutil ```pycon >>> import psutil >>> import sqlite3 >>> for f in psutil.Process().open_files(): print(f) ... >>> sqlite3.connect(""/tmp/blah.db"") >>> conn = _ >>> for f in psutil.Process().open_files(): print(f) ... popenfile(path='/private/tmp/blah.db', fd=3) >>> conn.close() >>> for f in psutil.Process().open_files(): print(f) ... >>> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291111357,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291111357,IC_kwDOCGYnMM5M9Mu9,9599,simonw,2022-10-25T20:36:06Z,2022-10-25T20:36:06Z,OWNER,... or maybe Windows doesn't like attempts to remove a file that the process has opened?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291103021,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291103021,IC_kwDOCGYnMM5M9Kst,9599,simonw,2022-10-25T20:32:01Z,2022-10-25T20:32:01Z,OWNER,"This test reliably fails on Windows with Python 3.11. I'm going to skip the test for the moment to get back to green CI... but I'll leave this issue open. This is definitely concerning, I just don't have the right local environment to solve this at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291093581,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291093581,IC_kwDOCGYnMM5M9IZN,9599,simonw,2022-10-25T20:23:00Z,2022-10-25T20:23:00Z,OWNER,"I'm not hugely happy with my fix there: https://github.com/simonw/sqlite-utils/blob/c5d7ec1dd71fa1dce829bc8bb82b639018befd63/sqlite_utils/db.py#L321-L328 The problem here was that in the case where the `os.remove()` failed the `self.conn` property was NOT being set to a valid connection - which caused `__repr__` to fail later on. So now I catch the `os.remove()` error, set `self.conn` to a memory connection, then raise the error again.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291088108,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291088108,IC_kwDOCGYnMM5M9HDs,9599,simonw,2022-10-25T20:17:36Z,2022-10-25T20:17:36Z,OWNER,"Now `mypy` is failing: ``` sqlite_utils/db.py:474: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:476: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:486: error: Item ""None"" of ""Optional[Any]"" has no attribute ""executescript"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:603: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:604: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:607: error: Item ""None"" of ""Optional[Any]"" has no attribute ""execute"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__enter__"" sqlite_utils/db.py:1082: error: Item ""None"" of ""Optional[Any]"" has no attribute ""__exit__"" sqlite_utils/db.py:1083: error: Item ""None"" of ""Optional[Any]"" has no attribute ""cursor"" sqlite_utils/db.py:1155: error: Item ""None"" of ""Optional[Any]"" has no attribute ""enable_load_extension"" sqlite_utils/db.py:1156: error: Item ""None"" of ""Optional[Any]"" has no attribute ""load_extension"" Found 12 errors in 1 file (checked 51 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291083188,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291083188,IC_kwDOCGYnMM5M9F20,9599,simonw,2022-10-25T20:12:52Z,2022-10-25T20:12:52Z,OWNER,"Failed again, but just noticed this: https://github.com/simonw/sqlite-utils/actions/runs/3323932266/jobs/5494890223 ``` > Database(filepath, recreate=True)[""t2""].insert({""foo"": ""bar""}) tests\test_recreate.py:31: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError(""'Database' object has no attribute 'conn'"") raised in repr()] Database object at 0x29fc125aa90> ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291076031,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291076031,IC_kwDOCGYnMM5M9EG_,9599,simonw,2022-10-25T20:06:28Z,2022-10-25T20:06:28Z,OWNER,"This is the failing test: https://github.com/simonw/sqlite-utils/blob/7b2d1c0ffd0b874e280292b926f328a61cb31e2c/tests/test_recreate.py#L21-L32 I'm going to try a different way of creating the temporary file: https://docs.pytest.org/en/7.1.x/how-to/tmp_path.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/sqlite-utils/issues/503#issuecomment-1291071627,https://api.github.com/repos/simonw/sqlite-utils/issues/503,1291071627,IC_kwDOCGYnMM5M9DCL,9599,simonw,2022-10-25T20:02:18Z,2022-10-25T20:02:18Z,OWNER,Passes on Windows with other Python versions for some reason.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423000702,test_recreate failing on Windows Python 3.11, https://github.com/simonw/datasette/issues/1854#issuecomment-1291047214,https://api.github.com/repos/simonw/datasette/issues/1854,1291047214,IC_kwDOBm6k_c5M89Eu,9599,simonw,2022-10-25T19:39:36Z,2022-10-25T19:39:48Z,OWNER,"This pattern should work (for the http server at least): ```python # Loop until port 8041 serves traffic while True: try: httpx.get(""http://localhost:8041/"") break except httpx.ConnectError: time.sleep(0.1) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http, https://github.com/simonw/datasette/issues/1854#issuecomment-1291046958,https://api.github.com/repos/simonw/datasette/issues/1854,1291046958,IC_kwDOBm6k_c5M89Au,9599,simonw,2022-10-25T19:39:22Z,2022-10-25T19:39:22Z,OWNER,"Here's the code that starts those various servers: https://github.com/simonw/datasette/blob/613ad05c095f92653221db267ef53d54d00cdfbb/tests/conftest.py#L104-L177 I don't like those `time.sleep(1.5)` lines much - I'm going to try polling for readiness instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http, https://github.com/simonw/datasette/issues/1854#issuecomment-1291045997,https://api.github.com/repos/simonw/datasette/issues/1854,1291045997,IC_kwDOBm6k_c5M88xt,9599,simonw,2022-10-25T19:38:28Z,2022-10-25T19:38:28Z,OWNER,"Also: ``` @pytest.mark.serial @pytest.mark.skipif( not hasattr(socket, ""AF_UNIX""), reason=""Requires socket.AF_UNIX support"" ) def test_serve_unix_domain_socket(ds_unix_domain_socket_server): _, uds = ds_unix_domain_socket_server transport = httpx.HTTPTransport(uds=uds) client = httpx.Client(transport=transport) > response = client.get(""http://localhost/_memory.json"") /home/runner/work/datasette/datasette/tests/test_cli_serve_server.py:35: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1039: in get return self.request( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:815: in request return self.send(request, auth=auth, follow_redirects=follow_redirects) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:902: in send response = self._send_handling_auth( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:930: in _send_handling_auth response = self._send_handling_redirects( /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:967: in _send_handling_redirects response = self._send_single_request(request) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_client.py:1003: in _send_single_request response = transport.handle_request(request) /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/site-packages/httpx/_transports/default.py:217: in handle_request with map_httpcore_exceptions(): /opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/contextlib.py:153: in __exit__ self.gen.throw(typ, value, traceback) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ @contextlib.contextmanager def map_httpcore_exceptions() -> typing.Iterator[None]: try: yield except Exception as exc: # noqa: PIE-786 mapped_exc = None for from_exc, to_exc in HTTPCORE_EXC_MAP.items(): if not isinstance(exc, from_exc): continue # We want to map to the most specific exception we can find. # Eg if `exc` is an `httpcore.ReadTimeout`, we want to map to # `httpx.ReadTimeout`, not just `httpx.TimeoutException`. if mapped_exc is None or issubclass(to_exc, mapped_exc): mapped_exc = to_exc if mapped_exc is None: # pragma: nocover raise message = str(exc) > raise mapped_exc(message) from exc E httpx.ConnectError: [Errno 2] No such file or directory ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422973111,Flaky test: test_serve_localhost_http, https://github.com/simonw/datasette/issues/1853#issuecomment-1291036623,https://api.github.com/repos/simonw/datasette/issues/1853,1291036623,IC_kwDOBm6k_c5M86fP,9599,simonw,2022-10-25T19:28:56Z,2022-10-25T19:28:56Z,OWNER,"Opened an issue here: - https://github.com/coleifer/pysqlite3/issues/43","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291032289,https://api.github.com/repos/simonw/datasette/issues/1853,1291032289,IC_kwDOBm6k_c5M85bh,9599,simonw,2022-10-25T19:24:27Z,2022-10-25T19:24:27Z,OWNER,https://latest.datasette.io/-/versions now shows 3.11.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/sqlite-utils/issues/502#issuecomment-1291029761,https://api.github.com/repos/simonw/sqlite-utils/issues/502,1291029761,IC_kwDOCGYnMM5M840B,9599,simonw,2022-10-25T19:21:44Z,2022-10-25T19:21:44Z,OWNER,"Replicated locally using a fresh virtual environment with Python 3.11 and: pytest -k test_query_invalid_function","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422954582,Fix tests for Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291023926,https://api.github.com/repos/simonw/datasette/issues/1853,1291023926,IC_kwDOBm6k_c5M83Y2,9599,simonw,2022-10-25T19:15:49Z,2022-10-25T19:15:49Z,OWNER,"This broke the deploy of `https://latest.datasette.io/` - because it tries to install `pysqlite3-binary` which doesn't have a 3.11 release yet: https://github.com/simonw/datasette/blob/2e9751672d4fe329b3c359d5b7b1992283185820/.github/workflows/deploy-latest.yml#L77 I started using that for the `latest.datasette.io` demo in https://github.com/simonw/datasette/commit/a970276b9999687b96c5e11ea1c817d814f5d267 because I wanted a version of SQLite that supported generated columns. Those were added in [SQLite 3.31.0](https://www.sqlite.org/changes.html#version_3_31_0) - and the SQLite version in the new base image is 3.34.1 - so I don't actually need `pysqlite3-binary` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291012637,https://api.github.com/repos/simonw/datasette/issues/1853,1291012637,IC_kwDOBm6k_c5M80od,9599,simonw,2022-10-25T19:04:03Z,2022-10-25T19:04:09Z,OWNER,"And tested `datasette package` like this: ``` datasette package fixtures.db -t datasette-package-python-upgrade-3-11 ``` Then: ``` docker run -p 8081:8001 datasette-package-python-upgrade-3-11 ``` And tested it like this: ``` curl http://localhost:8081/-/versions.json | jq ``` Output: ``` { ""python"": { ""version"": ""3.11.0"", ""full"": ""3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"" }, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291009987,https://api.github.com/repos/simonw/datasette/issues/1853,1291009987,IC_kwDOBm6k_c5M8z_D,9599,simonw,2022-10-25T19:01:23Z,2022-10-25T19:01:23Z,OWNER,"Also tested by running this locally: datasette publish cloudrun fixtures.db --service issue-1853 https://issue-1853-j7hipcg4aq-uc.a.run.app/-/versions now shows Python 3.11.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1291006149,https://api.github.com/repos/simonw/datasette/issues/1853,1291006149,IC_kwDOBm6k_c5M8zDF,9599,simonw,2022-10-25T18:57:33Z,2022-10-25T18:57:33Z,OWNER,"Ran the upgrade on the Datasette Cloud image first, works fine there. https://simon.datasette.cloud/-/versions shows me: ``` { ""python"": { ""version"": ""3.11.0"", ""full"": ""3.11.0 (main, Oct 25 2022, 05:00:36) [GCC 10.2.1 20210110]"" }, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1853#issuecomment-1290995178,https://api.github.com/repos/simonw/datasette/issues/1853,1290995178,IC_kwDOBm6k_c5M8wXq,9599,simonw,2022-10-25T18:46:33Z,2022-10-25T18:46:33Z,OWNER,"I ran a very crude benchmark on my laptop using Locust (against the official macOS packages from www.python.org for Python 3.10 and Python 3.11) and saw a substantial speed increase: 533.89 requests/second on 3.11 413.56 requests/second on 3.10 That was from running Locust against this `locustfile.py`: ```python from locust import HttpUser, task class CounterOne(HttpUser): @task def hello(self): self.client.get(""/-/static/app.css"") ``` Using: locust --headless --users 4 --spawn-rate 4 -H http://127.0.0.1:8001","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1422915587,Upgrade Datasette Docker to Python 3.11, https://github.com/simonw/datasette/issues/1851#issuecomment-1290615599,https://api.github.com/repos/simonw/datasette/issues/1851,1290615599,IC_kwDOBm6k_c5M7Tsv,25778,eyeseast,2022-10-25T14:05:12Z,2022-10-25T14:05:12Z,CONTRIBUTOR,"This could use a new plugin hook, too. I don't want to complicate your life too much, but for things like GIS, I'd want a way to turn regular JSON into SpatiaLite geometries or combine X/Y coordinates into point geometries and such. Happy to help however I can.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1851#issuecomment-1289865317,https://api.github.com/repos/simonw/datasette/issues/1851,1289865317,IC_kwDOBm6k_c5M4chl,9599,simonw,2022-10-25T01:42:47Z,2022-10-25T01:42:47Z,OWNER,"This is going to tie into Datasette's existing permissions mechanism, so plugins will be able to define their own custom mechanisms for tokens to be attached to a specific identity: https://docs.datasette.io/en/stable/authentication.html There's only one plugin for API tokens at the moment, which is this one: https://datasette.io/plugins/datasette-auth-tokens I'm actually planning on adding another, default token mechanism to Datasette itself as part of this work: - #1852 It may well be that `datasette-sandstorm-support` needs to add something custom here too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1852#issuecomment-1289776707,https://api.github.com/repos/simonw/datasette/issues/1852,1289776707,IC_kwDOBm6k_c5M4G5D,9599,simonw,2022-10-24T23:29:03Z,2022-10-24T23:29:03Z,OWNER,"I'm going to implement the first version of this token mechanism using permissions that exist already. Right now that's: https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions Here are the shortcuts I'll use for them: - `view-instance` - `vi` - `view-database` - `vd` - `view-database-download` - `vdd` - `view-table` - `vt` - `view-query` - `vq` - `execute-sql` - `es` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289775162,https://api.github.com/repos/simonw/datasette/issues/1852,1289775162,IC_kwDOBm6k_c5M4Gg6,9599,simonw,2022-10-24T23:27:00Z,2022-10-24T23:27:00Z,OWNER,"Might be neat for API tokens to be signed with an additional secret than can be rotated independently of `DATASETTE_SECRET` itself, in order to invalidate all tokens without needing to invalidate logged in users too. But again, I don't want to implement something like that until I see an actual need for it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289774183,https://api.github.com/repos/simonw/datasette/issues/1852,1289774183,IC_kwDOBm6k_c5M4GRn,9599,simonw,2022-10-24T23:25:52Z,2022-10-24T23:25:52Z,OWNER,"... also, maybe there should be a UI (perhaps on that page) for resetting the Datasette secret? Useful for emergency invalidation of all tokens. No, I'm not going to build that unless someone asks for it. Restarting the server with a fresh secret should be easy enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289773634,https://api.github.com/repos/simonw/datasette/issues/1852,1289773634,IC_kwDOBm6k_c5M4GJC,9599,simonw,2022-10-24T23:25:06Z,2022-10-24T23:25:06Z,OWNER,"If you start Datasette without providing a `DATASETTE_SECRET` environment variable of `--secret` option it creates a random signing secret that only lasts for the lifetime of the server. This means any signed API tokens you create will stop working if the server restarts. I think the `/-/create-token` UI should know when this happens and show a warning message about it, to avoid confusion.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289766513,https://api.github.com/repos/simonw/datasette/issues/1852,1289766513,IC_kwDOBm6k_c5M4EZx,9599,simonw,2022-10-24T23:16:00Z,2022-10-24T23:16:00Z,OWNER,"Here's what that example looks like signed: ```python from datasette.app import Datasette ds = Datasette() ds.sign('{""t"":{""a"":[""ir"",""ur"",""dr""],""d"":{""fixtures"":[""ir"",""ur"",""dr""]},""t"":{""fixtures"":{""searchable"":[""ir""]}}}}') ``` ``` .eJxTqo5RKolRsgJSiUAqOkYpsyhGSSdGqRRCpQCpWBANUZOWWVFSWpRajFNprQ7cPCS1QF5xamJRckZiUk4qQm9sLRAoAQCC8yph.O0Gaej6-VOLbbtPq7xU6T77jEO0 ``` That's 129 characters. Note that Datasette doesn't have its own mechanism for signing things for a specific duration yet: https://docs.datasette.io/en/stable/internals.html#sign-value-namespace-default So I'll need to add a `""e"": 1666739744` field with the UTC timestamp at which the token should expire.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1852#issuecomment-1289733483,https://api.github.com/repos/simonw/datasette/issues/1852,1289733483,IC_kwDOBm6k_c5M38Vr,9599,simonw,2022-10-24T22:54:37Z,2022-10-24T23:12:10Z,OWNER,"Token design concept: ```json { ""t"": { ""a"": [""ir"", ""ur"", ""dr""], ""d"": { ""fixtures"": [""ir"", ""ur"", ""dr""] }, ""t"": { ""fixtures"": { ""searchable"": [""ir""] } } } } ``` That JSON would be minified and signed. Minified version of the above looks like this (101 characters): `{""t"":{""a"":[""ir"",""ur"",""dr""],""d"":{""fixtures"":[""ir"",""ur"",""dr""]},""t"":{""fixtures"":{""searchable"":[""ir""]}}}}` The `""t""` key shows this is a token that as a default API key. `""a""` means ""all"" - these are permissions that have been granted on all tables and databases. `""d""` means ""databases"" - this is a way to set permissions for all tables in a specific database. `""t""` means ""tables"" - this lets you set permissions at a finely grained table level. Then the permissions themselves are two character codes which are shortened versions - so: - `ir` = `insert-row` - `ur` = `update-row` - `dr` = `delete-row`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1851#issuecomment-1289752130,https://api.github.com/repos/simonw/datasette/issues/1851,1289752130,IC_kwDOBm6k_c5M4A5C,4399499,ocdtrekkie,2022-10-24T23:07:30Z,2022-10-24T23:07:30Z,NONE,"How are you tying the bearer token to identity? I'm excited to see this feature, and since Sandstorm controls API access using the same header, it also will transparently support the API documentation here, but we strip the bearer before the request reaches the app (replacing it with our existing auth headers, of course).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1852#issuecomment-1289718660,https://api.github.com/repos/simonw/datasette/issues/1852,1289718660,IC_kwDOBm6k_c5M34uE,9599,simonw,2022-10-24T22:35:01Z,2022-10-24T22:35:01Z,OWNER,"Maybe these tokens can be restricted to specific databases and tables when they are first created? Since they're signed tokens, I could bundle a bunch of extra stuff in them - this token is allowed to do these permissions against these tables/rows for example. General wisdom seems to be that 8KB is a sensible maximum length for this kind of token, which is easily long enough to fit in a bunch of database / table / permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421552095,Default API token authentication mechanism, https://github.com/simonw/datasette/issues/1851#issuecomment-1289713513,https://api.github.com/repos/simonw/datasette/issues/1851,1289713513,IC_kwDOBm6k_c5M33dp,9599,simonw,2022-10-24T22:29:58Z,2022-10-24T22:30:15Z,OWNER,"Interesting open question: how should validation errors (if any) be returned? The two forms of validation I can think of at first are: - Missing keys which are marked as `not null` in the schema - Keys that do not match to existing columns (if you didn't pass `""alter"": true`, an option I am considering)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421544654,API to insert a single record into an existing table, https://github.com/simonw/datasette/issues/1850#issuecomment-1289707357,https://api.github.com/repos/simonw/datasette/issues/1850,1289707357,IC_kwDOBm6k_c5M319d,9599,simonw,2022-10-24T22:23:12Z,2022-10-24T22:23:12Z,OWNER,Here's the implementation of `datasette-auth-tokens`: https://github.com/simonw/datasette-auth-tokens/blob/main/datasette_auth_tokens/__init__.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289706439,https://api.github.com/repos/simonw/datasette/issues/1850,1289706439,IC_kwDOBm6k_c5M31vH,9599,simonw,2022-10-24T22:22:17Z,2022-10-24T22:22:17Z,OWNER,"API authentication will be via `Authorization: Bearer XXX` request headers. I'm inclined to add a default token mechanism to Datasette based on tokens that are signed with the `DATASETTE_SECRET`. Maybe the root user can access `/-/create-token` which provides a UI for generating a time-limited signed token? Could also have a `datasette create-token` command for creating such tokens at the command-line. Plugins can then define alternative ways of creating tokens, such as the existing https://datasette.io/plugins/datasette-auth-tokens plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289703432,https://api.github.com/repos/simonw/datasette/issues/1850,1289703432,IC_kwDOBm6k_c5M31AI,9599,simonw,2022-10-24T22:19:48Z,2022-10-24T22:19:48Z,OWNER,It may turn out that it makes sense to also add a UI for these actions as part of this project. That's still to be determined.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289702146,https://api.github.com/repos/simonw/datasette/issues/1850,1289702146,IC_kwDOBm6k_c5M30sC,9599,simonw,2022-10-24T22:19:04Z,2022-10-24T22:19:04Z,OWNER,"This is going to need a whole bunch of new permissions. To review: the existing set of permissions are listed here: https://docs.datasette.io/en/0.62/authentication.html#built-in-permissions - `view-instance` - `view-database` - `view-database-download` - `view-table` - `view-query` - `execute-sql` - `permissions-debug` - `debug-menu` I'm going to reuse database terminology for the new permissions. So first draft of those is: - `insert-row` - `update-row` - `delete-row` - `create-table` - `drop-table` - `alter-table`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1850#issuecomment-1289696171,https://api.github.com/repos/simonw/datasette/issues/1850,1289696171,IC_kwDOBm6k_c5M3zOr,9599,simonw,2022-10-24T22:15:57Z,2022-10-24T22:15:57Z,OWNER,"I'm going to treat this as a bit of a research spike, at least until I like the direction it is going enough to commit to it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723,Write API in Datasette core, https://github.com/simonw/datasette/issues/1849#issuecomment-1288384907,https://api.github.com/repos/simonw/datasette/issues/1849,1288384907,IC_kwDOBm6k_c5MyzGL,9599,simonw,2022-10-24T04:04:02Z,2022-10-24T04:04:02Z,OWNER,"Refs: - #1831 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1849#issuecomment-1288384098,https://api.github.com/repos/simonw/datasette/issues/1849,1288384098,IC_kwDOBm6k_c5Myy5i,9599,simonw,2022-10-24T04:03:09Z,2022-10-24T04:03:09Z,OWNER,"Looks like the new breadcrumbs code can't handle the case where `request` is `None`. Need a test that demonstrates this too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420174670,NoneType' object has no attribute 'actor', https://github.com/simonw/datasette/issues/1848#issuecomment-1288340476,https://api.github.com/repos/simonw/datasette/issues/1848,1288340476,IC_kwDOBm6k_c5MyoP8,9599,simonw,2022-10-24T02:50:29Z,2022-10-24T02:50:29Z,OWNER,"https://latest.datasette.io/_internal now looks like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420090659,Private database page should show padlock on every table, https://github.com/simonw/datasette/issues/1848#issuecomment-1288330238,https://api.github.com/repos/simonw/datasette/issues/1848,1288330238,IC_kwDOBm6k_c5Mylv-,9599,simonw,2022-10-24T02:34:41Z,2022-10-24T02:34:41Z,OWNER,"Tested my fix with this `metadata.yml`: ```yaml databases: fixtures: allow: id: root tables: 123_starts_with_digits: allow: true ``` Signed in as root I saw this - showing that the `123_starts_with_digits` table is public: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420090659,Private database page should show padlock on every table, https://github.com/simonw/datasette/issues/1848#issuecomment-1288327467,https://api.github.com/repos/simonw/datasette/issues/1848,1288327467,IC_kwDOBm6k_c5MylEr,9599,simonw,2022-10-24T02:30:48Z,2022-10-24T02:31:04Z,OWNER,"Here's the code at fault: https://github.com/simonw/datasette/blob/78dad236df730212aa7172f885fd8ec575f0d3ad/datasette/views/database.py#L67-L116 Those checks aren't doing the new cascading permissions thing added in #1829 which means they can't tell that an anonymous user would not be able to se those tbles and queries and views. Should do something like this instead: ```python view_visible, view_private = await self.ds.check_visibility( request.actor, permissions=[ (""view-table"", (database, view_name)), (""view-database"", database), ""view-instance"", ], ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420090659,Private database page should show padlock on every table, https://github.com/simonw/datasette/issues/1829#issuecomment-1288321630,https://api.github.com/repos/simonw/datasette/issues/1829,1288321630,IC_kwDOBm6k_c5Myjpe,9599,simonw,2022-10-24T02:22:49Z,2022-10-24T02:23:46Z,OWNER,"Visit https://latest.datasette.io/login-as-root and then: https://latest.datasette.io/ https://latest.datasette.io/_internal/columns https://latest.datasette.io/_internal/columns/_internal,columns,cid https://latest.datasette.io/_internal/from_hook That's all as it should be.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1829#issuecomment-1288320411,https://api.github.com/repos/simonw/datasette/issues/1829,1288320411,IC_kwDOBm6k_c5MyjWb,9599,simonw,2022-10-24T02:21:19Z,2022-10-24T02:21:19Z,OWNER,Updated docs for `check_visibility()`: https://docs.datasette.io/en/latest/internals.html#await-check-visibility-actor-action-none-resource-none-permissions-none,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/pull/1842#issuecomment-1288311852,https://api.github.com/repos/simonw/datasette/issues/1842,1288311852,IC_kwDOBm6k_c5MyhQs,9599,simonw,2022-10-24T02:11:12Z,2022-10-24T02:11:12Z,OWNER,"I'm going to construct a `metadata.yml` which makes various databases and tables visible or invisible, then browse them using the root user. `block-instance.yml`: ```yaml allow: id: root ``` `block-database.yml`: ```yaml databases: fixtures: allow: id: root ``` `block-table.yml`: ```yaml databases: fixtures: tables: searchable: allow: id: root ``` `block-query.yml`: ```yaml databases: fixtures: queries: two: sql: select 1 + 1 allow: id: root ``` https://gist.github.com/simonw/2d007ebe43de46d44499c77a2a291756 - checkout that Gist to get all four. I manually tested all four scenarios with root and non-root users and confirmed that they worked correctly and padlocks were shown in the right places.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408561039,check_visibility can now take multiple permissions into account, https://github.com/simonw/datasette/issues/1829#issuecomment-1288308945,https://api.github.com/repos/simonw/datasette/issues/1829,1288308945,IC_kwDOBm6k_c5MygjR,9599,simonw,2022-10-24T02:07:50Z,2022-10-24T02:07:50Z,OWNER,"Useful test: if you sign in as root to https://latest.datasette.io/_internal/columns/_internal,columns,database_name you can see there's no padlock icon on that page or on https://latest.datasette.io/_internal/columns - fixing this bug should fix that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/pull/1842#issuecomment-1278306180,https://api.github.com/repos/simonw/datasette/issues/1842,1278306180,IC_kwDOBm6k_c5MMWeE,22429695,codecov[bot],2022-10-14T00:11:46Z,2022-10-24T02:04:52Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.52**% // Head: **92.54**% // Increases project coverage by **`+0.02%`** :tada: > Coverage data is based on head [(`3623475`)](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`79aa0de`)](https://codecov.io/gh/simonw/datasette/commit/79aa0de083d38a9975915d5a4cc68ca6c74fbe3d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch coverage: 100.00% of modified lines in pull request are covered.
      Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1842 +/- ## ========================================== + Coverage 92.52% 92.54% +0.02% ========================================== Files 35 35 Lines 4415 4428 +13 ========================================== + Hits 4085 4098 +13 Misses 330 330 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `94.28% <100.00%> (+0.05%)` | :arrow_up: | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `95.29% <100.00%> (+0.06%)` | :arrow_up: | | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <100.00%> (ø)` | | | [datasette/views/row.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3Jvdy5weQ==) | `88.70% <100.00%> (+0.37%)` | :arrow_up: | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1842/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.20% <100.00%> (+0.01%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
      [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1842?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408561039,check_visibility can now take multiple permissions into account, https://github.com/simonw/datasette/pull/1842#issuecomment-1288304224,https://api.github.com/repos/simonw/datasette/issues/1842,1288304224,IC_kwDOBm6k_c5MyfZg,9599,simonw,2022-10-24T02:00:14Z,2022-10-24T02:00:14Z,OWNER,I need to do one last round of manual testing before I merge this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408561039,check_visibility can now take multiple permissions into account, https://github.com/simonw/datasette/issues/1847#issuecomment-1288296235,https://api.github.com/repos/simonw/datasette/issues/1847,1288296235,IC_kwDOBm6k_c5Mydcr,9599,simonw,2022-10-24T01:45:56Z,2022-10-24T01:45:56Z,OWNER,This bug here: https://github.com/simonw/datasette/blob/85d5d2762c13d2b5a8bd9c5ec81c77fe6577121f/tests/test_permissions.py#L485,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420055377,Both _local_metadata and _metadata_local?, https://github.com/simonw/datasette/issues/1847#issuecomment-1288295713,https://api.github.com/repos/simonw/datasette/issues/1847,1288295713,IC_kwDOBm6k_c5MydUh,9599,simonw,2022-10-24T01:45:13Z,2022-10-24T01:45:13Z,OWNER,"Turns out that was a bug I had introduced while working on that test, and it was the reason I was blocked on finishing work on: - #1829","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1420055377,Both _local_metadata and _metadata_local?, https://github.com/simonw/datasette/issues/1843#issuecomment-1288214953,https://api.github.com/repos/simonw/datasette/issues/1843,1288214953,IC_kwDOBm6k_c5MyJmp,9599,simonw,2022-10-23T22:22:52Z,2022-10-23T22:22:52Z,OWNER,This seems to have fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282830806,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282830806,IC_kwDOCGYnMM5MdnHW,9599,simonw,2022-10-18T18:23:36Z,2022-10-18T18:23:36Z,OWNER,"Tests pass now. Updated docs: - https://sqlite-utils.datasette.io/en/latest/cli.html#table-formatted-output - https://sqlite-utils.datasette.io/en/latest/cli-reference.html#query - and many other places on that page","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282819035,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282819035,IC_kwDOCGYnMM5MdkPb,9599,simonw,2022-10-18T18:15:05Z,2022-10-18T18:15:05Z,OWNER,I'm going to skip the cog test on Python 3.6 to address this. The documentation on the website will show the available list of options for 3.7 and higher.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282817901,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282817901,IC_kwDOCGYnMM5Mdj9t,9599,simonw,2022-10-18T18:14:35Z,2022-10-18T18:14:35Z,OWNER,"Now the 3.6 tests fail - because the new release of tabulate dropped support for that Python version (so on Python 3.6 you get an older version): https://github.com/simonw/sqlite-utils/actions/runs/3275842849/jobs/5391181675 https://github.com/astanin/python-tabulate/blame/20c6370d5da2dae89b305bfb6c7f12a0f8b7236c/pyproject.toml#L22 shows minimum is 3.7 now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/501#issuecomment-1282813168,https://api.github.com/repos/simonw/sqlite-utils/issues/501,1282813168,IC_kwDOCGYnMM5Mdizw,9599,simonw,2022-10-18T18:12:15Z,2022-10-18T18:12:15Z,OWNER,"Here's the new Tabulate release: - https://github.com/astanin/python-tabulate/releases/tag/v0.9.0 - https://github.com/astanin/python-tabulate/compare/v0.8.10...v0.9.0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413641049,Tests failing due to updated tabulate library, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282800547,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282800547,IC_kwDOCGYnMM5Mdfuj,9599,simonw,2022-10-18T18:02:09Z,2022-10-18T18:02:09Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/reference.html#sqlite-utils-utils-flatten,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 1, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282780770,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282780770,IC_kwDOCGYnMM5Mda5i,9599,simonw,2022-10-18T17:45:56Z,2022-10-18T17:46:05Z,OWNER,I think the public interface is a `flatten(row)` function that does `dict(_flatten(row))`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282779755,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282779755,IC_kwDOCGYnMM5Mdapr,9599,simonw,2022-10-18T17:45:10Z,2022-10-18T17:45:10Z,OWNER,It should go in `sqlite_utils.utils` - documented here: https://sqlite-utils.datasette.io/en/stable/reference.html#sqlite-utils-utils,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/sqlite-utils/issues/500#issuecomment-1282778928,https://api.github.com/repos/simonw/sqlite-utils/issues/500,1282778928,IC_kwDOCGYnMM5Mdacw,9599,simonw,2022-10-18T17:44:20Z,2022-10-18T17:44:20Z,OWNER,"Here's how it works: https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1847-L1848 https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/sqlite_utils/cli.py#L1082-L1088","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1413610718,Turn --flatten into a documented utility function, https://github.com/simonw/datasette/issues/1845#issuecomment-1279924827,https://api.github.com/repos/simonw/datasette/issues/1845,1279924827,IC_kwDOBm6k_c5MShpb,30636,kindly,2022-10-16T08:54:53Z,2022-10-16T08:54:53Z,NONE,"> It was part of a larger idea I was exploring around ensuring Datasette could be used to start interacting with CSV/JSON data out-of-the-box, without needing to first convert that data into SQLite using separate tools. This would be great. My organization deals with very nested JSON open data and I have been wanting to find a way to hook into datasette so that the analysts do not have to first convert to sqlite first. This can kind of be done with datasette-lite. From this random nested JSON API: https://api.nobelprize.org/v1/prize.json You can use the API of https://flatterer.herokuapp.com to return a multi table sqlite database: https://lite.datasette.io/?url=https://flatterer.herokuapp.com/api/convert?output_format=sqlite%26file_url=https://api.nobelprize.org/v1/prize.json This is great and fun, but it would be great if there was some plugin mechanism that you could feed a local datasette a nested JSON file directly, possibly hooking into other flattening tools for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1410305897,Reconsider the Datasette first-run experience, https://github.com/simonw/datasette/issues/1845#issuecomment-1279846110,https://api.github.com/repos/simonw/datasette/issues/1845,1279846110,IC_kwDOBm6k_c5MSObe,4399499,ocdtrekkie,2022-10-15T22:49:00Z,2022-10-15T22:49:00Z,NONE,"I think there's probably some core plugins a new user should just ""get"" maybe. I feel our Sandstorm package has really reasonable defaults, the ability to bring data in for instance. However, you also have to know that's in the hamburger menu, it's not necessarily intuitive where to start.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1410305897,Reconsider the Datasette first-run experience, https://github.com/simonw/datasette/issues/1845#issuecomment-1279842912,https://api.github.com/repos/simonw/datasette/issues/1845,1279842912,IC_kwDOBm6k_c5MSNpg,9599,simonw,2022-10-15T22:22:58Z,2022-10-15T22:22:58Z,OWNER,"I think this mechanism could go a long way towards helping here: - https://github.com/simonw/datasette/issues/1160 It was part of a larger idea I was exploring around ensuring Datasette could be used to start interacting with CSV/JSON data out-of-the-box, without needing to first convert that data into SQLite using separate tools.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1410305897,Reconsider the Datasette first-run experience, https://github.com/simonw/datasette/issues/1844#issuecomment-1279598878,https://api.github.com/repos/simonw/datasette/issues/1844,1279598878,IC_kwDOBm6k_c5MRSEe,9599,simonw,2022-10-14T23:51:46Z,2022-10-14T23:51:46Z,OWNER,Blogged about this here: https://simonwillison.net/2022/Oct/14/automating-screenshots/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279427618,https://api.github.com/repos/simonw/datasette/issues/1844,1279427618,IC_kwDOBm6k_c5MQoQi,9599,simonw,2022-10-14T20:25:45Z,2022-10-14T20:25:45Z,OWNER,Extracted a TIL: https://til.simonwillison.net/shot-scraper/subset-of-table-columns,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279415365,https://api.github.com/repos/simonw/datasette/issues/1844,1279415365,IC_kwDOBm6k_c5MQlRF,9599,simonw,2022-10-14T20:11:55Z,2022-10-14T20:11:55Z,OWNER,Twitter thread about this issue: https://twitter.com/simonw/status/1581012617526595584,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279406134,https://api.github.com/repos/simonw/datasette/issues/1844,1279406134,IC_kwDOBm6k_c5MQjA2,9599,simonw,2022-10-14T20:01:13Z,2022-10-14T20:01:13Z,OWNER,"Here's the YAML I added to https://github.com/simonw/datasette-screenshots/blob/main/shots.yml for this issue: ```yaml - url: https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date height: 585 width: 960 output: regmem-search.png - url: https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper selector: ""#export"" output: advanced-export.png padding: 10 - url: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10 selectors_all: - .suggested-facets a - tr:not(tr:nth-child(n+4)) td:not(:nth-child(n+11)) padding: 10 output: faceting-details.png - url: https://latest.datasette.io/fixtures/binary_data selector: table javascript: |- Array.from( document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el) ); padding: 10 output: binary-data.png ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279405429,https://api.github.com/repos/simonw/datasette/issues/1844,1279405429,IC_kwDOBm6k_c5MQi11,9599,simonw,2022-10-14T20:00:26Z,2022-10-14T20:00:26Z,OWNER,"New images are now live on these pages: - https://docs.datasette.io/en/latest/csv_export.html - https://docs.datasette.io/en/latest/binary_data.html - https://docs.datasette.io/en/latest/facets.html - https://docs.datasette.io/en/latest/full_text_search.html - https://docs.datasette.io/en/latest/changelog.html#v0-23 (was a duplicate of the advanced export image)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279392717,https://api.github.com/repos/simonw/datasette/issues/1844,1279392717,IC_kwDOBm6k_c5MQfvN,9599,simonw,2022-10-14T19:44:44Z,2022-10-14T19:45:54Z,OWNER,"OK, the URLs to use in the docs are: * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/advanced-export.png (retina) * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/non-retina/regmem-search.png * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/binary-data.png (retina) * https://raw.githubusercontent.com/simonw/datasette-screenshots/0.62/non-retina/faceting-details.png ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279349314,https://api.github.com/repos/simonw/datasette/issues/1844,1279349314,IC_kwDOBm6k_c5MQVJC,9599,simonw,2022-10-14T18:50:42Z,2022-10-14T19:34:37Z,OWNER,"I'm going to link the documentation screenshots directly to the images in the https://github.com/simonw/datasette-screenshots repository - but I don't want those images to reflect `main` when the documentation may reflect a specific version. So I'm going to start tagging releases of `datasette-screenshots` so I can get permanent URLs to those images.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279383121,https://api.github.com/repos/simonw/datasette/issues/1844,1279383121,IC_kwDOBm6k_c5MQdZR,9599,simonw,2022-10-14T19:33:49Z,2022-10-14T19:33:49Z,OWNER,"I'm going to tag `datasette-screenshots` with the current Datasette version, `0.62`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279311487,https://api.github.com/repos/simonw/datasette/issues/1844,1279311487,IC_kwDOBm6k_c5MQL5_,9599,simonw,2022-10-14T18:06:53Z,2022-10-14T19:33:24Z,OWNER,"I just spotted some other out-dated screenshots in the `docs/` directory: - [x] [advanced_export.png](https://github.com/simonw/datasette/blob/main/docs/advanced_export.png ""advanced_export.png"") - [x] [binary_data.png](https://github.com/simonw/datasette/blob/main/docs/binary_data.png ""binary_data.png"") - [x] [facets.png](https://github.com/simonw/datasette/blob/main/docs/facets.png ""facets.png"") - [x] [full_text_search.png](https://github.com/simonw/datasette/blob/main/docs/full_text_search.png ""full_text_search.png"") ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279382674,https://api.github.com/repos/simonw/datasette/issues/1844,1279382674,IC_kwDOBm6k_c5MQdSS,9599,simonw,2022-10-14T19:33:16Z,2022-10-14T19:33:16Z,OWNER,"That's the last two screenshots: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279348239,https://api.github.com/repos/simonw/datasette/issues/1844,1279348239,IC_kwDOBm6k_c5MQU4P,9599,simonw,2022-10-14T18:49:22Z,2022-10-14T18:49:22Z,OWNER,"This works: ``` shot-scraper 'https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10' \ -s '.suggested-facets a' \ --selector-all 'tr:not(tr:nth-child(n+4)) td:not(:nth-child(n+11))' \ -p 10 ``` ![congress-legislators-datasettes-com-legislators-legislator_terms 6](https://user-images.githubusercontent.com/9599/195919422-97616694-3ec0-4e05-afc2-c509275c767c.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279339124,https://api.github.com/repos/simonw/datasette/issues/1844,1279339124,IC_kwDOBm6k_c5MQSp0,9599,simonw,2022-10-14T18:38:22Z,2022-10-14T18:42:58Z,OWNER,"This seems to get every table cell in that table for the first 3 rows and the columns up to `party`: document.querySelectorAll('tr:not(:nth-child(n+4)) td:not(:nth-child(n+10))')","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279334694,https://api.github.com/repos/simonw/datasette/issues/1844,1279334694,IC_kwDOBm6k_c5MQRkm,9599,simonw,2022-10-14T18:33:41Z,2022-10-14T18:34:32Z,OWNER,"I'm going to use this page for the facets screenshot: https://congress-legislators.datasettes.com/legislators/legislator_terms?_facet=type&_facet=party&_facet=state&_facet_size=10 Trying for this bit: Which incorporates `.suggested-facets` but also the first 3 rows and 10 columns of the table, I wonder if I can specify that in a single selector?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279325685,https://api.github.com/repos/simonw/datasette/issues/1844,1279325685,IC_kwDOBm6k_c5MQPX1,9599,simonw,2022-10-14T18:23:22Z,2022-10-14T18:23:22Z,OWNER,"Here's the new advanced export image: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279325003,https://api.github.com/repos/simonw/datasette/issues/1844,1279325003,IC_kwDOBm6k_c5MQPNL,9599,simonw,2022-10-14T18:22:35Z,2022-10-14T18:22:35Z,OWNER,"So: ``` shot-scraper 'https://latest.datasette.io/fixtures/binary_data' \ -j ""Array.from(document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el));"" \ -s table -p 10 ``` ![latest-datasette-io-fixtures-binary_data 1](https://user-images.githubusercontent.com/9599/195915092-be81db43-5672-4375-bd66-4316211b1afc.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279323714,https://api.github.com/repos/simonw/datasette/issues/1844,1279323714,IC_kwDOBm6k_c5MQO5C,9599,simonw,2022-10-14T18:21:03Z,2022-10-14T18:21:03Z,OWNER,"For this image: https://latest.datasette.io/fixtures/binary_data has an extra row these days: This deletes every row past the first two (first three including the header row): ```javascipt Array.from(document.querySelectorAll('tr:nth-child(n+3)'), el => el.parentNode.removeChild(el)); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279314400,https://api.github.com/repos/simonw/datasette/issues/1844,1279314400,IC_kwDOBm6k_c5MQMng,9599,simonw,2022-10-14T18:10:07Z,2022-10-14T18:13:36Z,OWNER,"For the advanced export one: shot-scraper 'https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper' -s '#export' -p 10 Produces: ![register-of-members-interests-datasettes-com-regmem-items 2](https://user-images.githubusercontent.com/9599/195913614-448557aa-5ec6-4a83-98cf-8837d3117204.png) Current image is: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/simonw/datasette/issues/1844#issuecomment-1279316670,https://api.github.com/repos/simonw/datasette/issues/1844,1279316670,IC_kwDOBm6k_c5MQNK-,9599,simonw,2022-10-14T18:12:39Z,2022-10-14T18:12:39Z,OWNER,"New screenshot of FTS, from https://register-of-members-interests.datasettes.com/regmem/items?_search=hamper&_sort_desc=date ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1409679008,Update screenshots in documentation to match latest designs, https://github.com/dogsheep/twitter-to-sqlite/issues/60#issuecomment-1279249898,https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/60,1279249898,IC_kwDODEm0Qs5MP83q,7908073,chapmanjacobd,2022-10-14T16:58:26Z,2022-10-14T16:58:26Z,NONE,"You could try using `msys2`. I've had better luck running python CLIs within that system on Windows. Here is a guide: https://github.com/chapmanjacobd/lb/blob/main/Windows.md#prep","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1063982712,Execution on Windows, https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-1279224780,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51,1279224780,IC_kwDODFdgUs5MP2vM,7908073,chapmanjacobd,2022-10-14T16:34:07Z,2022-10-14T16:34:07Z,NONE,"also, it says that authenticated requests have a much higher ""rate limit"". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a ""rate limit"" (although I guess that is semantic equivalence) You would want to use `x-ratelimit-reset` ``` time.sleep(r['x-ratelimit-reset'] + 1 - time.time()) ``` But a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only `github-to-sqlite get` is using the `--auth=` CLI flag","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703246031,github-to-sqlite should handle rate limits better, https://github.com/simonw/datasette/issues/1843#issuecomment-1278537920,https://api.github.com/repos/simonw/datasette/issues/1843,1278537920,IC_kwDOBm6k_c5MNPDA,9599,simonw,2022-10-14T06:19:55Z,2022-10-14T06:20:06Z,OWNER,Maybe I need to explicitly close those SQLite connections held by the Datasette instance after this line: https://github.com/simonw/datasette/blob/79aa0de083d38a9975915d5a4cc68ca6c74fbe3d/tests/fixtures.py#L165,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1278480437,https://api.github.com/repos/simonw/datasette/issues/1843,1278480437,IC_kwDOBm6k_c5MNBA1,9599,simonw,2022-10-14T04:51:10Z,2022-10-14T04:51:10Z,OWNER,Extracted a TIL: https://til.simonwillison.net/python/too-many-open-files-psutil,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1843#issuecomment-1278478042,https://api.github.com/repos/simonw/datasette/issues/1843,1278478042,IC_kwDOBm6k_c5MNAba,9599,simonw,2022-10-14T04:46:29Z,2022-10-14T04:46:29Z,OWNER,"I did `pip install psutil` and then ran this in the debugger for one of these errors: ```python import psutil for f in psutil.Process().open_files(): print(f) ``` The output looked like this: ``` popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=11) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=12) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=13) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=14) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=15) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpq31d2af1/fixtures.db', fd=16) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=17) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=18) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=19) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfd3oyo10/fixtures.dot.db', fd=20) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=21) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpezwfu7w8/fixtures.db', fd=22) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=24) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/extra database.db', fd=25) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpz6e2anqw/fixtures.db', fd=26) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=27) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdp4we7hb/fixtures.db', fd=28) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=29) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4ljq_ai0/fixtures.db', fd=31) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=32) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/extra database.db', fd=33) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp907xmnzb/fixtures.db', fd=34) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpw32vwkjq/fixtures.db', fd=35) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo-bar.db', fd=36) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/foo.db', fd=37) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpzlwn6bqm/fixtures.db', fd=38) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=39) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp42e6vyj_/fixtures.db', fd=40) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoxdpxj6w/fixtures.db', fd=41) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=42) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=43) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=44) popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=45) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0w5jugqk/fixtures.db', fd=46) popenfile(path='/Users/simon/Dropbox/Development/datasette/tests/spatialite.db', fd=47) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmb9y9fba/fixtures.db', fd=48) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_create0/does_not_exist_yet.db', fd=49) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=92) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/fixtures.db', fd=93) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfwsbx941/data.db', fd=94) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=95) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=99) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_deduplicate_same_da0/db.db', fd=100) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=101) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=102) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/db.db', fd=103) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_serve_duplicate_database_0/nested/db.db', fd=104) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=105) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=106) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=107) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_test0/test-database (1).sqlite', fd=109) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=111) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test_weird_database_names_data0/database (1).sqlite', fd=113) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=117) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=118) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=119) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/demo.db', fd=120) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/immutable.db', fd=121) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/k.sqlite', fd=122) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/config-dir0/j.sqlite3', fd=123) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=124) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=125) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=126) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/extra database.db', fd=127) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmppju9w34z/fixtures.db', fd=128) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=129) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=130) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=131) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=132) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=133) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=134) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=135) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=136) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=137) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=138) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=139) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=140) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=141) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=142) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=143) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=144) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=145) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=146) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=147) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=148) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=149) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=150) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=151) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=152) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=153) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=154) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=155) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=156) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=157) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=158) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=159) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=160) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_10.db', fd=161) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_0.db', fd=162) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_1.db', fd=163) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_2.db', fd=164) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_3.db', fd=165) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_4.db', fd=166) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_5.db', fd=167) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_6.db', fd=168) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_7.db', fd=169) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_8.db', fd=170) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/dbs0/db_9.db', fd=171) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpky1jamnv/fixtures.db', fd=172) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=173) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpmzu4jbdx/fixtures.db', fd=174) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=175) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpskyh32wh/fixtures.db', fd=176) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpcl1edmyv/fixtures.db', fd=177) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp966705ec/fixtures.db', fd=178) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=179) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpwmewpy8e/fixtures.db', fd=180) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=181) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpkyn60obe/fixtures.db', fd=182) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4gctb4i6/fixtures.db', fd=183) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=184) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=185) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=187) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=188) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/extra database.db', fd=189) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmps7p_ee_e/fixtures.db', fd=190) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=191) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpf3_6i1d6/fixtures.db', fd=192) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=193) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp6_9vzq4o/fixtures.db', fd=197) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/extra database.db', fd=198) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpqtubcoah/fixtures.db', fd=211) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmprh373nt1/fixtures.db', fd=212) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp2gauogmy/fixtures.db', fd=215) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=216) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpd2tleqni/fixtures.db', fd=217) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmplpdc3wfl/fixtures.db', fd=219) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=222) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=223) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=224) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=225) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpdnsscl5v/fixtures.db', fd=226) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=227) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp4u5oo5ne/fixtures.db', fd=247) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmp0dgqc4wx/fixtures.db', fd=251) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpoehh1_tg/fixtures.db', fd=253) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmptjayd3pn/fixtures.db', fd=254) popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pytest-of-simon/pytest-14/test-view-names0/fixtures.db', fd=255) ``` Clearly something is bad with the way fixtures work in the tests - a huge number of `fixtures.db` database files are being created and left open!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1408757705,"Intermittent ""Too many open files"" error running tests", https://github.com/simonw/datasette/issues/1829#issuecomment-1278302478,https://api.github.com/repos/simonw/datasette/issues/1829,1278302478,IC_kwDOBm6k_c5MMVkO,9599,simonw,2022-10-14T00:06:19Z,2022-10-14T00:06:19Z,OWNER,"I'll finish this in a PR: - https://github.com/simonw/datasette/pull/1842","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1829#issuecomment-1278300241,https://api.github.com/repos/simonw/datasette/issues/1829,1278300241,IC_kwDOBm6k_c5MMVBR,9599,simonw,2022-10-14T00:03:52Z,2022-10-14T00:04:28Z,OWNER,"Here's what I've got so far: ```diff diff --git a/datasette/app.py b/datasette/app.py index 5fa4955c..df9eae49 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -1,5 +1,5 @@ import asyncio -from typing import Sequence, Union, Tuple +from typing import Sequence, Union, Tuple, Optional import asgi_csrf import collections import datetime @@ -707,7 +707,7 @@ class Datasette: Raises datasette.Forbidden() if any of the checks fail """""" - assert actor is None or isinstance(actor, dict) + assert actor is None or isinstance(actor, dict), ""actor must be None or a dict"" for permission in permissions: if isinstance(permission, str): action = permission @@ -732,23 +732,34 @@ class Datasette: else: raise Forbidden(action) - async def check_visibility(self, actor, action, resource): + async def check_visibility( + self, + actor: dict, + action: Optional[str] = None, + resource: Optional[str] = None, + permissions: Optional[ + Sequence[Union[Tuple[str, Union[str, Tuple[str, str]]], str]] + ] = None, + ): """"""Returns (visible, private) - visible = can you see it, private = can others see it too"""""" - visible = await self.permission_allowed( - actor, - action, - resource=resource, - default=True, - ) - if not visible: + if permissions: + assert ( + not action and not resource + ), ""Can't use action= or resource= with permissions="" + else: + permissions = [(action, resource)] + try: + await self.ensure_permissions(actor, permissions) + except Forbidden: return False, False - private = not await self.permission_allowed( - None, - action, - resource=resource, - default=True, - ) - return visible, private + # User can see it, but can the anonymous user see it? + try: + await self.ensure_permissions(None, permissions) + except Forbidden: + # It's visible but private + return True, True + # It's visible to everyone + return True, False async def execute( self, diff --git a/datasette/views/table.py b/datasette/views/table.py index 60c092f9..f73b0957 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -28,7 +28,7 @@ from datasette.utils import ( urlsafe_components, value_as_boolean, ) -from datasette.utils.asgi import BadRequest, NotFound +from datasette.utils.asgi import BadRequest, Forbidden, NotFound from datasette.filters import Filters from .base import DataView, DatasetteError, ureg from .database import QueryView @@ -213,18 +213,16 @@ class TableView(DataView): raise NotFound(f""Table not found: {table_name}"") # Ensure user has permission to view this table - await self.ds.ensure_permissions( + visible, private = await self.ds.check_visibility( request.actor, - [ + permissions=[ (""view-table"", (database_name, table_name)), (""view-database"", database_name), ""view-instance"", ], ) - - private = not await self.ds.permission_allowed( - None, ""view-table"", (database_name, table_name), default=True - ) + if not visible: + raise Forbidden(""You do not have permission to view this table"") # Handle ?_filter_column and redirect, if present redirect_params = filters_should_redirect(request.args) ``` Still needs tests and a documentation update. Also this fix is currently only applied on the table page - needs to be applied on database, row and query pages too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1829#issuecomment-1278237331,https://api.github.com/repos/simonw/datasette/issues/1829,1278237331,IC_kwDOBm6k_c5MMFqT,9599,simonw,2022-10-13T22:17:45Z,2022-10-13T22:19:22Z,OWNER,I think `check_visibility` should be changed to optionally accept `permissions=` which is the same list of tuples that can be passed to `ensure_permissions`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1396948693,Table/database that is private due to inherited permissions does not show padlock, https://github.com/simonw/datasette/issues/1831#issuecomment-1278198700,https://api.github.com/repos/simonw/datasette/issues/1831,1278198700,IC_kwDOBm6k_c5ML8Os,9599,simonw,2022-10-13T21:29:09Z,2022-10-13T21:29:09Z,OWNER,"I'm going to commit the code now, but then I need to add some extra tests to ensure the breadcrumb permission display logic works correctly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1278198145,https://api.github.com/repos/simonw/datasette/issues/1831,1278198145,IC_kwDOBm6k_c5ML8GB,9599,simonw,2022-10-13T21:28:30Z,2022-10-13T21:28:30Z,OWNER,"This has turned into a full refactor of how breadcrumbs work. I'm using my first ever Jinja macro for this - I import that at the top of `base.html` so that it will be available everywhere else: ```html+jinja {% import ""_crumbs.html"" as crumbs with context %} ``` The `with context` bit is needed so the macro can see the new `crumb_items()` function that I'm adding to the global template rendering scope. Here's the full content of `_crumbs.html`: ```html+jinja {% macro nav(request, database=None, table=None) -%} {% set items=crumb_items(request=request, database=database, table=table) %} {% if items %}

      {% for item in items %} {{ item.label }} {% if not loop.last %} / {% endif %} {% endfor %}

      {% endif %} {%- endmacro %} ``` This means custom template authors can use their own `_crumbs.html` template to do something different with the breadcrumbs. In the actual templates I display breadcrumbs like this: ```html+jinja {% block crumbs %} {{ crumbs.nav(request=request, database=database) }} {% endblock %} ``` Pass `database=` to get `home / database_name` - pass `table=` as well to get `home / database_name / table_name` - if you just send `request=` you just get `home`. I've also made the default base template show the `home` breadcrumbs - other pages such as `table.html` and `row.html` can then over-ride `{% block crumbs %}` to get different breadcrumbs. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1278175798,https://api.github.com/repos/simonw/datasette/issues/1831,1278175798,IC_kwDOBm6k_c5ML2o2,9599,simonw,2022-10-13T21:02:52Z,2022-10-13T21:02:52Z,OWNER,This patch to `default_permissions.py` made debugging easier: https://gist.github.com/simonw/daddf022e75a98ea6246ac1e12dc8759,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1276882359,https://api.github.com/repos/simonw/datasette/issues/1831,1276882359,IC_kwDOBm6k_c5MG623,9599,simonw,2022-10-13T00:36:09Z,2022-10-13T00:36:19Z,OWNER,"It's important that, however this works, it supports custom templates changing how the breadcrumbs are displayed. Probably needs a `_crumbs.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1276856836,https://api.github.com/repos/simonw/datasette/issues/1831,1276856836,IC_kwDOBm6k_c5MG0oE,9599,simonw,2022-10-12T23:57:28Z,2022-10-12T23:57:28Z,OWNER,"As part of this I think I want `request` to always be available in the template context, which will remove the need for https://datasette.io/plugins/datasette-template-request","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1397084281,If user can see table but NOT database/instance nav links should not display, https://github.com/simonw/datasette/issues/1831#issuecomment-1276706181,https://api.github.com/repos/simonw/datasette/issues/1831,1276706181,IC_kwDOBm6k_c5MGP2F,9599,simonw,2022-10-12T20:34:00Z,2022-10-12T23:43:39Z,OWNER,"Maybe new template functions: `table_crumbs(table, database)` and `database_crumbs(database)` and `instance_crumbs()` - which know how to both check the permissions and display the `
      .rows >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > /home/xk/github/xk/lb/xklb/check.py(11)test_transaction() 9 with db1.conn: 10 db1[""t""].insert({""foo"": 1}) ---> 11 assert list(db2[""t""].rows) == [] 12 assert list(db2[""t""].rows) == [{""foo"": 1}] ``` It fails because it is already inserted. btw if you put these two lines in you pyproject.toml you can get `ipdb` in pytest ``` [tool.pytest.ini_options] addopts = ""--pdbcls=IPython.terminal.debugger:TerminalPdb --ignore=tests/data --capture=tee-sys --log-cli-level=ERROR"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1149661489,`with db:` for transactions, https://github.com/simonw/sqlite-utils/issues/493#issuecomment-1264219650,https://api.github.com/repos/simonw/sqlite-utils/issues/493,1264219650,IC_kwDOCGYnMM5LWnYC,7908073,chapmanjacobd,2022-10-01T03:22:50Z,2022-10-01T03:23:58Z,CONTRIBUTOR,"this is likely what you are looking for: https://stackoverflow.com/a/51076749/697964 but yeah I would say just disable smart quotes","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386562662,Tiny typographical error in install/uninstall docs, https://github.com/simonw/datasette/pull/1827#issuecomment-1263570186,https://api.github.com/repos/simonw/datasette/issues/1827,1263570186,IC_kwDOBm6k_c5LUI0K,22429695,codecov[bot],2022-09-30T13:22:15Z,2022-09-30T13:22:15Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **92.50**% // Head: **92.50**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`1f0c557`)](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`34defdc`)](https://codecov.io/gh/simonw/datasette/commit/34defdc10aa293294ca01cfab70780755447e1d7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines.
      Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1827 +/- ## ======================================= Coverage 92.50% 92.50% ======================================= Files 35 35 Lines 4400 4400 ======================================= Hits 4070 4070 Misses 330 330 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
      [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1827?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1392426838,Bump furo from 2022.9.15 to 2022.9.29, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262920929,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1262920929,IC_kwDOCGYnMM5LRqTh,9599,simonw,2022-09-29T23:06:44Z,2022-09-29T23:06:44Z,OWNER,"Currently the only other use of `-t` is for this: ``` -t, --table Output as a formatted table ``` So I think it's OK to use it to mean something slightly different for this command, since `sqlite-utils insert` doesn't do any output of data in any format.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262918833,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1262918833,IC_kwDOCGYnMM5LRpyx,9599,simonw,2022-09-29T23:02:52Z,2022-09-29T23:02:52Z,OWNER,"The other nice thing about having this as a separate command is that I can implement a tiny subset of the overall `sqlite-utils insert` features at first, and then add additional features in subsequent releases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262917059,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1262917059,IC_kwDOCGYnMM5LRpXD,9599,simonw,2022-09-29T22:59:28Z,2022-09-29T22:59:28Z,OWNER,"I quite like `sqlite-utils fast-csv` - I think it's clear enough what it does, and running `--help` can clarify if needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262915322,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1262915322,IC_kwDOCGYnMM5LRo76,9599,simonw,2022-09-29T22:57:31Z,2022-09-29T22:57:42Z,OWNER,Maybe `sqlite-utils fast-csv` is right? Not entirely clear that's an insert though as opposed to a faster version of in-memory querying in the style of `sqlite-utils memory`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262914416,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1262914416,IC_kwDOCGYnMM5LRotw,9599,simonw,2022-09-29T22:56:53Z,2022-09-29T22:56:53Z,OWNER,"Potential names/designs: - `sqlite-utils fast data.db rows rows.csv` - `sqlite-utils insert-fast data.db rows rows.csv` - `sqlite-utils fast-csv data.db rows rows.csv` Or more interestingly... what if it could accept multiple CSV files to create multiple tables? - `sqlite-utils fast data.db rows.csv other.csv` Would still need to support creating tables with different names though. Maybe like this: - `sqlite-utils fast data.db -t mytable rows.csv -t othertable other.csv` I seem to be leaning towards `fast` as the command name, but as a standalone command name it's a bit meaningless - how do we know that's about CSV import and not about fast querying or similar?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/sqlite-utils/issues/297#issuecomment-1262913145,https://api.github.com/repos/simonw/sqlite-utils/issues/297,1262913145,IC_kwDOCGYnMM5LRoZ5,9599,simonw,2022-09-29T22:54:13Z,2022-09-29T22:54:13Z,OWNER,"After reviewing `sqlite-utils insert --help` I'm confident that MOST of these options wouldn't make sense for a ""fast"" moder that just supports CSV and works by piping directly to the `sqlite3` binary: https://github.com/simonw/sqlite-utils/blob/d792dad1cf5f16525da81b1e162fb71d469995f3/docs/cli-reference.rst#L251-L279 I'm going to implement a separate command instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",944846776,Option for importing CSV data using the SQLite .import mechanism, https://github.com/simonw/datasette/issues/370#issuecomment-1261930179,https://api.github.com/repos/simonw/datasette/issues/370,1261930179,IC_kwDOBm6k_c5LN4bD,72577720,MichaelTiemannOSC,2022-09-29T08:17:46Z,2022-09-29T08:17:46Z,CONTRIBUTOR,"Just watched this video which demonstrates the integration of *any* webapp into JupyterLab: https://youtu.be/FH1dKKmvFtc Maybe this is the answer?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",377155320,Integration with JupyterLab, https://github.com/simonw/datasette/issues/1624#issuecomment-1261194164,https://api.github.com/repos/simonw/datasette/issues/1624,1261194164,IC_kwDOBm6k_c5LLEu0,38532,palfrey,2022-09-28T16:54:22Z,2022-09-28T16:54:22Z,NONE,https://github.com/simonw/datasette-cors seems to workaround this,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1122427321,Index page `/` has no CORS headers, https://github.com/simonw/datasette/issues/1062#issuecomment-1260909128,https://api.github.com/repos/simonw/datasette/issues/1062,1260909128,IC_kwDOBm6k_c5LJ_JI,536941,fgregg,2022-09-28T13:22:53Z,2022-09-28T14:09:54Z,CONTRIBUTOR,"if you went this route: ```python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk ``` then `time_limit_ms` would probably have to be greatly extended, because the time spent in the loop will depend on the downstream processing. i wonder if this was why you were thinking this feature would need a dedicated connection? --- reading more, there's no real limit i can find on the number of active cursors (or more precisely active prepared statements objects, because sqlite doesn't really have cursors). maybe something like this would be okay? ```python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) # step through at least one to evaluate the statement, not sure if this is necessary yield c.execute.fetchone() for chunk in c.fetchmany(chunk_size): yield from chunk ``` this seems quite weird that there's not more of limit of the number of active prepared statements, but i haven't been able to find one. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows, https://github.com/simonw/datasette/issues/1062#issuecomment-1260829829,https://api.github.com/repos/simonw/datasette/issues/1062,1260829829,IC_kwDOBm6k_c5LJryF,536941,fgregg,2022-09-28T12:27:19Z,2022-09-28T12:27:19Z,CONTRIBUTOR,"for teaching `register_output_renderer` to stream it seems like the two options are to 1. a [nested query technique ](https://github.com/simonw/datasette/issues/526#issuecomment-505162238)to paginate through 2. a fetching model that looks like something ```python with sqlite_timelimit(conn, time_limit_ms): c.execute(query) for chunk in c.fetchmany(chunk_size): yield from chunk ``` currently `db.execute` is not a generator, so this would probably need a new method?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows, https://github.com/simonw/datasette/issues/1826#issuecomment-1260373403,https://api.github.com/repos/simonw/datasette/issues/1826,1260373403,IC_kwDOBm6k_c5LH8Wb,66709385,pjamargh,2022-09-28T04:30:27Z,2022-09-28T04:30:27Z,NONE,"I'm glad the bug report served some purpose. Frankly I just needed the method signature, that is why the documentation you mention wasn't read. On Tue, Sep 27, 2022, 9:05 PM Simon Willison ***@***.***> wrote: > Though now I notice that the copy right there needs to be updated to > reflect the new row parameter to render_cell! > > — > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1388631785,render_cell documentation example doesn't match the method signature, https://github.com/simonw/datasette/pull/1825#issuecomment-1260368537,https://api.github.com/repos/simonw/datasette/issues/1825,1260368537,IC_kwDOBm6k_c5LH7KZ,9599,simonw,2022-09-28T04:21:18Z,2022-09-28T04:21:18Z,OWNER,"This is great, thank you very much! https://datasette--1825.org.readthedocs.build/en/1825/deploying.html#running-datasette-using-openrc","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1388227245,Add documentation for serving via OpenRC, https://github.com/simonw/datasette/pull/1825#issuecomment-1260368122,https://api.github.com/repos/simonw/datasette/issues/1825,1260368122,IC_kwDOBm6k_c5LH7D6,22429695,codecov[bot],2022-09-28T04:20:28Z,2022-09-28T04:20:28Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report Base: **91.58**% // Head: **91.58**% // No change to project coverage :thumbsup: > Coverage data is based on head [(`b16eb2f`)](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) compared to base [(`5f9f567`)](https://codecov.io/gh/simonw/datasette/commit/5f9f567acbc58c9fcd88af440e68034510fb5d2b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > Patch has no changes to coverable lines. > :exclamation: Current head b16eb2f differs from pull request most recent head e7e96dc. Consider uploading reports for the commit e7e96dc to get more accurate results
      Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1825 +/- ## ======================================= Coverage 91.58% 91.58% ======================================= Files 36 36 Lines 4444 4444 ======================================= Hits 4070 4070 Misses 374 374 ``` Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)
      [:umbrella: View full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1825?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). :loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1388227245,Add documentation for serving via OpenRC, https://github.com/simonw/datasette/issues/1826#issuecomment-1260357878,https://api.github.com/repos/simonw/datasette/issues/1826,1260357878,IC_kwDOBm6k_c5LH4j2,9599,simonw,2022-09-28T04:05:45Z,2022-09-28T04:05:45Z,OWNER,Though now I notice that the copy right there needs to be updated to reflect the new `row` parameter to `render_cell`!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1388631785,render_cell documentation example doesn't match the method signature, https://github.com/simonw/datasette/issues/1826#issuecomment-1260357583,https://api.github.com/repos/simonw/datasette/issues/1826,1260357583,IC_kwDOBm6k_c5LH4fP,9599,simonw,2022-09-28T04:05:16Z,2022-09-28T04:05:16Z,OWNER,"This is deliberate. The Datasette plugin system allows you to specify only a subset of the parameters for a hook - in this example, only the `value` is needed so the others can be omitted. There's a note about this at the very top of that documentation page: https://docs.datasette.io/en/stable/plugin_hooks.html#plugin-hooks > When you implement a plugin hook you can accept any or all of the parameters that are documented as being passed to that hook. > > For example, you can implement the `render_cell` plugin hook like this even though the full documented hook signature is `render_cell(value, column, table, database, datasette)`: > ```python > @hookimpl > def render_cell(value, column): > if column == ""stars"": > return ""*"" * int(value) > ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1388631785,render_cell documentation example doesn't match the method signature, https://github.com/simonw/datasette/issues/526#issuecomment-1260355224,https://api.github.com/repos/simonw/datasette/issues/526,1260355224,IC_kwDOBm6k_c5LH36Y,9599,simonw,2022-09-28T04:01:25Z,2022-09-28T04:01:25Z,OWNER,"The ultimate protection against those memory bombs is to support more streaming output formats. Related issues: - #1177 - #1062","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1259718517,https://api.github.com/repos/simonw/datasette/issues/526,1259718517,IC_kwDOBm6k_c5LFcd1,536941,fgregg,2022-09-27T16:02:51Z,2022-09-27T16:04:46Z,CONTRIBUTOR,"i think that `max_returned_rows` **is** a defense mechanism, just not for connection exhaustion. `max_returned_rows` is a defense mechanism against **memory bombs**. if you are potentially yielding out hundreds of thousands or even millions of rows, you need to be quite careful about data flow to not run out of memory on the server, or on the client. you have a lot of places in your code that are protective of that right now, but `max_returned_rows` acts as the final backstop. so, given that, it makes sense to have removing `max_returned_rows` altogether be a non-goal, but instead allow for for specific codepaths (like streaming csv's) be able to bypass. that could dramatically lower the surface area for a memory-bomb attack.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1259693536,https://api.github.com/repos/simonw/datasette/issues/526,1259693536,IC_kwDOBm6k_c5LFWXg,9599,simonw,2022-09-27T15:42:55Z,2022-09-27T15:42:55Z,OWNER,"It's interesting to note WHY the time limit works against this so well. The time limit as-implemented looks like this: https://github.com/simonw/datasette/blob/5f9f567acbc58c9fcd88af440e68034510fb5d2b/datasette/utils/__init__.py#L181-L201 The key here is `conn.set_progress_handler(handler, n)` - which specifies that the handler function should be called every `n` SQLite operations. The handler function then checks to see if too much time has transpired and conditionally cancels the query. This also doubles up as a ""maximum number of operations"" guard, which is what's happening when you attempt to fetch an infinite number of rows from an infinite table. That limit code could even be extended to say ""exit the query after either 5s or 50,000,000 operations"". I don't think that's necessary though. To be honest I'm having trouble with the idea of dropping `max_returned_rows` mainly because what Datasette does (allow arbitrary untrusted SQL queries) is dangerous, so I've designed in multiple redundant defence-in-depth mechanisms right from the start.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258910228,https://api.github.com/repos/simonw/datasette/issues/526,1258910228,IC_kwDOBm6k_c5LCXIU,536941,fgregg,2022-09-27T03:11:07Z,2022-09-27T03:11:07Z,CONTRIBUTOR,"i think this feature would be safe, as its really only the time limit that can, and imo, should protect against long running queries, as it is pretty easy to make very expensive queries that don't return many rows. moving away from `max_returned_rows` will requires some thinking about: 1. memory usage and data flows to handle potentially very large result sets 2. how to avoid rendering tens or hundreds of thousands of [html rows](#1655).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258906440,https://api.github.com/repos/simonw/datasette/issues/526,1258906440,IC_kwDOBm6k_c5LCWNI,9599,simonw,2022-09-27T03:04:37Z,2022-09-27T03:04:37Z,OWNER,"It would be really neat if we could explore this idea in a plugin, but I don't think Datasette has plugin hooks in the right place for that at the moment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258905781,https://api.github.com/repos/simonw/datasette/issues/526,1258905781,IC_kwDOBm6k_c5LCWC1,9599,simonw,2022-09-27T03:03:35Z,2022-09-27T03:03:47Z,OWNER,"Yes good point, the time limit does already protect against that. I've been contemplating a permissioned-users-only relaxation of that time limit too, and I got that idea mixed up with this one in my head. On that basis maybe this feature would be safe after all? Would need to do some testing, but it may be that the existing time limit provides enough protection here already.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258878311,https://api.github.com/repos/simonw/datasette/issues/526,1258878311,IC_kwDOBm6k_c5LCPVn,536941,fgregg,2022-09-27T02:19:48Z,2022-09-27T02:19:48Z,CONTRIBUTOR,"this sql query doesn't trip up `maximum_returned_rows` but does timeout ```sql with recursive counter(x) as ( select 0 union select x + 1 from counter ) select * from counter LIMIT 10 OFFSET 100000000 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258871525,https://api.github.com/repos/simonw/datasette/issues/526,1258871525,IC_kwDOBm6k_c5LCNrl,536941,fgregg,2022-09-27T02:09:32Z,2022-09-27T02:14:53Z,CONTRIBUTOR,"thanks @simonw, i learned something i didn't know about sqlite's execution model! > Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever. why wouldn't the `sqlite_timelimit` guard prevent that? --- on my local version which has the code to [turn off truncations for query csv](#1820), `sqlite_timelimit` does protect me. ![Screenshot 2022-09-26 at 22-14-31 Error 500](https://user-images.githubusercontent.com/536941/192415680-94b32b7f-868f-4b89-8194-5752d45f6009.png) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258864140,https://api.github.com/repos/simonw/datasette/issues/526,1258864140,IC_kwDOBm6k_c5LCL4M,9599,simonw,2022-09-27T01:55:32Z,2022-09-27T01:55:32Z,OWNER,"That recursive query is a great example of the kind of thing having a maximum row limit protects against. Imagine if Datasette CSVs did allow unlimited retrievals. Someone could hit the CSV endpoint for that recursive query and tie up Datasette's SQL connection effectively forever. Even if this feature becomes a permission-guarded thing we still need to take that case into account. At the very least it would be good if the query could be cancelled if the client disconnects - so if someone accidentally starts an infinite query they can cancel the request and free up the server resources. It might be a good idea to implement a page that shows ""currently running"" queries and allows users with the right permission to terminate them from that page. Another option: a ""limit of last resource"" - either a very high row limit (10,000,000 perhaps) or even a time limit, saying that all queries will be cancelled if they take longer than thirty minutes or similar.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258860845,https://api.github.com/repos/simonw/datasette/issues/526,1258860845,IC_kwDOBm6k_c5LCLEt,9599,simonw,2022-09-27T01:48:31Z,2022-09-27T01:50:01Z,OWNER,"The protection is supposed to be from this line: ```python rows = cursor.fetchmany(max_returned_rows + 1) ``` By capping the call to `.fetchman()` at `max_returned_rows + 1` (the `+ 1` is to allow detection of whether or not there is a next page) I'm ensuring that Datasette never attempts to iterate over a huge result set. SQLite and the `sqlite3` library seem to handle this correctly. Here's an example: ```pycon >>> import sqlite3 >>> conn = sqlite3.connect("":memory:"") >>> cursor = conn.execute("""""" ... with recursive counter(x) as ( ... select 0 ... union ... select x + 1 from counter ... ) ... select * from counter"""""") >>> cursor.fetchmany(10) [(0,), (1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,), (10,)] ``` `counter` there is an infinitely long table ([see TIL](https://til.simonwillison.net/sqlite/simple-recursive-cte)) - but we can retrieve the first 10 results without going into an infinite loop. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258849766,https://api.github.com/repos/simonw/datasette/issues/526,1258849766,IC_kwDOBm6k_c5LCIXm,536941,fgregg,2022-09-27T01:27:03Z,2022-09-27T01:27:03Z,CONTRIBUTOR,"i agree with that concern! but if i'm understanding the code correctly, `maximum_returned_rows` does not protect against long-running queries in any way.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/issues/526#issuecomment-1258846992,https://api.github.com/repos/simonw/datasette/issues/526,1258846992,IC_kwDOBm6k_c5LCHsQ,9599,simonw,2022-09-27T01:21:41Z,2022-09-27T01:21:41Z,OWNER,"My main concern here is that public Datasette instances could easily have all of their available database connections consumed by long-running queries - either accidentally or deliberately. I do totally understand the need for this feature though. I think it can absolutely make sense provided it's protected by authentication and permissions. Maybe even limit the number of concurrent downloads at once such that there's always at least one database connection free for other requests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",459882902,Stream all results for arbitrary SQL and canned queries, https://github.com/simonw/datasette/pull/1823#issuecomment-1258828705,https://api.github.com/repos/simonw/datasette/issues/1823,1258828705,IC_kwDOBm6k_c5LCDOh,9599,simonw,2022-09-27T00:45:46Z,2022-09-27T00:45:46Z,OWNER,Also need to do a bit more of an audit to see if there is anywhere else that this style should be applied.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386917344,Keyword-only arguments for a bunch of internal methods, https://github.com/simonw/datasette/pull/1823#issuecomment-1258828509,https://api.github.com/repos/simonw/datasette/issues/1823,1258828509,IC_kwDOBm6k_c5LCDLd,9599,simonw,2022-09-27T00:45:26Z,2022-09-27T00:45:26Z,OWNER,I should update the documentation to reflect this change.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386917344,Keyword-only arguments for a bunch of internal methods, https://github.com/simonw/datasette/issues/1822#issuecomment-1258827688,https://api.github.com/repos/simonw/datasette/issues/1822,1258827688,IC_kwDOBm6k_c5LCC-o,9599,simonw,2022-09-27T00:44:04Z,2022-09-27T00:44:04Z,OWNER,I'll do this in a PR.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1386854246,Switch to keyword-only arguments for a bunch of internal methods, https://github.com/simonw/datasette/issues/1817#issuecomment-1258818028,https://api.github.com/repos/simonw/datasette/issues/1817,1258818028,IC_kwDOBm6k_c5LCAns,9599,simonw,2022-09-27T00:27:53Z,2022-09-27T00:27:53Z,OWNER,"Made a start on this: ```diff diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index 34e19664..fe0971e5 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -31,25 +31,29 @@ def prepare_jinja2_environment(env, datasette): @hookspec -def extra_css_urls(template, database, table, columns, view_name, request, datasette): +def extra_css_urls( + template, database, table, columns, sql, params, view_name, request, datasette +): """"""Extra CSS URLs added by this plugin"""""" @hookspec -def extra_js_urls(template, database, table, columns, view_name, request, datasette): +def extra_js_urls( + template, database, table, columns, sql, params, view_name, request, datasette +): """"""Extra JavaScript URLs added by this plugin"""""" @hookspec def extra_body_script( - template, database, table, columns, view_name, request, datasette + template, database, table, columns, sql, params, view_name, request, datasette ): """"""Extra JavaScript code to be included in