html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/datasette/issues/1939#issuecomment-1343734812,https://api.github.com/repos/simonw/datasette/issues/1939,1343734812,IC_kwDOBm6k_c5QF8Qc,9599,2022-12-09T01:57:07Z,2022-12-09T01:57:07Z,OWNER,"This search is better: datasette permission_allowed -user:simonw -path:datasette/** -path:docs/** -path:tests/** language:python That returns 11 results: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw+-path%3Adatasette%2F**+-path%3Adocs%2F**+-path%3Atests%2F**+language%3Apython 3 are forks of my repos. The rest are all by four users: - [20after4/ddd](https://github.com/20after4/ddd) - [emg110/datasette-graphql](https://github.com/emg110/datasette-graphql) - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer) - [next-LI/datasette-demo](https://github.com/next-LI/datasette-demo) - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config) - [next-LI/datasette-live-permissions](https://github.com/next-LI/datasette-live-permissions) - [next-LI/datasette-search-all](https://github.com/next-LI/datasette-search-all) - [next-LI/datasette-surveys](https://github.com/next-LI/datasette-surveys) - [next-LI/datasette-write](https://github.com/next-LI/datasette-write) - [rclement/datasette-dashboards](https://github.com/rclement/datasette-dashboards) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511, https://github.com/simonw/datasette/issues/1939#issuecomment-1343728929,https://api.github.com/repos/simonw/datasette/issues/1939,1343728929,IC_kwDOBm6k_c5QF60h,9599,2022-12-09T01:48:11Z,2022-12-09T01:52:33Z,OWNER,"This code search shows a bunch of repos I don't know about that would be affected by this change: https://cs.github.com/?scopeName=All+repos&scope=&q=datasette+permission_allowed+-user%3Asimonw# These (and likely more): Repositories - [20after4/ddd](https://github.com/20after4/ddd) - [next-LI/datasette-csv-importer](https://github.com/next-LI/datasette-csv-importer) - [digital-land/datasette](https://github.com/digital-land/datasette) - [mroswell/datasette](https://github.com/mroswell/datasette) - [next-LI/datasette-live-config](https://github.com/next-LI/datasette-live-config) - [keladhruv/datasette](https://github.com/keladhruv/datasette) - [RhetTbull/datasette](https://github.com/RhetTbull/datasette) - [chriswedgwood/datasette](https://github.com/chriswedgwood/datasette) - [boan-anbo/datasette](https://github.com/boan-anbo/datasette) - [MattTriano/datasette](https://github.com/MattTriano/datasette) - [incadenza/datasette](https://github.com/incadenza/datasette) - [robdyke/datasette](https://github.com/robdyke/datasette) - [ctb/datasette](https://github.com/ctb/datasette) - [eyeseast/datasette](https://github.com/eyeseast/datasette) - [symbol-management/api-match-audit](https://github.com/symbol-management/api-match-audit) Actually a lot of those are forks of Datasette itself - so maybe this is manageable? Would be nice if I could come up with a GitHub search that excluded any repos with ""datasette"" as their exact name. https://docs.github.com/en/search-github/github-code-search/understanding-github-code-search-syntax#using-qualifiers says: > **Note:** The new code search beta does not currently support regular expressions or partial matching for repository names, so you will have to type the entire repository name (including the user prefix) for the `repo:` qualifier to work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511, https://github.com/simonw/datasette/issues/1939#issuecomment-1343727184,https://api.github.com/repos/simonw/datasette/issues/1939,1343727184,IC_kwDOBm6k_c5QF6ZQ,9599,2022-12-09T01:45:15Z,2022-12-09T01:45:15Z,OWNER,"Moving the concept of the default for the permission into this registry warrants a redesign of this method anyway: https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/datasette/app.py#L706","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511, https://github.com/simonw/datasette/issues/1939#issuecomment-1343724732,https://api.github.com/repos/simonw/datasette/issues/1939,1343724732,IC_kwDOBm6k_c5QF5y8,9599,2022-12-09T01:40:44Z,2022-12-09T01:43:25Z,OWNER,"```python Permission = collections.namedtuple( ""Permission"", (""name"", ""abbr"", ""takes_database"", ""takes_table"", ""default"") ) ``` I don`t think that design is quite right. - Elsewhere in the code the concept is called an ""action"" rather than a ""permission"" - I think I can stick with the `Permission` name here though, it's pretty clear - `takes_database` - is `takes_` the right verb here? - `takes_table` can also refer to a SQL view or a canned named query A question that was raised by the work in #1938 is whether you should be able to grant a permission like `insert-row` at the instance or database level - and if so, what does that look like? I think you should be able to do that, it doesn't make sense to have to grant it explicitly for every single table. So maybe `takes_table` and `takes_database` are the right names here? But `table` is still bad because it doesn't reflect views and canned queries. One thought is to use `resource` - but that will require a bunch of breaking changes to the existing APIs which treat resource as a tuple. Now's the best time to do that though before Datasette 1.0.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511, https://github.com/simonw/datasette/issues/1881#issuecomment-1301645921,https://api.github.com/repos/simonw/datasette/issues/1881,1301645921,IC_kwDOBm6k_c5NlYph,9599,2022-11-03T05:10:05Z,2022-12-09T01:38:21Z,OWNER,"I'd love to come up with a good short name for the second part of the resource two-tuple, the thing which is usually the name of a table but could also be the name of a SQL view or the name of a canned query. Idea 8th December: why not call it resource? A resource could be a thing that lives inside a database.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1434094365, https://github.com/simonw/datasette/issues/1939#issuecomment-1343722020,https://api.github.com/repos/simonw/datasette/issues/1939,1343722020,IC_kwDOBm6k_c5QF5Ik,9599,2022-12-09T01:36:05Z,2022-12-09T01:36:16Z,OWNER,"I originally added `permissions.py` for the permission debug tool in https://github.com/simonw/datasette/commit/c51d9246b996a2831c9bd6a1e205f6cb48b9a5f3 - I don't think anything else uses it yet. - #1881","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511, https://github.com/simonw/datasette/issues/1939#issuecomment-1343721522,https://api.github.com/repos/simonw/datasette/issues/1939,1343721522,IC_kwDOBm6k_c5QF5Ay,9599,2022-12-09T01:35:15Z,2022-12-09T01:35:15Z,OWNER,"One concern I have about this: there are a bunch of existing plugins that do stuff with permissions that won't currently be using this hook. Do I break those plugins, forcing new releases of them for compatibility with Datasette 1.0? Or maybe I keep them working, but until they've upgraded to register their permissions there are things about them that won't work - e.g. you won't be able to configure their permissions in `metadata.yml` until they release something that does this hook. Best thing is probably for me to get this working in core first and then evaluate the impact it would have on existing plugins once I have some running code.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485757511, https://github.com/simonw/datasette/issues/1636#issuecomment-1343715746,https://api.github.com/repos/simonw/datasette/issues/1636,1343715746,IC_kwDOBm6k_c5QF3mi,9599,2022-12-09T01:27:41Z,2022-12-09T01:27:58Z,OWNER,"I may need to consult this file to figure out if the permission that is being checked can act at the database/table/instance level: https://github.com/simonw/datasette/blob/e539c1c024bc62d88df91d9107cbe37e7f0fe55f/datasette/permissions.py#L1-L19","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/issues/1636#issuecomment-1343446071,https://api.github.com/repos/simonw/datasette/issues/1636,1343446071,IC_kwDOBm6k_c5QE1w3,9599,2022-12-08T22:16:17Z,2022-12-08T22:16:17Z,OWNER,First draft of documentation: https://datasette--1938.org.readthedocs.build/en/1938/authentication.html#other-permissions-in-metadata,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/pull/1938#issuecomment-1343445885,https://api.github.com/repos/simonw/datasette/issues/1938,1343445885,IC_kwDOBm6k_c5QE1t9,9599,2022-12-08T22:16:03Z,2022-12-08T22:16:03Z,OWNER,Docs: https://datasette--1938.org.readthedocs.build/en/1938/authentication.html#other-permissions-in-metadata,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1485488236, https://github.com/simonw/datasette/issues/1636#issuecomment-1343440504,https://api.github.com/repos/simonw/datasette/issues/1636,1343440504,IC_kwDOBm6k_c5QE0Z4,9599,2022-12-08T22:10:28Z,2022-12-08T22:10:48Z,OWNER,"What if you want to grant `insert-row` to a user for ALL tables in a database, or even for all tables in all databases? You should be able to do that by putting that in the root `permissions:` block. Need to figure out how the implementation will handle that. Also: there are some permissions like `view-instance` or `debug-menu` for which putting them at the `database` or `table` or `query` level doesn't actually make any sense. Ideally the implementation would spot those on startup and refuse to start the server, with a helpful error message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/pull/1930#issuecomment-1343360006,https://api.github.com/repos/simonw/datasette/issues/1930,1343360006,IC_kwDOBm6k_c5QEgwG,9599,2022-12-08T21:12:28Z,2022-12-08T21:12:28Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473664029, https://github.com/simonw/datasette/issues/1867#issuecomment-1341874378,https://api.github.com/repos/simonw/datasette/issues/1867,1341874378,IC_kwDOBm6k_c5P-2DK,9599,2022-12-08T02:07:06Z,2022-12-08T02:28:02Z,OWNER,"Basic version of this allows you to rename a table: ``` POST /db/table/-/rename { ""name"": ""new_table_name"" } ``` If a table with that new name already exists you will get an error - unless you pass `""replace"": true` in which case that table will be dropped and replaced by the new one. ``` POST /db/table/-/rename { ""name"": ""new_table_name"", ""replace"": true } ``` This is useful because it allows for that atomic replacement operation: upload brand new data into a `_tmp_name` table, then atomic rename and replace after the upload has completed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1426080014, https://github.com/simonw/datasette/issues/1636#issuecomment-1341854373,https://api.github.com/repos/simonw/datasette/issues/1636,1341854373,IC_kwDOBm6k_c5P-xKl,9599,2022-12-08T01:43:35Z,2022-12-08T01:43:35Z,OWNER,I'm going to write the documentation for this first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/issues/1936#issuecomment-1341849735,https://api.github.com/repos/simonw/datasette/issues/1936,1341849735,IC_kwDOBm6k_c5P-wCH,9599,2022-12-08T01:35:54Z,2022-12-08T01:35:54Z,OWNER,"Running that twice produced this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483250004, https://github.com/simonw/datasette/issues/1936#issuecomment-1341849496,https://api.github.com/repos/simonw/datasette/issues/1936,1341849496,IC_kwDOBm6k_c5P-v-Y,9599,2022-12-08T01:35:35Z,2022-12-08T01:35:35Z,OWNER,"Related bug: you can send `""id"": null` and it works (it should throw an error): ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483250004, https://github.com/simonw/datasette/issues/1937#issuecomment-1341848525,https://api.github.com/repos/simonw/datasette/issues/1937,1341848525,IC_kwDOBm6k_c5P-vvN,9599,2022-12-08T01:34:03Z,2022-12-08T01:34:03Z,OWNER,Check should go somewhere around here: https://github.com/simonw/datasette/blob/dee18ed8ce7af2ab8699bcb5a51a99f48301bc42/datasette/views/database.py#L625-L631,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1483320357, https://github.com/simonw/datasette/pull/1931#issuecomment-1341825314,https://api.github.com/repos/simonw/datasette/issues/1931,1341825314,IC_kwDOBm6k_c5P-qEi,9599,2022-12-08T01:03:18Z,2022-12-08T01:03:18Z,OWNER,"I broke this test: ``` ds_write = @pytest.mark.asyncio async def test_insert_row(ds_write): token = write_token(ds_write) response = await ds_write.client.post( ""/data/docs/-/insert"", json={""row"": {""title"": ""Test"", ""score"": 1.2, ""age"": 5}}, headers={ ""Authorization"": ""***"".format(token), ""Content-Type"": ""application/json"", }, ) expected_row = {""id"": 1, ""title"": ""Test"", ""score"": 1.2, ""age"": 5} > assert response.status_code == 201 E assert 500 == 201 E + where 500 = .status_code /home/runner/work/datasette/datasette/tests/test_api_write.py:43: AssertionError ----------------------------- Captured stderr call ----------------------------- Traceback (most recent call last): File ""/home/runner/work/datasette/datasette/datasette/app.py"", line 1447, in route_path response = await view(request, send) File ""/home/runner/work/datasette/datasette/datasette/views/base.py"", line 151, in view return await self.dispatch_request(request) File ""/home/runner/work/datasette/datasette/datasette/views/base.py"", line 105, in dispatch_request response = await handler(request) File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in post row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] File ""/home/runner/work/datasette/datasette/datasette/views/table.py"", line 1228, in row_pk_values_for_later = [tuple(row[pk] for pk in pks) for row in rows] KeyError: 'id' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539, https://github.com/simonw/datasette/pull/1931#issuecomment-1341821213,https://api.github.com/repos/simonw/datasette/issues/1931,1341821213,IC_kwDOBm6k_c5P-pEd,9599,2022-12-08T00:58:21Z,2022-12-08T00:58:21Z,OWNER,"In the interests of shipping, I'm going to punt the API explorer to a later issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539, https://github.com/simonw/datasette/pull/1931#issuecomment-1339968514,https://api.github.com/repos/simonw/datasette/issues/1931,1339968514,IC_kwDOBm6k_c5P3kwC,9599,2022-12-06T20:28:47Z,2022-12-06T20:28:47Z,OWNER,"Should the `""return"": true` mode reflect the order in which the rows were provided when the API was called? I think it should. Since this is small enough to happily fit in Python memory (thanks to the `max_insert_rows` setting) I can load the fresh data from the database and then sort it in Python space before returning it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539, https://github.com/simonw/datasette/issues/1855#issuecomment-1339952692,https://api.github.com/repos/simonw/datasette/issues/1855,1339952692,IC_kwDOBm6k_c5P3g40,9599,2022-12-06T20:15:50Z,2022-12-06T20:16:00Z,OWNER,"That commit there https://github.com/simonw/datasette/commit/6da17d5529773dfe41b53ed4ce5a6ecb46ed2457 (which will be squash-merged in a PR later on) made it so that `_r` was correctly copied across from the token to the created actor, and fixed a bug in the code that checks permissions against it for resources. I needed that mechanism to write a test that exercised different API permissions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089, https://github.com/simonw/datasette/pull/1931#issuecomment-1339911152,https://api.github.com/repos/simonw/datasette/issues/1931,1339911152,IC_kwDOBm6k_c5P3Wvw,9599,2022-12-06T19:38:12Z,2022-12-06T19:38:12Z,OWNER,Documentation: https://datasette--1931.org.readthedocs.build/en/1931/json_api.html#upserting-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539, https://github.com/simonw/datasette/issues/1927#issuecomment-1339910494,https://api.github.com/repos/simonw/datasette/issues/1927,1339910494,IC_kwDOBm6k_c5P3Wle,9599,2022-12-06T19:37:39Z,2022-12-06T19:37:39Z,OWNER,"I'll finish this after I land: - #1931 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197, https://github.com/simonw/datasette/issues/1929#issuecomment-1339909159,https://api.github.com/repos/simonw/datasette/issues/1929,1339909159,IC_kwDOBm6k_c5P3WQn,9599,2022-12-06T19:36:23Z,2022-12-06T19:36:23Z,OWNER,https://docs.datasette.io/en/1.0a1/json_api.html đź‘Ť ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191, https://github.com/simonw/datasette/issues/1929#issuecomment-1339871933,https://api.github.com/repos/simonw/datasette/issues/1929,1339871933,IC_kwDOBm6k_c5P3NK9,9599,2022-12-06T19:23:48Z,2022-12-06T19:24:17Z,OWNER,"I can do that on this page: https://readthedocs.org/projects/datasette/versions/?version_filter=1.0 I'm making them both active and hidden: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191, https://github.com/simonw/datasette/issues/1929#issuecomment-1339867570,https://api.github.com/repos/simonw/datasette/issues/1929,1339867570,IC_kwDOBm6k_c5P3MGy,9599,2022-12-06T19:22:47Z,2022-12-06T19:22:47Z,OWNER,"Yeah I should deploy the docs for the alpha versions (the intention is that the docs exactly match the release you are using), thanks for spotting that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473659191, https://github.com/simonw/sqlite-utils/issues/516#issuecomment-1339834918,https://api.github.com/repos/simonw/sqlite-utils/issues/516,1339834918,IC_kwDOCGYnMM5P3EIm,9599,2022-12-06T19:00:18Z,2022-12-06T19:00:35Z,OWNER,"Right now the command produces no output at all. Maybe a `--verbose` mode that writes these numbers to standard error (or even standard output since it's an option)? Is there a better name than `--verbose` for this? `--summary` perhaps?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1479914599, https://github.com/simonw/datasette/pull/1931#issuecomment-1339784569,https://api.github.com/repos/simonw/datasette/issues/1931,1339784569,IC_kwDOBm6k_c5P2315,9599,2022-12-06T18:16:15Z,2022-12-06T18:17:56Z,OWNER,"Just noticed the insert API returns `{}` when it should return `{""ok"": true}` - will fix that here too. UPDATE: no it did that already, it was just the documentation that was wrong.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539, https://github.com/simonw/datasette/pull/1931#issuecomment-1339768422,https://api.github.com/repos/simonw/datasette/issues/1931,1339768422,IC_kwDOBm6k_c5P2z5m,9599,2022-12-06T18:04:59Z,2022-12-06T18:04:59Z,OWNER,"I realized this API should require both the `insert-row` AND the `update-row` permissions, since calls to it could do either one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473814539, https://github.com/simonw/datasette/issues/1855#issuecomment-1302815929,https://api.github.com/repos/simonw/datasette/issues/1855,1302815929,IC_kwDOBm6k_c5Np2S5,9599,2022-11-04T00:19:10Z,2022-12-03T07:05:51Z,OWNER,Added the tests.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1423336089, https://github.com/simonw/datasette/issues/1878#issuecomment-1336100218,https://api.github.com/repos/simonw/datasette/issues/1878,1336100218,IC_kwDOBm6k_c5Po0V6,9599,2022-12-03T07:02:15Z,2022-12-03T07:02:15Z,OWNER,"Moved this work to a PR: - #1931","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099588,https://api.github.com/repos/simonw/datasette/issues/1927,1336099588,IC_kwDOBm6k_c5Po0ME,9599,2022-12-03T06:58:14Z,2022-12-03T06:58:14Z,OWNER,I have not yet documented the new `insert` and `replace` options.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099533,https://api.github.com/repos/simonw/datasette/issues/1927,1336099533,IC_kwDOBm6k_c5Po0LN,9599,2022-12-03T06:57:52Z,2022-12-03T06:57:52Z,OWNER,I'm going to push what I have anyway. I'll keep this issue open while I think through the above comment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197, https://github.com/simonw/datasette/issues/1927#issuecomment-1336099368,https://api.github.com/repos/simonw/datasette/issues/1927,1336099368,IC_kwDOBm6k_c5Po0Io,9599,2022-12-03T06:56:36Z,2022-12-03T06:56:36Z,OWNER,"Neither of these options make sense if you didn't pass a `""pk""`. My initial implementation spotted if the `pk` was missing and looked it up from the table, but actually I don't think that makes sense - if you know the table exists and hence don't pass the `pk` you should be using `/-/insert` or `/-/upsert` instead. So maybe this work should expanded to include validation that checks if the table exists already - and if it does, confirms that the primary key (and maybe even the columns) are the same as for that existing table. Of course if you only send `row` or `rows` then checking `columns` doesn't completely make sense - but we could check that the rows you have sent are equal to or a subset of the columns in the table. We could even check the column types as well, as seen in: - #1910 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094562,https://api.github.com/repos/simonw/datasette/issues/1878,1336094562,IC_kwDOBm6k_c5Poy9i,9599,2022-12-03T06:27:50Z,2022-12-03T06:29:06Z,OWNER,"This adds it to the API explorer: ```diff diff --git a/datasette/views/special.py b/datasette/views/special.py index 1f84b094..1b4a9d3c 100644 --- a/datasette/views/special.py +++ b/datasette/views/special.py @@ -316,21 +316,37 @@ class ApiExplorerView(BaseView): request.actor, ""insert-row"", (name, table) ): pks = await db.primary_keys(table) - table_links.append( - { - ""path"": self.ds.urls.table(name, table) + ""/-/insert"", - ""method"": ""POST"", - ""label"": ""Insert rows into {}"".format(table), - ""json"": { - ""rows"": [ - { - column: None - for column in await db.table_columns(table) - if column not in pks - } - ] + table_links.extend( + [ + { + ""path"": self.ds.urls.table(name, table) + ""/-/insert"", + ""method"": ""POST"", + ""label"": ""Insert rows into {}"".format(table), + ""json"": { + ""rows"": [ + { + column: None + for column in await db.table_columns(table) + if column not in pks + } + ] + }, }, - } + { + ""path"": self.ds.urls.table(name, table) + ""/-/upsert"", + ""method"": ""POST"", + ""label"": ""Upsert rows into {}"".format(table), + ""json"": { + ""rows"": [ + { + column: None + for column in await db.table_columns(table) + if column not in pks + } + ] + }, + }, + ] ) if await self.ds.permission_allowed( request.actor, ""drop-table"", (name, table) ``` Except it doesn't quite, because the example JSON this generates is invalid as it does not include the primary key column. (Made me notice that the way example columns are created for `/-/insert` will fail for tables that don't have an auto-incrementing primary key)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094470,https://api.github.com/repos/simonw/datasette/issues/1878,1336094470,IC_kwDOBm6k_c5Poy8G,9599,2022-12-03T06:27:13Z,2022-12-03T06:27:13Z,OWNER,"Tests are going to need to cover both rowid-only and compound primary key tables, including all of the error states.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704, https://github.com/simonw/datasette/issues/1878#issuecomment-1336094381,https://api.github.com/repos/simonw/datasette/issues/1878,1336094381,IC_kwDOBm6k_c5Poy6t,9599,2022-12-03T06:26:25Z,2022-12-03T06:26:25Z,OWNER,"Initial prototype: ```diff diff --git a/datasette/app.py b/datasette/app.py index 125b4969..282c0984 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -40,7 +40,7 @@ from .views.special import ( PermissionsDebugView, MessagesDebugView, ) -from .views.table import TableView, TableInsertView, TableDropView +from .views.table import TableView, TableInsertView, TableUpsertView, TableDropView from .views.row import RowView, RowDeleteView, RowUpdateView from .renderer import json_renderer from .url_builder import Urls @@ -1292,6 +1292,10 @@ class Datasette: TableInsertView.as_view(self), r""/(?P[^\/\.]+)/(?P[^\/\.]+)/-/insert$"", ) + add_route( + TableUpsertView.as_view(self), + r""/(?P[^\/\.]+)/(?P
[^\/\.]+)/-/upsert$"", + ) add_route( TableDropView.as_view(self), r""/(?P[^\/\.]+)/(?P
[^\/\.]+)/-/drop$"", diff --git a/datasette/views/table.py b/datasette/views/table.py index 7ba78c11..ae0d6366 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -1074,9 +1074,15 @@ class TableInsertView(BaseView): def __init__(self, datasette): self.ds = datasette - async def _validate_data(self, request, db, table_name): + async def _validate_data(self, request, db, table_name, pks, upsert): errors = [] + pks_list = [] + if isinstance(pks, str): + pks_list = [pks] + else: + pks_list = list(pks) + def _errors(errors): return None, errors, {} @@ -1135,6 +1141,15 @@ class TableInsertView(BaseView): # Validate columns of each row columns = set(await db.table_columns(table_name)) for i, row in enumerate(rows): + if upsert: + # It MUST have the primary key + missing_pks = [pk for pk in pks_list if pk not in row] + if missing_pks: + errors.append( + 'Row {} is missing primary key column(s): ""{}""'.format( + i, '"", ""'.join(missing_pks) + ) + ) invalid_columns = set(row.keys()) - columns if invalid_columns: errors.append( @@ -1146,7 +1161,7 @@ class TableInsertView(BaseView): return _errors(errors) return rows, errors, extras - async def post(self, request): + async def post(self, request, upsert=False): try: resolved = await self.ds.resolve_table(request) except NotFound as e: @@ -1164,7 +1179,12 @@ class TableInsertView(BaseView): request.actor, ""insert-row"", resource=(database_name, table_name) ): return _error([""Permission denied""], 403) - rows, errors, extras = await self._validate_data(request, db, table_name) + + pks = await db.primary_keys(table_name) + + rows, errors, extras = await self._validate_data( + request, db, table_name, pks, upsert + ) if errors: return _error(errors, 400) @@ -1172,15 +1192,19 @@ class TableInsertView(BaseView): replace = extras.get(""replace"") should_return = bool(extras.get(""return"", False)) - # Insert rows - def insert_rows(conn): + + def insert_or_upsert_rows(conn): table = sqlite_utils.Database(conn)[table_name] + kwargs = {} + if upsert: + kwargs[""pk""] = pks[0] if len(pks) == 1 else pks + else: + kwargs = {""ignore"": ignore, ""replace"": replace} if should_return: rowids = [] + method = table.upsert if upsert else table.insert for row in rows: - rowids.append( - table.insert(row, ignore=ignore, replace=replace).last_rowid - ) + rowids.append(method(row, **kwargs).last_rowid) return list( table.rows_where( ""rowid in ({})"".format("","".join(""?"" for _ in rowids)), @@ -1188,10 +1212,11 @@ class TableInsertView(BaseView): ) ) else: - table.insert_all(rows, ignore=ignore, replace=replace) + method_all = table.upsert_all if upsert else table.insert_all + method_all(rows, **kwargs) try: - rows = await db.execute_write_fn(insert_rows) + rows = await db.execute_write_fn(insert_or_upsert_rows) except Exception as e: return _error([str(e)]) result = {""ok"": True} @@ -1200,6 +1225,13 @@ class TableInsertView(BaseView): return Response.json(result, status=201) +class TableUpsertView(TableInsertView): + name = ""table-upsert"" + + async def post(self, request): + return await super().post(request, upsert=True) + + class TableDropView(BaseView): name = ""table-drop"" ``` Manual testing reveals that this mostly works... but it's not doing the right thing for `""return"": true` - it always returns an empty list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704, https://github.com/simonw/datasette/issues/1878#issuecomment-1336073212,https://api.github.com/repos/simonw/datasette/issues/1878,1336073212,IC_kwDOBm6k_c5Potv8,9599,2022-12-03T05:38:49Z,2022-12-03T05:38:49Z,OWNER,And on Discord today: https://discord.com/channels/823971286308356157/823971286941302908/1048426072066236536,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704, https://github.com/simonw/datasette/issues/1878#issuecomment-1336070843,https://api.github.com/repos/simonw/datasette/issues/1878,1336070843,IC_kwDOBm6k_c5PotK7,9599,2022-12-03T05:37:53Z,2022-12-03T05:37:53Z,OWNER,Also requested here: https://news.ycombinator.com/item?id=33839894,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432013704, https://github.com/simonw/datasette/issues/1927#issuecomment-1335984268,https://api.github.com/repos/simonw/datasette/issues/1927,1335984268,IC_kwDOBm6k_c5PoYCM,9599,2022-12-03T00:26:26Z,2022-12-03T00:26:26Z,OWNER,"Also: the documentation should clarify that you can call this API multiple times when using the `rows` option. (It will probably grow `""alter"": true` soon too).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473411197, https://github.com/simonw/datasette/issues/1928#issuecomment-1335966329,https://api.github.com/repos/simonw/datasette/issues/1928,1335966329,IC_kwDOBm6k_c5PoTp5,9599,2022-12-02T23:47:11Z,2022-12-02T23:47:11Z,OWNER,Wrote about this extensively here: https://simonwillison.net/2022/Dec/2/datasette-write-api/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870889,https://api.github.com/repos/simonw/datasette/issues/1928,1335870889,IC_kwDOBm6k_c5Pn8Wp,9599,2022-12-02T21:41:09Z,2022-12-02T21:41:09Z,OWNER,"Got it working! https://simon.datasette.cloud/data/hacker_news_posts https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/submit-to-datasette-cloud.sh","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870887,https://api.github.com/repos/simonw/datasette/issues/1928,1335870887,IC_kwDOBm6k_c5Pn8Wn,9599,2022-12-02T21:25:16Z,2022-12-02T21:25:16Z,OWNER,"Here's the change that should submit data to Datasette Cloud: https://github.com/simonw/scrape-hacker-news-by-domain/commit/848bb7e835a9fb87cd656362591835179cd1dc1b I ran `delete from hacker_news_posts` against my instance so https://simon.datasette.cloud/data/hacker_news_posts is now empty.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870884,https://api.github.com/repos/simonw/datasette/issues/1928,1335870884,IC_kwDOBm6k_c5Pn8Wk,9599,2022-12-02T21:19:58Z,2022-12-02T21:19:58Z,OWNER,"But until I fix this issue: - https://github.com/simonw/datasette/issues/1927 I need to insert freshly scraped data like this: ```bash export ROWS=$( jq -n --argjson rows ""$(cat simonwillison-net.json)"" \ '{ ""rows"": $rows, ""replace"": true }' ) curl -X POST \ https://simon.datasette.cloud/data/hacker_news_posts/-/insert \ -H ""Content-Type: application/json"" \ -H ""Authorization: Bearer $DS_TOKEN"" \ -d $ROWS ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870883,https://api.github.com/repos/simonw/datasette/issues/1928,1335870883,IC_kwDOBm6k_c5Pn8Wj,9599,2022-12-02T21:19:10Z,2022-12-02T21:19:10Z,OWNER,"I created the `hacker_news_posts` table like this: ```bash export ROWS=$( jq -n --argjson rows ""$(cat simonwillison-net.json)"" \ '{ ""table"": ""hacker_news_posts"", ""rows"": $rows, ""pk"": ""id"", ""replace"": true }' ) # Use curl to POST some JSON to a URL curl -X POST \ https://simon.datasette.cloud/data/-/create \ -H ""Content-Type: application/json"" \ -H ""Authorization: Bearer $DS_TOKEN"" \ -d $ROWS ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870879,https://api.github.com/repos/simonw/datasette/issues/1928,1335870879,IC_kwDOBm6k_c5Pn8Wf,9599,2022-12-02T21:18:25Z,2022-12-02T21:18:25Z,OWNER,"This is the SQL view for the atom feed: ```sql CREATE VIEW hacker_news_posts_atom as select id as atom_id, title as atom_title, url, commentsUrl as atom_link, dt || 'Z' as atom_updated, 'Submitter: ' || submitter || ' - ' || points || ' points, ' || numComments || ' comments' as atom_content from hacker_news_posts order by dt desc limit 100; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262, https://github.com/simonw/datasette/issues/1928#issuecomment-1335870877,https://api.github.com/repos/simonw/datasette/issues/1928,1335870877,IC_kwDOBm6k_c5Pn8Wd,9599,2022-12-02T21:17:50Z,2022-12-02T21:17:50Z,OWNER,https://simon.datasette.cloud/data/hacker_news_posts_atom.atom is working now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1473481262, https://github.com/simonw/datasette/issues/1636#issuecomment-1334759315,https://api.github.com/repos/simonw/datasette/issues/1636,1334759315,IC_kwDOBm6k_c5Pjs-T,9599,2022-12-02T04:46:32Z,2022-12-02T04:46:32Z,OWNER,"Thankfully all of the logic for this already lives in just one place: https://github.com/simonw/datasette/blob/d7e5e3c9f98d194fdfb12f1ecc60ed5b3afbc464/datasette/default_permissions.py#L23-L59","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/issues/1636#issuecomment-1334758766,https://api.github.com/repos/simonw/datasette/issues/1636,1334758766,IC_kwDOBm6k_c5Pjs1u,9599,2022-12-02T04:45:16Z,2022-12-02T04:45:16Z,OWNER,"Also, this is another thing which should live in `config.yml` rather than being crammed into `metadata.yml` - but I can fix that when I address: - #493","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/issues/1636#issuecomment-1334757597,https://api.github.com/repos/simonw/datasette/issues/1636,1334757597,IC_kwDOBm6k_c5Pjsjd,9599,2022-12-02T04:42:35Z,2022-12-02T04:42:35Z,OWNER,"Should I call this key `permissions` or something else? Some options: - `permissions` - `perms` - shorter to type - `allow` - I like the word, but might be confusing to change its meaning since we use it already","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/issues/1636#issuecomment-1334673179,https://api.github.com/repos/simonw/datasette/issues/1636,1334673179,IC_kwDOBm6k_c5PjX8b,9599,2022-12-02T02:07:20Z,2022-12-02T04:27:07Z,OWNER,"So the new mechanism needs to extend that to handle all of the other permissions as well. The simplest design I can think of is this (here illustrated using YAML): ```yaml # instance-level permissions - give every logged in user the debug menu: permissions: debug-menu: id: * databases: content: # Allow bob to create-table in the content database permissions: create-table: id: bob ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/issues/1636#issuecomment-1334666806,https://api.github.com/repos/simonw/datasette/issues/1636,1334666806,IC_kwDOBm6k_c5PjWY2,9599,2022-12-02T01:58:40Z,2022-12-02T02:00:53Z,OWNER,"Current design: ```json { ""databases"": { ""private"": { ""allow"": { ""id"": ""*"" } } } } ``` This can be applied at the instance, database, table or query level within the nested JSON. https://docs.datasette.io/en/stable/authentication.html#controlling-access-to-specific-databases It's actually controlling the following permissions: - `view-instance` - `view-database` - `view-table` - `view-query` There's also a special case for allowing SQL queries,at the instance and database level: ```json { ""databases"": { ""mydatabase"": { ""allow_sql"": { ""id"": ""root"" } } } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1138008042, https://github.com/simonw/datasette/issues/1926#issuecomment-1334508062,https://api.github.com/repos/simonw/datasette/issues/1926,1334508062,IC_kwDOBm6k_c5Pivoe,9599,2022-12-01T22:06:12Z,2022-12-01T22:06:12Z,OWNER,Released: https://pypi.org/project/datasette/1.0a1/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1471969984, https://github.com/simonw/datasette/issues/1924#issuecomment-1334165225,https://api.github.com/repos/simonw/datasette/issues/1924,1334165225,IC_kwDOBm6k_c5Phb7p,9599,2022-12-01T18:15:15Z,2022-12-01T18:15:15Z,OWNER,Updated docs at the bottom of this section: https://docs.datasette.io/en/latest/json_api.html#inserting-rows,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936, https://github.com/simonw/datasette/issues/1924#issuecomment-1333042785,https://api.github.com/repos/simonw/datasette/issues/1924,1333042785,IC_kwDOBm6k_c5PdJ5h,9599,2022-12-01T02:00:06Z,2022-12-01T02:00:06Z,OWNER,"Looks like I did implement these already: https://github.com/simonw/datasette/blob/9a1536b52a07e32da5900652da1bd7894c58fa9f/tests/test_api_write.py#L270-L273 But forgot to document them!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936, https://github.com/simonw/datasette/issues/1924#issuecomment-1333025076,https://api.github.com/repos/simonw/datasette/issues/1924,1333025076,IC_kwDOBm6k_c5PdFk0,9599,2022-12-01T01:37:09Z,2022-12-01T01:37:09Z,OWNER,"Related question: what happens if you attempt to insert rows without either of these settings and 1:10 of them is an integrity error due to an existing primary key? Does the entire batch fail to be inserted? Should it? This may be the point that I need to think hard about how to use transactions here. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936, https://github.com/simonw/datasette/issues/1924#issuecomment-1333023273,https://api.github.com/repos/simonw/datasette/issues/1924,1333023273,IC_kwDOBm6k_c5PdFIp,9599,2022-12-01T01:34:48Z,2022-12-01T01:34:48Z,OWNER,"This will enable some very cool Datasette write API demos - for example, Git scrapers that insert-replace the most recent copy of the scraped data to a table somewhere (which can then produce an atom feed with https://datasette.io/plugins/datasette-atom)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470509936, https://github.com/simonw/datasette/issues/1922#issuecomment-1332903011,https://api.github.com/repos/simonw/datasette/issues/1922,1332903011,IC_kwDOBm6k_c5Pcnxj,9599,2022-11-30T23:45:29Z,2022-11-30T23:45:29Z,OWNER,"That worked for the preflight request - got this now: So it looks like error responses (in this case for permission denied) are missing CORS headers.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332855687,https://api.github.com/repos/simonw/datasette/issues/1922,1332855687,IC_kwDOBm6k_c5PccOH,9599,2022-11-30T23:09:31Z,2022-11-30T23:09:31Z,OWNER,"Still getting a CORS header. I tried it in Chrome, which outputs helpful messages to the console: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1923#issuecomment-1332851215,https://api.github.com/repos/simonw/datasette/issues/1923,1332851215,IC_kwDOBm6k_c5PcbIP,9599,2022-11-30T23:04:56Z,2022-11-30T23:04:56Z,OWNER,That fixed it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227, https://github.com/simonw/datasette/issues/1923#issuecomment-1332842435,https://api.github.com/repos/simonw/datasette/issues/1923,1332842435,IC_kwDOBm6k_c5PcY_D,9599,2022-11-30T22:58:33Z,2022-11-30T22:58:33Z,OWNER,https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562740#74562740 suggests trying Python 3.9.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227, https://github.com/simonw/datasette/issues/1923#issuecomment-1332835664,https://api.github.com/repos/simonw/datasette/issues/1923,1332835664,IC_kwDOBm6k_c5PcXVQ,9599,2022-11-30T22:50:10Z,2022-11-30T22:51:25Z,OWNER,https://stackoverflow.com/questions/74490465/github-actions-failing-for-setup-gcloud/74562526#74562526 suggests setting `version: '318.0.0'` of `google-github-actions/setup-gcloud`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1470320227, https://github.com/simonw/datasette/issues/1922#issuecomment-1332698636,https://api.github.com/repos/simonw/datasette/issues/1922,1332698636,IC_kwDOBm6k_c5Pb14M,9599,2022-11-30T20:25:50Z,2022-11-30T20:25:50Z,OWNER,"I just shipped this: Access-Control-Allow-Methods: GET, POST, HEAD, OPTIONS I'll try this out on `latest.datasette.io` - but I need to research more to check if this is a safe thing to do or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332689547,https://api.github.com/repos/simonw/datasette/issues/1922,1332689547,IC_kwDOBm6k_c5PbzqL,9599,2022-11-30T20:16:21Z,2022-11-30T20:16:46Z,OWNER,"That notebook: ```javascript viewof token = Inputs.text({ label: ""Your API token"" }) ``` ```javascript viewof createResponse = Inputs.button(""Create table"", { value: null, reduce: async () => { const response = await fetch( ""https://latest.datasette.io/ephemeral/-/create"", { method: ""POST"", mode: ""cors"", headers: { Authorization: `Bearer {$token}`, ""Content-Type"": ""application/json"" }, body: JSON.stringify({ table: ""my_new_table"", row: { task: ""Demonstrate a JSON creation from another domain"" } }) } ); return await response.json(); } }) ``` Based on this tip: https://talk.observablehq.com/t/best-pattern-for-click-here-to-submit-your-results-to-an-api-backend/7353/3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332688245,https://api.github.com/repos/simonw/datasette/issues/1922,1332688245,IC_kwDOBm6k_c5PbzV1,9599,2022-11-30T20:15:08Z,2022-11-30T20:15:08Z,OWNER,"Still getting a CORS error: My hunch is this is because I'm not sending `Access-Control-Allow-Methods: GET,HEAD,POST`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332585861,https://api.github.com/repos/simonw/datasette/issues/1922,1332585861,IC_kwDOBm6k_c5PbaWF,9599,2022-11-30T18:43:46Z,2022-11-30T18:43:46Z,OWNER,"Here's what Django Rest Framework does: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/views.py#L514-L521 ```python def options(self, request, *args, **kwargs): """""" Handler method for HTTP 'OPTIONS' request. """""" if self.metadata_class is None: return self.http_method_not_allowed(request, *args, **kwargs) data = self.metadata_class().determine_metadata(request, self) return Response(data, status=status.HTTP_200_OK) ``` That default `determine_metadata` method looks like this: https://github.com/encode/django-rest-framework/blob/1ae812ea209392ad76cc5d2f35f9f7fb337f63e4/rest_framework/metadata.py#L61-L71 ```python def determine_metadata(self, request, view): metadata = OrderedDict() metadata['name'] = view.get_view_name() metadata['description'] = view.get_view_description() metadata['renders'] = [renderer.media_type for renderer in view.renderer_classes] metadata['parses'] = [parser.media_type for parser in view.parser_classes] if hasattr(view, 'get_serializer'): actions = self.determine_actions(request, view) if actions: metadata['actions'] = actions return metadata ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332580395,https://api.github.com/repos/simonw/datasette/issues/1922,1332580395,IC_kwDOBm6k_c5PbZAr,9599,2022-11-30T18:38:22Z,2022-11-30T18:38:22Z,OWNER,"> [@simon](https://fedi.simonwillison.net/@simon) IMO, it should always be a 2XX series response, typically with no content & an extra `Allow` header with a list of HTTP verbs it responds to. https://mastodon.social/@daniellindsley/109434186252099323","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332572453,https://api.github.com/repos/simonw/datasette/issues/1922,1332572453,IC_kwDOBm6k_c5PbXEl,9599,2022-11-30T18:30:38Z,2022-11-30T18:30:54Z,OWNER,Started a conversation about how OPTIONS should work on Mastodon: https://fedi.simonwillison.net/@simon/109434148676475291,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332561813,https://api.github.com/repos/simonw/datasette/issues/1922,1332561813,IC_kwDOBm6k_c5PbUeV,9599,2022-11-30T18:20:05Z,2022-11-30T18:20:05Z,OWNER,"Weird, GitHub reply with a 404! ``` ~ % curl -X OPTIONS https://github.com/ -i HTTP/2 404 server: GitHub.com date: Wed, 30 Nov 2022 18:19:39 GMT content-type: text/html; charset=utf-8 content-length: 0 strict-transport-security: max-age=31536000; includeSubdomains; preload x-frame-options: deny x-content-type-options: nosniff x-xss-protection: 0 referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/ opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/ vary: Accept-Encoding, Accept, X-Requested-With x-github-request-id: DD6B:5ACA:102E8A6:1164A99:63879EBB ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332561059,https://api.github.com/repos/simonw/datasette/issues/1922,1332561059,IC_kwDOBm6k_c5PbUSj,9599,2022-11-30T18:19:20Z,2022-11-30T18:19:20Z,OWNER,"Two test failures: ``` ____________________________ test_homepage_options _____________________________ [gw0] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python app_client = def test_homepage_options(app_client): response = app_client.get(""/"", method=""OPTIONS"") > assert response.status == 405 E assert 200 == 405 E + where 200 = .status /home/runner/work/datasette/datasette/tests/test_html.py:58: AssertionError ______________________ test_client_methods[options-/-405] ______________________ [gw1] linux -- Python 3.11.0 /opt/hostedtoolcache/Python/3.11.0/x64/bin/python datasette = method = 'options', path = '/', expected_status = 405 @pytest.mark.asyncio @pytest.mark.parametrize( ""method,path,expected_status"", [ (""get"", ""/"", 200), (""options"", ""/"", 405), (""head"", ""/"", 200), (""put"", ""/"", 405), (""patch"", ""/"", 405), (""delete"", ""/"", 405), ], ) async def test_client_methods(datasette, method, path, expected_status): client_method = getattr(datasette.client, method) response = await client_method(path) assert isinstance(response, httpx.Response) > assert response.status_code == expected_status E assert 200 == 405 E + where 200 = .status_code /home/runner/work/datasette/datasette/tests/test_internals_datasette_client.py:29: AssertionError =============================== warnings summary =============================== tests/test_cli.py::test_inspect_cli_writes_to_file tests/test_cli.py::test_inspect_cli /home/runner/work/datasette/datasette/datasette/cli.py:163: DeprecationWarning: There is no current event loop loop = asyncio.get_event_loop() tests/test_cli_serve_get.py: 2 warnings tests/test_cli.py: 12 warnings tests/test_crossdb.py: 1 warning /home/runner/work/datasette/datasette/datasette/cli.py:591: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(ds.invoke_startup()) tests/test_cli_serve_get.py: 2 warnings tests/test_cli.py: 12 warnings tests/test_crossdb.py: 1 warning /home/runner/work/datasette/datasette/datasette/cli.py:594: DeprecationWarning: There is no current event loop asyncio.get_event_loop().run_until_complete(check_databases(ds)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================== short test summary info ============================ FAILED tests/test_html.py::test_homepage_options - assert 200 == 405 + where 200 = .status FAILED tests/test_internals_datasette_client.py::test_client_methods[options-/-405] - assert 200 == 405 + where 200 = .status_code ====== 2 failed, 1195 passed, 1 skipped, 32 warnings in 191.06s (0:03:11) ====== Error: Process completed with exit code 1. ``` On reading https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/OPTIONS I feel like I should be a bit more thoughtful about how I treat OPTIONS - maybe it should work for every URL on the site, but return a `204 No Content` header? Comparing a few different sites: ``` ~ % curl -X OPTIONS https://www.google.com/ -i HTTP/2 405 allow: GET, HEAD date: Wed, 30 Nov 2022 18:18:15 GMT content-type: text/html; charset=UTF-8 server: gws content-length: 1592 x-xss-protection: 0 x-frame-options: SAMEORIGIN alt-svc: h3="":443""; ma=2592000,h3-29="":443""; ma=2592000,h3-Q050="":443""; ma=2592000,h3-Q046="":443""; ma=2592000,h3-Q043="":443""; ma=2592000,quic="":443""; ma=2592000; v=""46,43"" Error 405 (Method Not Allowed)!!1

405. That’s an error.

The request method OPTIONS is inappropriate for the URL /. That’s all we know. ~ % curl -X OPTIONS https://www.mozilla.org/ -i HTTP/2 405 content-type: text/html; charset=utf-8 content-length: 0 server: meinheld/1.0.2 date: Wed, 30 Nov 2022 18:18:38 GMT allow: GET, HEAD x-frame-options: DENY content-security-policy: child-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; connect-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com region1.google-analytics.com logs.convertexperiments.com 1003350.metrics.convertexperiments.com 1003343.metrics.convertexperiments.com sentry.prod.mozaws.net o1069899.sentry.io o1069899.ingest.sentry.io https://accounts.firefox.com/ stage.cjms.nonprod.cloudops.mozgcp.net cjms.services.mozilla.com; frame-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com www.googletagmanager.com www.google-analytics.com www.youtube-nocookie.com trackertest.org www.surveygizmo.com accounts.firefox.com accounts.firefox.com.cn www.youtube.com; script-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' 'unsafe-eval' www.googletagmanager.com www.google-analytics.com tagmanager.google.com www.youtube.com s.ytimg.com cdn-3.convertexperiments.com app.convert.com data.track.convertexperiments.com 1003350.track.convertexperiments.com 1003343.track.convertexperiments.com; img-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com data: mozilla.org www.googletagmanager.com www.google-analytics.com adservice.google.com adservice.google.de adservice.google.dk creativecommons.org cdn-3.convertexperiments.com logs.convertexperiments.com images.ctfassets.net ad.doubleclick.net; style-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com 'unsafe-inline' app.convert.com; default-src 'self' *.mozilla.net *.mozilla.org *.mozilla.com; font-src 'self' cache-control: max-age=600 expires: Wed, 30 Nov 2022 18:28:38 GMT x-backend-server: bedrock-prod-web-b95bc569d-grd25.iowa-a strict-transport-security: max-age=31536000 x-content-type-options: nosniff x-xss-protection: 1; mode=block referrer-policy: strict-origin-when-cross-origin via: 1.1 google, 1.1 6c90b631453c435bd0022caa657b67e8.cloudfront.net (CloudFront) x-cache: Error from cloudfront x-amz-cf-pop: SFO5-P2 x-amz-cf-id: A6-9mLztaE2tz840CbV9bXYiBMZRKEamDj6jGGEl1U7sg8egWfsDqg== ~ % curl -X OPTIONS https://example.com -i HTTP/2 200 allow: OPTIONS, GET, HEAD, POST cache-control: max-age=604800 content-type: text/html; charset=UTF-8 date: Wed, 30 Nov 2022 18:18:59 GMT expires: Wed, 07 Dec 2022 18:18:59 GMT server: EOS (vny/0451) content-length: 0 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332504654,https://api.github.com/repos/simonw/datasette/issues/1922,1332504654,IC_kwDOBm6k_c5PbGhO,9599,2022-11-30T17:27:39Z,2022-11-30T17:27:39Z,OWNER,I'll test this once it's deployed to https://latest.datasette.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332493004,https://api.github.com/repos/simonw/datasette/issues/1922,1332493004,IC_kwDOBm6k_c5PbDrM,9599,2022-11-30T17:18:10Z,2022-11-30T17:18:10Z,OWNER,"Here's why: https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L71-L79 That's code in `BaseView` - but it turns out the code that adds CORS headers is in the `DataView` subclass of that (which the various write API endpoints do not use). https://github.com/simonw/datasette/blob/4ddd77e51254bda3bac990ea662bac2e6b29c5e0/datasette/views/base.py#L158-L162","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1922#issuecomment-1332492092,https://api.github.com/repos/simonw/datasette/issues/1922,1332492092,IC_kwDOBm6k_c5PbDc8,9599,2022-11-30T17:17:21Z,2022-11-30T17:17:21Z,OWNER,I tried running `fetch()` with a POST from a separate domain and got a browser error because it did a GET against the `/db/-/create` endpoint and the 405 method not supported response did not include the CORS headers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469973742, https://github.com/simonw/datasette/issues/1605#issuecomment-1331694246,https://api.github.com/repos/simonw/datasette/issues/1605,1331694246,IC_kwDOBm6k_c5PYAqm,9599,2022-11-30T06:18:41Z,2022-11-30T06:18:41Z,OWNER,"Those sounds to me like they should be promoted to documented, supported internals.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1108671952, https://github.com/simonw/datasette/issues/1918#issuecomment-1331658629,https://api.github.com/repos/simonw/datasette/issues/1918,1331658629,IC_kwDOBm6k_c5PX3-F,9599,2022-11-30T05:21:51Z,2022-11-30T05:21:51Z,OWNER,"Much better: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469044738, https://github.com/simonw/datasette/issues/1919#issuecomment-1331657404,https://api.github.com/repos/simonw/datasette/issues/1919,1331657404,IC_kwDOBm6k_c5PX3q8,9599,2022-11-30T05:19:43Z,2022-11-30T05:19:43Z,OWNER,"This is the test: https://github.com/simonw/datasette/blob/8404b21556d133c89eda4bd1bf5335ed9a0785d6/tests/test_api_write.py#L342-L401 I'm suspicious that there's a timing error of some sort but I can't think what it might be.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469062686, https://github.com/simonw/datasette/issues/1916#issuecomment-1331651721,https://api.github.com/repos/simonw/datasette/issues/1916,1331651721,IC_kwDOBm6k_c5PX2SJ,9599,2022-11-30T05:10:27Z,2022-11-30T05:10:27Z,OWNER,"They should return 405 method not allowed with an `{""ok"":false, ""error"": ""Method not allowed""}` body.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469015001, https://github.com/simonw/datasette/issues/1917#issuecomment-1331644751,https://api.github.com/repos/simonw/datasette/issues/1917,1331644751,IC_kwDOBm6k_c5PX0lP,9599,2022-11-30T04:59:22Z,2022-11-30T04:59:22Z,OWNER,"Yeah it looks like I introduced this bug here: https://github.com/simonw/datasette/commit/fb7e70d5e72a951efe4b29ad999d8915c032d021","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469043836, https://github.com/simonw/datasette/issues/1917#issuecomment-1331644078,https://api.github.com/repos/simonw/datasette/issues/1917,1331644078,IC_kwDOBm6k_c5PX0au,9599,2022-11-30T04:58:06Z,2022-11-30T04:58:06Z,OWNER,"The problem might actually be here: https://github.com/simonw/datasette/blob/9f5321ff1eca58c469a45cc406d7eb5ad05accbd/datasette/app.py#L280-L281 `is_mutable` defaults to `True`, so this line should probably be: ```python self.add_database(Database(self, is_mutable=False, is_memory=True), name=""_memory"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1469043836, https://github.com/simonw/datasette/issues/1915#issuecomment-1331479606,https://api.github.com/repos/simonw/datasette/issues/1915,1331479606,IC_kwDOBm6k_c5PXMQ2,9599,2022-11-30T00:09:06Z,2022-11-30T00:09:06Z,OWNER,One last feature: I want to show an indication on the table page that the table has X seconds left to live.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531, https://github.com/simonw/datasette/issues/1915#issuecomment-1331479328,https://api.github.com/repos/simonw/datasette/issues/1915,1331479328,IC_kwDOBm6k_c5PXMMg,9599,2022-11-30T00:08:41Z,2022-11-30T00:08:41Z,OWNER,Five minute has now passed and https://latest.datasette.io/ephemeral/new_table is gone.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531, https://github.com/simonw/datasette/issues/1915#issuecomment-1331476246,https://api.github.com/repos/simonw/datasette/issues/1915,1331476246,IC_kwDOBm6k_c5PXLcW,9599,2022-11-30T00:04:35Z,2022-11-30T00:08:24Z,OWNER,"The new https://github.com/simonw/datasette-ephemeral-tables plugin is live now: https://latest.datasette.io/ephemeral - you have to navigate through https://latest.datasette.io/login-as-root first It work! I created a table using https://latest.datasette.io/-/api#path=%2Fephemeral%2F-%2Fcreate&json=%7B%0A++%22table%22%3A+%22new_table%22%2C%0A++%22columns%22%3A+%5B%0A++++%7B%0A++++++%22name%22%3A+%22id%22%2C%0A++++++%22type%22%3A+%22integer%22%0A++++%7D%2C%0A++++%7B%0A++++++%22name%22%3A+%22name%22%2C%0A++++++%22type%22%3A+%22text%22%0A++++%7D%0A++%5D%2C%0A++%22pk%22%3A+%22id%22%0A%7D&method=POST The table should vanish in a few minutes.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531, https://github.com/simonw/datasette/issues/1915#issuecomment-1331478611,https://api.github.com/repos/simonw/datasette/issues/1915,1331478611,IC_kwDOBm6k_c5PXMBT,9599,2022-11-30T00:07:37Z,2022-11-30T00:07:37Z,OWNER,"Then I created an API token at https://latest.datasette.io/-/create-token and ran this: ``` curl -XPOST 'https://latest.datasette.io/ephemeral/new_table/-/insert' \ -H 'Authorization: Bearer xxx' \ -H 'Content-Type: application/json' \ -d '{""row"": {""name"": ""NAME""}}' ``` And it inserted a row into https://latest.datasette.io/ephemeral/new_table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531, https://github.com/simonw/datasette/issues/1915#issuecomment-1331432223,https://api.github.com/repos/simonw/datasette/issues/1915,1331432223,IC_kwDOBm6k_c5PXAsf,9599,2022-11-29T23:06:17Z,2022-11-29T23:06:17Z,OWNER,To (slightly) discourage abuse I'm going to make the demo database only visible to the root user - so people can't create tables with rude names and have them show to the public on https://latest.datasette.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531, https://github.com/simonw/datasette/issues/1915#issuecomment-1331331082,https://api.github.com/repos/simonw/datasette/issues/1915,1331331082,IC_kwDOBm6k_c5PWoAK,9599,2022-11-29T21:24:59Z,2022-11-29T21:34:53Z,OWNER,Maybe a plugin called `datasette-temporary-tables` or `datasette-demo-tables` or `datasette-demo-database`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468709531, https://github.com/simonw/datasette/issues/1850#issuecomment-1331238841,https://api.github.com/repos/simonw/datasette/issues/1850,1331238841,IC_kwDOBm6k_c5PWRe5,9599,2022-11-29T20:11:20Z,2022-11-29T20:11:20Z,OWNER,"Released this in Datasette 1.0a0: - #1913","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1421529723, https://github.com/simonw/datasette/issues/1913#issuecomment-1331238029,https://api.github.com/repos/simonw/datasette/issues/1913,1331238029,IC_kwDOBm6k_c5PWRSN,9599,2022-11-29T20:10:35Z,2022-11-29T20:10:35Z,OWNER,"Released: - https://pypi.org/project/datasette/1.0a0/ - https://docs.datasette.io/en/latest/changelog.html#a0-2022-11-29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331226346,https://api.github.com/repos/simonw/datasette/issues/1913,1331226346,IC_kwDOBm6k_c5PWObq,9599,2022-11-29T20:00:16Z,2022-11-29T20:00:36Z,OWNER,"Looks like a fix is coming: https://github.com/pypa/twine/issues/940#issuecomment-1331225509 > > Note that `must_decode` was defined in `pkg_info/_compat.py`, and was thus never an API: before 1.9.0, it was only imported and used in `pkginfo/distribution.py'. > > Nevertheless, I will push out a 1.9.1 release of `pkginfo` which restores a deprecated compatibility alias.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331225277,https://api.github.com/repos/simonw/datasette/issues/1913,1331225277,IC_kwDOBm6k_c5PWOK9,9599,2022-11-29T19:59:14Z,2022-11-29T19:59:34Z,OWNER,I deleted the tag and tried creating a new release. Now running here: https://github.com/simonw/datasette/actions/runs/3577554546,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331216652,https://api.github.com/repos/simonw/datasette/issues/1913,1331216652,IC_kwDOBm6k_c5PWMEM,9599,2022-11-29T19:54:22Z,2022-11-29T19:54:22Z,OWNER,Filed a bug report here: https://bugs.launchpad.net/pkginfo/+bug/1998249,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331208206,https://api.github.com/repos/simonw/datasette/issues/1913,1331208206,IC_kwDOBm6k_c5PWKAO,9599,2022-11-29T19:51:31Z,2022-11-29T19:51:31Z,OWNER,https://pypi.org/project/pkginfo/#history - 1.9.0 came out 39 minutes ago!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331207334,https://api.github.com/repos/simonw/datasette/issues/1913,1331207334,IC_kwDOBm6k_c5PWJym,9599,2022-11-29T19:50:37Z,2022-11-29T19:50:37Z,OWNER,"https://pypi.org/project/setuptools/65.6.3/ came out most recently - 23rd November (wheel and twine are older). No search results at all for that error message. This is very weird, I would have expected it to have been reported by now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331205613,https://api.github.com/repos/simonw/datasette/issues/1913,1331205613,IC_kwDOBm6k_c5PWJXt,9599,2022-11-29T19:48:52Z,2022-11-29T19:48:52Z,OWNER,https://github.com/simonw/datasette/blob/07aad511769da9242260c850e8d975cbd8c29552/.github/workflows/publish.yml#L52-L61,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331204360,https://api.github.com/repos/simonw/datasette/issues/1913,1331204360,IC_kwDOBm6k_c5PWJEI,9599,2022-11-29T19:47:40Z,2022-11-29T19:47:40Z,OWNER,"... but the last step of the deploy failed, when it was meant to push to PyPI! ``` Uploading distributions to https://upload.pypi.org/legacy/ Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.11.0/x64/bin/twine"", line 8, in sys.exit(main()) ^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/__main__.py"", line 33, in main error = cli.dispatch(sys.argv[1:]) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/cli.py"", line 123, in dispatch return main(args.args) ^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 198, in main return upload(upload_settings, parsed_args.dists) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 123, in upload packages_to_upload = [ ^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 124, in _make_package(filename, signatures, upload_settings) for filename in uploads ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/commands/upload.py"", line 77, in _make_package package = package_file.PackageFile.from_filename(filename, upload_settings.comment) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/package.py"", line 96, in from_filename meta = DIST_TYPES[dtype](filename) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py"", line 42, in __init__ self.extractMetadata() File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/pkginfo/distribution.py"", line 121, in extractMetadata self.parse(data) File ""/opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/site-packages/twine/wheel.py"", line 89, in parse fp = io.StringIO(distribution.must_decode(data)) ^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkginfo.distribution' has no attribute 'must_decode'. Did you mean: '_must_decode'? Error: Process completed with exit code 1. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1913#issuecomment-1331203997,https://api.github.com/repos/simonw/datasette/issues/1913,1331203997,IC_kwDOBm6k_c5PWI-d,9599,2022-11-29T19:47:13Z,2022-11-29T19:47:13Z,OWNER,"Weird, retrying the tests DID get them to pass. https://github.com/simonw/datasette/actions/runs/3577355358/jobs/6016518244","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468603401, https://github.com/simonw/datasette/issues/1877#issuecomment-1331201207,https://api.github.com/repos/simonw/datasette/issues/1877,1331201207,IC_kwDOBm6k_c5PWIS3,9599,2022-11-29T19:44:07Z,2022-11-29T19:44:07Z,OWNER,"I fixed the duplicate logic issue here: https://github.com/simonw/datasette/commit/ee64130fa8a5ff4a24791916c696e10cf2375102 - #1896 Decided not to address `views/table.py`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1432012302, https://github.com/simonw/datasette/pull/1912#issuecomment-1331196531,https://api.github.com/repos/simonw/datasette/issues/1912,1331196531,IC_kwDOBm6k_c5PWHJz,9599,2022-11-29T19:39:10Z,2022-11-29T19:39:10Z,OWNER,"Annoyingly it looks like I can't rebase this one, and I don't want to squash-merge and lose the commits, so I'm going to do a regular merge instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468592292, https://github.com/simonw/datasette/issues/1891#issuecomment-1331181922,https://api.github.com/repos/simonw/datasette/issues/1891,1331181922,IC_kwDOBm6k_c5PWDli,9599,2022-11-29T19:23:41Z,2022-11-29T19:23:41Z,OWNER,https://github.com/simonw/datasette/blob/4d49a5a39739476e1ada43f70a0029abcef07977/docs/changelog.rst#10a0-2022-11-29,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205, https://github.com/simonw/datasette/issues/1891#issuecomment-1331143292,https://api.github.com/repos/simonw/datasette/issues/1891,1331143292,IC_kwDOBm6k_c5PV6J8,9599,2022-11-29T18:57:40Z,2022-11-29T18:57:40Z,OWNER,I'm going to keep these short - they'll mostly be links to the documentation for the new features.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205, https://github.com/simonw/datasette/issues/1891#issuecomment-1331140747,https://api.github.com/repos/simonw/datasette/issues/1891,1331140747,IC_kwDOBm6k_c5PV5iL,9599,2022-11-29T18:55:42Z,2022-11-29T18:55:42Z,OWNER,"All features for the alpha are complete now. Release notes should be based on these commits: https://github.com/simonw/datasette/compare/0.63.2...6bda2257868a2cbd70b84b7a86a5bcb47dcc4874","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1450303205, https://github.com/simonw/datasette/issues/1911#issuecomment-1331135709,https://api.github.com/repos/simonw/datasette/issues/1911,1331135709,IC_kwDOBm6k_c5PV4Td,9599,2022-11-29T18:50:58Z,2022-11-29T18:50:58Z,OWNER,Updated docs: https://docs.datasette.io/en/1.0-dev/json_api.html#creating-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1468519699,