{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1304539296", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1304539296, "node_id": "IC_kwDOCGYnMM5NwbCg", "user": {"value": 559711, "label": "ryascott"}, "created_at": "2022-11-05T12:40:12Z", "updated_at": "2022-11-05T12:40:12Z", "author_association": "NONE", "body": "I had the problem this morning when running:\r\n\r\n`Python==3.9.6\r\n sqlite3.sqlite_version==3.37.0\r\n sqlite-utils==3.30\r\n`\r\n\r\nI upgraded to:\r\n`Python ==3.10.8 \r\n sqlite3.sqlite_version==3.37.2\r\n sqlite-utils==3.30\r\n`\r\n\r\nand the error did not appear anymore.\r\n\r\nHope this helps\r\nRyan\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304320521", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/511", "id": 1304320521, "node_id": "IC_kwDOCGYnMM5NvloJ", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-04T22:54:09Z", "updated_at": "2022-11-04T22:59:54Z", "author_association": "CONTRIBUTOR", "body": "I ran `PRAGMA integrity_check` and it returned `ok`. but then I tried restoring from a backup and I didn't get this `IntegrityError: constraint failed` error. So I think it was just something wrong with my database. If it happens again I will first try to reindex and see if that fixes the issue", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1436539554, "label": "[insert_all, upsert_all] IntegrityError: constraint failed"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/511#issuecomment-1304078945", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/511", "id": 1304078945, "node_id": "IC_kwDOCGYnMM5Nuqph", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-04T19:38:36Z", "updated_at": "2022-11-04T20:13:17Z", "author_association": "CONTRIBUTOR", "body": "Even more bizarre, the source db only has one record and the target table has no conflicting record:\r\n\r\n```\r\n875 0.3s lb:/ (main|\u271a2) [0|0]\ud83c\udf3a sqlite-utils tube_71.db 'select * from media where path = \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\"' | jq\r\n[\r\n {\r\n \"size\": null,\r\n \"time_created\": null,\r\n \"play_count\": 1,\r\n \"language\": null,\r\n \"view_count\": null,\r\n \"width\": null,\r\n \"height\": null,\r\n \"fps\": null,\r\n \"average_rating\": null,\r\n \"live_status\": null,\r\n \"age_limit\": null,\r\n \"uploader\": null,\r\n \"time_played\": 0,\r\n \"path\": \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\",\r\n \"id\": \"088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz/074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv\",\r\n \"ie_key\": \"ArchiveOrg\",\r\n \"playlist_path\": \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\",\r\n \"duration\": 1424.05,\r\n \"tags\": null,\r\n \"title\": \"074 - Home Away from Home, Rainy Day Robot, Odie the Amazing DVDRip XviD [PhZ].mkv\"\r\n }\r\n]\r\n875 0.3s lb:/ (main|\u271a2) [0|0]\ud83e\udd67 sqlite-utils video.db 'select * from media where path = \"https://archive.org/details/088ghostofachanceroygetssackedrevengeofthelivinglunchdvdripxvidphz\"' | jq\r\n[]\r\n```\r\n\r\nI've been able to use this code successfully several times before so not sure what's causing the issue.\r\n\r\nI guess the way that I'm handling multiple databases is an issue, though it hasn't ever inserted into the source db, not sure what's different. The only reasonable explanation is that it is trying to insert into the source db from the source db for some reason? Or maybe sqlite3 is checking the source db for primary key violation because the table name is the same", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1436539554, "label": "[insert_all, upsert_all] IntegrityError: constraint failed"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302818784", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302818784, "node_id": "IC_kwDOBm6k_c5Np2_g", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:25:18Z", "updated_at": "2022-11-04T16:12:39Z", "author_association": "OWNER", "body": "On that basis I think the core API design should change to this:\r\n```\r\nPOST /db/-/create\r\nAuthorization: Bearer xxx\r\nContent-Type: application/json\r\n{\r\n \"name\": \"my new table\",\r\n \"columns\": [\r\n {\r\n \"name\": \"id\",\r\n \"type\": \"integer\"\r\n },\r\n {\r\n \"name\": \"title\",\r\n \"type\": \"text\"\r\n }\r\n ]\r\n \"pk\": \"id\"\r\n}\r\n```\r\nThis leaves room for a `\"rows\": []` key at the root too. Having that as a child of `\"table\"` felt unintuitive to me, and I didn't like the way this looked either:\r\n\r\n```json\r\n{\r\n \"table\": {\r\n \"name\": \"my_new_table\"\r\n },\r\n \"rows\": [\r\n {\"id\": 1, \"title\": \"Title\"}\r\n ]\r\n}\r\n```\r\nWeird to have the table `name` nested inside `table` when `rows` wasn't.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/50#issuecomment-1303660293", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/50", "id": 1303660293, "node_id": "IC_kwDOCGYnMM5NtEcF", "user": {"value": 7908073, "label": "chapmanjacobd"}, "created_at": "2022-11-04T14:38:36Z", "updated_at": "2022-11-04T14:38:36Z", "author_association": "CONTRIBUTOR", "body": "where did you see the limit as 999? I believe the limit has been 32766 for quite some time. If you could detect which one this could speed up batch insert of some types of data significantly", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 473083260, "label": "\"Too many SQL variables\" on large inserts"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1217#issuecomment-1303301786", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1217", "id": 1303301786, "node_id": "IC_kwDOBm6k_c5Nrs6a", "user": {"value": 31312775, "label": "mattmalcher"}, "created_at": "2022-11-04T11:37:52Z", "updated_at": "2022-11-04T11:37:52Z", "author_association": "NONE", "body": "All seems to work well, but there are some glitches to do with proxies, see #1883 .\r\n\r\nExcited to use this :)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 802513359, "label": "Possible to deploy as a python app (for Rstudio connect server)?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1217#issuecomment-1303299509", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1217", "id": 1303299509, "node_id": "IC_kwDOBm6k_c5NrsW1", "user": {"value": 31312775, "label": "mattmalcher"}, "created_at": "2022-11-04T11:35:13Z", "updated_at": "2022-11-04T11:35:13Z", "author_association": "NONE", "body": "The following worked for deployment to RStudio / Posit Connect\r\n\r\nAn app.py along the lines of:\r\n\r\n```python\r\nfrom pathlib import Path\r\n\r\nfrom datasette.app import Datasette\r\n\r\nexample_db = Path(__file__).parent / \"data\" / \"example.db\"\r\n\r\n# use connect 'Content URL' setting here to set app to /datasette/\r\nds = Datasette(files=[example_db], settings={\"base_url\": \"/datasette/\"})\r\n\r\nds._startup_invoked = True\r\nds_app = ds.app()\r\n```\r\nThen to deploy, from within a virtualenv with `rsconnect-python`\r\n```sh\r\nrsconnect write-manifest fastapi -p $VIRTUAL_ENV/bin/python -e app:ds_app -o .\r\nrsconnect deploy manifest manifest.json -n -t \"Example Datasette\"\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 802513359, "label": "Possible to deploy as a python app (for Rstudio connect server)?"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302818153", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302818153, "node_id": "IC_kwDOBm6k_c5Np21p", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:23:58Z", "updated_at": "2022-11-04T00:23:58Z", "author_association": "OWNER", "body": "I made a decision here that this endpoint should also accept an optional `\"rows\": [...]` list which is used to automatically create the table using a schema derived from those example rows (which then get inserted):\r\n\r\n- https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1862#issuecomment-1302817807", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1862", "id": 1302817807, "node_id": "IC_kwDOBm6k_c5Np2wP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:23:13Z", "updated_at": "2022-11-04T00:23:13Z", "author_association": "OWNER", "body": "I don't like this on `/db/table/-/insert` - I think it makes more sense to optionally pass a `\"rows\"` key to the `/db/-/create` endpoint instead.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425011030, "label": "Create a new table from one or more records, `sqlite-utils` style"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1862#issuecomment-1302817500", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1862", "id": 1302817500, "node_id": "IC_kwDOBm6k_c5Np2rc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:22:31Z", "updated_at": "2022-11-04T00:22:31Z", "author_association": "OWNER", "body": "Maybe this is a feature added to the existing `/db/table/-/insert` endpoint?\r\n\r\nBit weird that you can call that endpoint for a table that doesn't exist yet, but it fits the `sqlite-utils` way of creating tables which I've found very pleasant over the past few years.\r\n\r\nSo perhaps the API looks like this:\r\n\r\n```\r\nPOST //
/-/insert\r\nContent-Type: application/json\r\nAuthorization: Bearer dstok_\r\n{\r\n \"create_table\": true,\r\n \"rows\": [\r\n {\r\n \"column1\": \"value1\",\r\n \"column2\": \"value2\"\r\n },\r\n {\r\n \"column1\": \"value3\",\r\n \"column2\": \"value4\"\r\n }\r\n ]\r\n}\r\n```\r\nThe `create_table` option will cause the table to be created if it doesn't already exist.\r\n\r\nThat means I probably also need a `\"pk\": \"...\"` column for setting a primary key if the table is being created ... and maybe other options that I invent for this other feature too?\r\n- #1882", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425011030, "label": "Create a new table from one or more records, `sqlite-utils` style"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1871#issuecomment-1302815105", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1871", "id": 1302815105, "node_id": "IC_kwDOBm6k_c5Np2GB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:17:23Z", "updated_at": "2022-11-04T00:17:23Z", "author_association": "OWNER", "body": "I'll probably enhance it a bit more though, I want to provide a UI that lists all the tables you can explore and lets you click to pre-fill the forms with them.\r\n\r\nThough at that point what should I do about the other endpoints? Probably list those too. Gets a bit complex, especially with the row-level update and delete endpoints.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1427293909, "label": "API explorer tool"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1871#issuecomment-1302814693", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1871", "id": 1302814693, "node_id": "IC_kwDOBm6k_c5Np1_l", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:16:36Z", "updated_at": "2022-11-04T00:16:36Z", "author_association": "OWNER", "body": "I can close this issue once I fix it so it no longer hard-codes a potentially invalid example endpoint:\r\n\r\nhttps://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L24-L26\r\n\r\nhttps://github.com/simonw/datasette/blob/bcc781f4c50a8870e3389c4e60acb625c34b0317/datasette/templates/api_explorer.html#L34-L35", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1427293909, "label": "API explorer tool"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1302813449", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1302813449, "node_id": "IC_kwDOBm6k_c5Np1sJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:14:07Z", "updated_at": "2022-11-04T00:14:07Z", "author_association": "OWNER", "body": "Tool is now live here: https://latest-1-0-dev.datasette.io/-/permissions\r\n\r\nNeeds root perms, so access this first: https://latest-1-0-dev.datasette.io/login-as-root", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1302812918", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1302812918, "node_id": "IC_kwDOBm6k_c5Np1j2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-04T00:13:05Z", "updated_at": "2022-11-04T00:13:05Z", "author_association": "OWNER", "body": "Has tests now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302790013", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302790013, "node_id": "IC_kwDOBm6k_c5Npv99", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T23:32:30Z", "updated_at": "2022-11-03T23:32:30Z", "author_association": "OWNER", "body": "I'm not going to allow updates to primary keys. If you need to do that, you can instead delete the record and then insert a new one with the new primary keys you wanted - or maybe use a custom SQL query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1851#issuecomment-1294224185", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1851", "id": 1294224185, "node_id": "IC_kwDOBm6k_c5NJEs5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-10-27T23:18:24Z", "updated_at": "2022-11-03T23:26:05Z", "author_association": "OWNER", "body": "So new API design is:\r\n\r\n```\r\nPOST /db/table/-/insert\r\nAuthorization: Bearer xxx\r\nContent-Type: application/json\r\n{\r\n \"row\": {\r\n \"id\": 1,\r\n \"name\": \"New record\"\r\n }\r\n}\r\n```\r\nReturns:\r\n```\r\n201 Created\r\n{\r\n \"row\": [{\r\n \"id\": 1,\r\n \"name\": \"New record\"\r\n }]\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1421544654, "label": "API to insert a single record into an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302785086", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302785086, "node_id": "IC_kwDOBm6k_c5Npuw-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T23:24:33Z", "updated_at": "2022-11-03T23:24:56Z", "author_association": "OWNER", "body": "Thinking more about validation: I'm considering if this should validate that columns which are defined as SQLite foreign keys are being updated to values that exist in those other tables.\r\n\r\nI like the sound of this. It seems like a sensible default behaviour for Datasette. And it fits with the fact that Datasette treats foreign keys specially elsewhere in the interface.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302760549", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302760549, "node_id": "IC_kwDOBm6k_c5Npoxl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T22:43:04Z", "updated_at": "2022-11-03T23:21:31Z", "author_association": "OWNER", "body": "The `id=(int, ...)` thing is weird, but is apparently Pydantic syntax for a required field?\r\n\r\nhttps://cs.github.com/starlite-api/starlite/blob/28ddc847c4cb072f0d5d21a9ecd5259711f12ec9/docs/usage/11-data-transfer-objects.md#L161 confirms:\r\n\r\n> 1. For required fields use a tuple of type + ellipsis, for example `(str, ...)`.\r\n> 2. For optional fields use a tuple of type + `None`, for example `(str, None)`\r\n> 3. To set a default value use a tuple of type + default value, for example `(str, \"Hello World\")`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302760382", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302760382, "node_id": "IC_kwDOBm6k_c5Npou-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T22:42:47Z", "updated_at": "2022-11-03T22:42:47Z", "author_association": "OWNER", "body": "```python\r\nprint(create_model('document', id=(int, ...), title=(str, None)).schema_json(indent=2))\r\n```\r\n```json\r\n{\r\n \"title\": \"document\",\r\n \"type\": \"object\",\r\n \"properties\": {\r\n \"id\": {\r\n \"title\": \"Id\",\r\n \"type\": \"integer\"\r\n },\r\n \"title\": {\r\n \"title\": \"Title\",\r\n \"type\": \"string\"\r\n }\r\n },\r\n \"required\": [\r\n \"id\"\r\n ]\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1863#issuecomment-1302759174", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1863", "id": 1302759174, "node_id": "IC_kwDOBm6k_c5NpocG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T22:40:47Z", "updated_at": "2022-11-03T22:40:47Z", "author_association": "OWNER", "body": "I'm considering Pydantic for this, see:\r\n- https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350\r\n\r\nIn particular the `create_model()` method: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation\r\n\r\nThis would give me good validation. It would also, weirdly, give me the ability to output JSON schema. Maybe I could have this as the JSON schema for a row?\r\n\r\n`/db/table/-/json-schema`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1425029242, "label": "Update a single record in an existing table"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302716350", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302716350, "node_id": "IC_kwDOBm6k_c5Npd--", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:51:14Z", "updated_at": "2022-11-03T22:35:54Z", "author_association": "OWNER", "body": "Validating this JSON object is getting a tiny bit complex. I'm tempted to adopt https://pydantic-docs.helpmanual.io/ at this point.\r\n\r\nThe `create_model` example on https://stackoverflow.com/questions/66168517/generate-dynamic-model-using-pydantic/66168682#66168682 is particularly relevant, especially when I work on this issue:\r\n\r\n- #1863\r\n\r\n```python\r\nfrom pydantic import create_model\r\n\r\nd = {\"strategy\": {\"name\": \"test_strat2\", \"periods\": 10}}\r\n\r\nStrategy = create_model(\"Strategy\", **d[\"strategy\"])\r\n\r\nprint(Strategy.schema_json(indent=2))\r\n```\r\n`create_model()`: https://pydantic-docs.helpmanual.io/usage/models/#dynamic-model-creation", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302721916", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302721916, "node_id": "IC_kwDOBm6k_c5NpfV8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:58:50Z", "updated_at": "2022-11-03T21:59:17Z", "author_association": "OWNER", "body": "Mocked up a quick HTML+JavaScript form for creating that JSON structure using some iteration against Copilot prompts:\r\n```html\r\n
\r\n/* JSON format:\r\n{\r\n \"table\": {\r\n \"name\": \"my new table\",\r\n \"columns\": [\r\n {\r\n \"name\": \"id\",\r\n \"type\": \"integer\"\r\n },\r\n {\r\n \"name\": \"title\",\r\n \"type\": \"text\"\r\n }\r\n ]\r\n \"pk\": \"id\"\r\n }\r\n}\r\n\r\nHTML form with Javascript for creating this JSON:\r\n*/
\r\n\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1882#issuecomment-1302715662", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1882", "id": 1302715662, "node_id": "IC_kwDOBm6k_c5Npd0O", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:50:27Z", "updated_at": "2022-11-03T21:50:27Z", "author_association": "OWNER", "body": "API design for this:\r\n```\r\nPOST /db/-/create\r\nAuthorization: Bearer xxx\r\nContent-Type: application/json\r\n{\r\n \"table\": {\r\n \"name\": \"my new table\",\r\n \"columns\": [\r\n {\r\n \"name\": \"id\",\r\n \"type\": \"integer\"\r\n },\r\n {\r\n \"name\": \"title\",\r\n \"type\": \"text\"\r\n }\r\n ]\r\n \"pk\": \"id\"\r\n }\r\n}\r\n```\r\nSupported column types are:\r\n\r\n- `integer`\r\n- `text`\r\n- `float` (even though SQLite calls it a \"real\")\r\n- `blob`\r\n\r\nThis matches my design for `sqlite-utils`: https://sqlite-utils.datasette.io/en/stable/cli.html#cli-create-table", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1435294468, "label": "`/db/-/create` API for creating tables"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302679026", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302679026, "node_id": "IC_kwDOBm6k_c5NpU3y", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:22:42Z", "updated_at": "2022-11-03T21:22:42Z", "author_association": "OWNER", "body": "Docs for the new `db.close()` method: https://docs.datasette.io/en/latest/internals.html#db-close", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302678384", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302678384, "node_id": "IC_kwDOBm6k_c5NpUtw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T21:21:59Z", "updated_at": "2022-11-03T21:21:59Z", "author_association": "OWNER", "body": "I added extra debug info to `/-/threads` to see this for myself:\r\n\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex 02bd38f1..16579e28 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -969,6 +969,13 @@ class Datasette:\r\n \"threads\": [\r\n {\"name\": t.name, \"ident\": t.ident, \"daemon\": t.daemon} for t in threads\r\n ],\r\n+ \"file_connections\": {\r\n+ db.name: [\r\n+ [dict(r) for r in conn.execute(\"pragma database_list\").fetchall()]\r\n+ for conn in db._all_file_connections\r\n+ ]\r\n+ for db in self.databases.values()\r\n+ },\r\n }\r\n # Only available in Python 3.7+\r\n if hasattr(asyncio, \"all_tasks\"):\r\n```\r\nOutput after hitting refresh on a few `/fixtures` tables to ensure more threads started:\r\n\r\n```\r\n \"file_connections\": {\r\n \"_internal\": [],\r\n \"fixtures\": [\r\n [\r\n {\r\n \"seq\": 0,\r\n \"name\": \"main\",\r\n \"file\": \"/Users/simon/Dropbox/Development/datasette/fixtures.db\"\r\n }\r\n ],\r\n [\r\n {\r\n \"seq\": 0,\r\n \"name\": \"main\",\r\n \"file\": \"/Users/simon/Dropbox/Development/datasette/fixtures.db\"\r\n }\r\n ],\r\n [\r\n {\r\n \"seq\": 0,\r\n \"name\": \"main\",\r\n \"file\": \"/Users/simon/Dropbox/Development/datasette/fixtures.db\"\r\n }\r\n ]\r\n ]\r\n },\r\n```\r\nI decided not to ship this feature though as it leaks the names of internal database files.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302634332", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302634332, "node_id": "IC_kwDOBm6k_c5NpJ9c", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T20:34:56Z", "updated_at": "2022-11-03T20:34:56Z", "author_association": "OWNER", "body": "Confirmed that calling `conn.close()` on each SQLite file-based connection is the way to fix this problem.\r\n\r\nI'm adding a `db.close()` method (sync, not async - I tried async first but it was really hard to cause every thread in the pool to close its threadlocal database connection) which loops through all known open file-based connections and closes them.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1843#issuecomment-1302574330", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1843", "id": 1302574330, "node_id": "IC_kwDOBm6k_c5No7T6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T19:30:22Z", "updated_at": "2022-11-03T19:30:22Z", "author_association": "OWNER", "body": "This is affecting me a lot at the moment, on my laptop (runs fine in CI).\r\n\r\nHere's a change to `conftest.py` which highlights the problem - it cause a failure the moment there are more than 5 open files according to `psutil`:\r\n\r\n```diff\r\ndiff --git a/tests/conftest.py b/tests/conftest.py\r\nindex f4638a14..21d433c1 100644\r\n--- a/tests/conftest.py\r\n+++ b/tests/conftest.py\r\n@@ -1,6 +1,7 @@\r\n import httpx\r\n import os\r\n import pathlib\r\n+import psutil\r\n import pytest\r\n import re\r\n import subprocess\r\n@@ -192,3 +193,8 @@ def ds_unix_domain_socket_server(tmp_path_factory):\r\n yield ds_proc, uds\r\n # Shut it down at the end of the pytest session\r\n ds_proc.terminate()\r\n+\r\n+\r\n+def pytest_runtest_teardown(item: pytest.Item) -> None:\r\n+ open_files = psutil.Process().open_files()\r\n+ assert len(open_files) < 5\r\n```\r\nThe first error I get from this with `pytest --pdb -x` is here:\r\n\r\n```\r\ntests/test_api.py ............E\r\n>>>>> traceback >>>>>\r\n\r\nitem = \r\n\r\n def pytest_runtest_teardown(item: pytest.Item) -> None:\r\n open_files = psutil.Process().open_files()\r\n> assert len(open_files) < 5\r\nE AssertionError: assert 5 < 5\r\nE + where 5 = len([popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmpfglrt4p2/fixtures.db', fd=14), popenfile(... fd=19), popenfile(path='/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/tmphdi5b250/fixtures.dot.db', fd=20)])\r\n\r\n/Users/simon/Dropbox/Development/datasette/tests/conftest.py:200: AssertionError\r\n>>>>> entering PDB >>>>>\r\n\r\n>>>>> PDB post_mortem (IO-capturing turned off) >>>>>\r\n> /Users/simon/Dropbox/Development/datasette/tests/conftest.py(200)pytest_runtest_teardown()\r\n-> assert len(open_files) < 5\r\n```\r\nThat's this test:\r\n\r\nhttps://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/test_api.py#L656-L673\r\n\r\nWhich uses this fixture:\r\n\r\nhttps://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L228-L231\r\n\r\nWhich calls this function:\r\n\r\nhttps://github.com/simonw/datasette/blob/2ec5583629005b32cb0877786f9681c5d43ca33f/tests/fixtures.py#L105-L122\r\n\r\nSo now I'm suspicious that, even though the fixture is meant to be session scoped, the way I'm using `with tempfile.TemporaryDirectory() as tmpdir:` is causing a whole load of files to be created and held open which are not later closed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1408757705, "label": "Intermittent \"Too many open files\" error running tests"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1855#issuecomment-1301646670", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1855", "id": 1301646670, "node_id": "IC_kwDOBm6k_c5NlY1O", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T05:11:26Z", "updated_at": "2022-11-03T05:11:26Z", "author_association": "OWNER", "body": "That still needs comprehensive tests before I land it.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1423336089, "label": "`datasette create-token` ability to create tokens with a reduced set of permissions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1855#issuecomment-1301646493", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1855", "id": 1301646493, "node_id": "IC_kwDOBm6k_c5NlYyd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T05:11:06Z", "updated_at": "2022-11-03T05:11:06Z", "author_association": "OWNER", "body": "Built a prototype of the above:\r\n\r\n```diff\r\ndiff --git a/datasette/default_permissions.py b/datasette/default_permissions.py\r\nindex 32b0c758..f68aa38f 100644\r\n--- a/datasette/default_permissions.py\r\n+++ b/datasette/default_permissions.py\r\n@@ -6,8 +6,8 @@ import json\r\n import time\r\n \r\n \r\n-@hookimpl(tryfirst=True)\r\n-def permission_allowed(datasette, actor, action, resource):\r\n+@hookimpl(tryfirst=True, specname=\"permission_allowed\")\r\n+def permission_allowed_default(datasette, actor, action, resource):\r\n async def inner():\r\n if action in (\r\n \"permissions-debug\",\r\n@@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource):\r\n return inner\r\n \r\n \r\n+@hookimpl(specname=\"permission_allowed\")\r\n+def permission_allowed_actor_restrictions(actor, action, resource):\r\n+ if actor is None:\r\n+ return None\r\n+ _r = actor.get(\"_r\")\r\n+ if not _r:\r\n+ # No restrictions, so we have no opinion\r\n+ return None\r\n+ action_initials = \"\".join([word[0] for word in action.split(\"-\")])\r\n+ # If _r is defined then we use those to further restrict the actor\r\n+ # Crucially, we only use this to say NO (return False) - we never\r\n+ # use it to return YES (True) because that might over-ride other\r\n+ # restrictions placed on this actor\r\n+ all_allowed = _r.get(\"a\")\r\n+ if all_allowed is not None:\r\n+ assert isinstance(all_allowed, list)\r\n+ if action_initials in all_allowed:\r\n+ return None\r\n+ # How about for the current database?\r\n+ if action in (\"view-database\", \"view-database-download\", \"execute-sql\"):\r\n+ database_allowed = _r.get(\"d\", {}).get(resource)\r\n+ if database_allowed is not None:\r\n+ assert isinstance(database_allowed, list)\r\n+ if action_initials in database_allowed:\r\n+ return None\r\n+ # Or the current table? That's any time the resource is (database, table)\r\n+ if not isinstance(resource, str) and len(resource) == 2:\r\n+ database, table = resource\r\n+ table_allowed = _r.get(\"t\", {}).get(database, {}).get(table)\r\n+ # TODO: What should this do for canned queries?\r\n+ if table_allowed is not None:\r\n+ assert isinstance(table_allowed, list)\r\n+ if action_initials in table_allowed:\r\n+ return None\r\n+ # This action is not specifically allowed, so reject it\r\n+ return False\r\n+\r\n+\r\n @hookimpl\r\n def actor_from_request(datasette, request):\r\n prefix = \"dstok_\"\r\n\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1423336089, "label": "`datasette create-token` ability to create tokens with a reduced set of permissions"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301639741", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301639741, "node_id": "IC_kwDOBm6k_c5NlXI9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:58:21Z", "updated_at": "2022-11-03T04:58:21Z", "author_association": "OWNER", "body": "The whole `database_name` or `(database_name, table_name)` tuple for resource is a bit of a code smell. Maybe this is a chance to tidy that up too?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301639370", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301639370, "node_id": "IC_kwDOBm6k_c5NlXDK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:57:21Z", "updated_at": "2022-11-03T04:57:21Z", "author_association": "OWNER", "body": "The plugin hook would be called `register_permissions()`, for consistency with `register_routes()` and `register_commands()`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301638918", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301638918, "node_id": "IC_kwDOBm6k_c5NlW8G", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:56:06Z", "updated_at": "2022-11-03T04:56:06Z", "author_association": "OWNER", "body": "I've also introduced a new concept of a permission abbreviation, which like the permission name needs to be globally unique.\r\n\r\nThat's a problem for plugins - they might just be able to guarantee that their permission long-form name is unique among other plugins (through sensible naming conventions) but the thing where they declare a initial-letters-only abbreviation is far more risky.\r\n\r\nI think abbreviations are optional - they are provided for core permissions but plugins are advised not to use them.\r\n\r\nAlso Datasette could check that the installed plugins do not provide conflicting permissions on startup and refuse to start if they do.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301638156", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301638156, "node_id": "IC_kwDOBm6k_c5NlWwM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:54:00Z", "updated_at": "2022-11-03T04:54:00Z", "author_association": "OWNER", "body": "If I have the permissions defined like this:\r\n```python\r\nPERMISSIONS = (\r\n Permission(\"view-instance\", \"vi\", False, False, True),\r\n Permission(\"view-database\", \"vd\", True, False, True),\r\n Permission(\"view-database-download\", \"vdd\", True, False, True),\r\n Permission(\"view-table\", \"vt\", True, True, True),\r\n Permission(\"view-query\", \"vq\", True, True, True),\r\n Permission(\"insert-row\", \"ir\", True, True, False),\r\n Permission(\"delete-row\", \"dr\", True, True, False),\r\n Permission(\"drop-table\", \"dt\", True, True, False),\r\n Permission(\"execute-sql\", \"es\", True, False, True),\r\n Permission(\"permissions-debug\", \"pd\", False, False, False),\r\n Permission(\"debug-menu\", \"dm\", False, False, False),\r\n)\r\n```\r\nInstead of just calling them by their undeclared names in places like this:\r\n```python\r\nawait self.ds.permission_allowed(\r\n request.actor, \"execute-sql\", database, default=True\r\n)\r\n```\r\nOn the one hand I can ditch that confusing `default=True` option - whether a permission is on by default becomes a characteristic of that `Permission()` itself, which feels much neater.\r\n\r\nOn the other hand though, plugins that introduce their own permissions - like https://datasette.io/plugins/datasette-edit-schema - will need a way to register those permissions with Datasette core. Probably another plugin hook.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301635906", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301635906, "node_id": "IC_kwDOBm6k_c5NlWNC", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:48:09Z", "updated_at": "2022-11-03T04:48:09Z", "author_association": "OWNER", "body": "I built this prototype on the http://127.0.0.1:8001/-/allow-debug page, which is open to anyone to visit.\r\n\r\nBut... I just realized that using this tool can leak information - you can use it to guess the names of invisible databases and tables and run theoretical permission checks against them.\r\n\r\nUsing the tool also pollutes the list of permission checks that show up on the root-anlo `/-/permissions` page.\r\n\r\nSo.... I'm going to restrict the usage of this tool to users with access to `/-/permissions` and put it on that page instead.\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1434094365, "label": "Tool for simulating permission checks against actors"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1881#issuecomment-1301635340", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1881", "id": 1301635340, "node_id": "IC_kwDOBm6k_c5NlWEM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-11-03T04:46:41Z", "updated_at": "2022-11-03T04:46:41Z", "author_association": "OWNER", "body": "Built this prototype:\r\n\r\n\r\n\r\nIn building it I realized I needed to know which permissions took a table, a database, both or neither. So I had to bake that into the code.\r\n\r\nHere's the prototype so far (which includes a prototype of the logic for the `_r` field on actor, see #1855):\r\n\r\n```diff\r\ndiff --git a/datasette/default_permissions.py b/datasette/default_permissions.py\r\nindex 32b0c758..f68aa38f 100644\r\n--- a/datasette/default_permissions.py\r\n+++ b/datasette/default_permissions.py\r\n@@ -6,8 +6,8 @@ import json\r\n import time\r\n \r\n \r\n-@hookimpl(tryfirst=True)\r\n-def permission_allowed(datasette, actor, action, resource):\r\n+@hookimpl(tryfirst=True, specname=\"permission_allowed\")\r\n+def permission_allowed_default(datasette, actor, action, resource):\r\n async def inner():\r\n if action in (\r\n \"permissions-debug\",\r\n@@ -57,6 +57,44 @@ def permission_allowed(datasette, actor, action, resource):\r\n return inner\r\n \r\n \r\n+@hookimpl(specname=\"permission_allowed\")\r\n+def permission_allowed_actor_restrictions(actor, action, resource):\r\n+ if actor is None:\r\n+ return None\r\n+ _r = actor.get(\"_r\")\r\n+ if not _r:\r\n+ # No restrictions, so we have no opinion\r\n+ return None\r\n+ action_initials = \"\".join([word[0] for word in action.split(\"-\")])\r\n+ # If _r is defined then we use those to further restrict the actor\r\n+ # Crucially, we only use this to say NO (return False) - we never\r\n+ # use it to return YES (True) because that might over-ride other\r\n+ # restrictions placed on this actor\r\n+ all_allowed = _r.get(\"a\")\r\n+ if all_allowed is not None:\r\n+ assert isinstance(all_allowed, list)\r\n+ if action_initials in all_allowed:\r\n+ return None\r\n+ # How about for the current database?\r\n+ if action in (\"view-database\", \"view-database-download\", \"execute-sql\"):\r\n+ database_allowed = _r.get(\"d\", {}).get(resource)\r\n+ if database_allowed is not None:\r\n+ assert isinstance(database_allowed, list)\r\n+ if action_initials in database_allowed:\r\n+ return None\r\n+ # Or the current table? That's any time the resource is (database, table)\r\n+ if not isinstance(resource, str) and len(resource) == 2:\r\n+ database, table = resource\r\n+ table_allowed = _r.get(\"t\", {}).get(database, {}).get(table)\r\n+ # TODO: What should this do for canned queries?\r\n+ if table_allowed is not None:\r\n+ assert isinstance(table_allowed, list)\r\n+ if action_initials in table_allowed:\r\n+ return None\r\n+ # This action is not specifically allowed, so reject it\r\n+ return False\r\n+\r\n+\r\n @hookimpl\r\n def actor_from_request(datasette, request):\r\n prefix = \"dstok_\"\r\ndiff --git a/datasette/templates/allow_debug.html b/datasette/templates/allow_debug.html\r\nindex 0f1b30f0..ae43f0f5 100644\r\n--- a/datasette/templates/allow_debug.html\r\n+++ b/datasette/templates/allow_debug.html\r\n@@ -35,7 +35,7 @@ p.message-warning {\r\n \r\n