![\"image\"](\"https://user-images.githubusercontent.com/9599/169594178-6a779d9b-1388-4ab9-837f-4b8bfbfc6011.png\")
\r\n\r\nThis helps a bit:\r\n\r\n```css\r\n.sidebar-logo-container {\r\n background-color: white;\r\n padding: 5px;\r\n opacity: 0.6;\r\n}\r\n```\r\n\r\n![\"image\"](\"https://user-images.githubusercontent.com/9599/169594107-9c153bee-e2e5-4ba1-90de-ecc0a853e608.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133210942", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133210942, "node_id": "IC_kwDOBm6k_c5Di20-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T18:49:40Z", "updated_at": "2022-05-20T18:49:40Z", "author_association": "OWNER", "body": "And for those local table of contents, do this:\r\n\r\n```rst\r\n.. contents::\r\n :local:\r\n :class: this-will-duplicate-information-and-it-is-still-useful-here\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1746#issuecomment-1133210651", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1746", "id": 1133210651, "node_id": "IC_kwDOBm6k_c5Di2wb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-20T18:49:11Z", "updated_at": "2022-05-20T18:49:11Z", "author_association": "OWNER", "body": "I found a workaround for the no-longer-nested left hand navigation: drop this into `_templates/sidebar/navigation.html`:\r\n```html+jinja\r\n![\"image\"](\"https://user-images.githubusercontent.com/9599/169592991-979865af-511b-43de-bb6c-13bc24e2c88e.png\")
\r\n\r\nI also still want to include those inline tables of contents on the two pages that have them:\r\n\r\n- https://docs.datasette.io/en/stable/installation.html\r\n- https://docs.datasette.io/en/stable/plugin_hooks.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1243498298, "label": "Switch documentation theme to Furo"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1744#issuecomment-1129251699", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1744", "id": 1129251699, "node_id": "IC_kwDOBm6k_c5DTwNz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T19:44:47Z", "updated_at": "2022-05-17T19:46:38Z", "author_association": "OWNER", "body": "Updated docs: https://docs.datasette.io/en/latest/getting_started.html#using-datasette-on-your-own-computer and https://docs.datasette.io/en/latest/cli-reference.html#datasette-serve-help", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239008850, "label": "`--nolock` feature for opening locked databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1745#issuecomment-1129252603", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1745", "id": 1129252603, "node_id": "IC_kwDOBm6k_c5DTwb7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T19:45:51Z", "updated_at": "2022-05-17T19:45:51Z", "author_association": "OWNER", "body": "Now documented here: https://docs.datasette.io/en/latest/contributing.html#running-cog", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239080102, "label": "Documentation on running cog"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1744#issuecomment-1129243427", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1744", "id": 1129243427, "node_id": "IC_kwDOBm6k_c5DTuMj", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T19:35:02Z", "updated_at": "2022-05-17T19:35:02Z", "author_association": "OWNER", "body": "One thing to note is that the `datasette-copy-to-memory` plugin broke with a locked file, because it does this: https://github.com/simonw/datasette-copy-to-memory/blob/d541c18a78ae6f707a8f9b1e7fc4c020a9f68f2e/datasette_copy_to_memory/__init__.py#L27\r\n```python\r\ntmp.execute(\"ATTACH DATABASE ? AS _copy_from\", [db.path])\r\n```\r\nThat would need to use a URI filename too for it to work with locked files.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239008850, "label": "`--nolock` feature for opening locked databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1744#issuecomment-1129241873", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1744", "id": 1129241873, "node_id": "IC_kwDOBm6k_c5DTt0R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T19:33:16Z", "updated_at": "2022-05-17T19:33:16Z", "author_association": "OWNER", "body": "I'm going to skip adding a test for this - the test logic would have to be pretty convoluted to exercise it properly, and it's a pretty minor and low-risk feature in the scheme of things.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239008850, "label": "`--nolock` feature for opening locked databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1744#issuecomment-1129241283", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1744", "id": 1129241283, "node_id": "IC_kwDOBm6k_c5DTtrD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T19:32:35Z", "updated_at": "2022-05-17T19:32:35Z", "author_association": "OWNER", "body": "I tried writing a test like this:\r\n\r\n```python\r\n@pytest.mark.parametrize(\"locked\", (True, False))\r\ndef test_locked_sqlite_db(tmp_path_factory, locked):\r\n dir = tmp_path_factory.mktemp(\"test_locked_sqlite_db\")\r\n test_db = str(dir / \"test.db\")\r\n sqlite3.connect(test_db).execute(\"create table t (id integer primary key)\")\r\n if locked:\r\n fp = open(test_db, \"w\")\r\n fcntl.lockf(fp.fileno(), fcntl.LOCK_EX)\r\n runner = CliRunner()\r\n result = runner.invoke(\r\n cli,\r\n [\r\n \"serve\",\r\n \"--memory\",\r\n \"--get\",\r\n \"/test\",\r\n ],\r\n catch_exceptions=False,\r\n )\r\n```\r\nBut it didn't work, because the test runs in the same process - so taking an exclusive lock on that file didn't cause an error when the test later tried to access it via Datasette!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239008850, "label": "`--nolock` feature for opening locked databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1744#issuecomment-1129187486", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1744", "id": 1129187486, "node_id": "IC_kwDOBm6k_c5DTgie", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T18:28:49Z", "updated_at": "2022-05-17T18:28:49Z", "author_association": "OWNER", "body": "I think I do that with `fcntl.flock()`: https://docs.python.org/3/library/fcntl.html#fcntl.flock", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239008850, "label": "`--nolock` feature for opening locked databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1744#issuecomment-1129185356", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1744", "id": 1129185356, "node_id": "IC_kwDOBm6k_c5DTgBM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T18:26:26Z", "updated_at": "2022-05-17T18:26:26Z", "author_association": "OWNER", "body": "Not sure how to test this - I'd need to open my own lock against a database file somehow.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239008850, "label": "`--nolock` feature for opening locked databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1744#issuecomment-1129184908", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1744", "id": 1129184908, "node_id": "IC_kwDOBm6k_c5DTf6M", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-17T18:25:57Z", "updated_at": "2022-05-17T18:25:57Z", "author_association": "OWNER", "body": "I knocked out a quick prototype of this and it worked!\r\n\r\n datasette ~/Library/Application\\ Support/Google/Chrome/Default/History --nolock\r\n\r\nHere's the prototype diff:\r\n\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex b7b8437..f43700d 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -213,6 +213,7 @@ class Datasette:\r\n config_dir=None,\r\n pdb=False,\r\n crossdb=False,\r\n+ nolock=False,\r\n ):\r\n assert config_dir is None or isinstance(\r\n config_dir, Path\r\n@@ -238,6 +239,7 @@ class Datasette:\r\n self.databases = collections.OrderedDict()\r\n self._refresh_schemas_lock = asyncio.Lock()\r\n self.crossdb = crossdb\r\n+ self.nolock = nolock\r\n if memory or crossdb or not self.files:\r\n self.add_database(Database(self, is_memory=True), name=\"_memory\")\r\n # memory_name is a random string so that each Datasette instance gets its own\r\ndiff --git a/datasette/cli.py b/datasette/cli.py\r\nindex 3c6e1b2..7e44665 100644\r\n--- a/datasette/cli.py\r\n+++ b/datasette/cli.py\r\n@@ -452,6 +452,11 @@ def uninstall(packages, yes):\r\n is_flag=True,\r\n help=\"Enable cross-database joins using the /_memory database\",\r\n )\r\n+@click.option(\r\n+ \"--nolock\",\r\n+ is_flag=True,\r\n+ help=\"Ignore locking and open locked files in read-only mode\",\r\n+)\r\n @click.option(\r\n \"--ssl-keyfile\",\r\n help=\"SSL key file\",\r\n@@ -486,6 +491,7 @@ def serve(\r\n open_browser,\r\n create,\r\n crossdb,\r\n+ nolock,\r\n ssl_keyfile,\r\n ssl_certfile,\r\n return_instance=False,\r\n@@ -545,6 +551,7 @@ def serve(\r\n version_note=version_note,\r\n pdb=pdb,\r\n crossdb=crossdb,\r\n+ nolock=nolock,\r\n )\r\n \r\n # if files is a single directory, use that as config_dir=\r\ndiff --git a/datasette/database.py b/datasette/database.py\r\nindex 44d3266..fa55804 100644\r\n--- a/datasette/database.py\r\n+++ b/datasette/database.py\r\n@@ -89,6 +89,8 @@ class Database:\r\n # mode=ro or immutable=1?\r\n if self.is_mutable:\r\n qs = \"?mode=ro\"\r\n+ if self.ds.nolock:\r\n+ qs += \"&nolock=1\"\r\n else:\r\n qs = \"?immutable=1\"\r\n assert not (write and not self.is_mutable)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1239008850, "label": "`--nolock` feature for opening locked databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1742#issuecomment-1128052948", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1742", "id": 1128052948, "node_id": "IC_kwDOBm6k_c5DPLjU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-16T19:28:31Z", "updated_at": "2022-05-16T19:28:31Z", "author_association": "OWNER", "body": "The trace mechanism is a bit gnarly - it's actually done by some ASGI middleware I wrote, so I'm pretty sure the bug is in there somewhere: https://github.com/simonw/datasette/blob/280ff372ab30df244f6c54f6f3002da57334b3d7/datasette/tracer.py#L73", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1237586379, "label": "?_trace=1 fails with datasette-geojson for some reason"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1742#issuecomment-1128033018", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1742", "id": 1128033018, "node_id": "IC_kwDOBm6k_c5DPGr6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-16T19:06:38Z", "updated_at": "2022-05-16T19:06:38Z", "author_association": "OWNER", "body": "The same URL with `.json` instead works fine: https://calands.datasettes.com/calands/CPAD_2020a_SuperUnits.json?_sort=id&id__exact=4&_labels=on&_trace=1", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1237586379, "label": "?_trace=1 fails with datasette-geojson for some reason"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1739#issuecomment-1117662420", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1739", "id": 1117662420, "node_id": "IC_kwDOBm6k_c5CnizU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-04T18:21:18Z", "updated_at": "2022-05-04T18:21:18Z", "author_association": "OWNER", "body": "That prototype is now public: https://github.com/simonw/datasette-lite", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223699280, "label": ".db downloads should be served with an ETag"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1739#issuecomment-1116215371", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1739", "id": 1116215371, "node_id": "IC_kwDOBm6k_c5CiBhL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-03T15:12:16Z", "updated_at": "2022-05-03T15:12:16Z", "author_association": "OWNER", "body": "That worked - both DBs are 304 for me now on a subsequent load of the page:\r\n\r\n![\"CleanShot](\"https://user-images.githubusercontent.com/9599/166481669-9570d225-78d4-461b-88c8-044f02b68a64.png\")
\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223699280, "label": ".db downloads should be served with an ETag"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1739#issuecomment-1116183369", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1739", "id": 1116183369, "node_id": "IC_kwDOBm6k_c5Ch5tJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-03T14:43:14Z", "updated_at": "2022-05-03T14:43:14Z", "author_association": "OWNER", "body": "Relevant tests start here: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/tests/test_html.py#L395", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223699280, "label": ".db downloads should be served with an ETag"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1739#issuecomment-1116180599", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1739", "id": 1116180599, "node_id": "IC_kwDOBm6k_c5Ch5B3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-03T14:40:32Z", "updated_at": "2022-05-03T14:40:32Z", "author_association": "OWNER", "body": "Database downloads are served here: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/datasette/views/database.py#L186-L192\r\n\r\nHere's `AsgiFileDownload`: https://github.com/simonw/datasette/blob/d60f163528f466b1127b2935c3b6869c34fd6545/datasette/utils/asgi.py#L410-L430\r\n\r\nI can add an `etag=` parameter to that and populate it with `db.hash`, if it is populated (which it always should be for immutable databases that can be downloaded).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223699280, "label": ".db downloads should be served with an ETag"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1739#issuecomment-1116178727", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1739", "id": 1116178727, "node_id": "IC_kwDOBm6k_c5Ch4kn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-03T14:38:46Z", "updated_at": "2022-05-03T14:38:46Z", "author_association": "OWNER", "body": "Reminded myself how this works by reviewing `conditional-get`: https://github.com/simonw/conditional-get/blob/db6dfec0a296080aaf68fcd80e55fb3f0714e738/conditional_get/cli.py#L33-L52\r\n\r\nSimply add a `If-None-Match: last-known-etag` header to the request and check that the response is a status 304 with an empty body.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223699280, "label": ".db downloads should be served with an ETag"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1739#issuecomment-1115760104", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1739", "id": 1115760104, "node_id": "IC_kwDOBm6k_c5CgSXo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-03T05:50:19Z", "updated_at": "2022-05-03T05:50:19Z", "author_association": "OWNER", "body": "Here's how Starlette does it: https://github.com/encode/starlette/blob/830f3486537916bae6b46948ff922adc14a22b7c/starlette/staticfiles.py#L213", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223699280, "label": ".db downloads should be served with an ETag"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1732#issuecomment-1115533820", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1732", "id": 1115533820, "node_id": "IC_kwDOBm6k_c5CfbH8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-03T01:42:25Z", "updated_at": "2022-05-03T01:42:25Z", "author_association": "OWNER", "body": "Thanks, this definitely sounds like a bug. Do you have simple steps to reproduce this?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1221849746, "label": "Custom page variables aren't decoded"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1737#issuecomment-1115470180", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1737", "id": 1115470180, "node_id": "IC_kwDOBm6k_c5CfLlk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T23:39:29Z", "updated_at": "2022-05-02T23:39:29Z", "author_association": "OWNER", "body": "Test ran in 38 seconds and passed! https://github.com/simonw/datasette/runs/6265954274?check_suite_focus=true\r\n\r\nI'm going to have it run on every commit and PR.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223459734, "label": "Automated test for Pyodide compatibility"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1737#issuecomment-1115468193", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1737", "id": 1115468193, "node_id": "IC_kwDOBm6k_c5CfLGh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T23:35:26Z", "updated_at": "2022-05-02T23:35:26Z", "author_association": "OWNER", "body": "https://github.com/simonw/datasette/runs/6265915080?check_suite_focus=true failed but looks like it passed because I forgot to use `set -e` at the start of the bash script.\r\n\r\nIt failed because it didn't have `build` available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223459734, "label": "Automated test for Pyodide compatibility"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1737#issuecomment-1115464097", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1737", "id": 1115464097, "node_id": "IC_kwDOBm6k_c5CfKGh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T23:27:40Z", "updated_at": "2022-05-02T23:27:40Z", "author_association": "OWNER", "body": "I'm going to start off by running this manually - I may run it on every commit once this is all a little bit more stable.\r\n\r\nI can base the workflow on https://github.com/simonw/scrape-hacker-news-by-domain/blob/main/.github/workflows/scrape.yml", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223459734, "label": "Automated test for Pyodide compatibility"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1737#issuecomment-1115462720", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1737", "id": 1115462720, "node_id": "IC_kwDOBm6k_c5CfJxA", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T23:25:03Z", "updated_at": "2022-05-02T23:25:03Z", "author_association": "OWNER", "body": "Here's a script that seems to work. It builds the wheel, starts a Python web server that serves the wheel, runs a test with `shot-scraper` and then shuts down the server again.\r\n\r\n```bash\r\n#!/bin/bash\r\n\r\n# Build the wheel\r\npython3 -m build\r\n\r\n# Find name of wheel\r\nwheel=$(basename $(ls dist/*.whl))\r\n# strip off the dist/\r\n\r\n\r\n# Create a blank index page\r\necho '\r\n\r\n' > dist/index.html\r\n\r\n# Run a server for that dist/ folder\r\ncd dist\r\npython3 -m http.server 8529 &\r\ncd ..\r\n\r\nshot-scraper javascript http://localhost:8529/ \"\r\nasync () => {\r\n let pyodide = await loadPyodide();\r\n await pyodide.loadPackage(['micropip', 'ssl', 'setuptools']);\r\n let output = await pyodide.runPythonAsync(\\`\r\n import micropip\r\n await micropip.install('h11==0.12.0')\r\n await micropip.install('http://localhost:8529/$wheel')\r\n import ssl\r\n import setuptools\r\n from datasette.app import Datasette\r\n ds = Datasette(memory=True, settings={'num_sql_threads': 0})\r\n (await ds.client.get('/_memory.json?sql=select+55+as+itworks&_shape=array')).text\r\n \\`);\r\n if (JSON.parse(output)[0].itworks != 55) {\r\n throw 'Got ' + output + ', expected itworks: 55';\r\n }\r\n return 'Test passed!';\r\n}\r\n\"\r\n\r\n# Shut down the server\r\npkill -f 'http.server 8529'\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223459734, "label": "Automated test for Pyodide compatibility"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115404729", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115404729, "node_id": "IC_kwDOBm6k_c5Ce7m5", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T21:49:01Z", "updated_at": "2022-05-02T21:49:38Z", "author_association": "OWNER", "body": "That alpha release works!\r\n\r\nhttps://pyodide.org/en/stable/console.html\r\n\r\n```pycon\r\nWelcome to the Pyodide terminal emulator \ud83d\udc0d\r\nPython 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import micropip\r\n>>> await micropip.install(\"datasette==0.62a0\")\r\n>>> import ssl\r\n>>> import setuptools\r\n>>> from datasette.app import Datasette\r\n>>> ds = Datasette(memory=True, settings={\"num_sql_threads\": 0})\r\n>>> await ds.client.get(\"/.json\")\r\n\r\n\\n\\n\\n\r\nr detailsClickedWithin = null;\\n while (target && target.tagName != \\'DETAILS\\') {\\n target = target.parentNode;\\\r\nn }\\n if (target && target.tagName == \\'DETAILS\\') {\\n detailsClickedWithin = target;\\n }\\n Array.from(d\r\nocument.getElementsByTagName(\\'details\\')).filter(\\n (details) => details.open && details != detailsClickedWithin\\n \r\n ).forEach(details => details.open = false);\\n});\\n\\n\\n\\n\\n\\n\\n\r\n'\r\n>>> \r\n```\r\n\r\nThat `ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed` error is annoying. I assume it's a `uvicorn` dependency clash of some sort, because I wasn't getting that when I removed `uvicorn` as a dependency.\r\n\r\nI can avoid it by running this first though:\r\n\r\n await micropip.install(\"h11==0.12\")", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1735#issuecomment-1115301733", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1735", "id": 1115301733, "node_id": "IC_kwDOBm6k_c5Ceidl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:57:19Z", "updated_at": "2022-05-02T19:59:03Z", "author_association": "OWNER", "body": "This code breaks if that setting is 0:\r\n\r\nhttps://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/app.py#L291-L293\r\n\r\nIt's used here:\r\n\r\nhttps://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/database.py#L188-L190", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223263540, "label": "Datasette setting to disable threading (for Pyodide)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115288284", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115288284, "node_id": "IC_kwDOBm6k_c5CefLc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:40:33Z", "updated_at": "2022-05-02T19:40:33Z", "author_association": "OWNER", "body": "I'll release this as a `0.62a0` as soon as it's ready, so I can start testing it out in Pyodide for real.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1734#issuecomment-1115283922", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1734", "id": 1115283922, "node_id": "IC_kwDOBm6k_c5CeeHS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:35:32Z", "updated_at": "2022-05-02T19:35:32Z", "author_association": "OWNER", "body": "I'll use my original from 2009: https://www.djangosnippets.org/snippets/1431/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223241647, "label": "Remove python-baseconv dependency"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1734#issuecomment-1115282773", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1734", "id": 1115282773, "node_id": "IC_kwDOBm6k_c5Ced1V", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:34:15Z", "updated_at": "2022-05-02T19:34:15Z", "author_association": "OWNER", "body": "I'm going to vendor it and update the documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223241647, "label": "Remove python-baseconv dependency"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115278325, "node_id": "IC_kwDOBm6k_c5Cecv1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:29:05Z", "updated_at": "2022-05-02T19:29:05Z", "author_association": "OWNER", "body": "I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case.\r\n\r\nI thought about adding a new setting, then I noticed this:\r\n\r\n datasette mydatabase.db --setting num_sql_threads 10\r\n\r\nI'm going to let users set that to `0` to disable threaded execution of SQL queries.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115268245", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115268245, "node_id": "IC_kwDOBm6k_c5CeaSV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:18:11Z", "updated_at": "2022-05-02T19:18:11Z", "author_association": "OWNER", "body": "Maybe I can leave `uvicorn` as a dependency? Installing it works OK, it only generates errors when you try to import it:\r\n\r\n```pycon\r\nWelcome to the Pyodide terminal emulator \ud83d\udc0d\r\nPython 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import micropip\r\n>>> await micropip.install(\"uvicorn\")\r\n>>> import uvicorn\r\nTraceback (most recent call last):\r\n File \"
Datasette
\\n\\n\\n\\n\\n\\n\r\n\r\nr detailsClickedWithin = null;\\n while (target && target.tagName != \\'DETAILS\\') {\\n target = target.parentNode;\\\r\nn }\\n if (target && target.tagName == \\'DETAILS\\') {\\n detailsClickedWithin = target;\\n }\\n Array.from(d\r\nocument.getElementsByTagName(\\'details\\')).filter(\\n (details) => details.open && details != detailsClickedWithin\\n \r\n ).forEach(details => details.open = false);\\n});\\n\\n\\n\\n\\n\\n\\n\r\n'\r\n>>> \r\n```\r\n\r\nThat `ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed` error is annoying. I assume it's a `uvicorn` dependency clash of some sort, because I wasn't getting that when I removed `uvicorn` as a dependency.\r\n\r\nI can avoid it by running this first though:\r\n\r\n await micropip.install(\"h11==0.12\")", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1735#issuecomment-1115301733", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1735", "id": 1115301733, "node_id": "IC_kwDOBm6k_c5Ceidl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:57:19Z", "updated_at": "2022-05-02T19:59:03Z", "author_association": "OWNER", "body": "This code breaks if that setting is 0:\r\n\r\nhttps://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/app.py#L291-L293\r\n\r\nIt's used here:\r\n\r\nhttps://github.com/simonw/datasette/blob/a29c1277896b6a7905ef5441c42a37bc15f67599/datasette/database.py#L188-L190", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223263540, "label": "Datasette setting to disable threading (for Pyodide)"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115288284", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115288284, "node_id": "IC_kwDOBm6k_c5CefLc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:40:33Z", "updated_at": "2022-05-02T19:40:33Z", "author_association": "OWNER", "body": "I'll release this as a `0.62a0` as soon as it's ready, so I can start testing it out in Pyodide for real.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1734#issuecomment-1115283922", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1734", "id": 1115283922, "node_id": "IC_kwDOBm6k_c5CeeHS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:35:32Z", "updated_at": "2022-05-02T19:35:32Z", "author_association": "OWNER", "body": "I'll use my original from 2009: https://www.djangosnippets.org/snippets/1431/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223241647, "label": "Remove python-baseconv dependency"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1734#issuecomment-1115282773", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1734", "id": 1115282773, "node_id": "IC_kwDOBm6k_c5Ced1V", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:34:15Z", "updated_at": "2022-05-02T19:34:15Z", "author_association": "OWNER", "body": "I'm going to vendor it and update the documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223241647, "label": "Remove python-baseconv dependency"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115278325", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115278325, "node_id": "IC_kwDOBm6k_c5Cecv1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:29:05Z", "updated_at": "2022-05-02T19:29:05Z", "author_association": "OWNER", "body": "I'm going to add a Datasette setting to disable threading entirely, designed for usage in this particular case.\r\n\r\nI thought about adding a new setting, then I noticed this:\r\n\r\n datasette mydatabase.db --setting num_sql_threads 10\r\n\r\nI'm going to let users set that to `0` to disable threaded execution of SQL queries.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115268245", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115268245, "node_id": "IC_kwDOBm6k_c5CeaSV", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:18:11Z", "updated_at": "2022-05-02T19:18:11Z", "author_association": "OWNER", "body": "Maybe I can leave `uvicorn` as a dependency? Installing it works OK, it only generates errors when you try to import it:\r\n\r\n```pycon\r\nWelcome to the Pyodide terminal emulator \ud83d\udc0d\r\nPython 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import micropip\r\n>>> await micropip.install(\"uvicorn\")\r\n>>> import uvicorn\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/lib/python3.10/site-packages/uvicorn/__init__.py\", line 1, in \r\n from uvicorn.config import Config\r\n File \"/lib/python3.10/site-packages/uvicorn/config.py\", line 8, in \r\n import ssl\r\n File \"/lib/python3.10/ssl.py\", line 98, in \r\n import _ssl # if we can't import it, let the error propagate\r\nModuleNotFoundError: No module named '_ssl'\r\n>>> import ssl\r\n>>> import uvicorn\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/lib/python3.10/site-packages/uvicorn/__init__.py\", line 2, in \r\n from uvicorn.main import Server, main, run\r\n File \"/lib/python3.10/site-packages/uvicorn/main.py\", line 24, in \r\n from uvicorn.supervisors import ChangeReload, Multiprocess\r\n File \"/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py\", line 3, in \r\n from uvicorn.supervisors.basereload import BaseReload\r\n File \"/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py\", line 12, in \r\n from uvicorn.subprocess import get_subprocess\r\n File \"/lib/python3.10/site-packages/uvicorn/subprocess.py\", line 14, in \r\n multiprocessing.allow_connection_pickling()\r\n File \"/lib/python3.10/multiprocessing/context.py\", line 170, in allow_connection_pickling\r\n from . import connection\r\n File \"/lib/python3.10/multiprocessing/connection.py\", line 21, in \r\n import _multiprocessing\r\nModuleNotFoundError: No module named '_multiprocessing'\r\n>>> import multiprocessing\r\n>>> import uvicorn\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/lib/python3.10/site-packages/uvicorn/__init__.py\", line 2, in \r\n from uvicorn.main import Server, main, run\r\n File \"/lib/python3.10/site-packages/uvicorn/main.py\", line 24, in \r\n from uvicorn.supervisors import ChangeReload, Multiprocess\r\n File \"/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py\", line 3, in \r\n from uvicorn.supervisors.basereload import BaseReload\r\n File \"/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py\", line 12, in \r\n from uvicorn.subprocess import get_subprocess\r\n File \"/lib/python3.10/site-packages/uvicorn/subprocess.py\", line 14, in \r\n multiprocessing.allow_connection_pickling()\r\n File \"/lib/python3.10/multiprocessing/context.py\", line 170, in allow_connection_pickling\r\n from . import connection\r\n File \"/lib/python3.10/multiprocessing/connection.py\", line 21, in \r\n import _multiprocessing\r\nModuleNotFoundError: No module named '_multiprocessing'\r\n>>> \r\n```\r\nSince the `import ssl` trick fixed the `_ssl` error I was hopeful that `import multiprocessing` could fix the `_multiprocessing` one, but sadly it did not.\r\n\r\nBut it looks like i can address this issue just by making `import uvicorn` in `app.py` an optional import.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115262218", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115262218, "node_id": "IC_kwDOBm6k_c5CeY0K", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:11:51Z", "updated_at": "2022-05-02T19:14:01Z", "author_association": "OWNER", "body": "Here's the full diff I applied to Datasette to get it fully working in Pyodide:\r\n\r\nhttps://github.com/simonw/datasette/compare/94a3171b01fde5c52697aeeff052e3ad4bab5391...8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81\r\n\r\nAnd as a visible diff:\r\n\r\n```diff\r\ndiff --git a/datasette/app.py b/datasette/app.py\r\nindex d269372..6c0c5fc 100644\r\n--- a/datasette/app.py\r\n+++ b/datasette/app.py\r\n@@ -15,7 +15,6 @@ import pkg_resources\r\n import re\r\n import secrets\r\n import sys\r\n-import threading\r\n import traceback\r\n import urllib.parse\r\n from concurrent import futures\r\n@@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer\r\n from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader\r\n from jinja2.environment import Template\r\n from jinja2.exceptions import TemplateNotFound\r\n-import uvicorn\r\n \r\n from .views.base import DatasetteError, ureg\r\n from .views.database import DatabaseDownload, DatabaseView\r\n@@ -813,7 +811,6 @@ class Datasette:\r\n },\r\n \"datasette\": datasette_version,\r\n \"asgi\": \"3.0\",\r\n- \"uvicorn\": uvicorn.__version__,\r\n \"sqlite\": {\r\n \"version\": sqlite_version,\r\n \"fts_versions\": fts_versions,\r\n@@ -854,23 +851,7 @@ class Datasette:\r\n ]\r\n \r\n def _threads(self):\r\n- threads = list(threading.enumerate())\r\n- d = {\r\n- \"num_threads\": len(threads),\r\n- \"threads\": [\r\n- {\"name\": t.name, \"ident\": t.ident, \"daemon\": t.daemon} for t in threads\r\n- ],\r\n- }\r\n- # Only available in Python 3.7+\r\n- if hasattr(asyncio, \"all_tasks\"):\r\n- tasks = asyncio.all_tasks()\r\n- d.update(\r\n- {\r\n- \"num_tasks\": len(tasks),\r\n- \"tasks\": [_cleaner_task_str(t) for t in tasks],\r\n- }\r\n- )\r\n- return d\r\n+ return {\"num_threads\": 0, \"threads\": []}\r\n \r\n def _actor(self, request):\r\n return {\"actor\": request.actor}\r\ndiff --git a/datasette/database.py b/datasette/database.py\r\nindex ba594a8..b50142d 100644\r\n--- a/datasette/database.py\r\n+++ b/datasette/database.py\r\n@@ -4,7 +4,6 @@ from pathlib import Path\r\n import janus\r\n import queue\r\n import sys\r\n-import threading\r\n import uuid\r\n \r\n from .tracer import trace\r\n@@ -21,8 +20,6 @@ from .utils import (\r\n )\r\n from .inspect import inspect_hash\r\n \r\n-connections = threading.local()\r\n-\r\n AttachedDatabase = namedtuple(\"AttachedDatabase\", (\"seq\", \"name\", \"file\"))\r\n \r\n \r\n@@ -43,12 +40,12 @@ class Database:\r\n self.hash = None\r\n self.cached_size = None\r\n self._cached_table_counts = None\r\n- self._write_thread = None\r\n- self._write_queue = None\r\n if not self.is_mutable and not self.is_memory:\r\n p = Path(path)\r\n self.hash = inspect_hash(p)\r\n self.cached_size = p.stat().st_size\r\n+ self._read_connection = None\r\n+ self._write_connection = None\r\n \r\n @property\r\n def cached_table_counts(self):\r\n@@ -134,60 +131,17 @@ class Database:\r\n return results\r\n \r\n async def execute_write_fn(self, fn, block=True):\r\n- task_id = uuid.uuid5(uuid.NAMESPACE_DNS, \"datasette.io\")\r\n- if self._write_queue is None:\r\n- self._write_queue = queue.Queue()\r\n- if self._write_thread is None:\r\n- self._write_thread = threading.Thread(\r\n- target=self._execute_writes, daemon=True\r\n- )\r\n- self._write_thread.start()\r\n- reply_queue = janus.Queue()\r\n- self._write_queue.put(WriteTask(fn, task_id, reply_queue))\r\n- if block:\r\n- result = await reply_queue.async_q.get()\r\n- if isinstance(result, Exception):\r\n- raise result\r\n- else:\r\n- return result\r\n- else:\r\n- return task_id\r\n-\r\n- def _execute_writes(self):\r\n- # Infinite looping thread that protects the single write connection\r\n- # to this database\r\n- conn_exception = None\r\n- conn = None\r\n- try:\r\n- conn = self.connect(write=True)\r\n- self.ds._prepare_connection(conn, self.name)\r\n- except Exception as e:\r\n- conn_exception = e\r\n- while True:\r\n- task = self._write_queue.get()\r\n- if conn_exception is not None:\r\n- result = conn_exception\r\n- else:\r\n- try:\r\n- result = task.fn(conn)\r\n- except Exception as e:\r\n- sys.stderr.write(\"{}\\n\".format(e))\r\n- sys.stderr.flush()\r\n- result = e\r\n- task.reply_queue.sync_q.put(result)\r\n+ # We always treat it as if block=True now\r\n+ if self._write_connection is None:\r\n+ self._write_connection = self.connect(write=True)\r\n+ self.ds._prepare_connection(self._write_connection, self.name)\r\n+ return fn(self._write_connection)\r\n \r\n async def execute_fn(self, fn):\r\n- def in_thread():\r\n- conn = getattr(connections, self.name, None)\r\n- if not conn:\r\n- conn = self.connect()\r\n- self.ds._prepare_connection(conn, self.name)\r\n- setattr(connections, self.name, conn)\r\n- return fn(conn)\r\n-\r\n- return await asyncio.get_event_loop().run_in_executor(\r\n- self.ds.executor, in_thread\r\n- )\r\n+ if self._read_connection is None:\r\n+ self._read_connection = self.connect()\r\n+ self.ds._prepare_connection(self._read_connection, self.name)\r\n+ return fn(self._read_connection)\r\n \r\n async def execute(\r\n self,\r\ndiff --git a/setup.py b/setup.py\r\nindex 7f0562f..c41669c 100644\r\n--- a/setup.py\r\n+++ b/setup.py\r\n@@ -44,20 +44,20 @@ setup(\r\n install_requires=[\r\n \"asgiref>=3.2.10,<3.6.0\",\r\n \"click>=7.1.1,<8.2.0\",\r\n- \"click-default-group~=1.2.2\",\r\n+ # \"click-default-group~=1.2.2\",\r\n \"Jinja2>=2.10.3,<3.1.0\",\r\n \"hupper~=1.9\",\r\n \"httpx>=0.20\",\r\n \"pint~=0.9\",\r\n \"pluggy>=1.0,<1.1\",\r\n- \"uvicorn~=0.11\",\r\n+ # \"uvicorn~=0.11\",\r\n \"aiofiles>=0.4,<0.9\",\r\n \"janus>=0.6.2,<1.1\",\r\n \"asgi-csrf>=0.9\",\r\n \"PyYAML>=5.3,<7.0\",\r\n \"mergedeep>=1.1.1,<1.4.0\",\r\n \"itsdangerous>=1.1,<3.0\",\r\n- \"python-baseconv==1.2.2\",\r\n+ # \"python-baseconv==1.2.2\",\r\n ],\r\n entry_points=\"\"\"\r\n [console_scripts]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1734#issuecomment-1115260999", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1734", "id": 1115260999, "node_id": "IC_kwDOBm6k_c5CeYhH", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:10:34Z", "updated_at": "2022-05-02T19:10:34Z", "author_association": "OWNER", "body": "This is actually mostly a documentation thing: here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time\r\n\r\nIn the code it's only used in these two places:\r\n\r\nhttps://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20\r\n\r\nhttps://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_auth.py#L56-L60", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223241647, "label": "Remove python-baseconv dependency"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115258737", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115258737, "node_id": "IC_kwDOBm6k_c5CeX9x", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:08:17Z", "updated_at": "2022-05-02T19:08:17Z", "author_association": "OWNER", "body": "I was going to vendor `baseconv.py`, but then I reconsidered - what if there are plugins out there that expect `import baseconv` to work because they have dependend on Datasette?\r\n\r\nI used https://cs.github.com/ and as far as I can tell there aren't any!\r\n\r\nSo I'm going to remove that dependency and work out a smarter way to do this - probably by providing a utility function within Datasette itself.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1733#issuecomment-1115256318", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1733", "id": 1115256318, "node_id": "IC_kwDOBm6k_c5CeXX-", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T19:05:55Z", "updated_at": "2022-05-02T19:05:55Z", "author_association": "OWNER", "body": "I released a `click-default-group-wheel` package to solve that dependency issue. I've already upgraded `sqlite-utils` to that, so now you can use that in Pyodide:\r\n\r\n- https://github.com/simonw/sqlite-utils/pull/429\r\n\r\n`python-baseconv` is only used for actor cookie expiration times:\r\n\r\nhttps://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/actor_auth_cookie.py#L16-L20\r\n\r\nDatasette never actually sets that cookie itself - it instead encourages plugins to set it in the authentication documentation here: https://docs.datasette.io/en/0.61.1/authentication.html#including-an-expiry-time", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223234932, "label": "Get Datasette compatible with Pyodide"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115196863", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429", "id": 1115196863, "node_id": "IC_kwDOCGYnMM5CeI2_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T18:03:47Z", "updated_at": "2022-05-02T18:52:42Z", "author_association": "OWNER", "body": "I made a build of this branch and tested it like this: https://pyodide.org/en/stable/console.html\r\n\r\n```pycon\r\n>>> import micropip\r\n>>> await micropip.install(\"https://s3.amazonaws.com/simonwillison-cors-allowed-public/sqlite_utils-3.26-py3-none-any.whl\")\r\n>>> import sqlite_utils\r\n>>> db = sqlite_utils.Database(memory=True)\r\n>>> list(db.query(\"select 32443 + 55\"))\r\n[{'32443 + 55': 32498}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223177069, "label": "Depend on click-default-group-wheel"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/pull/429#issuecomment-1115197644", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/429", "id": 1115197644, "node_id": "IC_kwDOCGYnMM5CeJDM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-05-02T18:04:28Z", "updated_at": "2022-05-02T18:04:28Z", "author_association": "OWNER", "body": "I'm going to ship this straight away as `3.26.1`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1223177069, "label": "Depend on click-default-group-wheel"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1114058210", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1114058210, "node_id": "IC_kwDOBm6k_c5CZy3i", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T21:39:34Z", "updated_at": "2022-04-30T21:39:34Z", "author_association": "OWNER", "body": "Something to consider if I look into subprocesses for parallel query execution:\r\n\r\nhttps://sqlite.org/howtocorrupt.html#_carrying_an_open_database_connection_across_a_fork_\r\n\r\n> Do not open an SQLite database connection, then fork(), then try to use that database connection in the child process. All kinds of locking problems will result and you can easily end up with a corrupt database. SQLite is not designed to support that kind of behavior. Any database connection that is used in a child process must be opened in the child process, not inherited from the parent. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114038259", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114038259, "node_id": "IC_kwDOBm6k_c5CZt_z", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T19:06:03Z", "updated_at": "2022-04-30T19:06:03Z", "author_association": "OWNER", "body": "> but actually the facet results would be better if they were a list rather than a dictionary\r\n\r\nI think `facet_results` in the JSON should match this (used by the HTML) instead:\r\n\r\nhttps://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/views/table.py#L737-L741\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114036946", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114036946, "node_id": "IC_kwDOBm6k_c5CZtrS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T18:56:25Z", "updated_at": "2022-04-30T19:04:03Z", "author_association": "OWNER", "body": "Related:\r\n- #1558\r\n\r\nWhich talks about how there was confusion in this example: https://latest.datasette.io/fixtures/facetable.json?_facet=created&_facet_date=created&_facet=tags&_facet_array=tags&_nosuggest=1&_size=0\r\n\r\nWhich I fixed in #625 by introducing `tags` and `tags_2` keys, but actually the facet results would be better if they were a list rather than a dictionary.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114037521", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114037521, "node_id": "IC_kwDOBm6k_c5CZt0R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T19:01:07Z", "updated_at": "2022-04-30T19:01:07Z", "author_association": "OWNER", "body": "I had to look up what `hideable` means - it means that you can't hide the current facet because it was defined in metadata, not as a `?_facet=` parameter:\r\n\r\nhttps://github.com/simonw/datasette/blob/4e47a2d894b96854348343374c8e97c9d7055cf6/datasette/facets.py#L228\r\n\r\nThat's a bit of a weird thing to expose in the API. Maybe change that to `source` so it can be `metadata` or `request`? That's very slightly less coupled to how the UI works.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1114013757", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1114013757, "node_id": "IC_kwDOBm6k_c5CZoA9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-30T16:15:51Z", "updated_at": "2022-04-30T18:54:39Z", "author_association": "OWNER", "body": "Deployed a preview of this here: https://latest-1-0-alpha.datasette.io/\r\n\r\nExamples:\r\n\r\n- https://latest-1-0-alpha.datasette.io/fixtures/facetable.json\r\n- https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count\r\n\r\nSecond example produces:\r\n\r\n```json\r\n{\r\n \"rows\": [],\r\n \"next\": null,\r\n \"next_url\": null,\r\n \"count\": 15,\r\n \"facet_results\": {\r\n \"state\": {\r\n \"name\": \"state\",\r\n \"type\": \"column\",\r\n \"hideable\": true,\r\n \"toggle_url\": \"/fixtures/facetable.json?_size=0&_extra=facet_results&_extra=count\",\r\n \"results\": [\r\n {\r\n \"value\": \"CA\",\r\n \"label\": \"CA\",\r\n \"count\": 10,\r\n \"toggle_url\": \"https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=CA\",\r\n \"selected\": false\r\n },\r\n {\r\n \"value\": \"MI\",\r\n \"label\": \"MI\",\r\n \"count\": 4,\r\n \"toggle_url\": \"https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MI\",\r\n \"selected\": false\r\n },\r\n {\r\n \"value\": \"MC\",\r\n \"label\": \"MC\",\r\n \"count\": 1,\r\n \"toggle_url\": \"https://latest-1-0-alpha.datasette.io/fixtures/facetable.json?_facet=state&_size=0&_extra=facet_results&_extra=count&state=MC\",\r\n \"selected\": false\r\n }\r\n ],\r\n \"truncated\": false\r\n }\r\n }\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1112889800", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1112889800, "node_id": "IC_kwDOBm6k_c5CVVnI", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-29T05:29:38Z", "updated_at": "2022-04-29T05:29:38Z", "author_association": "OWNER", "body": "OK, I just got the most incredible result with that!\r\n\r\nI started up a container running `bash` like this, from my `datasette` checkout. I'm mapping port 8005 on my laptop to port 8001 inside the container because laptop port 8001 was already doing something else:\r\n```\r\ndocker run -it --rm --name my-running-script -p 8005:8001 -v \"$PWD\":/usr/src/myapp \\\r\n -w /usr/src/myapp nogil/python bash\r\n```\r\nThen in `bash` I ran the following commands to install Datasette and its dependencies:\r\n```\r\npip install -e '.[test]'\r\npip install datasette-pretty-traces # For debug tracing\r\n```\r\nThen I started Datasette against my `github.db` database (from github-to-sqlite.dogsheep.net/github.db) like this:\r\n\r\n```\r\ndatasette github.db -h 0.0.0.0 --setting trace_debug 1\r\n```\r\nI hit the following two URLs to compare the parallel v.s. not parallel implementations:\r\n\r\n- `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10`\r\n- `http://127.0.0.1:8005/github/issues?_facet=milestone&_facet=repo&_trace=1&_size=10&_noparallel=1`\r\n\r\nAnd... the parallel one beat the non-parallel one decisively, on multiple page refreshes!\r\n\r\nNot parallel: 77ms\r\n\r\nParallel: 47ms\r\n\r\n![\"CleanShot](\"https://user-images.githubusercontent.com/9599/165889437-60d4200d-698a-4175-af23-7c03bb456e66.png\")
\r\n\r\n![\"CleanShot](\"https://user-images.githubusercontent.com/9599/165889445-2dfb8676-d823-405e-aecb-ad28ec3043da.png\")
\r\n\r\nSo yeah, I'm very confident this is a problem with the GIL. And I am absolutely **stunned** that @colesbury's fork ran Datasette (which has some reasonably tricky threading and async stuff going on) out of the box!", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1112879463", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1112879463, "node_id": "IC_kwDOBm6k_c5CVTFn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-29T05:03:58Z", "updated_at": "2022-04-29T05:03:58Z", "author_association": "OWNER", "body": "It would be _really_ fun to try running this with the in-development `nogil` Python from https://github.com/colesbury/nogil\r\n\r\nThere's a Docker container for it: https://hub.docker.com/r/nogil/python\r\n\r\nIt suggests you can run something like this:\r\n\r\n docker run -it --rm --name my-running-script -v \"$PWD\":/usr/src/myapp \\\r\n -w /usr/src/myapp nogil/python python your-daemon-or-script.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1112878955", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1112878955, "node_id": "IC_kwDOBm6k_c5CVS9r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-29T05:02:40Z", "updated_at": "2022-04-29T05:02:40Z", "author_association": "OWNER", "body": "Here's a very useful (recent) article about how the GIL works and how to think about it: https://pythonspeed.com/articles/python-gil/ - via https://lobste.rs/s/9hj80j/when_python_can_t_thread_deep_dive_into_gil\r\n\r\nFrom that article:\r\n\r\n> For example, let's consider an extension module written in C or Rust that lets you talk to a PostgreSQL database server.\r\n> \r\n> Conceptually, handling a SQL query with this library will go through three steps:\r\n> \r\n> 1. Deserialize from Python to the internal library representation. Since this will be reading Python objects, it needs to hold the GIL.\r\n> 2. Send the query to the database server, and wait for a response. This doesn't need the GIL.\r\n> 3. Convert the response into Python objects. This needs the GIL again.\r\n> \r\n> As you can see, how much parallelism you can get depends on how much time is spent in each step. If the bulk of time is spent in step 2, you'll get parallelism there. But if, for example, you run a `SELECT` and get a large number of rows back, the library will need to create many Python objects, and step 3 will have to hold GIL for a while.\r\n\r\nThat explains what I'm seeing here. I'm pretty convinced now that the reason I'm not getting a performance boost from parallel queries is that there's more time spent in Python code assembling the results than in SQLite C code executing the query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112734577", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112734577, "node_id": "IC_kwDOBm6k_c5CUvtx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:08:42Z", "updated_at": "2022-04-28T23:08:42Z", "author_association": "OWNER", "body": "That prototype is a very small amount of code so far:\r\n```diff\r\ndiff --git a/datasette/renderer.py b/datasette/renderer.py\r\nindex 4508949..b600e1b 100644\r\n--- a/datasette/renderer.py\r\n+++ b/datasette/renderer.py\r\n@@ -28,6 +28,10 @@ def convert_specific_columns_to_json(rows, columns, json_cols):\r\n \r\n def json_renderer(args, data, view_name):\r\n \"\"\"Render a response as JSON\"\"\"\r\n+ from pprint import pprint\r\n+\r\n+ pprint(data)\r\n+\r\n status_code = 200\r\n \r\n # Handle the _json= parameter which may modify data[\"rows\"]\r\n@@ -43,6 +47,41 @@ def json_renderer(args, data, view_name):\r\n if \"rows\" in data and not value_as_boolean(args.get(\"_json_infinity\", \"0\")):\r\n data[\"rows\"] = [remove_infinites(row) for row in data[\"rows\"]]\r\n \r\n+ # Start building the default JSON here\r\n+ columns = data[\"columns\"]\r\n+ next_url = data.get(\"next_url\")\r\n+ output = {\r\n+ \"rows\": [dict(zip(columns, row)) for row in data[\"rows\"]],\r\n+ \"next\": data[\"next\"],\r\n+ \"next_url\": next_url,\r\n+ }\r\n+\r\n+ extras = set(args.getlist(\"_extra\"))\r\n+\r\n+ extras_map = {\r\n+ # _extra= : data[field]\r\n+ \"count\": \"filtered_table_rows_count\",\r\n+ \"facet_results\": \"facet_results\",\r\n+ \"suggested_facets\": \"suggested_facets\",\r\n+ \"columns\": \"columns\",\r\n+ \"primary_keys\": \"primary_keys\",\r\n+ \"query_ms\": \"query_ms\",\r\n+ \"query\": \"query\",\r\n+ }\r\n+ for extra_key, data_key in extras_map.items():\r\n+ if extra_key in extras:\r\n+ output[extra_key] = data[data_key]\r\n+\r\n+ body = json.dumps(output, cls=CustomJSONEncoder)\r\n+ content_type = \"application/json; charset=utf-8\"\r\n+ headers = {}\r\n+ if next_url:\r\n+ headers[\"link\"] = f'<{next_url}>; rel=\"next\"'\r\n+ return Response(\r\n+ body, status=status_code, headers=headers, content_type=content_type\r\n+ )\r\n+\r\n+\r\n # Deal with the _shape option\r\n shape = args.get(\"_shape\", \"arrays\")\r\n # if there's an error, ignore the shape entirely\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112732563", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112732563, "node_id": "IC_kwDOBm6k_c5CUvOT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:05:03Z", "updated_at": "2022-04-28T23:05:03Z", "author_association": "OWNER", "body": "OK, the prototype of this is looking really good - it's very pleasant to use.\r\n\r\n`http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_sort=id&_size=5&_extra=query_ms&_extra=count&_col=body` returns this:\r\n\r\n```json\r\n{\r\n \"rows\": [\r\n {\r\n \"id\": 338854988,\r\n \"body\": \" /database-name/table-name?name__contains=simon&sort=id+desc\\r\\n\\r\\nNote that if there's a column called \\\"sort\\\" you can still do sort__exact=blah\\r\\n\\r\\n\"\r\n },\r\n {\r\n \"id\": 346427794,\r\n \"body\": \"Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.\\n\\nOn November 22, 2017 11:38:06 AM EST, Simon Willison wrote:\\n>I have a solution for FTS already, but I'm interested in apsw as a\\n>mechanism for allowing custom virtual tables to be written in Python\\n>(pysqlite only lets you write custom functions)\\n>\\n>Not having PyPI support is pretty tough though. I'm planning a\\n>plugin/extension system which would be ideal for things like an\\n>optional apsw mode, but that's a lot harder if apsw isn't in PyPI.\\n>\\n>-- \\n>You are receiving this because you authored the thread.\\n>Reply to this email directly or view it on GitHub:\\n>https://github.com/simonw/datasette/issues/144#issuecomment-346405660\\n\"\r\n },\r\n {\r\n \"id\": 348252037,\r\n \"body\": \"WOW!\\n\\n--\\nPaul Ford // (646) 369-7128 // @ftrain\\n\\nOn Thu, Nov 30, 2017 at 11:47 AM, Simon Willison \\nwrote:\\n\\n> Remaining work on this now lives in a milestone:\\n> https://github.com/simonw/datasette/milestone/6\\n>\\n> \u2014\\n> You are receiving this because you were mentioned.\\n> Reply to this email directly, view it on GitHub\\n> ,\\n> or mute the thread\\n> \\n> .\\n>\\n\"\r\n },\r\n {\r\n \"id\": 391141391,\r\n \"body\": \"I'm going to clean this up for consistency tomorrow morning so hold off\\nmerging until then please\\n\\nOn Tue, May 22, 2018 at 6:34 PM, Simon Willison \\nwrote:\\n\\n> Yeah let's try this without pysqlite3 and see if we still get the correct\\n> version.\\n>\\n> \u2014\\n> You are receiving this because you authored the thread.\\n> Reply to this email directly, view it on GitHub\\n> , or mute\\n> the thread\\n> \\n> .\\n>\\n\"\r\n },\r\n {\r\n \"id\": 391355030,\r\n \"body\": \"No objections;\\r\\nIt's good to go @simonw\\r\\n\\r\\nOn Wed, 23 May 2018, 14:51 Simon Willison, wrote:\\r\\n\\r\\n> @r4vi any objections to me merging this?\\r\\n>\\r\\n> \u2014\\r\\n> You are receiving this because you were mentioned.\\r\\n> Reply to this email directly, view it on GitHub\\r\\n> , or mute\\r\\n> the thread\\r\\n> \\r\\n> .\\r\\n>\\r\\n\"\r\n }\r\n ],\r\n \"next\": \"391355030,391355030\",\r\n \"next_url\": \"http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_size=5&_extra=query_ms&_extra=count&_col=body&_next=391355030%2C391355030&_sort=id\",\r\n \"count\": 57,\r\n \"query_ms\": 21.780223003588617\r\n}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112730416", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112730416, "node_id": "IC_kwDOBm6k_c5CUusw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:01:21Z", "updated_at": "2022-04-28T23:01:21Z", "author_association": "OWNER", "body": "I'm not sure what to do about the `\"truncated\": true/false` key.\r\n\r\nIt's not really relevant to table results, since they are paginated whether or not you ask for them to be.\r\n\r\nIt plays a role in query results, where you might run `select * from table` and get back 1000 results because Datasette truncates at that point rather than returning everything.\r\n\r\nAdding it to every table result and always setting it to `\"truncated\": false` feels confusing.\r\n\r\nI think I'm going to keep it exclusively in the default representation for the `/db?sql=...` query endpoint, and not return it at all for tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112721321", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112721321, "node_id": "IC_kwDOBm6k_c5CUsep", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:44:05Z", "updated_at": "2022-04-28T22:44:14Z", "author_association": "OWNER", "body": "I may be able to implement this mostly in the `json_renderer()` function: https://github.com/simonw/datasette/blob/94a3171b01fde5c52697aeeff052e3ad4bab5391/datasette/renderer.py#L29-L34", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112717745", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112717745, "node_id": "IC_kwDOBm6k_c5CUrmx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:38:39Z", "updated_at": "2022-04-28T22:39:05Z", "author_association": "OWNER", "body": "(I remain keen on the idea of shipping a plugin that restores the old default API shape to people who have written pre-Datasette-1.0 code against it, but I'll tackle that much later. I really like how jQuery has a culture of doing this.)", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112717210", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112717210, "node_id": "IC_kwDOBm6k_c5CUrea", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:37:37Z", "updated_at": "2022-04-28T22:37:37Z", "author_association": "OWNER", "body": "This means `filtered_table_rows_count` is going to become `count`. I had originally picked that terrible name to avoid confusion between the count of all rows in the table and the count of rows that were filtered.\r\n\r\nI'll add `?_extra=table_count` for getting back the full table count instead. I think `count` is clear enough!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112716611", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112716611, "node_id": "IC_kwDOBm6k_c5CUrVD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:36:24Z", "updated_at": "2022-04-28T22:36:24Z", "author_association": "OWNER", "body": "Then I'm going to implement the following `?_extra=` options:\r\n\r\n- `?_extra=facet_results` - to see facet results\r\n- `?_extra=suggested_facets` - for suggested facets\r\n- `?_extra=count` - for the count of total rows\r\n- `?_extra=columns` - for a list of column names\r\n- `?_extra=primary_keys` - for a list of primary keys\r\n- `?_extra=query` - a `{\"sql\" \"select ...\", \"params\": {}}` object\r\n\r\nI thought about having `?_extra=facet_results` returned automatically if the user specifies at least one `?_facet` - but that doesn't work for default facets configured in `metadata.json` - how can the user opt out of those being returned? So I'm going to say you don't see facets at all if you don't include `?_extra=facet_results`.\r\n\r\nI'm tempted to add `?_extra=_all` to return everything, but I can decide if that's a good idea later.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112713581", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112713581, "node_id": "IC_kwDOBm6k_c5CUqlt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:31:11Z", "updated_at": "2022-04-28T22:31:11Z", "author_association": "OWNER", "body": "I'm going to change the default API response to look like this: \r\n```json\r\n{\r\n \"rows\": [\r\n {\r\n \"pk\": 1,\r\n \"created\": \"2019-01-14 08:00:00\",\r\n \"planet_int\": 1,\r\n \"on_earth\": 1,\r\n \"state\": \"CA\",\r\n \"_city_id\": 1,\r\n \"_neighborhood\": \"Mission\",\r\n \"tags\": \"[\\\"tag1\\\", \\\"tag2\\\"]\",\r\n \"complex_array\": \"[{\\\"foo\\\": \\\"bar\\\"}]\",\r\n \"distinct_some_null\": \"one\",\r\n \"n\": \"n1\"\r\n },\r\n {\r\n \"pk\": 2,\r\n \"created\": \"2019-01-14 08:00:00\",\r\n \"planet_int\": 1,\r\n \"on_earth\": 1,\r\n \"state\": \"CA\",\r\n \"_city_id\": 1,\r\n \"_neighborhood\": \"Dogpatch\",\r\n \"tags\": \"[\\\"tag1\\\", \\\"tag3\\\"]\",\r\n \"complex_array\": \"[]\",\r\n \"distinct_some_null\": \"two\",\r\n \"n\": \"n2\"\r\n }\r\n ],\r\n \"next\": null,\r\n \"next_url\": null\r\n}\r\n```\r\nBasically https://latest.datasette.io/fixtures/facetable.json?_shape=objects but with just the `rows`, `next` and `next_url` fields returned by default.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1715#issuecomment-1112711115", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1715", "id": 1112711115, "node_id": "IC_kwDOBm6k_c5CUp_L", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T22:26:56Z", "updated_at": "2022-04-28T22:26:56Z", "author_association": "OWNER", "body": "I'm not going to use `asyncinject` in this refactor - at least not until I really need it. My research in these issues has put me off the idea ( in favour of `asyncio.gather()` or even not trying for parallel execution at all):\r\n\r\n- #1727", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1212823665, "label": "Refactor TableView to use asyncinject"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1112668411", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1112668411, "node_id": "IC_kwDOBm6k_c5CUfj7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T21:25:34Z", "updated_at": "2022-04-28T21:25:44Z", "author_association": "OWNER", "body": "The two most promising theories at the moment, from here and Twitter and the SQLite forum, are:\r\n\r\n- SQLite is I/O bound - it generally only goes as fast as it can load data from disk. Multiple connections all competing for the same file on disk are going to end up blocked at the file system layer. But maybe this means in-memory databases will perform better?\r\n- It's the GIL. The sqlite3 C code may release the GIL, but the bits that do things like assembling `Row` objects to return still happen in Python, and that Python can only run on a single core.\r\n\r\nA couple of ways to research the in-memory theory:\r\n\r\n- Use a RAM disk on macOS (or Linux). https://stackoverflow.com/a/2033417/6083 has instructions - short version:\r\n\r\n hdiutil attach -nomount ram://$((2 * 1024 * 100))\r\n diskutil eraseVolume HFS+ RAMDisk name-returned-by-previous-command (was `/dev/disk2` when I tried it)\r\n cd /Volumes/RAMDisk\r\n cp ~/fixtures.db .\r\n\r\n- Copy Datasette databases into an in-memory database on startup. I built a new plugin to do that here: https://github.com/simonw/datasette-copy-to-memory\r\n\r\nI need to do some more, better benchmarks using these different approaches.\r\n\r\nhttps://twitter.com/laurencerowe/status/1519780174560169987 also suggests:\r\n\r\n> Maybe try:\r\n> 1. Copy the sqlite file to /dev/shm and rerun (all in ram.)\r\n> 2. Create a CTE which calculates Fibonacci or similar so you can test something completely cpu bound (only return max value or something to avoid crossing between sqlite/Python.)\r\n\r\nI like that second idea a lot - I could use the mandelbrot example from https://www.sqlite.org/lang_with.html#outlandish_recursive_query_examples", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111726586", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111726586, "node_id": "IC_kwDOBm6k_c5CQ5n6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T04:17:16Z", "updated_at": "2022-04-28T04:19:31Z", "author_association": "OWNER", "body": "I could experiment with the `await asyncio.run_in_executor(processpool_executor, fn)` mechanism described in https://stackoverflow.com/a/29147750\r\n\r\nCode examples: https://cs.github.com/?scopeName=All+repos&scope=&q=run_in_executor+ProcessPoolExecutor", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111725638", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111725638, "node_id": "IC_kwDOBm6k_c5CQ5ZG", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T04:15:15Z", "updated_at": "2022-04-28T04:15:15Z", "author_association": "OWNER", "body": "Useful theory from Keith Medcalf https://sqlite.org/forum/forumpost/e363c69d3441172e\r\n\r\n> This is true, but the concurrency is limited to the execution which occurs with the GIL released (that is, in the native C sqlite3 library itself). Each row (for example) can be retrieved in parallel but \"constructing the python return objects for each row\" will be serialized (by the GIL).\r\n> \r\n> That is to say that if your have two python threads each with their own connection, and each one is performing a select that returns 1,000,000 rows (lets say that is 25% of the candidates for each select) then the difference in execution time between executing two python threads in parallel vs a single serial thead will not be much different (if even detectable at all). In fact it is possible that the multiple-threaded version takes longer to run both queries to completion because of the increased contention over a shared resource (the GIL).\r\n\r\nSo maybe this is a GIL thing.\r\n\r\nI should test with some expensive SQL queries (maybe big aggregations against large tables) and see if I can spot an improvement there.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111714665", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111714665, "node_id": "IC_kwDOBm6k_c5CQ2tp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:52:47Z", "updated_at": "2022-04-28T03:52:58Z", "author_association": "OWNER", "body": "Nice custom template/theme!\r\n\r\nYeah, for that I'd recommend hosting elsewhere - on a regular VPS (I use `systemd` like this: https://docs.datasette.io/en/stable/deploying.html#running-datasette-using-systemd ) or using Fly if you want to tub containers without managing a full server.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111708206", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111708206, "node_id": "IC_kwDOBm6k_c5CQ1Iu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:38:56Z", "updated_at": "2022-04-28T03:38:56Z", "author_association": "OWNER", "body": "In terms of this bug, there are a few potential fixes:\r\n\r\n1. Detect the write to a immutable database and show the user a proper, meaningful error message in the red error box at the top of the page\r\n2. Don't allow the user to even submit the form - show a message saying that this canned query is unavailable because the database cannot be written to\r\n3. Don't even allow Datasette to start running at all - if there's a canned query configured in `metadata.yml` and the database it refers to is in `-i` immutable mode throw an error on startup\r\n\r\nI'm not keen on that last one because it would be frustrating if you couldn't launch Datasette just because you had an old canned query lying around in your metadata file.\r\n\r\nSo I'm leaning towards option 2.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111707384", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111707384, "node_id": "IC_kwDOBm6k_c5CQ074", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:36:46Z", "updated_at": "2022-04-28T03:36:56Z", "author_association": "OWNER", "body": "A more realistic solution (which I've been using on several of my own projects) is to keep the data itself in GitHub and encourage users to edit it there - using the GitHub web interface to edit YAML files or similar.\r\n\r\nNeeds your users to be comfortable hand-editing YAML though! You can at least guard against critical errors by having CI run tests against their YAML before deploying.\r\n\r\nI have a dream of building a more friendly web forms interface which edits the YAML back on GitHub for the user, but that's just a concept at the moment.\r\n\r\nEven more fun would be if a user-friendly form could submit PRs for review without the user having to know what a PR is!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111706519", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111706519, "node_id": "IC_kwDOBm6k_c5CQ0uX", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:34:49Z", "updated_at": "2022-04-28T03:34:49Z", "author_association": "OWNER", "body": "I've wanted to do stuff like that on Cloud Run too. So far I've assumed that it's not feasible, but recently I've been wondering how hard it would be to have a small (like less than 100KB or so) Datasette instance which persists data to a backing GitHub repository such that when it starts up it can pull the latest copy and any time someone edits it can push their changes.\r\n\r\nI'm still not sure it would work well on Cloud Run due to the uncertainty at what would happen if Cloud Run decided to boot up a second instance - but it's still an interesting thought exercise.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111705069", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111705069, "node_id": "IC_kwDOBm6k_c5CQ0Xt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:31:33Z", "updated_at": "2022-04-28T03:31:33Z", "author_association": "OWNER", "body": "Confirmed - this is a bug where immutable databases fail to show a useful error if you write to them with a canned query.\r\n\r\nSteps to reproduce:\r\n```\r\necho '\r\ndatabases:\r\n writable:\r\n queries:\r\n add_name:\r\n sql: insert into names(name) values (:name)\r\n write: true\r\n' > write-metadata.yml\r\necho '{\"name\": \"Simon\"}' | sqlite-utils insert writable.db names -\r\ndatasette writable.db -m write-metadata.yml\r\n```\r\nThen visit http://127.0.0.1:8001/writable/add_name - adding names works.\r\n\r\nNow do this instead:\r\n\r\n```\r\ndatasette -i writable.db -m write-metadata.yml\r\n```\r\n\r\nAnd I'm getting a broken error:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111699175", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111699175, "node_id": "IC_kwDOBm6k_c5CQy7n", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:19:48Z", "updated_at": "2022-04-28T03:20:08Z", "author_association": "OWNER", "body": "I ran `py-spy` and then hammered refresh a bunch of times on the `http://127.0.0.1:8856/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=` page - it generated this SVG profile for me.\r\n\r\nThe area on the right is the threads running the DB queries:\r\n\r\n\r\n\r\nInteractive version here: https://static.simonwillison.net/static/2022/datasette-parallel-profile.svg", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111698307", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111698307, "node_id": "IC_kwDOBm6k_c5CQyuD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:18:02Z", "updated_at": "2022-04-28T03:18:02Z", "author_association": "OWNER", "body": "If the behaviour you are seeing is because the database is running in immutable mode then that's a bug - you should get a useful error message instead!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1728#issuecomment-1111697985", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1728", "id": 1111697985, "node_id": "IC_kwDOBm6k_c5CQypB", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T03:17:20Z", "updated_at": "2022-04-28T03:17:20Z", "author_association": "OWNER", "body": "How did you deploy to Cloud Run?\r\n\r\n`datasette publish cloudrun` defaults to running databases there in `-i` immutable mode, because if you managed to change a file on disk on Cloud Run those changes would be lost the next time your container restarted there.\r\n\r\nThat's why I upgraded `datasette-publish-fly` to provide a way of working with their volumes support - they're the best option I know of right now for running Datasette in a container with a persistent volume that can accept writes: https://simonwillison.net/2022/Feb/15/fly-volumes/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1218133366, "label": "Writable canned queries fail with useless non-error against immutable databases"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111683539", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111683539, "node_id": "IC_kwDOBm6k_c5CQvHT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T02:47:57Z", "updated_at": "2022-04-28T02:47:57Z", "author_association": "OWNER", "body": "Maybe this is the Python GIL after all?\r\n\r\nI've been hoping that the GIL won't be an issue because the `sqlite3` module releases the GIL for the duration of the execution of a SQL query - see https://github.com/python/cpython/blob/f348154c8f8a9c254503306c59d6779d4d09b3a9/Modules/_sqlite/cursor.c#L749-L759\r\n\r\nSo I've been hoping this means that SQLite code itself can run concurrently on multiple cores even when Python threads cannot.\r\n\r\nBut maybe I'm misunderstanding how that works?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111681513", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111681513, "node_id": "IC_kwDOBm6k_c5CQunp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T02:44:26Z", "updated_at": "2022-04-28T02:44:26Z", "author_association": "OWNER", "body": "I could try `py-spy top`, which I previously used here:\r\n- https://github.com/simonw/datasette/issues/1673", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111661331", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111661331, "node_id": "IC_kwDOBm6k_c5CQpsT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T02:07:31Z", "updated_at": "2022-04-28T02:07:31Z", "author_association": "OWNER", "body": "Asked on the SQLite forum about this here: https://sqlite.org/forum/forumpost/ffbfa9f38e", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111602802", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111602802, "node_id": "IC_kwDOBm6k_c5CQbZy", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T00:21:35Z", "updated_at": "2022-04-28T00:21:35Z", "author_association": "OWNER", "body": "Tried this but I'm getting back an empty JSON array of traces at the bottom of the page most of the time (intermittently it works correctly):\r\n\r\n```diff\r\ndiff --git a/datasette/database.py b/datasette/database.py\r\nindex ba594a8..d7f9172 100644\r\n--- a/datasette/database.py\r\n+++ b/datasette/database.py\r\n@@ -7,7 +7,7 @@ import sys\r\n import threading\r\n import uuid\r\n \r\n-from .tracer import trace\r\n+from .tracer import trace, trace_child_tasks\r\n from .utils import (\r\n detect_fts,\r\n detect_primary_keys,\r\n@@ -207,30 +207,31 @@ class Database:\r\n time_limit_ms = custom_time_limit\r\n \r\n with sqlite_timelimit(conn, time_limit_ms):\r\n- try:\r\n- cursor = conn.cursor()\r\n- cursor.execute(sql, params if params is not None else {})\r\n- max_returned_rows = self.ds.max_returned_rows\r\n- if max_returned_rows == page_size:\r\n- max_returned_rows += 1\r\n- if max_returned_rows and truncate:\r\n- rows = cursor.fetchmany(max_returned_rows + 1)\r\n- truncated = len(rows) > max_returned_rows\r\n- rows = rows[:max_returned_rows]\r\n- else:\r\n- rows = cursor.fetchall()\r\n- truncated = False\r\n- except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:\r\n- if e.args == (\"interrupted\",):\r\n- raise QueryInterrupted(e, sql, params)\r\n- if log_sql_errors:\r\n- sys.stderr.write(\r\n- \"ERROR: conn={}, sql = {}, params = {}: {}\\n\".format(\r\n- conn, repr(sql), params, e\r\n+ with trace(\"sql\", database=self.name, sql=sql.strip(), params=params):\r\n+ try:\r\n+ cursor = conn.cursor()\r\n+ cursor.execute(sql, params if params is not None else {})\r\n+ max_returned_rows = self.ds.max_returned_rows\r\n+ if max_returned_rows == page_size:\r\n+ max_returned_rows += 1\r\n+ if max_returned_rows and truncate:\r\n+ rows = cursor.fetchmany(max_returned_rows + 1)\r\n+ truncated = len(rows) > max_returned_rows\r\n+ rows = rows[:max_returned_rows]\r\n+ else:\r\n+ rows = cursor.fetchall()\r\n+ truncated = False\r\n+ except (sqlite3.OperationalError, sqlite3.DatabaseError) as e:\r\n+ if e.args == (\"interrupted\",):\r\n+ raise QueryInterrupted(e, sql, params)\r\n+ if log_sql_errors:\r\n+ sys.stderr.write(\r\n+ \"ERROR: conn={}, sql = {}, params = {}: {}\\n\".format(\r\n+ conn, repr(sql), params, e\r\n+ )\r\n )\r\n- )\r\n- sys.stderr.flush()\r\n- raise\r\n+ sys.stderr.flush()\r\n+ raise\r\n \r\n if truncate:\r\n return Results(rows, truncated, cursor.description)\r\n@@ -238,9 +239,8 @@ class Database:\r\n else:\r\n return Results(rows, False, cursor.description)\r\n \r\n- with trace(\"sql\", database=self.name, sql=sql.strip(), params=params):\r\n- results = await self.execute_fn(sql_operation_in_thread)\r\n- return results\r\n+ with trace_child_tasks():\r\n+ return await self.execute_fn(sql_operation_in_thread)\r\n \r\n @property\r\n def size(self):\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111597176", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111597176, "node_id": "IC_kwDOBm6k_c5CQaB4", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T00:11:44Z", "updated_at": "2022-04-28T00:11:44Z", "author_association": "OWNER", "body": "Though it would be interesting to also have the trace reveal how much time is spent in the functions that wrap that core SQL - the stuff that is being measured at the moment.\r\n\r\nI have a hunch that this could help solve the over-arching performance mystery.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111595319", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111595319, "node_id": "IC_kwDOBm6k_c5CQZk3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T00:09:45Z", "updated_at": "2022-04-28T00:11:01Z", "author_association": "OWNER", "body": "Here's where read queries are instrumented: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L241-L242\r\n\r\nSo the instrumentation is actually capturing quite a bit of Python activity before it gets to SQLite:\r\n\r\nhttps://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L179-L190\r\n\r\nAnd then:\r\n\r\nhttps://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L204-L233\r\n\r\nIdeally I'd like that `trace()` block to wrap just the `cursor.execute()` and `cursor.fetchmany(...)` or `cursor.fetchall()` calls.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111558204", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111558204, "node_id": "IC_kwDOBm6k_c5CQQg8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T22:58:39Z", "updated_at": "2022-04-27T22:58:39Z", "author_association": "OWNER", "body": "I should check my timing mechanism. Am I capturing the time taken just in SQLite or does it include time spent in Python crossing between async and threaded world and waiting for a thread pool worker to become available?\r\n\r\nThat could explain the longer query times.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111553029", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111553029, "node_id": "IC_kwDOBm6k_c5CQPQF", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T22:48:21Z", "updated_at": "2022-04-27T22:48:21Z", "author_association": "OWNER", "body": "I wonder if it would be worth exploring multiprocessing here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111551076", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111551076, "node_id": "IC_kwDOBm6k_c5CQOxk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T22:44:51Z", "updated_at": "2022-04-27T22:45:04Z", "author_association": "OWNER", "body": "Really wild idea: what if I created three copies of the SQLite database file - as three separate file names - and then balanced the parallel queries across all these? Any chance that could avoid any mysterious locking issues?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111535818", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111535818, "node_id": "IC_kwDOBm6k_c5CQLDK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T22:18:45Z", "updated_at": "2022-04-27T22:18:45Z", "author_association": "OWNER", "body": "Another avenue: https://twitter.com/weargoggles/status/1519426289920270337\r\n\r\n> SQLite has its own mutexes to provide thread safety, which as another poster noted are out of play in multi process setups. Perhaps downgrading from the \u201cserializable\u201d to \u201cmulti-threaded\u201d safety would be okay for Datasette? https://sqlite.org/c3ref/c_config_covering_index_scan.html#sqliteconfigmultithread\r\n\r\nDoesn't look like there's an obvious way to access that from Python via the `sqlite3` module though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111485722", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111485722, "node_id": "IC_kwDOBm6k_c5CP-0a", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T21:08:20Z", "updated_at": "2022-04-27T21:08:20Z", "author_association": "OWNER", "body": "Tried that and it didn't seem to make a difference either.\r\n\r\nI really need a much deeper view of what's going on here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111462442", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111462442, "node_id": "IC_kwDOBm6k_c5CP5Iq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T20:40:59Z", "updated_at": "2022-04-27T20:42:49Z", "author_association": "OWNER", "body": "This looks VERY relevant: [SQLite Shared-Cache Mode](https://www.sqlite.org/sharedcache.html):\r\n\r\n> SQLite includes a special \"shared-cache\" mode (disabled by default) intended for use in embedded servers. If shared-cache mode is enabled and a thread establishes multiple connections to the same database, the connections share a single data and schema cache. This can significantly reduce the quantity of memory and IO required by the system.\r\n\r\nEnabled as part of the URI filename:\r\n\r\n ATTACH 'file:aux.db?cache=shared' AS aux;\r\n\r\nTurns out I'm already using this for in-memory databases that have `.memory_name` set, but not (yet) for regular file-backed databases:\r\n\r\nhttps://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L75\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111460068", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111460068, "node_id": "IC_kwDOBm6k_c5CP4jk", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T20:38:32Z", "updated_at": "2022-04-27T20:38:32Z", "author_association": "OWNER", "body": "WAL mode didn't seem to make a difference. I thought there was a chance it might help multiple read connections operate at the same time but it looks like it really does only matter for when writes are going on.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111456500", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111456500, "node_id": "IC_kwDOBm6k_c5CP3r0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T20:36:01Z", "updated_at": "2022-04-27T20:36:01Z", "author_association": "OWNER", "body": "Yeah all of this is pretty much assuming read-only connections. Datasette has a separate mechanism for ensuring that writes are executed one at a time against a dedicated connection from an in-memory queue:\r\n- https://github.com/simonw/datasette/issues/682", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111442012", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111442012, "node_id": "IC_kwDOBm6k_c5CP0Jc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T20:19:00Z", "updated_at": "2022-04-27T20:19:00Z", "author_association": "OWNER", "body": "Something worth digging into: are these parallel queries running against the same SQLite connection or are they each rubbing against a separate SQLite connection?\r\n\r\nJust realized I know the answer: they're running against separate SQLite connections, because that's how the time limit mechanism works: it installs a progress handler for each connection which terminates it after a set time.\r\n\r\nThis means that if SQLite benefits from multiple threads using the same connection (due to shared caches or similar) then Datasette will not be seeing those benefits.\r\n\r\nIt also means that if there's some mechanism within SQLite that penalizes you for having multiple parallel connections to a single file (just guessing here, maybe there's some kind of locking going on?) then Datasette will suffer those penalties.\r\n\r\nI should try seeing what happens with WAL mode enabled.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111432375", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111432375, "node_id": "IC_kwDOBm6k_c5CPxy3", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T20:07:57Z", "updated_at": "2022-04-27T20:07:57Z", "author_association": "OWNER", "body": "Also useful: https://avi.im/blag/2021/fast-sqlite-inserts/ - from a tip on Twitter: https://twitter.com/ricardoanderegg/status/1519402047556235264", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111431785", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111431785, "node_id": "IC_kwDOBm6k_c5CPxpp", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T20:07:16Z", "updated_at": "2022-04-27T20:07:16Z", "author_association": "OWNER", "body": "I think I need some much more in-depth tracing tricks for this.\r\n\r\nhttps://www.maartenbreddels.com/perf/jupyter/python/tracing/gil/2021/01/14/Tracing-the-Python-GIL.html looks relevant - uses the `perf` tool on Linux.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111408273", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111408273, "node_id": "IC_kwDOBm6k_c5CPr6R", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T19:40:51Z", "updated_at": "2022-04-27T19:42:17Z", "author_association": "OWNER", "body": "Relevant: here's the code that sets up a Datasette SQLite connection: https://github.com/simonw/datasette/blob/7a6654a253dee243518dc542ce4c06dbb0d0801d/datasette/database.py#L73-L96\r\n\r\nIt's using `check_same_thread=False` - here's [the Python docs on that](https://docs.python.org/3/library/sqlite3.html#sqlite3.connect):\r\n\r\n> By default, *check_same_thread* is [`True`](https://docs.python.org/3/library/constants.html#True \"True\") and only the creating thread may use the connection. If set [`False`](https://docs.python.org/3/library/constants.html#False \"False\"), the returned connection may be shared across multiple threads. When using multiple threads with the same connection writing operations should be serialized by the user to avoid data corruption.\r\n\r\nThis is why Datasette reserves a single connection for write queries and queues them up in memory, [as described here](https://simonwillison.net/2020/Feb/26/weeknotes-datasette-writes/).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111390433", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111390433, "node_id": "IC_kwDOBm6k_c5CPnjh", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T19:21:02Z", "updated_at": "2022-04-27T19:21:02Z", "author_association": "OWNER", "body": "One weird thing: I noticed that in the parallel trace above the SQL query bars are wider. Mousover shows duration in ms, and I got 13ms for this query:\r\n\r\n select message as value, count(*) as n from (\r\n\r\nBut in the `?_noparallel=1` version that some query took 2.97ms.\r\n\r\nGiven those numbers though I would expect the overall page time to be MUCH worse for the parallel version - but the page load times are instead very close to each other, with parallel often winning.\r\n\r\nThis is super-weird.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111385875", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111385875, "node_id": "IC_kwDOBm6k_c5CPmcT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T19:16:57Z", "updated_at": "2022-04-27T19:16:57Z", "author_association": "OWNER", "body": "I just remembered the `--setting num_sql_threads` option... which defaults to 3! https://github.com/simonw/datasette/blob/942411ef946e9a34a2094944d3423cddad27efd3/datasette/app.py#L109-L113\r\n\r\nWould explain why the first trace never seems to show more than three SQL queries executing at once.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1111380282", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1111380282, "node_id": "IC_kwDOBm6k_c5CPlE6", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T19:10:27Z", "updated_at": "2022-04-27T19:10:27Z", "author_association": "OWNER", "body": "Wrote more about that here: https://simonwillison.net/2022/Apr/27/parallel-queries/\r\n\r\nCompare https://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1\r\n\r\n\r\n\r\nWith the same thing but with parallel execution disabled:\r\n\r\nhttps://latest-with-plugins.datasette.io/github/commits?_facet=repo&_facet=committer&_trace=1&_noparallel=1\r\n\r\n\r\n\r\nThose total page load time numbers are very similar. Is this parallel optimization worthwhile?\r\n\r\nMaybe it's only worth it on larger databases? Or maybe larger databases perform worse with this?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1724#issuecomment-1110585475", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1724", "id": 1110585475, "node_id": "IC_kwDOBm6k_c5CMjCD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-27T06:15:14Z", "updated_at": "2022-04-27T06:15:14Z", "author_association": "OWNER", "body": "Yeah, that page is 438K (but only 20K gzipped).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1216619276, "label": "?_trace=1 doesn't work on Global Power Plants demo"}, "performed_via_github_app": null}
\r\n\r\n\r\n\r\nSo yeah, I'm very confident this is a problem with the GIL. And I am absolutely **stunned** that @colesbury's fork ran Datasette (which has some reasonably tricky threading and async stuff going on) out of the box!", "reactions": "{\"total_count\": 2, \"+1\": 2, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1112879463", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1112879463, "node_id": "IC_kwDOBm6k_c5CVTFn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-29T05:03:58Z", "updated_at": "2022-04-29T05:03:58Z", "author_association": "OWNER", "body": "It would be _really_ fun to try running this with the in-development `nogil` Python from https://github.com/colesbury/nogil\r\n\r\nThere's a Docker container for it: https://hub.docker.com/r/nogil/python\r\n\r\nIt suggests you can run something like this:\r\n\r\n docker run -it --rm --name my-running-script -v \"$PWD\":/usr/src/myapp \\\r\n -w /usr/src/myapp nogil/python python your-daemon-or-script.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1727#issuecomment-1112878955", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1727", "id": 1112878955, "node_id": "IC_kwDOBm6k_c5CVS9r", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-29T05:02:40Z", "updated_at": "2022-04-29T05:02:40Z", "author_association": "OWNER", "body": "Here's a very useful (recent) article about how the GIL works and how to think about it: https://pythonspeed.com/articles/python-gil/ - via https://lobste.rs/s/9hj80j/when_python_can_t_thread_deep_dive_into_gil\r\n\r\nFrom that article:\r\n\r\n> For example, let's consider an extension module written in C or Rust that lets you talk to a PostgreSQL database server.\r\n> \r\n> Conceptually, handling a SQL query with this library will go through three steps:\r\n> \r\n> 1. Deserialize from Python to the internal library representation. Since this will be reading Python objects, it needs to hold the GIL.\r\n> 2. Send the query to the database server, and wait for a response. This doesn't need the GIL.\r\n> 3. Convert the response into Python objects. This needs the GIL again.\r\n> \r\n> As you can see, how much parallelism you can get depends on how much time is spent in each step. If the bulk of time is spent in step 2, you'll get parallelism there. But if, for example, you run a `SELECT` and get a large number of rows back, the library will need to create many Python objects, and step 3 will have to hold GIL for a while.\r\n\r\nThat explains what I'm seeing here. I'm pretty convinced now that the reason I'm not getting a performance boost from parallel queries is that there's more time spent in Python code assembling the results than in SQLite C code executing the query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1217759117, "label": "Research: demonstrate if parallel SQL queries are worthwhile"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112734577", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112734577, "node_id": "IC_kwDOBm6k_c5CUvtx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:08:42Z", "updated_at": "2022-04-28T23:08:42Z", "author_association": "OWNER", "body": "That prototype is a very small amount of code so far:\r\n```diff\r\ndiff --git a/datasette/renderer.py b/datasette/renderer.py\r\nindex 4508949..b600e1b 100644\r\n--- a/datasette/renderer.py\r\n+++ b/datasette/renderer.py\r\n@@ -28,6 +28,10 @@ def convert_specific_columns_to_json(rows, columns, json_cols):\r\n \r\n def json_renderer(args, data, view_name):\r\n \"\"\"Render a response as JSON\"\"\"\r\n+ from pprint import pprint\r\n+\r\n+ pprint(data)\r\n+\r\n status_code = 200\r\n \r\n # Handle the _json= parameter which may modify data[\"rows\"]\r\n@@ -43,6 +47,41 @@ def json_renderer(args, data, view_name):\r\n if \"rows\" in data and not value_as_boolean(args.get(\"_json_infinity\", \"0\")):\r\n data[\"rows\"] = [remove_infinites(row) for row in data[\"rows\"]]\r\n \r\n+ # Start building the default JSON here\r\n+ columns = data[\"columns\"]\r\n+ next_url = data.get(\"next_url\")\r\n+ output = {\r\n+ \"rows\": [dict(zip(columns, row)) for row in data[\"rows\"]],\r\n+ \"next\": data[\"next\"],\r\n+ \"next_url\": next_url,\r\n+ }\r\n+\r\n+ extras = set(args.getlist(\"_extra\"))\r\n+\r\n+ extras_map = {\r\n+ # _extra= : data[field]\r\n+ \"count\": \"filtered_table_rows_count\",\r\n+ \"facet_results\": \"facet_results\",\r\n+ \"suggested_facets\": \"suggested_facets\",\r\n+ \"columns\": \"columns\",\r\n+ \"primary_keys\": \"primary_keys\",\r\n+ \"query_ms\": \"query_ms\",\r\n+ \"query\": \"query\",\r\n+ }\r\n+ for extra_key, data_key in extras_map.items():\r\n+ if extra_key in extras:\r\n+ output[extra_key] = data[data_key]\r\n+\r\n+ body = json.dumps(output, cls=CustomJSONEncoder)\r\n+ content_type = \"application/json; charset=utf-8\"\r\n+ headers = {}\r\n+ if next_url:\r\n+ headers[\"link\"] = f'<{next_url}>; rel=\"next\"'\r\n+ return Response(\r\n+ body, status=status_code, headers=headers, content_type=content_type\r\n+ )\r\n+\r\n+\r\n # Deal with the _shape option\r\n shape = args.get(\"_shape\", \"arrays\")\r\n # if there's an error, ignore the shape entirely\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1219385669, "label": "Implement ?_extra and new API design for TableView"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1729#issuecomment-1112732563", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1729", "id": 1112732563, "node_id": "IC_kwDOBm6k_c5CUvOT", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-28T23:05:03Z", "updated_at": "2022-04-28T23:05:03Z", "author_association": "OWNER", "body": "OK, the prototype of this is looking really good - it's very pleasant to use.\r\n\r\n`http://127.0.0.1:8001/github_memory/issue_comments.json?_search=simon&_sort=id&_size=5&_extra=query_ms&_extra=count&_col=body` returns this:\r\n\r\n```json\r\n{\r\n \"rows\": [\r\n {\r\n \"id\": 338854988,\r\n \"body\": \" /database-name/table-name?name__contains=simon&sort=id+desc\\r\\n\\r\\nNote that if there's a column called \\\"sort\\\" you can still do sort__exact=blah\\r\\n\\r\\n\"\r\n },\r\n {\r\n \"id\": 346427794,\r\n \"body\": \"Thanks. There is a way to use pip to grab apsw, which also let's you configure it (flags to build extensions, use an internal sqlite, etc). Don't know how that works as a dependency for another package, though.\\n\\nOn November 22, 2017 11:38:06 AM EST, Simon Willison