{"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997272223", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997272223, "node_id": "IC_kwDOBm6k_c47cSqf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T19:17:13Z", "updated_at": "2021-12-18T19:17:13Z", "author_association": "OWNER", "body": "That's a good optimization. Still need to deal with the huge flurry of `PRAGMA` queries though before I can consider this done.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997267416", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997267416, "node_id": "IC_kwDOBm6k_c47cRfY", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T18:44:53Z", "updated_at": "2021-12-18T18:45:28Z", "author_association": "OWNER", "body": "Rather than adding a `executemany=True` parameter, I'm now thinking a better design might be to have three methods:\r\n\r\n- `db.execute_write(sql, params=None, block=False)`\r\n- `db.execute_writescript(sql, block=False)`\r\n- `db.execute_writemany(sql, params_seq, block=False)`", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997266100", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997266100, "node_id": "IC_kwDOBm6k_c47cRK0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T18:40:02Z", "updated_at": "2021-12-18T18:40:02Z", "author_association": "OWNER", "body": "The implementation of `cursor.executemany()` looks very efficient - it turns into a call to this C function with `multiple` set to `1`: https://github.com/python/cpython/blob/e002bbc6cce637171fb2b1391ffeca8643a13843/Modules/_sqlite/cursor.c#L468-L469", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997262475", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997262475, "node_id": "IC_kwDOBm6k_c47cQSL", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T18:34:18Z", "updated_at": "2021-12-18T18:34:18Z", "author_association": "OWNER", "body": "\"image\"\r\n\r\nUsing `executescript=True` that call now takes 1.89ms to create all of those tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997248364", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997248364, "node_id": "IC_kwDOBm6k_c47cM1s", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T18:20:10Z", "updated_at": "2021-12-18T18:20:10Z", "author_association": "OWNER", "body": "Idea: teach `execute_write` to accept an optional `executescript=True` parameter, like this:\r\n```diff\r\ndiff --git a/datasette/database.py b/datasette/database.py\r\nindex 468e936..1a424f5 100644\r\n--- a/datasette/database.py\r\n+++ b/datasette/database.py\r\n@@ -94,10 +94,14 @@ class Database:\r\n f\"file:{self.path}{qs}\", uri=True, check_same_thread=False\r\n )\r\n \r\n- async def execute_write(self, sql, params=None, block=False):\r\n+ async def execute_write(self, sql, params=None, executescript=False, block=False):\r\n+ assert not executescript and params, \"Cannot use params with executescript=True\"\r\n def _inner(conn):\r\n with conn:\r\n- return conn.execute(sql, params or [])\r\n+ if executescript:\r\n+ return conn.executescript(sql)\r\n+ else:\r\n+ return conn.execute(sql, params or [])\r\n \r\n with trace(\"sql\", database=self.name, sql=sql.strip(), params=params):\r\n results = await self.execute_write_fn(_inner, block=block)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997245301", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997245301, "node_id": "IC_kwDOBm6k_c47cMF1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T18:17:04Z", "updated_at": "2021-12-18T18:17:04Z", "author_association": "OWNER", "body": "One downside of `conn.executescript()` is that it won't be picked up by the tracing mechanism - in fact nothing that uses `await db.execute_write_fn(fn, block=True)` or `await db.execute_fn(fn, block=True)` gets picked up by tracing.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997241969", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997241969, "node_id": "IC_kwDOBm6k_c47cLRx", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T18:13:04Z", "updated_at": "2021-12-18T18:13:04Z", "author_association": "OWNER", "body": "Also: running all of those `CREATE TABLE IF NOT EXISTS` in a single call to `conn.executescript()` rather than as separate queries may speed things up too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997241645", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997241645, "node_id": "IC_kwDOBm6k_c47cLMt", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T18:12:26Z", "updated_at": "2021-12-18T18:12:26Z", "author_association": "OWNER", "body": "A simpler optimization would be just to turn all of those column and index reads into a single efficient UNION query against each database, then figure out the most efficient pattern to send them all as writes in one go as opposed to calling `.execute_write()` in a loop.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997235086", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997235086, "node_id": "IC_kwDOBm6k_c47cJmO", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T17:30:13Z", "updated_at": "2021-12-18T17:30:13Z", "author_association": "OWNER", "body": "Now that trace sees write queries (#1568) it's clear that there is a whole lot more DB activity then I had realized:\r\n\r\n\"image\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997234858", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997234858, "node_id": "IC_kwDOBm6k_c47cJiq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T17:28:44Z", "updated_at": "2021-12-18T17:28:44Z", "author_association": "OWNER", "body": "Maybe it would be worth exploring attaching each DB in turn to the _internal connection in order to perform these queries faster.\r\n\r\nI'm a bit worried about leaks though: the internal database isn't meant to be visible, even temporarily attaching another DB to it could cause SQL queries against that DB to be able to access the internal data.\r\n\r\nSo maybe instead the _internal connection gets to connect to the other DBs? There's a maximum of ten there I think, which is good for most but not all cases. But the cases with the most connected databases will see the worst performance!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997128508", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997128508, "node_id": "IC_kwDOBm6k_c47bvk8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T02:33:57Z", "updated_at": "2021-12-18T02:33:57Z", "author_association": "OWNER", "body": "Here's why - `trace` only applies to read, not write SQL operations: https://github.com/simonw/datasette/blob/7c8f8aa209e4ba7bf83976f8495d67c28fbfca24/datasette/database.py#L209-L211", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997128368", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997128368, "node_id": "IC_kwDOBm6k_c47bviw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T02:32:43Z", "updated_at": "2021-12-18T02:32:43Z", "author_association": "OWNER", "body": "I wonder why the `INSERT INTO` queries don't show up in that `?trace=1` view?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997128251", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997128251, "node_id": "IC_kwDOBm6k_c47bvg7", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T02:31:51Z", "updated_at": "2021-12-18T02:31:51Z", "author_association": "OWNER", "body": "I was thinking it might even be possible to convert this into a `insert into tables select from ...` query:\r\n\r\nhttps://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L102-L112\r\n\r\n But the `SELECT` runs against a separate database from the `INSERT INTO`, so I would have to setup a cross-database connection for this which feels a little too complicated.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1555#issuecomment-997128080", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1555", "id": 997128080, "node_id": "IC_kwDOBm6k_c47bveQ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-12-18T02:30:19Z", "updated_at": "2021-12-18T02:30:19Z", "author_association": "OWNER", "body": "I think all of these queries happen in one place - in the `populate_schema_tables()` function - so optimizing them might be localized to just that area of the code, which would be nice:\r\n\r\nhttps://github.com/simonw/datasette/blob/c00f29affcafce8314366852ba1a0f5a7dd25690/datasette/utils/internal_db.py#L97-L183", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1079149656, "label": "Optimize all those calls to index_list and foreign_key_list"}, "performed_via_github_app": null}