{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134386", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134386, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDM4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:20:28Z", "updated_at": "2021-08-13T00:42:30Z", "author_association": "OWNER", "body": "... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the `Variable` opcodes.\r\n\r\n[UPDATE: it did indeed do that, see #1421]", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-901475812", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 901475812, "node_id": "IC_kwDOBm6k_c41u23k", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-18T22:41:19Z", "updated_at": "2021-08-18T22:41:19Z", "author_association": "OWNER", "body": "> Maybe I split this out into a separate Python library that gets tested against _every_ SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?\r\n\r\nI'm going to do this, and call the Python library `sqlite-explain`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898065011", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898065011, "node_id": "IC_kwDOBm6k_c41h2Jz", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T00:36:30Z", "updated_at": "2021-08-13T00:36:30Z", "author_association": "OWNER", "body": "> https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:\r\n> \r\n> ```sql\r\n> CREATE VIEW paginated_view AS\r\n> SELECT\r\n> content,\r\n> '- ' || content || ' -' AS content_extra\r\n> FROM no_primary_key;\r\n> ```\r\n> \r\n> So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.\r\n\r\nHere's what it does for that:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113175", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113175, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzE3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:07:01Z", "updated_at": "2021-04-04T23:07:01Z", "author_association": "OWNER", "body": "A more promising route I found involved the `db.set_authorizer` method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!\r\n\r\n```pycon\r\n>>> def print_args(*args, **kwargs):\r\n... print(\"args\", args, \"kwargs\", kwargs)\r\n... return sqlite3.SQLITE_OK\r\n\r\n>>> db = sqlite3.connect(\"fixtures.db\")\r\n>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'content', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'created', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'planet_int', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'on_earth', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'state', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'city_id', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'neighborhood', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'tags', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'complex_array', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}\r\n```\r\nThose `20` values (where 20 is `SQLITE_READ`) looked like they were checking permissions for the columns in the order they would be returned!\r\n\r\nThen I found a snag:\r\n\r\n```pycon\r\nIn [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (31, None, 'max', None, None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'facetable', '', None, None) kwargs {}\r\n```\r\nOnce a subselect is involved the order of the `20` checks no longer matches the order in which the columns are returned from the query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-1235752140", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 1235752140, "node_id": "IC_kwDOBm6k_c5JqBTM", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-09-02T17:34:09Z", "updated_at": "2022-09-02T17:34:09Z", "author_association": "OWNER", "body": "Accidentally closed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898572065", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898572065, "node_id": "IC_kwDOBm6k_c41jx8h", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T16:13:16Z", "updated_at": "2021-08-13T16:13:16Z", "author_association": "OWNER", "body": "Aha! That `MakeRecord` line says `r[5..7]` - and r5 = neighborhood, r6 = facet_cities.name, r7 = facetable.state\r\n\r\nSo if the `MakeRecord` defines what goes into that pseudo-table column 2 of that pseudo-table would be `state` - which is what we want.\r\n\r\nThis is really convoluted. I'm no longer confident I can get this to work in a sensible way, especially since I've not started exploring what complex nested tables with CTEs and sub-selects do yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898527525", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898527525, "node_id": "IC_kwDOBm6k_c41jnEl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T15:08:03Z", "updated_at": "2021-08-13T15:08:03Z", "author_association": "OWNER", "body": "Am I going to need to look at the `ResultRow` and its columns but then wind back to that earlier `MakeRecord` and its columns?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898760808", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898760808, "node_id": "IC_kwDOBm6k_c41kgBo", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T23:03:01Z", "updated_at": "2021-08-13T23:03:01Z", "author_association": "OWNER", "body": "Another idea: strip out any `order by` clause to try and keep this simpler. I doubt that's going to cope with complex nested queries though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898936068", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898936068, "node_id": "IC_kwDOBm6k_c41lK0E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T17:44:54Z", "updated_at": "2021-08-14T17:44:54Z", "author_association": "OWNER", "body": "Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29\r\n\r\nThat one shows `VColumn` instead of `Column`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898541543", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898541543, "node_id": "IC_kwDOBm6k_c41jqfn", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T15:25:26Z", "updated_at": "2021-08-13T15:25:26Z", "author_association": "OWNER", "body": "But the debug output here seems to be saying what we want it to say:\r\n```\r\n17 SorterSort 2 24 0 00 \r\n18 SorterData 2 10 3 00 r[10]=data \r\n19 Column 3 2 8 00 r[8]=state \r\n20 Column 3 1 7 00 r[7]=facet_cities.name\r\n21 Column 3 0 6 00 r[6]=neighborhood\r\n22 ResultRow 6 3 0 00 output=r[6..8]\r\n```\r\nWe want to get back `neighborhood`, `facet_cities.name`, `state`.\r\n\r\nWhy then are we seeing `[('facet_cities', 'name'), ('facetable', 'state'), (None, None)]`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898524057", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898524057, "node_id": "IC_kwDOBm6k_c41jmOZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T15:06:37Z", "updated_at": "2021-08-13T15:06:37Z", "author_association": "OWNER", "body": "Comparing the `explain` for the two versions of that query - one with the order by and one without:\r\n\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898961535", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898961535, "node_id": "IC_kwDOBm6k_c41lRB_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-14T21:37:24Z", "updated_at": "2021-08-14T21:37:24Z", "author_association": "OWNER", "body": "Did some more research into building SQLite custom versions via `pysqlite3` - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-898536181", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 898536181, "node_id": "IC_kwDOBm6k_c41jpL1", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-08-13T15:17:20Z", "updated_at": "2021-08-13T15:20:33Z", "author_association": "OWNER", "body": "Documentation for `MakeRecord`: https://www.sqlite.org/opcode.html#MakeRecord\r\n\r\nRunning `explain` inside `sqlite3` provides extra comments and indentation which make it easier to understand:\r\n```\r\nsqlite> explain select neighborhood, facet_cities.name, state\r\n ...> from facetable\r\n ...> join facet_cities\r\n ...> on facetable.city_id = facet_cities.id\r\n ...> where neighborhood like '%bob%';\r\naddr opcode p1 p2 p3 p4 p5 comment \r\n---- ------------- ---- ---- ---- ------------- -- -------------\r\n0 Init 0 15 0 00 Start at 15 \r\n1 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable\r\n2 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities\r\n3 Rewind 0 14 0 00 \r\n4 Column 0 6 3 00 r[3]=facetable.neighborhood\r\n5 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3])\r\n6 IfNot 1 13 1 00 \r\n7 Column 0 5 4 00 r[4]=facetable.city_id\r\n8 SeekRowid 1 13 4 00 intkey=r[4] \r\n9 Column 0 6 5 00 r[5]=facetable.neighborhood\r\n10 Column 1 1 6 00 r[6]=facet_cities.name\r\n11 Column 0 4 7 00 r[7]=facetable.state\r\n12 ResultRow 5 3 0 00 output=r[5..7]\r\n13 Next 0 4 0 01 \r\n14 Halt 0 0 0 00 \r\n15 Transaction 0 0 35 0 01 usesStmtJournal=0\r\n16 String8 0 2 0 %bob% 00 r[2]='%bob%' \r\n17 Goto 0 1 0 00 \r\n```\r\nCompared with:\r\n```\r\nsqlite> explain select neighborhood, facet_cities.name, state\r\n ...> from facetable\r\n ...> join facet_cities\r\n ...> on facetable.city_id = facet_cities.id\r\n ...> where neighborhood like '%bob%' order by neighborhood\r\n ...> ;\r\naddr opcode p1 p2 p3 p4 p5 comment \r\n---- ------------- ---- ---- ---- ------------- -- -------------\r\n0 Init 0 25 0 00 Start at 25 \r\n1 SorterOpen 2 5 0 k(1,B) 00 \r\n2 OpenRead 0 43 0 7 00 root=43 iDb=0; facetable\r\n3 OpenRead 1 42 0 2 00 root=42 iDb=0; facet_cities\r\n4 Rewind 0 16 0 00 \r\n5 Column 0 6 3 00 r[3]=facetable.neighborhood\r\n6 Function0 1 2 1 like(2) 02 r[1]=func(r[2..3])\r\n7 IfNot 1 15 1 00 \r\n8 Column 0 5 4 00 r[4]=facetable.city_id\r\n9 SeekRowid 1 15 4 00 intkey=r[4] \r\n10 Column 1 1 6 00 r[6]=facet_cities.name\r\n11 Column 0 4 7 00 r[7]=facetable.state\r\n12 Column 0 6 5 00 r[5]=facetable.neighborhood\r\n13 MakeRecord 5 3 9 00 r[9]=mkrec(r[5..7])\r\n14 SorterInsert 2 9 5 3 00 key=r[9] \r\n15 Next 0 5 0 01 \r\n16 OpenPseudo 3 10 5 00 5 columns in r[10]\r\n17 SorterSort 2 24 0 00 \r\n18 SorterData 2 10 3 00 r[10]=data \r\n19 Column 3 2 8 00 r[8]=state \r\n20 Column 3 1 7 00 r[7]=facet_cities.name\r\n21 Column 3 0 6 00 r[6]=neighborhood\r\n22 ResultRow 6 3 0 00 output=r[6..8]\r\n23 SorterNext 2 18 0 00 \r\n24 Halt 0 0 0 00 \r\n25 Transaction 0 0 35 0 01 usesStmtJournal=0\r\n26 String8 0 2 0 %bob% 00 r[2]='%bob%' \r\n27 Goto 0 1 0 00 \r\n```\r\nSo actually it looks like the `SorterSort` may be key to understanding this.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813164282", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813164282, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T03:42:26Z", "updated_at": "2021-04-05T03:42:36Z", "author_association": "OWNER", "body": "Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the `explain select...` query. Using `defaultdict` seems to work there:\r\n\r\n```pycon\r\n>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))\r\n>>> [dict(r) for r in rows if r['opcode'] == 'Variable']\r\n[{'addr': 2,\r\n 'opcode': 'Variable',\r\n 'p1': 1,\r\n 'p2': 1,\r\n 'p3': 0,\r\n 'p4': ':id',\r\n 'p5': 0,\r\n 'comment': None}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134072", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134072, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:18:37Z", "updated_at": "2021-04-05T01:18:37Z", "author_association": "OWNER", "body": "Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by `explain select ...`.\r\n\r\nCopying the reply I posted to this thread:\r\n\r\nThat's really useful, thanks! It looks like it _might_ be possible for me to reconstruct where each column came from using the `explain select` output.\r\n\r\nHere's a complex example:
This data as {% for name, url in renderers.items() %}{{ name }}{{ \", \" if not loop.last }}{% endfor %}, CSV
\r\n