{"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813480043", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813480043, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzQ4MDA0Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T16:16:17Z", "updated_at": "2021-04-05T16:16:17Z", "author_association": "OWNER", "body": "https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:\r\n\r\n```sql\r\nCREATE VIEW paginated_view AS\r\n SELECT\r\n content,\r\n '- ' || content || ' -' AS content_extra\r\n FROM no_primary_key;\r\n```\r\nSo this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813445512", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813445512, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzQ0NTUxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T15:11:40Z", "updated_at": "2021-04-05T15:11:40Z", "author_association": "OWNER", "body": "Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813438771", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813438771, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzQzODc3MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T14:58:48Z", "updated_at": "2021-04-05T14:58:48Z", "author_association": "OWNER", "body": "I may need to do something special for rowid columns - there is a `RowId` opcode that might come into play here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813164282", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813164282, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T03:42:26Z", "updated_at": "2021-04-05T03:42:36Z", "author_association": "OWNER", "body": "Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the `explain select...` query. Using `defaultdict` seems to work there:\r\n\r\n```pycon\r\n>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))\r\n>>> [dict(r) for r in rows if r['opcode'] == 'Variable']\r\n[{'addr': 2,\r\n 'opcode': 'Variable',\r\n 'p1': 1,\r\n 'p2': 1,\r\n 'p3': 0,\r\n 'p4': ':id',\r\n 'p5': 0,\r\n 'comment': None}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813162622", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813162622, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzE2MjYyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T03:34:24Z", "updated_at": "2021-04-05T03:40:35Z", "author_association": "OWNER", "body": "This almost works, but throws errors with some queries (anything with a `rowid` column for example) - it needs a bunch of test coverage.\r\n```python\r\ndef columns_for_query(conn, sql):\r\n rows = conn.execute('explain ' + sql).fetchall()\r\n table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}\r\n names_by_rootpage = dict(\r\n conn.execute(\r\n 'select rootpage, name from sqlite_master where rootpage in ({})'.format(\r\n ', '.join(map(str, table_rootpage_by_register.values()))\r\n )\r\n )\r\n )\r\n columns_by_column_register = {}\r\n for row in rows:\r\n if row['opcode'] == 'Column':\r\n addr, opcode, table_id, cid, column_register, p4, p5, comment = row\r\n table = names_by_rootpage[table_rootpage_by_register[table_id]]\r\n columns_by_column_register[column_register] = (table, cid)\r\n result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]\r\n registers = list(range(result_row[\"p1\"], result_row[\"p1\"] + result_row[\"p2\"] - 1))\r\n all_column_names = {}\r\n for table in names_by_rootpage.values():\r\n table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()\r\n for row in table_xinfo:\r\n all_column_names[(table, row[\"cid\"])] = row[\"name\"]\r\n final_output = []\r\n for r in registers:\r\n try:\r\n table, cid = columns_by_column_register[r]\r\n final_output.append((table, all_column_names[table, cid]))\r\n except KeyError:\r\n final_output.append((None, None))\r\n return final_output\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134637", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134637, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDYzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:21:59Z", "updated_at": "2021-04-05T01:21:59Z", "author_association": "OWNER", "body": "http://www.sqlite.org/draft/lang_explain.html says:\r\n\r\n> Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.\r\n\r\nI'm going to keep exploring this though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134227", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134227, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDIyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:19:31Z", "updated_at": "2021-04-05T01:19:31Z", "author_association": "OWNER", "body": "| addr | opcode | p1 | p2 | p3 | p4 | p5 | comment |\r\n|--------|---------------|------|------|------|-----------------------|------|-----------|\r\n| 0 | Init | 0 | 47 | 0 | | 00 | |\r\n| 1 | OpenRead | 0 | 51 | 0 | 15 | 00 | |\r\n| 2 | Integer | 15 | 2 | 0 | | 00 | |\r\n| 3 | Once | 0 | 15 | 0 | | 00 | |\r\n| 4 | OpenEphemeral | 2 | 1 | 0 | k(1,) | 00 | |\r\n| 5 | VOpen | 1 | 0 | 0 | vtab:3E692C362158 | 00 | |\r\n| 6 | String8 | 0 | 5 | 0 | CPAD_2020a_SuperUnits | 00 | |\r\n| 7 | SCopy | 7 | 6 | 0 | | 00 | |\r\n| 8 | Integer | 2 | 3 | 0 | | 00 | |\r\n| 9 | Integer | 2 | 4 | 0 | | 00 | |\r\n| 10 | VFilter | 1 | 15 | 3 | | 00 | |\r\n| 11 | Rowid | 1 | 8 | 0 | | 00 | |\r\n| 12 | MakeRecord | 8 | 1 | 9 | C | 00 | |\r\n| 13 | IdxInsert | 2 | 9 | 8 | 1 | 00 | |\r\n| 14 | VNext | 1 | 11 | 0 | | 00 | |\r\n| 15 | Return | 2 | 0 | 0 | | 00 | |\r\n| 16 | Rewind | 2 | 46 | 0 | | 00 | |\r\n| 17 | Column | 2 | 0 | 1 | | 00 | |\r\n| 18 | IsNull | 1 | 45 | 0 | | 00 | |\r\n| 19 | SeekRowid | 0 | 45 | 1 | | 00 | |\r\n| 20 | Column | 0 | 2 | 11 | | 00 | |\r\n| 21 | Function0 | 1 | 10 | 9 | like(2) | 02 | |\r\n| 22 | IfNot | 9 | 45 | 1 | | 00 | |\r\n| 23 | Column | 0 | 14 | 13 | | 00 | |\r\n| 24 | Function0 | 1 | 12 | 9 | intersects(2) | 02 | |\r\n| 25 | Ne | 14 | 45 | 9 | | 51 | |\r\n| 26 | Column | 0 | 14 | 9 | | 00 | |\r\n| 27 | Function0 | 0 | 9 | 15 | asgeojson(1) | 01 | |\r\n| 28 | Rowid | 0 | 16 | 0 | | 00 | |\r\n| 29 | Column | 0 | 1 | 17 | | 00 | |\r\n| 30 | Column | 0 | 2 | 18 | | 00 | |\r\n| 31 | Column | 0 | 3 | 19 | | 00 | |\r\n| 32 | Column | 0 | 4 | 20 | | 00 | |\r\n| 33 | Column | 0 | 5 | 21 | | 00 | |\r\n| 34 | Column | 0 | 6 | 22 | | 00 | |\r\n| 35 | Column | 0 | 7 | 23 | | 00 | |\r\n| 36 | Column | 0 | 8 | 24 | | 00 | |\r\n| 37 | Column | 0 | 9 | 25 | | 00 | |\r\n| 38 | Column | 0 | 10 | 26 | | 00 | |\r\n| 39 | Column | 0 | 11 | 27 | | 00 | |\r\n| 40 | RealAffinity | 27 | 0 | 0 | | 00 | |\r\n| 41 | Column | 0 | 12 | 28 | | 00 | |\r\n| 42 | Column | 0 | 13 | 29 | | 00 | |\r\n| 43 | Column | 0 | 14 | 30 | | 00 | |\r\n| 44 | ResultRow | 15 | 16 | 0 | | 00 | |\r\n| 45 | Next | 2 | 17 | 0 | | 00 | |\r\n| 46 | Halt | 0 | 0 | 0 | | 00 | |\r\n| 47 | Transaction | 0 | 0 | 265 | 0 | 01 | |\r\n| 48 | Variable | 1 | 31 | 0 | :freedraw | 00 | |\r\n| 49 | Function0 | 1 | 31 | 7 | geomfromgeojson(1) | 01 | |\r\n| 50 | String8 | 0 | 10 | 0 | %mini% | 00 | |\r\n| 51 | Variable | 1 | 32 | 0 | :freedraw | 00 | |\r\n| 52 | Function0 | 1 | 32 | 12 | geomfromgeojson(1) | 01 | |\r\n| 53 | Integer | 1 | 14 | 0 | | 00 | |\r\n| 54 | Goto | 0 | 1 | 0 | | 00 | |\r\n\r\nEssential documentation for understanding that output: https://www.sqlite.org/opcode.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813134072", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813134072, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-05T01:18:37Z", "updated_at": "2021-04-05T01:18:37Z", "author_association": "OWNER", "body": "Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by `explain select ...`.\r\n\r\nCopying the reply I posted to this thread:\r\n\r\nThat's really useful, thanks! It looks like it _might_ be possible for me to reconstruct where each column came from using the `explain select` output.\r\n\r\nHere's a complex example: \r\n\r\nIt looks like the opcodes I need to inspect are `OpenRead`, `Column` and `ResultRow`.\r\n\r\n`OpenRead` tells me which tables are being opened - the `p2` value (in this case 51) corresponds to the `rootpage` column in `sqlite_master` here: - it gets assigned to the register in `p1`.\r\n\r\nThe `Column` opcodes tell me which columns are being read - `p1` is that table reference, and `p2` is the `cid` of the column within that table.\r\n\r\nThe `ResultRow` opcode then tells me which columns are used in the results. `15 16` means start at the 15th and then read the next 16 columns.\r\n\r\nI think this might work!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813116177", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813116177, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNjE3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:31:00Z", "updated_at": "2021-04-04T23:31:00Z", "author_association": "OWNER", "body": "Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:\r\n\r\n\"sql-metadata_-_Jupyter_Notebook\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813115607", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813115607, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNTYwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:25:15Z", "updated_at": "2021-04-04T23:25:15Z", "author_association": "OWNER", "body": "Oh wow, I just spotted https://github.com/macbre/sql-metadata\r\n\r\n> Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.\r\n\r\nIt's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813115414", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813115414, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNTQxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:23:34Z", "updated_at": "2021-04-04T23:23:34Z", "author_association": "OWNER", "body": "The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment `select *` came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the `*`.\r\n\r\nA Python parser approach might be good enough to handle a subset of queries - those that don't use `select *` for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813114933", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813114933, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExNDkzMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:19:22Z", "updated_at": "2021-04-04T23:19:22Z", "author_association": "OWNER", "body": "I asked about this on the SQLite forum: https://sqlite.org/forum/forumpost/0180277fb7", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113653", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113653, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzY1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:10:49Z", "updated_at": "2021-04-04T23:10:49Z", "author_association": "OWNER", "body": "One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?\r\n\r\nThen I could load that extension and run a SQL query something like this:\r\n\r\n```\r\nselect database, table, column from analyze_query(:sql_query)\r\n```\r\nWhere `analyze_query(...)` would be a fancy virtual table function of some sort that uses the underlying `sqlite3_column_database_name()` C functions to analyze the SQL query and return details of what it would return.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113403", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113403, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzQwMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:08:48Z", "updated_at": "2021-04-04T23:08:48Z", "author_association": "OWNER", "body": "Worth noting that adding `limit 0` to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:\r\n```pycon\r\nIn [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()\r\n ...: \r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'content', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'created', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'planet_int', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'on_earth', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'state', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'city_id', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'neighborhood', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'tags', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'complex_array', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}\r\nOut[20]: []\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113218", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113218, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzIxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:07:25Z", "updated_at": "2021-04-04T23:07:25Z", "author_association": "OWNER", "body": "Here are all of the available constants:\r\n```pycon\r\nIn [3]: for k in dir(sqlite3):\r\n ...: if k.startswith(\"SQLITE_\"):\r\n ...: print(k, getattr(sqlite3, k))\r\n ...: \r\nSQLITE_ALTER_TABLE 26\r\nSQLITE_ANALYZE 28\r\nSQLITE_ATTACH 24\r\nSQLITE_CREATE_INDEX 1\r\nSQLITE_CREATE_TABLE 2\r\nSQLITE_CREATE_TEMP_INDEX 3\r\nSQLITE_CREATE_TEMP_TABLE 4\r\nSQLITE_CREATE_TEMP_TRIGGER 5\r\nSQLITE_CREATE_TEMP_VIEW 6\r\nSQLITE_CREATE_TRIGGER 7\r\nSQLITE_CREATE_VIEW 8\r\nSQLITE_CREATE_VTABLE 29\r\nSQLITE_DELETE 9\r\nSQLITE_DENY 1\r\nSQLITE_DETACH 25\r\nSQLITE_DONE 101\r\nSQLITE_DROP_INDEX 10\r\nSQLITE_DROP_TABLE 11\r\nSQLITE_DROP_TEMP_INDEX 12\r\nSQLITE_DROP_TEMP_TABLE 13\r\nSQLITE_DROP_TEMP_TRIGGER 14\r\nSQLITE_DROP_TEMP_VIEW 15\r\nSQLITE_DROP_TRIGGER 16\r\nSQLITE_DROP_VIEW 17\r\nSQLITE_DROP_VTABLE 30\r\nSQLITE_FUNCTION 31\r\nSQLITE_IGNORE 2\r\nSQLITE_INSERT 18\r\nSQLITE_OK 0\r\nSQLITE_PRAGMA 19\r\nSQLITE_READ 20\r\nSQLITE_RECURSIVE 33\r\nSQLITE_REINDEX 27\r\nSQLITE_SAVEPOINT 32\r\nSQLITE_SELECT 21\r\nSQLITE_TRANSACTION 22\r\nSQLITE_UPDATE 23\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813113175", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813113175, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMzE3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:07:01Z", "updated_at": "2021-04-04T23:07:01Z", "author_association": "OWNER", "body": "A more promising route I found involved the `db.set_authorizer` method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!\r\n\r\n```pycon\r\n>>> def print_args(*args, **kwargs):\r\n... print(\"args\", args, \"kwargs\", kwargs)\r\n... return sqlite3.SQLITE_OK\r\n\r\n>>> db = sqlite3.connect(\"fixtures.db\")\r\n>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}\r\nargs (20, 'compound_primary_key', 'content', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'created', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'planet_int', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'on_earth', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'state', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'city_id', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'neighborhood', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'tags', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'complex_array', 'main', None) kwargs {}\r\nargs (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}\r\n```\r\nThose `20` values (where 20 is `SQLITE_READ`) looked like they were checking permissions for the columns in the order they would be returned!\r\n\r\nThen I found a snag:\r\n\r\n```pycon\r\nIn [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (31, None, 'max', None, None) kwargs {}\r\nargs (20, 'facetable', 'pk', 'main', None) kwargs {}\r\nargs (21, None, None, None, None) kwargs {}\r\nargs (20, 'facetable', '', None, None) kwargs {}\r\n```\r\nOnce a subselect is involved the order of the `20` checks no longer matches the order in which the columns are returned from the query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/1293#issuecomment-813112546", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/1293", "id": 813112546, "node_id": "MDEyOklzc3VlQ29tbWVudDgxMzExMjU0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2021-04-04T23:02:45Z", "updated_at": "2021-04-04T23:02:45Z", "author_association": "OWNER", "body": "I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.\r\n\r\nThe SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.\r\n\r\nMaybe it would be possible to use them via `ctypes`? My hunch is that I would have to re-implement the full `sqlite3` module with `ctypes`, which sounds daunting.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 849978964, "label": "Show column metadata plus links for foreign keys on arbitrary query results"}, "performed_via_github_app": null}