html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/datasette/issues/1293#issuecomment-813480043,https://api.github.com/repos/simonw/datasette/issues/1293,813480043,MDEyOklzc3VlQ29tbWVudDgxMzQ4MDA0Mw==,9599,2021-04-05T16:16:17Z,2021-04-05T16:16:17Z,OWNER,"https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because `paginated_view` is defined like this:
```sql
CREATE VIEW paginated_view AS
SELECT
content,
'- ' || content || ' -' AS content_extra
FROM no_primary_key;
```
So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813445512,https://api.github.com/repos/simonw/datasette/issues/1293,813445512,MDEyOklzc3VlQ29tbWVudDgxMzQ0NTUxMg==,9599,2021-04-05T15:11:40Z,2021-04-05T15:11:40Z,OWNER,"Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813438771,https://api.github.com/repos/simonw/datasette/issues/1293,813438771,MDEyOklzc3VlQ29tbWVudDgxMzQzODc3MQ==,9599,2021-04-05T14:58:48Z,2021-04-05T14:58:48Z,OWNER,I may need to do something special for rowid columns - there is a `RowId` opcode that might come into play here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/dogsheep/dogsheep-photos/issues/35#issuecomment-813249000,https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35,813249000,MDEyOklzc3VlQ29tbWVudDgxMzI0OTAwMA==,1151557,2021-04-05T07:37:57Z,2021-04-05T07:37:57Z,NONE,"There are trained ML models used in Photoprism:
- https://dl.photoprism.org/tensorflow/nasnet.zip
- https://dl.photoprism.org/tensorflow/nsfw.zip","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",842695374,
https://github.com/simonw/datasette/issues/620#issuecomment-813167335,https://api.github.com/repos/simonw/datasette/issues/620,813167335,MDEyOklzc3VlQ29tbWVudDgxMzE2NzMzNQ==,9599,2021-04-05T03:57:22Z,2021-04-05T03:57:22Z,OWNER,This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520667773,
https://github.com/simonw/datasette/issues/1293#issuecomment-813164282,https://api.github.com/repos/simonw/datasette/issues/1293,813164282,MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg==,9599,2021-04-05T03:42:26Z,2021-04-05T03:42:36Z,OWNER,"Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the `explain select...` query. Using `defaultdict` seems to work there:
```pycon
>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))
>>> [dict(r) for r in rows if r['opcode'] == 'Variable']
[{'addr': 2,
'opcode': 'Variable',
'p1': 1,
'p2': 1,
'p3': 0,
'p4': ':id',
'p5': 0,
'comment': None}]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813162622,https://api.github.com/repos/simonw/datasette/issues/1293,813162622,MDEyOklzc3VlQ29tbWVudDgxMzE2MjYyMg==,9599,2021-04-05T03:34:24Z,2021-04-05T03:40:35Z,OWNER,"This almost works, but throws errors with some queries (anything with a `rowid` column for example) - it needs a bunch of test coverage.
```python
def columns_for_query(conn, sql):
rows = conn.execute('explain ' + sql).fetchall()
table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
names_by_rootpage = dict(
conn.execute(
'select rootpage, name from sqlite_master where rootpage in ({})'.format(
', '.join(map(str, table_rootpage_by_register.values()))
)
)
)
columns_by_column_register = {}
for row in rows:
if row['opcode'] == 'Column':
addr, opcode, table_id, cid, column_register, p4, p5, comment = row
table = names_by_rootpage[table_rootpage_by_register[table_id]]
columns_by_column_register[column_register] = (table, cid)
result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
registers = list(range(result_row[""p1""], result_row[""p1""] + result_row[""p2""] - 1))
all_column_names = {}
for table in names_by_rootpage.values():
table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
for row in table_xinfo:
all_column_names[(table, row[""cid""])] = row[""name""]
final_output = []
for r in registers:
try:
table, cid = columns_by_column_register[r]
final_output.append((table, all_column_names[table, cid]))
except KeyError:
final_output.append((None, None))
return final_output
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134637,https://api.github.com/repos/simonw/datasette/issues/1293,813134637,MDEyOklzc3VlQ29tbWVudDgxMzEzNDYzNw==,9599,2021-04-05T01:21:59Z,2021-04-05T01:21:59Z,OWNER,"http://www.sqlite.org/draft/lang_explain.html says:
> Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.
I'm going to keep exploring this though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134227,https://api.github.com/repos/simonw/datasette/issues/1293,813134227,MDEyOklzc3VlQ29tbWVudDgxMzEzNDIyNw==,9599,2021-04-05T01:19:31Z,2021-04-05T01:19:31Z,OWNER,"| addr | opcode | p1 | p2 | p3 | p4 | p5 | comment |
|--------|---------------|------|------|------|-----------------------|------|-----------|
| 0 | Init | 0 | 47 | 0 | | 00 | |
| 1 | OpenRead | 0 | 51 | 0 | 15 | 00 | |
| 2 | Integer | 15 | 2 | 0 | | 00 | |
| 3 | Once | 0 | 15 | 0 | | 00 | |
| 4 | OpenEphemeral | 2 | 1 | 0 | k(1,) | 00 | |
| 5 | VOpen | 1 | 0 | 0 | vtab:3E692C362158 | 00 | |
| 6 | String8 | 0 | 5 | 0 | CPAD_2020a_SuperUnits | 00 | |
| 7 | SCopy | 7 | 6 | 0 | | 00 | |
| 8 | Integer | 2 | 3 | 0 | | 00 | |
| 9 | Integer | 2 | 4 | 0 | | 00 | |
| 10 | VFilter | 1 | 15 | 3 | | 00 | |
| 11 | Rowid | 1 | 8 | 0 | | 00 | |
| 12 | MakeRecord | 8 | 1 | 9 | C | 00 | |
| 13 | IdxInsert | 2 | 9 | 8 | 1 | 00 | |
| 14 | VNext | 1 | 11 | 0 | | 00 | |
| 15 | Return | 2 | 0 | 0 | | 00 | |
| 16 | Rewind | 2 | 46 | 0 | | 00 | |
| 17 | Column | 2 | 0 | 1 | | 00 | |
| 18 | IsNull | 1 | 45 | 0 | | 00 | |
| 19 | SeekRowid | 0 | 45 | 1 | | 00 | |
| 20 | Column | 0 | 2 | 11 | | 00 | |
| 21 | Function0 | 1 | 10 | 9 | like(2) | 02 | |
| 22 | IfNot | 9 | 45 | 1 | | 00 | |
| 23 | Column | 0 | 14 | 13 | | 00 | |
| 24 | Function0 | 1 | 12 | 9 | intersects(2) | 02 | |
| 25 | Ne | 14 | 45 | 9 | | 51 | |
| 26 | Column | 0 | 14 | 9 | | 00 | |
| 27 | Function0 | 0 | 9 | 15 | asgeojson(1) | 01 | |
| 28 | Rowid | 0 | 16 | 0 | | 00 | |
| 29 | Column | 0 | 1 | 17 | | 00 | |
| 30 | Column | 0 | 2 | 18 | | 00 | |
| 31 | Column | 0 | 3 | 19 | | 00 | |
| 32 | Column | 0 | 4 | 20 | | 00 | |
| 33 | Column | 0 | 5 | 21 | | 00 | |
| 34 | Column | 0 | 6 | 22 | | 00 | |
| 35 | Column | 0 | 7 | 23 | | 00 | |
| 36 | Column | 0 | 8 | 24 | | 00 | |
| 37 | Column | 0 | 9 | 25 | | 00 | |
| 38 | Column | 0 | 10 | 26 | | 00 | |
| 39 | Column | 0 | 11 | 27 | | 00 | |
| 40 | RealAffinity | 27 | 0 | 0 | | 00 | |
| 41 | Column | 0 | 12 | 28 | | 00 | |
| 42 | Column | 0 | 13 | 29 | | 00 | |
| 43 | Column | 0 | 14 | 30 | | 00 | |
| 44 | ResultRow | 15 | 16 | 0 | | 00 | |
| 45 | Next | 2 | 17 | 0 | | 00 | |
| 46 | Halt | 0 | 0 | 0 | | 00 | |
| 47 | Transaction | 0 | 0 | 265 | 0 | 01 | |
| 48 | Variable | 1 | 31 | 0 | :freedraw | 00 | |
| 49 | Function0 | 1 | 31 | 7 | geomfromgeojson(1) | 01 | |
| 50 | String8 | 0 | 10 | 0 | %mini% | 00 | |
| 51 | Variable | 1 | 32 | 0 | :freedraw | 00 | |
| 52 | Function0 | 1 | 32 | 12 | geomfromgeojson(1) | 01 | |
| 53 | Integer | 1 | 14 | 0 | | 00 | |
| 54 | Goto | 0 | 1 | 0 | | 00 | |
Essential documentation for understanding that output: https://www.sqlite.org/opcode.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,
https://github.com/simonw/datasette/issues/1293#issuecomment-813134072,https://api.github.com/repos/simonw/datasette/issues/1293,813134072,MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg==,9599,2021-04-05T01:18:37Z,2021-04-05T01:18:37Z,OWNER,"Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by `explain select ...`.
Copying the reply I posted to this thread:
That's really useful, thanks! It looks like it _might_ be possible for me to reconstruct where each column came from using the `explain select` output.
Here's a complex example:
It looks like the opcodes I need to inspect are `OpenRead`, `Column` and `ResultRow`.
`OpenRead` tells me which tables are being opened - the `p2` value (in this case 51) corresponds to the `rootpage` column in `sqlite_master` here: - it gets assigned to the register in `p1`.
The `Column` opcodes tell me which columns are being read - `p1` is that table reference, and `p2` is the `cid` of the column within that table.
The `ResultRow` opcode then tells me which columns are used in the results. `15 16` means start at the 15th and then read the next 16 columns.
I think this might work!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",849978964,