issue_comments

Did some more research into building SQLite custom versions via pysqlite3 - here's what I figured out for macOS (which should hopefully work for Linux too): https://til.simonwillison.net/sqlite/build-specific-sqlite-pysqlite-macos

Another interesting query to consider: https://latest.datasette.io/fixtures?sql=explain+select+*+from++pragma_table_info%28+%27123_starts_with_digits%27%29

That one shows VColumn instead of Column.

Maybe I split this out into a separate Python library that gets tested against every SQLite release I can possibly try it against, and then bakes out the supported release versions into the library code itself?

Datasette could depend on that library. The library could be released independently of Datasette any time a new SQLite version comes out.

I could even run a separate git scraper repo that checks for new SQLite releases and submits PRs against the library when a new release comes out.

I would feel a lot more comfortable about all of this if I had a robust mechanism for running the Datasette test suite against multiple versions of SQLite itself.

I think I need to care about the following:

• ResultRow and Column for the final result
• OpenRead for opening tables
• OpenEphemeral then MakeRecord and IdxInsert for writing records into ephemeral tables

Column may reference either a table (from OpenRead) or an ephemeral table (from OpenEphemeral).

That might be enough.

No visible backticks on https://sqlite-utils.datasette.io/en/latest/reference.html any more.

Tried a more complicated query: sql explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 Here's the explain: ``` sqlite> explain select pk, text1, text2, [name with . and spaces] from searchable where rowid in (select rowid from searchable_fts where searchable_fts match escape_fts(:search)) order by text1 desc limit 101 ...> ; addr opcode p1 p2 p3 p4 p5 comment

0 Init 0 41 0 00 Start at 41
1 OpenEphemeral 2 6 0 k(1,-B) 00 nColumn=6
2 Integer 101 1 0 00 r[1]=101; LIMIT counter 3 OpenRead 0 32 0 4 00 root=32 iDb=0; searchable 4 Integer 16 3 0 00 r[3]=16; return address 5 Once 0 16 0 00
6 OpenEphemeral 3 1 0 k(1,) 00 nColumn=1; Result of SELECT 1 7 VOpen 1 0 0 vtab:7FCBCA72BE80 00
8 Function0 1 7 6 unknown(-1) 01 r[6]=func(r[7]) 9 Integer 5 4 0 00 r[4]=5
10 Integer 1 5 0 00 r[5]=1
11 VFilter 1 16 4 00 iplan=r[4] zplan='' 12 Rowid 1 8 0 00 r[8]=rowid
13 MakeRecord 8 1 9 C 00 r[9]=mkrec(r[8]) 14 IdxInsert 3 9 8 1 00 key=r[9]
15 VNext 1 12 0 00
16 Return 3 0 0 00
17 Rewind 3 33 0 00
18 Column 3 0 2 00 r[2]=
19 IsNull 2 32 0 00 if r[2]==NULL goto 32 20 SeekRowid 0 32 2 00 intkey=r[2]
21 Column 0 1 10 00 r[10]=searchable.text1 22 Sequence 2 11 0 00 r[11]=cursor[2].ctr++ 23 IfNotZero 1 27 0 00 if r[1]!=0 then r[1]--, goto 27 24 Last 2 0 0 00
25 IdxLE 2 32 10 1 00 key=r[10]
26 Delete 2 0 0 00
27 Rowid 0 12 0 00 r[12]=rowid
28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16]
32 Next 3 18 0 00
33 Sort 2 40 0 00
34 Column 2 4 15 00 r[15]=[name with . and spaces] 35 Column 2 3 14 00 r[14]=text2
36 Column 2 0 13 00 r[13]=text1
37 Column 2 2 12 00 r[12]=pk
38 ResultRow 12 4 0 00 output=r[12..15] 39 Next 2 34 0 00
40 Halt 0 0 0 00
41 Transaction 0 0 35 0 01 usesStmtJournal=0 42 Variable 1 7 0 :search 00 r[7]=parameter(1,:search) 43 Goto 0 1 0 00
`` Here theResultRowis for registers12..15- but those all refer toColumnrecords in2- where2is the firstOpenEphemeraldeclared right at the start. I'm having enormous trouble figuring out how that ephemeral table gets populated by the other operations in a way that would let me derive which columns end up in theResultRow`.

Frustratingly SQLite seems to be able to figure that out just fine, see the column of comments on the right hand side - but I only get those in the sqlite3 CLI shell, they're not available to me with SQLite when called as a library from Python.

Maybe the key to that is this section: 27 Rowid 0 12 0 00 r[12]=rowid 28 Column 0 2 13 00 r[13]=searchable.text2 29 Column 0 3 14 00 r[14]=searchable.name with . and spaces 30 MakeRecord 10 5 16 00 r[16]=mkrec(r[10..14]) 31 IdxInsert 2 16 10 5 00 key=r[16] MakeRecord:

Convert P2 registers beginning with P1 into the record format use as a data record in a database table or as a key in an index. The Column opcode can decode the record later.

P4 may be a string that is P2 characters long. The N-th character of the string indicates the column affinity that should be used for the N-th field of the index key.

The mapping from character to affinity is given by the SQLITE_AFF_ macros defined in sqliteInt.h.

If P4 is NULL then all index fields have the affinity BLOB.

The meaning of P5 depends on whether or not the SQLITE_ENABLE_NULL_TRIM compile-time option is enabled:

• If SQLITE_ENABLE_NULL_TRIM is enabled, then the P5 is the index of the right-most table that can be null-trimmed.

• If SQLITE_ENABLE_NULL_TRIM is omitted, then P5 has the value OPFLAG_NOCHNG_MAGIC if the MakeRecord opcode is allowed to accept no-change records with serial_type 10. This value is only used inside an assert() and does not affect the end result.

IdxInsert:

Register P2 holds an SQL index key made using the MakeRecord instructions. This opcode writes that key into the index P1. Data for the entry is nil.

If P4 is not zero, then it is the number of values in the unpacked key of reg(P2). In that case, P3 is the index of the first register for the unpacked key. The availability of the unpacked key can sometimes be an optimization.

If P5 has the OPFLAG_APPEND bit set, that is a hint to the b-tree layer that this insert is likely to be an append.

If P5 has the OPFLAG_NCHANGE bit set, then the change counter is incremented by this instruction. If the OPFLAG_NCHANGE bit is clear, then the change counter is unchanged.

If the OPFLAG_USESEEKRESULT flag of P5 is set, the implementation might run faster by avoiding an unnecessary seek on cursor P1. However, the OPFLAG_USESEEKRESULT flag must only be set if there have been no prior seeks on the cursor or if the most recent seek used a key equivalent to P2.

This instruction only works for indices. The equivalent instruction for tables is Insert.

IdxLE:

The P4 register values beginning with P3 form an unpacked index key that omits the PRIMARY KEY or ROWID. Compare this key value against the index that P1 is currently pointing to, ignoring the PRIMARY KEY or ROWID on the P1 index.

If the P1 index entry is less than or equal to the key value then jump to P2. Otherwise fall through to the next instruction.