issue_comments

I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the LD_PRELOAD workaround above should work OK with the next release of sqlite-utils. Thanks for the detailed bug report!

Trying this alternative implementation of the register() method:

python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn With that fix, the following worked! ``` LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table table index_name seqno cid name desc coll key

countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```

That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400

OK I figured out what's going on here. First I added an extra print(sql) statement to the indexes command to see what SQL it was running: ``` (app-root) sqlite-utils indexes global.db --table

select
  sqlite_master.name as "table",
  indexes.name as index_name,
  xinfo.*
from sqlite_master
  join pragma_index_list(sqlite_master.name) indexes
  join pragma_index_xinfo(index_name) xinfo
where
  sqlite_master.type = 'table'
 and xinfo.key = 1

Error: near "(": syntax error This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version: (app-root) sqlite3 SQLite version 3.36.0 2021-06-18 18:36:39 ``` That version should be fine - it's the one you compiled in the Dockerfile.

Then I checked the version that sqlite-utils itself was using: (app-root) sqlite-utils memory 'select sqlite_version()' [{"sqlite_version()": "3.7.17"}] It's running SQLite 3.7.17!

So the problem here is that the Python in that Docker image is running a very old version of SQLite.

I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked:

(app-root) python3 -c 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())' ('3.7.17',) (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())' ('3.36.0',) But when I try to run sqlite-utils like that I get an error:

(app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db ... File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py", line 1624, in query db.register_fts4_bm25() File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 412, in register_fts4_bm25 self.register_function(rank_bm25, deterministic=True) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 408, in register_function register(fn) File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 401, in register self.conn.create_function(name, arity, fn, **kwargs) sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher

Aha! I was able to replicate the bug using your Dockerfile - thanks very much for providing that. (app-root) sqlite-utils indexes global.db --table Error: near "(": syntax error (That wa sbefore I even ran the extract command.)

To build your Dockerfile I copied it into an empty folder and ran the following: wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz docker build . -t centos-sqlite-utils docker run -it centos-sqlite-utils /bin/bash This gave me a shell in which I could replicate the bug.

I can't replicate the bug I'm afraid: ``` % wget "https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true"
... 2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038] % sqlite-utils insert global.db power_plants \
'global_power_plant_database.csv?raw=true' --csv [------------------------------------] 0% [###################################-] 99% 00:00:00% % sqlite-utils indexes global.db --table
table index_name seqno cid name desc coll key

% sqlite-utils extract global.db power_plants country country_long \ --table countries \ --fk-column country_id \ --rename country_long name % sqlite-utils indexes global.db --table
table index_name seqno cid name desc coll key

countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ```

Can you share either your whole global.db table or a shrunk down example that illustrates the bug?

My hunch is that you may have a table or column with a name that triggers the error.