html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1079407962,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1079407962,IC_kwDOCGYnMM5AVnVa,9599,simonw,2022-03-25T20:25:10Z,2022-03-25T20:25:18Z,OWNER,"Can you share either your whole `global.db` table or a shrunk down example that illustrates the bug?
My hunch is that you may have a table or column with a name that triggers the error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1081079506,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1081079506,IC_kwDOCGYnMM5Ab_bS,24938923,learning4life,2022-03-28T19:58:55Z,2022-03-28T20:05:57Z,NONE,"Sure, it is from the documentation example:
[Extracting columns into a separate table](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table)
```
wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true""
sqlite-utils insert global.db power_plants \
'global_power_plant_database.csv?raw=true' --csv
# Extract those columns:
sqlite-utils extract global.db power_plants country country_long \
--table countries \
--fk-column country_id \
--rename country_long name
```
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098288158,IC_kwDOCGYnMM5Bdowe,9599,simonw,2022-04-13T17:07:53Z,2022-04-13T17:07:53Z,OWNER,"I can't replicate the bug I'm afraid:
```
% wget ""https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true""
...
2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038]
% sqlite-utils insert global.db power_plants \
'global_power_plant_database.csv?raw=true' --csv
[------------------------------------] 0%
[###################################-] 99% 00:00:00%
% sqlite-utils indexes global.db --table
table index_name seqno cid name desc coll key
------- ------------ ------- ----- ------ ------ ------ -----
% sqlite-utils extract global.db power_plants country country_long \
--table countries \
--fk-column country_id \
--rename country_long name
% sqlite-utils indexes global.db --table
table index_name seqno cid name desc coll key
--------- -------------------------- ------- ----- ------- ------ ------ -----
countries idx_countries_country_name 0 1 country 0 BINARY 1
countries idx_countries_country_name 1 2 name 0 BINARY 1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098295517,IC_kwDOCGYnMM5Bdqjd,9599,simonw,2022-04-13T17:16:20Z,2022-04-13T17:16:20Z,OWNER,"Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that.
```
(app-root) sqlite-utils indexes global.db --table
Error: near ""("": syntax error
```
(That wa sbefore I even ran the `extract` command.)
To build your `Dockerfile` I copied it into an empty folder and ran the following:
```
wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz
docker build . -t centos-sqlite-utils
docker run -it centos-sqlite-utils /bin/bash
```
This gave me a shell in which I could replicate the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098531354,IC_kwDOCGYnMM5BekIa,9599,simonw,2022-04-13T22:08:20Z,2022-04-13T22:08:20Z,OWNER,"OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running:
```
(app-root) sqlite-utils indexes global.db --table
select
sqlite_master.name as ""table"",
indexes.name as index_name,
xinfo.*
from sqlite_master
join pragma_index_list(sqlite_master.name) indexes
join pragma_index_xinfo(index_name) xinfo
where
sqlite_master.type = 'table'
and xinfo.key = 1
Error: near ""("": syntax error
```
This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version:
```
(app-root) sqlite3
SQLite version 3.36.0 2021-06-18 18:36:39
```
That version should be fine - it's the one you compiled in the Dockerfile.
Then I checked the version that `sqlite-utils` itself was using:
```
(app-root) sqlite-utils memory 'select sqlite_version()'
[{""sqlite_version()"": ""3.7.17""}]
```
It's running SQLite 3.7.17!
So the problem here is that the Python in that Docker image is running a very old version of SQLite.
I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked:
```
(app-root) python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())'
('3.7.17',)
(app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect("":memory"").execute(""select sqlite_version()"").fetchone())'
('3.36.0',)
```
But when I try to run `sqlite-utils` like that I get an error:
```
(app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db
...
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py"", line 1624, in query
db.register_fts4_bm25()
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 412, in register_fts4_bm25
self.register_function(rank_bm25, deterministic=True)
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 408, in register_function
register(fn)
File ""/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py"", line 401, in register
self.conn.create_function(name, arity, fn, **kwargs)
sqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098532220,IC_kwDOCGYnMM5BekV8,9599,simonw,2022-04-13T22:09:52Z,2022-04-13T22:09:52Z,OWNER,That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098535531,IC_kwDOCGYnMM5BelJr,9599,simonw,2022-04-13T22:15:48Z,2022-04-13T22:15:48Z,OWNER,"Trying this alternative implementation of the `register()` method:
```python
def register(fn):
name = fn.__name__
arity = len(inspect.signature(fn).parameters)
if not replace and (name, arity) in self._registered_functions:
return fn
kwargs = {}
done = False
if deterministic:
# Try this, but fall back if sqlite3.NotSupportedError
try:
self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True))
done = True
except sqlite3.NotSupportedError:
pass
if not done:
self.conn.create_function(name, arity, fn, **kwargs)
self._registered_functions.add((name, arity))
return fn
```
With that fix, the following worked!
```
LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table
table index_name seqno cid name desc coll key
--------- -------------------------- ------- ----- ------- ------ ------ -----
countries idx_countries_country_name 0 1 country 0 BINARY 1
countries idx_countries_country_name 1 2 name 0 BINARY 1
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098548931,https://api.github.com/repos/simonw/sqlite-utils/issues/421,1098548931,IC_kwDOCGYnMM5BeobD,9599,simonw,2022-04-13T22:41:59Z,2022-04-13T22:41:59Z,OWNER,"I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the `LD_PRELOAD` workaround above should work OK with the next release of `sqlite-utils`. Thanks for the detailed bug report!","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1180427792,"""Error: near ""("": syntax error"" when using sqlite-utils indexes CLI",