github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531 | https://api.github.com/repos/simonw/sqlite-utils/issues/421 | 1098535531 | IC_kwDOCGYnMM5BelJr | 9599 | 2022-04-13T22:15:48Z | 2022-04-13T22:15:48Z | OWNER | Trying this alternative implementation of the `register()` method: ```python def register(fn): name = fn.__name__ arity = len(inspect.signature(fn).parameters) if not replace and (name, arity) in self._registered_functions: return fn kwargs = {} done = False if deterministic: # Try this, but fall back if sqlite3.NotSupportedError try: self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True)) done = True except sqlite3.NotSupportedError: pass if not done: self.conn.create_function(name, arity, fn, **kwargs) self._registered_functions.add((name, arity)) return fn ``` With that fix, the following worked! ``` LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1180427792 | |
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220 | https://api.github.com/repos/simonw/sqlite-utils/issues/421 | 1098532220 | IC_kwDOCGYnMM5BekV8 | 9599 | 2022-04-13T22:09:52Z | 2022-04-13T22:09:52Z | OWNER | That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1180427792 | |
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354 | https://api.github.com/repos/simonw/sqlite-utils/issues/421 | 1098531354 | IC_kwDOCGYnMM5BekIa | 9599 | 2022-04-13T22:08:20Z | 2022-04-13T22:08:20Z | OWNER | OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running: ``` (app-root) sqlite-utils indexes global.db --table select sqlite_master.name as "table", indexes.name as index_name, xinfo.* from sqlite_master join pragma_index_list(sqlite_master.name) indexes join pragma_index_xinfo(index_name) xinfo where sqlite_master.type = 'table' and xinfo.key = 1 Error: near "(": syntax error ``` This made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version: ``` (app-root) sqlite3 SQLite version 3.36.0 2021-06-18 18:36:39 ``` That version should be fine - it's the one you compiled in the Dockerfile. Then I checked the version that `sqlite-utils` itself was using: ``` (app-root) sqlite-utils memory 'select sqlite_version()' [{"sqlite_version()": "3.7.17"}] ``` It's running SQLite 3.7.17! So the problem here is that the Python in that Docker image is running a very old version of SQLite. I tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked: ``` (app-root) python3 -c 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())' ('3.7.17',) (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())' ('3.36.0',) ``` But when I try to run `sqlite-utils` like that I get an error: ``` (app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db ... File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py", line 1624, in query db.register_fts4_bm25() File "/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py", line 412, in register_fts… | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1180427792 | |
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517 | https://api.github.com/repos/simonw/sqlite-utils/issues/421 | 1098295517 | IC_kwDOCGYnMM5Bdqjd | 9599 | 2022-04-13T17:16:20Z | 2022-04-13T17:16:20Z | OWNER | Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that. ``` (app-root) sqlite-utils indexes global.db --table Error: near "(": syntax error ``` (That wa sbefore I even ran the `extract` command.) To build your `Dockerfile` I copied it into an empty folder and ran the following: ``` wget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz docker build . -t centos-sqlite-utils docker run -it centos-sqlite-utils /bin/bash ``` This gave me a shell in which I could replicate the bug. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1180427792 | |
https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158 | https://api.github.com/repos/simonw/sqlite-utils/issues/421 | 1098288158 | IC_kwDOCGYnMM5Bdowe | 9599 | 2022-04-13T17:07:53Z | 2022-04-13T17:07:53Z | OWNER | I can't replicate the bug I'm afraid: ``` % wget "https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true" ... 2022-04-13 10:06:29 (8.97 MB/s) - ‘global_power_plant_database.csv?raw=true’ saved [8856038/8856038] % sqlite-utils insert global.db power_plants \ 'global_power_plant_database.csv?raw=true' --csv [------------------------------------] 0% [###################################-] 99% 00:00:00% % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key ------- ------------ ------- ----- ------ ------ ------ ----- % sqlite-utils extract global.db power_plants country country_long \ --table countries \ --fk-column country_id \ --rename country_long name % sqlite-utils indexes global.db --table table index_name seqno cid name desc coll key --------- -------------------------- ------- ----- ------- ------ ------ ----- countries idx_countries_country_name 0 1 country 0 BINARY 1 countries idx_countries_country_name 1 2 name 0 BINARY 1 ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1180427792 |