{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1079407962", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1079407962, "node_id": "IC_kwDOCGYnMM5AVnVa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-03-25T20:25:10Z", "updated_at": "2022-03-25T20:25:18Z", "author_association": "OWNER", "body": "Can you share either your whole `global.db` table or a shrunk down example that illustrates the bug?\r\n\r\nMy hunch is that you may have a table or column with a name that triggers the error.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1081079506", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1081079506, "node_id": "IC_kwDOCGYnMM5Ab_bS", "user": {"value": 24938923, "label": "learning4life"}, "created_at": "2022-03-28T19:58:55Z", "updated_at": "2022-03-28T20:05:57Z", "author_association": "NONE", "body": "Sure, it is from the documentation example:\r\n[Extracting columns into a separate table](https://sqlite-utils.datasette.io/en/stable/cli.html#extracting-columns-into-a-separate-table)\r\n\r\n\r\n```\r\nwget \"https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true\"\r\n\r\nsqlite-utils insert global.db power_plants \\\r\n    'global_power_plant_database.csv?raw=true' --csv\r\n# Extract those columns:\r\nsqlite-utils extract global.db power_plants country country_long \\\r\n    --table countries \\\r\n    --fk-column country_id \\\r\n    --rename country_long name\r\n```\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098288158", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1098288158, "node_id": "IC_kwDOCGYnMM5Bdowe", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-13T17:07:53Z", "updated_at": "2022-04-13T17:07:53Z", "author_association": "OWNER", "body": "I can't replicate the bug I'm afraid:\r\n```\r\n% wget \"https://github.com/wri/global-power-plant-database/blob/232a6666/output_database/global_power_plant_database.csv?raw=true\"               \r\n...\r\n2022-04-13 10:06:29 (8.97 MB/s) - \u2018global_power_plant_database.csv?raw=true\u2019 saved [8856038/8856038]\r\n% sqlite-utils insert global.db power_plants \\                      \r\n    'global_power_plant_database.csv?raw=true' --csv\r\n  [------------------------------------]    0%\r\n  [###################################-]   99%  00:00:00%\r\n% sqlite-utils indexes global.db --table                            \r\ntable    index_name    seqno    cid    name    desc    coll    key\r\n-------  ------------  -------  -----  ------  ------  ------  -----\r\n% sqlite-utils extract global.db power_plants country country_long \\\r\n    --table countries \\\r\n    --fk-column country_id \\\r\n    --rename country_long name\r\n% sqlite-utils indexes global.db --table                            \r\ntable      index_name                    seqno    cid  name       desc  coll      key\r\n---------  --------------------------  -------  -----  -------  ------  ------  -----\r\ncountries  idx_countries_country_name        0      1  country       0  BINARY      1\r\ncountries  idx_countries_country_name        1      2  name          0  BINARY      1\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098295517", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1098295517, "node_id": "IC_kwDOCGYnMM5Bdqjd", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-13T17:16:20Z", "updated_at": "2022-04-13T17:16:20Z", "author_association": "OWNER", "body": "Aha! I was able to replicate the bug using your `Dockerfile` - thanks very much for providing that.\r\n```\r\n(app-root) sqlite-utils indexes global.db --table\r\nError: near \"(\": syntax error\r\n```\r\n(That wa sbefore I even ran the `extract` command.)\r\n\r\nTo build your `Dockerfile` I copied it into an empty folder and ran the following:\r\n```\r\nwget https://www.sqlite.org/2021/sqlite-autoconf-3360000.tar.gz\r\ndocker build . -t centos-sqlite-utils\r\ndocker run -it centos-sqlite-utils /bin/bash\r\n```\r\nThis gave me a shell in which I could replicate the bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098531354", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1098531354, "node_id": "IC_kwDOCGYnMM5BekIa", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-13T22:08:20Z", "updated_at": "2022-04-13T22:08:20Z", "author_association": "OWNER", "body": "OK I figured out what's going on here. First I added an extra `print(sql)` statement to the `indexes` command to see what SQL it was running:\r\n```\r\n(app-root) sqlite-utils indexes global.db --table\r\n\r\n    select\r\n      sqlite_master.name as \"table\",\r\n      indexes.name as index_name,\r\n      xinfo.*\r\n    from sqlite_master\r\n      join pragma_index_list(sqlite_master.name) indexes\r\n      join pragma_index_xinfo(index_name) xinfo\r\n    where\r\n      sqlite_master.type = 'table'\r\n     and xinfo.key = 1\r\nError: near \"(\": syntax error\r\n```\r\nThis made me suspicious that the SQLite version being used here didn't support joining against the `pragma_index_list(...)` table-valued functions in that way. So I checked the version:\r\n```\r\n(app-root) sqlite3\r\nSQLite version 3.36.0 2021-06-18 18:36:39\r\n```\r\nThat version should be fine - it's the one you compiled in the Dockerfile.\r\n\r\nThen I checked the version that `sqlite-utils` itself was using:\r\n```\r\n(app-root) sqlite-utils memory 'select sqlite_version()'\r\n[{\"sqlite_version()\": \"3.7.17\"}]\r\n```\r\nIt's running SQLite 3.7.17!\r\n\r\nSo the problem here is that the Python in that Docker image is running a very old version of SQLite.\r\n\r\nI tried using the trick in https://til.simonwillison.net/sqlite/ld-preload as a workaround, and it almost worked:\r\n\r\n```\r\n(app-root) python3 -c 'import sqlite3; print(sqlite3.connect(\":memory\").execute(\"select sqlite_version()\").fetchone())'\r\n('3.7.17',)\r\n(app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so python3 -c 'import sqlite3; print(sqlite3.connect(\":memory\").execute(\"select sqlite_version()\").fetchone())'\r\n('3.36.0',)\r\n```\r\nBut when I try to run `sqlite-utils` like that I get an error:\r\n\r\n```\r\n(app-root) LD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db \r\n...\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/cli.py\", line 1624, in query\r\n    db.register_fts4_bm25()\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 412, in register_fts4_bm25\r\n    self.register_function(rank_bm25, deterministic=True)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 408, in register_function\r\n    register(fn)\r\n  File \"/opt/app-root/lib64/python3.8/site-packages/sqlite_utils/db.py\", line 401, in register\r\n    self.conn.create_function(name, arity, fn, **kwargs)\r\nsqlite3.NotSupportedError: deterministic=True requires SQLite 3.8.3 or higher\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098532220", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1098532220, "node_id": "IC_kwDOCGYnMM5BekV8", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-13T22:09:52Z", "updated_at": "2022-04-13T22:09:52Z", "author_association": "OWNER", "body": "That error is weird - it's not supposed to happen according to this code here: https://github.com/simonw/sqlite-utils/blob/95522ad919f96eb6cc8cd3cd30389b534680c717/sqlite_utils/db.py#L389-L400", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098535531", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1098535531, "node_id": "IC_kwDOCGYnMM5BelJr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-13T22:15:48Z", "updated_at": "2022-04-13T22:15:48Z", "author_association": "OWNER", "body": "Trying this alternative implementation of the `register()` method:\r\n\r\n```python\r\n        def register(fn):\r\n            name = fn.__name__\r\n            arity = len(inspect.signature(fn).parameters)\r\n            if not replace and (name, arity) in self._registered_functions:\r\n                return fn\r\n            kwargs = {}\r\n            done = False\r\n            if deterministic:\r\n                # Try this, but fall back if sqlite3.NotSupportedError\r\n                try:\r\n                    self.conn.create_function(name, arity, fn, **dict(kwargs, deterministic=True))\r\n                    done = True\r\n                except sqlite3.NotSupportedError:\r\n                    pass\r\n            if not done:\r\n                self.conn.create_function(name, arity, fn, **kwargs)\r\n            self._registered_functions.add((name, arity))\r\n            return fn\r\n```\r\nWith that fix, the following worked!\r\n```\r\nLD_PRELOAD=./build/sqlite-autoconf-3360000/.libs/libsqlite3.so sqlite-utils indexes /tmp/global.db --table\r\ntable      index_name                    seqno    cid  name       desc  coll      key\r\n---------  --------------------------  -------  -----  -------  ------  ------  -----\r\ncountries  idx_countries_country_name        0      1  country       0  BINARY      1\r\ncountries  idx_countries_country_name        1      2  name          0  BINARY      1\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/421#issuecomment-1098548931", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/421", "id": 1098548931, "node_id": "IC_kwDOCGYnMM5BeobD", "user": {"value": 9599, "label": "simonw"}, "created_at": "2022-04-13T22:41:59Z", "updated_at": "2022-04-13T22:41:59Z", "author_association": "OWNER", "body": "I'm going to close this ticket since it looks like this is a bug in the way the Dockerfile builds Python, but I'm going to ship a fix for that issue I found so the `LD_PRELOAD` workaround above should work OK with the next release of `sqlite-utils`. Thanks for the detailed bug report!", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1180427792, "label": "\"Error: near \"(\": syntax error\" when using sqlite-utils indexes CLI"}, "performed_via_github_app": null}