html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606415188,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606415188,IC_kwDOCGYnMM5fv_NU,9599,simonw,2023-06-26T01:46:47Z,2023-06-26T01:47:01Z,OWNER,"I just tested this in a brand new virtual environment using the macOS Python 3: ```bash pipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3 ``` Then in that virtual environment I ran: ```bash pip install sqlite-utils # Confirm the right one is on the path: which sqlite-utils curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \ sqlite-utils insert meteorites.db meteorites - --pk=id sqlite-utils extract meteorites.db meteorites recclass ``` This threw the same error reported above. Then I did this: ```bash rm meteorites.db pip install sqlean.py curl ""https://data.nasa.gov/resource/y77d-th95.json"" | \ sqlite-utils insert meteorites.db meteorites - --pk=id sqlite-utils extract meteorites.db meteorites recclass ``` And that second time it worked correctly.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606411508,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1606411508,IC_kwDOCGYnMM5fv-T0,9599,simonw,2023-06-26T01:42:10Z,2023-06-26T01:42:22Z,OWNER,https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.,"{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1604379952,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1604379952,IC_kwDOCGYnMM5foOUw,9599,simonw,2023-06-23T14:39:55Z,2023-06-23T15:39:32Z,OWNER,"Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment: sqlite-utils install pysqlite3-binary But `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people. The ""easiest"" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504288134,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504288134,IC_kwDOCGYnMM5ZqZ2G,9599,simonw,2023-04-11T23:55:06Z,2023-04-12T03:34:32Z,OWNER,"Also checked the official Datasette Docker image - I had to run that in Codespaces because it doesn't currently work on my M2 Mac: ``` codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker pull datasetteproject/datasette Using default tag: latest ... codespace@codespaces-112c61:/workspaces/sqlite-utils$ docker run -it datasetteproject/datasette / bin/bash root@75ba34f501ec:/# python Python 3.11.0 (main, Dec 6 2022, 13:31:55) [GCC 10.2.1 20210110] on linux Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import sqlite3 .executescript("""""" PRAGMA writable_schema = 1; UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)'; PRAGMA writable_schema = 0; """""")>>> db = sqlite3.connect("":memory:"") >>> db.executescript("""""" ... PRAGMA writable_schema = 1; ... UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)'; ... PRAGMA writable_schema = 0; ... """""") >>> ``` So that confirms that the official image also has a Python with a SQLite that's not in defensive mode.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504245029,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1504245029,IC_kwDOCGYnMM5ZqPUl,9599,simonw,2023-04-11T23:13:41Z,2023-04-11T23:14:39Z,OWNER,"I also tested this against the current `ubuntu:latest` Docker image (on an M2 Mac), in Python 3.10 and 3.11: ``` docker run -it ubuntu:latest /bin/bash ``` Then in the container: ``` apt-get update apt-get install python3 python3 # pasted in the above recipe apt install software-properties-common add-apt-repository ppa:deadsnakes/ppa apt install python3.11 python3.11 # pasted it in again ``` In both cases the Python code did not raise an exception, which suggests that on Ubuntu those two Python versions do not have the defensive mode set.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502559442,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502559442,IC_kwDOCGYnMM5ZjzzS,9599,simonw,2023-04-11T01:32:30Z,2023-04-11T01:33:27Z,OWNER,"This seems to work: ```python import sqlite3 db = sqlite3.connect("":memory:"") db.executescript("""""" PRAGMA writable_schema = 1; UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)'; PRAGMA writable_schema = 0; """""") ``` It succeeds on my Python 3.11 and raises the following exception on my broken Python 3.9: ``` sqlite3.OperationalError: table sqlite_master may not be modified ``` Removing the `PRAGMA writable_schema = 1;` causes the same exception to be raised on both Pythons.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502557629,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502557629,IC_kwDOCGYnMM5ZjzW9,9599,simonw,2023-04-11T01:30:12Z,2023-04-11T01:30:12Z,OWNER,"I'll ask on the SQLite forum if it's possible to toggle that mode on and off using regular SQL. My hunch is that it isn't. In which case `sqlite-utils` should at least know how to catch this error and display a much more readable error message, maybe with a link to further documentation. A utility function that can detect this mode would be really useful too. I'd probably have to do a test that tries to modify `sqlite_master` on a new in-memory database to catch if it's possible or not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502556111,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1502556111,IC_kwDOCGYnMM5Zjy_P,9599,simonw,2023-04-11T01:28:41Z,2023-04-11T01:28:41Z,OWNER,"Investigating this one now. The `sqlite-utils` test suite passes without errors on my Python 3.11.2 installation... but it fails with this error on a Python 3.9.6 installation. In the broken version's virtual environment directory I ran this: ``` cat pyvenv.cfg home = /Applications/Xcode.app/Contents/Developer/usr/bin implementation = CPython version_info = 3.9.6.final.0 virtualenv = 20.17.1 include-system-site-packages = false base-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9 base-exec-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9 base-executable = /Applications/Xcode.app/Contents/Developer/usr/bin/python3 ``` So it looks like the Xcode `python3` has ""defensive"" mode turned on for SQLite. As far as I can tell there's no way to turn it OFF again in Python. My virtual environment that DOES work has this: ``` home = /opt/homebrew/opt/python@3.11/bin implementation = CPython version_info = 3.11.2.final.0 virtualenv = 20.17.1 include-system-site-packages = false base-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 base-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11 base-executable = /opt/homebrew/opt/python@3.11/bin/python3.11 ``` So the Python 3.11 I installed through Homebrew doesn't exhibit this bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1495780111,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1495780111,IC_kwDOCGYnMM5ZJ8sP,19786848,Thomascountz,2023-04-04T11:09:56Z,2023-04-04T11:13:40Z,NONE,"@wpears' workaround also worked for me, but also required me to manually set `PRAGMA writable_schema`. ```sql PRAGMA writable_schema = 1; UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (...)' PRAGMA writable_schema = 0; ``` ```shell $ python --version Python 3.11.2 $ sqlite3 --version 3.41.2 2023-03-22 11:56:21 0d1fc92f94cb6b76bffe3ec34d69cffde2924203304e8ffc4155597af0c191da $ sqlite-utils --version sqlite-utils, version 3.30 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1492777509,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1492777509,IC_kwDOCGYnMM5Y-fol,9020979,hydrosquall,2023-04-01T01:31:48Z,2023-04-01T01:31:48Z,NONE,"My current workaround is to use this library from a python script instead of as a CLI tool. This lets me set the foreign key constraint at table creation time, instead of trying to modify an existing table. [docs](https://sqlite-utils.datasette.io/en/stable/python-api.html#specifying-foreign-keys) I found this [stackoverflow helpful](https://stackoverflow.com/a/1884841/5129731), as it explained that Sqlite doesn't support modifying existing tables directly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354192168,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354192168,IC_kwDOCGYnMM5Qt1Uo,9020979,hydrosquall,2022-12-16T04:35:30Z,2022-12-16T04:35:38Z,NONE,"A related historical problem: https://github.com/tekartik/sqflite/issues/525#issuecomment-714500720 I wonder if the version of Sqlite or Python for Intel chip have defensive mode disabled by default, whereas M1 chips versions have it enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1354160286,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1354160286,IC_kwDOCGYnMM5Qttie,9020979,hydrosquall,2022-12-16T03:51:19Z,2022-12-16T03:52:13Z,NONE,"Hi @ryascott, thanks for sharing this! How did you upgrade your sqlite3 version? I'm running into this issue (also on an m1) with `Python ==3.10.7 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30` Unfortunately, `3.10.8` isn't listed in `pyenv` so I'm unable to install it. For me, the trigger is trying to use the `add-foreign-key` command on its own: ```bash sqlite-utils add-foreign-key library.db book_creators creator_id creators id ``` Some stackoverflow searching suggests that brew installing sqlite may fix it ( https://stackoverflow.com/questions/26345972/how-do-i-upgrade-the-sqlite-version-used-by-pythons-sqlite3-module-on-mac ), but I don't want to risk breaking the version of sqlite used by some other system, I'd only like to upgrade sqlite3 inside my current virtual environment. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1304539296,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1304539296,IC_kwDOCGYnMM5NwbCg,559711,ryascott,2022-11-05T12:40:12Z,2022-11-05T12:40:12Z,NONE,"I had the problem this morning when running: `Python==3.9.6 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30 ` I upgraded to: `Python ==3.10.8 sqlite3.sqlite_version==3.37.2 sqlite-utils==3.30 ` and the error did not appear anymore. Hope this helps Ryan ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1223378004,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1223378004,IC_kwDOCGYnMM5I60RU,1558033,wpears,2022-08-23T00:44:11Z,2022-08-23T00:44:11Z,NONE,"This bug affects me as well. Env: ``` Python 3.8.12 sqlite-utils, version 3.28 sqlite3 3.32.3 MacOS Big Sur 11.6.7 Intel ``` Similar to @mdrovdahl, I was able to work around this bug by piping the SQL string constructed in `add_foreign_keys` to the `sqlite3` command itself. Specifically, if you're trying to patch this yourself, replace [lines 1026-1039 of db.py in your site packages](https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1026-L1039) with something similar to the following: ``` print(""PRAGMA writable_schema = 1;"") for table_name, new_sql in table_sql.items(): print(""UPDATE sqlite_master SET sql = '{}' WHERE name = '{}';"".format( new_sql, table_name) ) print(""PRAGMA writable_schema = 0;"") print(""VACUUM;"") ``` Then from your terminal: `db-to-sqlite """" your.db --all > output.sql && sqlite3 your.db < output.sql` If you want to run this with `-p`, you'll have to actually open a file in code to write to instead of redirecting the output.","{""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1206241356,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1206241356,IC_kwDOCGYnMM5H5chM,503614,lfdebrux,2022-08-05T09:26:15Z,2022-08-05T09:29:42Z,NONE,"I am getting the same error when using github-to-sqlite (which uses sqlite-utils internally). I am also using an M1 MacBook Pro, with macOS Monterey 12.5, with Python 3.10.6 for arm64 installed using pyenv. I have sqlite-utils 3.28 installed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1198414383,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1198414383,IC_kwDOCGYnMM5Hblov,474467,mdrovdahl,2022-07-28T17:10:06Z,2022-07-28T17:10:06Z,NONE,"I was able to fight through this by capturing the SQL commands from the `add_foreign_keys()` function in sqlite-utils and then executing them manually via the sqlite3 client, first setting PRAGMA writable_schema on and then updating the sqlite_master table. Still no clue why they were failing when run in context...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1172766270,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1172766270,IC_kwDOCGYnMM5F5v4-,474467,mdrovdahl,2022-07-01T22:40:26Z,2022-07-01T22:40:26Z,NONE,"Note, I do not get this issue using my Intel MacBook Pro =/ Environment ``` markd@Marks-MBP metabase % python3 --version Python 3.9.13 markd@Marks-MBP metabase % sqlite3 --version 3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl markd@Marks-MBP metabase % sqlite-utils --version sqlite-utils, version 3.27 markd@Marks-MBP metabase % sqlite3 github.db SQLite version 3.37.0 2021-12-09 01:34:53 Enter "".help"" for usage hints. sqlite> .dbconfig defensive off dqs_ddl on dqs_dml on enable_fkey off enable_qpsg off enable_trigger on enable_view on fts3_tokenizer off legacy_alter_table on legacy_file_format off load_extension off no_ckpt_on_close off reset_database off trigger_eqp off trusted_schema on writable_schema off ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified, https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1172697090,https://api.github.com/repos/simonw/sqlite-utils/issues/235,1172697090,IC_kwDOCGYnMM5F5fAC,474467,mdrovdahl,2022-07-01T20:37:40Z,2022-07-01T20:37:55Z,NONE,"I just ran into what appears to be the same issue on a MacBook Pro, M1 Pro. Environment: ``` markd@Marks-MacBook-Pro metabase % python --version Python 3.8.9 markd@Marks-MacBook-Pro metabase % sqlite3 --version 3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl markd@Marks-MacBook-Pro metabase % sqlite-utils --version sqlite-utils, version 3.27 markd@Marks-MacBook-Pro metabase % sqlite3 gh.db SQLite version 3.37.0 2021-12-09 01:34:53 Enter "".help"" for usage hints. sqlite> .dbconfig defensive off dqs_ddl on dqs_dml on enable_fkey off enable_qpsg off enable_trigger on enable_view on fts3_tokenizer off legacy_alter_table on legacy_file_format off load_extension off no_ckpt_on_close off reset_database off trigger_eqp off trusted_schema on writable_schema off ``` Error ``` markd@Marks-MacBook-Pro metabase % github-to-sqlite repos gh.db a8cteam51 Traceback (most recent call last): File ""/Users/markd/Library/Python/3.8/bin/github-to-sqlite"", line 8, in sys.exit(cli()) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1130, in __call__ return self.main(*args, **kwargs) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1055, in main rv = self.invoke(ctx) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/click/core.py"", line 760, in invoke return __callback(*args, **kwargs) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/cli.py"", line 268, in repos utils.ensure_db_shape(db) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py"", line 688, in ensure_db_shape ensure_foreign_keys(db) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/github_to_sqlite/utils.py"", line 682, in ensure_foreign_keys db[table].add_foreign_key(column, table2, column2) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py"", line 2004, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File ""/Users/markd/Library/Python/3.8/lib/python/site-packages/sqlite_utils/db.py"", line 1019, in add_foreign_keys cursor.execute( sqlite3.OperationalError: table sqlite_master may not be modified ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",810618495,Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified,