{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606415188", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1606415188, "node_id": "IC_kwDOCGYnMM5fv_NU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-26T01:46:47Z", "updated_at": "2023-06-26T01:47:01Z", "author_association": "OWNER", "body": "I just tested this in a brand new virtual environment using the macOS Python 3:\r\n```bash\r\npipenv shell --python /Applications/Xcode.app/Contents/Developer/usr/bin/python3\r\n```\r\nThen in that virtual environment I ran:\r\n```bash\r\npip install sqlite-utils\r\n# Confirm the right one is on the path:\r\nwhich sqlite-utils\r\n\r\ncurl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n    sqlite-utils insert meteorites.db meteorites - --pk=id\r\nsqlite-utils extract meteorites.db meteorites  recclass\r\n```\r\nThis threw the same error reported above. Then I did this:\r\n```bash\r\nrm meteorites.db\r\npip install sqlean.py\r\n\r\ncurl \"https://data.nasa.gov/resource/y77d-th95.json\" | \\\r\n    sqlite-utils insert meteorites.db meteorites - --pk=id\r\nsqlite-utils extract meteorites.db meteorites  recclass\r\n```\r\nAnd that second time it worked correctly.", "reactions": "{\"total_count\": 1, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 1, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1606411508", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1606411508, "node_id": "IC_kwDOCGYnMM5fv-T0", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-26T01:42:10Z", "updated_at": "2023-06-26T01:42:22Z", "author_association": "OWNER", "body": "https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-33 - upgrading to `sqlite-utils>=3.33` and then installing both `sqlean.py` and `sqlite-dump` in the same virtual environment as `sqlite-utils` should fix this issue.", "reactions": "{\"total_count\": 2, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 2, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1604379952", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1604379952, "node_id": "IC_kwDOCGYnMM5foOUw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-06-23T14:39:55Z", "updated_at": "2023-06-23T15:39:32Z", "author_association": "OWNER", "body": "Ideally a workaround for this right now would be to install `pysqlite3` in the same virtual environment:\r\n\r\n    sqlite-utils install pysqlite3-binary\r\n\r\nBut `pysqlite3-binary` doesn't yet ship a wheel for macOS so this probably won't work for most people.\r\n\r\nThe \"easiest\" fix at the moment is to use Python from Homebrew - so `brew install sqlite-utils` for example won't suffer from this problem. Not a great solution for people who aren't using Homebrew though!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504288134", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1504288134, "node_id": "IC_kwDOCGYnMM5ZqZ2G", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T23:55:06Z", "updated_at": "2023-04-12T03:34:32Z", "author_association": "OWNER", "body": "Also checked the official Datasette Docker image - I had to run that in Codespaces because it doesn't currently work on my M2 Mac:\r\n\r\n```\r\ncodespace@codespaces-112c61:/workspaces/sqlite-utils$ docker pull datasetteproject/datasette\r\nUsing default tag: latest\r\n...\r\ncodespace@codespaces-112c61:/workspaces/sqlite-utils$ docker run -it datasetteproject/datasette /\r\nbin/bash\r\nroot@75ba34f501ec:/# python\r\nPython 3.11.0 (main, Dec  6 2022, 13:31:55) [GCC 10.2.1 20210110] on linux\r\nType \"help\", \"copyright\", \"credits\" or \"license\" for more information.\r\n>>> import sqlite3\r\n.executescript(\"\"\"\r\nPRAGMA writable_schema = 1;\r\nUPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';\r\nPRAGMA writable_schema = 0;\r\n\"\"\")>>> db = sqlite3.connect(\":memory:\")\r\n>>> db.executescript(\"\"\"\r\n... PRAGMA writable_schema = 1;\r\n... UPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';\r\n... PRAGMA writable_schema = 0;\r\n... \"\"\")\r\n<sqlite3.Cursor object at 0x7fd9b0561140>\r\n>>>\r\n```\r\nSo that confirms that the official image also has a Python with a SQLite that's not in defensive mode.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1504245029", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1504245029, "node_id": "IC_kwDOCGYnMM5ZqPUl", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T23:13:41Z", "updated_at": "2023-04-11T23:14:39Z", "author_association": "OWNER", "body": "I also tested this against the current `ubuntu:latest` Docker image (on an M2 Mac), in Python 3.10 and 3.11:\r\n```\r\ndocker run -it ubuntu:latest /bin/bash\r\n```\r\nThen in the container:\r\n```\r\napt-get update\r\napt-get install python3\r\npython3\r\n# pasted in the above recipe\r\napt install software-properties-common\r\nadd-apt-repository ppa:deadsnakes/ppa\r\napt install python3.11\r\npython3.11 \r\n# pasted it in again\r\n```\r\nIn both cases the Python code did not raise an exception, which suggests that on Ubuntu those two Python versions do not have the defensive mode set.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502559442", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1502559442, "node_id": "IC_kwDOCGYnMM5ZjzzS", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T01:32:30Z", "updated_at": "2023-04-11T01:33:27Z", "author_association": "OWNER", "body": "This seems to work:\r\n```python\r\nimport sqlite3\r\ndb = sqlite3.connect(\":memory:\")\r\ndb.executescript(\"\"\"\r\nPRAGMA writable_schema = 1;\r\nUPDATE sqlite_master SET sql = 'CREATE TABLE [foos] (id integer primary key)';\r\nPRAGMA writable_schema = 0;\r\n\"\"\")\r\n```\r\nIt succeeds on my Python 3.11 and raises the following exception on my broken Python 3.9:\r\n```\r\nsqlite3.OperationalError: table sqlite_master may not be modified\r\n```\r\nRemoving the `PRAGMA writable_schema = 1;` causes the same exception to be raised on both Pythons.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502557629", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1502557629, "node_id": "IC_kwDOCGYnMM5ZjzW9", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T01:30:12Z", "updated_at": "2023-04-11T01:30:12Z", "author_association": "OWNER", "body": "I'll ask on the SQLite forum if it's possible to toggle that mode on and off using regular SQL. My hunch is that it isn't.\r\n\r\nIn which case `sqlite-utils` should at least know how to catch this error and display a much more readable error message, maybe with a link to further documentation.\r\n\r\nA utility function that can detect this mode would be really useful too. I'd probably have to do a test that tries to modify `sqlite_master` on a new in-memory database to catch if it's possible or not.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/235#issuecomment-1502556111", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/235", "id": 1502556111, "node_id": "IC_kwDOCGYnMM5Zjy_P", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-04-11T01:28:41Z", "updated_at": "2023-04-11T01:28:41Z", "author_association": "OWNER", "body": "Investigating this one now.\r\n\r\nThe `sqlite-utils` test suite passes without errors on my Python 3.11.2 installation... but it fails with this error on a Python 3.9.6 installation.\r\n\r\nIn the broken version's virtual environment directory I ran this:\r\n```\r\ncat pyvenv.cfg \r\nhome = /Applications/Xcode.app/Contents/Developer/usr/bin\r\nimplementation = CPython\r\nversion_info = 3.9.6.final.0\r\nvirtualenv = 20.17.1\r\ninclude-system-site-packages = false\r\nbase-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9\r\nbase-exec-prefix = /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9\r\nbase-executable = /Applications/Xcode.app/Contents/Developer/usr/bin/python3\r\n```\r\nSo it looks like the Xcode `python3` has \"defensive\" mode turned on for SQLite.\r\n\r\nAs far as I can tell there's no way to turn it OFF again in Python.\r\n\r\nMy virtual environment that DOES work has this:\r\n\r\n```\r\nhome = /opt/homebrew/opt/python@3.11/bin\r\nimplementation = CPython\r\nversion_info = 3.11.2.final.0\r\nvirtualenv = 20.17.1\r\ninclude-system-site-packages = false\r\nbase-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11\r\nbase-exec-prefix = /opt/homebrew/opt/python@3.11/Frameworks/Python.framework/Versions/3.11\r\nbase-executable = /opt/homebrew/opt/python@3.11/bin/python3.11\r\n```\r\nSo the Python 3.11 I installed through Homebrew doesn't exhibit this bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 810618495, "label": "Extract columns cannot create foreign key relation: sqlite3.OperationalError: table sqlite_master may not be modified"}, "performed_via_github_app": null}