html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1843586503,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1843586503,IC_kwDOCGYnMM5t4uXH,9599,2023-12-06T19:49:10Z,2023-12-06T19:49:29Z,OWNER,"This looks really great on first glance - design is good, implementation is solid, tests and documentation look great. Looks like a couple of `mypy` failures in the tests at the moment: ``` mypy sqlite_utils tests sqlite_utils/db.py:543: error: Incompatible types in assignment (expression has type ""type[Table]"", variable has type ""type[View]"") [assignment] tests/test_lookup.py:156: error: Name ""test_lookup_new_table"" already defined on line 5 [no-redef] Found 2 errors in 2 files (checked 54 source files) Error: Process completed with exit code 1. ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157, https://github.com/simonw/sqlite-utils/pull/604#issuecomment-1846560096,https://api.github.com/repos/simonw/sqlite-utils/issues/604,1846560096,IC_kwDOCGYnMM5uEEVg,9599,2023-12-08T05:16:44Z,2023-12-08T05:17:20Z,OWNER,"Also tested this manually like so: ```bash sqlite-utils create-table strict.db strictint id integer size integer --strict sqlite-utils create-table strict.db notstrictint id integer size integer sqlite-utils install sqlite-utils-shell sqlite-utils shell strict.db ``` ``` Attached to strict.db Type 'exit' to exit. sqlite-utils> insert into strictint (size) values (4); 1 row affected sqlite-utils> insert into strictint (size) values ('four'); An error occurred: cannot store TEXT value in INTEGER column strictint.size sqlite-utils> insert into notstrictint (size) values ('four'); 1 row affected sqlite-utils> commit; Done ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",2001006157, https://github.com/simonw/sqlite-utils/issues/603#issuecomment-1846555822,https://api.github.com/repos/simonw/sqlite-utils/issues/603,1846555822,IC_kwDOCGYnMM5uEDSu,9599,2023-12-08T05:09:55Z,2023-12-08T05:10:31Z,OWNER,"I'm unable to replicate this issue. This is with a fresh install of `sqlite-utils==3.35.2`: ``` (base) ~ python3.12 Python 3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin Type ""help"", ""copyright"", ""credits"" or ""license"" for more information. >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db[""foo""].insert({""bar"": 1}) >>> import sys >>> sys.version '3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)]' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1988525411, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793263638,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793263638,IC_kwDOCGYnMM5q4wgW,9599,2023-11-04T00:19:58Z,2023-11-04T00:19:58Z,OWNER,Thanks for this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793265952,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793265952,IC_kwDOCGYnMM5q4xEg,9599,2023-11-04T00:25:34Z,2023-11-04T00:25:34Z,OWNER,The tests failed because they found a spelling mistake in a completely unrelated area of the code - not sure why that had not been caught before.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793268126,IC_kwDOCGYnMM5q4xme,9599,2023-11-04T00:31:34Z,2023-11-04T00:31:34Z,OWNER,"Testing this manually on macOS using Docker Desk top like this: ```bash docker run -it --rm arm64v8/ubuntu /bin/bash ``` Then inside the container: ```bash uname -m ``` Outputs: `aarch64` Then: ```bash apt install spatialite-bin libsqlite3-mod-spatialite git python3 python3-venv -y cd /tmp git clone https://github.com/simonw/sqlite-utils cd sqlite-utils python3 -m venv venv source venv/bin/activate pip install -e '.[test]' sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Which output: ``` Traceback (most recent call last): File ""/tmp/sqlite-utils/venv/bin/sqlite-utils"", line 33, in sys.exit(load_entry_point('sqlite-utils', 'console_scripts', 'sqlite-utils')()) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1157, in __call__ return self.main(*args, **kwargs) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1078, in main rv = self.invoke(ctx) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/tmp/sqlite-utils/venv/lib/python3.10/site-packages/click/core.py"", line 783, in invoke return __callback(*args, **kwargs) File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 1959, in memory _load_extensions(db, load_extension) File ""/tmp/sqlite-utils/sqlite_utils/cli.py"", line 3232, in _load_extensions if "":"" in ext: TypeError: argument of type 'NoneType' is not iterable ``` Then I ran this: ```bash git checkout -b MikeCoats-spatialite-paths-linux-arm main git pull https://github.com/MikeCoats/sqlite-utils.git spatialite-paths-linux-arm ``` And now: ```bash sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Outputs: ```json [{""spatialite_version()"": ""5.0.1""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379, https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793269219,https://api.github.com/repos/simonw/sqlite-utils/issues/600,1793269219,IC_kwDOCGYnMM5q4x3j,9599,2023-11-04T00:34:33Z,2023-11-04T00:34:33Z,OWNER,"The GIS tests now pass in that container too: ```bash pytest tests/test_gis.py ``` ``` ======================== test session starts ========================= platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0 rootdir: /tmp/sqlite-utils plugins: hypothesis-6.88.1 collected 12 items tests/test_gis.py ............ [100%] ========================= 12 passed in 0.48s ========================= ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1977004379, https://github.com/simonw/sqlite-utils/issues/6#issuecomment-457978729,https://api.github.com/repos/simonw/sqlite-utils/issues/6,457978729,MDEyOklzc3VlQ29tbWVudDQ1Nzk3ODcyOQ==,9599,2019-01-28T02:12:19Z,2019-01-28T02:12:19Z,OWNER,Will need to solve #7 for this to become truly efficient.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",403624090, https://github.com/simonw/sqlite-utils/issues/599#issuecomment-1793268750,https://api.github.com/repos/simonw/sqlite-utils/issues/599,1793268750,IC_kwDOCGYnMM5q4xwO,9599,2023-11-04T00:33:25Z,2023-11-04T00:33:25Z,OWNER,"See details of how I tested this here: - https://github.com/simonw/sqlite-utils/pull/600#issuecomment-1793268126 Short version: having applied this fix, the following command (on simulated `aarch64`): ```bash sqlite-utils memory ""select spatialite_version()"" --load-extension=spatialite ``` Outputs: ```json [{""spatialite_version()"": ""5.0.1""}] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1976986318, https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793272429,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793272429,IC_kwDOCGYnMM5q4ypt,9599,2023-11-04T00:40:34Z,2023-11-04T00:40:34Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132, https://github.com/simonw/sqlite-utils/pull/598#issuecomment-1793274485,https://api.github.com/repos/simonw/sqlite-utils/issues/598,1793274485,IC_kwDOCGYnMM5q4zJ1,9599,2023-11-04T00:46:55Z,2023-11-04T00:46:55Z,OWNER,"Manually tested. Before: ![cursor-bug](https://github.com/simonw/sqlite-utils/assets/9599/3bdd30ea-1a54-4fec-b42d-793130a17bc1) After: ![cursor-fix](https://github.com/simonw/sqlite-utils/assets/9599/015d4e4e-b40c-4a93-81f5-1a4adef69b11) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1926729132, https://github.com/simonw/sqlite-utils/pull/596#issuecomment-1793274869,https://api.github.com/repos/simonw/sqlite-utils/issues/596,1793274869,IC_kwDOCGYnMM5q4zP1,9599,2023-11-04T00:47:55Z,2023-11-04T00:47:55Z,OWNER,Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1919296686, https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714919806,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714919806,IC_kwDOCGYnMM5mN5l-,9599,2023-09-12T03:49:41Z,2023-09-12T03:49:41Z,OWNER,"Digging in a bit more: ```pycon >>> pprint(list(db.query('PRAGMA foreign_key_list(courses)'))) [{'from': 'campus_name', 'id': 0, 'match': 'NONE', 'on_delete': 'NO ACTION', 'on_update': 'NO ACTION', 'seq': 0, 'table': 'departments', 'to': 'campus_name'}, {'from': 'dept_code', 'id': 0, 'match': 'NONE', 'on_delete': 'NO ACTION', 'on_update': 'NO ACTION', 'seq': 1, 'table': 'departments', 'to': 'dept_code'}] ``` I think the way you tell it's a compound foreign key is that both of those have the same `id` value - of `0` - but they then have two different `seq` values of `0` and `1`. Right now I ignore those columns entirely: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L1523-L1540","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971, https://github.com/simonw/sqlite-utils/issues/594#issuecomment-1714920708,https://api.github.com/repos/simonw/sqlite-utils/issues/594,1714920708,IC_kwDOCGYnMM5mN50E,9599,2023-09-12T03:51:13Z,2023-09-12T03:51:13Z,OWNER,"Changing this without breaking backwards compatibility (and forcing a 4.0 release) will be tricky, because `ForeignKey()` is a `namedtuple`: https://github.com/simonw/sqlite-utils/blob/622c3a5a7dd53a09c029e2af40c2643fe7579340/sqlite_utils/db.py#L148-L150 I could swap it out for a `dataclass` and add those extra columns, but I need to make sure that code like this still works: ```python for table, column, other_table, other_column in table.foreign_keys: # ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1891614971, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710930934,IC_kwDOCGYnMM5l-rv2,9599,2023-09-08T00:47:57Z,2023-09-08T00:47:57Z,OWNER,"That's odd, I wrote a test for this just now and it passes already: ```python def test_transform_preserves_rowids(fresh_db): # Create a rowid table fresh_db[""places""].insert_all( ( {""name"": ""Paris"", ""country"": ""France""}, {""name"": ""London"", ""country"": ""UK""}, {""name"": ""New York"", ""country"": ""USA""}, ), ) assert fresh_db[""places""].use_rowid previous_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, name from places"") ) # Transform it fresh_db[""places""].transform(column_order=(""country"", ""name"")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, name from places"") ) assert previous_rows == next_rows ``` So maybe I'm wrong about the cause of that bug?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710931605,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710931605,IC_kwDOCGYnMM5l-r6V,9599,2023-09-08T00:49:02Z,2023-09-08T00:49:02Z,OWNER,"I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710933716,IC_kwDOCGYnMM5l-sbU,9599,2023-09-08T00:52:42Z,2023-09-08T00:52:42Z,OWNER,"I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key. The reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows. But I tried this test and it passed, too: ```python def test_transform_preserves_rowids(fresh_db): fresh_db[""places""].insert_all( [ {""id"": ""1"", ""name"": ""Paris"", ""country"": ""France""}, {""id"": ""2"", ""name"": ""London"", ""country"": ""UK""}, {""id"": ""3"", ""name"": ""New York"", ""country"": ""USA""}, ], pk=""id"", ) previous_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"") ) # Transform it fresh_db[""places""].transform(column_order=(""country"", ""name"")) # Should be the same next_rows = list( tuple(row) for row in fresh_db.execute(""select rowid, id, name from places"") ) assert previous_rows == next_rows ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710934448,IC_kwDOCGYnMM5l-smw,9599,2023-09-08T00:54:03Z,2023-09-08T00:54:03Z,OWNER,"Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3. If I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1710935270,IC_kwDOCGYnMM5l-szm,9599,2023-09-08T00:55:30Z,2023-09-08T00:55:30Z,OWNER,"Yes! That recreated the bug: ``` > assert previous_rows == next_rows E AssertionError: assert equals failed E [ [ E (1, '1', 'Paris'), (1, '1', 'Paris'), E (3, '3', 'New York'), (2, '3', 'New York'), E (4, '4', 'London'), (3, '4', 'London'), E ] ... E ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493, https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580,https://api.github.com/repos/simonw/sqlite-utils/issues/592,1712895580,IC_kwDOCGYnMM5mGLZc,9599,2023-09-10T17:46:41Z,2023-09-10T17:46:41Z,OWNER,"In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`: https://www.sqlite.org/lang_vacuum.html#how_vacuum_works > The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid). So this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1886771493, https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1708695907,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1708695907,IC_kwDOCGYnMM5l2KFj,9599,2023-09-06T16:15:59Z,2023-09-06T16:19:14Z,OWNER,"The test failure was while installing `numpy`, relating to importing `distutils` - maybe relevant: - https://github.com/pypa/setuptools/issues/3661 ``` 25h Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' ERROR: Exception: Traceback (most recent call last): ... File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_internal/utils/misc.py"", line 697, in get_requires_for_build_wheel return super().get_requires_for_build_wheel(config_settings=cs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 166, in get_requires_for_build_wheel return self._call_hook('get_requires_for_build_wheel', { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_impl.py"", line 321, in _call_hook raise BackendUnavailable(data.get('traceback', '')) pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last): File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py"", line 77, in _build_backend obj = import_module(mod_path) ^^^^^^^^^^^^^^^^^^^^^^^ File ""/opt/hostedtoolcache/Python/3.12.0-rc.2/x64/lib/python3.12/importlib/__init__.py"", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File """", line 1381, in _gcd_import File """", line 1354, in _find_and_load File """", line 1304, in _find_and_load_unlocked File """", line 488, in _call_with_frames_removed File """", line 1381, in _gcd_import File """", line 1354, in _find_and_load File """", line 1325, in _find_and_load_unlocked File """", line 929, in _load_unlocked File """", line 994, in exec_module File """", line 488, in _call_with_frames_removed File ""/tmp/pip-build-env-x9nyg3kd/overlay/lib/python3.12/site-packages/setuptools/__init__.py"", line 10, in import distutils.core ModuleNotFoundError: No module named 'distutils' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789, https://github.com/simonw/sqlite-utils/pull/591#issuecomment-1793278279,https://api.github.com/repos/simonw/sqlite-utils/issues/591,1793278279,IC_kwDOCGYnMM5q40FH,9599,2023-11-04T00:58:03Z,2023-11-04T00:58:03Z,OWNER,I'm going to abandon this PR and ship the 3.12 testing change directly to `main`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1884335789, https://github.com/simonw/sqlite-utils/issues/590#issuecomment-1704387161,https://api.github.com/repos/simonw/sqlite-utils/issues/590,1704387161,IC_kwDOCGYnMM5lluJZ,9599,2023-09-03T19:50:36Z,2023-09-03T19:50:36Z,OWNER,"Maybe just populate `db.memory: bool` and `db.memory_name: Optional[str]` for this, then document them.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879214365, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704383901,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704383901,IC_kwDOCGYnMM5lltWd,9599,2023-09-03T19:34:05Z,2023-09-03T19:34:05Z,OWNER,"For that particular case I realized I'd quite like to have a mechanism for applying functions for a block of code and then de-registering them at the end - a context manager. I played with this idea a bit: ```python with db.register_functions(md5, md5_random): db.query(...) ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384111,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384111,IC_kwDOCGYnMM5lltZv,9599,2023-09-03T19:35:03Z,2023-09-03T19:35:03Z,OWNER,"Normally in Python/`sqlite3` you de-register a function by passing `None` to it. You can't do that with `db.register_function()` at the moment because a `fn` of `None` does something else: https://github.com/simonw/sqlite-utils/blob/1260bdc7bfe31c36c272572c6389125f8de6ef71/sqlite_utils/db.py#L461-L464","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560, https://github.com/simonw/sqlite-utils/issues/589#issuecomment-1704384393,https://api.github.com/repos/simonw/sqlite-utils/issues/589,1704384393,IC_kwDOCGYnMM5llteJ,9599,2023-09-03T19:36:34Z,2023-09-03T19:36:34Z,OWNER,"Here's a prototype: https://github.com/simonw/sqlite-utils/commit/62f673835c4a66f87cf6f949eaff43c8b014619b Still needs tests and documentation (and some more thought to make sure it's doing the right thing).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1879209560, https://github.com/simonw/sqlite-utils/issues/588#issuecomment-1694823972,https://api.github.com/repos/simonw/sqlite-utils/issues/588,1694823972,IC_kwDOCGYnMM5lBPYk,9599,2023-08-28T00:41:54Z,2023-08-28T00:41:54Z,OWNER,Tips on typing `**kwargs`: https://adamj.eu/tech/2021/05/11/python-type-hints-args-and-kwargs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1868713944, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096129,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096129,IC_kwDOCGYnMM5kcIbB,9599,2023-08-19T20:03:00Z,2023-08-19T20:03:00Z,OWNER,"Simplest possible recreation of the bug: ```bash python -c ' import sqlite_utils db = sqlite_utils.Database(memory=True) db.execute("""""" CREATE TABLE ""logs"" ( [id] INTEGER PRIMARY KEY, [chat_id] INTEGER REFERENCES [log]([id]), [reply_to_id] INTEGER ); """""") db[""logs""].add_foreign_key(""reply_to_id"", ""logs"", ""id"") ' ``` That `chat_id` line is the line that causes the problem - because it is defining a reference to a table that no longer exists!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096284,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096284,IC_kwDOCGYnMM5kcIdc,9599,2023-08-19T20:03:59Z,2023-08-19T20:03:59Z,OWNER,"Although this is revealing a problem in the underlying code (that schema is invalid), it also represents a regression: `sqlite-utils 3.34` ran this just fine, but it fails on `sqlite-utils 3.35` due to the change made in: - #577","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384, https://github.com/simonw/sqlite-utils/issues/587#issuecomment-1685096381,https://api.github.com/repos/simonw/sqlite-utils/issues/587,1685096381,IC_kwDOCGYnMM5kcIe9,9599,2023-08-19T20:04:32Z,2023-08-19T20:04:32Z,OWNER,I'm inclined to say this isn't a bug in `sqlite-utils` though - it's a bug in the code that calls it. So I'm not going to fix it here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1857851384, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683396150,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683396150,IC_kwDOCGYnMM5kVpY2,9599,2023-08-18T06:02:18Z,2023-08-18T06:06:31Z,OWNER,"More notes in here: - https://github.com/simonw/datasette-edit-schema/issues/35#issuecomment-1683392873 Not all Python/SQLite installations exhibit this problem by default! It turns out this is controlled by the `legacy_alter_table` pragma: https://sqlite.org/pragma.html#pragma_legacy_alter_table If that PRAGMA is turned on (default in newer SQLites) then `alter table` will error if you try to rename a table that is referenced in a view. Here's a one-liner to test if it is on or not: ```bash python -c 'import sqlite3; print(sqlite3.connect("":memory:"").execute(""PRAGMA legacy_alter_table"").fetchall())' ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683398866,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683398866,IC_kwDOCGYnMM5kVqDS,9599,2023-08-18T06:05:50Z,2023-08-18T06:06:42Z,OWNER,"Options: - Provide a `recreate_views: bool` parameter to `table.transform()` controlling if views that might reference this table are stashed and dropped and recreated within a transaction as part of the operation. But should that be `True` or `False` by default? - Read that `PRAGMA` and automatically do that view workaround if it's turned on - Toggle that `PRAGMA` off for the duration of the `.transform()` operation and on again at the end. Does it only affect the current connection? - Try the `transform()` in a transaction, detect the `""error in view""`, `""no such table""`error, if spotted then do the VIEW workaround and try again I'm on the fence as to which of these I like the most. I'm tempted to go with the one which just drops VIEWS and recreates them all the time, because it feels simpler.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668, https://github.com/simonw/sqlite-utils/issues/586#issuecomment-1683404978,https://api.github.com/repos/simonw/sqlite-utils/issues/586,1683404978,IC_kwDOCGYnMM5kVriy,9599,2023-08-18T06:13:46Z,2023-08-18T06:13:46Z,OWNER,"I shipped the view recreating fix in `datasette-edit-schema`, so at least I can start exercising that fix and see if it has any weird issues.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1856075668, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683195669,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683195669,IC_kwDOCGYnMM5kU4cV,9599,2023-08-18T01:24:57Z,2023-08-18T01:24:57Z,OWNER,"Currently: ```bash sqlite-utils transform --help ``` ``` Usage: sqlite-utils transform [OPTIONS] PATH TABLE Transform a table beyond the capabilities of ALTER TABLE Example: sqlite-utils transform mydb.db mytable \ --drop column1 \ --rename column2 column_renamed Options: --type ... Change column type to INTEGER, TEXT, FLOAT or BLOB --drop TEXT Drop this column --rename ... Rename this column to X -o, --column-order TEXT Reorder columns --not-null TEXT Set this column to NOT NULL --not-null-false TEXT Remove NOT NULL from this column --pk TEXT Make this column the primary key --pk-none Remove primary key (convert to rowid table) --default ... Set default value for this column --default-none TEXT Remove default from this column --drop-foreign-key TEXT Drop foreign key constraint for this column --sql Output SQL without executing it --load-extension TEXT Path to SQLite extension, with optional :entrypoint -h, --help Show this message and exit. ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683197882,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683197882,IC_kwDOCGYnMM5kU4-6,9599,2023-08-18T01:25:53Z,2023-08-18T01:25:53Z,OWNER,"Probably most relevant here is this snippet from: ```bash sqlite-utils create-table --help ``` ``` --default ... Default value that should be set for a column --fk ... Column, other table, other column to set as a foreign key ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683198740,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683198740,IC_kwDOCGYnMM5kU5MU,9599,2023-08-18T01:26:47Z,2023-08-18T01:26:47Z,OWNER,"The only CLI feature that supports providing just the column name appears to be this: ```bash sqlite-utils add-foreign-key --help ``` ``` Usage: sqlite-utils add-foreign-key [OPTIONS] PATH TABLE COLUMN [OTHER_TABLE] [OTHER_COLUMN] Add a new foreign key constraint to an existing table Example: sqlite-utils add-foreign-key my.db books author_id authors id WARNING: Could corrupt your database! Back up your database file first. ``` I can drop that WARNING now since I'm not writing to `sqlite_master` any more.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683200128,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683200128,IC_kwDOCGYnMM5kU5iA,9599,2023-08-18T01:29:00Z,2023-08-18T01:29:00Z,OWNER,I'm not going to implement the `foreign_keys=` option that entirely replaces existing foreign keys - I'll just do a `--add-foreign-key` multi-option.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683201239,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683201239,IC_kwDOCGYnMM5kU5zX,9599,2023-08-18T01:30:46Z,2023-08-18T01:30:46Z,OWNER,"Help can now look like this: ``` --drop-foreign-key TEXT Drop foreign key constraint for this column --add-foreign-key ... Add a foreign key constraint from a column to another table with another column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683212074,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683212074,IC_kwDOCGYnMM5kU8cq,9599,2023-08-18T01:43:54Z,2023-08-18T01:43:54Z,OWNER,"Some manual testing: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); ``` ```bash sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [places] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER ); CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); ``` ```bash sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id sqlite-utils schema /tmp/t.db ``` ```sql CREATE TABLE [country] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [city] ( [id] INTEGER, [name] TEXT ); CREATE TABLE [continent] ( [id] INTEGER, [name] TEXT ); CREATE TABLE ""places"" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); ``` ```bash sqlite-utils transform /tmp/t.db places --drop-foreign-key country sqlite-utils schema /tmp/t.db places ``` ```sql CREATE TABLE ""places"" ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER, [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222, https://github.com/simonw/sqlite-utils/issues/585#issuecomment-1683217284,https://api.github.com/repos/simonw/sqlite-utils/issues/585,1683217284,IC_kwDOCGYnMM5kU9uE,9599,2023-08-18T01:50:21Z,2023-08-18T01:50:21Z,OWNER,"And a test of the `--sql` option: ```bash sqlite-utils create-table /tmp/t.db places id integer name text country integer city integer continent integer --pk id sqlite-utils create-table /tmp/t.db country id integer name text sqlite-utils create-table /tmp/t.db city id integer name text sqlite-utils create-table /tmp/t.db continent id integer name text sqlite-utils transform /tmp/t.db places --add-foreign-key country country id --add-foreign-key continent continent id --sql ``` Outputs: ```sql CREATE TABLE [places_new_6a705d2f5a13] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [country] INTEGER REFERENCES [country]([id]), [city] INTEGER, [continent] INTEGER REFERENCES [continent]([id]) ); INSERT INTO [places_new_6a705d2f5a13] ([id], [name], [country], [city], [continent]) SELECT [id], [name], [country], [city], [continent] FROM [places]; DROP TABLE [places]; ALTER TABLE [places_new_6a705d2f5a13] RENAME TO [places]; ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855894222, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left: ``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): > raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) E sqlite_utils.db.AlterError: No such column: breeds.rowid sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem: ```python from sqlite_utils import Database fresh_db = Database(memory=True) fresh_db.create_table(""dogs"", {""name"": str}) fresh_db.create_table(""breeds"", {""name"": str}) fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"") ``` Traceback: ``` Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql self.db.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line: ```python fresh_db.create_table(""breeds"", {""name"": str}) ``` Has this schema: ```sql CREATE TABLE [breeds] ( [name] TEXT ); ``` SQLite creates an invisible `rowid` column for it automatically. On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added: ```sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); ``` But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884 Here's what the debugger reveals about this code: ```python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) ``` ``` (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures: ``` sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[, ] = ..."") [var-annotated] sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform"" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] Found 5 errors in 1 file (checked 56 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683137259,IC_kwDOCGYnMM5kUqLr,9599,2023-08-18T00:06:59Z,2023-08-18T00:06:59Z,OWNER,"The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683138953,IC_kwDOCGYnMM5kUqmJ,9599,2023-08-18T00:09:20Z,2023-08-18T00:09:20Z,OWNER,"Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop: ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683139304,IC_kwDOCGYnMM5kUqro,9599,2023-08-18T00:09:56Z,2023-08-18T00:09:56Z,OWNER,"Upgrading `flake8` locally replicated the error: ``` pip install -U flake8 flake8 ``` ``` ./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96 ./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683143723,IC_kwDOCGYnMM5kUrwr,9599,2023-08-18T00:14:52Z,2023-08-18T00:14:52Z,OWNER,"Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times: > The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries. > > `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up. > > Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`. That doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145110,IC_kwDOCGYnMM5kUsGW,9599,2023-08-18T00:16:28Z,2023-08-18T00:16:48Z,OWNER,"One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`: https://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708 I should write tests for them too.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683145819,IC_kwDOCGYnMM5kUsRb,9599,2023-08-18T00:17:26Z,2023-08-18T00:17:26Z,OWNER,Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683164661,IC_kwDOCGYnMM5kUw31,9599,2023-08-18T00:45:53Z,2023-08-18T00:45:53Z,OWNER,"More updated documentation: - https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform - https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on: - #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914, https://github.com/simonw/sqlite-utils/issues/581#issuecomment-1652496702,https://api.github.com/repos/simonw/sqlite-utils/issues/581,1652496702,IC_kwDOCGYnMM5ifxk-,9599,2023-07-26T21:07:45Z,2023-07-26T21:07:45Z,OWNER,Docs: https://sqlite-utils.datasette.io/en/latest/cli.html#using-the-debugger,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1823160748, https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648323482,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648323482,IC_kwDOCGYnMM5iP2ua,9599,2023-07-24T17:31:56Z,2023-07-24T17:31:56Z,OWNER,"The main blocker here is coming up with a design. The challenge is cleanly integrating it with the existing format options: https://github.com/simonw/sqlite-utils/blob/8bee14588687b66c54c7a3dfae5de2f9cc7cac3f/docs/cli-reference.rst#L114-L132 I'd like to avoid breaking backwards compatibility. The most obvious solution is to add a `--format x` option. This is slightly confusing as `--fmt` already exists. Or... `--fmt` could be enhanced to handle plugins too - and the existing set of formats (from the `tabulate` integration) could be refactored to use the new hook. If I ever do release `sqlite-utils 4` I'm tempted to move that tabulate stuff into a plugin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648324312,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648324312,IC_kwDOCGYnMM5iP27Y,9599,2023-07-24T17:32:21Z,2023-07-24T17:32:21Z,OWNER,I'm currently leaning very slightly towards `--format geojson` - and explaining in the docs that `--format` is for formats added by plugins.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294, https://github.com/simonw/sqlite-utils/issues/578#issuecomment-1648325682,https://api.github.com/repos/simonw/sqlite-utils/issues/578,1648325682,IC_kwDOCGYnMM5iP3Qy,9599,2023-07-24T17:33:10Z,2023-07-24T17:33:10Z,OWNER,"A related feature would be support for plugins to add new ways of _ingesting_ data - currently `sqlite-utils insert` works against JSON, newline-JSON, CSV and TSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1818838294, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work. It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872 Improving this code to support adding foreign keys as well would be pretty simple. And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others: - https://github.com/simonw/llm/issues/60 - https://github.com/simonw/llm/issues/116 - https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149 At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`. Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments: ```python def transform( self, *, # ... # This one exists already: drop_foreign_keys: Optional[Iterable] = None, # These two are new. This one specifies keys to add: add_foreign_keys: Optional[ForeignKeysType] = None, # Or this one causes them all to be replaced with the new definitions: foreign_keys: Optional[ForeignKeysType] = None, ``` There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this: ```python drop_foreign_keys: Optional[Iterable] = None, ``` Should be this: ```python drop_foreign_keys: Optional[Iterable[str]] = None, ``` Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683076325,IC_kwDOCGYnMM5kUbTl,9599,2023-08-17T22:48:36Z,2023-08-17T22:48:36Z,OWNER,"I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism. It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133 Needs updating like so: ```diff diff --git a/tests/test_extract.py b/tests/test_extract.py index 70ad0cf..fd52534 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db): assert fresh_db[""tree""].schema == ( 'CREATE TABLE ""tree"" (\n' "" [name] TEXT,\n"" - "" [common_name_latin_name_id] INTEGER,\n"" - "" FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n"" + "" [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n"" "")"" ) assert ( ``` Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in. I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1684235760,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1684235760,IC_kwDOCGYnMM5kY2Xw,9599,2023-08-18T17:43:11Z,2023-08-18T17:43:11Z,OWNER,"Here's a new plugin that brings back the `sqlite_master` modifying version, for those that can use it: https://github.com/simonw/sqlite-utils-fast-fks","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646752814,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646752814,IC_kwDOCGYnMM5iJ3Qu,9599,2023-07-23T05:43:13Z,2023-07-23T05:43:13Z,OWNER,"Using `llm` in my `sqlite-utils` local checkout: ```bash git diff 0.2 0.3 | llm -s 'release notes as markdown bullet points' ``` > - Added a new `Table` class in the `db` module which represents a table in a SQLite database > - The `Table` class has properties that provide information about the table, such as column names, foreign keys, and schema > - The `Table` class also has a `count` property that returns the number of rows in the table > - The `Table` class has a `create` method that can be used to create a new table with specified columns, primary key, and foreign keys > - Updated the `Database` class in the `__init__` module to include the `Table` class and expose it via the `__all__` variable > - Added new tests for creating tables and introspecting tables ```bash git diff 0.3 0.3.1 | llm -s 'release notes as markdown bullet points' ``` > - Updated PyPI version badge to display the latest version > - Updated Travis CI badge to link to the master branch > - Added a badge for documentation status > - Added a license badge > - Added a description of the package > - Added installation instructions > - Added a link to the documentation > - Added a ""Related projects"" section and linked to Datasette and csvs-to-sqlite > - Updated the version in setup.py to 0.3.1 ```bash git diff 0.3.1 0.4 | llm -s 'release notes as markdown bullet points' ``` > - Added an `enable_fts` method to the `Table` class that enables full-text search on specified columns > - Added a `populate_fts` method to the `Table` class that refreshes the search index for full-text search > - Added a `search` method to the `Table` class that allows searching the table using full-text search > - Updated the `setup.py` script to version 0.4 > - Updated the `db.py` file to include the `enable_fts`, `populate_fts`, and `search` methods > - Added test cases for the `enable_fts`, `populate_fts`, and `search` methods in the `test_enable_fts.py` file > - Added test cases for the `count`, `columns`, and `schema` properties in the `test_introspect.py` file > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390, https://github.com/simonw/sqlite-utils/issues/576#issuecomment-1646753678,https://api.github.com/repos/simonw/sqlite-utils/issues/576,1646753678,IC_kwDOCGYnMM5iJ3eO,9599,2023-07-23T05:49:51Z,2023-07-23T05:49:51Z,OWNER,Done - bottom of https://sqlite-utils.datasette.io/en/latest/changelog.html#id116,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816997390, https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646687461,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646687461,IC_kwDOCGYnMM5iJnTl,9599,2023-07-22T23:01:44Z,2023-07-22T23:01:44Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/3f80a026983d3e634f05a46f2a6da162b5139dd9/sqlite_utils/db.py#L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568, https://github.com/simonw/sqlite-utils/issues/575#issuecomment-1646688288,https://api.github.com/repos/simonw/sqlite-utils/issues/575,1646688288,IC_kwDOCGYnMM5iJngg,9599,2023-07-22T23:08:22Z,2023-07-22T23:08:22Z,OWNER,Documented here: https://sqlite-utils.datasette.io/en/latest/plugins.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816919568, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646686477,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646686477,IC_kwDOCGYnMM5iJnEN,9599,2023-07-22T22:52:56Z,2023-07-22T22:52:56Z,OWNER,"Alex built this in: - #573","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688339,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688339,IC_kwDOCGYnMM5iJnhT,9599,2023-07-22T23:08:49Z,2023-07-22T23:08:49Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#prepare-connection-conn,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185, https://github.com/simonw/sqlite-utils/issues/574#issuecomment-1646688748,https://api.github.com/repos/simonw/sqlite-utils/issues/574,1646688748,IC_kwDOCGYnMM5iJnns,9599,2023-07-22T23:12:03Z,2023-07-22T23:13:14Z,OWNER,"Just tried this out by adding the example from the docs to my `sqlite-utils-hello-world` plugin and running: ```bash sqlite-utils install -e ../sqlite-utils-hello-world ``` ``` Obtaining file:///Users/simon/Dropbox/Development/sqlite-utils-hello-world ... ``` Then: ```bash sqlite-utils memory ""select hello('simon')"" ``` ```json [{""hello('simon')"": ""Hello, simon!""}] ``` Also: ```pycon >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> list(db.query(""select hello('simon')"")) [{""hello('simon')"": 'Hello, simon!'}] >>> db2 = sqlite_utils.Database(memory=True, execute_plugins=False) >>> list(db2.query(""select hello('simon')"")) Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 494, in query cursor = self.execute(sql, params or tuple()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 512, in execute return self.conn.execute(sql, parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ sqlean.dbapi2.OperationalError: no such function: hello ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816918185, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686332,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686332,IC_kwDOCGYnMM5iJnB8,9599,2023-07-22T22:52:01Z,2023-07-22T22:52:01Z,OWNER,I was literally seconds away from shipping version 3.34 but I this looks good so I'm going to try and get it in there.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646686675,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646686675,IC_kwDOCGYnMM5iJnHT,9599,2023-07-22T22:54:38Z,2023-07-22T22:54:38Z,OWNER," Glitch in the rendered documentation from https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522, https://github.com/simonw/sqlite-utils/pull/573#issuecomment-1646687103,https://api.github.com/repos/simonw/sqlite-utils/issues/573,1646687103,IC_kwDOCGYnMM5iJnN_,9599,2023-07-22T22:58:35Z,2023-07-22T22:58:35Z,OWNER,"https://sqlite-utils--573.org.readthedocs.build/en/573/plugins.html#prepare-connection-conn ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816917522, https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646660777,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646660777,IC_kwDOCGYnMM5iJgyp,9599,2023-07-22T19:59:13Z,2023-07-22T19:59:13Z,OWNER,I don't know where that is coming from! I can't see `textual` as a dependency we are pulling in anywhere.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910, https://github.com/simonw/sqlite-utils/issues/572#issuecomment-1646681192,https://api.github.com/repos/simonw/sqlite-utils/issues/572,1646681192,IC_kwDOCGYnMM5iJlxo,9599,2023-07-22T22:12:08Z,2023-07-22T22:12:08Z,OWNER,Found it: https://github.com/simonw/sqlite-utils/blob/18f190e28334d821be78a1dbbf31d7610fc1f9c1/.github/workflows/test.yml#L29-L31,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816877910, https://github.com/simonw/sqlite-utils/issues/571#issuecomment-1646682686,https://api.github.com/repos/simonw/sqlite-utils/issues/571,1646682686,IC_kwDOCGYnMM5iJmI-,9599,2023-07-22T22:23:47Z,2023-07-22T22:23:47Z,OWNER,"Demo: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].insert({""name"": ""Cleo""})
>>> print(db[""foo""].schema) CREATE TABLE [foo] ( [name] TEXT ) >>> db[""foo""].transform(not_null={""name""})
>>> print(db[""foo""].schema) CREATE TABLE ""foo"" ( [name] TEXT NOT NULL ) >>> db[""foo""].transform(types={""name"": int}, keep_table=""kept"")
>>> print(db.schema) CREATE TABLE ""kept"" ( [name] TEXT NOT NULL ); CREATE TABLE ""foo"" ( [name] INTEGER NOT NULL ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816876211, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527091278,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527091278,MDEyOklzc3VlQ29tbWVudDUyNzA5MTI3OA==,9599,2019-09-02T10:13:32Z,2019-09-02T10:13:32Z,OWNER,"This is brilliant! One tiny suggestion: I like `--create-triggers` and `create_triggers=True` better for this as they are shorter but still capture what it dies - especially since the underlying SQL uses `CREATE TRIGGER` statements.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527258212,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527258212,MDEyOklzc3VlQ29tbWVudDUyNzI1ODIxMg==,9599,2019-09-02T23:40:56Z,2019-09-02T23:40:56Z,OWNER,"This is fantastic, thanks so much. I spotted a Datasette bug triggered by this: Datasette has its own version of the `detect_fts` function - at https://github.com/simonw/datasette/blob/d224ee2c98ac39c2c6e21a0ac0c62e5c3e1ccd11/datasette/utils/__init__.py#L466-L479 - which fails to pick up FTS tables created using the new escaping pattern. It's a bug in Datasette, not sqlite-utils - so I'll open an issue there.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958, https://github.com/simonw/sqlite-utils/pull/57#issuecomment-527266798,https://api.github.com/repos/simonw/sqlite-utils/issues/57,527266798,MDEyOklzc3VlQ29tbWVudDUyNzI2Njc5OA==,9599,2019-09-03T01:03:59Z,2019-09-03T01:03:59Z,OWNER,Released in 1.11 - thanks again! https://sqlite-utils.readthedocs.io/en/latest/changelog.html#v1-11,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",487987958, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646643676,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646643676,IC_kwDOCGYnMM5iJcnc,9599,2023-07-22T18:18:24Z,2023-07-22T18:18:24Z,OWNER,"Here's where I added that to LLM: https://github.com/simonw/llm/commit/a396950f7934e82a9968703bb3ce9ab7ab62f7f8 - https://github.com/simonw/llm/issues/49","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646645990,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646645990,IC_kwDOCGYnMM5iJdLm,9599,2023-07-22T18:29:40Z,2023-07-22T18:30:16Z,OWNER,"Example plugin: https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf ```bash sqlite-utils install https://gist.github.com/simonw/ccfbbf9b384a38ac7535b01849f57daf/archive/cd7960e476c441a3c5f619e2a44a641a39b91467.zip ``` `pyproject.toml`: ```toml [project] name = ""sqlite-utils-hello-world"" version = ""0.1"" [project.entry-points.sqlite_utils] hello_world = ""sqlite_utils_hello_world"" ``` `sqlite_utils_hello_world.py`: ```python import click import sqlite_utils @sqlite_utils.hookimpl def register_commands(cli): @cli.command() def hello_world(): ""Say hello world"" click.echo(""Hello world!"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402, https://github.com/simonw/sqlite-utils/issues/569#issuecomment-1646654275,https://api.github.com/repos/simonw/sqlite-utils/issues/569,1646654275,IC_kwDOCGYnMM5iJfND,9599,2023-07-22T19:19:35Z,2023-07-22T19:19:35Z,OWNER,Documentation: https://sqlite-utils.datasette.io/en/latest/plugins.html#register-commands-cli,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816852402, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642666,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642666,IC_kwDOCGYnMM5iJcXq,9599,2023-07-22T18:13:19Z,2023-07-22T18:13:19Z,OWNER,"https://sqlite-utils.datasette.io/en/stable/cli-reference.html#create-table ```bash sqlite-utils create-table ... --replace ``` That also has `--ignore`: ``` --ignore If table already exists, do nothing --replace If table already exists, replace it --transform If table already exists, try to transform the schema ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646642959,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646642959,IC_kwDOCGYnMM5iJccP,9599,2023-07-22T18:14:49Z,2023-07-22T18:14:49Z,OWNER,Here's where those are implemented for the `create-table` CLI command: https://github.com/simonw/sqlite-utils/blob/f7af23837deab5c98dae9441d1f68318065d7d8c/sqlite_utils/cli.py#L1543-L1564,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646652105,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646652105,IC_kwDOCGYnMM5iJerJ,9599,2023-07-22T19:05:13Z,2023-07-22T19:05:13Z,OWNER,I think this is `replace=True` and `ignore=True` to match the CLI. And refactoring the CLI to use them.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653382,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653382,IC_kwDOCGYnMM5iJe_G,9599,2023-07-22T19:13:20Z,2023-07-22T19:13:20Z,OWNER,"Demo: ```pycon >>> from sqlite_utils import Database >>> db = Database(memory=True) >>> db[""foo""].create({""id"": int})
>>> db[""foo""].create({""id"": int}) Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1647, in create self.db.create_table( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1030, in create_table self.execute(sql) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 510, in execute return self.conn.execute(sql) ^^^^^^^^^^^^^^^^^^^^^^ sqlean.dbapi2.OperationalError: table [foo] already exists >>> db[""foo""].create({""id"": int}, ignore=True)
>>> db[""foo""].create({""id"": int, ""name"": str}, replace=True)
>>> db[""foo""].create({""id"": int, ""name"": str, ""age"": int}, transform=True)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646653610,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646653610,IC_kwDOCGYnMM5iJfCq,9599,2023-07-22T19:14:56Z,2023-07-22T19:14:56Z,OWNER,"Manual testing of CLI command as well: ``` $ sqlite-utils create-table /tmp/f.db foo id integer $ sqlite-utils create-table /tmp/f.db foo id integer Error: Table ""foo"" already exists. Use --replace to delete and replace it. $ sqlite-utils create-table /tmp/f.db foo id integer --replace $ sqlite-utils create-table /tmp/f.db foo id $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER ); $ sqlite-utils create-table /tmp/f.db foo id integer name str --transform Error: column types must be one of ('INTEGER', 'TEXT', 'FLOAT', 'BLOB') $ sqlite-utils create-table /tmp/f.db foo id integer name text --transform $ sqlite-utils schema /tmp/f.db CREATE TABLE ""foo"" ( [id] INTEGER, [name] TEXT ); $ sqlite-utils create-table /tmp/f.db foo id integer name text --ignore $ sqlite-utils create-table /tmp/f.db foo id integer name text --replace $ sqlite-utils schema /tmp/f.db CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646654818,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646654818,IC_kwDOCGYnMM5iJfVi,9599,2023-07-22T19:22:40Z,2023-07-22T19:22:40Z,OWNER,"I think this broke a test: ``` $ pytest tests/test_tracer.py =============================================== test session starts ================================================ platform darwin -- Python 3.11.4, pytest-7.2.2, pluggy-1.0.0 rootdir: /Users/simon/Dropbox/Development/sqlite-utils plugins: icdiff-0.6, hypothesis-6.68.2 collected 2 items tests/test_tracer.py F. [100%] ===================================================== FAILURES ===================================================== ___________________________________________________ test_tracer ____________________________________________________ def test_tracer(): collected = [] db = Database( memory=True, tracer=lambda sql, params: collected.append((sql, params)) ) db[""dogs""].insert({""name"": ""Cleopaws""}) db[""dogs""].enable_fts([""name""]) db[""dogs""].search(""Cleopaws"") > assert collected == [ (""PRAGMA recursive_triggers=on;"", None), (""select name from sqlite_master where type = 'view'"", None), (""select name from sqlite_master where type = 'table'"", None), (""select name from sqlite_master where type = 'view'"", None), (""CREATE TABLE [dogs] (\n [name] TEXT\n);\n "", None), (""select name from sqlite_master where type = 'view'"", None), (""INSERT INTO [dogs] ([name]) VALUES (?);"", [""Cleopaws""]), (""select name from sqlite_master where type = 'view'"", None), ( ""CREATE VIRTUAL TABLE [dogs_fts] USING FTS5 (\n [name],\n content=[dogs]\n)"", None, ), ( ""INSERT INTO [dogs_fts] (rowid, [name])\n SELECT rowid, [name] FROM [dogs];"", None, ), (""select name from sqlite_master where type = 'view'"", None), ] E assert equals failed E [ [ E ('PRAGMA recursive_triggers=on;', None), ('PRAGMA recursive_triggers=on;', None), E ( E ""select name from sqlite_master where type = E 'view'"", E None, ... E E ...Full output truncated (13 lines hidden), use '-vv' to show tests/test_tracer.py:12: AssertionError ============================================= short test summary info ============================================== FAILED tests/test_tracer.py::test_tracer - assert equals failed =========================================== 1 failed, 1 passed in 0.05s ============================================ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056, https://github.com/simonw/sqlite-utils/issues/568#issuecomment-1646655272,https://api.github.com/repos/simonw/sqlite-utils/issues/568,1646655272,IC_kwDOCGYnMM5iJfco,9599,2023-07-22T19:25:35Z,2023-07-22T19:25:35Z,OWNER,"Here's why that test broke: https://github.com/simonw/sqlite-utils/blob/58b577279fcd5ef6ce88f88b28668dffebfe7f44/sqlite_utils/db.py#L960-L964 I added an extra `if self[name].exists()` check to the `db.create_table()` method.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1816851056, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1638926655,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1638926655,IC_kwDOCGYnMM5hsAk_,9599,2023-07-17T21:42:37Z,2023-07-17T21:42:37Z,OWNER,"I really like this. I'm also interested in: - Plugins that make new custom SQL functions available - similar to this Datasette hook: https://docs.datasette.io/en/stable/plugin_hooks.html#prepare-connection-conn-database-datasette - Plugins that register functions that can be used as recipes for `sqlite-utils convert` https://sqlite-utils.datasette.io/en/stable/cli.html#sqlite-utils-convert-recipes The upload-data-to-Datasette problem is planned to be solved by a future version of https://github.com/simonw/dclient ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643379,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643379,IC_kwDOCGYnMM5iJciz,9599,2023-07-22T18:16:54Z,2023-07-22T18:16:54Z,OWNER,"> Would this possibly make a bunch of `x-to-sqlite` tools obsolete? Or nudge some to become plugins? Yeah, it could do! That's not a terrible idea to be honest, those things have really been proliferating. Alternatively, they could each register themselves as plugins in addition - so if you install e.g. `pocket-to-sqlite` you could then optionally also run it as `sqlite-utils pocket-to-sqlite ...` The benefit there is for people who install `sqlite-utils` from Homebrew, where it gets its own virtual environment. They could run: ```bash brew install sqlite-utils sqlite-utils install pocket-to-sqlite sqlite-utils pocket-to-sqlite ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646643450,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646643450,IC_kwDOCGYnMM5iJcj6,9599,2023-07-22T18:17:18Z,2023-07-22T18:17:18Z,OWNER,I'm going to start by adding the `register_command` hook using the exact same pattern as Datasette and LLM.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646654383,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646654383,IC_kwDOCGYnMM5iJfOv,9599,2023-07-22T19:20:16Z,2023-07-22T19:20:16Z,OWNER,"Here's documentation for the new plugins mechanism, including a very short tutorial on writing a new plugin (inspired by https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html): https://sqlite-utils.datasette.io/en/latest/plugins.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646686424,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646686424,IC_kwDOCGYnMM5iJnDY,9599,2023-07-22T22:52:34Z,2023-07-22T22:52:34Z,OWNER,Splitting off an issue for `prepare_connection()` since Alex got the PR in seconds before I shipped 3.34!,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1801394744, https://github.com/simonw/sqlite-utils/issues/567#issuecomment-1646687219,https://api.github.com/repos/simonw/sqlite-utils/issues/567,1646687219,IC_kwDOCGYnMM5iJnPz,9599,2023-07-22T22:59:36Z,2023-07-22T22:59:36Z,OWNER,Now that we have two plugin hooks I'm closing this issue (we can open other issues for further hooks).,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1801394744, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627597872,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627597872,IC_kwDOCGYnMM5hAyww,9599,2023-07-09T04:09:56Z,2023-07-09T04:09:56Z,OWNER,"Thanks, looks like a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865, https://github.com/simonw/sqlite-utils/issues/566#issuecomment-1627598570,https://api.github.com/repos/simonw/sqlite-utils/issues/566,1627598570,IC_kwDOCGYnMM5hAy7q,9599,2023-07-09T04:13:34Z,2023-07-09T04:13:34Z,OWNER,On consulting https://pypi.org/project/tabulate/ it looks like most of those formats don't actually makes sense without headers - so the right thing here might be to raise an error if `--fmt` and `--no-headers` are used at the same time.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1795219865,