html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures: ``` sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[, ] = ..."") [var-annotated] sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform"" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type] Found 5 errors in 1 file (checked 56 source files) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line: ```python fresh_db.create_table(""breeds"", {""name"": str}) ``` Has this schema: ```sql CREATE TABLE [breeds] ( [name] TEXT ); ``` SQLite creates an invisible `rowid` column for it automatically. On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added: ```sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); ``` But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884 Here's what the debugger reveals about this code: ```python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) ``` ``` (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem: ```python from sqlite_utils import Database fresh_db = Database(memory=True) fresh_db.create_table(""dogs"", {""name"": str}) fresh_db.create_table(""breeds"", {""name"": str}) fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"") ``` Traceback: ``` Traceback (most recent call last): File """", line 1, in File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql self.db.create_table_sql( File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left: ``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): > raise AlterError( ""No such column: {}.{}"".format(fk.other_table, fk.other_column) ) E sqlite_utils.db.AlterError: No such column: breeds.rowid sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223, https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on: - #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133 Needs updating like so: ```diff diff --git a/tests/test_extract.py b/tests/test_extract.py index 70ad0cf..fd52534 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db): assert fresh_db[""tree""].schema == ( 'CREATE TABLE ""tree"" (\n' "" [name] TEXT,\n"" - "" [common_name_latin_name_id] INTEGER,\n"" - "" FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n"" + "" [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n"" "")"" ) assert ( ``` Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in. I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments: ```python def transform( self, *, # ... # This one exists already: drop_foreign_keys: Optional[Iterable] = None, # These two are new. This one specifies keys to add: add_foreign_keys: Optional[ForeignKeysType] = None, # Or this one causes them all to be replaced with the new definitions: foreign_keys: Optional[ForeignKeysType] = None, ``` There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this: ```python drop_foreign_keys: Optional[Iterable] = None, ``` Should be this: ```python drop_foreign_keys: Optional[Iterable[str]] = None, ``` Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149 At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`. Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others: - https://github.com/simonw/llm/issues/60 - https://github.com/simonw/llm/issues/116 - https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521, https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work. It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872 Improving this code to support adding foreign keys as well would be pretty simple. And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,