issue_comments

Oops, mypy failures:

sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for "by_table" (hint: "by_table: Dict[<type>, <type>] = ...") [var-annotated] sqlite_utils/db.py:1169: error: Item "View" of "Union[Table, View]" has no attribute "transform" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing> [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing> [arg-type] Found 5 errors in 1 file (checked 56 source files)

The problem here is that the table created by this line: python fresh_db.create_table("breeds", {"name": str}) Has this schema: sql CREATE TABLE [breeds] ( [name] TEXT ); SQLite creates an invisible rowid column for it automatically.

On the main branch with the old implementation that table ends up looking like this after the foreign key has been added: sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884

Here's what the debugger reveals about this code: python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( "No such column: {}.{}".format(fk.other_table, fk.other_column) ) (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]

Just these three lines recreate the problem: ```python from sqlite_utils import Database

fresh_db = Database(memory=True)

fresh_db.create_table("dogs", {"name": str}) fresh_db.create_table("breeds", {"name": str}) fresh_db["dogs"].add_column("breed_id", fk="breeds") Traceback: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1896, in transform_sql self.db.create_table_sql( File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ```

Full test:

https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484

Just one failing test left:

``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ):

          raise AlterError(
                "No such column: {}.{}".format(fk.other_table, fk.other_column)
            )

E sqlite_utils.db.AlterError: No such column: breeds.rowid

sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ```

Spotted this while working on: - #577

An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133

Needs updating like so: diff diff --git a/tests/test_extract.py b/tests/test_extract.py index 70ad0cf..fd52534 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db): assert fresh_db["tree"].schema == ( 'CREATE TABLE "tree" (\n' " [name] TEXT,\n" - " [common_name_latin_name_id] INTEGER,\n" - " FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n" + " [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n" ")" ) assert ( Unfortunately this means it may break other test suites that depend on sqlite-utils that have schema tests like this baked in.

I don't think this should count as a breaking change release though, but it's still worth noting.

I'm inclined to just go with the .transform() method and not attempt to keep around the method that involves updating sqlite_master and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism.

It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.

Actually I think table.transform() might get the following optional arguments: python def transform( self, *, # ... # This one exists already: drop_foreign_keys: Optional[Iterable] = None, # These two are new. This one specifies keys to add: add_foreign_keys: Optional[ForeignKeysType] = None, # Or this one causes them all to be replaced with the new definitions: foreign_keys: Optional[ForeignKeysType] = None, There should be validation that forbids you from using foreign_keys= at the same time as either drop_foreign_keys= or add_foreign_keys= because the point of foreign_keys= is to define the keys for the new table all in one go.

As a reminder: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165

Maybe this: python drop_foreign_keys: Optional[Iterable] = None, Should be this: python drop_foreign_keys: Optional[Iterable[str]] = None, Because it takes a list of column names that should have their foreign keys dropped.

Looking at the whole of the .add_foreign_keys() method, the first section of it can remain unchanged - it's just a bunch of validation:

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149

At that point we have foreign_keys_to_create as the ones that are new, but we should instead try to build up a foreign_keys which is both new and old, ready to be passed to .transform().

Here's the rest of that function, which will be replaced by a called to .transform(foreign_keys=foreign_keys):

https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177

This would help address these issues, among potentially many others: - https://github.com/simonw/llm/issues/60 - https://github.com/simonw/llm/issues/116 - https://github.com/simonw/llm/issues/123

I'm certain this could work.

It turns out the .transform() method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the drop_foreign_keys= parameter:

https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872

Improving this code to support adding foreign keys as well would be pretty simple.

And then the .add_foreign_keys() and .add_foreign_key() methods could be updated to use .transform(...) under the hood instead.