html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures:
```
sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"") [assignment]
sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[, ] = ..."") [var-annotated]
sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform"" [union-attr]
sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type]
sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type]
Found 5 errors in 1 file (checked 56 source files)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line:
```python
fresh_db.create_table(""breeds"", {""name"": str})
```
Has this schema:
```sql
CREATE TABLE [breeds] (
[name] TEXT
);
```
SQLite creates an invisible `rowid` column for it automatically.
On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added:
```sql
(Pdb) print(fresh_db.schema)
CREATE TABLE [dogs] (
[name] TEXT
, [breed_id] INTEGER,
FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid])
);
CREATE TABLE [breeds] (
[name] TEXT
);
```
But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884
Here's what the debugger reveals about this code:
```python
for fk in foreign_keys:
if fk.other_table == name and columns.get(fk.other_column):
continue
if not any(
c for c in self[fk.other_table].columns if c.name == fk.other_column
):
raise AlterError(
""No such column: {}.{}"".format(fk.other_table, fk.other_column)
)
```
```
(Pdb) fk
ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid')
(Pdb) self[fk.other_table].columns
[Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem:
```python
from sqlite_utils import Database
fresh_db = Database(memory=True)
fresh_db.create_table(""dogs"", {""name"": str})
fresh_db.create_table(""breeds"", {""name"": str})
fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"")
```
Traceback:
```
Traceback (most recent call last):
File """", line 1, in
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column
self.add_foreign_key(col_name, fk, fk_col)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key
self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys
self[table].transform(add_foreign_keys=fks)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform
sqls = self.transform_sql(
^^^^^^^^^^^^^^^^^^^
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql
self.db.create_table_sql(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql
raise AlterError(
sqlite_utils.db.AlterError: No such column: breeds.rowid
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test:
https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left:
```
# Soundness check foreign_keys point to existing tables
for fk in foreign_keys:
if fk.other_table == name and columns.get(fk.other_column):
continue
if not any(
c for c in self[fk.other_table].columns if c.name == fk.other_column
):
> raise AlterError(
""No such column: {}.{}"".format(fk.other_table, fk.other_column)
)
E sqlite_utils.db.AlterError: No such column: breeds.rowid
sqlite_utils/db.py:882: AlterError
==== short test summary info ====
FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid
==== 1 failed, 378 deselected in 0.49s ====
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,
https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on:
- #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133
Needs updating like so:
```diff
diff --git a/tests/test_extract.py b/tests/test_extract.py
index 70ad0cf..fd52534 100644
--- a/tests/test_extract.py
+++ b/tests/test_extract.py
@@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db):
assert fresh_db[""tree""].schema == (
'CREATE TABLE ""tree"" (\n'
"" [name] TEXT,\n""
- "" [common_name_latin_name_id] INTEGER,\n""
- "" FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n""
+ "" [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n""
"")""
)
assert (
```
Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in.
I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments:
```python
def transform(
self,
*,
# ...
# This one exists already:
drop_foreign_keys: Optional[Iterable] = None,
# These two are new. This one specifies keys to add:
add_foreign_keys: Optional[ForeignKeysType] = None,
# Or this one causes them all to be replaced with the new definitions:
foreign_keys: Optional[ForeignKeysType] = None,
```
There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder:
https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this:
```python
drop_foreign_keys: Optional[Iterable] = None,
```
Should be this:
```python
drop_foreign_keys: Optional[Iterable[str]] = None,
```
Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation:
https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149
At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`.
Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`:
https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others:
- https://github.com/simonw/llm/issues/60
- https://github.com/simonw/llm/issues/116
- https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work.
It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter:
https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872
Improving this code to support adding foreign keys as well would be pretty simple.
And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,