html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683122767,IC_kwDOCGYnMM5kUmpP,9599,simonw,2023-08-17T23:46:09Z,2023-08-17T23:46:09Z,OWNER,"Oops, `mypy` failures:
```
sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type ""Tuple[Any, ...]"", variable has type ""Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]"") [assignment]
sqlite_utils/db.py:1164: error: Need type annotation for ""by_table"" (hint: ""by_table: Dict[, ] = ..."") [var-annotated]
sqlite_utils/db.py:1169: error: Item ""View"" of ""Union[Table, View]"" has no attribute ""transform"" [union-attr]
sqlite_utils/db.py:1813: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type]
sqlite_utils/db.py:1824: error: Argument 1 to ""append"" of ""list"" has incompatible type ""ForeignKey""; expected [arg-type]
Found 5 errors in 1 file (checked 56 source files)
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683118376,IC_kwDOCGYnMM5kUlko,9599,simonw,2023-08-17T23:41:10Z,2023-08-17T23:41:10Z,OWNER,"The problem here is that the table created by this line:
```python
fresh_db.create_table(""breeds"", {""name"": str})
```
Has this schema:
```sql
CREATE TABLE [breeds] (
[name] TEXT
);
```
SQLite creates an invisible `rowid` column for it automatically.
On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added:
```sql
(Pdb) print(fresh_db.schema)
CREATE TABLE [dogs] (
[name] TEXT
, [breed_id] INTEGER,
FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid])
);
CREATE TABLE [breeds] (
[name] TEXT
);
```
But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884
Here's what the debugger reveals about this code:
```python
for fk in foreign_keys:
if fk.other_table == name and columns.get(fk.other_column):
continue
if not any(
c for c in self[fk.other_table].columns if c.name == fk.other_column
):
raise AlterError(
""No such column: {}.{}"".format(fk.other_table, fk.other_column)
)
```
```
(Pdb) fk
ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid')
(Pdb) self[fk.other_table].columns
[Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683114719,IC_kwDOCGYnMM5kUkrf,9599,simonw,2023-08-17T23:36:02Z,2023-08-17T23:36:02Z,OWNER,"Just these three lines recreate the problem:
```python
from sqlite_utils import Database
fresh_db = Database(memory=True)
fresh_db.create_table(""dogs"", {""name"": str})
fresh_db.create_table(""breeds"", {""name"": str})
fresh_db[""dogs""].add_column(""breed_id"", fk=""breeds"")
```
Traceback:
```
Traceback (most recent call last):
File """", line 1, in
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2170, in add_column
self.add_foreign_key(col_name, fk, fk_col)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 2273, in add_foreign_key
self.db.add_foreign_keys([(self.name, column, other_table, other_column)])
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1169, in add_foreign_keys
self[table].transform(add_foreign_keys=fks)
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1728, in transform
sqls = self.transform_sql(
^^^^^^^^^^^^^^^^^^^
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 1896, in transform_sql
self.db.create_table_sql(
File ""/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py"", line 882, in create_table_sql
raise AlterError(
sqlite_utils.db.AlterError: No such column: breeds.rowid
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112857,IC_kwDOCGYnMM5kUkOZ,9599,simonw,2023-08-17T23:33:58Z,2023-08-17T23:33:58Z,OWNER,"Full test:
https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298,https://api.github.com/repos/simonw/sqlite-utils/issues/584,1683112298,IC_kwDOCGYnMM5kUkFq,9599,simonw,2023-08-17T23:33:14Z,2023-08-17T23:33:14Z,OWNER,"Just one failing test left:
```
# Soundness check foreign_keys point to existing tables
for fk in foreign_keys:
if fk.other_table == name and columns.get(fk.other_column):
continue
if not any(
c for c in self[fk.other_table].columns if c.name == fk.other_column
):
> raise AlterError(
""No such column: {}.{}"".format(fk.other_table, fk.other_column)
)
E sqlite_utils.db.AlterError: No such column: breeds.rowid
sqlite_utils/db.py:882: AlterError
==== short test summary info ====
FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid
==== 1 failed, 378 deselected in 0.49s ====
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855838223,.transform() instead of modifying sqlite_master for add_foreign_keys,
https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636,https://api.github.com/repos/simonw/sqlite-utils/issues/583,1683110636,IC_kwDOCGYnMM5kUjrs,9599,simonw,2023-08-17T23:31:27Z,2023-08-17T23:31:27Z,OWNER,"Spotted this while working on:
- #577 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1855836914,Get rid of test.utils.collapse_whitespace,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683098094,IC_kwDOCGYnMM5kUgnu,9599,simonw,2023-08-17T23:15:36Z,2023-08-17T23:15:36Z,OWNER,"An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133
Needs updating like so:
```diff
diff --git a/tests/test_extract.py b/tests/test_extract.py
index 70ad0cf..fd52534 100644
--- a/tests/test_extract.py
+++ b/tests/test_extract.py
@@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db):
assert fresh_db[""tree""].schema == (
'CREATE TABLE ""tree"" (\n'
"" [name] TEXT,\n""
- "" [common_name_latin_name_id] INTEGER,\n""
- "" FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n""
+ "" [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n""
"")""
)
assert (
```
Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in.
I don't think this should count as a breaking change release though, but it's still worth noting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683076325,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683076325,IC_kwDOCGYnMM5kUbTl,9599,simonw,2023-08-17T22:48:36Z,2023-08-17T22:48:36Z,OWNER,"I'm inclined to just go with the `.transform()` method and not attempt to keep around the method that involves updating `sqlite_master` and then add code to detect if that's possible (or catch if it fails) and fall back on the other mechanism.
It would be nice to drop some code complexity, plus I don't yet have a way of running automated tests against Python + SQLite versions that exhibit the problem.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074009,IC_kwDOCGYnMM5kUavZ,9599,simonw,2023-08-17T22:45:29Z,2023-08-17T22:47:08Z,OWNER,"Actually I think `table.transform()` might get the following optional arguments:
```python
def transform(
self,
*,
# ...
# This one exists already:
drop_foreign_keys: Optional[Iterable] = None,
# These two are new. This one specifies keys to add:
add_foreign_keys: Optional[ForeignKeysType] = None,
# Or this one causes them all to be replaced with the new definitions:
foreign_keys: Optional[ForeignKeysType] = None,
```
There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074857,IC_kwDOCGYnMM5kUa8p,9599,simonw,2023-08-17T22:46:40Z,2023-08-17T22:46:40Z,OWNER,"As a reminder:
https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683074546,IC_kwDOCGYnMM5kUa3y,9599,simonw,2023-08-17T22:46:18Z,2023-08-17T22:46:18Z,OWNER,"Maybe this:
```python
drop_foreign_keys: Optional[Iterable] = None,
```
Should be this:
```python
drop_foreign_keys: Optional[Iterable[str]] = None,
```
Because it takes a list of column names that should have their foreign keys dropped.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683071519,IC_kwDOCGYnMM5kUaIf,9599,simonw,2023-08-17T22:42:28Z,2023-08-17T22:42:28Z,OWNER,"Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation:
https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149
At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`.
Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`:
https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683068505,IC_kwDOCGYnMM5kUZZZ,9599,simonw,2023-08-17T22:39:17Z,2023-08-17T22:39:38Z,OWNER,"This would help address these issues, among potentially many others:
- https://github.com/simonw/llm/issues/60
- https://github.com/simonw/llm/issues/116
- https://github.com/simonw/llm/issues/123","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934,https://api.github.com/repos/simonw/sqlite-utils/issues/577,1683066934,IC_kwDOCGYnMM5kUZA2,9599,simonw,2023-08-17T22:37:18Z,2023-08-17T22:37:18Z,OWNER,"I'm certain this could work.
It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter:
https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872
Improving this code to support adding foreign keys as well would be pretty simple.
And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",1817289521,Get `add_foreign_keys()` to work without modifying `sqlite_master`,
https://github.com/simonw/datasette/pull/2142#issuecomment-1682272492,https://api.github.com/repos/simonw/datasette/issues/2142,1682272492,IC_kwDOBm6k_c5kRXDs,22429695,codecov[bot],2023-08-17T13:18:08Z,2023-08-17T13:18:08Z,NONE,"## [Codecov](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report
Patch and project coverage have no change.
> Comparison is base [(`943df09`)](https://app.codecov.io/gh/simonw/datasette/commit/943df09dcca93c3b9861b8c96277a01320db8662?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06% compared to head [(`6b617f7`)](https://app.codecov.io/gh/simonw/datasette/pull/2142?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 92.06%.
Additional details and impacted files
```diff
@@ Coverage Diff @@
## main #2142 +/- ##
=======================================
Coverage 92.06% 92.06%
=======================================
Files 40 40
Lines 5937 5937
=======================================
Hits 5466 5466
Misses 471 471
```