github
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683122767 | IC_kwDOCGYnMM5kUmpP | 9599 | 2023-08-17T23:46:09Z | 2023-08-17T23:46:09Z | OWNER | Oops, `mypy` failures: ``` sqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type "Tuple[Any, ...]", variable has type "Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]") [assignment] sqlite_utils/db.py:1164: error: Need type annotation for "by_table" (hint: "by_table: Dict[<type>, <type>] = ...") [var-annotated] sqlite_utils/db.py:1169: error: Item "View" of "Union[Table, View]" has no attribute "transform" [union-attr] sqlite_utils/db.py:1813: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing> [arg-type] sqlite_utils/db.py:1824: error: Argument 1 to "append" of "list" has incompatible type "ForeignKey"; expected <nothing> [arg-type] Found 5 errors in 1 file (checked 56 source files) ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683118376 | IC_kwDOCGYnMM5kUlko | 9599 | 2023-08-17T23:41:10Z | 2023-08-17T23:41:10Z | OWNER | The problem here is that the table created by this line: ```python fresh_db.create_table("breeds", {"name": str}) ``` Has this schema: ```sql CREATE TABLE [breeds] ( [name] TEXT ); ``` SQLite creates an invisible `rowid` column for it automatically. On the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added: ```sql (Pdb) print(fresh_db.schema) CREATE TABLE [dogs] ( [name] TEXT , [breed_id] INTEGER, FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid]) ); CREATE TABLE [breeds] ( [name] TEXT ); ``` But I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884 Here's what the debugger reveals about this code: ```python for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): raise AlterError( "No such column: {}.{}".format(fk.other_table, fk.other_column) ) ``` ``` (Pdb) fk ForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid') (Pdb) self[fk.other_table].columns [Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)] ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683114719 | IC_kwDOCGYnMM5kUkrf | 9599 | 2023-08-17T23:36:02Z | 2023-08-17T23:36:02Z | OWNER | Just these three lines recreate the problem: ```python from sqlite_utils import Database fresh_db = Database(memory=True) fresh_db.create_table("dogs", {"name": str}) fresh_db.create_table("breeds", {"name": str}) fresh_db["dogs"].add_column("breed_id", fk="breeds") ``` Traceback: ``` Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2170, in add_column self.add_foreign_key(col_name, fk, fk_col) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 2273, in add_foreign_key self.db.add_foreign_keys([(self.name, column, other_table, other_column)]) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1169, in add_foreign_keys self[table].transform(add_foreign_keys=fks) File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1728, in transform sqls = self.transform_sql( ^^^^^^^^^^^^^^^^^^^ File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 1896, in transform_sql self.db.create_table_sql( File "/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py", line 882, in create_table_sql raise AlterError( sqlite_utils.db.AlterError: No such column: breeds.rowid ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683112857 | IC_kwDOCGYnMM5kUkOZ | 9599 | 2023-08-17T23:33:58Z | 2023-08-17T23:33:58Z | OWNER | Full test: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298 | https://api.github.com/repos/simonw/sqlite-utils/issues/584 | 1683112298 | IC_kwDOCGYnMM5kUkFq | 9599 | 2023-08-17T23:33:14Z | 2023-08-17T23:33:14Z | OWNER | Just one failing test left: ``` # Soundness check foreign_keys point to existing tables for fk in foreign_keys: if fk.other_table == name and columns.get(fk.other_column): continue if not any( c for c in self[fk.other_table].columns if c.name == fk.other_column ): > raise AlterError( "No such column: {}.{}".format(fk.other_table, fk.other_column) ) E sqlite_utils.db.AlterError: No such column: breeds.rowid sqlite_utils/db.py:882: AlterError ==== short test summary info ==== FAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid ==== 1 failed, 378 deselected in 0.49s ==== ``` | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855838223 | |
https://github.com/simonw/sqlite-utils/issues/583#issuecomment-1683110636 | https://api.github.com/repos/simonw/sqlite-utils/issues/583 | 1683110636 | IC_kwDOCGYnMM5kUjrs | 9599 | 2023-08-17T23:31:27Z | 2023-08-17T23:31:27Z | OWNER | Spotted this while working on: - #577 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1855836914 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683098094 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683098094 | IC_kwDOCGYnMM5kUgnu | 9599 | 2023-08-17T23:15:36Z | 2023-08-17T23:15:36Z | OWNER | An interesting side-effect of this change is that it does result in a slightly different schema - e.g. this test: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/tests/test_extract.py#L118-L133 Needs updating like so: ```diff diff --git a/tests/test_extract.py b/tests/test_extract.py index 70ad0cf..fd52534 100644 --- a/tests/test_extract.py +++ b/tests/test_extract.py @@ -127,8 +127,7 @@ def test_extract_rowid_table(fresh_db): assert fresh_db["tree"].schema == ( 'CREATE TABLE "tree" (\n' " [name] TEXT,\n" - " [common_name_latin_name_id] INTEGER,\n" - " FOREIGN KEY([common_name_latin_name_id]) REFERENCES [common_name_latin_name]([id])\n" + " [common_name_latin_name_id] INTEGER REFERENCES [common_name_latin_name]([id])\n" ")" ) assert ( ``` Unfortunately this means it may break other test suites that depend on `sqlite-utils` that have schema tests like this baked in. I don't think this should count as a breaking change release though, but it's still worth noting. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074009 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683074009 | IC_kwDOCGYnMM5kUavZ | 9599 | 2023-08-17T22:45:29Z | 2023-08-17T22:47:08Z | OWNER | Actually I think `table.transform()` might get the following optional arguments: ```python def transform( self, *, # ... # This one exists already: drop_foreign_keys: Optional[Iterable] = None, # These two are new. This one specifies keys to add: add_foreign_keys: Optional[ForeignKeysType] = None, # Or this one causes them all to be replaced with the new definitions: foreign_keys: Optional[ForeignKeysType] = None, ``` There should be validation that forbids you from using `foreign_keys=` at the same time as either `drop_foreign_keys=` or `add_foreign_keys=` because the point of `foreign_keys=` is to define the keys for the new table all in one go. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074857 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683074857 | IC_kwDOCGYnMM5kUa8p | 9599 | 2023-08-17T22:46:40Z | 2023-08-17T22:46:40Z | OWNER | As a reminder: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L159-L165 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683074546 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683074546 | IC_kwDOCGYnMM5kUa3y | 9599 | 2023-08-17T22:46:18Z | 2023-08-17T22:46:18Z | OWNER | Maybe this: ```python drop_foreign_keys: Optional[Iterable] = None, ``` Should be this: ```python drop_foreign_keys: Optional[Iterable[str]] = None, ``` Because it takes a list of column names that should have their foreign keys dropped. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683071519 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683071519 | IC_kwDOCGYnMM5kUaIf | 9599 | 2023-08-17T22:42:28Z | 2023-08-17T22:42:28Z | OWNER | Looking at the whole of the `.add_foreign_keys()` method, the first section of it can remain unchanged - it's just a bunch of validation: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1106-L1149 At that point we have `foreign_keys_to_create` as the ones that are new, but we should instead try to build up a `foreign_keys` which is both new and old, ready to be passed to `.transform()`. Here's the rest of that function, which will be replaced by a called to `.transform(foreign_keys=foreign_keys)`: https://github.com/simonw/sqlite-utils/blob/13ebcc575d2547c45e8d31288b71a3242c16b886/sqlite_utils/db.py#L1151-L1177 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683068505 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683068505 | IC_kwDOCGYnMM5kUZZZ | 9599 | 2023-08-17T22:39:17Z | 2023-08-17T22:39:38Z | OWNER | This would help address these issues, among potentially many others: - https://github.com/simonw/llm/issues/60 - https://github.com/simonw/llm/issues/116 - https://github.com/simonw/llm/issues/123 | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 | |
https://github.com/simonw/sqlite-utils/issues/577#issuecomment-1683066934 | https://api.github.com/repos/simonw/sqlite-utils/issues/577 | 1683066934 | IC_kwDOCGYnMM5kUZA2 | 9599 | 2023-08-17T22:37:18Z | 2023-08-17T22:37:18Z | OWNER | I'm certain this could work. It turns out the `.transform()` method already has code that creates the new table with a copy of foreign keys from the old one - dropping any foreign keys that were specified in the `drop_foreign_keys=` parameter: https://github.com/simonw/sqlite-utils/blob/1dc6b5aa644a92d3654f7068110ed7930989ce71/sqlite_utils/db.py#L1850-L1872 Improving this code to support adding foreign keys as well would be pretty simple. And then the `.add_foreign_keys()` and `.add_foreign_key()` methods could be updated to use `.transform(...)` under the hood instead. | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
1817289521 |