{"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112298", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683112298, "node_id": "IC_kwDOCGYnMM5kUkFq", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-17T23:33:14Z", "updated_at": "2023-08-17T23:33:14Z", "author_association": "OWNER", "body": "Just one failing test left:\r\n\r\n```\r\n # Soundness check foreign_keys point to existing tables\r\n for fk in foreign_keys:\r\n if fk.other_table == name and columns.get(fk.other_column):\r\n continue\r\n if not any(\r\n c for c in self[fk.other_table].columns if c.name == fk.other_column\r\n ):\r\n> raise AlterError(\r\n \"No such column: {}.{}\".format(fk.other_table, fk.other_column)\r\n )\r\nE sqlite_utils.db.AlterError: No such column: breeds.rowid\r\n\r\nsqlite_utils/db.py:882: AlterError\r\n==== short test summary info ====\r\nFAILED tests/test_create.py::test_add_column_foreign_key - sqlite_utils.db.AlterError: No such column: breeds.rowid\r\n==== 1 failed, 378 deselected in 0.49s ====\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683112857", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683112857, "node_id": "IC_kwDOCGYnMM5kUkOZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-17T23:33:58Z", "updated_at": "2023-08-17T23:33:58Z", "author_association": "OWNER", "body": "Full test:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/tests/test_create.py#L468-L484", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683114719", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683114719, "node_id": "IC_kwDOCGYnMM5kUkrf", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-17T23:36:02Z", "updated_at": "2023-08-17T23:36:02Z", "author_association": "OWNER", "body": "Just these three lines recreate the problem:\r\n```python\r\nfrom sqlite_utils import Database\r\n\r\nfresh_db = Database(memory=True)\r\n\r\nfresh_db.create_table(\"dogs\", {\"name\": str})\r\nfresh_db.create_table(\"breeds\", {\"name\": str})\r\nfresh_db[\"dogs\"].add_column(\"breed_id\", fk=\"breeds\")\r\n```\r\nTraceback:\r\n```\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2170, in add_column\r\n self.add_foreign_key(col_name, fk, fk_col)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 2273, in add_foreign_key\r\n self.db.add_foreign_keys([(self.name, column, other_table, other_column)])\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 1169, in add_foreign_keys\r\n self[table].transform(add_foreign_keys=fks)\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 1728, in transform\r\n sqls = self.transform_sql(\r\n ^^^^^^^^^^^^^^^^^^^\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 1896, in transform_sql\r\n self.db.create_table_sql(\r\n File \"/Users/simon/Dropbox/Development/sqlite-utils/sqlite_utils/db.py\", line 882, in create_table_sql\r\n raise AlterError(\r\nsqlite_utils.db.AlterError: No such column: breeds.rowid\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683118376", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683118376, "node_id": "IC_kwDOCGYnMM5kUlko", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-17T23:41:10Z", "updated_at": "2023-08-17T23:41:10Z", "author_association": "OWNER", "body": "The problem here is that the table created by this line:\r\n```python\r\nfresh_db.create_table(\"breeds\", {\"name\": str})\r\n```\r\nHas this schema:\r\n```sql\r\nCREATE TABLE [breeds] (\r\n [name] TEXT\r\n);\r\n```\r\nSQLite creates an invisible `rowid` column for it automatically.\r\n\r\nOn the `main` branch with the old implementation that table ends up looking like this after the foreign key has been added:\r\n```sql\r\n(Pdb) print(fresh_db.schema)\r\nCREATE TABLE [dogs] (\r\n [name] TEXT\r\n, [breed_id] INTEGER,\r\n FOREIGN KEY([breed_id]) REFERENCES [breeds]([rowid])\r\n);\r\nCREATE TABLE [breeds] (\r\n [name] TEXT\r\n);\r\n```\r\nBut I think this validation check is failing now: https://github.com/simonw/sqlite-utils/blob/842b61321fc6a9f0bdb913ab138e39d71bf42e00/sqlite_utils/db.py#L875-L884\r\n\r\nHere's what the debugger reveals about this code:\r\n```python\r\n for fk in foreign_keys:\r\n if fk.other_table == name and columns.get(fk.other_column):\r\n continue\r\n if not any(\r\n c for c in self[fk.other_table].columns if c.name == fk.other_column\r\n ):\r\n raise AlterError(\r\n \"No such column: {}.{}\".format(fk.other_table, fk.other_column)\r\n )\r\n```\r\n```\r\n(Pdb) fk\r\nForeignKey(table='dogs', column='breed_id', other_table='breeds', other_column='rowid')\r\n(Pdb) self[fk.other_table].columns\r\n[Column(cid=0, name='name', type='TEXT', notnull=0, default_value=None, is_pk=0)]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122490", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683122490, "node_id": "IC_kwDOCGYnMM5kUmk6", "user": {"value": 22429695, "label": "codecov[bot]"}, "created_at": "2023-08-17T23:45:44Z", "updated_at": "2023-08-18T00:46:07Z", "author_association": "NONE", "body": "## [Codecov](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report\nPatch coverage: **`92.85%`** and project coverage change: **`-0.07%`** :warning:\n> Comparison is base [(`1dc6b5a`)](https://app.codecov.io/gh/simonw/sqlite-utils/commit/1dc6b5aa644a92d3654f7068110ed7930989ce71?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.82% compared to head [(`2915050`)](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) 95.76%.\n> Report is 1 commits behind head on main.\n\n
Additional details and impacted files\n\n\n```diff\n@@ Coverage Diff @@\n## main #584 +/- ##\n==========================================\n- Coverage 95.82% 95.76% -0.07% \n==========================================\n Files 8 8 \n Lines 2829 2834 +5 \n==========================================\n+ Hits 2711 2714 +3 \n- Misses 118 120 +2 \n```\n\n\n| [Files Changed](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage \u0394 | |\n|---|---|---|\n| [sqlite\\_utils/db.py](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-c3FsaXRlX3V0aWxzL2RiLnB5) | `97.22% <92.85%> (-0.15%)` | :arrow_down: |\n\n\n
\n\n[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/simonw/sqlite-utils/pull/584?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). \n:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison).\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683122767", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683122767, "node_id": "IC_kwDOCGYnMM5kUmpP", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-17T23:46:09Z", "updated_at": "2023-08-17T23:46:09Z", "author_association": "OWNER", "body": "Oops, `mypy` failures:\r\n\r\n```\r\nsqlite_utils/db.py:781: error: Incompatible types in assignment (expression has type \"Tuple[Any, ...]\", variable has type \"Union[str, ForeignKey, Tuple[str, str], Tuple[str, str, str], Tuple[str, str, str, str]]\") [assignment]\r\nsqlite_utils/db.py:1164: error: Need type annotation for \"by_table\" (hint: \"by_table: Dict[, ] = ...\") [var-annotated]\r\nsqlite_utils/db.py:1169: error: Item \"View\" of \"Union[Table, View]\" has no attribute \"transform\" [union-attr]\r\nsqlite_utils/db.py:1813: error: Argument 1 to \"append\" of \"list\" has incompatible type \"ForeignKey\"; expected [arg-type]\r\nsqlite_utils/db.py:1824: error: Argument 1 to \"append\" of \"list\" has incompatible type \"ForeignKey\"; expected [arg-type]\r\nFound 5 errors in 1 file (checked 56 source files)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683137259", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683137259, "node_id": "IC_kwDOCGYnMM5kUqLr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-18T00:06:59Z", "updated_at": "2023-08-18T00:06:59Z", "author_association": "OWNER", "body": "The docs still describe the old trick, I need to update that: https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-foreign-key-constraints", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683138953", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683138953, "node_id": "IC_kwDOCGYnMM5kUqmJ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-18T00:09:20Z", "updated_at": "2023-08-18T00:09:20Z", "author_association": "OWNER", "body": "Weird, I'm getting a `flake8` problem in CI which doesn't occur on my laptop:\r\n```\r\n./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96\r\n./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683139304", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683139304, "node_id": "IC_kwDOCGYnMM5kUqro", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-18T00:09:56Z", "updated_at": "2023-08-18T00:09:56Z", "author_association": "OWNER", "body": "Upgrading `flake8` locally replicated the error:\r\n```\r\npip install -U flake8\r\nflake8\r\n```\r\n```\r\n./tests/test_recipes.py:99:9: F811 redefinition of unused 'fn' from line 96\r\n./tests/test_recipes.py:127:9: F811 redefinition of unused 'fn' from line 124\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683143723", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683143723, "node_id": "IC_kwDOCGYnMM5kUrwr", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-18T00:14:52Z", "updated_at": "2023-08-18T00:14:52Z", "author_association": "OWNER", "body": "Another docs update: this bit in here https://sqlite-utils.datasette.io/en/3.34/python-api.html#adding-multiple-foreign-key-constraints-at-once talks about how `.add_foreign_keys()` is a performance optimization to avoid having to run VACUUM a bunch of separate times:\r\n\r\n> The final step in adding a new foreign key to a SQLite database is to run `VACUUM`, to ensure the new foreign key is available in future introspection queries.\r\n> \r\n> `VACUUM` against a large (multi-GB) database can take several minutes or longer. If you are adding multiple foreign keys using `table.add_foreign_key(...)` these can quickly add up.\r\n> \r\n> Instead, you can use `db.add_foreign_keys(...)` to add multiple foreign keys within a single transaction. This method takes a list of four-tuples, each one specifying a `table`, `column`, `other_table` and `other_column`.\r\n\r\nThat doesn't apply any more - the new mechanism using `.transform()` works completely differently, so this issue around running VACUUM no longer applies.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145110", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683145110, "node_id": "IC_kwDOCGYnMM5kUsGW", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-18T00:16:28Z", "updated_at": "2023-08-18T00:16:48Z", "author_association": "OWNER", "body": "One last piece of documentation: need to document the new option to `table.transform()` and `table.transform_sql()`:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/0771ac61fe5c2aca74075b20b1a99b9bd4c65661/sqlite_utils/db.py#L1706-L1708\r\n\r\nI should write tests for them too.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683145819", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683145819, "node_id": "IC_kwDOCGYnMM5kUsRb", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-18T00:17:26Z", "updated_at": "2023-08-18T00:17:26Z", "author_association": "OWNER", "body": "Updated documentation: https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#adding-foreign-key-constraints", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/584#issuecomment-1683164661", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/584", "id": 1683164661, "node_id": "IC_kwDOCGYnMM5kUw31", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-08-18T00:45:53Z", "updated_at": "2023-08-18T00:45:53Z", "author_association": "OWNER", "body": "More updated documentation:\r\n- https://sqlite-utils--584.org.readthedocs.build/en/584/reference.html#sqlite_utils.db.Table.transform\r\n- https://sqlite-utils--584.org.readthedocs.build/en/584/python-api.html#python-api-transform-add-foreign-key-constraints", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1855838223, "label": ".transform() instead of modifying sqlite_master for add_foreign_keys"}, "performed_via_github_app": null}