{"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710930934", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710930934, "node_id": "IC_kwDOCGYnMM5l-rv2", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:47:57Z", "updated_at": "2023-09-08T00:47:57Z", "author_association": "OWNER", "body": "That's odd, I wrote a test for this just now and it passes already:\r\n```python\r\ndef test_transform_preserves_rowids(fresh_db):\r\n    # Create a rowid table\r\n    fresh_db[\"places\"].insert_all(\r\n        (\r\n            {\"name\": \"Paris\", \"country\": \"France\"},\r\n            {\"name\": \"London\", \"country\": \"UK\"},\r\n            {\"name\": \"New York\", \"country\": \"USA\"},\r\n        ),\r\n    )\r\n    assert fresh_db[\"places\"].use_rowid\r\n    previous_rows = list(\r\n        tuple(row) for row in fresh_db.execute(\"select rowid, name from places\")\r\n    )\r\n    # Transform it\r\n    fresh_db[\"places\"].transform(column_order=(\"country\", \"name\"))\r\n    # Should be the same\r\n    next_rows = list(\r\n        tuple(row) for row in fresh_db.execute(\"select rowid, name from places\")\r\n    )\r\n    assert previous_rows == next_rows\r\n```\r\nSo maybe I'm wrong about the cause of that bug?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710931605", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710931605, "node_id": "IC_kwDOCGYnMM5l-r6V", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:49:02Z", "updated_at": "2023-09-08T00:49:02Z", "author_association": "OWNER", "body": "I tried bumping that up to 10,000 rows instead of just 3 but the test still passed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710933716", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710933716, "node_id": "IC_kwDOCGYnMM5l-sbU", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:52:42Z", "updated_at": "2023-09-08T00:52:42Z", "author_association": "OWNER", "body": "I just noticed that the table where I encountered this bug wasn't actually a `rowid` table after all - it had an `id` column that was a text primary key.\r\n\r\nThe reason the `rowid` was important is that's how the FTS mechanism in Datasette relates FTS entries to their rows.\r\n\r\nBut I tried this test and it passed, too:\r\n```python\r\ndef test_transform_preserves_rowids(fresh_db):\r\n    fresh_db[\"places\"].insert_all(\r\n        [\r\n            {\"id\": \"1\", \"name\": \"Paris\", \"country\": \"France\"},\r\n            {\"id\": \"2\", \"name\": \"London\", \"country\": \"UK\"},\r\n            {\"id\": \"3\", \"name\": \"New York\", \"country\": \"USA\"},\r\n        ],\r\n        pk=\"id\",\r\n    )\r\n    previous_rows = list(\r\n        tuple(row) for row in fresh_db.execute(\"select rowid, id, name from places\")\r\n    )\r\n    # Transform it\r\n    fresh_db[\"places\"].transform(column_order=(\"country\", \"name\"))\r\n    # Should be the same\r\n    next_rows = list(\r\n        tuple(row) for row in fresh_db.execute(\"select rowid, id, name from places\")\r\n    )\r\n    assert previous_rows == next_rows\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710934448", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710934448, "node_id": "IC_kwDOCGYnMM5l-smw", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:54:03Z", "updated_at": "2023-09-08T00:54:03Z", "author_association": "OWNER", "body": "Oh! Maybe the row ID preservation here is a coincidence because the tables are created from scratch and count 1, 2, 3.\r\n\r\nIf I delete a row from the table and then insert some more - breaking the `rowid` sequence - it might show the bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1710935270", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1710935270, "node_id": "IC_kwDOCGYnMM5l-szm", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-08T00:55:30Z", "updated_at": "2023-09-08T00:55:30Z", "author_association": "OWNER", "body": "Yes! That recreated the bug:\r\n```\r\n>       assert previous_rows == next_rows\r\nE       AssertionError: assert equals failed\r\nE         [                                                                [                                                               \r\nE           (1, '1', 'Paris'),                                               (1, '1', 'Paris'),                                            \r\nE           (3, '3', 'New York'),                                            (2, '3', 'New York'),                                         \r\nE           (4, '4', 'London'),                                              (3, '4', 'London'),                                           \r\nE         ]                                                       ...\r\nE         \r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/592#issuecomment-1712895580", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/592", "id": 1712895580, "node_id": "IC_kwDOCGYnMM5mGLZc", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-09-10T17:46:41Z", "updated_at": "2023-09-10T17:46:41Z", "author_association": "OWNER", "body": "In working on this I learned that `rowid` values in SQLite are way less stable than I had thought - in particular, they are often entirely rewritten on a `VACUUM`:\r\n\r\nhttps://www.sqlite.org/lang_vacuum.html#how_vacuum_works\r\n\r\n> The VACUUM command may change the [ROWIDs](https://www.sqlite.org/lang_createtable.html#rowid) of entries in any tables that do not have an explicit [INTEGER PRIMARY KEY](https://www.sqlite.org/lang_createtable.html#rowid).\r\n\r\nSo this fix wasn't as valuable as I thought. I need to move away from ever assuming that a `rowid` is a useful foreign key for anything.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1886771493, "label": "`table.transform()` should preserve `rowid` values"}, "performed_via_github_app": null}