{"id": 1720096994, "node_id": "I_kwDOCGYnMM5mhpji", "number": 554, "title": "`IndexError` when doing `.insert(..., pk='id')` after `insert_all`", "user": {"value": 1231935, "label": "xavdid"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-05-22T17:13:02Z", "updated_at": "2023-05-22T17:18:33Z", "closed_at": null, "author_association": "NONE", "pull_request": null, "body": "I believe this is related to https://github.com/simonw/sqlite-utils/issues/98.\r\n\r\nWhen `pk` is specified by table A's `insert` call, it throws an index error if a different table has written a row with a higher rowid than exists in the first table. Here's a basic example:\r\n\r\n```py\r\nfrom sqlite_utils import Database\r\n\r\n\r\ndef test_pk_for_insert(fresh_db):\r\n    user = {\"id\": \"abc\", \"name\": \"david\"}\r\n\r\n    fresh_db[\"users\"].insert(user, pk=\"id\")\r\n\r\n    fresh_db[\"comments\"].insert_all(\r\n        [\r\n            {\"id\": \"def\", \"text\": \"ok\"},\r\n            {\"id\": \"ghi\", \"text\": \"great\"},\r\n        ],\r\n    )\r\n\r\n    fresh_db[\"users\"].insert(\r\n        user,\r\n        ignore=True,\r\n        # BUG: when specifying pk on the second insert call \r\n        # db.py goes into a block it doesn't expect and we get the error\r\n        pk=\"id\",\r\n    )\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    db = Database(\"bug.db\")\r\n    if db[\"users\"].exists():\r\n        raise ValueError(\r\n            \"bug only shows on a new database - remove bug.db before running the script\"\r\n        )\r\n    test_pk_for_insert(db)\r\n```\r\n\r\nThe error is:\r\n\r\n```py\r\n  File \"/Users/david/projects/reddit-to-sqlite/.venv/lib/python3.11/site-packages/sqlite_utils/db.py\", line 2960, in insert_chunk\r\n    row = list(self.rows_where(\"rowid = ?\", [self.last_rowid]))[0]\r\n          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^\r\nIndexError: list index out of range\r\n```\r\n\r\nThe issue is in this block: \r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/2747257a3334d55e890b40ec58fada57ae8cfbfd/sqlite_utils/db.py#L2954-L2958\r\n\r\nrelevant locals are:\r\n\r\n- `pk`: `'id'`\r\n- `result.lastrowid`: `2`\r\n\r\nWhat's most interesting is the comment `# self.last_rowid will be 0 if a \"INSERT OR IGNORE\" happened`, which doesn't seem to be the case here. ", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/554/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": null}
{"id": 1695428235, "node_id": "I_kwDOCGYnMM5lDi6L", "number": 538, "title": "`table.upsert_all` fails to write rows when `not_null` is present", "user": {"value": 1231935, "label": "xavdid"}, "state": "closed", "locked": 0, "assignee": null, "milestone": null, "comments": 9, "created_at": "2023-05-04T07:30:38Z", "updated_at": "2023-05-08T20:06:35Z", "closed_at": "2023-05-08T19:27:02Z", "author_association": "NONE", "pull_request": null, "body": "I found an odd bug today, where calls to `table.upsert_all` don't write rows if you include the `not_null` kwarg.\r\n\r\n## Repro Example\r\n\r\n```py\r\nfrom sqlite_utils import Database\r\n\r\ndb = Database(\"upsert-test.db\")\r\n\r\ndb[\"comments\"].upsert_all(\r\n    [{\"id\": 1, \"name\": \"david\"}],\r\n    pk=\"id\",\r\n    not_null=[\"name\"],\r\n)\r\n\r\nassert list(db[\"comments\"].rows) # err!\r\n```\r\n\r\nThe schema is correctly created:\r\n\r\n```sql\r\nCREATE TABLE [comments] (\r\n   [id] INTEGER PRIMARY KEY,\r\n   [name] TEXT NOT NULL\r\n)\r\n```\r\n\r\nBut no rows are created. Removing either the `not_null` kwargs works as expected, as does an `insert_all` call.\r\n\r\n## Version Info\r\n\r\n- Python: `3.11.0`\r\n- sqlite-utils: `3.30`\r\n- sqlite: `3.39.5 2022-10-14`", "repo": {"value": 140912432, "label": "sqlite-utils"}, "type": "issue", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/simonw/sqlite-utils/issues/538/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": null, "state_reason": "completed"}
{"id": 1641117021, "node_id": "PR_kwDODtX3eM5M66op", "number": 6, "title": "Add permalink virtual field to items table", "user": {"value": 1231935, "label": "xavdid"}, "state": "open", "locked": 0, "assignee": null, "milestone": null, "comments": 1, "created_at": "2023-03-26T22:22:38Z", "updated_at": "2023-03-29T18:38:52Z", "closed_at": null, "author_association": "FIRST_TIME_CONTRIBUTOR", "pull_request": "dogsheep/hacker-news-to-sqlite/pulls/6", "body": "I added a virtual column (no storage overhead) to the output that easily links back to the source. It works nicely out of the box with datasette:\r\n\r\n![](https://cdn.zappy.app/faf43661d539ee0fee02c0421de22d65.png)\r\n\r\nI got bit a bit by https://github.com/simonw/sqlite-utils/issues/411, so I went with a manual `table_xinfo` and creating the table via execute. Happy to adjust if that issue moves, but this seems like it works.\r\n\r\nI also added my best-guess instructions for local development on this package. I'm shooting in the dark, so feel free to replace with how you work on it locally.", "repo": {"value": 248903544, "label": "hacker-news-to-sqlite"}, "type": "pull", "active_lock_reason": null, "performed_via_github_app": null, "reactions": "{\"url\": \"https://api.github.com/repos/dogsheep/hacker-news-to-sqlite/issues/6/reactions\", \"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "draft": 0, "state_reason": null}