{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538793817", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538793817, "node_id": "IC_kwDOCGYnMM5buCFZ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T17:55:10Z", "updated_at": "2023-05-08T17:55:10Z", "author_association": "OWNER", "body": "Confirmed - I added this test and it fails:\r\n```python\r\ndef test_upsert_all_not_null(fresh_db):\r\n    # https://github.com/simonw/sqlite-utils/issues/538\r\n    fresh_db[\"comments\"].upsert_all(\r\n        [{\"id\": 1, \"name\": \"Cleo\"}],\r\n        pk=\"id\",\r\n        not_null=[\"name\"],\r\n    )\r\n    assert list(fresh_db[\"comments\"].rows) == [{\"id\": 1, \"name\": \"Cleo\"}]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538801855", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538801855, "node_id": "IC_kwDOCGYnMM5buEC_", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T18:00:17Z", "updated_at": "2023-05-08T18:00:17Z", "author_association": "OWNER", "body": "From time in the debugger, after creating the table it ends up doing this:\r\n```\r\n(Pdb) queries_and_params\r\n[\r\n  ('INSERT OR IGNORE INTO [comments]([id]) VALUES(?);', [1]),\r\n  ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538887361", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538887361, "node_id": "IC_kwDOCGYnMM5buY7B", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:01:20Z", "updated_at": "2023-05-08T19:01:20Z", "author_association": "OWNER", "body": "Here's the problem:\r\n```python\r\nimport sqlite3\r\ndb = sqlite3.connect(\":memory:\")\r\ndb.execute('create table foo (id integer primary key, name not null)')\r\ndb.execute('insert into foo (id) values (1)')\r\n```\r\nProduces:\r\n```\r\nIntegrityError: NOT NULL constraint failed: foo.name\r\n```\r\nBut this:\r\n```python\r\ndb.execute('insert or ignore into foo (id) values (1)')\r\n```\r\nCompletes without an exception.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538889482", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538889482, "node_id": "IC_kwDOCGYnMM5buZcK", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:02:38Z", "updated_at": "2023-05-08T19:02:38Z", "author_association": "OWNER", "body": "Here's the code at fault:\r\nhttps://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2774-L2788", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538893329", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538893329, "node_id": "IC_kwDOCGYnMM5buaYR", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:04:47Z", "updated_at": "2023-05-08T19:04:47Z", "author_association": "OWNER", "body": "This feels like a fundamental flaw in the way upserts are implemented by `sqlite-utils`.\r\n\r\nOne fix would be to switch to using the `UPSERT` feature in SQLite: https://www.sqlite.org/lang_UPSERT.html\r\n\r\nBut...\r\n\r\n> UPSERT syntax was added to SQLite with version 3.24.0 (2018-06-04).\r\n\r\nI still want to support SQLite versions earlier than that.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538903556", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538903556, "node_id": "IC_kwDOCGYnMM5buc4E", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:11:24Z", "updated_at": "2023-05-08T19:13:23Z", "author_association": "OWNER", "body": "I could detect if this happens using `cursor.rowcount` - not sure how I would recover from it though.\r\n\r\nThis would also require some major re-engineering, since currently it all works by generating a list of SQL queries in advance and applying them inside a loop in `.insert_chunk()`:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/80763edaa2bdaf1113717378b8d62075c4dcbcfb/sqlite_utils/db.py#L2839-L2878\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538910894", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538910894, "node_id": "IC_kwDOCGYnMM5buequ", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:16:52Z", "updated_at": "2023-05-08T19:17:00Z", "author_association": "OWNER", "body": "How about if I had logic which checked that all not-null columns were provided in the call to `upsert_all()` - and if they were, modified the `INSERT OR IGNORE INTO` to include a placeholder value for those columns that would then be fixed by the later `UPDATE`?\r\n\r\nSomething like this:\r\n\r\n```python\r\n[\r\n  ('INSERT OR IGNORE INTO [comments]([id], name) VALUES(?, ?);', [1, '']),\r\n  ('UPDATE [comments] SET [name] = ? WHERE [id] = ?', ['Cleo', 1])\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538921774", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538921774, "node_id": "IC_kwDOCGYnMM5buhUu", "user": {"value": 9599, "label": "simonw"}, "created_at": "2023-05-08T19:24:41Z", "updated_at": "2023-05-08T19:24:41Z", "author_association": "OWNER", "body": "That fix seems to work!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}
{"html_url": "https://github.com/simonw/sqlite-utils/issues/538#issuecomment-1538975545", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/538", "id": 1538975545, "node_id": "IC_kwDOCGYnMM5buuc5", "user": {"value": 1231935, "label": "xavdid"}, "created_at": "2023-05-08T20:06:35Z", "updated_at": "2023-05-08T20:06:35Z", "author_association": "NONE", "body": "perfect, thank you!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 1695428235, "label": "`table.upsert_all` fails to write rows when `not_null` is present"}, "performed_via_github_app": null}